load data

Question 1

… (existing content unchanged) …

Question 2

How can you load data from existing Databricks tables in Lakeflow?

Answer 2

Use a CREATE OR REFRESH MATERIALIZED VIEW or define a DLT table that reads from the existing table. This supports further transformation within the pipeline.

Question 3

How do you load data from cloud object storage using Auto Loader in SQL?

Answer 3

Use the read_files function in a CREATE OR REFRESH STREAMING TABLE query, specifying the file path and format.

Question 4

How do you ingest data from Kafka into a streaming table?

Answer 4

Use read_kafka in a SQL CREATE OR REFRESH STREAMING TABLE statement, providing Kafka bootstrap servers and topic name.

Question 5

Can Lakeflow ingest data from systems like PostgreSQL?

Answer 5

Yes, using Python and Spark DataFrame readers (e.g., .format("postgresql") with connection options), you can read external database tables.

Question 6

How can you ingest small or static files directly into a materialized view?

Answer 6

Use read_files in a CREATE OR REFRESH MATERIALIZED VIEW SQL command to load the file contents.

Question 7

How do you ignore updates and deletes in a streaming table source?

Answer 7

Use the skipChangeCommits option in spark.readStream.option() to ignore change operations from the source table.

Question 8

How do you securely load data from Azure Data Lake using secrets?

Answer 8

Store credentials in Databricks Secrets and use them in the spark.hadoop config. Then define a DLT table using Auto Loader to read from the ADLS path.