Exam Certified Data Engineer Professional topic 1 question 77 discussion

suretly E

Evolve Schema = mergeSchema option is needed ; Incrementally = checkpointing is needed; Real-Time = WriteStream with default trigger . The only option that catches all of these is E

write is not for spark streaming

Reference: https://docs.databricks.com/en/ingestion/auto-loader/schema.html writeStream: Ensures real-time streaming write capabilities, which is essential f or near real-time workloads. checkpointLocation: Necessary for fault tolerance and tracking progress. mergeSchema: Ensures automatic schema evolution, allowing new columns to be detected and added to the target table. Why Option 'C ' is incorrect? Uses write instead of writeStream, which is for batch processing, making it inappropriate for real-time streaming. Why Option 'B ' is incorrect? Although it includes checkpointLocation and mergeSchema, the addition of trigger(once=True) is not necessary in this context, and it is better suited for batch-like processing. Reference: https://docs.databricks.com/en/ingestion/auto-loader/schema.html

streamRead & StreamWrite shares the schema using checkpoint location so cloudFiles.schemaLocation needs to be same for checkpointLocation so that we dont need to specify it manually also mergeSchema True make sure if any new column detected , it will be added in the target table https://docs.databricks.com/en/ingestion/auto-loader/schema.html

https://notebooks.databricks.com/demos/auto-loader/01-Auto-loader-schema-evolution-Ingestion.html

This response correctly fills in the blank to meet the specified requirements of using Databricks Auto Loader for automatic schema detection and evolution in a near real-time streaming context.

Please refer: https://docs.databricks.com/en/ingestion/auto-loader/schema.html

It does not mention to write as stream, it mentions to write incrementally, so option C looks correct for me

Correct answer is E, it is a streaming write, and the default outputMode is Append (so if it's optional in this case)

there is a type in the statement. Is it schema or checkpoint ? Provided answer is not correct. It has to be a writestream, with mode append