Exam DP-203 topic 1 question 76 discussion

Auto Loader provides a Structured Streaming source called cloudFiles. Plus, it supports schema drift. Hence, Auto Loader is the correct answer. https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/

Auto Loader

Auto Loader (option C) est la solution recommandée pour configurer storage1 comme source de streaming structurée dans Azure Databricks.

Auto Loader is correc

autoloader is correct-:Incremental processing: Auto Loader can automatically detect and incrementally process new files as they are uploaded to Azure Data Lake Storage Gen2. Minimize implementation and maintenance effort: Auto Loader is designed for simplicity, requiring minimal setup and automatically handling file management. It reduces operational overhead by automating many of the tasks required to manage a streaming source. Minimize the cost of processing millions of files: Auto Loader efficiently scales to handle millions of files and minimizes costs by using a directory listing mode or a more optimized file notification mode with Azure Event Grid. Support schema inference and schema drift: Auto Loader automatically infers schema and can handle schema drift, which allows it to dynamically adapt to changes in the file structure without requiring constant updates to the processing logic.

Auto Loader is correct

Reference: https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/#incremental-ingestion-using-auto-loader-with-delta-live-tables

Auto Loader seems more correct. Copy Into focuses on loading from storage to a Delta table

To configure storage1 as a structured streaming source that incrementally processes new files as they are uploaded minimizes implementation and maintenance effort, minimizes the cost of processing millions of files, and supports schema inference and schema drift, you should use Apache Spark FileStreamSource

Bing explains the following: The best option is C. Auto Loader. Auto Loader is a feature in Azure Databricks that uses a cloudFiles data source to incrementally and efficiently process new data files as they arrive in Azure Data Lake Storage Gen2. It supports schema inference and schema evolution (drift). It also minimizes implementation and maintenance effort, as it simplifies the ETL pipeline by reducing the complexity of identifying new files for processing. Other options do not meet the requirements because: A. COPY INTO: does not incrementally process new files as they are uploaded, which is one of your requirements. B. Azure Data Factory: does not natively support schema inference and schema drift. The incremental processing of new files would need to be manually implemented, which could increase implementation and maintenance effort. D. Apache Spark FileStreamSource: requires manual setup and does not natively support schema inference or schema drift. It also may not minimize the cost of processing millions of files as efficiently as Auto Loader.

Auto Loader

C is correct

To configure Azure Data Lake Storage Gen2 account (storage1) as a structured streaming source in Azure Databricks workspace, while meeting the given requirements, you should include the following in the recommendation: C. Auto Loader Auto Loader is a feature provided by Azure Databricks that automatically discovers and processes new files as they are uploaded to a specified directory in Azure Data Lake Storage Gen2. It provides an efficient and cost-effective way to incrementally process new files without the need for manual intervention. Auto Loader also supports schema inference and schema drift, allowing you to handle changes in the file schema over time. By using Auto Loader, you can minimize implementation and maintenance effort as it takes care of monitoring the storage directory for new files and processing them in an optimized manner. It also helps to minimize the cost of processing millions of files as it leverages the efficient processing capabilities of Databricks. Therefore, the correct answer is C. Auto Loader.

Auto Loader

D according to ChatGPT

Ans : B DF supports Schema Drift - https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift