Exam DP-203 topic 2 question 34 discussion

The ABS-AQS source is deprecated. For new streams, we recommend using Auto Loader instead.

Why not partition by date? What does the auto loader have to do with streaming jobs?

Watermark. For those saying it can't be watermark please consider that watermark has different meaning based on the context. I.e. different concept when used in a streaming or loading context. Please see the below link for why watermark is used and how it is used for incremental loading. https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

Partitioning by DateTime is the most effective approach to minimize storage costs and speed up incremental load times.

B. Sink to Azure Queue storage: Queue storage is primarily for messaging and not ideal for storing large volumes of streaming data efficiently. C. Include a watermark column: While a watermark column is useful for processing streaming data, it does not directly address storage costs or load times. D. Use a JSON format for physical data storage: JSON can be easy to work with, but it tends to use more storage space compared to more efficient formats like Parquet, which can negatively impact storage costs.

Partitioning by DateTime fields helps in organizing the data efficiently, which can significantly reduce the time required for incremental loads. It allows you to quickly access and process only the relevant partitions, rather than scanning the entire dataset1. Including a watermark column (Option C) is also important for managing late-arriving data and ensuring that only the most recent data is processed. However, it doesn’t directly address storage costs or incremental load times2. Sinking to Azure Queue storage (Option B) is not suitable for this scenario as it is more appropriate for message queuing rather than persistent storage for large volumes of data3. Using a JSON format for physical data storage (Option D) is not recommended because JSON is not optimized for storage efficiency or query performance. Instead, using a columnar storage format like Parquet or Delta Lake would be more efficient4.

A watermark column is essential for implementing event time-based processing in streaming data scenarios. It helps track the progress of event ingestion and ensures that only the latest data is processed, thereby enabling efficient incremental loading.

Watermark column could reduce the storage

Well, Azure Queue Storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account. Queues are commonly used to create a backlog of work to process asynchronously, like in the Web-Queue-Worker architectural style. I believe the correct answer is A.

Incremental key and time stamp matching watermark behaviour https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

chat gpt o design an efficient Azure Databricks table for ingesting an average of 20 million streaming events per day while minimizing storage costs and incremental load times, you should consider the following: A. Partition by DateTime fields. Explanation: Partitioning by DateTime fields is a common practice for time-series data in Azure

should be B

option B

Sinking to Azure Queue storage is not necessary for persisting the events in the Azure Databricks table. Azure Queue storage is typically used for decoupling and asynchronous messaging scenarios and may not directly contribute to minimizing storage costs or incremental load times for the Databricks table.

Probably it is B: Partition by date&time is not the best, immagine events with each single partition because of (day, hour, minute, second) => the requirement is clear, minimiuze the space, etc.. You use Watermark when you need to reduce the amount of state data to improve latency during a long-running steaming operation. JSON I would exclude because how it is formulated. My answer is B, even if it's deprecated, it's clear that this question is an old one, but looking at the commnents, we can still get in the exam.

A. Partition by DateTime fields: Partitioning the table on frequently used columns such as DateTime fields can improve query performance and reduce incremental load times. Partitioning by DateTime can help to reduce the amount of data scanned during query execution and facilitate incremental loading.

is the question outdated?