Exam Certified Data Engineer Professional topic 1 question 150 discussion

Correct Answer: B The Change Data Feed (CDF) feature in Delta Lake enables reading only the changes (inserts and updates) to a Delta table. This would allow the function to focus on new or modified data since the last trigger, making it ideal for processing only the new records that have not been processed yet. This directly meets the requirement for identifying and manipulating new records efficiently.

Looks like B is more suitable for batch processing with Delta (default for table) and CDF

any tables can become a stream in Databricks, therefore answer A. It cannot be answer B as the table named bronze does not have CDF enabled

The function new_records() is meant to retrieve new records that have not yet been processed in the pipeline. Delta Lake provides a feature called the Change Data Feed (CDF), which allows you to track changes (inserts, updates, and deletes) in a Delta table. By using .option("readChangeFeed", "true"), the function can read only the new changes from the Delta table named "bronze". This ensures that only unprocessed records are returned, aligning with the pipeline's requirement.

A - append only table works for streaming, B is missing the starting version, C - timestamp during the insertion will be different from the one during the next step, D - that was a "like" type query (or maybe a substring on the file name == the directory), then yes - actual file names only start with the pattern presented, but are longer.

Option A seems to be the correct answer. Option B seems not to be the right one, because is missing the version. Based on the documentation 'Change data feed also supports batch execution, which requires specifying a starting version'. https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed#batch Option D , in my opinion, is not correct, as the function definition above is not having any input parameter.

Correct answer is A as we have append-only mode writes which are ideal for simple Structured Streaming as a next step ;)

delta table returns new records in streaming read.

B. Set the skipChangeCommits flag to true on raw_iot Let's break down the requirements and explain why this is the best solution: Retain manually deleted or updated records in raw_iot: The skipChangeCommits flag, when set to true, tells Delta Live Tables (DLT) to ignore any manual changes (updates or deletes) made to the table outside of the pipeline. This means that even if records are manually deleted or updated in the raw_iot table, these changes won't be reflected in the table when the pipeline runs again. Recompute downstream bpm_stats table: By default, DLT will recompute downstream tables when their upstream dependencies change. Since bpm_stats is based on raw_iot, it will naturally be recomputed when the pipeline updates, without any special configuration. Why the other options are not correct: A. Setting pipelines.reset.allowed to false on raw_iot would prevent the table from being reset, but it wouldn't address the requirement to retain manually deleted or updated records.

You have to know the CDF's current version and last processed the version in order to get not processed records. B does not provide those versions. It will just return content from the bronze table with CDF turned on. D is only possible solution.

I did not test it. But i think D is wrong as it filtering agenst directory path using ==

Seems D