Exam Certified Data Engineer Professional topic 1 question 95 discussion

Only A seems logical

It's A, rows are deduplicated only in 2hrs window, therefore final table may eventually contain duplicates

- The code snippet uses **`withWatermark("time", "2 hours")`**, which means that Spark will only process records with a timestamp (`time` field) that falls within a 2-hour window from the most recent watermark. Any records arriving **more than 2 hours late** will be ignored and won't be written to the `orders` table. - The **`dropDuplicates(["customer_id", "order_id"])`** ensures that duplicate records (based on the composite key of `customer_id` and `order_id`) are removed **within the 2-hour window**. This guarantees no duplicates will be present in the `orders` table for data falling within the watermark range.

E. The orders table will not contain duplicates, but records arriving more than 2 hours late will be ignored and missing from the table. This matches Spark’s actual behavior with watermark + dropDuplicates: the stateful deduplication ensures no duplicate (customer_id, order_id) rows are ever written to the Delta table, and the 2-hour watermark means any record that comes in more than 2 hours behind the stream’s current event-time will be dropped (i.e. not processed/output)​

A is correct. The watermark ensures that data arriving up to 2h late will be deduplicated in the incoming batch.

A - since the same [customer_id, order_id] record can be emitted twice with the "time" value set apart by more than 2 hours. The target table will then contain duplicates (duplicates from the perspective of the composite key).

Per me è la cipolla

Seems that E should be the correct answer. As time is the time it was queued in the *source_system*. And withWatermark ignores records that have a "time" more than 2 hours old.

When watermarking is set to 2 hours, the system will wait for up to 2 hours for late data to arrive. Any data that arrives within this 2-hour window will be considered for processing and deduplication. However, data that arrives later than 2 hours after the event time will be considered too late and will be discarded. This ensures that the state store does not grow indefinitely, but it also means that any records arriving more than 2 hours late will not be included in the orders table.

Every Stream micro-batch is executed on all of the new data that arrived after the last run 2 hours ago by the .trigger(once=True) option. Deduplication is done for it based on the combined key fields values but all records older than 2 hours based on the 'time' field will be ignored thanks to the .withWatermark() option/function. So target table will have deduplicated data withOUT the late records arriving more than 2 hours later based on the 2 hours watermark buffer set for the readStream.

Data arrive outside of watermark will be dropped.

Watermark("time", "2 hours") --> does'nt let records arriving more than 2 hours late to be written dropDuplicates --> removes duplicate records from the records that are read

Should be E. Watermarking (withWatermark("time", "2 hours")): This sets a 2-hour watermark on the time column. The watermark specifies the event time threshold for data completeness, meaning that data older than 2 hours will be considered late and may be dropped. Deduplication (dropDuplicates(["customer_id", "order_id"])): This operation removes duplicates based on the composite key (customer_id and order_id). However, it only works within the window defined by the watermark.