Exam Professional Data Engineer topic 1 question 62 discussion

Option A: Set a single global window to capture all the data is not a good idea because it may not allow for late or out-of-order data to be processed. Option B: Set sliding windows to capture all the lagged data is not suitable for the case where you want to process the data over a predictable time period. Sliding windows are used when you want to process data over a period of time that is continuously moving forward, not a fixed period. Option D: Ensure every datasource type (stream or batch) has a timestamp, and use the timestamps to define the logic for lagged data is a good practice but not a complete solution, because it only ensures that data is ordered correctly, but it does not account for data that may be late.

The naming in Apache Beam is: Fixed, Sliding, Session In Dataflow it's: Tumbling, Hopping, Session. I was very confused at first too when I saw "hopping" in a question.

I highly doubt, DataFlow windowing is divided into three(3) types: 1. Fixed 2. Sliding 3. Session

with watermarks, when the late data arrives, it goes into its rightful window and gets in order

delete this answer. The answer belongs to another question

C is, I think, the correct answer: Watermark is different from late data. Watermark in implementation is a monotonically increasing timestamp. When Beam/Dataflow see a record with an event timestamp that is earlier than the watermark, the record is treated as late data. I’ll try to explain: Late data is inherent to Beam’s model for out-of-order processing. What does it mean for data to be late? The definition and its properties are intertwined with watermarks that track the progress of each computation across the event time domain. The simple intuition behind handling lateness is this: only late input should result in late data anywhere in the pipeline. So, is not easy to decide between C and D. If you ask me I’d say C since for D we ought to make some suppositions.