Exam AWS Certified Data Engineer - Associate DEA-C01 topic 1 question 118 discussion

• B (checkpointing) prevents reprocessing beyond checkpoints but doesn’t stop duplicates created at ingestion. • C isn’t realistic during outages/retries. • D is unnecessary; switching to EMR doesn’t solve producer-side duplicates.

Amazon Kinesis Data Streams does not provide exactly-once delivery natively. It ensures at-least-once delivery, meaning that under certain conditions (e.g., network failures, retries), duplicate records can occur. To achieve exactly-once processing, deduplication must be handled at the application level.

A. Design the application so it can remove duplicates during processing by embedding a unique ID in each record at the source.

A. Design the application so it can remove duplicates during processing by embedding a unique ID in each record at the source. Explanation: Exactly-Once Delivery: Ensuring exactly-once delivery is a challenge in distributed systems, especially in the presence of network outages and retries. By embedding a unique ID in each record at the source, you can track and identify duplicate records during processing. This approach allows you to implement idempotent processing, where duplicate records can be detected and discarded, ensuring that each record is processed exactly once. De-duplication Logic: Implementing de-duplication logic based on unique IDs ensures that even if the same record is ingested multiple times due to retries or network issues, it will be processed only once by the downstream applications.

A. Design the application so it can remove duplicates during processing by embedding a unique ID in each record at the source. This approach ensures that even if a record is sent more than once due to network outages or other issues, it will only be processed once because the unique ID can be used to identify and remove any duplicates. This is a common pattern for achieving exactly-once processing semantics in distributed systems. The other options do not guarantee exactly-once delivery across the entire pipeline. Option B is partially correct but it only avoids duplicate processing within the Amazon Managed Service for Apache Flink, not across the entire pipeline. Option C is not always feasible because network issues and other factors can lead to events being ingested into Kinesis Data Streams multiple times. Option D involves changing the entire technology stack, which is not necessary to achieve the desired outcome and could introduce additional complexity and cost.