Exam Certified Data Engineer Professional topic 1 question 75 discussion

Actual exam question from Databricks's Certified Data Engineer Professional

Question #: 75
Topic #: 1

[All Certified Data Engineer Professional Questions]

A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.

In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

A. Set the configuration delta.deduplicate = true.
B. VACUUM the Delta table after each batch completes.
C. Perform an insert-only merge with a matching condition on a unique key.
D. Perform a full outer join on a unique key and overwrite existing data.
E. Rely on Delta Lake schema enforcement to prevent duplicate records.

Show Suggested Answer

Suggested Answer: C 🗳️

by Crocjun at Oct. 22, 2023, 7:26 p.m.

Comments

Submit Cancel

aragorn_brego

Highly Voted 1 year, 8 months ago

Selected Answer: C

To handle deduplication against previously processed records in a Delta table, the MERGE INTO command can be used to perform an upsert operation. This means that if the incoming data has a record that matches an existing record based on a unique key, the MERGE INTO operation can update the existing record (if needed) or simply ignore the duplicate. If there is no match (i.e., the record is new), then the record will be inserted

upvoted 6 times

...