Exam Certified Data Engineer Professional topic 1 question 20 discussion

Actual exam question from Databricks's Certified Data Engineer Professional

Question #: 20
Topic #: 1

[All Certified Data Engineer Professional Questions]

A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams.
The proposed directory structure is displayed below:

Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?

A. No; Delta Lake manages streaming checkpoints in the transaction log.
B. Yes; both of the streams can share a single checkpoint directory.
C. No; only one stream can write to a Delta Lake table.
D. Yes; Delta Lake supports infinite concurrent writers.
E. No; each of the streams needs to have its own checkpoint directory.

Show Suggested Answer

Suggested Answer: E 🗳️

by Eertyy at Aug. 30, 2023, 1:04 p.m.

Comments

Submit Cancel

thxsgod

Highly Voted 1 year, 11 months ago

Selected Answer: E

Correct, E. Source: https://docs.databricks.com/en/optimizations/isolation-level.html#:~:text=If%20a%20streaming%20query%20using%20the%20same%20checkpoint%20location%20is%20started%20multiple%20times%20concurrently%20and%20tries%20to%20write%20to%20the%20Delta%20table%20at%20the%20same%20time.%20You%20should%20never%20have%20two%20streaming%20queries%20use%20the%20same%20checkpoint%20location%20and%20run%20at%20the%20same%20time.

upvoted 11 times

...

KadELbied

Most Recent 3 months ago

Selected Answer: E

Suretly E

upvoted 1 times

...

benni_ale

9 months, 3 weeks ago

Selected Answer: E

E is the correct

upvoted 1 times

...

imatheushenrique

1 year, 2 months ago

E. No; each of the streams needs to have its own checkpoint directory. The checkpoint directory is 1 to 1

upvoted 2 times

...

svik

1 year, 2 months ago

Selected Answer: B

It is not clear from the question that year_week=2020_01 and year_week=2020_02 are used by stream 1 and stream 2 respectively. If they use the common parent checkpoint directory with individual sub folders for checkpointing, that should work fine. In that case the answer should be B

upvoted 2 times