Exam Certified Data Engineer Professional topic 1 question 25 discussion

Actual exam question from Databricks's Certified Data Engineer Professional

Question #: 25
Topic #: 1

[All Certified Data Engineer Professional Questions]

A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.
Which situation is causing increased duration of the overall job?

A. Task queueing resulting from improper thread pool assignment.
B. Spill resulting from attached volume storage being too small.
C. Network latency due to some cluster nodes being in different regions from the source data
D. Skew caused by more data being assigned to a subset of spark-partitions.
E. Credential validation errors while pulling data from an external system.

Show Suggested Answer

Suggested Answer: D 🗳️

by Eertyy at Sept. 21, 2023, 9:15 p.m.

Comments

Submit Cancel

KadELbied

2 months ago

Selected Answer: D

Suretly D

upvoted 1 times

...

arekm

6 months, 1 week ago

Selected Answer: D

D - other answers don't make sense. In particular C - all nodes of the cluster must be in the same region (at least on AWS and Azure; GCP - I don't know, but they have networks spanning regions, so maybe it is possible).

upvoted 2 times

...

benni_ale

7 months, 1 week ago

Selected Answer: D

D is correct

upvoted 1 times

...

AndreFR

10 months, 2 weeks ago

A excluded because task queueing does not increase the duration of a task B excluded, spill is writing to storage when a memory is insufficient (not storage insufficient) C excluded, region cannot have a 100 times impact on duration E excluded, no errors mentioned in question

upvoted 1 times

...

1 year, 9 months ago

D is the correct answer

upvoted 3 times

...