Exam Certified Data Engineer Professional topic 1 question 26 discussion

Option A, question is about maximum performance. Wide transformation will result in often expensive shuffle. With one executor this problem will be resolved. https://docs.databricks.com/en/clusters/cluster-config-best-practices.html#complex-batch-etl

How Option (C) Excels? ✅ More Executors (16 vs. 8 in Option B) → Faster parallel execution. ✅ Fewer Cores per Executor (10 vs. 20 in Option B) → Prevents CPU contention and scheduling delays. ✅ Better Memory Management (25GB vs. 50GB in Option B) → Reduces GC overhead. Final Verdict Option (C) is the "Best" configuration for handling a job with wide transformations.

See aswer from robson90

one worker no shuffle, if we target wide tr explicitly, if we go for balance than c

Suretly A

The question talks about higher performance for one large wide transformation. This needs fewer large VMs/Executor. Therefore, one needs to choose the largest possible option.

overly large executors are bad due to large Garbage Collection (GC) overhead and inefficient parallelism option C provides the best balance of parallelism, memory utilization and performance efficiency

A is correct only one VM per Job.

Number of workers Choosing the right number of workers requires some trials and iterations to figure out the compute and memory needs of a Spark job. Here are some guidelines to help you start: Never choose a single worker for a production job, as it will be the single point for failure Start with 2-4 workers for small workloads (for example, a job with no wide transformations like joins and aggregations) Start with 8-10 workers for medium to big workloads that involve wide transformations like joins and aggregations, then scale up if necessary https://www.databricks.com/discover/pages/optimize-data-workloads-guide#number-workers

Maximum performance - A guarantees no shuffles between nodes in the cluster. Only processes on one VM.

Answer B offers a good balance with 8 executors, providing a decent amount of memory and cores per executor, allowing for significant parallel processing. Option C increases the number of executors further but at the cost of reduced memory and cores per executor, which might not be as effective for wide transformations.

for wide transformations, leveraging multiple executors typically results in better performance, resource utilization, and fault tolerance.

B is a correct Answer based on https://www.databricks.com/discover/pages/optimize-data-workloads-guide#all-purpose

Besides that A & E do not provide enough parallelism & fault tolerance, I can't explain why, but the correct answer is B. I got the same question during the exam and got 100% at tooling with answer B. (B is the answer provided by other sites similar to examtopics) Choosing between B, C & D is tricky !

B "Number of workers Choosing the right number of workers requires some trials and iterations to figure out the compute and memory needs of a Spark job. Here are some guidelines to help you start: Never choose a single worker for a production job, as it will be the single point for failure Start with 2-4 workers for small workloads (for example, a job with no wide transformations like joins and aggregations) Start with 8-10 workers for medium to big workloads that involve wide transformations like joins and aggregations, then scale up if necessary"

Wide transformation falls under complex etl which means Option A is correct in the documentation didn't mention to do otherwise in this scenario.

A is correct