Exam Professional Data Engineer topic 1 question 87 discussion

Should be A: https://stackoverflow.com/questions/42918663/is-it-better-to-have-one-large-parquet-file-or-lots-of-smaller-parquet-files https://www.dremio.com/tuning-parquet/ C & D will improve performance but need to pay more $$

Answer should be D

C is correct. It cannot be D as increasing boot disk size does not impact shuffle performance much. We need local SSDs specifically attached for shuffle storage (temporary fast storage), not just a bigger persistent boot disk. C is the correct answer because SSD + HDFS shuffle layer = fastest for Spark shuffle-heavy jobs, while still using preemptibles and keeping costs down. The job as mentioned is shuffle-intensive. In Spark, shuffling (moving data between nodes) is heavily disk I/O intensive. Faster local storage (i.e., SSDs) can dramatically speed up shuffle operations compared to using standard HDDs. GCS is great for object storage, not shuffle storage.

C is the correct answer

Improves shuffle management by using HDFS instead of GCS. SSDs speed up access to temporary data. Compatible with Dataproc's preemptible cost model, without requiring more non-preemptible workers.

Preemptible Cost Considerations Using preemptibles (with 2 non-preemptible workers) is cost-effective, but shuffle operations still need fast local storage. SSDs improve reliability without increasing instance costs significantly

A. Not D because it doesn't make sens to move to SSD when cost-senstive

A is the answer

I think either A or C. The problem is occured by I/O performance. Option A is feasible, which reduces the number of files leading better parallel processing. Option C tries to handle I/O performance issue. Taking other factors like budget and no mention of HDD/SSD, option A is possible the correct answer.

There's no mention of a drive type used, only GCS. That means A is the only sensible option.

Question doesn't actually say they are using HDD in the scenario, for that reason I choose A

A We don't know if HDD was used, so we can know what to do about that, but we know that the parquet files are small and much, and we can act on that by increasing the sizes to have lesser number of it.

Should be A: https://stackoverflow.com/questions/42918663/is-it-better-to-have-one-large-parquet-file-or-lots-of-smaller-parquet-files

https://stackoverflow.com/questions/42918663/is-it-better-to-have-one-large-parquet-file-or-lots-of-smaller-parquet-files Cost effective is the key in the question.

Preemptible VMs can't be used for HDFS storage. As a default, preemptible VMs are created with a smaller boot disk size, and you might want to override this configuration if you are running shuffle-heavy workloads.

Should NOT be A as: 1. The file size is already at the optimal size 2. If the current file size works well in the current Hadoop, it is expected to have similar performance in Dataproc The only difference between the current and Dataproc is that Dataproc is using preemptible nodes. So yes, it may incur a bit more cost by using SSD but assuming using the preemptible already save most of it, so we want to save less to improve the performance

Cost sensitive is the keyword.