Exam Certified Data Engineer Professional topic 1 question 70 discussion

This strategy aims to control the size of the output Parquet files without shuffling the data. The spark.sql.files.maxPartitionBytes parameter sets the maximum size of a partition that Spark will read. By setting it to 512 MB, you are aligning the read partition size with the desired output file size. Since the transformations are narrow (meaning they do not require shuffling), the number of partitions should roughly correspond to the number of output files when writing out to Parquet, assuming the data is evenly distributed and there is no data expansion during processing.

D is the only one that does the trick. Note, we can not do shuffling. Wrong answers: A: spark.sql.files.maxPartitionBytes is about reading, not writing.(The maximum number of bytes to pack into a single partition when reading files. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. ) B: spark.sql.adaptive.advisoryPartitionSizeInBytes takes effect while shuffling and sorting does not make sense (The advisory size in bytes of the shuffle partition during adaptive optimization (when spark.sql.adaptive.enabled is true). It takes effect when Spark coalesces small shuffle partitions or splits skewed shuffle partition.) C: Would work but spark.sql.adaptive.advisoryPartitionSizeInBytes would need shuffling. E. spark.sql.shuffle.partitions (Configures the number of partitions to use when shuffling data for joins or aggregations.) is not about writing.

This is still only kinda correct. A is wrong because there may be no correlation to the size of the data being read versus the size of the data being written and this only controls read. The reason I say this is only partially correct is because it makes a false assumption that if you simply take a 1TB dataset and divide it up that the sum of the pieces will equal 1TB which is not true. The size of dataset will be dependent on the cardinality of the columns, but by definition any time you increase the number of parquet files you will increase total data size. If you don't believe me try it yourself. Take a 5GB dataframe and save it as 1 parquet vs 500 parquets. you will see the size is very different

You want target Parquet file sizes around 512 MB without causing a full shuffle. Using coalesce instead of repartition ensures that no shuffle occurs — perfect for performance-sensitive jobs that don’t need data redistribution. The spark.sql.adaptive.advisoryPartitionSizeInBytes helps optimize physical plan execution under Adaptive Query Execution (AQE) without enforcing a hard shuffle. Writing to 2,048 partitions (1 TB ÷ 512 MB) gives the best chance of achieving the desired file sizes while minimizing write latency.

I found this question in anothers Exam test and all of them it's looks like B

Answer D. Explicitly repartitioning to 2,048 partitions ensures that the output files are close to the desired size of 512 MB, provided the data distribution is relatively even. Repartitioning directly addresses the problem by controlling the number of partitions, which directly affects the output file size Why not option A ? Misinterpretation of spark.sql.files.maxPartitionBytes in Option A: The assessment incorrectly states that this configuration controls the maximum size of files when writing to Parquet. This setting controls the size of partitions when reading data, not during writing.

spark.sql.adaptive.advisoryPartitionSizeInBytes - The advisory size in bytes of the shuffle partition during adaptive optimization (when spark.sql.adaptive.enabled is true). It takes effect when Spark coalesces small shuffle partitions or splits skewed shuffle partition. And then we do coalesce - without shuffle - so have to work!

I though D, but default num of partitions is 200, so you cant do coalesce (2048) (you cant increase numb of partitions through coalesce), so its not possible to do it without repartitioning and shuffle. Only A can be done without Shuffle

Option A spark.sql.files.maxPartitionBytes controls the maximum size of partitions during reading on the Spark cluster, and that reducing this value could lead to more partitions and thus potentially more output files. The key point is that it works best when no shuffles occur, which aligns with the scenario of having narrow transformations only.

D is not correct as it will create 2048 target files of 0.5 MB each Only A will do the job as it will read this file in 2 partition ( 1 TB = 512*2 MB) and as we are not doing any shuffling(not mentioned in option) it will create those many partition file i.e 2 part files

ChatGPT says D: This strategy directly addresses the desired part-file size by repartitioning the data. It avoids shuffling during narrow transformations. Recommended for achieving the desired part-file size without unnecessary shuffling.

D is mot suitable.

This approach ensures that each partition will be approximately the target part-file size, which can improve the efficiency of the data write. It also avoids the need for a shuffle operation, which can be expensive in terms of performance.

С is correct

Rest of the answers trigger shuffles

A is correct. The question states Which strategy will yield the best performance without shuffling data. The other options involve shuffling either manually or through AQE

C is correct answer