Exam Certified Data Engineer Professional topic 1 question 136 discussion

The key to efficiently converting a large JSON dataset to Parquet files of a specific size without shuffling data lies in controlling the size of the output files directly. Setting spark.sql.files.maxPartitionBytes to 512 MB configures Spark to process data in chunks of 512 MB. This setting directly influences the size of the part-files in the output, aligning with the target file size. Narrow transformations (which do not involve shuffling data across partitions) can then be applied to this data. Writing the data out to Parquet will result in files that are approximately the size specified by spark.sql.files.maxPartitionBytes, in this case, 512 MB. The other options involve unnecessary shuffles or repartitions (B, C, D) or an incorrect setting for this specific requirement (E).

Correct answer is D: Why Not Other Options?: A. Set spark.sql.files.maxPartitionBytes: This configuration controls how many bytes Spark reads per input partition during a file scan, not the output file size. It does not help in controlling Parquet file sizes during writing. B. Set spark.sql.shuffle.partitions and sort data: While sorting data can optimize performance in some cases, it introduces unnecessary overhead for this scenario. Additionally, spark.sql.shuffle.partitions controls the number of shuffle partitions, not directly the output partitioning of the data. C. Use spark.sql.adaptive.advisoryPartitionSizeInBytes: Adaptive Query Execution (AQE) optimizes queries at runtime, but this configuration does not directly control Parquet file sizes. It dynamically adjusts partition sizes for shuffle stages, not for the write output.

arekm explanation

Definitely A - no repartitioning and subsequent shuffle (which the question is asking about). The parameter defines how many bytes per partition to read, tasks will read in those chunks, since only narrow operations performed (per definition - no shuffle), we just write what we read. The target files size is 512MBs and we did not shuffle.

I think, "execute the narrow transformations, coalesce to" is key words here - because coalesce is not cause shuffling.

It's D, because A primarily affects the reading of the data

A does not shuffle while D shuffles

Answer A as narrow transformations like union, filter and map do not cause shuffle across partitions.

The correct answer is D. Ingest the data, execute the narrow transformations, repartition to 2,048 partitions (1TB * 1024 * 1024 / 512), and then write to Parquet. Explanation: In this case, the goal is to write a 1 TB dataset to Parquet with a target file size of 512 MB without incurring the overhead of data shuffling. To achieve optimal performance, we must balance the number of partitions to match the file size requirements while avoiding expensive shuffle operations. Narrow transformations: These transformations (such as map, filter) don’t require shuffling the data, which keeps the operation efficient. Repartition to 2,048 partitions: Given that the desired part-file size is 512 MB and the total dataset size is 1 TB, repartitioning the dataset into 2,048 partitions ensures that each partition will be approximately 512 MB in size, which matches the target file size. This avoids shuffle operations and allows for an efficient write.

Not A because spark.sql.files.maxPartitionBytes primarily affects the reading of data, not the writing. It determines the maximum size of a partition when reading files, not when writing them.

A, D will not prevent shuffling data. C using coalesce to reduce shuffling data.

best performance without shuffling data

option D

Correct Answer D: Repartition to 2,048 partitions and write to Parquet This option directly controls the number of output files by repartitioning the data into 2,048 partitions, assuming that 1TB/512MB per file roughly translates to 2,048 files. Repartitioning the data involves shuffling, but it's a deliberate shuffle designed to achieve a specific partitioning beneficial for writing. After repartitioning, the data is written to Parquet files, each expected to be approximately 512 MB if the data is uniformly distributed across partitions.