Exam Professional Data Engineer topic 1 question 150 discussion

Correct: B Local HDFS storage is a good option if: Your jobs require a lot of metadata operations—for example, you have thousands of partitions and directories, and each file size is relatively small. You modify the HDFS data frequently or you rename directories. (Cloud Storage objects are immutable, so renaming a directory is an expensive operation because it consists of copying all objects to a new key and deleting them afterwards.) You heavily use the append operation on HDFS files. You have workloads that involve heavy I/O. For example, you have a lot of partitioned writes, such as the following: spark.read().write.partitionBy(...).parquet("gs://") You have I/O workloads that are especially sensitive to latency. For example, you require single-digit millisecond latency per storage operation.

Answer B Its google recommended approach to use LocalDisk/HDFS to store Intermediate result and use Cloud Storage for initial and final results.

Local HDFS storage is a good option if: - You have workloads that involve heavy I/O. For example, you have a lot of partitioned writes such as the following: spark.read().write.partitionBy(...).parquet("gs://") - You have I/O workloads that are especially sensitive to latency. For example, you require single-digit millisecond latency per storage operation. - Your jobs require a lot of metadata operations—for example, you have thousands of partitions and directories, and each file size is relatively small. - You modify the HDFS data frequently or you rename directories. (Cloud Storage objects are immutable, so renaming a directory is an expensive operation because it consists of copying all objects to a new key and deleting them afterwards.) - You heavily use the append operation on HDFS files.

The correct answer is B.

Disk I/O Performance: In a Cloud Dataproc cluster, the default setup uses local persistent disks for HDFS storage. These disks offer good disk I/O performance and are well-suited for storing intermediate data generated during Hadoop jobs. Data Locality: Storing intermediate data on native HDFS allows for better data locality. This means that the data is stored on the same nodes where computation occurs, reducing the need for data transfer over the network. This can significantly improve the performance of disk I/O-intensive jobs. Scalability: Cloud Dataproc clusters can be easily scaled up or down to meet the specific requirements of your jobs. You can allocate additional disk space as needed to accommodate the intermediate data generated by this particular Hadoop job.

Correct: A I'd choose A as the doc states adding more SSDs are good for disk-intensive jobs especially those with many individual read and write operations https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-jobs

I would choose A. Google Storage is faster than HDFS in many cases. https://cloud.google.com/architecture/hadoop#:~:text=It%27s%20faster%20than%20HDFS%20in%20many%20cases. The question mention '(8-core nodes with 100-GB RAM)' on-premises Hadoop. the problem may caused by insufficient memory, and does not mention cost would be an issue, so A 'memory' approach would be a better option.

Best option is B. However allocating sufficient persistent disk space to the Hadoop cluster, and storing the intermediate data of that particular Hadoop job on native HDFS, would not improve the performance of the Hadoop job. In fact, it might even slow down the Hadoop job, as the data would have to be read and written to disk twice.

B is the answer. https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-jobs#choosing_primary_disk_options If your job is disk-intensive and is executing slowly on individual nodes, you can add more primary disk space. For particularly disk-intensive jobs, especially those with many individual read and write operations, you might be able to improve operation by adding local SSDs. Add enough SSDs to contain all of the space you need for local execution. Your local execution directories are spread across however many SSDs you add.

https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-jobs#choosing_primary_disk_options

B is not the right answer. The problem says that for intermediate data cloud storage is to be used, while B option says: B ... the intermediate data of that particular Hadoop job on native HDFS A is the right answer. If you have enough memory then the shuffle wont spill on the disk.

B should be the right answer: https://cloud.google.com/compute/docs/disks/performance#optimize_disk_performance

https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-jobs

Why there are so many wrong answers? Examtopics.com are you enjoying paid subscription by giving random answers from people? Ans: B

If your job is disk-intensive and is executing slowly on individual nodes, you can add more primary disk space. For particularly disk-intensive jobs, especially those with many individual read and write operations, you might be able to improve operation by adding local SSDs. Add enough SSDs to contain all of the space you need for local execution. Your local execution directories are spread across however many SSDs you add. Its B https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-jobs

yes B is correct

Correct Answer is Option B - Adding persistent disk space, reasons:- - The question mentions that the particular job is "disk I/O intensive - so the word "disk" is explicitly mentioned. - Option B also mentions about local HDFS storage, which is ideally a good option of general I/O intensive work. -