Exam Professional Data Engineer topic 1 question 156 discussion

Confused between A and B. For r/w intensive jobs need to use SSDs. But questions doesnt state anything about the nature of the jobs. So better to start with a default option. Choose A

Answer is A…Cloud Dataproc for Managed Cloud native application and HDD for cost-effective solution.

Options A is the right answer. Option B using SSD persistent disk which will add more cost than default HDD Option C & D are out of scope.

Dataproc Managed Service: Dataproc is a fully managed service for running Apache Hadoop and Spark. It provides ease of management and automation. Standard Persistent Disk: Using standard persistent disks for Dataproc workers ensures durability and is cost-effective compared to SSDs. Preemptible Workers: By using 50% preemptible workers, you can significantly reduce costs while maintaining fault tolerance. Preemptible VMs are cheaper but can be preempted by Google, so having a mix of preemptible and non-preemptible workers provides cost savings with redundancy. Storing Data in Cloud Storage: Storing data in Cloud Storage is highly durable, scalable, and cost-effective. It also makes data accessible to Dataproc clusters, and you can leverage native connectors for reading data from Cloud Storage. Changing References to gs://: Updating your scripts to reference data in Cloud Storage using gs:// ensures that your jobs work seamlessly with the cloud storage infrastructure.

Apache Hadoop -> Dataproc or Compute Engine with proper SW installation cost-effective -> use standard persistent disk + store data in Cloud Storage batch -> Dataproc or Compute Engine with proper SW installation managed service -> Dataproc

A is the answer. https://cloud.google.com/dataproc/docs/concepts/overview Dataproc is a managed Spark and Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. Dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them. With less time and money spent on administration, you can focus on your jobs and your data.

it says cost effective , hence no SSD

Correct: A

Correct : A Option B is usefull if you use HDFS, and in this case as you use preemtible machines it isn't worth use SSD disks.

Answer - B

Look at this link. https://cloud.google.com/bigtable/docs/choosing-ssd-hdd At the First look I chose Option-B as they mentioned SSD is cost-effective on most cases. But after reading the whole page, they also mentioned that for batch workloads, HDD is suggested as long as not heavy read. So I changed my mind to Option-A (I assumed this is not ready heavy process?).

Option B - SSD disks, reasons:- The question asks "fault-tolerant and cost-effective as possible for long-running batch job". 3 Key words are - fault tolerant / cost effective / long running batch jobs.. The cost efficiency part mentioned in the question could be addressed by 50% preemptible disks and storing the data in cloud storage than HDFS. For long running batch jobs and as standard approach for Dataproc - we should always go with SSD disk types as per google recommendations.

Ans is B, for long running SDD suitable. HDD maintenance will be additional charge for long running jobs

Answer A