Exam Professional Data Engineer topic 1 question 313 discussion

Priority is "minimize installation and management effort" which is done via Dataproc Serverless. Furthermore, with Dataproc serverless you can still specify resource settings for your job, such as the number of vCPUs and memory per executor (https://cloud.google.com/dataproc-serverless/docs/concepts/properties)

I would go with (C) : "and you want to be able to choose similar settings" not applicable for (D)

" where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings." minimize effort by using dataproc not GKE or vms..

D is the best answer - It allows Dataproc serverless allows to specify executor configurations like vCPU and memory settings (e.g., executor cores and memory) to match the current setup as is specified. C is a valid but sub-optimal choice — while we can specify vCPUs and memory similar to our on-prem setup but it requires provisioning and managing clusters, which we want to avoid. it requires slightly more effort compared to Dataproc Serverless.

I agree that minimizing installation and management means using Dataproc Serverles. Also, Serverles can be configured with up to 16 VPU and up to 29696m of memory in for the premium tier. https://cloud.google.com/dataproc-serverless/docs/concepts/properties#:~:text=Total%20driver%20memory%20per%20driver%20core%2C%20including%20driver%20memory%20overhead%2C%20which%20must%20be%20between%201024m%20and%207424m%20for%20the%20Standard%20compute%20tier%20(24576m%20for%20the%20Premium%20compute%20tier).%20For%20example%2C%20if%20spark.driver.cores%20%3D%204%2C%20then%204096m%20%3C%3D%20spark.driver.memory%20%2B%20spark.driver.memoryOverhead%20%3C%3D%2029696m.

Needs to be able to configure "similar settings"

It's not possible to specify a machine type using Dataproc Serverless

I choice "C", just: "where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings"

Dataproc Serverless allows you to run Spark jobs without needing to manage the underlying infrastructure. It automatically handles resource provisioning and scaling, which simplifies the process and reduces management overhead

Dataproc supports Spark 3, ensuring compatibility with your existing job. It also allows you to customize the cluster configuration, including the number of executors, vCPUs, and memory per executor, to match your on-premises setup (8 vCPU and 16 GB memory)