You need to migrate Hadoop jobs for your company's Data Science team without modifying the underlying infrastructure. You want to minimize costs and infrastructure management effort. What should you do?
A.
Create a Dataproc cluster using standard worker instances.
B.
Create a Dataproc cluster using preemptible worker instances.
C.
Manually deploy a Hadoop cluster on Compute Engine using standard instances.
D.
Manually deploy a Hadoop cluster on Compute Engine using preemptible instances.
Should be B, you want to minimize costs.
https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms#preemptible_and_non-preemptible_secondary_workers
Agree, the migration guide also recommends to think about preemptible worker nodes: https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-jobs#using_preemptible_worker_nodes
I think it's A.
The question does not mention anything about minimize the costs, all the questions in GCP exams that require minimize the costs as requirement literally mention that in the question.
Also in order to minimize the costs you need to build jobs that are fault tolerant, as workers instances are preemptible. This also requires some kind of Dev investment of work. So if not mentioned in the question fault tolerant and minimize costs then is not required/needed.
Doc states below:
Only use preemptible nodes for jobs that are fault-tolerant or that are low enough priority that occasional job failure won't disrupt your business.
Hi TotoroChina,
I had the same thought when I first read the question - the problem I see is, in real business I think you would try to mix preemtible instances and on-demand instances... Here you have to choose between only preemtible instances and on-demand instances... Preemptible instances have some downsides - so we would need more details and ideally a mixed approach. That's why both answers might be correcy, a) and b)...
Do you see that different?
Thanks!
Cheers,
D.
A cost-savings consideration: Using preemptible VMs does not always save costs since preemptions can cause longer job execution with resulting higher job costs. This is mentioned in above link So I think Ans should be A
It's A, the primary workers can only be standard, where secondary workers can be preemtible.------In addition to using standard Compute Engine VMs as Dataproc workers (called "primary" workers), Dataproc clusters can use "secondary" workers.
There are two types of secondary workers: preemptible and non-preemptible. All secondary workers in your cluster must be of the same type, either preemptible or non-preemptible. The default is preemptible.
A. Create a Dataproc cluster using standard worker instances.
you can use premtible instnace however should be less then 50% totaak wroker instances... could impact staiblirty...
cons transient task failures.
follow best practice:
General Recommendations:
Start Conservatively: Begin with a smaller percentage of preemptible workers (e.g., less than 30% of the total worker nodes). Monitor your jobs closely to see how they perform and adjust as needed.
Don't Exceed 50%: Google generally recommends keeping the number of preemptible workers below 50% of the total worker nodes in your cluster. 1 This helps maintain stability and reduces the risk of significant job disruptions.
Probably A
- Primary workers must be standard
- Preemptible doesn't always save cost
- Infrastructure on prem doesn't have spot machines
- You cannot choose spot/preemptible when creating the cluster, only when provisioning secondary nodes, which are actually preemptible by default.
- They do not store data, only do data processing
B: being a multiple-choice question, we need to focus on explicit keywords here.
Management effort => Managed service -> Dataproc.
Cost-optimization => Preemptible.
For ones who say "but that also requires fault toleration": well, there is no explicit keyword in the question says "we have critical jobs" or "out data scientists team has not takes into account toleration". So we must not assume that's needed.
A - only secondary workers can be preemptible and "Using preemptible VMs does not always save costs since preemptions can cause longer job execution with resulting higher job costs" according to: https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms#preemptible_and_non-preemptible_secondary_workers
Answer is B.
The secondary worker type instance for default Dataproc cluster is preemptible VMs.
https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms
Dataproc: Dataproc is a fully managed Apache Spark and Hadoop service on Google Cloud Platform. It allows you to run clusters without the need to manually deploy and manage Hadoop clusters on Compute Engine.
Preemptible Worker Instances: Preemptible instances are short-lived, cost-effective virtual machine instances that are suitable for fault-tolerant and batch processing workloads. Since Hadoop jobs can often tolerate interruptions, using preemptible instances can significantly reduce costs.
Option B leverages the benefits of Dataproc for managing Hadoop clusters without the need for manual deployment and takes advantage of preemptible instances to minimize costs. This aligns well with the goal of minimizing both costs and infrastructure management efforts.
Using standard Compute Engine VMs as Dataproc workers (called "primary" workers), Preemptible can be only used for secondary workers hence A is valid answer
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
TotoroChina
Highly Voted 3 years, 10 months agoJ19G
3 years, 6 months agoale_brd_111
2 years, 5 months agoAmrx
6 months, 1 week agogrejao
2 years agoXDevX
3 years, 10 months agokopper2019
3 years, 9 months agoHenkH
2 years, 5 months agoSukon_Desknot
2 years, 6 months agoYogi42
2 years, 3 months agofirecloud
Highly Voted 3 years, 9 months agoManh
3 years, 7 months agodavid_tay
Most Recent 2 months, 1 week ago3a7557a
2 months, 3 weeks agoplumbig11
3 months, 4 weeks agoLestrang
7 months agopcamaster
7 months agoafsarkhan
9 months, 3 weeks agoGino17m
1 year agodija123
1 year agoDiwz
1 year, 1 month agoshashii82
1 year, 1 month agoVidhyaBupesh
1 year, 2 months agoAmrita2012
1 year, 2 months agoPime13
1 year, 2 months agod0094d6
1 year, 2 months agod0094d6
1 year, 2 months ago