Exam Professional Data Engineer topic 1 question 177 discussion

The question is C but not because the SQL Syntax, as you can perfectly use SparkSQL on Dataproc reading files from GCS. It's because the "serverless" requirement.

A) Looks more suitable , serverless approach for handling and performance.

Option C is the best approach to meet the stated requirements. Here's why: BigQuery SQL provides a fast, scalable, and serverless method for transforming structured data, easier to develop than PySpark. Directly ingesting the raw Cloud Storage data into BigQuery avoids needing an intermediate processing cluster like Dataproc. Transforming the data via BigQuery SQL queries will be faster than PySpark, especially since the data is already loaded into BigQuery. Writing the transformed results to a new BigQuery table keeps the original raw data intact and provides a clean output. So migrating to BigQuery SQL for transformations provides a fully managed serverless architecture that can significantly expedite development and reduce pipeline runtime versus PySpark. The ability to avoid clusters and conduct transformations completely within BigQuery is the most efficient approach here.

Note: Dataproc by itself is not serverless https://cloud.google.com/dataproc-serverless/docs/overview

because of serverless nature

Answer C: need to setup SQL based job means transformation in not very complex. And Biqquery sql are faster than spark sql context. (google claims) However, i will make a test by myself to check it.

In the GCP Machine Learning Engineer practice question (Q4) there's the same question with similar answers and the correct answer is A since B "is incorrect, here transformation is done on Cloud SQL, which wouldn’t scale the process" and C "is incorrect as this process wouldn’t scale the data transformation routine. And, it is always better to transform data during ingestion": https://medium.com/@gcpguru/google-google-cloud-professional-machine-learning-engineer-practice-questions-part-1-3ee4a2b3f0a4

C D-is incorrect because you are rebuild your batch pipeline for structured data on Google Cloud.

This same question is there on Google's Professional Machine Learning Engineer, Question 4 Answer is C.

I choose C BigQuery SQL is more performant but more expensive. Here, it's a performance issue ( time reduction) Source : https://medium.com/paypal-tech/comparing-bigquery-processing-and-spark-dataproc-4c90c10e31ac

C is the most likely , bigquery is severless and sql D is dataflow severless but it is wrong at using python sdk but using sql beam rthen it will be correct

Answer C

A - You have to maintain PySpark Code -> Proc

C. Ingest your data into BigQuery from Cloud Storage, convert your PySpark commands into BigQuery SQL queries to transform the data, and then write the transformations to a new table. Keys: "Serverless" and "SQL"