Exam Professional Data Engineer topic 1 question 104 discussion

I'd pick D because it's the only option which allows variable execution (since we need to execute the dataprep job only after the prior load job). Although D suggests the export of Dataflow templates, this discussion suggests that the export option is no longer available (https://stackoverflow.com/questions/72544839/how-to-get-the-dataflow-template-of-a-dataprep-job), there are already Airflow Operators for Dataprep which we should be using instead - https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/dataprep.html

Since the load job execution time is unexpected, schedule the Dataprep based on a fixed time window may not work. When the Dataprep job run the first time, we can find the Dataflow job for that in the console. We can use that to create the Template --> With the help of the Composer to determine if the load job is completed, we can then trigger the Dataflow job

Dataprep by Trifacta allows you to schedule the execution of recipes. You can set up a cron schedule directly within Dataprep to automatically run your recipe at specified intervals, such as daily. WHY NOT D ? : This option involves significant additional complexity. Exporting the Dataprep job as a Dataflow template and then incorporating it into a Composer (Apache Airflow) job is a more complicated process and is typically used for more complex orchestration needs that go beyond simple scheduling.

We have external dependency "after the load job with variable execution time completes" which requires DAG -> Airflow (Cloud Composer) The reasons: A scheduler like Cloud Scheduler won't handle the dependency on the BigQuery load completion time Using Composer allows creating a DAG workflow that can: Trigger the BigQuery load Wait for BigQuery load to complete Trigger the Dataprep Dataflow job Dataflow template allows easy reuse of the Dataprep transformation logic Composer coordinates everything based on the dependencies in an automated workflow

I'd pick D because it's the only option which allows variable execution

The key here is "after the load job with variable execution time completes" which means the execution of this job depends on the completion of another job which has a variable execution time. Hence D

This approach ensures the dynamic triggering of the Dataprep job based on the completion of the preceding load job, ensuring data is processed accurately and in sequen

A is correct. D is too complicated. A is correct, because you can schedule a job right from Dataprep UI. https://cloud.google.com/blog/products/gcp/scheduling-and-sampling-arrive-for-google-cloud-dataprep Scheduling and sampling arrive for Google Cloud Dataprep Throughout our early releases, users’ most common request has been Flow scheduling. As of Thursday’s release, Flows can be scheduled with minute granularity at any frequency.

I think C it is more straighforward

Answer C: Use Recipe Template feature of dataprep. Don't need to change the service.

Why not C?

D is the answer.

It's A. You can set it directly in Dataprep a job and it will use Dataflow under the hood.

It's A. You can set it directly in Dataprep a job and it will use Dataflow under the hood. No need to export nor incorporate into a Composer job. Dataprep by trifacta - https://docs.trifacta.com/display/DP/cron+Schedule+Syntax+Reference Dataprep job uses dataflow - https://cloud.google.com/dataprep

It's D

Dataprep and Dataflow are same famitly

D. Export the Dataprep job as a Dataflow template, and incorporate it into a Composer job. Reveal Solution