Exam Professional Data Engineer topic 1 question 143 discussion

It's D. → Your engineers have a new version of the pipeline with a different windowing algorithm and triggering strategy. New version is mayor changes. Stop and drain and then launch the new code is a lot is the safer way. We recommend that you attempt only smaller changes to your pipeline's windowing, such as changing the duration of fixed- or sliding-time windows. Making major changes to windowing or triggers, like changing the windowing algorithm, might have unpredictable results on your pipeline output. https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#changing_windowing

D seems the right way to go

Option A is the first approach to try, as it allows for an in-flight update with minimal disruption. However, if the changes in the new version of the pipeline are not compatible with an in-flight update (due to significant changes in windowing or triggering), then option D should be used. The Drain option ensures a graceful shutdown of the existing pipeline, reducing the risk of data loss, and then a new job can be started with the updated code.

A is not an option as "You want to ensure that no data is lost during the update. ": Making major changes to windowing or triggers, like changing the windowing algorithm, might have unpredictable results on your pipeline output. https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#change_windowing

Drain Option: The "Drain" option allows the existing Dataflow job to complete processing of any in-flight data before stopping the job. This ensures that no data is lost during the transition to the new version. Create a New Job: After draining the existing job, you create a new Cloud Dataflow job with the updated code. This new job starts fresh and continues processing data from where the old job left off. Option A (updating the inflight pipeline with the --update option) may not guarantee no data loss, as the update could disrupt the existing job's operation and potentially cause data loss. Option B (updating the inflight pipeline with the --update option and a new job name) is similar to option A and may not provide data loss guarantees. Option C (stopping the pipeline with the Cancel option and creating a new job) will abruptly stop the existing job without draining, potentially leading to data loss.

Look D after seeing some docs. please check the below link https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline

I will go with option D - If you want to minimize the impact of the update, then option A is the best option. However, if you are not concerned about a temporary interruption in processing, then option D is also a valid option. Option Pros Cons A Does not stop the pipeline, so no data is lost. Requires you to create a new version of the pipeline. B Creates a new job with the updated code, so you do not have to update the running pipeline. Can lead to data loss if the new job does not process all of the data that was in the running pipeline. C Stops the pipeline and drains any data that is currently in flight, so no data is lost. Causes a temporary interruption in processing.

A is not recommeded for major changes in pipeline.

Answer A: ```gcloud dataflow jobs update <JOB_ID> --update <GCS_PATH_TO_UPDATED_PIPELINE> --region <REGION>``` --update flag does not miss any data and you can execute this command even yourpipeline is running. Its safe any fast, you can continuously make some change and update this command. no problem. Stop and Drain, is required when you want to test the pipeline and stop it without losing the data.

agree with odacir

vote A D: drain doesn't mention about update dataflow job just stop and preserve data A: replace existing job and preserve data (When you update your job, the Dataflow service performs a compatibility check between your currently-running job and your potential replacement job. The compatibility check ensures that things like intermediate state information and buffered data can be transferred from your prior job to your replacement job.) https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline

A is the answer. https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching To update your job, launch a new job to replace the ongoing job. When you launch your replacement job, set the following pipeline options to perform the update process in addition to the job's regular options: - Pass the --update option. - Set the --jobName option in PipelineOptions to the same name as the job you want to update.

D A-is not because The Dataflow service retains the job name, but runs the replacement job with an updated Job ID. Description: When you update a job on the Dataflow service, you replace the existing job with a new job that runs your updated pipeline code. The Dataflow service retains the job name, but runs the replacement job with an updated Job ID. This process can cause downtime while the existing job stops, the compatibility check runs, and the new job starts.' https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#python:~:text=When%20you%20update%20a,has%20the%20following%20transforms%3A D is correct Drain ->clone -> update -> run

Changing windowing algorithm may break the pipeline. https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#changing_windowing

No, do not drain the current job.

in this scenario pipline is streaming pipline with windowing algorithm and triggering strategy changes to new one without loss of data,so better to go with Drain option as it fullfile all precondition described in scenario which is :- 1.streaming 2.code changes with windowing algorithm and triggering strategy to new way 3.no loss of data during update Referances:- https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain Drain a job. This method applies only to streaming pipelines. Draining a job enables the Dataflow service to finish processing the buffered data while simultaneously ceasing the ingestion of new data. For more information, see Draining a job.

D: They want to preserve data and updates might not be predictable. https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#changing_windowing