Exam Professional Machine Learning Engineer topic 1 question 208 discussion

The correct answer is: A. Use the Kubeflow Pipelines SDK to implement the pipeline. Use the BigQueryJobOp component to run the preprocessing script and the CustomTrainingJobOp component to launch a Vertex AI training job. Kubeflow Pipelines (KFP) is a good choice for orchestrating training pipelines, especially since the requirement is to minimize model development and training time. The BigQueryJobOp component is appropriate because the data preprocessing is already performed in BigQuery using SQL scripts. Using BigQueryJobOp avoids unnecessary additional processing layers. The CustomTrainingJobOp component allows launching a Vertex AI training job, which aligns with the need for scalable and managed model training.

C is the best option because: - TFX is designed for ML pipelines, reducing custom code needs and development time and training time as the statement requires - ExampleGen with the BQ executor eliminate data export needs; - Trainer component seamlessly integrates with Vertex AI, leveraging its managed infrastructure for training, further reducing development and operational overhead. A & B uses Kubeflow Pipelines, which would mean more development time and code customization (your model is in TensorFlow); D puts preprocessing inside input_fn, which is generally less efficient for large datasets and complex transformations.

"If you use TensorFlow in an ML workflow that processes terabytes of structured data or text data, we recommend that you build your pipeline using TFX." https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline Google recommends TFX for large amount of structured data. Use input_fn for the Tensorflow model as it will output a tf.data.Dataset object. Note: As it is not mentionned that we are working with terabytes of data, Kubeflow is a viable option and i would choose answer A but i'll stick to google's recommendations

Option C is the most suitable because TFX provides a comprehensive MLOps framework, seamlessly integrating data ingestion, preprocessing, and model training, while also offering strong support for Vertex AI, making it the most efficient solution for the given use case.

C. Use the TensorFlow Extended SDK to implement the pipeline. Use the ExampleGen component with the BigQuery executor to ingest the data, the Transform component to preprocess the data, and the Trainer component to launch a Vertex AI training job.

I go with A Kubeflow Pipelines SDK: supports machine learning and includes components specifically for tasks like data preprocessing, model training, and validation. BigQueryJobOp: enabling you to preprocess data using SQL scripts efficiently within BigQuery.

Example Gen directly ingest data from BigQuery and the transform component makes it more efficient than using an input fn. I chose C over A and B because kubeflow pipelines is more sophisticated and requires more setup and effort because of it's customizability.

agree with guilhermebutzke

agree with guilhermebutzke

Given the requirement to minimize model development and training time while creating a training pipeline for a wide and deep model trained on datasets preprocessed using a SQL script in BigQuery, the most suitable option is: C. Use the TensorFlow Extended SDK to implement the pipeline. Use the ExampleGen component with the BigQuery executor to ingest the data, the Transform component to preprocess the data, and the Trainer component to launch a Vertex AI training job. This option leverages TensorFlow Extended (TFX), which is designed for scalable and production-ready machine learning pipelines. The ExampleGen component with the BigQuery executor efficiently ingests data from BigQuery. The Transform component applies preprocessing steps to the data, and the Trainer component launches a Vertex AI training job, minimizing the time and effort required for model development and training.

Why not C?

My Answer: A According with this documentation: https://cloud.google.com/vertex-ai/docs/tabular-data/tabular-workflows/overview A: CORRECT: BigQueryJobOp for running the existing preprocessing script that already resides there, CustomTrainingJobOp for launching custom training jobs on Vertex AI, which aligns with the requirement of using the pre-trained TensorFlow model. B: Not Correct: While DataflowPythonJobOp can be used for preprocessingthis increasing development time compared to the simpler BigQueryJobOp approach. C and D: Not Correct: While possible, using the TensorFlow Extended SDK with its components introduces unnecessary complexity for this specific scenario. For example, why use ExampleGen? Implementing preprocessing within the model's input_fn is generally not recommended due to potential efficiency drawbacks and training-serving skew issues.

D is wrong. Google doesn't recommend to use input_fn for preprocessing https://www.tensorflow.org/tfx/guide/tft_bestpractices#preprocessing_options_summary

Addressing Limitations of Other Options: Kubeflow Pipelines (A and B): While Kubeflow offers flexibility, it might require more setup and configuration, potentially increasing development time compared to TFX's integrated approach. Separate Preprocessing (C): Using a separate Transform component for preprocessing can add complexity and potential overheads, especially for instance-level transformations that can often be directly integrated within the model's input pipeline.