Exam Professional Machine Learning Engineer topic 1 question 263 discussion

"Full-pass stateful transformations aren't suitable for implementation in BigQuery. If you use BigQuery for full-pass transformations, you need auxiliary tables to store quantities needed by stateful transformations, such as means and variances to scale numerical features. Further, implementation of full-pass transformations using SQL on BigQuery creates increased complexity in the SQL scripts, and creates intricate dependency between training and the scoring SQL scripts." https://www.tensorflow.org/tfx/guide/tft_bestpractices#where_to_do_preprocessing

With multiple epochs, the data is passed through the model multiple times. If you pre-encoded the categorical features (as in options C and D), you would be storing and repeatedly reading a much larger dataset (due to the one-hot encoding). This significantly increases storage costs and I/O overhead. By performing the one-hot encoding within TensorFlow during training (as in option B), the encoding happens on-the-fly for each batch of data during each epoch.

B & D as top 2 choices, C is including Dataflow unnecessarily. D as "minimize the effort and cost of your solution", still some room for B but I selected D

Option D minimizes effort and cost by using BigQuery to handle both the scaling and one-hot encoding. BigQuery is efficient for these types of preprocessing tasks, especially when dealing with large datasets. By preparing the data in BigQuery, you avoid the need to export data to other services or use additional resources for preprocessing, such as Dataflow. This approach provides a streamlined workflow by creating a preprocessed view in BigQuery, which can then be directly fed into Vertex AI Training without extra transformation steps. This helps optimize cost and simplicity while handling large tabular data effectively.

multiple epochs --> need to persist data after preprocessing

Option D since it says minimize effort and cost following that adding something rather than BQ will increase complexity.

Option C uses TFX (TensorFlow Extended) components with Dataflow, which is a great way to perform complex data preprocessing tasks like one-hot encoding and scaling. This approach allows you to process your data in a scalable and efficient manner, using Cloud Storage as the output location. By exporting the results as TFRecords, you can easily feed this preprocessed data into Vertex AI Training for model development.

agree with TFX components with Dataflow

GPT says D, Gemini says B, Perplexity says C.... I say D : stay in one tool, BQ, which is cheap and natively scalable. B has a risk of out of memory error.

BigQuery for Preprocessing: BigQuery is a serverless data warehouse optimized for large datasets.expand_more It can handle scaling numerical features using built-in functions like SCALE or QUANTILE_SCALE, reducing the need for complex custom logic or separate lookup tables. TensorFlow for One-Hot Encoding: TensorFlow excels at in-memory processing. One-hot encoding of categorical features, especially text features like SKU names, can be efficiently performed within your TensorFlow model during training. This avoids unnecessary data movement or transformations in BigQuery. Vertex AI Training: By feeding the preprocessed data (scaled numerical features) directly into Vertex AI Training, you leverage its managed infrastructure for training your custom TensorFlow model.

Agree with b1a8fae

agree with daidai75

My Answer: B 1. Use BigQuery to scale the numerical features.: Simpler and cheaper then use TFX components with Dataflow to scale the numerical features 2. Feed the features into Vertex AI Training. 3. Allow TensorFlow to perform the one-hot text encoding: TensorFlow handles the one-hot text encoding better than BQ.

key messages: "contains hundreds of millions of rows, and includes both categorical and numerical features. You need to use a MaxMin scaler on some numerical features, and apply a one-hot encoding to some categorical features such as SKU names". Option B is not suitable for the big volume of data processing. Option C is better.

Inclined to choose C over B. By using TFX components with Dataflow, you can perform feature engineering on large-scale tabular data in a distributed and efficient way. You can use the Transform component to apply the MaxMin scaler and the one-hot encoding to the numerical and categorical features, respectively. You can also use the ExampleGen component to read data from BigQuery and the Trainer component to train your TensorFlow model.

Option A: Involves creating a separate lookup table and deploying a Hugging Face model in BigQuery, increasing complexity and cost. Option C: While TFX offers robust preprocessing capabilities, it adds overhead for this use case and requires knowledge of Dataflow. Option D: Performing one-hot encoding in BigQuery can be less efficient than TensorFlow's optimized implementation.