Exam Professional Machine Learning Engineer topic 1 question 152 discussion

The crucial point is that the same encoding scheme must be applied to both the training and test sets.

I've very grudgingly ticket C, as the question is missing "handle the missing category by one hot encoding all zeros for the missing feature column". It otherwise doesn't make sense as will have the wrong amount of entries.

The correct approach is to handle the missing category during one-hot encoding of the test data. Here's how to address this issue: Identify the Missing Category: After applying one-hot encoding to the training set, compare the categories (unique values) present in the training data with the categories in the test data. This will reveal the missing category. Add a Column for the Missing Category in the Test Data: Include a new column in the test data specifically for the missing category. Initialize the values in this column with 0. Apply One-Hot Encoding to the Test Data: Now that the test data includes a column for the missing category, proceed with one-hot encoding the categorical variables in the test data. This will ensure the test data has the same structure as the encoded training data.

Answer C

Answer options analysis: C. Since one categorical variable is missing from the test set, (As I understand: “a categorical variable is in the test but not in train”) apply one hot encoding (trained with the train set?) to the test set, for the variables not present in train we just would obtain an array of all 0’s, so that would be OK. D. That data collection could be not feasible depending on the real-world-problem. B. Randomness would not always fix the problem. A. Not recommended to use different representations for train/test. Sparse representation doesn't magically recover missing categories; it's a way to efficiently store data with a large number of zeros. I would go with C.

C but not really sure

You must apply one hot enconding alsto for the test dataset. However, i find this answer incomplete.

Add data to the test set to get the same OHE

“Rows are selected for a data split randomly, but deterministically. (…) Training a new model with the same training data results in the same data split.” https://cloud.google.com/vertex-ai/docs/tabular-data/data-splits#classification-random. “Randomly redistribute data” [Option B] with different fractions, will result in a different data split. Having a higher fraction split of 70% for the training set will additionally help the model to better generalize (compared to only 50%), thus perform better when testing, the ultimate goal.

By using a sparse representation, you will be losing the information contained in the missing categorical variable. This could lead to the model making incorrect predictions on the test set.

I agree with formazioneQl that if a different one hot encoding is used for the test set compared to the train set then the results would be poor. However, there is no problem with not having all combinations in the test set if all possibilities are present in the training set. So assuming that we are using the same mapping of data in the train and test set, I would vote C. If we don't encode the test set, the variable is meaningless anyways. So I would lean C.

Since one categorical variable is missing from the test set, C would result in a different number of columns in the training and test sets.

C. Apply one-hot encoding on the categorical variables in the test data. When using one-hot encoding on categorical variables, each unique value of the variable is represented as a separate binary variable. Therefore, it is important to ensure that the same set of binary variables is present in both the training and test datasets. Since one categorical variable is missing in the test set, the recommended approach is to apply one-hot encoding on the categorical variables in the test set to ensure that the same set of binary variables is present in both datasets.

Answer C