Exam Professional Data Engineer topic 1 question 27 discussion

Ans: B A: correlated to output means that feature can contribute a lot to the model. so not a good idea. C: you need to run with almost same number, but you will iterate twice, once for averaging and second time to feed the averaged value. D: removing features even if it 50% nulls is not good idea, unless you prove that it is not at all correlated to output. But this is nowhere so can remove.

Trying to find a reason why it is B and not D, found this and it seems the answer is D. https://cloud.google.com/architecture/data-preprocessing-for-ml-with-tf-transform-pt1 Feature selection. Selecting a subset of the input features for training the model, and ignoring the irrelevant or redundant ones, using filter or wrapper methods. This can also involve simply dropping features if the features are missing a large number of values.

I think the best option is B. D could be an option but what if the feature is very correlated to the result ?

B: combine the dependent features. It is more like PCA (principal component analysis). D: could be the answer, but what if that feature is very important, or how often do you get a feature with more than 50% NULL values?

I am not into ML, to be honest, so I will rely on community opinion and choose B

B. Combine highly co-dependent features into one representative feature. Combining highly correlated features into a single representative feature can reduce the dimensionality of your dataset, making the training process faster while preserving relevant information. This approach often helps eliminate redundancy in the input data. Option A (eliminating features that are highly correlated to the output labels) can be counterproductive, as you want to maintain features that are informative for your prediction task. Removing features that are correlated with the output may reduce model accuracy. Option C (averaging feature values in batches of 3) is not a common technique for reducing dimensionality, and it could lead to loss of important information. Option D (removing features with null values for more than 50% of training records) can help reduce the dimensionality and may be useful if you have a large number of features with missing data, but it may not necessarily address co-dependency among features.

B. Combine highly co-dependent features into one representative feature. This is the best choice.

"D" is wrong, and very dangerous. For instance, it might represent modern measurements only installed in <50% of weather stations, but very very precise and valuable. Nulls are not a problem for models, out-of-the-box or with transformations models can handle nulls just fine.

wrong question. there are two answers B, D

Answer is Combine highly co-dependent features into one representative feature. A: correlated to output means that feature can contribute a lot to the model. so not a good idea. C: you need to run with almost same number, but you will iterate twice, once for averaging and second time to feed the averaged value. D: removing features even if it 50% nulls is not good idea, unless you prove that it is not at all correlated to output. But this is nowhere so can remove.

I have a doubt, instead of combining highly corelated features why cant we remove corelated features which may give much more simplified dataset?

Answer: B Data that is co-dependent is high corelated is some kind of reduldant information in some cases. If the features x1, x2 and x3 are x2 = x1 + 1 and x3 = 2*x1, for example, x2 and x3 are reduldant because can be explained with x1 feature, so can be excluded of the the model. Other option is to group this features. There is a lot of ways to resolve, but the main ideia is to use data engineer in co-depedent features to reduce the number of features in the model

This method is called Data Engineering, that you combine two or more values to get a custom info, this will avoid that the model read an extra column on the training and probably increase it's accuracy.

Answer: B

Co-dependent -> correlated -> correlated info = already present info in other variable.

B null values can have many meanings and need different approach to handle, otherwise it causes inaccurate model, so not D

For me the best option is B