Exam DP-100 topic 7 question 6 discussion

Actual exam question from Microsoft's DP-100

Question #: 6
Topic #: 8

HOTSPOT -
You need to configure the Permutation Feature Importance module for the model training requirements.
What should you do? To answer, select the appropriate options in the dialog box in the answer area.
NOTE: Each correct selection is worth one point.
Hot Area:

Show Suggested Answer

Suggested Answer:

Box 1: 500 -
For Random seed, type a value to use as seed for randomization. If you specify 0 (the default), a number is generated based on the system clock.
A seed value is optional, but you should provide a value if you want reproducibility across runs of the same experiment.
Here we must replicate the findings.

Box 2: Mean Absolute Error -
Scenario: Given a trained model and a test dataset, you must compute the Permutation Feature Importance scores of feature variables. You need to set up the
Permutation Feature Importance module to select the correct metric to investigate the model's accuracy and replicate the findings.
Regression. Choose one of the following: Precision, Recall, Mean Absolute Error, Root Mean Squared Error, Relative Absolute Error, Relative Squared Error,

Coefficient of Determination -
Reference:
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/permutation-feature-importance

by Zhuo at May 17, 2020, 7:02 p.m.

Comments

Submit Cancel

podval

Highly Voted 3 years, 11 months ago

RMSE indicates the absolute fit of the model to the data–how close the observed data points are to the model's predicted values. Whereas R-squared is a relative measure of fit, RMSE is an absolute measure of fit. See: "You must be determined the absolute fit of the model".

upvoted 23 times

...

Zhuo

Highly Voted 4 years ago

Mean Absolute Error , Root Mean Squared Error, r- squared are all correct.

upvoted 16 times

...

Mckay_

Most Recent 1 year, 7 months ago

MAE seems like the best choice since RMSE is more sensitive to outlier.

upvoted 2 times

...

[Removed]

2 years, 1 month ago

MAE is correct. RMSE has the benefit of penalizing large errors more so can be more appropriate in some cases, for example, if being off by 10 is more than twice as bad as being off by 5. But if being off by 10 is just twice as bad as being off by 5, then MAE is more appropriate. From an interpretation standpoint, MAE is clearly the winner. RMSE does not describe average error alone and has other implications that are more difficult to tease out and understand. On the other hand, one distinct advantage of RMSE over MAE is that RMSE avoids the use of taking the absolute value, which is undesirable in many mathematical calculations.

upvoted 5 times

BTAB

1 year, 4 months ago

Excellent evaluation and I concur. Going with MAE.

upvoted 1 times

...

Zwi3b3l

3 years, 3 months ago

Should be RMSE. https://www.theanalysisfactor.com/assessing-the-fit-of-regression-models/

upvoted 2 times

...

Alexandra

3 years, 11 months ago

if the findings should be replicated, than the seed should be 0 also another question for this case study had r-squared as correct evaluation method for regression model...

upvoted 2 times

111ssy

3 years, 6 months ago

If it is 0, the seed is generated by the system clock meaning that it won't be replicable and keep changing like the time, hence would be 500?

upvoted 11 times

phdykd

10 months ago

where does it ask to be replicable?

upvoted 1 times

...