Exam DP-100 topic 7 question 11 discussion

Actual exam question from Microsoft's DP-100

Question #: 11
Topic #: 8

DRAG DROP -
You need to implement an early stopping criteria policy for model training.
Which three code segments should you use to develop the solution? To answer, move the appropriate code segments from the list of code segments to the answer area and arrange them in the correct order.
NOTE: More than one order of answer choices is correct. You will receive credit for any of the correct orders you select.
Select and Place:

Show Suggested Answer

Suggested Answer:

You need to implement an early stopping criterion on models that provides savings without terminating promising jobs.
Truncation selection cancels a given percentage of lowest performing runs at each evaluation interval. Runs are compared based on their performance on the primary metric and the lowest X% are terminated.
Example:
from azureml.train.hyperdrive import TruncationSelectionPolicy early_termination_policy = TruncationSelectionPolicy(evaluation_interval=1, truncation_percentage=20, delay_evaluation=5)
Incorrect Answers:
Bandit is a termination policy based on slack factor/slack amount and evaluation interval. The policy early terminates any runs where the primary metric is not within the specified slack factor / slack amount with respect to the best performing training run.
Example:
from azureml.train.hyperdrive import BanditPolicy
early_termination_policy = BanditPolicy(slack_factor = 0.1, evaluation_interval=1, delay_evaluation=5
Reference:
https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters

by Lucario95 at May 19, 2021, 7:37 a.m.

Comments

Submit Cancel

james2033

8 months, 3 weeks ago

Code snippet in structure with 3 steps: 1) from ... 2) import ... 3) call method Has only "from...." from choices list --> choose "from azureml.train.hyperdrive" without any afraid. Next, "BanditPolicy" or ""TruncationSelectionPolicy"? all of two for "early termination policy". https://learn.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.banditpolicy?view=azure-ml-py https://learn.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.truncationselectionpolicy?view=azure-ml-py Keywords in the question "must implement an early stopping criterion on models that provides savings without terminating promising jobs." SAVING WITHOUT TERMINATING PROMISSING JOBS. See comparig from Google Bard https://g.co/bard/share/09f818b51866 . Choose TruncationSeclectionPloicy

upvoted 1 times

...

phdykd

1 year, 4 months ago

from azureml.train.hyperdrive, import BanditPolicy, early_termination_policy= BanditPolicy(). TruncationSelectionPolicy does have an option to cancel the lowest performing runs. However, this policy does not take into account the performance of previous runs, so it may not be the most effective early stopping policy. The policy simply cancels runs if their performance falls below a certain threshold. In contrast, the BanditPolicy considers the performance of previous runs when deciding which runs to stop early. It uses a Bayesian approach to estimate the best performing configuration and stops runs that are unlikely to perform better than the current best configuration. This makes it a more effective early stopping policy in many cases. So while the TruncationSelectionPolicy may be a valid option for some situations, the BanditPolicy is generally a better choice for implementing an effective early stopping policy.

upvoted 3 times

...

Lucario95

3 years, 1 month ago

Could anyone explain why the correct answer is Truncation instead of Bandit policy?

upvoted 1 times

prashantjoge

3 years, 1 month ago

For a conservative policy that provides savings without terminating promising jobs, consider a Median Stopping Policy with evaluation_interval 1 and delay_evaluation 5. These are conservative settings, that can provide approximately 25%-35% savings with no loss on primary metric (based on our evaluation data). For more aggressive savings, use Bandit Policy with a smaller allowable slack or Truncation Selection Policy with a larger truncation percentage.

upvoted 4 times

...

audun

2 years, 3 months ago

it says "NOTE: More than one order of answer choices is correct." considering there's really only two combinations that would work we can assume both are correct

upvoted 2 times

...

azure1000

2 years, 11 months ago

Because question says "this configuration should cancel the lowest performing runs", which only happens in truncation

upvoted 4 times

lcgcastro96

1 year ago

wdym? Bandit Policy cancels runs whose best results fall below a certain mark (related with the goal and the slack factor), so it also does that. Besides that, Truncation will stop runs that fall below a certain threshold right away, going against the requirement of the case study which reinforces that promising runs should not be terminated

upvoted 1 times

lcgcastro96

1 year ago

PS - I understand that the main core of Truncation is canceling the lowest performing runs per se, my point is just that Bandit Policy also does it but "less blindly" if this makes sense, so the possible solution can be either imo

upvoted 1 times

...

silva_831

1 year, 7 months ago

You're right. This phrase was mentioned in the section Hyperparameter.

upvoted 1 times

...