Exam AI-100 topic 4 question 20 discussion

The answer should be no. Data collection monitors the input data drift over time and gives an indication when to review and retrain the AI model. no statement can be made about the compute performance. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-enable-data-collection

B. No Enabling Model data collection does not directly address the goal of monitoring the compute performance of the AI models deployed in a production environment. Model data collection is a feature in Azure Machine Learning that allows you to collect data about model invocations, inputs, and outputs for the purpose of model monitoring and analysis. It helps you understand how the model is performing and identify any issues or deviations from expected behavior. However, to monitor the compute performance of the models, you would need to utilize other monitoring mechanisms, such as Azure Monitor, which provides comprehensive monitoring and diagnostics capabilities for Azure resources, including Azure Machine Learning. Azure Monitor can help you track and analyze various performance metrics, such as CPU and memory usage, request latency, and resource utilization.

The answer is correct https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment Capture the governance data required for capturing the end-to-end ML lifecycle Azure ML gives you the capability to track the end-to-end audit trail of all of your ML assets by using metadata. Azure ML integrates with Git to track information on which repository / branch / commit your code came from. Azure ML Datasets help you track, profile, and version data. Interpretability allows you to explain your models, meet regulatory compliance, and understand how models arrive at a result for given input. Azure ML Run history stores a snapshot of the code, data, and computes used to train a model. The Azure ML Model Registry captures all of the metadata associated with your model (which experiment trained it, where it is being deployed, if its deployments are healthy). Integration with Azure allows you to act on events in the ML lifecycle. For example, model registration, deployment, data drift, and training (run) events.

I really wish this test was more straight forward. I've looked into data collection and application insights, but "compute performance of the models" is too ambiguous. Are they saying the actual performance of the compute i.e. how its resources are being used? Or are they asking about the performance of the models? Even with time to study the possible answers, this question is confusing.

Actually, I think compute performance of the models means how well the model is performing it's own task

For computer performance, it has to be configuring/enabling Azure Monitor, so it's a no for me. https://docs.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning

I feel the given answer is correct. From the Azure Application Insights page, you can use Application Insights for monitoring output data, responses, requests stats, exceptions, failure rates, dependency rates, etc. Although "output data" sounds convincing, this is very different from "compute performance", which is what we want to monitor. Collecting output data doesn't necessarily tell us about model performance. Model data collection, on the other hand, allows for tracking "data drift", which is how the performance of your model is changing over time, so I would say this is probably what is meant by monitoring "compute performance". Anyway, that's just my intepretation. Source: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets