Measuring the Impact of Azure AI Services

Welcome back to our blog series on Azure AI Services. Our earlier posts discussed implementing Azure AI Services and the best practices to follow. Today, we will discuss tools to measure the impact of Azure AI Services on your business and how to interpret these metrics.

Tools to Measure the Impact

Azure supplies several tools to help you measure the impact of AI Services on your business. These include:

  1. Azure Monitor: This service supplies full-stack monitoring, allowing you to collect, analyze, and act on telemetry data from your Azure and on-premises environments.
  2. Azure Application Insights: This is an extensible application performance management (APM) service that can help you understand the performance and usage of your live web applications.
  3. Azure Log Analytics: This service helps you collect and analyze data generated by resources in your cloud and on-premises environments.

Interpreting the Metrics

Interpreting the metrics involves understanding each metric and how it relates to the performance and impact of Azure AI Services on your business. Here are a few tips:

  1. Understand the Metrics: Each tool supplies different metrics. Understand what each metric stands for. For example, the number of API calls might stand for the usage of a service, while the response time might represent the performance of a service.
  2. Analyze Trends: Look for trends in the metrics. For example, increasing API calls might show increasing service usage.
  3. Correlate Metrics: Try to correlate different metrics. For example, if the response time increases as the number of API calls increases, it might show that the service is struggling to handle the load.

Microsoft Learn has some good guidance for performance monitoring. For example, for Azure Search, see the article Analyze performance – Azure AI Search. Here are examples of performance metrics used to measure the success of various Azure AI Services:

Azure OpenAI

  • HTTP Requests: The number of HTTP requests made to the service.
  • Tokens-Based Usage: The number of tokens used in the requests.
  • PTU Utilization: The PTU (Premium Turing Units) use for the service.
  • Fine-tuning Data: The data used for fine-tuning the models.

Azure AI Search

  • Query Performance: Latency and throughput performance metrics.
  • Indexing Volume: The volume of data indexed by the service.

Azure Vision

  • Precision: The percentage of identified classifications that were correct.
  • Recall: Actual classifications correctly identified (percentage).
  • Mean Average Precision (mAP): The average value of the average precision.

Azure Speech

  • Latency: The time taken to transcribe speech to text.
  • Requests Per Second (RPS): The number of requests made to the service per second.

Azure Language

  • Precision: Measures how precise/accurate your model is.
  • Recall: Measures the model’s ability to predict actual positive classes.
  • F1 score: The F1 score is a function of Precision and Recall.

Azure Translator

  • Translation Accuracy: The accuracy of the translations provided by the service6.
  • Latency: The time taken to translate text.

Azure Video Indexer

  • Transcription Accuracy: The accuracy of the transcriptions provided by the service8.
  • Speaker Recognition Accuracy: The accuracy of the speaker recognition feature8.

Azure Immersive Reader

  • User Engagement: The number of users engaging with the service and the duration of their engagement.
  • Reading Comprehension Improvement: The improvement in reading comprehension for service users.

Azure Content Safety

  • Severity Indication: A unique ‘Severity’ metric that shows the severity of specific content on a scale ranging from 0 to 7.
  • Technical metrics (latency, accuracy, recall) and business metrics (block rate, block volume, category proportions, language proportions, and more).

We hope this post has helped you understand how to measure the impact of Azure AI Services on your business and the metrics to consider for monitoring these services. As we progress through this series, we aim to provide you with a comprehensive understanding of these services and how they can help your business. Remember, the future of your business could be powered by AI, and with Azure AI Services, that future is within your reach.