Azure Data Factory Monitoring Choice?
May 19, 2021
If you are using Azure Data Factory, you can export platform logs and metrics to a log analytics workspace. Sending logs and metrics to a Log Analytics workspace allows you to analyze them with other monitoring data collected by Azure Monitor using powerful log queries and leveraging other Azure Monitor features such as alerts and visualizations.
You have 2 choices for destination table:
- Azure diagnostics
- Resource specific
Which one should you use?
‘Azure diagnostics’ mode collects all of the logs into a single table. There are limits to the number of columns supported by the Azure Monitor Service Log Analytics Workspace of 500 columns. As each resource has its own metrics, the unique columns are appended to the table; there is potential to violate the 500 columns hard limit.
‘Resource specific’ create individual tables in the workspace. This destination is recommended since it makes it much easier to work with the data in log queries, provides better discoverability of schemas and structure, improves performance across both ingestion latency and query times, and grants Azure RBAC rights on a specific table. All Azure services will eventually migrate to the ‘Resource specific’ mode.
This documentation compares the two destination tables.
Information about the Azure Monitor service limits, including the number of columns, is here.
Does both options work with the ‘AzureDataFactoryAnalytics’ solution?
The Azure Data Factory Analytics will work with either destination ‘Azure diagnostics’ or ‘Resource specific’.
Monitor data factories using Azure Monitor – Azure Data Factory | Microsoft Docs
Summary
All Azure services will eventually use the ‘Resource specific’ mode. As part of this transition, some resources still allow you to select either destination in the Diagnostic setting. Specify the ‘Resource specific’ mode for any new diagnostic settings since this makes the data easier to manage and may help you avoid complex migrations later.
Credit
Chris Coulthrust from Microsoft helped me out by answering some of my questions.
