Monitoring is an important architectural concern that should be part of any solution, big or small, mission‑critical or not, cloud‑based or not—it should not be neglected.
Monitoring refers to the act of keeping track of solutions and capturing various telemetry information, processing it, identifying the information that qualifies for alerts based on rules, and raising them. Generally, an agent is deployed within the environment and monitors it, sending telemetry information to a centralized server, where the rest of the processing of generating alerts and notifying stakeholders takes place.
Monitoring takes both proactive and reactive actions and measures against a solution. It is also the first step toward auditing a solution. Without the ability to monitor log records, it is difficult to audit a system from various perspectives, such as security, performance, and availability.
Monitoring helps us identify availability, performance, and scalability issues before they arise. Hardware failure, software misconfiguration, and patch update challenges can be discovered well before they impact users through monitoring, and performance degradation can be fixed before it happens.
Monitoring reactively logs pinpoint areas and locations that are causing issues,
identifies the issues, and enables faster and better repairs.
Teams can identify patterns of issues using monitoring telemetry information and eliminate them by innovating new solutions and features.
Azure is a rich cloud environment that provides multiple rich monitoring features and resources to monitor not only cloud‑based deployment but also on‑premises deployment.
Azure monitoring
The first question that should be answered is, “What must we monitor?” This question becomes more important for solutions that are deployed on the cloud because of the constrained control over them.
There are some important components that should be monitored. They include the following:
- Custom applications
- Azure resources
- Guest OSes (VMs)
- Host OSes (Azure physical servers)
- Azure infrastructure
There are different Azure logging and monitoring services for these components, and they are discussed in the following sections.
Azure activity logs
Previously known as audit logs and operational logs, activity logs are control‑plane events on the Azure platform. They provide information and telemetry information at the subscription level, instead of the individual resource level. They track information about all changes that happen at the subscription level, such as creating, deleting, and updating resources using Azure Resource Manager (ARM). Activity logs help
us discover the identity of (such as service principal, users, or groups), and perform actions on (such as write or update), resources (for example, storage, virtual machines, or SQL databases) at any given point in time. They provide information about resources that are modified in their configuration, but not their inner workings and execution. For example, you can get the logs for starting a VM, resizing a VM, or stopping a VM.
The next topic that we are going to discuss is diagnostic logs.
Azure diagnostic logs
The information originating within the inner workings of Azure resources is captured in what are known as diagnostic logs. They provide telemetry information about the operations of resources that are inherent to the resources. Not every resource provides diagnostic logs, and resources that provide logs on their own content are completely different from other resources. Diagnostic logs are configured individually for each resource. Examples of diagnostic logs include storing a file in a container in a blob in a storage account.
The next type of log that we are going to discuss is application logs.
Azure application logs
Application logs can be captured by Application Insights resources and can be managed centrally. They get information about the inner workings of custom applications, such as their performance metrics and availability, and users can get insights from them in order to manage them better.
Lastly, we have guest and host OS logs. Let’s understand what these are.
Guest and host OS logs
Both guest and host OS logs are offered to users using Azure Monitor. They provide information about the statuses of host and guest OSes:
Figure 2.16: Logging in Azure
The important Azure resources related to monitoring are Azure Monitor, Azure Application Insights, and Log Analytics, previously known as Operational Insights.
There are other tools, such as System Center Operations Manager (SCOM), that are not part of the cloud feature but can be deployed on IaaS‑based VMs to monitor any workload on Azure or an on‑premises datacenter. Let’s discuss the three monitoring resources in the following section.
Azure Monitor
Azure Monitor is a central tool and resource that provides complete management features that allow you to monitor an Azure subscription. It provides management features for activity logs, diagnostic logs, metrics, Application Insights, and Log Analytics. It should be treated as a dashboard and management resource for all other monitoring capabilities.
Our next topic is Azure Application Insights.
Azure Application Insights
Azure Application Insights provides centralized, Azure‑scale monitoring, logs, and metrics capabilities to custom applications. Custom applications can send metrics, logs, and other telemetry information to Azure Application Insights. It also provides rich reporting, dashboarding, and analytics capabilities to get insights from incoming data and act on them.
Now that we have covered Application Insights, let’s look at another similar service called Azure Log Analytics.
Azure Log Analytics
Azure Log Analytics enables the centralized processing of logs and generates insights and alerts from them. Activity logs, diagnostic logs, application logs, event logs, and even custom logs can send information to Log Analytics, which can further provide rich reporting, dashboarding, and analytics capabilities to get insights from incoming data and act on them.
Now that we know the purpose of Log Analytics, let’s discuss how logs are stored in a Log Analytics workspace and how they can be queried.
Logs
A Log Analytics workspace provides search capabilities to search for specific log entries, export all telemetry data to Excel and/or Power BI, and search a query language called Kusto Query Language (KQL), which is similar to SQL.
The Log Search screen is shown here:
Figure 2.17: Log search in a Log Analytics workspace
In the next section, we will be covering Log Analytics solutions, which are like additional capabilities in a Log Analytics workspace.
Solutions
Solutions in Log Analytics are further capabilities that can be added to a workspace, capturing additional telemetry data that is not captured by default. When these solutions are added to a workspace, appropriate management packs are sent to all the agents connected to the workspace so that they can configure themselves to capture solution-specific data from VMs and containers and then send it to the Log Analytics workspace. Monitoring solutions from Microsoft and partners are available from Azure Marketplace.
Azure provides lots of Log Analytics solutions for tracking and monitoring different aspects of environments and applications. At a minimum, a set of solutions that are generic and applicable to almost any environment should be added to the workspace:
- Capacity and performance
- Agent health
- Change tracking
- Containers
- Security and audit
- Update management
- Network performance monitoring