How to create an alert rule in Grafana 10.1
You may have built an alert rule with Grafana Alerting and then grappled with routing, reconfiguring, and managing the different alerts your team set up. To address this challenge, we’ve implemented a series of improvements to set up and maintain alert rules in Grafana. Watch how the new alerting workflow works. Grafana 10.1 release: https://grafana.com/blog/2023/08/24/grafana-10.1-release-all-the-latest-features/ Grafana Alerting documentation: https://grafana.com/docs/grafana/latest/alerting/
How to create an alert rule in Grafana 10.1
Introduction and Problem Statement
In this section, Sonia introduces herself as a software engineer on the ltin squad and discusses the problems that the new alert rule creation workflow in Grafana 10.1 aims to address.
Problems with Previous Workflow
- Users found certain steps, like choosing the type of rule to create, to be mandatory and added decision-making overhead.
- Opportunities for improvements were identified in the alert rule form to enhance user experience during the creation process.
- Confusion exists among users regarding how alerts are routed to specific contact points.
Improvements in Alert Rule Form
Sonia explains the changes made in the alert rule form to improve user experience during the creation process.
Changes Made
- The "New Alert Recording Rule" action button has been moved outside of the rule creation workflow since a recording rule is not an alert rule.
- Several improvements have been made in the alert rule form, including options for time range, max data points, min interval, and fixing expandability issues with queries and expressions.
- Expressions can no longer be edited; users can only remove or create new ones with their desired type.
Deciding Rule Type and Data Source Selection
Sonia discusses changes related to deciding the rule type and selecting a data source.
Changes Made
- Deciding the rule type is no longer mandatory. The first decision users need to make is selecting a data source.
- By default, rules will be Grafana-managed alerts. However, if a data source allows cloud rules, users can switch to cloud rules after selecting it.
Previewing Potential Alerts
Sonia introduces a new feature that enables users to preview how potential alerts will be routed based on the current alert definition and alert manager configuration.
Previewing Alerts
- Users can now preview how potential alerts will be routed depending on labels and the configured alert managers.
- This feature is only available for Grafana-managed alerts, as control over routing in cloud alerts is limited.
Reviewing Alert Manager Configuration
Sonia explains how to review the alert manager configuration and notification policies.
Alert Manager Configuration
- In the admin tab, under Grafana Alert Manager Configuration, both internal and external alert managers are listed.
- The external alert manager has a specific notification policy named "Magic Label" with different routing configurations based on label matching.
Routing of Potential Alerts
Sonia demonstrates how potential alerts are routed based on defined labels in the notification policies of both internal and external alert managers.
Routing Example
- For the internal alert manager, potential alerts without matching labels are sent to the default policy.
- For the external alert manager, an instance with a specific label ("magical label") is routed to a different notification policy named "Not Desired Level" and sent to a contact point named "Grafana Alerting."
Conclusion
Sonia concludes by inviting viewers to reach out to the alerting team if they have any questions or need further assistance.