Custom Prometheus Alert Rules & Grafana Dashboards
23 minAdvanced
This creates billable resources. Run it in a dev environment and destroy it when you finish. Set a budget alarm first.
Success criteria
0 of 1
What you are building#
What are PrometheusRules & ServiceMonitors?#
- PrometheusRule: Custom Kubernetes CRD defining alert threshold expressions evaluated dynamically against scraped time-series metrics.
- ServiceMonitor: CRD telling Prometheus which Kubernetes services to discover and scrape for metrics.
- Grafana Dashboard JSON: Declarative visualization layout displaying RED metrics (Request Rate, Error Rate, Duration/Latency).
Steps#
Step 1: Apply Alert Rules & ServiceMonitor#
Terminal
cd 07-Monitoring-Observability/Lab18-Alerts-Grafana-Dashboards
kubectl apply -f alerts/nti-devops-rules.yaml -f alerts/servicemonitor.yamlVerify it worked#
Terminal
kubectl get prometheusrule -n monitoringExpected Output:
text
NAME AGE
nti-devops-alert-rules 1mClean up#
Run this even if you did not finish. Everything above is destroyable, and an account full of half-built experiments is how a surprise bill starts.
Destructive — This removes real resources. Check which environment you are in first.
Terminal
kubectl delete prometheusrule --all -n monitoring
kubectl delete configmap -l grafana_dashboard -n monitoringCost of this lab: Depends on an existing cluster. Alert rules and dashboards are configuration and cost nothing; the Prometheus stack under them holds EBS volumes.