Alerts
What your alert rules found, where alerts are sent, and Telopi's built-in alerts.
Make alert rules
Alert rules live in your Grafana. Open Alerting → Alert rules → New alert
rule and make a data source-managed rule on Mimir (your metrics) or
Loki (your logs). Use any namespace except telopi, which is Telopi's.
See what fired
The portal's Alerts page shows what your rules found, for every role:
- Firing now, with each alert's severity and the labels that tell one firing from another.
- Last 30 days: every alert that started in that time, and how long it lasted.
When Telopi's agent investigates an alert, the alert links to its findings.
Manage rules in Grafana opens your rules. If the page says Not switched on here yet, your rules still run, and their alerts are in your Grafana.
Where alerts go
Where alerts go sends every alert on to up to ten destinations, each with a name you choose:
| Destination | What you give |
|---|---|
| Up to ten addresses, one a line. | |
| Slack | An incoming webhook from your Slack workspace, for the channel alerts should reach. |
| PagerDuty | The Integration Key of an Events API v2 integration on your service. |
| Microsoft Teams | A channel workflow's webhook URL ("Send webhook alerts to a channel"). |
| Webhook | An https address on port 443 that takes Alertmanager's JSON, optionally with a bearer token or a username and password. |
| Discord | A channel's webhook URL. |
| Telegram | A bot token from @BotFather and the chat's numeric ID. Add the bot to the chat first. |
Owners and admins Add a destination, switch one off and on, Send test to see a test alert arrive, and Remove one. Everyone else sees the list. An email destination can be sent a test once every 5 minutes.
Every URL, key, token and password is kept encrypted and never shown again, only a hint such as a webhook's host or the last four characters. To change one, remove the destination and add it again. Webhook and Teams addresses must be on the public internet.
Every destination that is on gets every alert. Alerts of one rule arrive together, and one that keeps firing is sent again every 4 hours. Each destination is also told when an alert resolves. A change takes effect right away; if it cannot be applied at once, we retry every 5 minutes.
When Telopi's agent has investigated an alert, its root cause follows to the same destinations, except PagerDuty, which has paged already.
Built-in alerts
Telopi keeps sets of rules for every customer, beside your own, in your
Grafana's telopi namespace. A change made there by hand is put back within
five minutes. Among them:
| Set | What it watches |
|---|---|
| Uptime | A checked site that stops answering, and a certificate about to expire. |
| Website | JavaScript errors, and Core Web Vitals turning poor. |
| Traces | A service whose requests fail, or are slow. |
| Logs | A service that suddenly logs many errors. |
| Machines | A machine that stops sending, or runs out of disk, memory or CPU. |
| Kubernetes | Pods crash-looping, unable to pull their image or not starting; deployments and stateful sets short of pods; daemon sets missing on some nodes; containers out of memory or held back by their CPU limit; failed jobs; nodes not ready; volumes nearly full. Each alert names its environment. |
An integration you connect can bring a set of its own. A set fires only on
telemetry your environments send. Owners and admins switch
a set off with Send these alerts, or change its thresholds. A threshold is
a number such as 5 or a duration such as 5m; leave it empty to keep the
default, which is shown beside it. Everyone else sees the sets.