# Alerts

> What your alert rules found, where alerts are sent, and Telopi's built-in alerts.

## Make alert rules

Alert rules live in your Grafana. Open **Alerting → Alert rules → New alert
rule** and make a *data source-managed* rule on **Mimir** (your metrics) or
**Loki** (your logs). Use any namespace except `telopi`, which is Telopi's.

## See what fired

The portal's **Alerts** page shows what your rules found, for every role:

- **Firing now**, with each alert's severity and the labels that tell one
  firing from another.
- **Last 30 days**: every alert that started in that time, and how long it
  lasted.

When Telopi's agent investigates an alert, the alert links to its findings.

**Manage rules in Grafana** opens your rules. If the page says **Not switched
on here yet**, your rules still run, and their alerts are in your Grafana.

## Where alerts go

**Where alerts go** sends every alert on to up to ten destinations, each with a
name you choose:

| Destination | What you give |
| --- | --- |
| Email | Up to ten addresses, one a line. |
| Slack | An [incoming webhook](https://api.slack.com/messaging/webhooks) from your Slack workspace, for the channel alerts should reach. |
| PagerDuty | The Integration Key of an Events API v2 integration on your service. |
| Microsoft Teams | A channel workflow's webhook URL ("Send webhook alerts to a channel"). |
| Webhook | An `https` address on port 443 that takes Alertmanager's JSON, optionally with a bearer token or a username and password. |
| Discord | A channel's webhook URL. |
| Telegram | A bot token from @BotFather and the chat's numeric ID. Add the bot to the chat first. |

Owners and admins **Add a destination**, switch one off and on, **Send test**
to see a test alert arrive, and **Remove** one. Everyone else sees the list.
An email destination can be sent a test once every 5 minutes.

Every URL, key, token and password is kept encrypted and never shown again,
only a hint such as a webhook's host or the last four characters. To change
one, remove the destination and add it again. Webhook and Teams addresses must
be on the public internet.

Every destination that is on gets every alert. Alerts of one rule arrive
together, and one that keeps firing is sent again every 4 hours. Each
destination is also told when an alert resolves. A change takes effect right
away; if it cannot be applied at once, we retry every 5 minutes.

When Telopi's agent has investigated an alert, its root cause follows to the
same destinations, except PagerDuty, which has paged already.

## Built-in alerts

Telopi keeps sets of rules for every customer, beside your own, in your
Grafana's `telopi` namespace. A change made there by hand is put back within
five minutes. Among them:

| Set | What it watches |
| --- | --- |
| Uptime | A checked site that stops answering, and a certificate about to expire. |
| Website | JavaScript errors, and Core Web Vitals turning poor. |
| Traces | A service whose requests fail, or are slow. |
| Logs | A service that suddenly logs many errors. |
| Machines | A machine that stops sending, or runs out of disk, memory or CPU. |
| Kubernetes | Pods crash-looping, unable to pull their image or not starting; deployments and stateful sets short of pods; daemon sets missing on some nodes; containers out of memory or held back by their CPU limit; failed jobs; nodes not ready; volumes nearly full. Each alert names its environment. |

An integration you connect can bring a set of its own. A set fires only on
telemetry your environments send. Owners and admins switch
a set off with **Send these alerts**, or change its thresholds. A threshold is
a number such as `5` or a duration such as `5m`; leave it empty to keep the
default, which is shown beside it. Everyone else sees the sets.