# Alerts & Notifications > Configure alert rules with threshold monitoring and multi-channel notifications. URL: https://docs.serverbee.app/en/docs/alerts ServerBee includes a flexible alerting system that evaluates metric thresholds and security events, then sends notifications through multiple channels when conditions are met. ## How Alerts Work [#how-alerts-work] A background task evaluates all enabled rules every 60 seconds: 1. Resolve which servers each rule covers. 2. Check whether the rule's conditions are met for each covered server. 3. Send a notification when a rule triggers (subject to debounce). 4. Clear the alert state when a previously triggered rule recovers. Alert state is persisted in the database and survives server restarts. Event-driven rules (IP change, SSH login, brute-force, port scan, temporary capability grant) are evaluated when an agent reports the event, not on the 60-second cycle. ## Creating Alert Rules [#creating-alert-rules] An alert rule consists of: * **Name** -- A descriptive label. * **Rules** -- One or more metric conditions. All conditions must be true simultaneously (AND logic). * **Trigger mode** -- `always` (repeat with debounce) or `once` (notify only on the first trigger). * **Cover type** -- Which servers the rule applies to. * **Notification group** -- Where notifications are sent. * **Trigger/recover tasks** -- Optional. Remote commands run automatically when the rule triggers or recovers. * **Block source IP** -- Optional. For security-event rules, automatically instructs the agent's firewall to block the offending source IP when the rule triggers. ## Supported Metric Types [#supported-metric-types] ### Resource Thresholds [#resource-thresholds] | Rule Type | Description | `min` Threshold Meaning | | --------------- | ------------------------ | ---------------------------- | | `cpu` | CPU usage percentage | Triggers when usage >= value | | `memory` | Memory used (bytes) | Triggers when usage >= value | | `swap` | Swap used (bytes) | Triggers when usage >= value | | `disk` | Disk used (bytes) | Triggers when usage >= value | | `load1` | 1-minute load average | Triggers when load >= value | | `load5` | 5-minute load average | Triggers when load >= value | | `load15` | 15-minute load average | Triggers when load >= value | | `temperature` | CPU temperature (C) | Triggers when temp >= value | | `gpu` | GPU utilization (%) | Triggers when usage >= value | | `tcp_conn` | TCP connections | Triggers when count >= value | | `udp_conn` | UDP connections | Triggers when count >= value | | `process` | Process count | Triggers when count >= value | | `net_in_speed` | Download speed (bytes/s) | Triggers when speed >= value | | `net_out_speed` | Upload speed (bytes/s) | Triggers when speed >= value | ### Traffic Cycle [#traffic-cycle] Used to monitor cumulative traffic over a time window: | Rule Type | Description | | -------------------- | --------------------------------------------- | | `transfer_in_cycle` | Inbound traffic accumulated per cycle | | `transfer_out_cycle` | Outbound traffic accumulated per cycle | | `transfer_all_cycle` | Combined inbound + outbound traffic per cycle | Cycle options: `hour` / `day` / `week` / `month` / `year` ### Network Quality [#network-quality] | Rule Type | Description | | --------------------- | -------------------------------------------------------------- | | `network_latency` | Triggers when average probe latency (ms) exceeds the threshold | | `network_packet_loss` | Triggers when the packet loss percentage exceeds the threshold | ### Offline, Expiration & Events [#offline-expiration--events] | Rule Type | Description | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `offline` | Triggers when the server has been offline for `duration` seconds | | `expiration` | Triggers when `expired_at` is within `duration` days | | `ip_changed` | Event-driven; triggers when the agent reports an IP change event (not part of the 60-second cycle) | | `ssh_login_detected` | Event-driven; successful SSH login. Supports `first_seen_only` filter. See [Security Events](/en/docs/security-events) | | `ssh_brute_force_detected` | Event-driven; SSH brute-force burst. Supports `severity_min` (medium/high/critical) and `exclude_cidrs` | | `port_scan_detected` | Event-driven; port scan from a single source IP. Supports `severity_min` and `exclude_cidrs` | | `capability_grant_detected` | Event-driven; fires when a high-risk capability (`terminal` / `exec` / `file` / `docker`) is **temporarily granted** on a server via the host-local `serverbee-agent grant` CLI. Expiry, revoke, and low-risk grants are audited but do not alert. See [Capabilities → Temporary grants](/en/docs/capabilities#temporary-grants) | The Alerts page offers a one-click **preset card** for `capability_grant_detected` ("Capability Temporarily Granted") with sensible defaults; the rule type is also selectable directly in the rule editor. It carries no `severity_min` / `exclude_cidrs` filters and no auto-block action — it fires on any high-risk temporary grant — but it follows the same cover-type and maintenance-suppression behavior as the other event-driven rules. ## Threshold Configuration [#threshold-configuration] Each rule item supports these fields: ```json { "rule_type": "cpu", "min": 90.0, "max": null, "duration": null, "cycle_interval": null, "cycle_limit": null } ``` * **`min`** -- Lower bound. For most metrics, the alert triggers when the value is at or above this threshold. * **`max`** -- Upper bound. When both `min` and `max` are set, the alert triggers when the value falls within the range. * **`duration`** -- Used by `offline` (seconds before triggering) and `expiration` (days threshold). * **`cycle_interval`** -- Time window for transfer cycle rules: `hour`, `day`, `week`, `month`, `year`. * **`cycle_limit`** -- Traffic limit in bytes for transfer cycle rules. ### Examples [#examples] **CPU over 90%:** ```json { "rule_type": "cpu", "min": 90.0 } ``` **Memory over 8 GB:** ```json { "rule_type": "memory", "min": 8589934592 } ``` **Server offline for more than 2 minutes:** ```json { "rule_type": "offline", "duration": 120 } ``` **Monthly outbound traffic over 1 TB:** ```json { "rule_type": "transfer_out_cycle", "cycle_interval": "month", "cycle_limit": 1099511627776 } ``` **Server expiring within 7 days:** ```json { "rule_type": "expiration", "duration": 7 } ``` **Multiple conditions (AND logic):** a rule with several conditions triggers only when all of them hold. The following fires when CPU usage is at or above 90% *and* memory usage is at or above 8 GB: ```json [ { "rule_type": "cpu", "min": 90.0 }, { "rule_type": "memory", "min": 8589934592 } ] ``` ## Sampling and Trigger Logic [#sampling-and-trigger-logic] For resource threshold alerts (CPU, memory, disk, load, etc.), ServerBee does not trigger on a single spike: 1. The evaluator reads all raw metric records from the **last 10 minutes** (one sample per minute, so up to 10 samples). 2. It counts how many samples exceed the threshold. 3. The alert triggers only if **70% or more** of the samples exceed the threshold (at least 7 of 10). This prevents false positives from brief, transient spikes. ## Cover Types [#cover-types] Each alert rule specifies which servers it applies to: | Cover Type | Behavior | | ---------- | ----------------------------------------------- | | `all` | Applies to every server in the system | | `include` | Applies only to the specified server IDs | | `exclude` | Applies to all servers except the specified IDs | ## Trigger Modes and Debounce [#trigger-modes-and-debounce] ### `always` Mode (Default) [#always-mode-default] * Sends a notification every time the condition is evaluated as true. * **5-minute debounce** prevents notification spam: for the same (rule + server) combination, after a notification is sent, the next one is suppressed for 5 minutes even if the condition persists. ### `once` Mode [#once-mode] * Sends a notification only on the **first trigger**. * No further notifications are sent until the condition recovers and triggers again. ### Renewal Expiration Reminders [#renewal-expiration-reminders] For a server with a stored renewal occurrence, an expiration rule in `once` mode sends at most one reminder for each rule, server and occurrence. A new monthly deadline can therefore send its own reminder even when a 60-day window stays continuously true. Retries, Server restart and disabling automatic tracking do not resend the same occurrence. The threshold uses the stored billing timezone's local expiry date. A zero-day reminder remains eligible throughout that date. Automatic tracking catches up to the current deadline after downtime without sending a backlog for missed occurrences. Previous automatic targets appear as **Superseded** in alert history; advancement sends no recovery notification and does not confirm provider renewal, payment or service availability. An explicitly edited date or a condition that stops matching can still produce the normal recovery transition. Authenticated Web and native iOS views refetch the Server-owned deadline, alert history and cost advisories after catalog changes and reconnects. Clients do not calculate recurring deadlines. ### State Persistence [#state-persistence] Trigger state is persisted to the `alert_states` table in SQLite and loaded into a hot cache on startup. As a result: * A `once` rule that is already triggered will not re-fire after a server restart. * Triggered-but-not-yet-recovered alerts are restored automatically after a restart. ### Recovery [#recovery] When a previously triggered alert recovers (the condition is no longer met): 1. The alert is marked as recovered, and its state is cleared in both the in-memory cache and the database. 2. If recover tasks (`recover_trigger_tasks`) are configured, the corresponding remote commands run. 3. If the condition triggers again later, notifications fire according to the trigger mode. ## Maintenance Suppression [#maintenance-suppression] Alert notifications are suppressed while the affected server is in an active maintenance window. Rule evaluation still runs, but ServerBee skips notification delivery for that server until the window ends. Event-driven rules follow the same cover-type and maintenance suppression rules as polling-based rules. ## Blocking Source IPs [#blocking-source-ips] Security-event rules (`ssh_brute_force_detected`, `port_scan_detected`) can enable **Block source IP**. When the rule triggers, ServerBee tells the affected agent's firewall to block the offending IP, turning detection into automatic mitigation. This requires the agent to have the `firewall_block` capability (`CAP_FIREWALL_BLOCK`), which is on by default but can be disabled on the agent host via `[capabilities]`. See [Security Events](/en/docs/security-events) and [Firewall](/en/docs/firewall). ## Notification Channels [#notification-channels] ServerBee supports five notification channel types. Each channel is a separate entity that can be reused across multiple notification groups. ### Webhook [#webhook] Send HTTP requests to any URL. Supports custom methods, headers, and body templates. ```json { "url": "https://hooks.slack.com/services/xxx", "method": "POST", "headers": { "Content-Type": "application/json" }, "body_template": "{\"text\": \"{{server_name}} {{event}}: {{message}}\"}" } ``` If no `body_template` is provided, the default template is used. If no `Content-Type` header is set, it defaults to `application/json`. ### Telegram [#telegram] Send messages to a Telegram chat via the Bot API. Messages are sent with HTML parse mode enabled. ```json { "bot_token": "123456:ABC-DEF", "chat_id": "-1001234567890" } ``` ### Bark [#bark] Send push notifications to iOS devices via [Bark](https://github.com/Finb/Bark). ```json { "server_url": "https://api.day.app", "device_key": "your-device-key" } ``` ### Email (via Resend) [#email-via-resend] Email notifications are delivered through [Resend](https://resend.com/). Two steps before use: 1. Set `SERVERBEE_RESEND__API_KEY` on the server (see the [Configuration](/en/docs/configuration) page). 2. Add and verify your sender domain at [resend.com/domains](https://resend.com/domains). The `from` address on each channel must belong to a verified domain. Channel config: ```json { "from": "alerts@yourdomain.com", "to": ["ops@example.com", "oncall@example.com"] } ``` `to` is an array — a single channel can deliver to multiple recipients in one API call. The subject follows the format `[ServerBee] {server_name} {event}`; the body is HTML with a plain-text fallback. ### APNs [#apns] Send native Apple Push Notification service pushes to registered mobile devices. ```json { "key_id": "ABC123DEFG", "team_id": "TEAM999888", "private_key": "-----BEGIN PRIVATE KEY-----...", "bundle_id": "com.example.serverbee", "sandbox": false } ``` APNs requires an Apple developer key, team ID, bundle ID, and private key. Set `sandbox: true` only for development builds. ### Template Variables [#template-variables] Notification messages support the following template variables: | Variable | Description | | ----------------- | -------------------------------- | | `{{server_name}}` | Name of the affected server | | `{{server_id}}` | Unique ID of the affected server | | `{{rule_name}}` | Name of the triggered alert rule | | `{{event}}` | Event type (e.g., "triggered") | | `{{message}}` | Human-readable alert message | | `{{time}}` | Timestamp of the event (UTC) | | `{{cpu}}` | Current CPU usage string | | `{{memory}}` | Current memory usage string | The default notification template is: ``` [ServerBee] {{server_name}} {{event}} {{message}} Time: {{time}} ``` ### Testing a Channel [#testing-a-channel] After creating a channel, verify its configuration with the test endpoint: ``` POST /api/notifications/:id/test ``` ## Notification Groups [#notification-groups] Notification channels are organized into **groups**. An alert rule is linked to a notification group, and when the rule triggers, all enabled channels in the group are dispatched. This allows you to: * Send the same alert to multiple channels (e.g., Telegram + Email). * Reuse channel configurations across different alert rules. * Enable or disable individual channels without modifying alert rules. ## Offline Detection [#offline-detection] Offline status is determined by a dedicated background task and works together with `offline` rules: * The task scans agent connection state every 10 seconds. * A server is marked offline when its last report is more than 30 seconds old. * The `offline` rule's `duration` sets how many seconds a server must stay offline before the alert fires. For example, `{ "rule_type": "offline", "duration": 60 }` triggers after 60 seconds offline. ## Example: Complete Alert Setup [#example-complete-alert-setup] Here is a typical setup for monitoring CPU usage across all servers with Telegram notifications: 1. **Create a Telegram notification channel** with your bot token and chat ID. 2. **Create a notification group** that includes the Telegram channel. 3. **Create an alert rule:** * Name: "High CPU Usage" * Rules: `[{"rule_type": "cpu", "min": 90.0}]` * Trigger mode: `always` * Cover type: `all` * Notification group: (the group you created) Now, when any server's CPU stays above 90% for at least 70% of a 10-minute sampling window, you will receive a Telegram message. Subsequent notifications are debounced to once every 5 minutes while the condition persists.