Alerts & Notifications

ServerBee includes a flexible alerting system that evaluates metric thresholds and security events, then sends notifications through multiple channels when conditions are met.

How Alerts Work

A background task evaluates all enabled rules every 60 seconds:

1. Resolve which servers each rule covers.
2. Check whether the rule's conditions are met for each covered server.
3. Send a notification when a rule triggers (subject to debounce).
4. Clear the alert state when a previously triggered rule recovers.

Alert state is persisted in the database and survives server restarts. Event-driven rules (IP change, SSH login, brute-force, port scan, temporary capability grant) are evaluated when an agent reports the event, not on the 60-second cycle.

Creating Alert Rules

An alert rule consists of:

• Name -- A descriptive label.
• Rules -- One or more metric conditions. All conditions must be true simultaneously (AND logic).
• Trigger mode -- always (repeat with debounce) or once (notify only on the first trigger).
• Cover type -- Which servers the rule applies to.
• Notification group -- Where notifications are sent.
• Trigger/recover tasks -- Optional. Remote commands run automatically when the rule triggers or recovers.
• Block source IP -- Optional. For security-event rules, automatically instructs the agent's firewall to block the offending source IP when the rule triggers.

Supported Metric Types

Resource Thresholds

Traffic Cycle

Used to monitor cumulative traffic over a time window:

Cycle options: hour / day / week / month / year

Network Quality

Offline, Expiration & Events

The Alerts page offers a one-click preset card for capability_grant_detected ("Capability Temporarily Granted") with sensible defaults; the rule type is also selectable directly in the rule editor. It carries no severity_min / exclude_cidrs filters and no auto-block action — it fires on any high-risk temporary grant — but it follows the same cover-type and maintenance-suppression behavior as the other event-driven rules.

Threshold Configuration

Each rule item supports these fields:

{
  "rule_type": "cpu",
  "min": 90.0,
  "max": null,
  "duration": null,
  "cycle_interval": null,
  "cycle_limit": null
}

• min -- Lower bound. For most metrics, the alert triggers when the value is at or above this threshold.
• max -- Upper bound. When both min and max are set, the alert triggers when the value falls within the range.
• duration -- Used by offline (seconds before triggering) and expiration (days threshold).
• cycle_interval -- Time window for transfer cycle rules: hour, day, week, month, year.
• cycle_limit -- Traffic limit in bytes for transfer cycle rules.

Examples

CPU over 90%:

{ "rule_type": "cpu", "min": 90.0 }

Memory over 8 GB:

{ "rule_type": "memory", "min": 8589934592 }

Server offline for more than 2 minutes:

{ "rule_type": "offline", "duration": 120 }

Monthly outbound traffic over 1 TB:

{
  "rule_type": "transfer_out_cycle",
  "cycle_interval": "month",
  "cycle_limit": 1099511627776
}

Server expiring within 7 days:

{ "rule_type": "expiration", "duration": 7 }

Multiple conditions (AND logic): a rule with several conditions triggers only when all of them hold. The following fires when CPU usage is at or above 90% and memory usage is at or above 8 GB:

[
  { "rule_type": "cpu", "min": 90.0 },
  { "rule_type": "memory", "min": 8589934592 }
]

Sampling and Trigger Logic

For resource threshold alerts (CPU, memory, disk, load, etc.), ServerBee does not trigger on a single spike:

1. The evaluator reads all raw metric records from the last 10 minutes (one sample per minute, so up to 10 samples).
2. It counts how many samples exceed the threshold.
3. The alert triggers only if 70% or more of the samples exceed the threshold (at least 7 of 10).

This prevents false positives from brief, transient spikes.

Cover Types

Each alert rule specifies which servers it applies to:

Trigger Modes and Debounce

always Mode (Default)

• Sends a notification every time the condition is evaluated as true.
• 5-minute debounce prevents notification spam: for the same (rule + server) combination, after a notification is sent, the next one is suppressed for 5 minutes even if the condition persists.

once Mode

• Sends a notification only on the first trigger.
• No further notifications are sent until the condition recovers and triggers again.

Renewal Expiration Reminders

For a server with a stored renewal occurrence, an expiration rule in once mode sends at most one reminder for each rule, server and occurrence. A new monthly deadline can therefore send its own reminder even when a 60-day window stays continuously true. Retries, Server restart and disabling automatic tracking do not resend the same occurrence.

The threshold uses the stored billing timezone's local expiry date. A zero-day reminder remains eligible throughout that date. Automatic tracking catches up to the current deadline after downtime without sending a backlog for missed occurrences. Previous automatic targets appear as Superseded in alert history; advancement sends no recovery notification and does not confirm provider renewal, payment or service availability. An explicitly edited date or a condition that stops matching can still produce the normal recovery transition.

Authenticated Web and native iOS views refetch the Server-owned deadline, alert history and cost advisories after catalog changes and reconnects. Clients do not calculate recurring deadlines.

State Persistence

Trigger state is persisted to the alert_states table in SQLite and loaded into a hot cache on startup. As a result:

• A once rule that is already triggered will not re-fire after a server restart.
• Triggered-but-not-yet-recovered alerts are restored automatically after a restart.

Recovery

When a previously triggered alert recovers (the condition is no longer met):

1. The alert is marked as recovered, and its state is cleared in both the in-memory cache and the database.
2. If recover tasks (recover_trigger_tasks) are configured, the corresponding remote commands run.
3. If the condition triggers again later, notifications fire according to the trigger mode.

Maintenance Suppression

Alert notifications are suppressed while the affected server is in an active maintenance window. Rule evaluation still runs, but ServerBee skips notification delivery for that server until the window ends. Event-driven rules follow the same cover-type and maintenance suppression rules as polling-based rules.

Blocking Source IPs

Security-event rules (ssh_brute_force_detected, port_scan_detected) can enable Block source IP. When the rule triggers, ServerBee tells the affected agent's firewall to block the offending IP, turning detection into automatic mitigation. This requires the agent to have the firewall_block capability (CAP_FIREWALL_BLOCK), which is on by default but can be disabled on the agent host via [capabilities]. See Security Events and Firewall.

Notification Channels

ServerBee supports five notification channel types. Each channel is a separate entity that can be reused across multiple notification groups.

Webhook

Send HTTP requests to any URL. Supports custom methods, headers, and body templates.

{
  "url": "https://hooks.slack.com/services/xxx",
  "method": "POST",
  "headers": {
    "Content-Type": "application/json"
  },
  "body_template": "{\"text\": \"{{server_name}} {{event}}: {{message}}\"}"
}

If no body_template is provided, the default template is used. If no Content-Type header is set, it defaults to application/json.

Telegram

Send messages to a Telegram chat via the Bot API. Messages are sent with HTML parse mode enabled.

{
  "bot_token": "123456:ABC-DEF",
  "chat_id": "-1001234567890"
}

Bark

Send push notifications to iOS devices via Bark.

{
  "server_url": "https://api.day.app",
  "device_key": "your-device-key"
}

Email (via Resend)

Email notifications are delivered through Resend. Two steps before use:

1. Set SERVERBEE_RESEND__API_KEY on the server (see the Configuration page).
2. Add and verify your sender domain at resend.com/domains. The from address on each channel must belong to a verified domain.

Channel config:

{
  "from": "alerts@yourdomain.com",
  "to": ["ops@example.com", "oncall@example.com"]
}

to is an array — a single channel can deliver to multiple recipients in one API call. The subject follows the format [ServerBee] {server_name} {event}; the body is HTML with a plain-text fallback.

APNs

Send native Apple Push Notification service pushes to registered mobile devices.

{
  "key_id": "ABC123DEFG",
  "team_id": "TEAM999888",
  "private_key": "-----BEGIN PRIVATE KEY-----...",
  "bundle_id": "com.example.serverbee",
  "sandbox": false
}

APNs requires an Apple developer key, team ID, bundle ID, and private key. Set sandbox: true only for development builds.

Template Variables

Notification messages support the following template variables:

The default notification template is:

[ServerBee] {{server_name}} {{event}}
{{message}}
Time: {{time}}

Testing a Channel

After creating a channel, verify its configuration with the test endpoint:

POST /api/notifications/:id/test

Notification Groups

Notification channels are organized into groups. An alert rule is linked to a notification group, and when the rule triggers, all enabled channels in the group are dispatched.

This allows you to:

• Send the same alert to multiple channels (e.g., Telegram + Email).
• Reuse channel configurations across different alert rules.
• Enable or disable individual channels without modifying alert rules.

Offline Detection

Offline status is determined by a dedicated background task and works together with offline rules:

• The task scans agent connection state every 10 seconds.
• A server is marked offline when its last report is more than 30 seconds old.
• The offline rule's duration sets how many seconds a server must stay offline before the alert fires. For example, { "rule_type": "offline", "duration": 60 } triggers after 60 seconds offline.

Example: Complete Alert Setup

Here is a typical setup for monitoring CPU usage across all servers with Telegram notifications:

1. Create a Telegram notification channel with your bot token and chat ID.
2. Create a notification group that includes the Telegram channel.
3. Create an alert rule:
   • Name: "High CPU Usage"
   • Rules: [{"rule_type": "cpu", "min": 90.0}]
   • Trigger mode: always
   • Cover type: all
   • Notification group: (the group you created)

Now, when any server's CPU stays above 90% for at least 70% of a 10-minute sampling window, you will receive a Telegram message. Subsequent notifications are debounced to once every 5 minutes while the condition persists.