67e26be2d9
The public tree and its history contain only the listed paths. Earlier projection history remains preserved internally. Source-Sha: 913fde029b935671833254797f0f20f1eb9fabba Policy-Sha: 913fde029b935671833254797f0f20f1eb9fabba Tree-Digest: 180ae530c1058a2a5c89837bdce2d323ae83e669e38590ca72e75b8d92b7262f
74 lines
2.3 KiB
Markdown
74 lines
2.3 KiB
Markdown
# Heartbeat System
|
|
|
|
**Agent health monitoring via system event polling.**
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
RedFlag monitors agent health through a dedicated heartbeat mechanism. Agents periodically report status, and the server tracks operational state.
|
|
|
|
---
|
|
|
|
## Heartbeat Types
|
|
|
|
| Type | Trigger | Source | Purpose |
|
|
|------|---------|--------|---------|
|
|
| `manual` | Agent-initiated status check | Agent | Regular health report |
|
|
| `system` | Auto-triggered at dispatch | Server | Confirm command receipt |
|
|
| `command` | Command execution complete | Agent | Report execution status |
|
|
| `stuck` | Timeout exceeded | Server | Alert on stalled operations |
|
|
|
|
**Implementation:**
|
|
- `agent/internal/orchestrator/system_scanner.go`
|
|
- `server/internal/services/timeout.go`
|
|
|
|
---
|
|
|
|
## Heartbeat Flow
|
|
|
|
```
|
|
1. Agent executes command
|
|
2. Agent triggers heartbeat (system or command)
|
|
3. Server receives heartbeat event
|
|
4. Server updates agent metadata (heartbeat_source, last_heartbeat)
|
|
5. Server updates operational state (update_stuck_minutes countdown)
|
|
6. Dashboard displays current status
|
|
```
|
|
|
|
**Implementation:**
|
|
- Agent: `agent/internal/orchestrator/system_scanner.go:queueSystemHeartbeat()`
|
|
- Server: `server/internal/handlers/agents.go:HandleSystemHeartbeat()`
|
|
- Metadata: `agent.metadata.heartbeat_source`
|
|
|
|
---
|
|
|
|
## Timeout Configuration
|
|
|
|
**Operational timeouts:**
|
|
- `operational.update_stuck_minutes`: Default 5 minutes
|
|
- `operational.agent_update_timeout_minutes`: Default 15 minutes
|
|
- Configured via `security_settings` table
|
|
|
|
**Watchdog behavior:**
|
|
- If command exceeds `update_stuck_minutes` → mark as stuck
|
|
- If update exceeds `agent_update_timeout_minutes` → trigger rollback
|
|
- Timeout events logged to history table
|
|
|
|
---
|
|
|
|
## Footer: Assumptions & Connections
|
|
|
|
**Assumption:** Heartbeat is a best-effort mechanism — agent crashes will be detected on next poll cycle.
|
|
|
|
**Connection:** Heartbeat system (`flows/04-heartbeat.md`) implements ETHOS #3 (assume failure).
|
|
|
|
**Connection:** Timeout configuration (`flows/04-heartbeat.md`) ties to security settings (`security_settings.operational`).
|
|
|
|
**Connection:** System events (`flows/04-heartbeat.md`) published to history table for audit trail.
|
|
|
|
**Connection:** Auto-heartbeat gates (`flows/04-heartbeat.md`) controlled by `policy.auto_heartbeat_enabled`.
|
|
|
|
---
|
|
|
|
*Last reviewed: 2026-05-26*
|