Agent SRE Basic Runbook 实战指南用 agent-governance-toolkit 为 AI Agent 构建 SLO 监控、事故检测与审批门禁驱动的恢复演练【免费下载链接】agent-governance-toolkitAI Agent Governance Toolkit — Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.项目地址: https://gitcode.com/GitHub_Trending/ag/agent-governance-toolkit本指南围绕 examples/agent-sre/basic-runbook 演示完整讲解如何基于agent-sreAI Agent 站点可靠性工程为自治 Agent 建立定义 SLO → 观测健康状态迁移 → 检测错误预算耗尽 → 执行带审批门禁的多步恢复 Runbook的闭环流程。读完本文你将掌握SLO/ErrorBudget/TaskSuccessRate/SLODashboard的配置与底层语义、IncidentDetector的信号分级与去重机制以及RunbookExecutor的审批门禁、超时与回滚实现并能独立把该示例落地到本地环境。示例概览三根支柱串起一条最小可靠性命周期basic-runbook是一个单文件、无异步、不调用外部服务的端到端演示完整代码见 examples/agent-sre/basic-runbook/main.py。它把agent-sre的工作流浓缩成三步对应关系如下步骤使用的 API输出结果定义并监控 SLOSLO、TaskSuccessRate、ErrorBudget、SLODashboard健康状态按unknown→healthy→exhausted迁移检测故障IncidentDetector.ingest_signal(...)由ERROR_BUDGET_EXHAUSTED信号自动创建p1级事故修复恢复RunbookExecutor.execute(runbook, incident, approve_callback...)依次执行三个步骤其中恢复步骤经过审批门禁这三根支柱分别对应agent-sre源码中的三大模块slo/SLO 引擎、incidents/detector.py事故检测、incidents/runbook.py 与 incidents/runbook_executor.pyRunbook 执行。示例所属模块的完整能力清单可参考 agent-governance-python/agent-sre/README.md同目录的示例索引见 examples/agent-sre/README.md。前置条件与环境准备Python 3.10pip示例不要求任何 API Key。安装依赖后完全本地运行不会调用外部服务。示例依赖被固定在 examples/agent-sre/basic-runbook/requirements.txtagent-governance-toolkit-cli4.0.0,5.0也就是说通过安装agent-governance-toolkit-cli4.x 即可获得agent_sre包agent-sre是 Agent Governance Toolkit Python 系列中的 SRE 子包。快速开始三步跑通演示cd examples/agent-sre/basic-runbook python -m venv .venv # 根据你的 shell 激活虚拟环境 # Windows PowerShell: .venv\Scripts\Activate.ps1 # macOS / Linux / Git Bash: source .venv/bin/activate pip install -r requirements.txt python main.py运行后无需任何清理——示例在运行时既不写文件也不打开 socket。如需移除本地虚拟环境直接删除.venv/目录即可。预期输出与运行规律Agent-SRE Basic Runbook Initial health : unknown After warmup : healthy After failure : exhausted [!] Incident created: error_budget_exhausted: Error budget exhausted after task failure Severity: p1 Incident ID: hex12 Runbook execution: completed [ok] Check health (durations) - health summary refreshed (statusexhausted) [ok] Restart agent (durations) - agent restarted [ok] Verify recovery (durations) - recovery checks passed关于逐次运行的差异需要明确以下几点健康状态行unknown、healthy、exhausted是确定性的每次运行结果一致。Severity 固定为p1因为ERROR_BUDGET_EXHAUSTED通过Signal.severity_hint映射到IncidentSeverity.P1。Incident ID 是随机的由uuid.uuid4().hex[:12]生成 12 位十六进制字符串每次运行不同。各步骤耗时为亚毫秒级且略有浮动不属于输出契约的一部分。源码走读三步链路是如何实现的第一步定义 SLO 并观测健康状态迁移main.py中的核心定义如下success_rate TaskSuccessRate(target0.95, window24h) slo SLO( namedemo-agent, descriptionDemo agent reliability target, indicators[success_rate], error_budgetErrorBudget(total0.05, burn_rate_critical10.0), ) dashboard SLODashboard() dashboard.register_slo(slo)随后示例通过两条记录路径驱动状态迁移# 预热20 个成功任务 → healthy for _ in range(20): success_rate.record_task(successTrue) slo.record_event(goodTrue) # 注入一个失败事件 → exhausted success_rate.record_task(successFalse) slo.record_event(goodFalse)为什么要同时调用record_task与record_event从源码可以清楚看到两者分工不同TaskSuccessRate.record_task(success)见 slo/indicators.py维护任务计数器_total/_success计算滑动成功率并写入MeasurementStore属于SLI指标采集层面SLO.record_event(good)见 slo/objectives.py把好坏事件记入ErrorBudget的有界deque(maxlen100_000)并在每次记录后立即调用self.evaluate()属于错误预算Error Budget层面。健康状态是怎么算出来的SLO.evaluate()见 slo/objectives.py按以下优先级判定error_budget.is_exhausted为真 →EXHAUSTED有critical级别燃烧率告警在触发 →CRITICAL有warning级别燃烧率告警在触发 →WARNING所有 SLI 都没有当前测量值 →UNKNOWN即数据不足否则 →HEALTHY。示例中初始没有任何测量因此为unknown预热后预算未耗尽且无告警因此为healthy一次失败事件使consumed(1.0) total(0.05)因此直接跳变到exhausted。ErrorBudget的关键字段与默认值汇总如下见 slo/objectives.py参数默认值含义total0.0未显式传入时由 SLO 最严格 SLI target 推导为1 - min_target错误预算总额 1 - SLO 目标consumed0.0已消耗预算window_seconds259200030 天预算窗口burn_rate_alert2.0触发 warning 的燃烧率阈值burn_rate_critical10.0触发 critical 的燃烧率阈值exhaustion_actionExhaustionAction.ALERT预算耗尽时的动作alert/freeze_deployments/circuit_break/throttlemax_events100_000事件缓冲上限防止长期运行内存无限增长此外SLODashboard.health_summary()见 slo/dashboard.py会按SLOStatus枚举统计total_slos/healthy/warning/critical/exhausted/unknown计数并给出每个 SLO 的状态字典——main.py中print_health打印的正是这个摘要中slos[slo_name]字段。关于 SLI 的更多能力agent-sre内置了 8 种 SLI 类型除TaskSuccessRate外还包括ToolCallAccuracy、ResponseLatency、CostPerTask、PolicyCompliance、DelegationChainDepth、HallucinationRate、CalibrationDeltaSLI全部注册在SLIRegistry见 slo/indicators.py。TimeWindow枚举支持1h/6h/24h/7d/30d五种窗口slo/indicators.py。第二步信号 → 事故p1是怎么来的detector IncidentDetector(correlation_window_seconds60) signal Signal( signal_typeSignalType.ERROR_BUDGET_EXHAUSTED, sourceslo.name, messageError budget exhausted after task failure, ) incident detector.ingest_signal(signal)从源码看Signal.severity_hint见 incidents/detector.py把信号类型映射到事故等级ERROR_BUDGET_EXHAUSTED、POLICY_VIOLATION、TRUST_REVOCATION→P1立即 pageSLO_BREACH、COST_ANOMALY、LATENCY_SPIKE→P2告警其余 →P3通知。IncidentDetector.ingest_signal()incidents/detector.py的处理逻辑是先追加信号并裁剪过期信号再做去重检查dedup_window_seconds600内同源同类型的重复信号不会重复建单最后只对P1/P2信号创建事故——P3/P4仅记录日志。创建事故时还会尝试在correlation_window_seconds窗口内聚合同源的不同类型信号生成Correlated事故并取其中最高严重级。示例中ERROR_BUDGET_EXHAUSTED直接命中P1因此必然返回一个Incident若返回Nonemain.py会抛出RuntimeError。Incident对象包含incident_id12 位 hex、title、severity、state初始DETECTED、agent_id、signals、actions、notes等字段并支持acknowledge/investigate/mitigate/resolve的完整生命周期状态迁移incidents/detector.py。第三步Runbook 执行、审批门禁与回滚runbook Runbook( namedemo-agent recovery, descriptionCheck health, restart the agent (with approval), verify recovery., steps[ RunbookStep(nameCheck health, actioncheck_health), RunbookStep( nameRestart agent, actionrestart_agent, requires_approvalTrue, ), RunbookStep(nameVerify recovery, actionverify_recovery), ], ) executor RunbookExecutor() execution executor.execute( runbookrunbook, incidentincident, approve_callbacklambda step, inc: True, )审批门禁的语义RunbookStep见 incidents/runbook.py支持timeout_seconds默认 300、requires_approval、rollback_action。RunbookExecutor.execute()incidents/runbook_executor.py在遇到requires_approvalTrue的步骤时有三条分支未提供approve_callback执行暂停状态置为WAITING_APPROVAL等待外部审批回调返回False该步骤被标记为SKIPPEDoutputApproval denied继续后续步骤回调返回True正常执行。示例传入lambda step, inc: True模拟人工审批通过。生产环境中这个回调应接入真实的人工审批流例如工单系统、IM 机器人确认等。每一步执行后RunbookExecutor都会向event_log写入审计事件execution_started/step_started/step_completed/step_failed/approval_waiting/approval_denied/rollback_*等形成可追踪的审计轨迹incidents/runbook_executor.py。超时与回滚机制若某步骤卡死_invoke_with_timeout会通过ThreadPoolExecutor(max_workers1)按step.timeout_seconds强制执行超时超时后抛出TimeoutError使该步骤标记为FAILEDincidents/runbook_executor.py。一旦任一步骤失败_rollback会逆序执行所有已完成步骤的rollback_action并把执行状态置为ROLLED_BACKincidents/runbook_executor.py。示例中所有步骤都成功因此execution.status为completed每个StepResult输出step_name、duration_seconds与output。示例文件清单文件作用examples/agent-sre/basic-runbook/main.py可运行演示单文件、无异步、无外部服务examples/agent-sre/basic-runbook/requirements.txt固定agent-governance-toolkit-cli4.0.0,5.0examples/agent-sre/basic-runbook/README.md本示例说明文档与 SDK 的映射关系示例用到的能力SDK 实现位置SLO 健康状态slo/objectives.pySLO、ErrorBudget、SLOStatus成功率指标slo/indicators.pyTaskSuccessRate及其他 7 种 SLI健康面板slo/dashboard.pySLODashboard事故检测incidents/detector.pyIncidentDetector、Signal、SignalTypeRunbook 模型incidents/runbook.pyRunbook、RunbookStepRunbook 执行incidents/runbook_executor.pyRunbookExecutoragent-sre包还预置了 4 个可直接加载的 YAML Runbook 模板restart_agent、revoke_trust、rollback_version、throttle_traffic见 agent-governance-python/agent-sre/src/agent_sre/incidents/runbooks/并内置了 4 个领域 SLO 模板coding-agent、customer-support-agent、data-pipeline-agent、research-agent见 agent-governance-python/agent-sre/src/agent_sre/specs/可作为把本示例扩展为生产策略时的起点。把示例扩展为生产实践的建议真实审批流将approve_callback替换为调用工单/IM/邮件审批服务的实现并为高风险步骤配置rollback_action把RunbookExecutor的审计event_log接入你的审计归档。持久化与告警SLI支持注入MeasurementStore内存或 SQLite 等持久化后端见 slo/indicators.py 与 slo/persistence.pySLO可挂载AlertManager在状态恶化warning/critical/exhausted或恢复时自动发送带dedup_key的告警slo/objectives.py。多信号聚合IncidentDetector的register_response可为特定信号类型注册自动响应动作如manual_rollback、notify_oncall配合correlation_window_seconds把同一 Agent 的多种异常聚合成一条事故减少告警风暴。与治理生态联动agent-sre属于 Agent Governance Toolkit 的可观测性/成本治理层其政策违规、信任评分、会话事件都可作为 SLO 的输入详见 agent-governance-python/agent-sre/README.md 的 Ecosystem Integration 章节相关的 SRE 面板教程可参考 docs/tutorials/06-sre-dashboards.md 与 docs/tutorials/51-cost-governance.md。【免费下载链接】agent-governance-toolkitAI Agent Governance Toolkit — Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.项目地址: https://gitcode.com/GitHub_Trending/ag/agent-governance-toolkit创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考