Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d457ac3961 | ||
|
|
9fa7511bb5 |
@@ -1,220 +0,0 @@
|
||||
# 事故复盘:Gitea 分支元数据漂移导致自主运作系统停止
|
||||
|
||||
日期:2026-07-10
|
||||
|
||||
事故等级:S1 / Stop-the-line
|
||||
|
||||
状态:已恢复;已通过 GitOps runtime 读写验收。
|
||||
|
||||
## 事故定级原则
|
||||
|
||||
凡是导致自主运作系统停止运转、无法继续闭环执行、无法写入 GitOps 状态、或必须人工介入恢复的事件,一律视为事故处理。
|
||||
|
||||
本事件满足该条件,因此按事故复盘、可靠性改进和防复发机制处理,而不是仅作为一次普通 Gitea 修复记录。
|
||||
|
||||
## 影响
|
||||
|
||||
- 受影响系统:自主运作系统、GitOps runtime、Gitea 分支查询和同步路径。
|
||||
- 受影响仓库:`aios/aios`。
|
||||
- 受影响主机 / 入口:`g.cnnetmkt.top`,SSH alias `hw`。
|
||||
- 受影响接口:
|
||||
- `GET /api/v1/repos/aios/aios/branches?page=1&limit=50`
|
||||
- `GET /api/v1/repos/aios/aios/branches?page=2&limit=50`
|
||||
- 用户可见影响:自主运作系统停止,需要人工检查和恢复。
|
||||
|
||||
## 现场结论
|
||||
|
||||
初始判断曾倾向于“Gitea 完全挂死、可能需要重启”。后续证据显示,更准确的判断是:
|
||||
|
||||
> Gitea 裸仓库 refs 与 Gitea SQLite `branch` 表 live 行之间发生元数据漂移,导致 branch API 分页结果异常,进而影响自主运作系统继续运转。
|
||||
|
||||
本次没有足够证据证明 Gitea 进程死锁。不能把“重启 Gitea”作为首选修复路径;应优先确认进程、API、裸仓库 refs、数据库 `branch` 表之间的一致性。
|
||||
|
||||
## 关键证据
|
||||
|
||||
来自修复记录:
|
||||
|
||||
- 裸仓库 refs 数量:`55`。
|
||||
- 修复前数据库 `branch` live 行数:`50`。
|
||||
- 修复后数据库 `branch` live 行数:`55`。
|
||||
- 修复前 API:
|
||||
- page 1 返回 `50`。
|
||||
- page 2 返回 `0`。
|
||||
- 修复后 API:
|
||||
- page 1 返回 `50`。
|
||||
- page 2 返回 `5`。
|
||||
- `gitea doctor check --run check-db-consistency` 修复后返回 `OK`。
|
||||
|
||||
缺失并已补回的 live 分支:
|
||||
|
||||
1. `patch/agent-event-bus-unified`
|
||||
2. `patch/agent-gpt-message-plane`
|
||||
3. `patch/gitops-change-archive-audit`
|
||||
4. `patch/group-owner-agent-context-isolation`
|
||||
5. `patch/user-profile-isolation`
|
||||
|
||||
## 恢复动作
|
||||
|
||||
1. 备份 Gitea SQLite 数据库:
|
||||
- `/data/aios-gitops/runtime/gitea-bin/data/gitea.db.bak-20260710-0105`
|
||||
2. 对比裸仓库 refs 与数据库 `branch` live 行。
|
||||
3. 用 SQLite 事务补回缺失的 5 条 live 分支记录。
|
||||
4. 复验 API 分页。
|
||||
5. 执行 Gitea doctor consistency check。
|
||||
6. 通过 GitOps runtime 接受测试。
|
||||
|
||||
## 恢复后接受测试
|
||||
|
||||
接受测试顺序:
|
||||
|
||||
1. `getRuntimeStatus`
|
||||
2. `getBaseline`
|
||||
3. `createBranch`
|
||||
4. `createPatch`
|
||||
|
||||
测试结果:
|
||||
|
||||
- Runtime 状态:`active`。
|
||||
- Baseline:`V1R4C3`。
|
||||
- 测试分支:`test/gitea-runtime-acceptance-20260710`。
|
||||
- 测试分支 commit:`7d6e497`。
|
||||
- 测试分支 sync:`synced`。
|
||||
- Patch:`gitea-runtime-acceptance-20260710`。
|
||||
- Patch commit:`59144bc`。
|
||||
- Patch sync:`synced`。
|
||||
|
||||
因此,事故当前状态为:已恢复。
|
||||
|
||||
## 根因判断
|
||||
|
||||
已证实根因:
|
||||
|
||||
- 裸仓库 `refs/heads` 与 Gitea `branch` 表 live 行不一致。
|
||||
- Gitea branch API 依赖的元数据状态与底层 Git refs 状态不一致,导致分页结果异常。
|
||||
|
||||
高可信风险假设:
|
||||
|
||||
- 自主控制器在高频创建、更新或同步分支时,可能触发了 refs 与 Gitea 元数据表之间的漂移。
|
||||
- 控制器缺少操作后的读回确认和一致性闸门。
|
||||
- 控制器缺少发现异常后的熔断机制,可能继续执行写操作并扩大影响。
|
||||
|
||||
尚未证实:
|
||||
|
||||
- Gitea 进程死锁。
|
||||
- Gitea 必须通过重启恢复。
|
||||
- Gitea 裸仓库数据损坏。
|
||||
|
||||
## 可靠性缺口
|
||||
|
||||
1. 缺少“refs 数量 vs branch live 行数 vs API 分页结果”的持续一致性检查。
|
||||
2. 缺少自主控制器写入后的 read-after-write 验证。
|
||||
3. 缺少控制器异常时的自动熔断和安全模式。
|
||||
4. 缺少 branch metadata drift 的专用告警。
|
||||
5. 缺少“不要先重启,先判定数据一致性”的事故处理规则。
|
||||
6. 缺少对 Gitea/GitOps 写链路的标准接受测试记录。
|
||||
|
||||
## 防复发要求
|
||||
|
||||
### 1. 一致性哨兵
|
||||
|
||||
必须增加周期性检查:
|
||||
|
||||
```text
|
||||
bare_repo_refs_count == gitea_branch_live_count == api_branch_total_count
|
||||
```
|
||||
|
||||
至少检查:
|
||||
|
||||
- 裸仓库 `refs/heads` 数量。
|
||||
- SQLite `branch` 表中 `is_deleted=0` 数量。
|
||||
- Gitea branch API 分页总数。
|
||||
- `gitea doctor check --run check-db-consistency` 结果。
|
||||
|
||||
### 2. 写后确认
|
||||
|
||||
自主控制器每次创建或删除分支后,必须执行 read-after-write 验证:
|
||||
|
||||
1. Git ref 存在 / 不存在。
|
||||
2. Gitea API 可见 / 不可见。
|
||||
3. `branch` live 行状态符合预期。
|
||||
4. 远端同步结果为 `synced`。
|
||||
|
||||
任一失败时,不得继续下一轮自治写入。
|
||||
|
||||
### 3. 熔断机制
|
||||
|
||||
发现以下任一条件时,控制器必须进入安全模式:
|
||||
|
||||
- refs 数量与 branch live 行数不一致。
|
||||
- API 分页总数与 refs 数量不一致。
|
||||
- `gitea doctor` 返回非 OK。
|
||||
- 连续 Git push / fetch / branch API 失败。
|
||||
- GitOps runtime 无法完成 `createBranch` 或 `createPatch`。
|
||||
|
||||
安全模式行为:
|
||||
|
||||
- 停止 branch 创建、删除、批量 patch 写入等突变操作。
|
||||
- 允许只读诊断。
|
||||
- 生成事故记录或 patch 请求。
|
||||
- 等待 owner/admin 审核恢复。
|
||||
|
||||
### 4. 单写者与限速
|
||||
|
||||
Gitea 分支元数据相关操作必须满足:
|
||||
|
||||
- 同一仓库同一时间只有一个自治写者。
|
||||
- 分支创建、删除、批量同步必须有锁。
|
||||
- 控制器需要限速,避免短时间大量 refs 变更。
|
||||
- 锁超时或异常退出后必须有清理机制。
|
||||
|
||||
### 5. 修复前备份与最小修复
|
||||
|
||||
如再次发生漂移:
|
||||
|
||||
1. 先备份 `gitea.db`。
|
||||
2. 对比 refs、branch 表、API。
|
||||
3. 只补齐缺失 live 行或只修正明确错误行。
|
||||
4. 不做无证据的重启、重建仓库或批量删除。
|
||||
5. 修复后执行 doctor 与 GitOps runtime 接受测试。
|
||||
|
||||
### 6. 重启规则
|
||||
|
||||
Gitea 重启不得作为默认第一步。
|
||||
|
||||
只有在以下证据存在时才考虑重启:
|
||||
|
||||
- Gitea 进程不存在。
|
||||
- HTTP 本地端口无响应。
|
||||
- 日志显示明确死锁或长期阻塞。
|
||||
- 数据库一致且 Git refs 一致,但服务仍无法响应。
|
||||
- owner/admin 明确批准。
|
||||
|
||||
## 标准运行手册
|
||||
|
||||
再次出现自主运作系统停止时,按顺序执行:
|
||||
|
||||
1. 检查 GitOps runtime:`getRuntimeStatus`。
|
||||
2. 检查 baseline:`getBaseline`。
|
||||
3. 检查 Gitea 进程和本地 HTTP 端口。
|
||||
4. 对比裸仓库 refs 数量。
|
||||
5. 对比 SQLite `branch` live 行数量。
|
||||
6. 对比 Gitea branch API 分页总数。
|
||||
7. 执行 `gitea doctor check --run check-db-consistency`。
|
||||
8. 判断是否是数据一致性问题,而不是直接重启。
|
||||
9. 修复后运行接受测试:`getRuntimeStatus -> getBaseline -> createBranch -> createPatch`。
|
||||
10. 将事故、根因和防复发动作提交到 GitOps。
|
||||
|
||||
## 后续行动项
|
||||
|
||||
- [ ] 实现 Gitea branch consistency sentinel。
|
||||
- [ ] 为自主控制器增加 branch 操作后的 read-after-write 验证。
|
||||
- [ ] 为自主控制器增加 Gitea/GitOps 写入熔断。
|
||||
- [ ] 增加单写者锁和限速。
|
||||
- [ ] 将本事故加入可靠性经验库。
|
||||
- [ ] 将“导致自主运作系统停止即为事故”写入运行准则。
|
||||
|
||||
## 最终结论
|
||||
|
||||
本事故当前已恢复,但具备复发可能。
|
||||
|
||||
防复发重点不是单纯重启 Gitea,而是建立 refs、Gitea branch 表、API 分页、GitOps runtime 写链路之间的一致性哨兵、写后确认和控制器熔断机制。
|
||||
@@ -0,0 +1,212 @@
|
||||
# V1R4C5 Acceptance Evidence
|
||||
|
||||
Date: 2026-07-11
|
||||
|
||||
Status: release candidate evidence in progress
|
||||
|
||||
Current active baseline: V1R4C3
|
||||
|
||||
V1R4C5 has not been promoted yet.
|
||||
|
||||
## Owner Decision
|
||||
|
||||
Owner approved continued V1R4C5 work under the following constraints:
|
||||
|
||||
```text
|
||||
允许无 force-push 的分支 reconcile
|
||||
memory search 保持关闭
|
||||
先完成 SR-01/SR-02/数据隔离验收
|
||||
再进入 merge/deploy/tag owner gate
|
||||
```
|
||||
|
||||
## Scope
|
||||
|
||||
V1R4C5 release candidate scope:
|
||||
|
||||
1. SR-01 Gitea reliability and branch consistency guardrails.
|
||||
2. SR-02 watcher/stuck command cleanup.
|
||||
3. User data isolation redline and enforcement gates.
|
||||
4. SR-04 admin workbench read-only visibility preparation only.
|
||||
|
||||
Explicitly excluded from V1R4C5 activation:
|
||||
|
||||
- memory search activation;
|
||||
- full SR-04 portal production rollout;
|
||||
- bulk merge of old test/smoke/auth branches;
|
||||
- force push reconciliation;
|
||||
- Gitea DB mutation without owner/admin approval.
|
||||
|
||||
## Runtime Acceptance Evidence
|
||||
|
||||
### Runtime status
|
||||
|
||||
Observed:
|
||||
|
||||
```json
|
||||
{
|
||||
"service": "aios-gitops-model-api",
|
||||
"status": "active",
|
||||
"workspace_branch": "patch/v1r4c5-runtime-acceptance-clean-20260711-01"
|
||||
}
|
||||
```
|
||||
|
||||
### Baseline
|
||||
|
||||
Observed:
|
||||
|
||||
```json
|
||||
{
|
||||
"current_baseline": "V1R4C3",
|
||||
"previous_baseline": "V1R3C1",
|
||||
"promoted_by": "openclaw"
|
||||
}
|
||||
```
|
||||
|
||||
### Clean createBranch acceptance
|
||||
|
||||
Branch:
|
||||
|
||||
```text
|
||||
test/v1r4c5-runtime-acceptance-clean-20260711-01
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
status: ok
|
||||
commit: 6ae7058
|
||||
sync: synced
|
||||
```
|
||||
|
||||
### Clean createPatch acceptance
|
||||
|
||||
Patch:
|
||||
|
||||
```text
|
||||
patch/v1r4c5-runtime-acceptance-clean-20260711-01
|
||||
```
|
||||
|
||||
Patch id:
|
||||
|
||||
```text
|
||||
patch-1783706123789-f5ca8b69
|
||||
```
|
||||
|
||||
Commit:
|
||||
|
||||
```text
|
||||
ba0391e
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
synced
|
||||
```
|
||||
|
||||
## SR-01 Evidence
|
||||
|
||||
Known accepted evidence:
|
||||
|
||||
- Gitea branch consistency sentinel deployed.
|
||||
- Sentinel reports:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"unhealthy_checks": []
|
||||
}
|
||||
```
|
||||
|
||||
- systemd timer runs every 5 minutes.
|
||||
- service runs with:
|
||||
|
||||
```text
|
||||
--emit-safe-mode --json
|
||||
```
|
||||
|
||||
- broken symlink issue resolved by replacing the external reliability symlink with workspace-local files.
|
||||
|
||||
Required before final promotion:
|
||||
|
||||
- confirm latest sentinel report remains healthy after acceptance writes.
|
||||
|
||||
## SR-02 Evidence
|
||||
|
||||
Known accepted evidence:
|
||||
|
||||
```text
|
||||
11 / 11 tests pass
|
||||
```
|
||||
|
||||
Fix summary:
|
||||
|
||||
- stuck command detection reads parsed timestamps through `data.get('ts', 0)`;
|
||||
- `.md` fixture mtime is aligned with stored stuck timestamp via `os.utime(path, (ts, ts))`;
|
||||
- behavior is consistent across `.json` and `.md` state formats.
|
||||
|
||||
Required before final promotion:
|
||||
|
||||
- re-run SR-02 tests on the final integration candidate.
|
||||
|
||||
## User Data Isolation Evidence
|
||||
|
||||
Implemented controls:
|
||||
|
||||
- user data isolation redline;
|
||||
- memory scope guard;
|
||||
- provider gate;
|
||||
- activation gate;
|
||||
- audit service/timer;
|
||||
- CI acceptance scripts;
|
||||
- memory search activation check.
|
||||
|
||||
Required safety state:
|
||||
|
||||
```text
|
||||
AIOS_MEMORY_ENABLED=false
|
||||
```
|
||||
|
||||
Required before memory/search activation:
|
||||
|
||||
- isolation audit healthy;
|
||||
- no denied records;
|
||||
- no invalid records;
|
||||
- provider filters by scope before ranking;
|
||||
- provenance required;
|
||||
- global index disabled;
|
||||
- activation gate returns allowed;
|
||||
- explicit owner approval.
|
||||
|
||||
V1R4C5 does not approve memory search activation.
|
||||
|
||||
## SR-04 Admin Workbench Evidence
|
||||
|
||||
Current user-visible gap is confirmed:
|
||||
|
||||
- administrator cannot yet see task progress;
|
||||
- administrator cannot yet see flow/process diagrams;
|
||||
- SR-04 remains activation-ready but gated.
|
||||
|
||||
V1R4C5 scope allows only read-only visibility preparation and acceptance design, not full production rollout.
|
||||
|
||||
Required before SR-04 deployment:
|
||||
|
||||
- SR-01 green;
|
||||
- SR-02 green;
|
||||
- user data isolation gate accepted;
|
||||
- admin RBAC read-only check green.
|
||||
|
||||
## Remaining Required Gates Before Promotion
|
||||
|
||||
- [ ] Latest Gitea sentinel report remains healthy.
|
||||
- [ ] SR-02 watcher tests pass on final candidate.
|
||||
- [ ] User data isolation acceptance passes.
|
||||
- [ ] Admin workbench read-only RBAC acceptance passes or remains explicitly deferred.
|
||||
- [ ] No force-push was used.
|
||||
- [ ] Memory search remains disabled.
|
||||
- [ ] Owner approves merge/deploy/tag/promotion.
|
||||
|
||||
## Promotion Rule
|
||||
|
||||
V1R4C5 may not be promoted until all required gates are green or explicitly deferred with owner approval.
|
||||
@@ -0,0 +1,54 @@
|
||||
# V1R4C5 Final Owner Approval
|
||||
|
||||
Date: 2026-07-11
|
||||
|
||||
Status: approved for final promotion
|
||||
|
||||
## Owner Decision
|
||||
|
||||
Owner approved final V1R4C5 promotion.
|
||||
|
||||
Memory search must remain disabled.
|
||||
|
||||
```text
|
||||
允许最终晋升
|
||||
继续保持 memory search 关闭
|
||||
```
|
||||
|
||||
## Promotion Scope
|
||||
|
||||
Approved for V1R4C5:
|
||||
|
||||
1. SR-01 Gitea reliability and branch consistency guardrails.
|
||||
2. SR-02 watcher/stuck command cleanup.
|
||||
3. User data isolation redline and enforcement gates.
|
||||
4. SR-04 admin workbench read-only visibility preparation only.
|
||||
|
||||
## Explicit Non-Activation
|
||||
|
||||
The following are not approved by this promotion:
|
||||
|
||||
```text
|
||||
memory search activation
|
||||
full cross-session memory recall
|
||||
global vector index
|
||||
full SR-04 production portal rollout beyond read-only/admin visibility preparation
|
||||
bulk merge of old test/smoke/auth branches
|
||||
force-push based reconciliation
|
||||
```
|
||||
|
||||
## Required Runtime State
|
||||
|
||||
Memory remains disabled:
|
||||
|
||||
```text
|
||||
AIOS_MEMORY_ENABLED=false
|
||||
```
|
||||
|
||||
If a provider or deployment attempts to enable memory/search, it must be blocked unless a future owner approval explicitly allows it after the user-data isolation activation gate returns allowed.
|
||||
|
||||
## Final Promotion Rule
|
||||
|
||||
V1R4C5 may proceed to tag/promotion under owner gate.
|
||||
|
||||
If any final gate fails, promotion must stop and memory/search must remain disabled.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Gitea Autonomous Controller Reliability Guardrails
|
||||
|
||||
## Purpose
|
||||
|
||||
Prevent autonomous operations from stopping due to Gitea branch metadata drift or uncontrolled Git state mutation.
|
||||
|
||||
## Required Controls
|
||||
|
||||
### 1. Consistency Sentinel
|
||||
|
||||
Continuously compare:
|
||||
|
||||
```text
|
||||
bare repository refs
|
||||
==
|
||||
Gitea branch live metadata
|
||||
==
|
||||
branch API visible state
|
||||
```
|
||||
|
||||
Alert on any drift.
|
||||
|
||||
### 2. Write Barrier
|
||||
|
||||
Every autonomous branch mutation must include:
|
||||
|
||||
1. Git operation result check.
|
||||
2. Gitea API read-back verification.
|
||||
3. Metadata consistency verification.
|
||||
4. Remote sync verification.
|
||||
|
||||
Failed verification blocks further autonomous writes.
|
||||
|
||||
### 3. Circuit Breaker
|
||||
|
||||
Controller enters safe mode when:
|
||||
|
||||
- branch metadata drift detected;
|
||||
- repeated Git/Gitea API failures occur;
|
||||
- consistency checks fail;
|
||||
- write acceptance tests fail.
|
||||
|
||||
Safe mode:
|
||||
|
||||
- disable mutation operations;
|
||||
- preserve read-only diagnostics;
|
||||
- create incident record;
|
||||
- require controlled recovery.
|
||||
|
||||
### 4. Single Writer Rule
|
||||
|
||||
For branch metadata operations:
|
||||
|
||||
- only one autonomous writer per repository;
|
||||
- use operation locks;
|
||||
- limit mutation frequency;
|
||||
- recover abandoned locks safely.
|
||||
|
||||
### 5. Recovery Order
|
||||
|
||||
Do not restart first.
|
||||
|
||||
Order:
|
||||
|
||||
1. Check service health.
|
||||
2. Check process state.
|
||||
3. Check Git refs.
|
||||
4. Check Gitea metadata.
|
||||
5. Check API visibility.
|
||||
6. Repair minimal inconsistent state.
|
||||
7. Run acceptance tests.
|
||||
|
||||
Restart is reserved for proven process-level failure.
|
||||
|
||||
### 6. Acceptance Test
|
||||
|
||||
After recovery:
|
||||
|
||||
```text
|
||||
getRuntimeStatus
|
||||
->
|
||||
getBaseline
|
||||
->
|
||||
createBranch
|
||||
->
|
||||
createPatch
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- read operations return active runtime information;
|
||||
- write operations return commit, revision and sync status.
|
||||
|
||||
## Operational Principle
|
||||
|
||||
Any condition that stops autonomous operation is an incident. Reliability improvements must prevent recurrence rather than only restore service.
|
||||
Reference in New Issue
Block a user