docs(search): document generic search gateway resilience

2026-07-10 18:42:18 +08:00
parent 630c19d079
commit a72f70652e
1 changed files with 128 additions and 0 deletions
@@ -0,0 +1,128 @@
+# Search Gateway Resilience
+
+## Positioning
+
+SR-02 is a generic Search Gateway capability improvement. It is not tied to education, exams, geography, policy, product search, or any single content vertical.
+
+The goal is to convert Search Gateway from single-shot retrieval into an evidence-first retrieval pipeline:
+
+```text
+query normalization
+→ source planning
+→ official/direct adapters
+→ provider fallback
+→ extraction
+→ evidence cache
+→ confidence scoring
+→ failure diagnosis
+→ human evidence request when needed
+```
+
+## Generic capabilities
+
+### 1. Multi-source fallback
+
+Search Gateway should not fail just because one provider is blocked, truncated, rate-limited, or dynamically rendered.
+
+Default fallback chain:
+
+```text
+direct_official_adapter
+→ official_site_search
+→ structured_api_or_feed
+→ search_engine_provider
+→ authoritative_mirror_or_media
+→ cached_evidence
+→ user_supplied_evidence_request
+```
+
+### 2. Source tiering
+
+Every source is classified before being trusted:
+
+```text
+OFFICIAL_PRIMARY
+OFFICIAL_SECONDARY
+AUTHORITATIVE_MEDIA
+SPECIALIZED_DATABASE
+GENERAL_WEB
+USER_SUPPLIED_EVIDENCE
+```
+
+The same tiering model works for public policy, software documentation, finance data, product specs, standards, academic sources, sports, travel, and user-supplied documents.
+
+### 3. Failure diagnosis
+
+Search failures must be structured, not vague.
+
+Examples:
+
+```text
+SEARCH_PROVIDER_TIMEOUT
+SEARCH_PROVIDER_CAPTCHA
+SEARCH_RESULT_TRUNCATED
+SOURCE_BLOCKED_BY_CDN
+SOURCE_DYNAMIC_PAGE_UNREADABLE
+OFFICIAL_SOURCE_FOUND_BUT_EXTRACTION_FAILED
+SOURCE_CONFLICT_NEEDS_REVIEW
+EVIDENCE_CACHE_STALE
+HUMAN_EVIDENCE_REQUIRED
+```
+
+### 4. Evidence cache
+
+A successful retrieval should leave structured evidence behind:
+
+```yaml
+evidence_id: ev-...
+normalized_query: ...
+source_tier: OFFICIAL_PRIMARY
+publisher: ...
+url: ...
+extracted_at: ...
+published_at: ...
+answer_fields: {...}
+confidence: high
+```
+
+Cache entries must carry staleness rules by topic type. Time-sensitive information should expire faster than static references.
+
+### 5. Confidence scoring
+
+Search Gateway should return both the answer and confidence metadata:
+
+```json
+{
+  "answer": {},
+  "confidence": 0.95,
+  "confidence_level": "high",
+  "sources": [],
+  "fallback_used": true,
+  "failure_codes": [],
+  "evidence_ids": []
+}
+```
+
+### 6. Human evidence intake
+
+When automated retrieval is blocked, Search Gateway should ask for a screenshot, URL, PDF, or pasted text and ingest it as `USER_SUPPLIED_EVIDENCE`, rather than silently failing.
+
+## Non-goals
+
+- No vertical-specific hardcoding.
+- No CAPTCHA bypass.
+- No access-control bypass.
+- No scraping of protected/private content.
+- No treating search snippets as authoritative without source classification.
+
+## Acceptance
+
+SR-02 is accepted when:
+
+```text
+AIOS_SEARCH_GATEWAY_MULTI_SOURCE_READY
+AIOS_SEARCH_GATEWAY_FAILURE_DIAGNOSIS_READY
+AIOS_SEARCH_GATEWAY_EVIDENCE_CACHE_READY
+AIOS_SEARCH_GATEWAY_CONFIDENCE_SCORING_READY
+AIOS_SEARCH_GATEWAY_GENERIC_RESILIENCE_READY
+```