AI Project 3 — RAG + Vector Search
Frontier AI
Intelligence
A Retrieval-Augmented Generation system querying 30+ documents across frontier AI labs,
safety evaluations, and regulatory frameworks — with hybrid BM25 + vector search,
Field CISO Mode, live model leaderboard, and a Zero Trust audit trail on every query.
Claude API
Cloud SQL
pgvector
Vertex AI
Python
Streamlit
GCP
BM25 + RRF
How it works
🕷️
Ingest
Auto-watcher fetches 9 AI lab sources, hash-detects changes
🧮
Embed
Vertex AI text-embedding-004 generates 768-dim vectors per chunk
🔍
Retrieve
Hybrid BM25 + vector search fused with Reciprocal Rank Fusion
🧠
Synthesize
Claude reads top chunks, forms position with citations
💬
Query
6-tab Streamlit app — standard or Field CISO mode
Key Features
🛡️
Field CISO Mode
Every query reframed through security, procurement, and compliance lens. Same corpus, different synthesis angle.
⚡
Find Disagreements
Explicitly maps tensions between sources — where Anthropic, Google, and regulators diverge on the same topic.
📊
Live Model Leaderboard
Scraped from llm-stats.com — Intelligence, Reasoning, Coding, Agent scores, pricing. Refreshes on demand.
🔄
Auto-Watcher
Monitors 9 AI lab URLs daily. Hash-based change detection — only re-ingests when content actually changes.
➕
Add Content
Paste any URL directly in the app — ingests immediately. Optionally add to the watch list for ongoing monitoring.
📐
Benchmark Leaders
GPQA Diamond, SWE-bench, AIME, Humanity's Last Exam — the 2026 benchmarks that actually differentiate frontier models.
Architecture
DATABASE
Cloud SQL PostgreSQL 15 on GCP (us-east4). pgvector 0.8.1 extension for 768-dim embeddings. ivfflat index for approximate nearest neighbor search.
EMBEDDINGS
Vertex AI text-embedding-004 — 768 dimensions, RETRIEVAL_DOCUMENT task type for ingestion, RETRIEVAL_QUERY for search. Service account auth for Streamlit Cloud deployment.
RETRIEVAL
Hybrid BM25 + vector search with Reciprocal Rank Fusion (k=60, 60/40 weighting). Diversity filter limits 2 chunks per document. Field CISO mode appends security context to query embeddings.
SYNTHESIS
Claude Sonnet reads top 8 chunks across sources and forms a synthesized answer with inline citations. Disagreement mode explicitly maps conflicting positions.
CORPUS
30+ documents across Anthropic, OpenAI, Google DeepMind, Meta, Mistral, DeepSeek, Alibaba, Microsoft, Amazon, CISA, UK AI Safety Institute, METR, Apollo Research, NIST.
🔐 claude-zt Foundation Tier — Zero Trust for AI Agents
Every query runs through a Zero Trust wrapper: AgentIdentity (UUID v4 per session, role-scoped),
InputValidator (10 prompt injection patterns blocked, 2000-char limit),
and AuditLogger (every query logged to PostgreSQL with agent_id, role, chunks retrieved, latency).
This is the Foundation Tier of a progressive ZT architecture that scales through P7.
What I learned
RAG CORE
Chunking strategy matters more than model choice. Paragraph-aware chunking with 200-char overlap + title prepending significantly improves retrieval accuracy over naive fixed-size splits.
HYBRID SEARCH
BM25 contributes most for exact technical terms (model names, acronyms). Vector search dominates for conceptual queries. RRF fusion beats either alone — even with simple 60/40 weighting.
DEPLOYMENT
Streamlit Cloud secrets are NOT exposed as OS environment variables — must use st.secrets directly. Cloud SQL requires service account credentials on cloud deployments, not ADC.
USE CASE
RAG adds genuine value over Claude + web search only when the corpus is proprietary, specialized, or not well-indexed. Public blog posts are a weak corpus. The leaderboard and ZT audit trail are the most differentiated features.