CodeRED-Astra/QUICK_REFERENCE.md

CodeRED-Astra Quick Reference

System Overview

Two-worker architecture for ISS document RAG:

1. FileWorker: Analyzes uploaded files (Flash → Pro → Embed → Qdrant)
2. QueryWorker: Answers queries (Embed → Search → Relationships → Answer)

Both workers are resumable and automatically recover from crashes.

Core Data Flow

Upload PDF → Storage → MySQL (pending) → FileWorker → Qdrant → MySQL (ready)
                                                          ↓
User Query → MySQL (queued) → QueryWorker → Search Qdrant → Gemini → Result

Module Map

Module	Purpose	Key Functions
main.rs	Entry point	Spawns workers, serves API
db.rs	Database init	Creates files/queries tables
api.rs	HTTP endpoints	Upload, list, delete, query CRUD
file_worker.rs	File analysis	Flash→Pro→embed→upsert
worker.rs	Query processing	Search→relationships→answer
gemini_client.rs	AI integration	Text generation, embeddings
vector_db.rs	Qdrant client	Upsert, search, delete
storage.rs	File management	Save/delete files
models.rs	Data structures	FileRecord, QueryRecord

API Endpoints

Files

• POST /api/files - Upload file
• POST /api/files/import-demo?force=1 - Bulk import demo PDFs
• GET /api/files/list - List all files with status
• GET /api/files/delete?id=<uuid> - Delete file

Queries

• POST /api/query/create - Create query
• GET /api/query/status?id=<uuid> - Check status
• GET /api/query/result?id=<uuid> - Get result
• GET /api/query/cancel?id=<uuid> - Cancel query

Database Schema

files

• id - UUID primary key
• filename - Original filename
• path - Storage path
• description - Gemini Flash description
• pending_analysis - FALSE when ready for search
• analysis_status - Queued/InProgress/Completed/Failed

queries

• id - UUID primary key
• status - Queued/InProgress/Completed/Cancelled/Failed
• payload - JSON query params {"q": "...", "top_k": 5}
• result - JSON result {"summary": "...", "related_files": [...], "relationships": "...", "final_answer": "..."}

Environment Variables

Required

• GEMINI_API_KEY - Gemini API key
• DATABASE_URL - MySQL connection string
• QDRANT_URL - Qdrant URL (default: http://qdrant:6333)

Optional

• ASTRA_STORAGE - Storage directory (default: /app/storage)
• DEMO_DATA_DIR - Demo data directory (default: /app/demo-data)
• GEMINI_MODEL - Override Gemini model (default: gemini-1.5-pro)

Worker States

FileWorker

1. Queued - File uploaded, awaiting processing
2. InProgress - Currently being analyzed
3. Completed - Ready for search (pending_analysis=FALSE)
4. Failed - Error during processing

QueryWorker

1. Queued - Query created, awaiting processing
2. InProgress - Currently searching/analyzing
3. Completed - Result available
4. Cancelled - User cancelled
5. Failed - Error during processing

Gemini Prompts

FileWorker Stage 1 (Flash)

Describe the file '{filename}' and extract all key components, keywords, 
and details for later vectorization. Be comprehensive and factual.

FileWorker Stage 2 (Pro)

Given the file '{filename}' and its description: {desc}
Generate a set of vector graph data (keywords, use cases, relationships) 
that can be used for broad and precise search. Only include what is 
directly supported by the file.

QueryWorker Stage 4 (Relationships)

You are an assistant analyzing relationships STRICTLY within the provided files.
Query: {query}
Files: {file_list}
Tasks:
1) Summarize key details from the files relevant to the query.
2) Describe relationships and linkages strictly supported by these files.
3) List important follow-up questions that could be answered only using the provided files.
Rules: Do NOT guess or invent. If information is insufficient in the files, explicitly state that.

QueryWorker Stage 5 (Final Answer)

You are to compose a final answer to the user query using only the information from the files.
Query: {query}
Files considered: {file_list}
Relationship analysis: {relationships}
Requirements:
- Use only information present in the files and analysis above.
- If the answer is uncertain or cannot be determined from the files, clearly state that limitation.
- Avoid speculation or assumptions.
Provide a concise, structured answer.

Docker Architecture

Services

• rust-engine - Warp API + workers (port 8000)
• web-app - Express + React (port 3001)
• mysql - MySQL 9.1 (port 3306)
• qdrant - Qdrant vector DB (port 6333)
• phpmyadmin - DB admin UI (port 8080)

Volumes (Production)

• rust-storage:/app/storage - File storage (writable)
• /var/www/codered-astra/rust-engine/demo-data:/app/demo-data:ro - Demo PDFs (read-only)
• ~/astra-logs:/var/log - Log files

Common Issues

1. SQL Syntax Error

Problem: error near 'CREATE TABLE' Cause: Multiple CREATE TABLE in one query Fix: Split into separate sqlx::query() calls

2. Permission Denied

Problem: Permission denied (os error 13) Cause: Container user can't write to storage Fix: Use Docker volume, ensure ownership matches container user

3. Worker Not Processing

Problem: Files/queries stuck in Queued Cause: Worker crashed or not started Fix: Check logs, ensure workers spawned in main.rs

4. Qdrant Connection Failed

Problem: qdrant upsert/search failed Cause: Qdrant not running or wrong URL Fix: Verify QDRANT_URL, check qdrant container health

Development Commands

# Build and run locally
cd rust-engine
cargo build
cargo run

# Check code
cargo check

# Run with logs
RUST_LOG=info cargo run

# Docker compose (dev)
docker-compose up --build

# Docker compose (production)
docker-compose -f docker-compose.prod.yml up -d

# View logs
docker logs rust-engine -f

# Rebuild single service
docker-compose build rust-engine
docker-compose up -d rust-engine

Testing Flow

1. Start services: docker-compose up -d
2. Import demo data: curl -X POST http://localhost:3001/api/files/import-demo
3. Wait for FileWorker to complete (~30 seconds for 20 files)
4. Check file status: curl http://localhost:3001/api/files/list
5. Create query: curl -X POST http://localhost:3001/api/query/create -H "Content-Type: application/json" -d '{"q": "ISS main bus voltage", "top_k": 5}'
6. Check status: curl http://localhost:3001/api/query/status?id=<id>
7. Get result: curl http://localhost:3001/api/query/result?id=<id>

Performance Notes

• FileWorker: ~1-2 sec per file (demo embeddings)
• QueryWorker: ~3-5 sec per query (search + 2 Gemini calls)
• Qdrant search: <100ms for 1000s of vectors
• MySQL queries: <10ms for simple selects

Security Considerations

• Store GEMINI_API_KEY in GitHub Secrets (production)
• Use environment variables for all credentials
• Don't commit .env files
• Restrict phpmyadmin to internal network only
• Use HTTPS in production deployment