219 lines
7 KiB
Markdown
219 lines
7 KiB
Markdown
# CodeRED-Astra Quick Reference
|
|
|
|
## System Overview
|
|
|
|
**Two-worker architecture for ISS document RAG:**
|
|
|
|
1. **FileWorker**: Analyzes uploaded files (Flash → Pro → Embed → Qdrant)
|
|
2. **QueryWorker**: Answers queries (Embed → Search → Relationships → Answer)
|
|
|
|
Both workers are **resumable** and automatically recover from crashes.
|
|
|
|
## Core Data Flow
|
|
|
|
```
|
|
Upload PDF → Storage → MySQL (pending) → FileWorker → Qdrant → MySQL (ready)
|
|
↓
|
|
User Query → MySQL (queued) → QueryWorker → Search Qdrant → Gemini → Result
|
|
```
|
|
|
|
## Module Map
|
|
|
|
| Module | Purpose | Key Functions |
|
|
|--------|---------|---------------|
|
|
| `main.rs` | Entry point | Spawns workers, serves API |
|
|
| `db.rs` | Database init | Creates files/queries tables |
|
|
| `api.rs` | HTTP endpoints | Upload, list, delete, query CRUD |
|
|
| `file_worker.rs` | File analysis | Flash→Pro→embed→upsert |
|
|
| `worker.rs` | Query processing | Search→relationships→answer |
|
|
| `gemini_client.rs` | AI integration | Text generation, embeddings |
|
|
| `vector_db.rs` | Qdrant client | Upsert, search, delete |
|
|
| `storage.rs` | File management | Save/delete files |
|
|
| `models.rs` | Data structures | FileRecord, QueryRecord |
|
|
|
|
## API Endpoints
|
|
|
|
### Files
|
|
- `POST /api/files` - Upload file
|
|
- `POST /api/files/import-demo?force=1` - Bulk import demo PDFs
|
|
- `GET /api/files/list` - List all files with status
|
|
- `GET /api/files/delete?id=<uuid>` - Delete file
|
|
|
|
### Queries
|
|
- `POST /api/query/create` - Create query
|
|
- `GET /api/query/status?id=<uuid>` - Check status
|
|
- `GET /api/query/result?id=<uuid>` - Get result
|
|
- `GET /api/query/cancel?id=<uuid>` - Cancel query
|
|
|
|
## Database Schema
|
|
|
|
### files
|
|
- `id` - UUID primary key
|
|
- `filename` - Original filename
|
|
- `path` - Storage path
|
|
- `description` - Gemini Flash description
|
|
- `pending_analysis` - FALSE when ready for search
|
|
- `analysis_status` - Queued/InProgress/Completed/Failed
|
|
|
|
### queries
|
|
- `id` - UUID primary key
|
|
- `status` - Queued/InProgress/Completed/Cancelled/Failed
|
|
- `payload` - JSON query params `{"q": "...", "top_k": 5}`
|
|
- `result` - JSON result `{"summary": "...", "related_files": [...], "relationships": "...", "final_answer": "..."}`
|
|
|
|
## Environment Variables
|
|
|
|
### Required
|
|
- `GEMINI_API_KEY` - Gemini API key
|
|
- `DATABASE_URL` - MySQL connection string
|
|
- `QDRANT_URL` - Qdrant URL (default: http://qdrant:6333)
|
|
|
|
### Optional
|
|
- `ASTRA_STORAGE` - Storage directory (default: /app/storage)
|
|
- `DEMO_DATA_DIR` - Demo data directory (default: /app/demo-data)
|
|
- `GEMINI_MODEL` - Override Gemini model (default: gemini-1.5-pro)
|
|
|
|
## Worker States
|
|
|
|
### FileWorker
|
|
1. **Queued** - File uploaded, awaiting processing
|
|
2. **InProgress** - Currently being analyzed
|
|
3. **Completed** - Ready for search (pending_analysis=FALSE)
|
|
4. **Failed** - Error during processing
|
|
|
|
### QueryWorker
|
|
1. **Queued** - Query created, awaiting processing
|
|
2. **InProgress** - Currently searching/analyzing
|
|
3. **Completed** - Result available
|
|
4. **Cancelled** - User cancelled
|
|
5. **Failed** - Error during processing
|
|
|
|
## Gemini Prompts
|
|
|
|
### FileWorker Stage 1 (Flash)
|
|
```
|
|
Describe the file '{filename}' and extract all key components, keywords,
|
|
and details for later vectorization. Be comprehensive and factual.
|
|
```
|
|
|
|
### FileWorker Stage 2 (Pro)
|
|
```
|
|
Given the file '{filename}' and its description: {desc}
|
|
Generate a set of vector graph data (keywords, use cases, relationships)
|
|
that can be used for broad and precise search. Only include what is
|
|
directly supported by the file.
|
|
```
|
|
|
|
### QueryWorker Stage 4 (Relationships)
|
|
```
|
|
You are an assistant analyzing relationships STRICTLY within the provided files.
|
|
Query: {query}
|
|
Files: {file_list}
|
|
Tasks:
|
|
1) Summarize key details from the files relevant to the query.
|
|
2) Describe relationships and linkages strictly supported by these files.
|
|
3) List important follow-up questions that could be answered only using the provided files.
|
|
Rules: Do NOT guess or invent. If information is insufficient in the files, explicitly state that.
|
|
```
|
|
|
|
### QueryWorker Stage 5 (Final Answer)
|
|
```
|
|
You are to compose a final answer to the user query using only the information from the files.
|
|
Query: {query}
|
|
Files considered: {file_list}
|
|
Relationship analysis: {relationships}
|
|
Requirements:
|
|
- Use only information present in the files and analysis above.
|
|
- If the answer is uncertain or cannot be determined from the files, clearly state that limitation.
|
|
- Avoid speculation or assumptions.
|
|
Provide a concise, structured answer.
|
|
```
|
|
|
|
## Docker Architecture
|
|
|
|
### Services
|
|
- **rust-engine** - Warp API + workers (port 8000)
|
|
- **web-app** - Express + React (port 3001)
|
|
- **mysql** - MySQL 9.1 (port 3306)
|
|
- **qdrant** - Qdrant vector DB (port 6333)
|
|
- **phpmyadmin** - DB admin UI (port 8080)
|
|
|
|
### Volumes (Production)
|
|
- `rust-storage:/app/storage` - File storage (writable)
|
|
- `/var/www/codered-astra/rust-engine/demo-data:/app/demo-data:ro` - Demo PDFs (read-only)
|
|
- `~/astra-logs:/var/log` - Log files
|
|
|
|
## Common Issues
|
|
|
|
### 1. SQL Syntax Error
|
|
**Problem**: `error near 'CREATE TABLE'`
|
|
**Cause**: Multiple CREATE TABLE in one query
|
|
**Fix**: Split into separate `sqlx::query()` calls
|
|
|
|
### 2. Permission Denied
|
|
**Problem**: `Permission denied (os error 13)`
|
|
**Cause**: Container user can't write to storage
|
|
**Fix**: Use Docker volume, ensure ownership matches container user
|
|
|
|
### 3. Worker Not Processing
|
|
**Problem**: Files/queries stuck in Queued
|
|
**Cause**: Worker crashed or not started
|
|
**Fix**: Check logs, ensure workers spawned in main.rs
|
|
|
|
### 4. Qdrant Connection Failed
|
|
**Problem**: `qdrant upsert/search failed`
|
|
**Cause**: Qdrant not running or wrong URL
|
|
**Fix**: Verify QDRANT_URL, check qdrant container health
|
|
|
|
## Development Commands
|
|
|
|
```bash
|
|
# Build and run locally
|
|
cd rust-engine
|
|
cargo build
|
|
cargo run
|
|
|
|
# Check code
|
|
cargo check
|
|
|
|
# Run with logs
|
|
RUST_LOG=info cargo run
|
|
|
|
# Docker compose (dev)
|
|
docker-compose up --build
|
|
|
|
# Docker compose (production)
|
|
docker-compose -f docker-compose.prod.yml up -d
|
|
|
|
# View logs
|
|
docker logs rust-engine -f
|
|
|
|
# Rebuild single service
|
|
docker-compose build rust-engine
|
|
docker-compose up -d rust-engine
|
|
```
|
|
|
|
## Testing Flow
|
|
|
|
1. Start services: `docker-compose up -d`
|
|
2. Import demo data: `curl -X POST http://localhost:3001/api/files/import-demo`
|
|
3. Wait for FileWorker to complete (~30 seconds for 20 files)
|
|
4. Check file status: `curl http://localhost:3001/api/files/list`
|
|
5. Create query: `curl -X POST http://localhost:3001/api/query/create -H "Content-Type: application/json" -d '{"q": "ISS main bus voltage", "top_k": 5}'`
|
|
6. Check status: `curl http://localhost:3001/api/query/status?id=<id>`
|
|
7. Get result: `curl http://localhost:3001/api/query/result?id=<id>`
|
|
|
|
## Performance Notes
|
|
|
|
- FileWorker: ~1-2 sec per file (demo embeddings)
|
|
- QueryWorker: ~3-5 sec per query (search + 2 Gemini calls)
|
|
- Qdrant search: <100ms for 1000s of vectors
|
|
- MySQL queries: <10ms for simple selects
|
|
|
|
## Security Considerations
|
|
|
|
- Store GEMINI_API_KEY in GitHub Secrets (production)
|
|
- Use environment variables for all credentials
|
|
- Don't commit `.env` files
|
|
- Restrict phpmyadmin to internal network only
|
|
- Use HTTPS in production deployment
|