Corrected sql communication issues causing rust panic
This commit is contained in:
Christbru
parent
db99d625c4
commit
a9dd6c032e
3 changed files with 497 additions and 3 deletions
268
ARCHITECTURE.md
Normal file
268
ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# CodeRED-Astra Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
CodeRED-Astra is a Retrieval-Augmented Generation (RAG) system for querying ISS technical documentation using vector search, MySQL metadata storage, and Gemini AI for analysis and response generation.
|
||||
|
||||
## System Components
|
||||
|
||||
### 1. **Rust Backend** (`rust-engine/`)
|
||||
High-performance Rust backend using Warp for HTTP, SQLx for MySQL, and Reqwest for external API calls.
|
||||
|
||||
#### Modules
|
||||
|
||||
**`main.rs`** - Entry point
|
||||
- Initializes tracing, database, storage
|
||||
- Spawns FileWorker and QueryWorker background tasks
|
||||
- Serves API routes on port 8000
|
||||
|
||||
**`db.rs`** - Database initialization
|
||||
- Connects to MySQL
|
||||
- Creates `files` table (id, filename, path, description, pending_analysis, analysis_status)
|
||||
- Creates `queries` table (id, status, payload, result, timestamps)
|
||||
|
||||
**`api.rs`** - HTTP endpoints
|
||||
- `POST /api/files` - Upload file (multipart/form-data)
|
||||
- `POST /api/files/import-demo` - Bulk import from demo-data directory
|
||||
- `GET /api/files/list` - List all files with status
|
||||
- `GET /api/files/delete?id=` - Delete file and remove from Qdrant
|
||||
- `POST /api/query/create` - Create new query (returns query ID)
|
||||
- `GET /api/query/status?id=` - Check query status
|
||||
- `GET /api/query/result?id=` - Get query result
|
||||
- `GET /api/query/cancel?id=` - Cancel in-progress query
|
||||
|
||||
**`file_worker.rs`** - File analysis pipeline
|
||||
- **Background worker** that processes files with `pending_analysis = TRUE`
|
||||
- Claims stale/queued files (requeues if stuck >10 min)
|
||||
- **Stage 1**: Call Gemini 1.5 Flash for initial description
|
||||
- **Stage 2**: Call Gemini 1.5 Pro for deep vector graph data (keywords, relationships)
|
||||
- **Stage 3**: Generate embedding and upsert to Qdrant
|
||||
- **Stage 4**: Mark file as ready (`pending_analysis = FALSE`, `analysis_status = 'Completed'`)
|
||||
- Resumable: Can recover from crashes/restarts
|
||||
|
||||
**`worker.rs`** - Query processing pipeline
|
||||
- **Background worker** that processes queries with `status = 'Queued'`
|
||||
- Requeues stale InProgress jobs (>10 min)
|
||||
- **Stage 1**: Embed query text
|
||||
- **Stage 2**: Search top-K similar vectors in Qdrant
|
||||
- **Stage 3**: Fetch file metadata from MySQL (only completed files)
|
||||
- **Stage 4**: Call Gemini to analyze relationships between files
|
||||
- **Stage 5**: Call Gemini for final answer synthesis (strict: no speculation)
|
||||
- **Stage 6**: Save results to database
|
||||
- Supports cancellation checks between stages
|
||||
|
||||
**`gemini_client.rs`** - Gemini API integration
|
||||
- `generate_text(prompt)` - Text generation with model switching via GEMINI_MODEL env var
|
||||
- `demo_text_embedding(text)` - Demo 64-dim embeddings (replace with real Gemini embeddings)
|
||||
- Falls back to demo responses if GEMINI_API_KEY not set
|
||||
|
||||
**`vector_db.rs`** - Qdrant client
|
||||
- `ensure_files_collection(dim)` - Create 'files' collection with Cosine distance
|
||||
- `upsert_point(id, vector)` - Store file embedding
|
||||
- `search_top_k(vector, k)` - Find k nearest neighbors
|
||||
- `delete_point(id)` - Remove file from index
|
||||
|
||||
**`storage.rs`** - File storage utilities
|
||||
- `storage_dir()` - Get storage path from ASTRA_STORAGE env or default `/app/storage`
|
||||
- `ensure_storage_dir()` - Create storage directory if missing
|
||||
- `save_file(filename, contents)` - Save file to storage
|
||||
- `delete_file(path)` - Remove file from storage
|
||||
|
||||
**`models.rs`** - Data structures
|
||||
- `FileRecord` - File metadata (mirrors files table)
|
||||
- `QueryRecord` - Query metadata (mirrors queries table)
|
||||
- `QueryStatus` enum - Queued, InProgress, Completed, Cancelled, Failed
|
||||
|
||||
### 2. **Web App** (`web-app/`)
|
||||
React + Vite frontend with Express backend for API proxying.
|
||||
|
||||
#### Backend (`server.mjs`)
|
||||
- Express server that proxies API calls to rust-engine:8000
|
||||
- Serves React static build from `/dist`
|
||||
- **Why needed**: Docker networking - React can't call rust-engine directly from browser
|
||||
|
||||
#### Frontend (`src/`)
|
||||
- `App.jsx` - Main chat interface component
|
||||
- `components/ui/chat/chat-header.jsx` - Header with debug-only "Seed Demo Data" button (visible with `?debug=1`)
|
||||
- Calls `/api/files/import-demo` endpoint to bulk-load ISS PDFs
|
||||
|
||||
### 3. **MySQL Database**
|
||||
Two tables for metadata storage:
|
||||
|
||||
**`files` table**
|
||||
```sql
|
||||
id VARCHAR(36) PRIMARY KEY
|
||||
filename TEXT NOT NULL
|
||||
path TEXT NOT NULL
|
||||
description TEXT
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
pending_analysis BOOLEAN DEFAULT TRUE
|
||||
analysis_status VARCHAR(32) DEFAULT 'Queued'
|
||||
```
|
||||
|
||||
**`queries` table**
|
||||
```sql
|
||||
id VARCHAR(36) PRIMARY KEY
|
||||
status VARCHAR(32) NOT NULL
|
||||
payload JSON
|
||||
result JSON
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
```
|
||||
|
||||
### 4. **Qdrant Vector Database**
|
||||
- Collection: `files`
|
||||
- Dimension: 64 (demo) - replace with real Gemini embedding dimension
|
||||
- Distance: Cosine similarity
|
||||
- Stores file embeddings for semantic search
|
||||
|
||||
### 5. **Demo Data** (`rust-engine/demo-data/`)
|
||||
~20 ISS technical PDFs organized by subsystem:
|
||||
- Electrical Power System (EPS)
|
||||
- Environmental Control & Life Support (ECLSS)
|
||||
- Command & Data Handling (C&DH)
|
||||
- Structures & Mechanisms
|
||||
|
||||
## Data Flow
|
||||
|
||||
### File Upload & Analysis
|
||||
```
|
||||
1. User uploads PDF → POST /api/files
|
||||
2. API saves file to storage, inserts DB record (pending_analysis=true)
|
||||
3. FileWorker claims pending file
|
||||
4. Gemini 1.5 Flash generates description
|
||||
5. Gemini 1.5 Pro generates vector graph data
|
||||
6. Embed text → upsert to Qdrant
|
||||
7. Mark file as ready (pending_analysis=false)
|
||||
```
|
||||
|
||||
### Query Processing
|
||||
```
|
||||
1. User submits query → POST /api/query/create
|
||||
2. API inserts query record (status='Queued')
|
||||
3. QueryWorker claims queued query
|
||||
4. Embed query text
|
||||
5. Search Qdrant for top-K similar files
|
||||
6. Fetch file metadata from MySQL
|
||||
7. Gemini analyzes relationships between files
|
||||
8. Gemini synthesizes final answer (no speculation)
|
||||
9. Save results to database
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Development (`docker-compose.yml`)
|
||||
- Local testing with hot-reload
|
||||
- Bind mounts for code
|
||||
|
||||
### Production (`docker-compose.prod.yml`)
|
||||
- Used by GitHub Actions for deployment
|
||||
- Runs rust-engine as user "1004" (github-actions)
|
||||
- Docker volume: `rust-storage` → `/app/storage`
|
||||
- Bind mount: `/var/www/codered-astra/rust-engine/demo-data` → `/app/demo-data:ro`
|
||||
- Environment variables:
|
||||
- `ASTRA_STORAGE=/app/storage`
|
||||
- `DEMO_DATA_DIR=/app/demo-data`
|
||||
- `QDRANT_URL=http://qdrant:6333`
|
||||
- `GEMINI_API_KEY=<secret>`
|
||||
- `DATABASE_URL=mysql://astraadmin:password@mysql:3306/astra`
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. **Two-Stage Analysis (Flash → Pro)**
|
||||
- Flash is faster/cheaper for initial description
|
||||
- Pro is better for deep analysis and relationship extraction
|
||||
- Enables cost-effective scaling
|
||||
|
||||
### 2. **Resumable Workers**
|
||||
- Workers requeue stale jobs (>10 min in InProgress)
|
||||
- Survives container restarts without data loss
|
||||
- Atomic state transitions via SQL
|
||||
|
||||
### 3. **Separation of Concerns**
|
||||
- FileWorker: Makes files searchable
|
||||
- QueryWorker: Answers user queries
|
||||
- Independent scaling and failure isolation
|
||||
|
||||
### 4. **Strict Answer Generation**
|
||||
- Gemini prompted to not speculate
|
||||
- Must state uncertainty when info is insufficient
|
||||
- Prevents hallucination in critical ISS documentation
|
||||
|
||||
### 5. **Demo Embeddings**
|
||||
- Current: 64-dim deterministic embeddings from text hash
|
||||
- Production: Replace with real Gemini text embeddings API
|
||||
- Allows development/testing without embedding API credits
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### Upload File
|
||||
```bash
|
||||
curl -F "file=@document.pdf" http://localhost:3001/api/files
|
||||
```
|
||||
|
||||
### Import Demo Data
|
||||
```bash
|
||||
curl -X POST http://localhost:3001/api/files/import-demo
|
||||
```
|
||||
|
||||
### Create Query
|
||||
```bash
|
||||
curl -X POST http://localhost:3001/api/query/create \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"q": "What is the voltage of the ISS main bus?", "top_k": 5}'
|
||||
```
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
curl http://localhost:3001/api/query/status?id=<query-id>
|
||||
```
|
||||
|
||||
### Get Result
|
||||
```bash
|
||||
curl http://localhost:3001/api/query/result?id=<query-id>
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### High Priority
|
||||
1. Real Gemini text embeddings (replace demo embeddings)
|
||||
2. File status UI panel (show processing progress)
|
||||
3. Health check endpoint (`/health`)
|
||||
4. Data purge endpoint (clear all files/queries)
|
||||
|
||||
### Medium Priority
|
||||
1. Streaming query responses (SSE/WebSocket)
|
||||
2. Query result caching
|
||||
3. File chunking for large PDFs
|
||||
4. User authentication
|
||||
|
||||
### Low Priority
|
||||
1. Multi-collection support (different document types)
|
||||
2. Query history UI
|
||||
3. File preview in chat
|
||||
4. Export results to PDF
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Storage Permission Errors
|
||||
- Ensure `/app/storage` is owned by container user
|
||||
- Docker volume must be writable by user 1004 in production
|
||||
|
||||
### SQL Syntax Errors
|
||||
- MySQL requires separate `CREATE TABLE` statements
|
||||
- Cannot combine multiple DDL statements in one `sqlx::query()`
|
||||
|
||||
### Qdrant Connection Issues
|
||||
- Check QDRANT_URL environment variable
|
||||
- Ensure qdrant service is running and healthy
|
||||
- Verify network connectivity between containers
|
||||
|
||||
### Worker Not Processing
|
||||
- Check logs: `docker logs rust-engine`
|
||||
- Verify database connectivity
|
||||
- Look for stale InProgress jobs in queries/files tables
|
||||
|
||||
## Demo Presentation (3 minutes)
|
||||
|
||||
See `rust-engine/DEMODETAILS.md` for curated demo script with example queries.
|
||||
Loading…
Add table
Add a link
Reference in a new issue