81 lines
2.5 KiB
Markdown
81 lines
2.5 KiB
Markdown
# Rust Engine API and Worker
|
|
|
|
## Overview
|
|
|
|
- HTTP API (warp) under /api for file management and query lifecycle
|
|
- MySQL for metadata, Qdrant for vector similarity
|
|
- Background worker resumes queued work and re-queues stale InProgress jobs at startup
|
|
|
|
## Environment variables
|
|
|
|
- DATABASE_URL: mysql://USER:PASS@HOST:3306/DB
|
|
- QDRANT_URL: default <http://qdrant:6333>
|
|
- GEMINI_API_KEY: used for Gemini content generation (optional in demo)
|
|
|
|
## Endpoints (JSON)
|
|
|
|
- POST /api/files (multipart)
|
|
- Form: file=@path
|
|
- Response: {"success": true}
|
|
|
|
- GET /api/files/list
|
|
- Response: {"files": [{"id","filename","path","description"}]}
|
|
|
|
- GET /api/files/delete?id=<file_id>
|
|
- Response: {"deleted": true|false}
|
|
|
|
- POST /api/query/create
|
|
- Body: {"q": "text", "top_k": 5}
|
|
- Response: {"id": "uuid"}
|
|
|
|
- GET /api/query/status?id=<query_id>
|
|
- Response: {"status": "Queued"|"InProgress"|"Completed"|"Cancelled"|"Failed"|"not_found"}
|
|
|
|
- GET /api/query/result?id=<query_id>
|
|
- Response (Completed):
|
|
{
|
|
"result": {
|
|
"summary": "Found N related files",
|
|
"related_files": [
|
|
{"id","filename","path","description","score"}
|
|
],
|
|
"relationships": "...",
|
|
"final_answer": "..."
|
|
}
|
|
}
|
|
|
|
- GET /api/query/cancel?id=<query_id>
|
|
- Response: {"cancelled": true}
|
|
|
|
## Worker behavior
|
|
|
|
- Ensures Qdrant collection exists (dim 64, cosine)
|
|
- Re-queues InProgress older than 10 minutes
|
|
- Processing stages:
|
|
1) Set InProgress
|
|
2) Embed query text (demo now; pluggable Gemini later)
|
|
3) Search Qdrant top_k (default 5)
|
|
4) Join file metadata (MySQL)
|
|
5) Gemini step: relationship analysis (strictly from provided files)
|
|
6) Gemini step: final answer (no speculation; say unknown if insufficient)
|
|
7) Persist result (JSON) and set Completed
|
|
- Checks for cancellation between stages
|
|
|
|
## Local quickstart
|
|
|
|
1. docker compose up -d mysql qdrant
|
|
2. set env DATABASE_URL and QDRANT_URL
|
|
3. cargo run
|
|
4. (optional) import demo PDFs
|
|
- Ensure demo files are located in `rust-engine/demo-data` (default) or set `DEMO_DATA_DIR` env var to a folder containing PDFs.
|
|
- Call the endpoint:
|
|
- POST <http://localhost:8000/api/files/import-demo>
|
|
- Optional query `?force=1` to overwrite existing by filename
|
|
- Or run the PowerShell helper:
|
|
- `./scripts/import_demo.ps1` (adds all PDFs in demo-data)
|
|
- `./scripts/import_demo.ps1 -Force` (overwrite existing)
|
|
|
|
## Notes
|
|
|
|
- Replace demo embeddings with real Gemini calls for production
|
|
- Add auth to endpoints if needed (API key/JWT)
|