CodeRED-Astra/rust-engine

Rust Engine API and Worker

Overview

• HTTP API (warp) under /api for file management and query lifecycle
• MySQL for metadata, Qdrant for vector similarity
• Background worker resumes queued work and re-queues stale InProgress jobs at startup

Environment variables

• DATABASE_URL: mysql://USER:PASS@HOST:3306/DB
• QDRANT_URL: default http://qdrant:6333
• GEMINI_API_KEY: used for Gemini content generation (optional in demo)

Endpoints (JSON)

• POST /api/files (multipart)

  • Form: file=@path
  • Response: {"success": true}

• GET /api/files/list

  • Response: {"files": [{"id","filename","path","description"}]}

• GET /api/files/delete?id=<file_id>

  • Response: {"deleted": true|false}

• POST /api/query/create

  • Body: {"q": "text", "top_k": 5}
  • Response: {"id": "uuid"}

• GET /api/query/status?id=<query_id>

  • Response: {"status": "Queued"|"InProgress"|"Completed"|"Cancelled"|"Failed"|"not_found"}

• GET /api/query/result?id=<query_id>

  • Response (Completed): { "result": { "summary": "Found N related files", "related_files": [ {"id","filename","path","description","score"} ], "relationships": "...", "final_answer": "..." } }

• GET /api/query/cancel?id=<query_id>

  • Response: {"cancelled": true}

Worker behavior

• Ensures Qdrant collection exists (dim 64, cosine)
• Re-queues InProgress older than 10 minutes
• Processing stages:
  1. Set InProgress
  2. Embed query text (demo now; pluggable Gemini later)
  3. Search Qdrant top_k (default 5)
  4. Join file metadata (MySQL)
  5. Gemini step: relationship analysis (strictly from provided files)
  6. Gemini step: final answer (no speculation; say unknown if insufficient)
  7. Persist result (JSON) and set Completed
  • Checks for cancellation between stages

Local quickstart

1. docker compose up -d mysql qdrant
2. set env DATABASE_URL and QDRANT_URL
3. cargo run
4. (optional) import demo PDFs
   • Ensure demo files are located in rust-engine/demo-data (default) or set DEMO_DATA_DIR env var to a folder containing PDFs.
   • Call the endpoint:
     • POST http://localhost:8000/api/files/import-demo
     • Optional query ?force=1 to overwrite existing by filename
   • Or run the PowerShell helper:
     • ./scripts/import_demo.ps1 (adds all PDFs in demo-data)
     • ./scripts/import_demo.ps1 -Force (overwrite existing)

Notes

• Replace demo embeddings with real Gemini calls for production
• Add auth to endpoints if needed (API key/JWT)