Private AI Stack
Reference builds for self-hosted, private LLM systems — the stack a small company runs when its data can't leave the building. Every claim is backed by a number a script in the repo produced.
Created · Updated Source ↗
llmragdockerlitellmpythonmlx
Runnable, runbook-style reference builds for private LLM systems. Each subproject stands alone:
- stack — a one-command platform: a local engine (oMLX on Apple Silicon, Ollama on Linux), a LiteLLM router with per-key budgets, Open WebUI, a Postgres audit log of every request, backups, and optional Prometheus/Loki/Grafana observability.
- rag — hybrid retrieval (BM25 + vectors), reranking, page-level citations, and an eval harness that publishes real recall and MRR numbers.
- extract — forms and invoices to schema-validated JSON with per-field confidence and evidence, a review queue, and a small model that defers the answers that matter to a larger one.
- finetune — LoRA fine-tune → eval → GGUF on Apple Silicon with MLX-LM.
The rule throughout: measured, not asserted. Each “verified” result comes from a script in the repo, including the results that came out badly.