KnowledgeLoop

Problem

Teams bury answers in runbooks and wikis. Chatbots that skip citations and evaluation sound fluent while being wrong — and nobody can prove retrieval quality.

Product

KnowledgeLoop is a documentation Q&A system: upload docs → ask with citations → evaluate quality on a fixed golden set. The public demo uses a fictional Northstar Runbooks corpus; operators can upload their own Markdown/PDF behind login.

Architecture

ingest → chunk → embed → retrieve → generate → log → evaluate

Engineering decisions

• One embedding model, one vector store, one answer path — no dual backends or disabled “segfault” branches in the happy path.
• No Celery/Redis on v1 — sync ingest keeps a free single-web deploy simple; async returns with Slack/Outline adapters.
• Eval harness over agent demos — recruiters and clients can inspect hit-rate, refusal, and latency instead of trusting a chat screenshot.
• Public chat = seed only — uploads stay behind session auth to control spend and abuse.

Evaluation

No published eval results yet. Index the seed corpus, then run python manage.py evaluate_rag.

• Retrieval hit@k — expected filename in top-k chunk metadata.
• Keyword coverage — cheap faithfulness proxy (must-include terms in answer + citations).
• Refusal precision — out-of-corpus items should refuse / low-confidence.
• No LLM-as-judge on the free-tier path — keep the harness cheap and reproducible.

Tradeoffs & next

• Slack Socket Mode and Outline sync are frozen in-tree for v1.1 adapters — not required for the public demo.
• Self-learning escalation loops stay secondary; measurable retrieval is the portfolio differentiator.
• Free-tier hosts may cold-start; the Loom and case study document the warm path.

Stack

Django · Django templates + HTMX · OpenAI (text-embedding-3-small, gpt-4o-mini) · Chroma · SQLite (demo) · single web process for free-tier deploy.