SpaceQuest: Hybrid RAG over Space-Exploration Articles

Python 3.12OpenAI gpt-4o-mini SQLite knowledge basedlt ingestion Streamlit UI + dashboardDocker

End-to-end RAG capstone for the LLM Zoomcamp. 30 curated Wikipedia articles (~762 chunks) about space exploration are ingested with dlt into SQLite, embedded with OpenAI, retrieved via BM25 / dense / hybrid RRF, and answered by a grounded GPT-4o-mini prompt with inline citations. Every query and 👍/👎 rating is logged and charted in a monitoring dashboard.

Overview

Problem statement

General-purpose chatbots answer space-exploration questions from pre-training memory, so mission facts get hallucinated — wrong crew counts, confused rover timelines, invented dates. SpaceQuest constrains generation to retrieved evidence: each question first retrieves relevant chunks from a dedicated knowledge base; the LLM may only use that context and cites sources inline. Retrieval quality was measured across six approaches and generation quality across two prompt designs; the best variant shipped.

Architecture

1 · Ingestion pipeline (dlt)

src/ingest_dlt.py runs a dlt pipeline (course automation tool) that loads the committed corpus and chunked children into SQLite with write_disposition="replace":

pipeline = dlt.pipeline(
    pipeline_name="wikipedia_ingestion",
    destination=dlt.destinations.sqlalchemy(f"sqlite:///{DB_PATH}"),
    dataset_name="kb",
)
pipeline.run([load_articles()], table_name="documents", write_disposition="replace")
pipeline.run([chunks], table_name="chunks", write_disposition="replace")   # 762 chunks

Chunking (src/chunking.py): 300-word windows, 50-word overlap; chunks under 30 words dropped.

2 · Knowledge base (SQLite)

Single file data/knowledge_base__kb.db holds the corpus, the chunk index, the OpenAI embeddings (text-embedding-3-small, 1536-d, written back by src/indexing.py), and the monitoring tables (query_log, feedback). Retrieval loads everything into memory once:

SELECT rowid, doc_title, url, chunk_id, text, embedding
FROM chunks WHERE embedding IS NOT NULL

3 · Hybrid retrieval

src/retrieval.py implements three search modes fused with Reciprocal Rank Fusion (k = 60): BM25 (rank-bm25, tokenized [a-z0-9]+), dense cosine similarity over normalized embeddings, and the RRF combination:

for rank, (i, _) in enumerate(dense):
    scores[i] = scores.get(i, 0.0) + 1.0 / (RRF_K + rank + 1)
for rank, (i, _) in enumerate(sparse):
    scores[i] = scores.get(i, 0.0) + 1.0 / (RRF_K + rank + 1)

The UI lets the user switch method live; BM25 is the default because it won the evaluation below.

4 · Query rewriting (best practice)

rewrite_query() asks gpt-4o-mini (temperature 0) to expand abbreviations and resolve pronouns before retrieval. Evaluated on the full ground-truth set — it did not beat plain BM25 on this corpus, so it ships as an opt-in checkbox rather than the default.

5 · Document re-ranking (best practice)

src/rerank.py prompts gpt-4o-mini to score each candidate chunk 0–3 for question relevance and reorders. Evaluated: hit-rate unchanged (0.9885), MRR dropped (0.9636 → 0.882) — the lexical ordering was already right, so re-ranking is available but not the default. Both attempts are reported honestly in the README.

6 · Grounded generation

src/generate.py builds the context (top-5 chunks with titles + URLs) and calls gpt-4o-mini at temperature 0.1. Two prompt variants were judged; the winner rules: answer only from context, cite as [title], refuse when absent:

Answer ONLY using the context below. Cite sources as [title].
If the context doesn't contain the answer, say you don't know.

Verified output: “Twelve people have walked on the Moon during the Apollo program [Apollo program].”

7 · Evaluation

Ground truth

eval/build_ground_truth.py generated 87 Q&A pairs from the corpus itself (gpt-4o-mini, JSON response format), committed to data/eval/ground_truth.json.

Retrieval evaluation — hit-rate@5 / MRR@5

LLM output evaluation — LLM-as-judge (30 questions, 0–5)

8 · Interface

app/app.py — Streamlit chat with a retrieval-method selector, rewrite toggle, expandable chunk-level sources, and 👍/👎 buttons. A terminal CLI (scripts/ask.py) covers the same flow headlessly. Run with streamlit run app/app.py.

9 · Monitoring

src/monitoring.py writes every query (question, rewritten question, method, answer, latency, token usage, sources) and each 👍/👎 into SQLite. The dashboard tab renders four KPIs (volume, avg latency, avg tokens, 👍/👎) and five charts: questions per day, latency per query, queries by retrieval method, top retrieved documents, feedback split.

10 · Containerization & reproducibility

docker compose up --build     # app at http://localhost:8501 (APP_PORT override)

Dockerfile (python:3.12-slim) + compose service with data/ and logs/ volumes; secrets via .env. Corpus and evaluation data are committed, requirements.txt is fully pinned, and the README's quick start rebuilds the KB from scratch with four commands.

Rubric scorecard