RAG Retriever Experiment: Reproducible Evaluation of Retrieval Architectures for Question Answering

This repository contains the source code, configurations, data preparation scripts, experimental protocols, and reproducibility artifacts developed for the master's dissertation entitled “Análise Comparativa de Arquiteturas de Recuperação em Sistemas Retrieval-Augmented Generation para Question Answering”. The project provides a reproducible experimental pipeline for evaluating the impact of different retrieval architectures on Retrieval-Augmented Generation (RAG) systems applied to Question Answering. The evaluated retrieval approaches include BM25, Dense Passage Retrieval (DPR), BAAI General Embedding (BGE), and ColBERT, representing lexical, dense, and late-interaction retrieval paradigms. The experimental design combines four retrieval architectures, three large language models, and different values of the top-k retrieval parameter. The pipeline supports retrieval evaluation using Recall@5, Recall@10, and Mean Reciprocal Rank at 10 (MRR@10), as well as RAG-specific evaluation metrics, including Faithfulness, Answer Relevancy, Context Precision, and Context Recall. The repository includes: - source code implemented in Python and LangChain;- configurations for the factorial experimental design;- scripts for obtaining and preprocessing Wikipedia and Natural Questions data;- persistent index structures for BM25, DPR, BGE, and ColBERT;- integration with local language models through Ollama;- RAGAS-based evaluation;- retrieval and generation latency collection;- per-query and aggregated experimental results;- dataset and index provenance registries;- environment, dependency, hardware, seed, and Git commit recording;- SHA-256 checksums for artifact integrity;- Docker and Docker Compose configurations;- files prepared for publication and citation through Zenodo. The complete Wikipedia and Natural Questions datasets are not redistributed in this deposit due to their size and original distribution conditions. Instead, the repository provides automated scripts and fixed configurations to download, validate, preprocess, and associate the datasets with the corresponding experimental artifacts. The demonstration result files included in the package are synthetic examples intended to validate the execution pipeline, file schemas, aggregation procedures, and reproducibility infrastructure. They must not be interpreted as the empirical results reported in the dissertation. This software supports the controlled and reproducible investigation of how retrieval architectures influence answer quality, retrieval effectiveness, computational cost, and the interaction between retrieval and generation components in RAG systems.rag_retriever_experiment/├── configs/│ ├── experiment.yaml│ └── datasets.yaml├── data/│ ├── wikipedia/│ │ ├── download.py│ │ ├── preprocess.py│ │ ├── metadata.sample.json│ │ └── README.md│ ├── natural_questions/│ │ ├── download.py│ │ ├── align_relevance.py│ │ └── metadata.sample.json│ ├── corpus.jsonl│ ├── queries.jsonl│ └── README.md├── indices/│ ├── bm25/│ ├── dpr/│ ├── bge/│ └── colbert/├── artifacts/│ └── example_run/│ ├── manifest.json│ ├── resolved_config.yaml│ ├── dataset_registry.json│ ├── index_registry.json│ ├── results.jsonl│ ├── summary.csv│ ├── retriever_summary.csv│ ├── data_dictionary.json│ ├── SHA256SUMS.txt│ └── BASES_E_ARTEFATOS.md├── scripts/│ ├── prepare_all_data.py│ ├── build_dataset_registry.py│ ├── build_index_registry.py│ ├── generate_checksums.py│ ├── run_experiment.py│ └── summarize_results.py├── docker-compose.yml├── Dockerfile├── environment.yml├── requirements.txt├── pyproject.toml├── CITATION.cff└── README.md