Vector Database Management System for Similarity Search on Commodity Hardware

Vector database management systems support semantic search, recommendation, and retrieval-augmented generation, but many available deployments assume cloud infrastructure, specialist accelerators, or server-scale memory. This creates a practical problem for students, researchers, and small development teams who need reproducible vector search on ordinary workstations with limited memory and no GPU support. This study developed and evaluated a lightweight vector database management system for similarity search on commodity hardware. The system integrated HDF5-based persistent vector storage, FAISS IndexIVFPQ approximate nearest neighbour indexing, a modular query-processing pipeline, and correctness validation against exhaustive FAISS IndexFlatL2 ground truth. Evaluation used pre-trained GloVe embeddings with 50, 100, 200, and 300 dimensions, covering dataset sizes between 10,000 and 200,000 vectors. The evaluation reports retrieval accuracy, index memory footprint, query latency, and correctness validation. Across all datasets, the system achieved an average recall@10 of 0.8978, with all configurations exceeding the 0.85 retrieval target and recall@1 reaching 1.0000. IndexIVFPQ memory footprint remained between 0.37 MB and 30.50 MB, giving 5.2x to 7.5x compression relative to IndexFlatL2. Query latency ranged from 0.102 ms to 14.558 ms, with three of the four configurations remaining less than 10 ms. Correctness validation on the 100d and 300d benchmark configurations produced recall@10 values greater than 0.90 against exact search. The results indicate that an integrated HDF5 and FAISS IndexIVFPQ architecture provides a reproducible local vector search workflow on commodity hardware, although larger and higher-dimensional workloads require further tuning of index construction and query latency.

vector database, similarity search, FAISS, IndexIVFPQ, HDF5

1. Pan, J. J., Wang, J., & Li, G. (2024). Survey of vector database management systems. The VLDB Journal, 33(5), 1591–1615. [Google Scholar] [Crossref]

2. Taipalus, T. (2024). Vector database management systems: Fundamental concepts, use- cases, and current challenges. Cognitive Systems Research, 85, 101216. [Google Scholar] [Crossref]

3. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., & Jégou, H. (2024). The Faiss library. arXiv, 2401.08281. [Google Scholar] [Crossref]

4. Gao, J., Gou, Y., Xu, Y., Yang, Y., Long, C., & Wong, R. C.-W. (2024). Practical and asymptotically optimal quantization of high-dimensional vectors in Euclidean space for approximate nearest neighbor search. arXiv, 2409.09913. [Google Scholar] [Crossref]

5. Subramanya, S. J., Devvrit, F., Simhadri, H. V., Krishnawamy, R., & Kadekodi, R. (2019). DiskANN: Fast accurate billion-point nearest neighbor search on a single node. In Advances in Neural Information Processing Systems, 32, 13748–13758. [Google Scholar] [Crossref]

6. Harwood, B., Dezfouli, A., Chades, I., & Sanderson, C. (2025). Approximate nearest neighbour search on dynamic datasets: An investigation. In AI 2024: Advances in Artificial Intelligence, 15443, 95–106. Springer. [Google Scholar] [Crossref]

7. Wei, J., Peng, B., Lee, X., & Palpanas, T. (2024). DET-LSH: A locality-sensitive hashing scheme with dynamic encoding tree for approximate nearest neighbor search. Proceedings of the VLDB Endowment, 17(9), 2241–2254. [Google Scholar] [Crossref]

8. Kuffo, L., Krippner, E., & Boncz, P. (2025). PDX: A data layout for vector similarity search. Proceedings of the ACM on Management of Data, 3(3), 196, 1–26. [Google Scholar] [Crossref]

9. Kuffo, L., & Boncz, P. (2025). Bang for the buck: Vector search on cloud CPUs. In Proceedings of the 21st International Workshop on Data Management on New Hardware, 4, 1–8. [Google Scholar] [Crossref]

10. Wang, S., Zhao, Y., Xie, Y., Liu, Z., Hou, X., Zou, Q., & Wang, H. (2025). Towards reliable vector database management systems: A software testing roadmap for 2030. arXiv, 2502.20812. [Google Scholar] [Crossref]

11. Xie, Y., Hou, X., Zhao, Y., Wang, S., Chen, K., & Wang, H. (2025). Toward understanding bugs in vector database management systems. arXiv, 2506.02617. [Google Scholar] [Crossref]

12. Mohoney, J., Pacaci, A., Chowdhury, S. R., Minhas, U. F., Pound, J., Renggli, C., Reyhani, N., Ilyas, I. F., Rekatsinas, T., & Venkataraman, S. (2024). Incremental IVF index maintenance for streaming vector search. arXiv, 2411.00970. [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet