Retrieval Systems

Retrieval systems are a critical component of Retrieval Augmented Generation (RAG) that extract relevant information from extensive textual corpora, including knowledge bases, web pages, and other structured or unstructured data sources.
Retrieval systems enable RAG models to access and incorporate external knowledge, enhancing the accuracy, relevance, and factual grounding of generated outputs. This process significantly improves the model's ability to provide informed, up-to-date, and context-aware responses across a wide range of applications and domains.
The primary function of retrieval systems is to efficiently identify and rank the most pertinent passages, documents, or data points in response to input queries, contextual prompts, and specific information needs.


Retrieval Systems techniques:

  • Vector Embedding
  • Inverted Indices:
    - Definition: Data structure that maps terms to their locations in a document or collection.
    • Purpose: Enable fast full-text search and retrieval.
    • Components:
      • Dictionary of terms
      • Postings lists (document IDs or positions where each term occurs)
    • Advantages: Efficient for Boolean queries and phrase searches
  • Relevance Scoring:
    - Definition: Methods to quantify how well a document matches a query.
    • Purpose: Rank search results by their relevance to the user's query.
    • Common techniques:
      • TF-IDF
      • BM25 (Best Matching 25)
      • Language models (e.g., query likelihood)
      • Learning to Rank algorithms
    • Factors considered: Term Frequency (TF), document length, query term importance