Question Answering, Information Retrieval, and RAG

Question Answering, Information Retrieval, and RAG

Have you ever wondered how search engines quickly find relevant information or how AI assistants answer your questions? The secret lies in a combination of techniques, including Question Answering (QA), Information Retrieval (IR), and Retrieval-Augmented Generation (RAG). Let's break down these concepts in a way that's easy to understand.

Information Retrieval: Finding What You Need

Imagine you have a vast library of documents and you need to find the ones that are relevant to a specific topic. That's essentially what Information Retrieval (IR) does. It's the process of finding documents that match a user's query.

Term Frequency-Inverse Document Frequency (TF-IDF)

One of the classic methods for IR is TF-IDF. It helps determine how important a word is to a document in a collection of documents. It works in two parts:

• Term Frequency (TF): How often a term appears in a document. The more a word appears, the more important it might be to that document.
• Inverse Document Frequency (IDF): Measures how rare a word is across all documents. Words that appear in many documents are considered less important for distinguishing between documents.

TF-IDF combines these two measures to give a weight to each word in a document. Here's how each component works:

Term Frequency (TF)

Instead of just counting the raw number of times a word appears in a document, we often use a log-weighted term frequency. This helps to reduce the impact of very frequent words. The formula looks like this:

tft,d = 1 + log₁₀(count(t,d)) if count(t,d) > 0, otherwise 0

For example, if a word appears once in a document, its TF would be 1. If it appears 10 times, its TF would be 2, and so on.

Inverse Document Frequency (IDF)

The inverse document frequency measures how rare a word is across the entire collection of documents. Words that appear in many documents are considered less important. The formula for IDF is:

idf_t = log₁₀(N / df_t)

Where N is the total number of documents, and df_t is the number of documents in which term t occurs.

Here's a table showing example IDF values for words in Shakespeare's plays:

Word	Document Frequency (DF)	Inverse Document Frequency (IDF)
Romeo	1	1.57
Salad	2	1.27
Falstaff	4	0.967
Forest	12	0.489
Good	37	0

Real-World Example: Imagine searching for "best running shoes" on an e-commerce website. TF-IDF helps the search engine prioritize products where the words "running" and "shoes" appear frequently (high TF) and where the word "best" isn't overly common across all products (high IDF).

Document Scoring

Once we have the TF-IDF values for each term in the query and the documents, we can score each document by the cosine of its vector with the query vector:

score(q,d) = cos(q,d) = q · d / (|q||d|)

This score represents the similarity between the query and the document.

BM25: A More Advanced Scoring Function

BM25 (Best Matching 25) is an advanced scoring function that improves upon TF-IDF by adding parameters to fine-tune the balance between term frequency, IDF, and document length normalization. The BM25 score of a document d given a query q is:

BM25 = Σ_t∈q IDF(t) * (tf_t,d / (k * (1 - b + b * (|d| / |d_avg|)) + tf_t,d))

Where:

• |d_avg| is the length of the average document.
• k is a parameter that adjusts the balance between term frequency and IDF.
• b is a parameter that controls the importance of document length normalization.

Reasonable values are k = [1.2,2] and b = 0.75.

Stop Words

Stop words are common words (like "the," "a," "is") that are often removed from the query and documents before processing. The idea is that these words don't contribute much to the meaning and can be ignored. However, modern systems often rely on IDF weighting to downweight these words instead.

Inverted Index

To quickly find documents containing specific words, IR systems use an "inverted index." Think of it as a dictionary where each word points to a list of documents that contain that word. This makes searching much faster.

Real-World Example: A library uses an inverted index to locate books. Instead of checking every book in the library for a specific word, they can quickly find the books that contain the word using the index.

Evaluating Information Retrieval Systems

So, how do we know if an IR system is performing well? We use metrics like:

• Precision: Of the documents returned, what percentage are actually relevant?
• Recall: Of all the relevant documents in the collection, what percentage did the system return?
• Mean Average Precision (MAP): Provides a single metric to compare the relevance and ranking quality of the retrieved documents.

Imagine a user searches for "Jaguar". A good system will return articles about the animal and not just car reviews.

Information Retrieval with Dense Vectors

While TF-IDF and BM25 are good, they only work if the query and document share the exact same words. What if someone uses a synonym? That's where dense vectors come in. Instead of using word counts, we use word embeddings to represent the meaning of words and documents.

One way to create dense vectors is to use models like BERT. Here are two common approaches:

• Joint Encoding: Pass both the query and document through BERT at the same time. This allows BERT to understand the relationship between the query and document. However, it's computationally expensive.
• Bi-Encoders: Use separate BERT models to encode the query and document independently. This is faster, as you can pre-compute document embeddings.

A tradeoff is done in computation power but gives less accurate since the relevance decision can’t take full advantage of all the possible meaning interactions between all the tokens in the query and the tokens in the document.

Real-World Example: Suppose you search "delicious apple pie recipe." A system using dense vectors could still return results containing "tasty apple tart" because the embeddings capture the semantic similarity between "pie" and "tart," and "delicious" and "tasty."

ColBERT

ColBERT is an intermediate approach that separately encodes the query and document into contextual representations for each token, pre-storing BERT representations of each document word. For each token in the query, it finds the most contextually similar token in the document, and then sums up these similarities to provide a similarity score.

Answering Questions with RAG: Retrieval-Augmented Generation

RAG is a powerful framework for question answering. It involves two main steps:

• Retrieval: Find relevant documents or passages using IR techniques (like TF-IDF or dense vectors).
• Generation: Use a large language model (LLM) to generate an answer based on the retrieved documents.

Real-World Example: You ask, "Who invented the telephone?" RAG would first retrieve documents about Alexander Graham Bell and his invention. Then, the LLM would use those documents to generate the answer, "Alexander Graham Bell invented the telephone."

Retrieval-Augmented Generation (RAG)

The main idea behind RAG is to improve the quality of answers by conditioning the generation on retrieved passages. This can address hallucination, provide textual evidence, and answer questions from proprietary data.

A language model is given a question and a token suggesting an answer:

Q: Who wrote the book "The Origin of Species"? A:

The model then generates an answer based on this text, autoregressively.

To address the issues of hallucination and lack of evidence, RAG conditions the generation on retrieved passages:

Schematic of a RAG Prompt

retrieved passage 1

retrieved passage 2

...

retrieved passage n

Based on these texts, answer this question:

Q: Who wrote the book "The Origin of Species"?

A:

Question Answering Datasets

Many question answering datasets are used for training and evaluating language models. These datasets vary in their original purpose, whether they were natural information-seeking questions or designed for probing.

• Natural Questions: A set of anonymized English queries to the Google search engine and their answers.
• MS MARCO: A collection of datasets including 1 million real anonymized English questions from Microsoft Bing query logs together with a human-generated answer and 9 million passages.
• MMLU (Massive Multitask Language Understanding): A dataset of knowledge and reasoning questions in 57 areas including medicine, mathematics, computer science, law, and others.

Evaluating Question Answering

Here are three common techniques to evaluate question-answering systems:

• Exact Match: The % of predicted answers that match the gold answer exactly.
• F1 Score: The average token overlap between predicted and gold answers.
• Mean Reciprocal Rank (MRR): is designed for systems that return a short ranked list of answers or passages for each test set question, which we can compare against the (human-labeled) correct answer.

Summary

QA, IR, and RAG are powerful tools for finding and understanding information. By combining these techniques, we can build intelligent systems that can answer questions, find relevant documents, and provide valuable insights. Next time you use a search engine or chat with an AI assistant, remember the magic happening behind the scenes!