Question answering in Prompt Engineering / GenAI - Deep Dive

Overview - Question answering

What is it?

Question answering is a technology that lets computers read or listen to information and then answer questions about it. It works by understanding the question, finding the right information, and giving a clear answer. This can happen with text, speech, or images as the source of information. It helps people get quick answers without searching through lots of data themselves.

Why it matters

Without question answering, people would spend much more time searching for information manually, which can be slow and frustrating. This technology makes it easier to access knowledge instantly, improving learning, customer support, and decision-making. It powers virtual assistants, search engines, and educational tools, making information more accessible and useful in everyday life.

Where it fits

Before learning question answering, you should understand basic natural language processing concepts like text representation and simple language understanding. After mastering question answering, you can explore advanced topics like conversational AI, knowledge graphs, and multi-modal AI systems that combine text, images, and speech.

Mental Model

Core Idea

Question answering is about teaching a computer to understand a question, find the right information, and give a clear, direct answer.

Think of it like...

It's like asking a knowledgeable friend a question and they quickly skim their memory or books to give you the best answer without you having to search yourself.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Question    │─────▶│ Understanding │─────▶│ Information   │
│   Input       │      │   Module      │      │ Retrieval    │
└───────────────┘      └───────────────┘      └───────────────┘
                                         │
                                         ▼
                                ┌───────────────┐
                                │ Answer Output │
                                └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Question Answering

Concept: Introduce the basic idea of question answering as a task where a computer answers questions from text or data.

Question answering means a computer reads or listens to some information and then answers questions about it. For example, if you ask 'What is the capital of France?', the computer should answer 'Paris'. This is different from just searching because the computer tries to understand the question and give a direct answer.

Result

You understand that question answering is about getting direct answers from information using a computer.

Understanding the goal of question answering helps you see why it is different from simple search or keyword matching.

2

Foundation

Types of Question Answering Systems

Concept: Learn about the main types: extractive, abstractive, and knowledge-based question answering.

Extractive QA finds exact words or sentences from a text to answer questions. Abstractive QA creates new sentences to answer, like a summary. Knowledge-based QA uses facts stored in databases or graphs to answer. Each type works differently but aims to give the best answer.

Result

You can identify different QA types and their basic working style.

Knowing QA types helps you understand the variety of approaches and when each is useful.

3

Intermediate

How Models Understand Questions

🤔Before reading on: do you think models understand questions by memorizing words or by grasping meaning? Commit to your answer.

Concept: Explore how models convert questions into numbers and capture meaning using embeddings and language understanding.

Models turn words into numbers called embeddings that capture meaning. They use these to understand the question's intent, not just the words. For example, 'Where is the Eiffel Tower?' and 'Location of Eiffel Tower?' have similar embeddings, so the model knows they ask the same thing.

Result

You see how models represent questions to find relevant answers.

Understanding embeddings is key to grasping how models 'understand' language beyond just matching words.

4

Intermediate

Finding Answers in Text

🤔Before reading on: do you think the model reads the whole text or just parts to find answers? Commit to your answer.

Concept: Learn how models scan or attend to parts of text to locate the answer span or generate a response.

Models use attention mechanisms to focus on parts of the text that likely contain the answer. In extractive QA, the model predicts the start and end positions of the answer in the text. This lets it pick exact phrases as answers.

Result

You understand how models pinpoint answers inside large texts.

Knowing attention helps explain how models handle long texts and find precise answers.

5

Intermediate

Training Question Answering Models

🤔Before reading on: do you think models learn from random text or from question-answer pairs? Commit to your answer.

Concept: Understand that models learn by seeing many examples of questions and correct answers during training.

QA models are trained on datasets with questions and their correct answers. The model adjusts itself to predict answers that match the training data. This process is called supervised learning. The more diverse the data, the better the model can answer new questions.

Result

You see how training data shapes model ability to answer questions.

Recognizing the role of training data explains why QA models improve with more and better examples.

6

Advanced

Handling Ambiguous or Complex Questions

🤔Before reading on: do you think models can always answer tricky questions perfectly? Commit to your answer.

Concept: Explore challenges when questions are unclear, multi-part, or need reasoning beyond text retrieval.

Some questions need combining facts, understanding context, or reasoning steps. Models may struggle if the question is vague or requires knowledge not in the text. Advanced QA systems use multiple steps, external knowledge, or reasoning modules to handle these cases.

Result

You understand the limits of basic QA and the need for complex reasoning.

Knowing these challenges prepares you to appreciate advanced QA techniques and their complexity.

7

Expert

Integrating Knowledge Graphs with QA Models

🤔Before reading on: do you think QA models only use text or can they also use structured knowledge? Commit to your answer.

Concept: Learn how combining text-based QA with knowledge graphs improves accuracy and reasoning.

Knowledge graphs store facts as connected entities and relationships. Integrating them with QA models lets the system check facts, answer questions needing logic, and handle queries beyond text. This hybrid approach is used in production systems for better reliability.

Result

You see how combining different data sources enhances QA capabilities.

Understanding hybrid QA systems reveals how experts build robust, real-world question answering.

Under the Hood

Question answering models process input questions by converting words into numerical vectors that capture meaning. They then use neural networks, often transformers, to attend to relevant parts of the input text or knowledge base. The model predicts answer spans or generates answers by learning patterns from large datasets. Attention mechanisms allow focusing on important words, while training adjusts model weights to improve accuracy.

Why designed this way?

QA systems evolved from simple keyword search to deep learning to better understand natural language and context. Early methods lacked understanding and gave poor answers. Neural networks with attention were designed to mimic human focus on relevant information. Combining text with structured knowledge was introduced to overcome limits of text-only models and improve reasoning.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Question    │─────▶│ Embedding     │─────▶│ Transformer   │
│   Input       │      │ Layer         │      │ Layers       │
└───────────────┘      └───────────────┘      └───────────────┘
                                         │
                                         ▼
                                ┌───────────────┐
                                │ Attention     │
                                │ Mechanism     │
                                └───────────────┘
                                         │
                                         ▼
                                ┌───────────────┐
                                │ Answer Output │
                                └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do QA models always find the exact answer in the text? Commit yes or no.

Common Belief: QA models always find the exact answer phrase in the text.

Reality: Some QA models generate answers by rephrasing or summarizing, not just extracting exact text.

Why it matters: Assuming all answers come from exact text can limit understanding of abstractive QA and cause wrong expectations.

Quick: Do you think QA models understand questions like humans do? Commit yes or no.

Common Belief: QA models truly understand questions like a human would.

Reality: Models use statistical patterns and embeddings to approximate understanding but do not have true comprehension or reasoning like humans.

Why it matters: Believing models fully understand can lead to overtrust and ignoring their limitations or errors.

Quick: Do you think more data always means better QA performance? Commit yes or no.

Common Belief: More training data always improves QA model performance.

Reality: More data helps but quality, diversity, and relevance of data matter more; poor data can harm performance.

Why it matters: Ignoring data quality can waste resources and produce biased or inaccurate models.

Quick: Do you think QA systems can answer any question perfectly? Commit yes or no.

Common Belief: QA systems can answer any question perfectly if trained well.

Reality: QA systems struggle with ambiguous, multi-step, or knowledge-outside-training questions and can make mistakes.

Why it matters: Overestimating QA abilities can cause critical errors in applications like healthcare or law.

Expert Zone

1

QA models often rely heavily on pretraining on large language corpora before fine-tuning on QA tasks, which greatly affects performance.

2

The choice of context length and how to chunk long documents impacts the model's ability to find correct answers in large texts.

3

Hybrid QA systems that combine neural models with symbolic reasoning or knowledge graphs can handle complex queries better but require careful integration.

When NOT to use

Question answering is not ideal when questions require deep reasoning, personal opinions, or real-time sensory data. In such cases, rule-based systems, expert systems, or interactive dialogue agents may be better.

Production Patterns

In production, QA systems often use a pipeline: question understanding, document retrieval, answer extraction, and answer ranking. They integrate with search engines and knowledge bases, use caching for speed, and apply confidence scoring to decide when to ask for human help.

Connections

Information Retrieval

Question answering builds on information retrieval by adding understanding and direct answering rather than just listing documents.

Knowing how search engines find documents helps understand the first step in many QA systems.

Cognitive Psychology

QA models mimic aspects of human memory and attention to find and recall information.

Understanding human attention and memory helps explain why attention mechanisms improve QA performance.

Legal Reasoning

Both QA and legal reasoning require extracting relevant facts and applying rules to answer questions.

Studying legal reasoning shows how complex, multi-step question answering can be structured and verified.

Common Pitfalls

#1 Expecting the model to answer questions outside its training data.

Wrong approach:

Asking a QA model about very recent events or niche topics without updating its knowledge base.

Correct approach:

Regularly update or fine-tune the model with new data or use external knowledge sources for fresh information.

Root cause:

Misunderstanding that QA models rely on learned data and cannot guess unknown facts.

#2 Feeding very long documents without preprocessing.

Wrong approach:

Inputting entire books or articles directly into the QA model without chunking or summarizing.

Correct approach:

Split long texts into manageable chunks or use retrieval to select relevant parts before answering.

Root cause:

Not knowing model input size limits and how attention works.

#3 Ignoring ambiguous questions and expecting precise answers.

Wrong approach:

Using QA models on vague questions like 'Tell me about history' expecting a single answer.

Correct approach:

Clarify or rephrase questions to be specific or use dialogue systems to gather more details.

Root cause:

Assuming QA models can handle all question types equally well.

Key Takeaways

Question answering teaches computers to understand questions and find direct answers from information.

Different QA types exist: extractive, abstractive, and knowledge-based, each suited for different tasks.

Models use embeddings and attention to grasp question meaning and locate answers in text.

Training on diverse question-answer pairs is essential for good performance but has limits.

Advanced QA combines text with knowledge graphs and reasoning to handle complex questions.