Bird
Raised Fist0

Prompt Engineering - Chain-of-Thought, Few-Shot, RAG Prompting Patterns

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Silent Recall Collapse
The issue started after a recent update to the prompt templates last Friday.

A customer support chatbot using a language model with Chain-of-Thought and Few-Shot prompting patterns to resolve user queries.

Symptom
The chatbot's accuracy remains high, but recall for resolving user queries has dropped to 0.
📡
Signals
metric
Recall metric for user query resolution
Recall dropped from 0.75 to 0.00 after the update
log
Logs of user interactions with the chatbot
Increased frequency of fallback responses post-update
distribution
Distribution of response types
Significant increase in 'I don't know' responses
code
Recent changes in prompt templates
New prompt templates deployed last Friday
feature_importance
Importance of prompt features in model output
Feature importance for 'example completions' dropped from 0.22 to 0.05
🔬
Hypotheses
The new prompt templates are not effectively guiding the model.
Supporting Signals
Recall dropped from 0.75 to 0.00New prompt templates deployed last Friday
Verify: Check if the new templates are missing critical examples or instructions.
with open('prompt_templates.txt') as f: print(f.read())
The Chain-of-Thought pattern is not being triggered correctly.
Supporting Signals
Increased frequency of fallback responsesSignificant increase in 'I don't know' responses
Verify: Analyze if the Chain-of-Thought logic is being bypassed.
print(model.chain_of_thought_triggered_count())
🎯
Root Cause
The new prompt templates inadvertently omitted key examples necessary for Chain-of-Thought reasoning.
Why It Happened

A deployment change introduced incomplete prompt templates that lacked necessary examples for effective reasoning.

🔧
Fix Strategy
ImmediateRollback to the previous prompt templates to restore functionality.
Short-termReview and update the prompt templates to include all necessary examples and instructions.
Long-termImplement a validation step for prompt templates to ensure completeness before deployment.
🐍
Python Fix
Python
with open('prompt_templates.txt', 'w') as f: f.write(previous_templates)
✗
Common Wrong Diagnoses
▼ Show
The model architecture is flawed.
Why candidates think this: The drastic drop in recall suggests a fundamental issue.
Why wrong: The architecture was unchanged; the issue arose after prompt template updates.
The data pipeline is corrupt.
Why candidates think this: A sudden drop in performance often points to data issues.
Why wrong: Data remained consistent; only prompt templates were altered.
📊
Monitoring to Add
  • Alert on significant drops in recall metrics.
  • Monitor distribution of response types for anomalies.
  • Track changes in feature importance related to prompt features.
📋
Process Change

Introduce a review process for prompt changes involving multiple stakeholders.

🏗️
Architectural Change

Develop a testing framework for prompt templates to simulate real-world interactions before deployment.

💡
Key Insight
Prompt templates are critical in guiding model reasoning; missing examples can silently collapse recall.

Practice

(1/5)
1. In the context of K-Means clustering, which initialization method is most likely to avoid poor local minima?
easy
A. K-Means++ initialization
B. Random initialization
C. Using centroids from a previous run
D. Initializing all centroids to the same point

Solution

  1. Step 1: Understand initialization impact

    K-Means++ initialization helps in spreading initial centroids to avoid poor local minima.

  2. Step 2: Eliminate wrong options

    Random initialization can lead to poor local minima. Using centroids from a previous run might not improve if the previous run was suboptimal. Initializing all centroids to the same point is clearly incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    K-Means++ spreads centroids well [OK]
Hint: K-Means++ spreads centroids well [OK]
Common Mistakes:
2. Which optimizer is most suitable for handling sparse data due to its adaptive learning rate per parameter?
easy
A. AdaGrad
B. Adam
C. SGD
D. RMSProp

Solution

  1. Step 1: Identify the optimizer's strength.

    AdaGrad adapts the learning rate for each parameter, making it suitable for sparse data.

  2. Step 2: Eliminate wrong options.

    SGD does not adapt learning rates. Adam and RMSProp adapt but are not specifically designed for sparse data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    AdaGrad adapts per parameter for sparse data. [check]
Hint: AdaGrad adapts per parameter for sparse data. [check]
Common Mistakes:
3. In the context of evaluating models on imbalanced datasets, which metric is more sensitive to changes in the minority class predictions?
easy
A. ROC-AUC
B. Accuracy
C. PR-AUC
D. F1-Score

Solution

  1. Step 1: Understand metric sensitivity

    PR-AUC focuses on precision and recall, which are directly affected by the minority class predictions.

  2. Step 2: Eliminate wrong options

    ROC-AUC is less sensitive to class imbalance. Accuracy can be misleading in imbalanced datasets. F1-Score is related but not as directly sensitive as PR-AUC.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    PR-AUC focuses on minority class. [OK]
Hint: PR-AUC focuses on minority class. [OK]
Common Mistakes:
4. When choosing between LSTM and GRU for a sequence modeling task, which scenario would favor using GRU over LSTM?
medium
A. When computational efficiency is a priority.
B. When the model needs to capture long-term dependencies.
C. When the dataset is extremely large.
D. When the model requires fine-grained control over memory updates.

Solution

  1. Step 1: Compare LSTM and GRU.

    GRUs are simpler and computationally more efficient than LSTMs.

  2. Step 2: Match to scenario.

    GRU is preferred when computational efficiency is a priority.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GRU is simpler, faster than LSTM. [check]
Hint: GRU is simpler, faster than LSTM. [check]
Common Mistakes:
5. You are tasked with building a model for a financial dataset with many features, some of which are irrelevant. Which regularization technique would you apply and why?
hard
A. L1 Lasso, because it can eliminate irrelevant features
B. L2 Ridge, because it handles multicollinearity well
C. Elastic Net, because it balances feature selection and multicollinearity
D. No regularization, because it avoids biasing the model

Solution

  1. Step 1: Identify the need for feature selection and handling multicollinearity.

    Financial datasets often have correlated features and irrelevant ones.

  2. Step 2: Choose the technique that addresses both issues.

    Elastic Net combines L1 for feature selection and L2 for handling multicollinearity.

  3. Step 3: Eliminate wrong options.

    L1 Lasso struggles with multicollinearity. L2 Ridge does not eliminate features. No regularization ignores the problem.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Elastic Net = L1 + L2 for finance. [OK]
Hint: Elastic Net = L1 + L2 for finance. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard