What Generative AI actually is in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - What Generative AI actually is

Generative AI creates new content like text, images, or music by learning patterns from existing data. It starts with input data, learns features, trains a model to generate similar outputs, and then produces new examples.

Data Flow - 5 Stages

1Data Collection

10000 samples x variable length text→Gather diverse text data from books, articles, and websites→10000 samples x variable length text

"The sun rises in the east."

↓

2Preprocessing

10000 samples x variable length text→Clean text, tokenize words, convert to numbers→10000 samples x 50 tokens

[101, 2023, 2003, 1037, 2742, 102]

↓

3Feature Engineering

10000 samples x 50 tokens→Create embeddings representing word meanings→10000 samples x 50 tokens x 768 features

[[0.12, -0.05, ..., 0.33], [...], ...]

↓

4Model Training

10000 samples x 50 tokens x 768 features→Train a neural network to predict next word→Trained model weights

Model learns to predict 'sun' after 'The'

↓

5Generation

Seed text tokens→Use trained model to generate new text tokens→Generated text sequence

"The sun rises in the morning and shines brightly."

Training Trace - Epoch by Epoch

Loss
2.5 |*****
2.0 |****
1.5 |***
1.0 |**
0.5 |*
    +------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	2.5	0.30	Model starts learning basic word patterns
2	1.8	0.45	Model improves predicting next words
3	1.3	0.60	Model captures grammar and simple context
4	1.0	0.70	Model generates more coherent sentences
5	0.8	0.78	Model produces fluent and relevant text

Prediction Trace - 5 Layers

Layer 1: Input Tokenization

Layer 2: Embedding Layer

Layer 3: Neural Network Layers

Layer 4: Softmax Activation

Layer 5: Word Selection

Model Quiz - 3 Questions

What does the embedding layer do in Generative AI?

A Cleans the input text

B Generates the final text output

C Turns words into numbers representing their meaning

D Splits text into sentences

Key Insight

Generative AI learns patterns from data to create new content by predicting what comes next. It transforms raw input into meaningful features, trains a model to understand context, and then generates new, coherent outputs.

Practice

1. What is the main purpose of Generative AI?

easy

A. To store large amounts of data efficiently

B. To delete irrelevant information from datasets

C. To only classify existing data into categories

D. To create new content by learning from examples

Solution

1. Step 1: Understand the role of Generative AI

   Generative AI learns patterns from data and creates new content based on those patterns.

2. Step 2: Compare options with the definition

   Only To create new content by learning from examples describes creating new content by learning from examples, which matches the main purpose.

3. Final Answer:

   To create new content by learning from examples -> Option D

4. Quick Check:

   Generative AI = create new content [OK]

Hint: Generative AI makes new stuff from learned data [OK]

Common Mistakes:

• Confusing Generative AI with data storage
• Thinking it only classifies data
• Believing it deletes data

2. Which of the following is the correct way to describe Generative AI in simple code terms?

easy

A. Train a model, then generate new outputs

B. Only collect data without processing

C. Manually write all new content

D. Delete old models before training

Solution

1. Step 1: Identify the typical workflow of Generative AI

   Generative AI involves training a model on data and then using it to create new outputs.

2. Step 2: Match options to this workflow

   Train a model, then generate new outputs correctly states this process, while others describe unrelated or incorrect actions.

3. Final Answer:

   Train a model, then generate new outputs -> Option A

4. Quick Check:

   Train then generate = correct process [OK]

Hint: Generative AI = train model + create new data [OK]

Common Mistakes:

• Thinking Generative AI only collects data
• Assuming manual content creation is AI
• Confusing training with deleting models

3. Consider this Python-like pseudocode for a simple Generative AI process:

model = train(data)
new_content = model.generate()

What will new_content most likely contain?

medium

A. A new example similar to the training data

B. The original training data unchanged

C. An error message because generate() is undefined

D. An empty output with no content

Solution

1. Step 1: Understand the code steps

   The code trains a model on data, then calls generate() to create new content.

2. Step 2: Predict the output of generate()

   Generate() produces new content similar to what the model learned, not the original data or errors.

3. Final Answer:

   A new example similar to the training data -> Option A

4. Quick Check:

   generate() = new similar content [OK]

Hint: generate() creates new data like training examples [OK]

Common Mistakes:

• Thinking generate() returns original data
• Assuming generate() causes an error
• Expecting empty output

4. The following code is intended to train a Generative AI model and generate new content:

model = train(data)
new_content = model.generate(data)

What is the likely problem here?

medium

A. model should be a list, not a model object

B. train() should not take data as input

C. generate() should not take data as input after training

D. new_content should be assigned before training

Solution

1. Step 1: Review typical usage of generate()

   After training, generate() usually creates new content without needing input data again.

2. Step 2: Identify misuse in code

   Passing data to generate() is incorrect; it should generate based on learned patterns alone.

3. Final Answer:

   generate() should not take data as input after training -> Option C

4. Quick Check:

   generate() no input needed [OK]

Hint: generate() uses learned model, no extra data input [OK]

Common Mistakes:

• Thinking train() shouldn't take data
• Confusing model type
• Assigning new_content before training

5. You want to create a Generative AI that writes short poems. Which steps best describe the process?

hard

A. Write poems manually, then use AI to classify them

B. Collect poem examples, train model on them, generate new poems

C. Train model on random text, then delete training data

D. Generate poems first, then collect examples to train

Solution

1. Step 1: Understand the goal of Generative AI for poems

   The AI needs to learn from existing poems to create new ones.

2. Step 2: Identify the correct sequence of actions

   Collecting examples, training the model, then generating new poems is the correct order.

3. Final Answer:

   Collect poem examples, train model on them, generate new poems -> Option B

4. Quick Check:

   Learn from examples, then create new [OK]

Hint: Train on examples first, then generate new content [OK]

Common Mistakes:

• Trying to generate before training
• Confusing classification with generation
• Deleting training data too early