NLP vs NLU vs NLG - Experiment Comparison

Experiment - NLP vs NLU vs NLG

Problem:You want to understand the differences between NLP (Natural Language Processing), NLU (Natural Language Understanding), and NLG (Natural Language Generation) by building simple models that demonstrate each concept.

Current Metrics:No models built yet; no metrics available.

Issue:Lack of practical understanding of how NLP, NLU, and NLG differ and relate to each other.

Your Task

Build three simple models: one for NLP text preprocessing, one for NLU intent classification, and one for NLG text generation. Show how each works and compare their outputs.

Use Python with basic libraries like sklearn and transformers.

Keep models simple and runnable on a small dataset.

Do not use complex deep learning architectures.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import re
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

# NLP: Text preprocessing function
def preprocess_text(text):
    text = text.lower()
    text = re.sub(r'[^a-z0-9 ]', '', text)
    tokens = text.split()
    return tokens

# Sample texts
texts = ["Hello, how are you?", "Book a flight to Paris", "What is the weather today?"]

# NLP step: preprocess texts
preprocessed_texts = [preprocess_text(t) for t in texts]

# NLU: Intent classification
# Sample data
X_train = ["book flight", "weather today", "hello there"]
y_train = ["BookFlight", "GetWeather", "Greeting"]

# Simple pipeline: vectorizer + classifier
model = make_pipeline(CountVectorizer(), MultinomialNB())
model.fit(X_train, y_train)

# Predict intents for sample texts
intents = model.predict(texts)

# NLG: Simple template-based generation
def generate_response(intent):
    responses = {
        "BookFlight": "I can help you book a flight. Where do you want to go?",
        "GetWeather": "The weather today is sunny with a high of 25°C.",
        "Greeting": "Hello! How can I assist you today?"
    }
    return responses.get(intent, "Sorry, I don't understand.")

responses = [generate_response(i) for i in intents]

# Output results
print("NLP Preprocessed Texts:", preprocessed_texts)
print("NLU Predicted Intents:", intents)
print("NLG Generated Responses:", responses)

Created a text preprocessing function to demonstrate NLP basics.

Built a simple intent classifier using Naive Bayes to show NLU.

Implemented a template-based response generator to illustrate NLG.

Results Interpretation

Before: No understanding or models for NLP, NLU, NLG.

After:

• NLP: Text is cleaned and tokenized: [['hello', 'how', 'are', 'you'], ['book', 'a', 'flight', 'to', 'paris'], ['what', 'is', 'the', 'weather', 'today']]
• NLU: Intents predicted for each text: ['Greeting', 'BookFlight', 'GetWeather']
• NLG: Responses generated based on intent: ['Hello! How can I assist you today?', 'I can help you book a flight. Where do you want to go?', 'The weather today is sunny with a high of 25°C.']

NLP prepares and cleans text data, NLU understands the meaning or intent behind text, and NLG creates new text responses. Together, they form a pipeline for machines to interact with human language.

Bonus Experiment

Try replacing the template-based NLG with a small pretrained language model to generate more natural responses.

💡 Hint

Use Hugging Face's transformers library with a lightweight model like GPT-2 small for text generation.