Challenges in language processing in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is ambiguity in language processing?

Ambiguity happens when a word or sentence can have more than one meaning, making it hard for computers to understand the right one.

beginner

Why is context important in language processing?

Context helps computers understand the meaning of words or sentences by looking at surrounding words or the situation, just like people do.

intermediate

What makes sarcasm difficult for language models to detect?

Sarcasm often means the opposite of the words used, and it depends on tone or situation, which computers find hard to recognize.

intermediate

Explain the challenge of handling idioms in language processing.

Idioms are phrases whose meaning is different from the literal words, so computers need special knowledge to understand them correctly.

beginner

How does language diversity pose a challenge in language processing?

Different languages have unique grammar, vocabulary, and expressions, so models trained on one language may not work well on others.

What does ambiguity in language processing mean?

A A word or sentence has multiple meanings

B A sentence is too long

C A word is misspelled

D A sentence is grammatically incorrect

Ambiguity means a word or sentence can have more than one meaning, causing confusion.

Why is context important for understanding language?

A It makes sentences shorter

B It helps computers guess the meaning based on surrounding words

C It removes all punctuation

D It translates words into numbers

Context helps computers understand meaning by looking at nearby words or situation.

Which of these is a challenge for detecting sarcasm?

A Sarcasm is only in written text

B Sarcasm uses very long sentences

C Sarcasm always uses slang

D Sarcasm often means the opposite of the words used

Sarcasm often means the opposite of what is said, which is hard for computers to detect.

What makes idioms hard for language models?

A Their meaning is different from the literal words

B They are always in foreign languages

C They contain numbers

D They are always questions

Idioms have meanings that are not obvious from the words themselves.

How does language diversity affect language processing?

A Languages do not affect processing

B All languages use the same alphabet

C Different languages have unique grammar and vocabulary

D Only English is used in language processing

Different languages have unique rules and words, making it hard for one model to work for all.

Describe three main challenges computers face when processing human language.

Think about meanings that can be confusing or hidden.

Explain why language diversity is a problem for building language models.

Consider how languages differ from each other.

Practice

1. Why is language processing challenging for computers?

easy

A. Because computers do not have enough memory

B. Because computers cannot store large amounts of data

C. Because language has only one fixed meaning per word

D. Because words can have multiple meanings depending on context

Solution

1. Step 1: Understand word ambiguity in language

   Words often have several meanings, which depend on the context they appear in.

2. Step 2: Relate ambiguity to computer difficulty

   Computers struggle to pick the correct meaning without understanding context, making language processing hard.

3. Final Answer:

   Because words can have multiple meanings depending on context -> Option D

4. Quick Check:

   Word ambiguity = D [OK]

Hint: Remember: words change meaning with context [OK]

Common Mistakes:

• Thinking each word has only one meaning
• Assuming computers lack memory causes difficulty
• Confusing data storage with language understanding

2. Which of the following is the correct way to represent a sentence tokenization step in Python using NLTK?

easy

A. tokens = nltk.word_tokenize(sentence)

B. tokens = nltk.sentence_tokenize(sentence)

C. tokens = nltk.tokenize_words(sentence)

D. tokens = nltk.split(sentence)

Solution

1. Step 1: Recall NLTK tokenization functions

   NLTK uses word_tokenize() to split sentences into words (tokens).

2. Step 2: Identify correct function for word tokenization

   word_tokenize() is the correct function; sentence_tokenize() does not exist, and others are invalid.

3. Final Answer:

   tokens = nltk.word_tokenize(sentence) -> Option A

4. Quick Check:

   NLTK word tokenization = C [OK]

Hint: Use word_tokenize() for splitting sentence into words [OK]

Common Mistakes:

• Using sentence_tokenize() which is not a valid function
• Confusing word_tokenize() with tokenize_words()
• Trying to split sentence with split() method

3. Given the code below, what will be the output?

sentence = "I saw her duck." 
tokens = sentence.split()
print(tokens)

medium

A. ['I', 'saw', 'her', 'duck.']

B. ['I', 'saw', 'her', 'duck']

C. ['I', 'saw', 'her', 'duck', '.']

D. ['I saw her duck']

Solution

1. Step 1: Understand split() behavior on string

   split() divides the string by spaces, keeping punctuation attached to words.

2. Step 2: Apply split() to the sentence

   Splitting "I saw her duck." by spaces results in ['I', 'saw', 'her', 'duck.'] with the period attached to 'duck.'

3. Final Answer:

   ['I', 'saw', 'her', 'duck.'] -> Option A

4. Quick Check:

   split() keeps punctuation attached = A [OK]

Hint: split() keeps punctuation with words [OK]

Common Mistakes:

• Assuming split() removes punctuation
• Expecting punctuation as separate token
• Confusing split() with word_tokenize()

4. The following code tries to remove stopwords from a list of tokens but does not work as expected. What is the error?

stopwords = ['the', 'is', 'at']
tokens = ['the', 'cat', 'is', 'on', 'the', 'mat']
filtered = [word for word in tokens if word not in stopwords()]
print(filtered)

medium

A. tokens should be converted to a set before filtering

B. The list comprehension syntax is incorrect

C. stopwords is a list, not a function; should not use parentheses

D. The print statement is missing parentheses

Solution

1. Step 1: Identify the error in stopwords usage

   stopwords is a list, but the code uses stopwords() as if it were a function.

2. Step 2: Correct the usage of stopwords

   Remove parentheses to use stopwords as a list: use 'word not in stopwords' instead of 'stopwords()'.

3. Final Answer:

   stopwords is a list, not a function; should not use parentheses -> Option C

4. Quick Check:

   stopwords list misuse = B [OK]

Hint: Lists are not functions; avoid parentheses [OK]

Common Mistakes:

• Using parentheses after list variable
• Thinking tokens must be sets to filter
• Misreading list comprehension syntax

5. Which challenge best explains why idioms like "kick the bucket" are hard for AI to understand?

hard

A. Idioms are always spelled incorrectly

B. Idioms have meanings different from the literal words

C. Idioms contain rare words not in dictionaries

D. Idioms are too long for AI to process

Solution

1. Step 1: Understand idioms in language

   Idioms are phrases whose meaning is not the sum of their individual words.

2. Step 2: Relate idioms to AI language challenges

   AI struggles because it cannot infer the non-literal meaning from the literal words alone.

3. Final Answer:

   Idioms have meanings different from the literal words -> Option B

4. Quick Check:

   Idioms = non-literal meaning = A [OK]

Hint: Idioms mean more than their words [OK]

Common Mistakes:

• Thinking idioms are misspelled
• Assuming idioms use rare words
• Believing idioms are too long to process