Bidirectional LSTM in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output shape of Bidirectional LSTM layer

Consider the following Keras code snippet that creates a Bidirectional LSTM layer. What is the shape of the output tensor after passing an input batch of shape (32, 10, 8) through this layer?

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.models import Sequential

model = Sequential()
model.add(Bidirectional(LSTM(16, return_sequences=False), input_shape=(10, 8)))

output = model.layers[0].output_shape

NLP

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.models import Sequential

model = Sequential()
model.add(Bidirectional(LSTM(16, return_sequences=False), input_shape=(10, 8)))

output = model.layers[0].output_shape

A (None, 16)

B (None, 32)

C (None, 64)

D (None, 10, 32)

💡 Hint

Remember that Bidirectional doubles the units of the LSTM output when return_sequences=False.

The LSTM has 16 units. Bidirectional wraps it and concatenates forward and backward outputs, doubling the units to 32. Since return_sequences=False, output shape is (batch_size, 32). Batch size is None (unspecified).

❓ Model Choice

intermediate

Choosing Bidirectional LSTM for sequence classification

You want to build a model to classify movie reviews as positive or negative based on the text. Which model architecture below best uses a Bidirectional LSTM for this task?

A Embedding -> Dense(64, activation='relu') -> Bidirectional LSTM (units=64) -> Dense(1, activation='sigmoid')

B Embedding -> LSTM (units=64, return_sequences=True) -> Dense(1, activation='sigmoid')

C Embedding -> Bidirectional LSTM (units=64, return_sequences=True) -> Dense(1, activation='sigmoid')

D Embedding -> Bidirectional LSTM (units=64, return_sequences=False) -> Dense(1, activation='sigmoid')

💡 Hint

For classification, the LSTM should output a single vector per sample, not a sequence.

Option D uses Bidirectional LSTM with return_sequences=False, producing one vector per input sequence, suitable for classification. Options A and C output sequences, which is not ideal here. Option D incorrectly places a Dense layer before LSTM, which is not standard.

❓ Hyperparameter

advanced

Effect of return_sequences in Bidirectional LSTM

What is the effect of setting return_sequences=True in a Bidirectional LSTM layer in Keras?

A The layer returns the input sequence unchanged.

B The layer returns the cell state instead of the hidden state.

C The layer returns the hidden state for each time step, producing a 3D tensor output.

D The layer returns only the last hidden state as a 2D tensor output.

💡 Hint

Think about whether the output keeps the time dimension or not.

When return_sequences=True, the LSTM returns the hidden state for every time step, so the output shape is (batch_size, time_steps, units*2) for Bidirectional. If False, it returns only the last hidden state (batch_size, units*2).

❓ Metrics

advanced

Interpreting training metrics of Bidirectional LSTM

You train a Bidirectional LSTM model for sentiment analysis. After 10 epochs, training accuracy is 95% but validation accuracy is 60%. What does this indicate?

A The model is overfitting the training data and not generalizing well.

B The model is underfitting and needs more training epochs.

C The model has a bug causing incorrect validation evaluation.

D The validation data is easier than the training data.

💡 Hint

High training accuracy but low validation accuracy usually means the model memorizes training data.

High training accuracy with low validation accuracy means the model learned the training data too well but fails to generalize to new data, a classic sign of overfitting.

🔧 Debug

expert

Debugging shape mismatch in Bidirectional LSTM model

You have this Keras model code snippet:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dense

model = Sequential()
model.add(Embedding(input_dim=1000, output_dim=64, input_length=20))
model.add(Bidirectional(LSTM(32)))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy')

# You try to train with labels shape (batch_size, 20, 10) but get a shape error.

What is the cause of the shape error?

NLP

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dense

model = Sequential()
model.add(Embedding(input_dim=1000, output_dim=64, input_length=20))
model.add(Bidirectional(LSTM(32)))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy')

# Training labels shape: (batch_size, 20, 10)

A The model outputs shape (batch_size, 10) but labels have shape (batch_size, 20, 10), causing mismatch.

B The Embedding layer output shape is incompatible with LSTM input shape.

C The Dense layer activation 'softmax' is invalid for multi-class classification.

D The loss function 'categorical_crossentropy' requires labels to be integers, not one-hot encoded.

💡 Hint

Check the output shape of the Bidirectional LSTM and the shape of the labels.

The Bidirectional LSTM without return_sequences outputs (batch_size, 64). The Dense layer outputs (batch_size, 10). But the labels have shape (batch_size, 20, 10), implying a sequence of labels per time step. This mismatch causes the error.

Practice

1. What is the main advantage of using a Bidirectional LSTM compared to a standard LSTM?

easy

A. It only reads the sequence backward for better performance.

B. It uses fewer parameters, making the model faster to train.

C. It processes the input sequence in both forward and backward directions to capture more context.

D. It replaces LSTM cells with simpler RNN cells.

Solution

1. Step 1: Understand LSTM directionality

   A standard LSTM reads the input sequence only in the forward direction, from start to end.

2. Step 2: Analyze Bidirectional LSTM behavior

   A Bidirectional LSTM reads the sequence both forward and backward, capturing information from past and future context.

3. Final Answer:

   It processes the input sequence in both forward and backward directions to capture more context. -> Option C

4. Quick Check:

   Bidirectional means forward + backward = C [OK]

Hint: Bidirectional means reading sequence both ways [OK]

Common Mistakes:

• Thinking it only reads backward
• Assuming it reduces parameters
• Confusing it with simpler RNNs

2. Which of the following is the correct way to add a Bidirectional LSTM layer in Keras?

easy

A. model.add(Bidirectional(LSTM(units=64)))

B. model.add(LSTM(Bidirectional(units=64)))

C. model.add(Bidirectional(units=64, LSTM()))

D. model.add(LSTM(units=64, bidirectional=True))

Solution

1. Step 1: Recall Keras Bidirectional syntax

   In Keras, the Bidirectional wrapper takes an RNN layer like LSTM as its argument.

2. Step 2: Check each option

   model.add(Bidirectional(LSTM(units=64))) correctly wraps LSTM inside Bidirectional. The other options misuse the syntax or parameters.

3. Final Answer:

   model.add(Bidirectional(LSTM(units=64))) -> Option A

4. Quick Check:

   Bidirectional wraps LSTM layer = A [OK]

Hint: Bidirectional wraps LSTM layer, not the other way [OK]

Common Mistakes:

• Putting Bidirectional inside LSTM
• Passing units to Bidirectional instead of LSTM
• Using bidirectional=True parameter in LSTM

3. Consider this code snippet using TensorFlow Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Bidirectional, Dense

model = Sequential()
model.add(Bidirectional(LSTM(10, return_sequences=False), input_shape=(5, 8)))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy')

import numpy as np
x = np.random.random((2, 5, 8))
pred = model.predict(x)
print(pred.shape)

What will be the shape of pred?

medium

A. (2, 10)

B. (2, 1)

C. (5, 1)

D. (2, 20)

Solution

1. Step 1: Understand model output shape

   The Bidirectional LSTM with 10 units outputs 20 features (10 forward + 10 backward) per timestep. Since return_sequences=False, it outputs only the last timestep's features, shape (batch_size, 20).

2. Step 2: Dense layer output shape

   The Dense layer with 1 unit outputs shape (batch_size, 1). Input batch size is 2, so output shape is (2, 1).

3. Final Answer:

   (2, 1) -> Option B

4. Quick Check:

   Batch size 2, Dense 1 unit = (2, 1) [OK]

Hint: Dense(1) outputs shape (batch_size, 1) [OK]

Common Mistakes:

• Confusing return_sequences=True vs False
• Forgetting bidirectional doubles units
• Mixing batch and timestep dimensions

4. You wrote this code but get an error:

model = Sequential()
model.add(Bidirectional(LSTM(32), input_shape=(10, 16)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# Training data
X_train = np.random.random((100, 10, 16))
y_train = np.random.random((100,))

model.fit(X_train, y_train, epochs=5)

The error says: ValueError: Error when checking target: expected dense_1 to have shape (None, 1) but got array with shape (100,)
What is the fix?

medium

A. Change Dense layer units to 100.

B. Remove Bidirectional wrapper.

C. Set return_sequences=True in LSTM layer.

D. Change y_train shape to (100, 1) by reshaping it.

Solution

1. Step 1: Understand error message

   The model expects targets with shape (batch_size, 1) because Dense(1) outputs shape (None, 1). But y_train has shape (100,), missing the last dimension.

2. Step 2: Fix target shape

   Reshape y_train to (100, 1) to match model output shape. This fixes the mismatch error.

3. Final Answer:

   Change y_train shape to (100, 1) by reshaping it. -> Option D

4. Quick Check:

   Target shape matches output shape = B [OK]

Hint: Targets must match model output shape exactly [OK]

Common Mistakes:

• Changing model output units instead of target shape
• Setting return_sequences=True unnecessarily
• Removing Bidirectional without reason

5. You want to build a sentiment analysis model using a Bidirectional LSTM on text sequences of length 100. Which of these model designs best captures full context and outputs a fixed-size vector for classification?

hard

A. Embedding -> Bidirectional(LSTM with return_sequences=True) -> GlobalMaxPooling1D -> Dense

B. Embedding -> Bidirectional(LSTM with return_sequences=False) -> Dense

C. Embedding -> LSTM with return_sequences=False -> Dense

D. Embedding -> Bidirectional(LSTM with return_sequences=True) -> Dense

Solution

1. Step 1: Understand context capture

   Bidirectional LSTM reads sequences forward and backward, capturing full context.

2. Step 2: Fixed-size vector output

   Using return_sequences=True outputs a sequence, so applying GlobalMaxPooling1D converts it to a fixed-size vector summarizing important features.

3. Step 3: Compare options

   Embedding -> Bidirectional(LSTM with return_sequences=True) -> GlobalMaxPooling1D -> Dense uses Bidirectional LSTM with return_sequences=True plus pooling, best for full context and fixed vector. Embedding -> Bidirectional(LSTM with return_sequences=False) -> Dense skips pooling, output is last timestep only. Embedding -> LSTM with return_sequences=False -> Dense is unidirectional. Embedding -> Bidirectional(LSTM with return_sequences=True) -> Dense outputs sequence but no pooling, so Dense gets sequence input, causing shape issues.

4. Final Answer:

   Embedding -> Bidirectional(LSTM with return_sequences=True) -> GlobalMaxPooling1D -> Dense -> Option A

5. Quick Check:

   Pooling after bidirectional LSTM = A [OK]

Hint: Use pooling after return_sequences=True for fixed vector [OK]

Common Mistakes:

• Using return_sequences=False loses sequence info
• Skipping pooling leads to shape mismatch
• Using unidirectional LSTM loses backward context