LSTM layer in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output shape of LSTM layer with return_sequences

Consider the following TensorFlow code snippet creating an LSTM layer:

import tensorflow as tf

lstm_layer = tf.keras.layers.LSTM(10, return_sequences=True)
input_tensor = tf.random.uniform((32, 5, 8))  # batch=32, time_steps=5, features=8
output = lstm_layer(input_tensor)
print(output.shape)

What is the printed output shape?

TensorFlow

import tensorflow as tf

lstm_layer = tf.keras.layers.LSTM(10, return_sequences=True)
input_tensor = tf.random.uniform((32, 5, 8))
output = lstm_layer(input_tensor)
print(output.shape)

A (5, 32, 10)

B (32, 10)

C (32, 5, 10)

D (32, 8, 10)

💡 Hint

Remember that return_sequences=True makes the LSTM return output for each time step.

The input shape is (batch_size=32, time_steps=5, features=8). With return_sequences=True, the output shape is (batch_size, time_steps, units), so (32, 5, 10).

❓ Model Choice

intermediate

Choosing LSTM for sequence data

You want to build a model to predict the next word in a sentence based on previous words. Which model layer is best suited for this task?

A Convolutional layer

B LSTM layer

C Dense layer with ReLU activation

D Dropout layer

💡 Hint

Think about which layer can remember information over time steps.

LSTM layers are designed to handle sequence data and remember information over time, making them ideal for predicting the next word in a sentence.

❓ Hyperparameter

advanced

Effect of increasing LSTM units

What is the most likely effect of increasing the number of units in an LSTM layer from 50 to 200?

A Model capacity increases, possibly improving accuracy but increasing risk of overfitting

B Model will train faster due to fewer parameters

C Model will ignore earlier time steps

D Model will require less memory

💡 Hint

More units mean more parameters and complexity.

Increasing units increases model capacity and parameters, which can improve accuracy but also increase overfitting risk and training time.

❓ Metrics

advanced

Interpreting LSTM training loss curve

During training an LSTM model, you observe the training loss steadily decreases but the validation loss starts increasing after some epochs. What does this indicate?

A There is a bug in the loss calculation

B The model is underfitting the training data

C The model has converged perfectly

D The model is overfitting the training data

💡 Hint

Think about what it means when validation loss worsens but training loss improves.

When training loss decreases but validation loss increases, the model is learning the training data too well and not generalizing, which is overfitting.

🔧 Debug

expert

Identifying error in LSTM input shape

Given this code snippet:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.LSTM(32, input_shape=(10,)),
    tf.keras.layers.Dense(1)
])

input_data = tf.random.uniform((64, 10, 5))
output = model(input_data)
print(output.shape)

What error will this code raise?

A ValueError: Input 0 of layer 'lstm' is incompatible with the layer: expected shape=(None, 10), found shape=(64, 10, 5)

B TypeError: Cannot convert input data to tensor

C RuntimeError: LSTM layer requires 3D input but got 2D

D No error, output shape is (64, 1)

💡 Hint

Check the input_shape parameter and the actual input data shape.

The LSTM layer expects input shape (time_steps, features). Here, input_shape=(10,) means time_steps=10 but features dimension is missing. The input data has shape (64, 10, 5), so features=5. This mismatch causes a ValueError.

Practice

1. What is the main purpose of an LSTM layer in a neural network?

easy

A. To remember important information from past sequence data

B. To perform image classification

C. To reduce the size of input data

D. To generate random numbers

Solution

1. Step 1: Understand LSTM function

   LSTM layers are designed to handle sequence data by remembering important past information, unlike regular layers.

2. Step 2: Compare options

   Options A, B, and C describe unrelated tasks. Only To remember important information from past sequence data correctly describes LSTM's purpose.

3. Final Answer:

   To remember important information from past sequence data -> Option A

4. Quick Check:

   LSTM purpose = Remember past info [OK]

Hint: LSTM = memory for sequences, not general tasks [OK]

Common Mistakes:

• Confusing LSTM with convolutional layers
• Thinking LSTM reduces data size
• Assuming LSTM generates random outputs

2. Which of the following is the correct way to add an LSTM layer with 50 units using the `units` keyword in TensorFlow Keras?

easy

A. tf.keras.layers.LSTM(50)

B. tf.keras.layers.LSTM(units=50)

C. tf.keras.layers.LSTM(50, return_sequences=True)

D. tf.keras.layers.LSTM(50, dropout=0.2)

Solution

1. Step 1: Recall LSTM syntax

   The LSTM layer requires the number of units as a named argument 'units=50' for clarity and correctness.

2. Step 2: Evaluate options

   tf.keras.layers.LSTM(50) is allowed but less clear. Options C and D add extra parameters but the question asks for the correct basic syntax using the 'units' keyword.

3. Final Answer:

   tf.keras.layers.LSTM(units=50) -> Option B

4. Quick Check:

   LSTM syntax = units=number [OK]

Hint: Use named argument 'units' for clarity in LSTM layer [OK]

Common Mistakes:

• Omitting 'units=' keyword
• Confusing dropout with units parameter
• Adding unnecessary parameters when not asked

3. What will be the output shape of the following LSTM layer if the input shape is (batch_size, 10, 8)?

tf.keras.layers.LSTM(16, return_sequences=False)

medium

A. (batch_size, 8)

B. (batch_size, 10, 16)

C. (batch_size, 16)

D. (batch_size, 10, 8)

Solution

1. Step 1: Understand return_sequences effect

   When return_sequences=False, the LSTM returns only the last output for the sequence, so output shape is (batch_size, units).

2. Step 2: Apply to given parameters

   Units=16, so output shape is (batch_size, 16). Options B and D include sequence length, which is incorrect here.

3. Final Answer:

   (batch_size, 16) -> Option C

4. Quick Check:

   return_sequences=False output = (batch_size, units) [OK]

Hint: return_sequences=False means output shape excludes time steps [OK]

Common Mistakes:

• Assuming output keeps sequence length when return_sequences=False
• Confusing input shape with output shape
• Mixing units and input feature dimensions

4. Identify the error in this LSTM layer code snippet:

model = tf.keras.Sequential()
model.add(tf.keras.layers.LSTM(32, input_shape=(20, 5), return_sequences=False))
model.add(tf.keras.layers.LSTM(16))
model.add(tf.keras.layers.Dense(1))

medium

A. The first LSTM layer must have return_sequences=True

B. The input_shape should be (5, 20) instead of (20, 5)

C. The second LSTM layer should have return_sequences=True

D. Dense layer cannot follow LSTM layers

Solution

1. Step 1: Check LSTM stacking rules

   When stacking LSTM layers, all except the last must have return_sequences=True to pass full sequence outputs to the next layer.

2. Step 2: Analyze given code

   The first LSTM has return_sequences=False, outputting only (batch_size, 32). The second LSTM expects 3D sequence input but receives 2D, causing a shape incompatibility error.

3. Final Answer:

   The first LSTM layer must have return_sequences=True -> Option A

4. Quick Check:

   Stacked LSTM earlier layers need return_sequences=True [OK]

Hint: Only last LSTM can have return_sequences=False [OK]

Common Mistakes:

• Forgetting return_sequences=True in middle LSTM layers
• Mixing input_shape dimensions
• Thinking Dense can't follow LSTM

5. You want to build an LSTM model that outputs a sequence of the same length as input (10 time steps) with 20 features each. Which configuration achieves this?

hard

A. tf.keras.layers.LSTM(20, return_sequences=False, input_shape=(10, 20))

B. tf.keras.layers.LSTM(20, return_sequences=True, input_shape=(20, 10))

C. tf.keras.layers.LSTM(10, return_sequences=True, input_shape=(20, 20))

D. tf.keras.layers.LSTM(20, return_sequences=True, input_shape=(10, 20))

Solution

1. Step 1: Understand input and output shapes

   Input shape is (10, 20) meaning 10 time steps, 20 features. To output sequence of same length, return_sequences must be True.

2. Step 2: Check options

   tf.keras.layers.LSTM(20, return_sequences=True, input_shape=(10, 20)) matches input shape and sets return_sequences=True with units=20, so output shape is (batch_size, 10, 20). Options A, C and D have wrong return_sequences or input shapes.

3. Final Answer:

   tf.keras.layers.LSTM(20, return_sequences=True, input_shape=(10, 20)) -> Option D

4. Quick Check:

   return_sequences=True + correct input shape = desired output [OK]

Hint: Use return_sequences=True to keep sequence length [OK]

Common Mistakes:

• Setting return_sequences=False loses sequence length
• Mixing time steps and feature dimensions in input_shape
• Confusing units with input feature size