Why multimodal combines text, image, and audio in Prompt Engineering / GenAI - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use multimodal data in AI?

Why do AI systems combine text, images, and audio together?

A Because combining different types of data helps AI understand information more like humans do.

B Because using only one type of data is always faster and more accurate.

C Because text data is always better than images and audio for AI tasks.

D Because images and audio cannot be processed by AI without text.

💡 Hint

Think about how humans use multiple senses to understand the world.

Multimodal AI uses text, images, and audio together to get a fuller understanding, similar to how humans combine sight, hearing, and language.

❓ Predict Output

intermediate

Output of combining text and image features

What is the shape of the combined feature vector after concatenating a text feature vector of shape (1, 300) and an image feature vector of shape (1, 512)?

Prompt Engineering / GenAI

import numpy as np
text_feat = np.random.rand(1, 300)
image_feat = np.random.rand(1, 512)
combined_feat = np.concatenate((text_feat, image_feat), axis=1)
print(combined_feat.shape)

A (1, 300)

B (1, 812)

C (812, 1)

D (300, 512)

💡 Hint

Concatenation along axis=1 joins columns side by side.

Concatenating along axis=1 adds the feature dimensions, so 300 + 512 = 812 features in one row.

❓ Model Choice

advanced

Best model type for multimodal AI

Which model architecture is best suited to process and combine text, image, and audio data in one system?

A A decision tree trained only on text features.

B A simple linear regression model trained on raw pixels only.

C A model that uses separate encoders for each data type and a fusion layer to combine them.

D A convolutional neural network designed only for images.

💡 Hint

Think about how to handle different data types before combining.

Separate encoders extract features from each modality, then a fusion layer combines them for joint understanding.

❓ Metrics

advanced

Evaluating multimodal model performance

Which metric is most appropriate to evaluate a multimodal classification model that predicts categories from text, image, and audio inputs?

A Accuracy, because it measures correct predictions over total samples.

B Mean Squared Error, because it measures distance between predicted and true values.

C BLEU score, because it measures text generation quality.

D Perplexity, because it measures language model uncertainty.

💡 Hint

Consider the task is classification, not generation or regression.

Accuracy is suitable for classification tasks to measure how many predictions are correct.

🔧 Debug

expert

Debugging multimodal data mismatch error

Given this code snippet, what error will occur when concatenating text and audio features with different batch sizes? import numpy as np text_feat = np.random.rand(4, 300) audio_feat = np.random.rand(5, 128) combined_feat = np.concatenate((text_feat, audio_feat), axis=1)

Prompt Engineering / GenAI

import numpy as np
text_feat = np.random.rand(4, 300)
audio_feat = np.random.rand(5, 128)
combined_feat = np.concatenate((text_feat, audio_feat), axis=1)

A TypeError: unsupported operand type(s) for +: 'int' and 'str'

B No error, concatenation succeeds

C IndexError: index out of range

D ValueError: all the input array dimensions except for the concatenation axis must match exactly

💡 Hint

Check if the batch sizes (first dimension) match before concatenation.

Concatenation along axis=1 requires the first dimension (batch size) to be equal; here 4 != 5 causes ValueError.