LoRA and QLoRA concepts in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define a low-rank adaptation (LoRA) layer with a rank of 4.

Prompt Engineering / GenAI

lora_layer = LoRA(rank=[1])

Drag options to blanks, or click blank then click option'

A128

B16

C64

D4

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing a very large rank defeats the purpose of LoRA's parameter efficiency.

Using zero or negative values for rank.

LoRA uses a small rank value like 4 to reduce the number of trainable parameters while adapting large models.

2fill in blank

medium

Complete the code to quantize a model using QLoRA with 4-bit precision.

Prompt Engineering / GenAI

quantized_model = QLoRA(model, bits=[1])

Drag options to blanks, or click blank then click option'

A8

B2

C4

D16

Attempts:

3 left

💡 Hint

Common Mistakes

Using 8-bit quantization which is not QLoRA's typical setting.

Using 2-bit which is too low and uncommon for QLoRA.

QLoRA uses 4-bit quantization to reduce memory usage while maintaining model quality.

3fill in blank

hard

Fix the error in the code to apply LoRA adapters to a transformer model.

Prompt Engineering / GenAI

model = Transformer()
model.apply_adapters(adapter_type=[1])

Drag options to blanks, or click blank then click option'

A'LoRA'

B'quantization'

C'dropout'

D'batchnorm'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'quantization' which is unrelated to adapter application.

Using 'dropout' or 'batchnorm' which are regularization techniques, not adapters.

To apply LoRA adapters, the adapter_type must be set to 'LoRA'. Other options are unrelated.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that stores LoRA adapter weights only if their norm is greater than 0.1.

Prompt Engineering / GenAI

adapter_weights = {name: weight for name, weight in model.adapters.items() if weight.norm() [1] [2]

Drag options to blanks, or click blank then click option'

A>

B<

C0.1

D1.0

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' which selects weights with small norms.

Using 1.0 which is a higher threshold and may exclude relevant weights.

We want weights with norm greater than 0.1, so the condition is weight.norm() > 0.1.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps adapter names to their scaled weights if the scale is less than 0.5.

Prompt Engineering / GenAI

scaled_adapters = {name: weight * [1] for name, weight in model.adapters.items() if [2] [3] 0.5}

Drag options to blanks, or click blank then click option'

A0.1

Bscale

C<

Dweight

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'weight' in the condition instead of 'scale'.

Using '>' instead of '<' which reverses the condition.

We multiply weights by 0.1 and filter where scale < 0.5, so blanks are 0.1, scale, and <.