Inpainting and outpainting in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Inpainting and outpainting

Problem:You have a model that fills missing parts inside images (inpainting) and extends images beyond their borders (outpainting). Currently, the model works well on training images but performs poorly on new images, showing blurry and incorrect fills.

Current Metrics:Training loss: 0.02, Validation loss: 0.15, Training PSNR: 35 dB, Validation PSNR: 22 dB

Issue:The model overfits the training data, causing poor generalization on validation images. The validation loss is much higher and image quality is low.

Your Task

Reduce overfitting to improve validation PSNR to at least 28 dB while keeping training PSNR below 33 dB.

Keep the same model architecture (U-Net based).

Do not increase training data size.

Adjust only training hyperparameters and regularization.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Define U-Net model with dropout
inputs = layers.Input(shape=(128, 128, 3))

# Encoder
c1 = layers.Conv2D(64, (3,3), activation='relu', padding='same')(inputs)
c1 = layers.Dropout(0.3)(c1)
c1 = layers.Conv2D(64, (3,3), activation='relu', padding='same')(c1)
p1 = layers.MaxPooling2D((2,2))(c1)

c2 = layers.Conv2D(128, (3,3), activation='relu', padding='same')(p1)
c2 = layers.Dropout(0.3)(c2)
c2 = layers.Conv2D(128, (3,3), activation='relu', padding='same')(c2)
p2 = layers.MaxPooling2D((2,2))(c2)

# Bottleneck
b = layers.Conv2D(256, (3,3), activation='relu', padding='same')(p2)
b = layers.Dropout(0.4)(b)
b = layers.Conv2D(256, (3,3), activation='relu', padding='same')(b)

# Decoder
u1 = layers.UpSampling2D((2,2))(b)
u1 = layers.concatenate([u1, c2])
c3 = layers.Conv2D(128, (3,3), activation='relu', padding='same')(u1)
c3 = layers.Dropout(0.3)(c3)
c3 = layers.Conv2D(128, (3,3), activation='relu', padding='same')(c3)

u2 = layers.UpSampling2D((2,2))(c3)
u2 = layers.concatenate([u2, c1])
c4 = layers.Conv2D(64, (3,3), activation='relu', padding='same')(u2)
c4 = layers.Dropout(0.3)(c4)
c4 = layers.Conv2D(64, (3,3), activation='relu', padding='same')(c4)

outputs = layers.Conv2D(3, (1,1), activation='sigmoid')(c4)

model = models.Model(inputs, outputs)

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005), loss='mse')

# Data augmentation
train_datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=20)

# Assume X_train, y_train, X_val, y_val are prepared numpy arrays

# Early stopping
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Train model
model.fit(train_datagen.flow(X_train, y_train, batch_size=32), epochs=50, validation_data=(X_val, y_val), callbacks=[early_stop])

Added dropout layers after convolutional layers to reduce overfitting.

Applied data augmentation with flips and rotations to increase data variety.

Reduced learning rate from 0.001 to 0.0005 for smoother training.

Added early stopping to prevent over-training.

Results Interpretation

Before: Training PSNR 35 dB, Validation PSNR 22 dB (large gap, overfitting)

After: Training PSNR 31 dB, Validation PSNR 29 dB (smaller gap, better generalization)

Adding dropout and data augmentation reduces overfitting, improving validation image quality while slightly lowering training performance. Early stopping and lower learning rate help training stability.

Bonus Experiment

Try using a smaller U-Net model with fewer filters to reduce model complexity and see if validation performance improves further.

💡 Hint

Reducing model size can prevent overfitting by limiting capacity, but watch for underfitting if too small.

Practice

1. What is the main difference between inpainting and outpainting in image editing?

easy

A. Both inpainting and outpainting only remove unwanted parts from images.

B. Inpainting adds new areas around an image, outpainting removes parts inside it.

C. Inpainting fills missing parts inside an image, outpainting adds new areas around it.

D. Inpainting and outpainting are the same process with different names.

Solution

1. Step 1: Understand inpainting

   Inpainting is used to fill missing or unwanted parts inside an image, like fixing scratches or holes.

2. Step 2: Understand outpainting

   Outpainting extends the image by adding new content around the edges, making the image bigger.

3. Final Answer:

   Inpainting fills missing parts inside an image, outpainting adds new areas around it. -> Option C

4. Quick Check:

   Inpainting = fill inside, Outpainting = add outside [OK]

Hint: Inpainting fixes inside; outpainting grows outside [OK]

Common Mistakes:

• Confusing inpainting with outpainting
• Thinking both remove parts only
• Believing they are identical

2. Which of the following is the correct way to describe the input for an inpainting model?

easy

A. Only the new areas to add around the image.

B. An image with missing or masked areas to fill.

C. A complete image with no missing parts.

D. A text description of the image content.

Solution

1. Step 1: Identify input for inpainting

   Inpainting models require an image with missing or masked parts that need filling.

2. Step 2: Check other options

   Complete images or text descriptions are not direct inputs for inpainting; new areas relate to outpainting.

3. Final Answer:

   An image with missing or masked areas to fill. -> Option B

4. Quick Check:

   Inpainting input = image with holes [OK]

Hint: Inpainting needs holes in image input [OK]

Common Mistakes:

• Choosing complete images without masks
• Confusing input with outpainting requirements
• Selecting text descriptions as input

3. Given this Python pseudocode for outpainting, what will be the shape of the output image if the input image is 256x256 and the model adds 64 pixels on each side?

input_image = load_image('photo.png')  # shape (256, 256)
output_image = outpaint_model(input_image, border=64)
print(output_image.shape)

medium

A. (256, 256)

B. (192, 192)

C. (320, 320)

D. (384, 384)

Solution

1. Step 1: Calculate added pixels

   The model adds 64 pixels on each side, so total added width = 64 * 2 = 128 pixels.

2. Step 2: Calculate new image size

   Original size 256 + 128 = 384 pixels. This is 256 + 64 + 64 = 384, since 64 pixels on each side means adding 64 left and 64 right.

3. Step 3: Re-check options

   (384, 384) matches calculation. (320, 320) is 256 + 64, adding only one side.

4. Final Answer:

   (384, 384) -> Option D

5. Quick Check:

   256 + 64*2 = 384 [OK]

Hint: Add border pixels twice (both sides) to original size [OK]

Common Mistakes:

• Adding border only once
• Confusing inpainting with outpainting size change
• Ignoring both width and height increase

4. You run an inpainting model but the output image still has visible holes where the mask was applied. What is the most likely cause?

medium

A. The mask was not correctly applied to the input image.

B. The model was trained only for outpainting, not inpainting.

C. The input image was too large for the model to process.

D. The output image format does not support transparency.

Solution

1. Step 1: Check mask application

   If holes remain, the mask likely was not properly set, so the model didn't know where to fill.

2. Step 2: Evaluate other options

   Model type mismatch or image size issues usually cause errors or poor quality, not visible holes. Output format affects display but not hole filling.

3. Final Answer:

   The mask was not correctly applied to the input image. -> Option A

4. Quick Check:

   Mask error = holes remain [OK]

Hint: Check mask covers missing parts fully [OK]

Common Mistakes:

• Ignoring mask correctness
• Blaming model type without checking input
• Assuming output format causes holes

5. You want to create a larger scenic image by extending the edges of a 512x512 photo using outpainting. You also want to remove a small unwanted object inside the photo using inpainting. Which approach correctly combines both tasks?

hard

A. First apply inpainting on the original image to remove the object, then apply outpainting to extend the image edges.

B. Apply outpainting first to extend the image, then apply inpainting on the extended edges to remove the object.

C. Only use outpainting because it can both remove objects and extend images.

D. Only use inpainting because it can extend images and remove objects.

Solution

1. Step 1: Remove unwanted object first

   Inpainting fixes inside the image, so remove the object before changing image size.

2. Step 2: Extend image after cleanup

   Outpainting adds new areas around the cleaned image, so apply it after inpainting.

3. Step 3: Evaluate other options

   Outpainting cannot remove inside objects; inpainting cannot add new edges. Order matters for best results.

4. Final Answer:

   First apply inpainting on the original image to remove the object, then apply outpainting to extend the image edges. -> Option A

5. Quick Check:

   Fix inside first, then grow outside [OK]

Hint: Clean inside first (inpainting), then extend outside (outpainting) [OK]

Common Mistakes:

• Applying outpainting before inpainting
• Thinking one method does both tasks
• Ignoring task order importance