Module sources (local, registry, git) in Terraform - Deep Dive

Overview - Module sources (local, registry, git)

What is it?

In Terraform, modules are reusable pieces of configuration that help organize and share infrastructure code. Module sources tell Terraform where to find these modules. They can be stored locally on your computer, in a public or private registry, or in a Git repository. This lets you use the same module code in different projects easily.

Why it matters

Without module sources, you would have to copy and paste the same infrastructure code everywhere, making it hard to maintain and update. Module sources solve this by letting you reuse and share code safely and efficiently. This saves time, reduces errors, and helps teams work together better.

Where it fits

Before learning module sources, you should understand basic Terraform configuration and how modules work conceptually. After this, you can learn about module versioning, publishing modules to registries, and advanced module composition techniques.

Mental Model

Core Idea

Module sources are the addresses that tell Terraform where to find reusable infrastructure code, whether on your computer, a shared library, or a code repository.

Think of it like...

It's like a recipe book: the source tells you if the recipe is in your kitchen drawer (local), a popular cookbook store (registry), or a friend's house (Git). You just follow the address to get the recipe you need.

Module Sources
┌───────────────┐
│ Terraform     │
│ Configuration │
└──────┬────────┘
       │
       ▼
┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ Local Folder  │      │ Registry      │      │ Git Repo      │
│ (./modules/)  │      │ (registry URL)│      │ (git URL)     │
└───────────────┘      └───────────────┘      └───────────────┘

Build-Up - 6 Steps

1

Foundation

What is a Terraform Module Source

Concept: Introduce the idea that a module source is a location pointer for reusable code.

Terraform modules are like building blocks for your infrastructure. A module source tells Terraform where to find these blocks. It can be a folder on your computer, a shared library online, or a code repository. This helps Terraform load the right code when you run it.

Result

You understand that module sources are addresses for reusable infrastructure code.

Understanding module sources is key to reusing code and avoiding repetition in infrastructure management.

2

Foundation

Using Local Module Sources

Concept: Learn how to reference modules stored on your local machine.

You can keep modules in folders on your computer. To use one, you write a source path like "./modules/network" in your Terraform configuration. Terraform looks in that folder to find the module code when you apply your configuration.

Result

Terraform loads module code from your local folder and applies it.

Knowing local sources lets you organize your code neatly and test modules before sharing.

3

Intermediate

Using Modules from the Terraform Registry

🤔Before reading on: do you think Terraform Registry modules require manual download or are fetched automatically? Commit to your answer.

Concept: Learn how to use modules published in the official Terraform Registry automatically.

The Terraform Registry is like an app store for modules. You specify a source like "hashicorp/aws" and Terraform downloads the module automatically. You can also specify versions to control updates. This makes sharing and using community or company modules easy.

Result

Terraform fetches and uses modules from the registry without manual steps.

Understanding registry sources unlocks easy sharing and version control of modules.

4

Intermediate

Using Git as a Module Source

🤔Before reading on: do you think Terraform can use any Git repository as a module source or only special ones? Commit to your answer.

Concept: Learn how to use modules stored in Git repositories, including specifying branches or tags.

You can point Terraform to a Git repository URL as a module source, like "git::https://github.com/user/module.git". You can also specify a branch, tag, or commit to use a specific version. Terraform clones the repo and uses the module code inside.

Result

Terraform downloads the module from Git and uses the specified version.

Knowing Git sources allows flexible sharing and versioning of modules outside the registry.

5

Advanced

Combining Multiple Module Sources

🤔Before reading on: can you mix local, registry, and Git sources in one Terraform project? Commit to your answer.

Concept: Learn how to use different module sources together in one project for flexibility.

Terraform lets you use modules from local folders, registries, and Git repos all in the same configuration. This helps you pick the best source for each module, like local for development, registry for stable modules, and Git for custom versions.

Result

Your Terraform project can reuse modules from multiple sources seamlessly.

Understanding mixed sources helps build flexible and maintainable infrastructure codebases.

6

Expert

Handling Module Source Versions and Updates

🤔Before reading on: do you think Terraform automatically updates modules from Git or registry sources every run? Commit to your answer.

Concept: Learn how Terraform manages module versions and when it updates module code from sources.

Terraform caches downloaded modules locally. For registry modules, you specify versions to control updates. For Git, you can pin to commits or tags. Terraform only updates modules when you run 'terraform init' with the '-upgrade' flag or change the source version. This prevents unexpected changes.

Result

You control when and how module code updates happen, avoiding surprises.

Knowing module update behavior prevents accidental infrastructure changes and supports stable deployments.

Under the Hood

Terraform reads the module source string and determines the type: local path, registry address, or Git URL. For local paths, it reads files directly. For registry sources, it queries the Terraform Registry API to download the module archive. For Git sources, it uses Git commands to clone or fetch the repository. Terraform caches downloaded modules in a local folder to speed up future runs. When applying, Terraform loads the module code from the cache or local path to build the infrastructure plan.

Why designed this way?

This design balances flexibility and performance. Local paths allow fast iteration during development. The registry provides a centralized, versioned module sharing system. Git support enables custom or private modules without publishing. Caching avoids repeated downloads, improving speed and reliability. Alternatives like always downloading modules would slow runs and increase network dependency.

Terraform Module Source Handling
┌─────────────────────────────┐
│ Terraform Configuration File │
└──────────────┬──────────────┘
               │
               ▼
      ┌───────────────────┐
      │ Parse module source│
      └─────────┬─────────┘
                │
   ┌────────────┼─────────────┐
   │            │             │
   ▼            ▼             ▼
┌────────┐  ┌────────────┐ ┌───────────┐
│ Local  │  │ Registry   │ │ Git Repo  │
│ Folder │  │ API        │ │ Clone     │
└────────┘  └────────────┘ └───────────┘
   │            │             │
   └─────┬──────┴─────┬───────┘
         ▼            ▼
   ┌─────────────────────────┐
   │ Cache module locally    │
   └────────────┬────────────┘
                │
                ▼
       ┌─────────────────┐
       │ Use module code  │
       └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Terraform automatically update Git modules every time you run 'terraform apply'? Commit yes or no.

Common Belief: Terraform always fetches the latest code from Git repositories every time it runs.

Reality: Terraform caches Git modules locally and only updates them when you run 'terraform init' with the '-upgrade' flag or change the source reference.

Why it matters: Believing this causes confusion when changes in Git don't appear immediately, leading to wasted time troubleshooting.

Quick: Can you use any folder on your computer as a module source without restrictions? Commit yes or no.

Common Belief: Any local folder can be used as a module source without needing a proper module structure.

Reality: The folder must contain valid Terraform module files (like main.tf) for Terraform to use it as a module source.

Why it matters: Using a folder without module files causes errors and wasted effort figuring out why Terraform fails.

Quick: Does specifying a module source without a version always use the latest version? Commit yes or no.

Common Belief: If you don't specify a version for a registry module, Terraform always uses the newest version available.

Reality: Terraform uses the latest version matching constraints or the default version, but this can lead to unexpected upgrades if not pinned.

Why it matters: Not pinning versions can cause sudden infrastructure changes and instability in production.

Quick: Can you use private Git repositories as module sources without extra setup? Commit yes or no.

Common Belief: Terraform can access any Git repository as a module source without authentication setup.

Reality: Private Git repositories require proper authentication (SSH keys or tokens) configured on the machine running Terraform.

Why it matters: Ignoring authentication leads to access errors and deployment failures.

Expert Zone

1

Terraform caches modules in a hidden folder (.terraform/modules) which can cause stale code if not cleaned or upgraded properly.

2

Git module sources support specifying subdirectories inside the repo using the 'subdir' query parameter, allowing multiple modules in one repo.

3

Registry modules support semantic versioning constraints, enabling flexible but controlled upgrades.

When NOT to use

Avoid using local module sources for shared production modules because they lack version control and sharing capabilities. Instead, use the Terraform Registry or Git repositories with versioning. For highly dynamic or experimental modules, local sources are fine during development.

Production Patterns

In production, teams often publish stable modules to a private Terraform Registry or Git with version tags. They pin module versions in configurations to ensure repeatable deployments. Local modules are used mainly for development or quick testing. CI/CD pipelines run 'terraform init -upgrade' selectively to update modules under controlled conditions.

Connections

Package Managers (e.g., npm, pip)

Module sources in Terraform are like package sources in software package managers.

Understanding how package managers fetch and cache code helps grasp Terraform's module source fetching and versioning.

Git Version Control

Terraform uses Git repositories as module sources, leveraging Git's versioning and branching features.

Knowing Git concepts like branches, tags, and commits helps manage module versions and updates effectively.

Library Linking in Programming

Module sources are similar to linking external libraries in programming languages, where code is reused from external locations.

Recognizing this parallel clarifies why module sources improve code reuse and maintainability in infrastructure as code.

Common Pitfalls

#1 Using a local path without a proper module structure causes errors.

Wrong approach:

module "network" {
source = "./myfolder"
}

# But ./myfolder has no main.tf or outputs.tf

Correct approach:

module "network" {
source = "./myfolder"
}

# Ensure ./myfolder contains valid Terraform module files like main.tf

Root cause:

Misunderstanding that any folder can be a module without required Terraform files.

#2 Not specifying a version for registry modules leads to unexpected upgrades.

Wrong approach:

module "vpc" {
source = "terraform-aws-modules/vpc/aws"
# no version specified
}

Correct approach:

module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "3.14.0"
}

Root cause:

Assuming Terraform always uses a fixed version without explicit version pinning.

#3 Using a private Git repo without authentication causes access failure.

Wrong approach:

module "custom" {
source = "git::https://github.com/private/repo.git"
}

Correct approach:

module "custom" {
source = "git::ssh://[e-mail]/private/repo.git"
}
# Ensure SSH keys or tokens are configured

Root cause:

Ignoring the need for authentication when accessing private repositories.

Key Takeaways

Module sources tell Terraform where to find reusable infrastructure code, enabling code sharing and reuse.

You can use local folders, the Terraform Registry, or Git repositories as module sources, each with its own benefits.

Terraform caches downloaded modules and updates them only when explicitly told, preventing unexpected changes.

Pinning module versions is crucial to maintain stable and predictable infrastructure deployments.

Understanding module sources helps you build flexible, maintainable, and collaborative infrastructure codebases.