Cloning with submodules in Git - Deep Dive

Overview - Cloning with submodules

What is it?

Cloning with submodules means copying a Git repository that contains other Git repositories inside it. These smaller repositories are called submodules. When you clone a project with submodules, you get the main project and links to the submodules, which you then fetch separately. This helps keep related projects organized but separate.

Why it matters

Without submodules, managing projects that depend on other projects would be messy and error-prone. You would have to copy all code manually or mix unrelated histories. Submodules let you keep dependencies clean and updated independently, saving time and avoiding mistakes. Without this, collaboration and code reuse would be much harder.

Where it fits

Before learning this, you should understand basic Git cloning and repositories. After this, you can learn about updating submodules, branching with submodules, and advanced Git workflows involving multiple repositories.

Mental Model

Core Idea

Cloning with submodules means copying a main project and separately fetching its linked smaller projects to keep them organized but connected.

Think of it like...

Imagine buying a furniture set where the main table comes with separate chairs packed in their own boxes. Cloning the set means you get the table box and instructions to get the chair boxes separately, so each piece stays neat and can be updated or replaced on its own.

Main Repo (cloned)
│
├─ Submodule A (linked, needs separate fetch)
│
└─ Submodule B (linked, needs separate fetch)

Build-Up - 7 Steps

1

Foundation

Understanding Git repositories and cloning

Concept: Learn what a Git repository is and how cloning copies it.

A Git repository is a folder with all your project files and history. Cloning means copying this repository from a remote server to your computer. You use the command: git clone . This copies the entire project so you can work on it locally.

Result

You get a full copy of the project on your computer with all files and history.

Understanding cloning is essential because submodules build on this concept by adding linked repositories inside the main one.

2

Foundation

What are Git submodules?

Concept: Submodules are Git repositories inside another Git repository.

Sometimes projects depend on other projects. Instead of copying their code directly, Git lets you link to them as submodules. Each submodule has its own repository and history. The main project keeps a reference to a specific commit of the submodule.

Result

You know that submodules are separate projects linked inside a main project.

Knowing submodules are separate repositories helps you understand why cloning them requires extra steps.

3

Intermediate

Cloning a repo with submodules basics

🤔Before reading on: do you think 'git clone' alone fetches submodules automatically? Commit to yes or no.

Concept: Cloning a repo with submodules requires extra commands to fetch the submodules' content.

When you run git clone on a repo with submodules, Git copies the main project but leaves submodules empty. To get submodules, you run git submodule update --init --recursive. This downloads the submodules at the commits the main project expects.

Result

You have the main project and all its submodules fully downloaded and ready to use.

Understanding that submodules are not cloned automatically prevents confusion when submodule folders appear empty after cloning.

4

Intermediate

Using git clone with --recurse-submodules

🤔Before reading on: do you think 'git clone --recurse-submodules' clones submodules in one step? Commit to yes or no.

Concept: Git offers a shortcut to clone the main repo and all submodules in one command.

Instead of cloning then updating submodules separately, you can run git clone --recurse-submodules . This clones the main project and automatically fetches all submodules recursively. It saves time and reduces mistakes.

Result

You get a complete copy of the main project and all submodules in one step.

Knowing this shortcut improves efficiency and reduces errors in cloning projects with submodules.

5

Intermediate

Understanding recursive submodules

Concept: Submodules can themselves have submodules, requiring recursive fetching.

Sometimes a submodule contains its own submodules. To get all nested submodules, you use the --recursive flag with git submodule update or git clone. This ensures every linked repository, no matter how deep, is fetched.

Result

All levels of submodules are downloaded and ready to use.

Recognizing recursive submodules helps avoid incomplete clones in complex projects.

6

Advanced

Handling submodule updates after cloning

🤔Before reading on: do you think submodules update automatically when you pull the main repo? Commit to yes or no.

Concept: Submodules do not update automatically; you must update them manually after pulling changes.

When you pull new commits in the main repo, submodules might point to new commits. You need to run git submodule update --recursive to fetch and checkout those new commits in submodules. Otherwise, submodules stay at old versions.

Result

Submodules match the main project's expected versions after updates.

Knowing manual submodule updates prevent bugs caused by mismatched code versions.

7

Expert

Common pitfalls and advanced submodule workflows

🤔Before reading on: do you think submodules always track the latest commit on their branches automatically? Commit to yes or no.

Concept: Submodules track specific commits, not branches, requiring careful management in workflows.

Submodules record a fixed commit, not a branch tip. If you want to update a submodule to a newer commit, you must enter the submodule folder, checkout or pull the desired commit, then commit the change in the main repo. This explicit control avoids unexpected changes but requires discipline. Also, conflicts can arise if multiple people update submodules differently.

Result

You understand how to manage submodules carefully in team environments and avoid common errors.

Understanding fixed-commit tracking in submodules is key to avoiding confusion and merge conflicts in complex projects.

Under the Hood

Git stores submodules as special entries in the main repository's index and .gitmodules file. These entries record the URL and the exact commit of the submodule. When cloning, Git fetches the main repo normally but leaves submodule folders empty until you explicitly fetch and checkout the recorded commits. This separation keeps histories clean and allows independent versioning.

Why designed this way?

Submodules were designed to keep projects modular and maintain separate histories. This avoids mixing unrelated code and allows teams to update dependencies independently. Alternatives like copying code directly or merging histories were rejected because they cause duplication, confusion, and harder collaboration.

Main Repo
├─ .gitmodules (stores submodule URLs)
├─ Submodule Folder (empty after clone)
│  └─ .git (separate repo)
└─ Index (records submodule commit)

Clone → fetch main repo → submodule folders empty
Update → fetch submodule commits → checkout submodule commits

Myth Busters - 4 Common Misconceptions

Quick: does 'git clone' automatically fetch submodules? Commit yes or no.

Common Belief: Running 'git clone' downloads the main project and all submodules automatically.

Reality: 'git clone' alone only downloads the main project. Submodules remain empty until you run extra commands.

Why it matters: Assuming submodules are cloned automatically leads to confusion and broken builds when submodule code is missing.

Quick: do submodules track the latest branch commits automatically? Commit yes or no.

Common Belief: Submodules always point to the latest commit on their branches automatically.

Reality: Submodules track a fixed commit, not a branch tip. You must update them manually to change versions.

Why it matters: Believing submodules update automatically causes unexpected code versions and bugs in development.

Quick: can you update submodules by just pulling the main repo? Commit yes or no.

Common Belief: Pulling the main repo also updates all submodules to their latest commits.

Reality: You must run 'git submodule update' after pulling to sync submodules to the expected commits.

Why it matters: Not updating submodules after pull leads to mismatched code and runtime errors.

Quick: do submodules merge their history into the main repo? Commit yes or no.

Common Belief: Submodules merge their commit history into the main repository's history.

Reality: Submodules keep their own separate history and repository. The main repo only records a pointer to a commit.

Why it matters: Misunderstanding this causes confusion about project size and history complexity.

Expert Zone

1

Submodules do not track branches by default, but you can configure them to track branches with 'branch' option in .gitmodules, though this is rarely used and can cause confusion.

2

When cloning with --recurse-submodules, Git fetches submodules in parallel, improving speed but sometimes causing network or permission issues that require manual intervention.

3

Submodule commits are recorded in the main repo's tree, so changing a submodule commit requires committing that change in the main repo, which can cause merge conflicts if multiple developers update submodules differently.

When NOT to use

Avoid submodules when dependencies change frequently or require complex branching. Instead, use package managers, monorepos, or Git subtree merges which integrate dependencies more tightly and simplify workflows.

Production Patterns

In production, teams often pin submodules to stable commits and update them only after testing. CI pipelines include explicit submodule update steps. Some projects use submodules for large binary assets or third-party libraries to keep main repo size small.

Connections

Package Management

Alternative approach to managing dependencies

Understanding submodules helps compare Git-based dependency linking with package managers like npm or pip, which automate versioning and updates differently.

Monorepo Architecture

Different strategy for managing multiple projects

Knowing submodules clarifies why some teams prefer monorepos that keep all code in one repo to simplify dependency management and CI/CD.

Modular Design in Software Engineering

Shared principle of separation and independent updates

Recognizing submodules as a form of modular design connects Git workflows to broader software engineering practices emphasizing loose coupling and clear interfaces.

Common Pitfalls

#1 Cloning a repo with submodules but forgetting to fetch submodules.

Wrong approach:

git clone https://example.com/project.git

Correct approach:

git clone --recurse-submodules https://example.com/project.git

Root cause:

Assuming 'git clone' fetches everything including submodules, missing the extra step needed.

#2 Pulling updates in main repo but not updating submodules.

Wrong approach:

git pull

Correct approach:

git pull
git submodule update --recursive

Root cause:

Believing 'git pull' updates submodules automatically, which it does not.

#3 Trying to update submodules by pulling inside submodule folders without committing in main repo.

Wrong approach:

cd submodule
git pull

Correct approach:

cd submodule
git pull
cd ..
git add submodule
git commit -m 'Update submodule to new commit'

Root cause:

Not realizing main repo tracks submodule commit pointers, so changes must be committed there.

Key Takeaways

Cloning a Git repository with submodules requires extra steps to fetch the linked repositories separately.

Submodules track specific commits, not branches, so they do not update automatically with the main project.

Using 'git clone --recurse-submodules' simplifies cloning by fetching all submodules in one command.

After pulling changes, you must run 'git submodule update --recursive' to sync submodules to the expected commits.

Managing submodules carefully avoids common pitfalls like missing code, version mismatches, and merge conflicts.