GitOps with ArgoCD in Kubernetes - Deep Dive

Overview - GitOps with ArgoCD

What is it?

GitOps with ArgoCD is a way to manage and deploy applications on Kubernetes using Git as the single source of truth. ArgoCD watches your Git repository and automatically applies changes to your Kubernetes cluster to match what is defined in Git. This means your cluster state is always in sync with your code, making deployments predictable and easy to track.

Why it matters

Without GitOps and tools like ArgoCD, managing Kubernetes deployments can be manual, error-prone, and hard to track. Changes might be made directly on the cluster, causing confusion and drift from the desired state. GitOps ensures every change is recorded in Git, making deployments auditable, repeatable, and safer, which is crucial for reliable software delivery.

Where it fits

Before learning GitOps with ArgoCD, you should understand basic Kubernetes concepts like pods, deployments, and manifests, as well as Git basics like repositories and commits. After mastering GitOps, you can explore advanced topics like multi-cluster management, progressive delivery, and integrating GitOps with CI/CD pipelines.

Mental Model

Core Idea

GitOps with ArgoCD treats Git as the single source of truth and automatically syncs Kubernetes clusters to match the Git state.

Think of it like...

Imagine Git as a recipe book and ArgoCD as a chef who constantly checks the recipe book and makes sure the kitchen (Kubernetes cluster) is cooking exactly what the recipe says, no more, no less.

┌─────────────┐      ┌───────────────┐      ┌─────────────────────┐
│   Developer  │─────▶│     Git Repo   │─────▶│     ArgoCD Agent     │
└─────────────┘      └───────────────┘      └─────────┬───────────┘
                                                      │
                                                      ▼
                                             ┌─────────────────┐
                                             │ Kubernetes Cluster│
                                             └─────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Git as Source of Truth

Concept: Git stores the desired state of your Kubernetes applications as code.

Git is a system that tracks changes to files over time. In GitOps, you store Kubernetes manifests (configuration files) in Git. These files describe how your applications should run in the cluster.

Result

You have a clear, versioned record of your application's desired state.

Understanding Git as the single source of truth is key because it makes your infrastructure changes trackable and reversible.

2

Foundation

Basics of Kubernetes Manifests

Concept: Kubernetes manifests define the desired state of resources like pods and services.

Manifests are YAML files that describe what your application needs, such as how many replicas, what image to use, and network settings. Kubernetes reads these files to create and maintain resources.

Result

You can describe your application setup in files that Kubernetes understands.

Knowing how manifests work lets you control your cluster declaratively, which is essential for GitOps.

3

Intermediate

How ArgoCD Watches Git Repositories

🤔Before reading on: do you think ArgoCD pushes changes to Git or pulls changes from Git? Commit to your answer.

Concept: ArgoCD continuously monitors Git repositories and pulls changes to apply them to Kubernetes.

ArgoCD runs inside your cluster and regularly checks the Git repo for updates. When it detects changes, it compares the desired state in Git with the actual cluster state and applies updates to match Git.

Result

Your cluster automatically updates to reflect changes committed in Git.

Knowing ArgoCD pulls changes ensures you understand it reacts to Git commits, not manual cluster changes.

4

Intermediate

Syncing and Drift Detection

🤔Before reading on: do you think ArgoCD automatically fixes all differences between Git and cluster, or does it require manual approval? Commit to your answer.

Concept: ArgoCD detects when the cluster state drifts from Git and can sync them automatically or manually.

If someone changes the cluster directly, ArgoCD notices the difference (drift). Depending on settings, it can auto-correct by applying Git state or alert you to approve the sync.

Result

Your cluster stays consistent with Git, preventing unexpected changes.

Understanding drift detection helps prevent configuration drift, a common cause of bugs and outages.

5

Intermediate

Declarative vs Imperative Deployment

Concept: GitOps uses declarative deployment, describing desired state, unlike imperative commands that tell what to do step-by-step.

In imperative deployment, you run commands like 'kubectl apply' manually. In declarative GitOps, you declare the desired state in Git, and ArgoCD ensures the cluster matches it automatically.

Result

Deployments become more predictable and less error-prone.

Knowing the difference clarifies why GitOps reduces manual errors and improves automation.

6

Advanced

Managing Multiple Environments with ArgoCD

🤔Before reading on: do you think one Git repo can manage multiple environments like dev and prod, or do you need separate repos? Commit to your answer.

Concept: ArgoCD supports managing multiple environments from one or multiple Git repositories using application definitions and project scopes.

You can organize manifests in folders or branches for different environments. ArgoCD applications point to these locations and deploy accordingly, enabling environment-specific configurations.

Result

You can safely manage dev, staging, and production environments with clear separation and control.

Understanding environment management helps scale GitOps practices across teams and stages.

7

Expert

Handling Secrets and Sensitive Data Securely

🤔Before reading on: do you think storing secrets directly in Git is safe with GitOps? Commit to your answer.

Concept: GitOps requires special handling of secrets to avoid exposing sensitive data in Git repositories.

Tools like Sealed Secrets or External Secrets integrate with ArgoCD to encrypt or fetch secrets securely. This prevents secrets from being stored as plain text in Git while keeping deployments automated.

Result

Your deployments remain secure without sacrificing GitOps automation benefits.

Knowing secure secret management prevents critical security risks in GitOps workflows.

Under the Hood

ArgoCD runs as a Kubernetes controller that continuously polls configured Git repositories. It fetches manifests, compares them with the live cluster state using Kubernetes APIs, and applies Kubernetes manifests via the API server to reconcile differences. It maintains an internal cache of application states and reports sync status and health to users.

Why designed this way?

ArgoCD was designed to leverage Git's version control and Kubernetes' declarative API to automate deployments reliably. Pull-based syncing reduces security risks compared to push models, and continuous reconciliation ensures cluster state consistency. Alternatives like manual kubectl apply or CI/CD pipelines lacked this continuous, automated state enforcement.

┌───────────────┐       ┌───────────────┐       ┌─────────────────────┐
│   Git Server  │──────▶│   ArgoCD Agent │──────▶│ Kubernetes API Server│
└───────────────┘       └───────────────┘       └─────────────┬───────┘
                                                           │
                                                           ▼
                                                  ┌─────────────────┐
                                                  │ Kubernetes Cluster│
                                                  └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does ArgoCD push changes to the cluster or pull them from Git? Commit to your answer.

Common Belief: ArgoCD pushes changes to the cluster whenever you tell it to.

Reality: ArgoCD pulls changes from Git and applies them to the cluster automatically or on demand.

Why it matters: Thinking ArgoCD pushes changes can lead to confusion about how updates happen and security concerns about cluster access.

Quick: Is it safe to store Kubernetes secrets as plain text in Git with GitOps? Commit to your answer.

Common Belief: Storing secrets directly in Git is fine because GitOps automates deployments.

Reality: Storing secrets in plain text in Git is insecure; special tools or encryption must be used.

Why it matters: Exposing secrets can lead to security breaches and data leaks.

Quick: Does ArgoCD automatically fix all drift without user intervention? Commit to your answer.

Common Belief: ArgoCD always auto-syncs and fixes any drift immediately without approval.

Reality: ArgoCD can be configured for manual approval before syncing to prevent unintended changes.

Why it matters: Assuming auto-sync can cause unexpected disruptions in production environments.

Quick: Can one Git repository manage multiple Kubernetes environments easily? Commit to your answer.

Common Belief: You must use separate Git repositories for each environment like dev and prod.

Reality: One Git repository can manage multiple environments using folders, branches, or overlays.

Why it matters: Believing otherwise can lead to unnecessary repo sprawl and complex management.

Expert Zone

1

ArgoCD's reconciliation loop frequency affects how quickly drift is detected and fixed, balancing responsiveness and resource use.

2

Using ArgoCD projects allows fine-grained access control and resource limits, which is critical in multi-team environments.

3

ArgoCD supports custom health checks and hooks, enabling complex deployment strategies beyond simple sync.

When NOT to use

GitOps with ArgoCD is not ideal for applications requiring frequent manual interventions or dynamic runtime changes. Alternatives like imperative kubectl commands or CI/CD pipelines with manual steps may be better for such cases.

Production Patterns

In production, teams use ArgoCD with branching strategies for environment promotion, integrate it with CI pipelines for automated testing, and combine it with secret management tools for secure deployments. Multi-cluster setups use ArgoCD ApplicationSets to manage many clusters efficiently.

Connections

Infrastructure as Code (IaC)

GitOps builds on IaC principles by using Git to declare infrastructure state and automate deployments.

Understanding IaC helps grasp how GitOps treats infrastructure changes as code, enabling automation and version control.

Continuous Integration/Continuous Deployment (CI/CD)

GitOps complements CI/CD by automating deployment steps after code passes tests, ensuring cluster state matches Git.

Knowing CI/CD pipelines clarifies how GitOps fits into the software delivery lifecycle for faster, safer releases.

Version Control Systems (VCS)

GitOps relies on VCS like Git to track and manage changes, making VCS knowledge foundational.

Understanding version control concepts like commits, branches, and merges is essential to use GitOps effectively.

Common Pitfalls

#1 Applying changes directly to the Kubernetes cluster instead of through Git.

Wrong approach:

kubectl apply -f updated-manifest.yaml

Correct approach:

git commit -am 'Update manifest' && git push origin main

Root cause:

Misunderstanding that Git is the source of truth leads to manual cluster changes that cause drift.

#2 Storing Kubernetes secrets as plain text in Git repositories.

Wrong approach:

apiVersion: v1
kind: Secret
metadata:
name: my-secret
data:
password: cGFzc3dvcmQ= # base64 encoded password

Correct approach:

Use Sealed Secrets or External Secrets to encrypt or fetch secrets securely instead of storing them directly.

Root cause:

Lack of awareness about security risks and proper secret management in GitOps workflows.

#3 Assuming ArgoCD will always auto-sync changes without configuration.

Wrong approach:

Relying on ArgoCD default settings without setting sync policies or approvals.

Correct approach:

Configure ArgoCD sync policies explicitly to require manual approval in sensitive environments.

Root cause:

Overlooking ArgoCD's configurable sync behavior can cause unexpected production changes.

Key Takeaways

GitOps with ArgoCD uses Git as the single source of truth to manage Kubernetes deployments declaratively and automatically.

ArgoCD continuously monitors Git repositories and syncs the cluster state to match, preventing configuration drift.

Proper secret management is critical in GitOps to avoid exposing sensitive data in Git repositories.

Understanding the difference between declarative and imperative deployment helps appreciate GitOps benefits.

Advanced GitOps practices include managing multiple environments, access controls, and integrating with CI/CD pipelines for production readiness.

Practice

1. What is the main purpose of using ArgoCD in a Kubernetes environment with GitOps?

easy

A. To replace Kubernetes with a new container orchestration system

B. To manually deploy applications without using Git

C. To store container images in a private registry

D. To automatically sync Kubernetes cluster state with Git repository configurations

Solution

1. Step 1: Understand GitOps concept

   GitOps uses Git as the single source of truth for application configurations.

2. Step 2: Role of ArgoCD in GitOps

   ArgoCD automatically syncs Kubernetes cluster state to match the Git repo, ensuring deployments are consistent and automated.

3. Final Answer:

   To automatically sync Kubernetes cluster state with Git repository configurations -> Option D

4. Quick Check:

   GitOps = Auto-sync cluster with Git [OK]

Hint: Remember: ArgoCD syncs cluster state from Git automatically [OK]

Common Mistakes:

• Thinking ArgoCD replaces Kubernetes
• Believing ArgoCD deploys apps manually
• Confusing ArgoCD with container registries

2. Which of the following is the correct ArgoCD CLI command to create an application named myapp syncing from Git repo https://github.com/example/repo.git with path k8s and target cluster https://kubernetes.default.svc?

easy

A. argocd app create myapp --repo https://github.com/example/repo.git --path k8s --dest-server https://kubernetes.default.svc --dest-namespace default

B. argocd create app myapp --repository https://github.com/example/repo.git --directory k8s --cluster https://kubernetes.default.svc

C. kubectl create app myapp --repo https://github.com/example/repo.git --path k8s

D. argocd app new myapp --repo-url https://github.com/example/repo.git --folder k8s --server https://kubernetes.default.svc

Solution

1. Step 1: Identify correct ArgoCD CLI syntax

   The correct command uses argocd app create with flags --repo, --path, --dest-server, and --dest-namespace.

2. Step 2: Compare options

   argocd app create myapp --repo https://github.com/example/repo.git --path k8s --dest-server https://kubernetes.default.svc --dest-namespace default matches the official syntax exactly. Others use incorrect commands or flags.

3. Final Answer:

   argocd app create myapp --repo https://github.com/example/repo.git --path k8s --dest-server https://kubernetes.default.svc --dest-namespace default -> Option A

4. Quick Check:

   Correct CLI syntax = argocd app create myapp --repo https://github.com/example/repo.git --path k8s --dest-server https://kubernetes.default.svc --dest-namespace default [OK]

Hint: Remember: 'argocd app create' plus repo, path, dest-server flags [OK]

Common Mistakes:

• Using kubectl instead of argocd CLI
• Wrong flag names like --repository or --folder
• Incorrect command order or missing --dest-namespace

3. Given the following ArgoCD Application YAML snippet, what will happen when ArgoCD syncs this app?

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-app
spec:
  source:
    repoURL: https://github.com/example/app-configs.git
    path: production
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated: {}

medium

A. ArgoCD will reject the app because syncPolicy is empty

B. ArgoCD will only show the app status but not deploy it automatically

C. ArgoCD will automatically deploy and keep the app in sync with the Git repo

D. ArgoCD will deploy the app once but not update it on Git changes

Solution

1. Step 1: Understand syncPolicy automated

   The syncPolicy: automated: {} means ArgoCD will auto-sync changes from Git to cluster.

2. Step 2: Effect on deployment

   ArgoCD will deploy the app initially and keep it updated automatically when Git changes.

3. Final Answer:

   ArgoCD will automatically deploy and keep the app in sync with the Git repo -> Option C

4. Quick Check:

   syncPolicy automated = auto deploy and sync [OK]

Hint: syncPolicy automated means auto deploy and keep in sync [OK]

Common Mistakes:

• Thinking empty braces mean no automation
• Assuming manual sync is required
• Confusing syncPolicy with app rejection

4. You created an ArgoCD app but it shows OutOfSync status even after you committed changes to Git. Which of the following is the most likely cause?

medium

A. The app's syncPolicy is not set to automated, so manual sync is needed

B. The Git repository URL is incorrect and unreachable

C. The Kubernetes cluster is down and cannot apply changes

D. The app name in ArgoCD does not match the Git repo name

Solution

1. Step 1: Understand OutOfSync status

   OutOfSync means cluster state differs from Git repo state.

2. Step 2: Check syncPolicy effect

   If syncPolicy is not automated, ArgoCD won't auto-apply changes; manual sync is required.

3. Final Answer:

   The app's syncPolicy is not set to automated, so manual sync is needed -> Option A

4. Quick Check:

   OutOfSync + no automated sync = manual sync needed [OK]

Hint: OutOfSync often means no automated sync set [OK]

Common Mistakes:

• Assuming cluster is down without checking
• Blaming Git URL without error evidence
• Thinking app name mismatch causes OutOfSync

5. You want to ensure that ArgoCD only deploys changes to your Kubernetes cluster after manual approval, but still track changes in Git automatically. Which configuration should you use in your ArgoCD Application YAML?

hard

A. Set syncPolicy: automated: {} to enable auto deploy

B. Omit syncPolicy and use manual sync to approve changes

C. Set syncPolicy: automated: null to disable auto deploy

D. Set syncPolicy: automated: { prune: true, selfHeal: true } for auto deploy with cleanup

Solution

1. Step 1: Understand manual approval need

   To require manual approval before deployment, auto sync must be disabled.

2. Step 2: Configure syncPolicy accordingly

   Omitting syncPolicy disables auto sync, so ArgoCD tracks Git but waits for manual sync.

3. Final Answer:

   Omit syncPolicy and use manual sync to approve changes -> Option B

4. Quick Check:

   No syncPolicy = manual approval required [OK]

Hint: No syncPolicy means manual sync only [OK]

Common Mistakes:

• Using automated sync disables manual approval
• Setting automated with prune/selfHeal still auto deploys
• Setting automated: null is invalid syntax