Custom resources concept in Kubernetes - Deep Dive

Overview - Custom resources concept

What is it?

Custom resources in Kubernetes let you add your own types of objects to the system. These objects behave like built-in ones such as Pods or Services but are defined by you. This allows you to extend Kubernetes with new features without changing its core code. You create a Custom Resource Definition (CRD) to tell Kubernetes about your new object type.

Why it matters

Without custom resources, Kubernetes would only manage the built-in objects it knows about. This limits what you can automate or control. Custom resources let you tailor Kubernetes to your needs, managing new kinds of applications or infrastructure. This makes Kubernetes a flexible platform that grows with your projects and teams.

Where it fits

Before learning custom resources, you should understand basic Kubernetes objects like Pods, Deployments, and Services. After mastering custom resources, you can learn about Operators, which use custom resources to automate complex tasks. This topic fits in the journey of extending Kubernetes beyond its default capabilities.

Mental Model

Core Idea

Custom resources let you teach Kubernetes new object types so it can manage anything you want, just like its built-in objects.

Think of it like...

Imagine Kubernetes as a toolbox with fixed tools. Custom resources are like adding your own custom tools to that box, so you can fix or build things that the original tools can't handle.

┌─────────────────────────────┐
│        Kubernetes API        │
├─────────────┬───────────────┤
│ Built-in    │ Custom        │
│ Resources   │ Resources     │
│ (Pods,      │ (Your own     │
│ Services)   │ objects)      │
└─────────────┴───────────────┘
          ↑                 ↑
          │                 │
   Managed by core     Defined by CRDs
   Kubernetes system   and controllers

Build-Up - 6 Steps

1

Foundation

Understanding Kubernetes Objects

Concept: Learn what Kubernetes objects are and how they represent system state.

Kubernetes uses objects like Pods, Services, and Deployments to represent applications and infrastructure. Each object has a specification describing the desired state and a status showing the current state. The Kubernetes control plane works to make reality match the desired state.

Result

You understand that Kubernetes manages resources by watching and adjusting these objects.

Knowing that Kubernetes works by managing objects helps you see why adding new object types can extend its power.

2

Foundation

What Are Custom Resource Definitions

Concept: Introduce CRDs as the way to add new object types to Kubernetes.

A Custom Resource Definition (CRD) is a Kubernetes object that defines a new kind of resource. It tells Kubernetes about the name, schema, and behavior of your custom objects. Once a CRD is created, you can create instances of your custom resource just like built-in ones.

Result

You can create and manage your own resource types in Kubernetes.

Understanding CRDs is key because they are the gateway to extending Kubernetes without changing its core.

3

Intermediate

Creating and Using Custom Resources

🤔Before reading on: do you think creating a custom resource requires changing Kubernetes source code or just applying YAML? Commit to your answer.

Concept: Learn how to define and use custom resources with YAML manifests.

You write a CRD YAML file describing your new resource's group, version, and schema. After applying it with kubectl, Kubernetes knows your new resource type. Then you create custom resource instances using YAML files that follow the CRD's schema. You can list, get, and delete these resources using kubectl commands.

Result

You can manage your custom resources using standard Kubernetes tools.

Knowing that you only need YAML files and kubectl to add new resource types makes extending Kubernetes accessible and safe.

4

Intermediate

Custom Controllers and Automation

🤔Before reading on: do you think custom resources work automatically or need extra code to act on them? Commit to your answer.

Concept: Understand that custom resources often need controllers to automate actions based on their state.

A custom controller is a program that watches your custom resources and takes actions to make reality match the desired state. For example, it might create Pods or update configurations. Controllers use the Kubernetes API to watch for changes and respond accordingly. Without a controller, custom resources are just data with no behavior.

Result

You realize custom resources plus controllers enable powerful automation.

Understanding the role of controllers prevents the misconception that custom resources alone provide automation.

5

Advanced

Versioning and Schema Validation in CRDs

🤔Before reading on: do you think CRDs can handle multiple versions of a resource at once? Commit to your answer.

Concept: Learn how CRDs support multiple versions and schema validation to evolve safely.

CRDs can define multiple versions (like v1alpha1, v1beta1, v1) to support changes over time. Kubernetes can convert between versions when clients request different ones. CRDs also support OpenAPI schemas to validate resource fields, preventing invalid data. This helps maintain stability and compatibility as your custom resources evolve.

Result

You can safely update custom resource definitions without breaking existing users.

Knowing about versioning and validation helps you design maintainable and robust custom resources.

6

Expert

Advanced CRD Features and Limitations

🤔Before reading on: do you think CRDs can replace all Kubernetes built-in resources? Commit to your answer.

Concept: Explore advanced CRD features and understand their limits compared to built-in resources.

CRDs support features like subresources (status, scale), additional printer columns for kubectl, and conversion webhooks for complex versioning. However, they have performance and scalability limits compared to built-in resources. Some Kubernetes features like finalizers or admission controllers behave differently with CRDs. Understanding these nuances is critical for production use.

Result

You gain a realistic view of what CRDs can and cannot do in large-scale systems.

Knowing CRD limits prevents overusing them and guides when to use native resources or other extensions.

Under the Hood

When you create a CRD, Kubernetes stores its definition in etcd like other resources. The API server dynamically adds endpoints for your custom resource type. When you create instances, they are stored and served by the API server. Controllers watch these instances via the API server's watch mechanism and act accordingly. Validation and version conversion happen inside the API server based on the CRD schema and settings.

Why designed this way?

Kubernetes was designed to be extensible without changing its core code. CRDs provide a declarative way to add new resource types dynamically. This avoids the complexity and risk of modifying the Kubernetes source. The design balances flexibility with stability by using standard API server mechanisms and schemas.

┌───────────────┐        ┌───────────────┐
│   etcd Store  │◀───────│ API Server    │
│ (CRDs &      │        │ (Dynamic APIs)│
│  Resources)  │        └───────┬───────┘
└───────────────┘                │
          ▲                      │
          │                      ▼
   ┌─────────────┐        ┌─────────────┐
   │ Custom      │        │ Controllers │
   │ Resource    │        │ (Watch &    │
   │ Instances   │        │  Act)       │
   └─────────────┘        └─────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Do custom resources automatically perform actions like built-in controllers? Commit to yes or no.

Common Belief: Custom resources automatically manage their own lifecycle and actions without extra code.

Reality: Custom resources are just data objects; they need custom controllers to automate behavior.

Why it matters: Without controllers, custom resources do nothing beyond storing data, leading to confusion and failed automation.

Quick: Can you create custom resources by modifying Kubernetes source code? Commit to yes or no.

Common Belief: To add new resource types, you must change Kubernetes source code and rebuild it.

Reality: You can add new resource types dynamically using CRDs without touching Kubernetes code.

Why it matters: Believing otherwise discourages users from extending Kubernetes and slows innovation.

Quick: Do CRDs have the same performance and feature set as built-in resources? Commit to yes or no.

Common Belief: Custom resources behave exactly like built-in resources in all ways.

Reality: CRDs have some performance limits and lack certain built-in features, so they are not always a full replacement.

Why it matters: Ignoring these limits can cause scalability issues or unexpected behavior in production.

Expert Zone

1

CRDs support subresources like status and scale, but their implementation differs from built-in resources, affecting how controllers interact with them.

2

Conversion webhooks allow complex version upgrades for CRDs, but require careful design to avoid data loss or API errors.

3

CRDs do not support all admission controller features equally, so some security or validation policies may need custom solutions.

When NOT to use

Avoid using CRDs for very high-performance or core Kubernetes features where built-in resources are optimized. Instead, use native resources or contribute upstream. For complex automation, consider Operators that combine CRDs with controllers. If you need temporary or simple data storage, ConfigMaps or annotations may be better.

Production Patterns

In production, CRDs are often paired with custom controllers or Operators to manage complex applications like databases or message queues. Teams use versioned CRDs with conversion webhooks to upgrade APIs smoothly. Monitoring and alerting are set up on custom resources to track health. CRDs are also used to extend Kubernetes for multi-cloud or hybrid environments.

Connections

Operators in Kubernetes

Builds-on

Understanding custom resources is essential to grasp Operators, which automate complex tasks by combining CRDs with controllers.

API Design Principles

Shares patterns

CRDs follow API design principles like versioning and schema validation, showing how good API design enables extensibility.

Object-Oriented Programming

Analogous pattern

Custom resources are like classes in programming, defining new object types with properties and behaviors, helping bridge software design and infrastructure management.

Common Pitfalls

#1 Creating a custom resource without a controller expecting automation.

Wrong approach:

kubectl apply -f mycustomresource.yaml
# Expecting Kubernetes to act automatically

Correct approach:

# Also deploy a custom controller that watches and acts on the resource
kubectl apply -f mycustomresource.yaml
kubectl apply -f mycontroller.yaml

Root cause:

Misunderstanding that custom resources alone do not provide behavior or automation.

#2 Defining a CRD without schema validation leading to invalid resource data.

Wrong approach:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema: {}

Correct approach:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
field1:
type: string

Root cause:

Skipping schema validation allows invalid data, causing runtime errors and confusion.

#3 Using CRDs for very high-frequency, core Kubernetes features.

Wrong approach:

Replacing built-in resources like Pods with CRDs for performance reasons.

Correct approach:

Use built-in resources for core features; use CRDs for extensions where performance is less critical.

Root cause:

Not recognizing the performance and feature limitations of CRDs compared to native resources.

Key Takeaways

Custom resources let you add new object types to Kubernetes without changing its core code.

A Custom Resource Definition (CRD) declares your new resource's schema and API details.

Custom resources need controllers to automate actions; otherwise, they are just data holders.

CRDs support versioning and validation to evolve safely but have limits compared to built-in resources.

Understanding custom resources is essential for extending Kubernetes and building Operators.