Durable Functions for workflows in Azure - Deep Dive

Overview - Durable Functions for workflows

What is it?

Durable Functions are a way to write long-running workflows in the cloud that keep track of their progress automatically. They let you break complex tasks into smaller steps that run one after another or at the same time. These workflows can pause and resume without losing their place, even if the system restarts. This helps build reliable applications that handle processes like order processing or data pipelines.

Why it matters

Without Durable Functions, managing long tasks in the cloud is hard because you must manually save progress and handle failures. This can lead to lost work or complicated code. Durable Functions solve this by automatically saving state and retrying steps, making workflows more reliable and easier to build. This means businesses can trust their cloud apps to complete important jobs without errors or manual fixes.

Where it fits

Before learning Durable Functions, you should understand basic Azure Functions and serverless computing concepts. After mastering Durable Functions, you can explore advanced workflow patterns, integrate with other Azure services like Logic Apps, and optimize for scale and cost.

Mental Model

Core Idea

Durable Functions let you write cloud workflows that remember their progress and can pause and resume automatically.

Think of it like...

Imagine writing a to-do list where each task can be checked off one by one, and if you stop halfway, the list remembers exactly where you left off so you can continue later without losing track.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Start        │──────▶│ Step 1       │──────▶│ Step 2       │
└───────────────┘       └───────────────┘       └───────────────┘
       │                      │                       │
       ▼                      ▼                       ▼
  (Pause/Resume)         (Pause/Resume)          (Pause/Resume)
       │                      │                       │
       └───────────────────────────────────────────────┘
                        Workflow State

Build-Up - 7 Steps

1

Foundation

Understanding Azure Functions Basics

Concept: Learn what Azure Functions are and how they run small pieces of code in the cloud on demand.

Azure Functions let you write simple code that runs when triggered by events like HTTP requests or timers. They are serverless, meaning you don't manage servers. This makes it easy to run code without worrying about infrastructure.

Result

You can create small, event-driven programs that run automatically in the cloud.

Understanding Azure Functions is key because Durable Functions build on this to manage more complex, long-running tasks.

2

Foundation

What is Workflow State and Why It Matters

Concept: Introduce the idea of workflow state as the memory of a process that tracks what has happened so far.

When a process has many steps, it needs to remember which steps are done and which are next. This memory is called state. Without saving state, if the process stops, it must start over. Saving state lets workflows pause and continue smoothly.

Result

You understand why workflows need to save progress to handle interruptions.

Knowing about state helps you see why Durable Functions are useful—they handle state automatically.

3

Intermediate

How Durable Functions Manage State Automatically

🤔Before reading on: do you think Durable Functions store state in memory or somewhere else? Commit to your answer.

Concept: Durable Functions save workflow state outside of memory so workflows can survive restarts and failures.

Durable Functions use a storage system (like Azure Storage) to save the current status of each workflow step. This means even if the function app restarts or crashes, the workflow can pick up where it left off. This is called durable state management.

Result

Workflows become reliable and can run for hours, days, or longer without losing progress.

Understanding external state storage explains how Durable Functions achieve reliability beyond normal functions.

4

Intermediate

Orchestrator Functions and Activity Functions

🤔Before reading on: do you think the orchestrator runs all steps itself or calls other functions? Commit to your answer.

Concept: Durable Functions separate workflow logic (orchestrator) from individual tasks (activities).

An orchestrator function defines the workflow steps and their order. It calls activity functions, which do the actual work like sending emails or processing data. The orchestrator waits for activities to finish before moving on. This separation keeps workflows organized and manageable.

Result

You can design workflows that coordinate many tasks cleanly and reliably.

Knowing this separation helps you write clear, maintainable workflows and understand how Durable Functions run tasks efficiently.

5

Intermediate

Common Workflow Patterns with Durable Functions

🤔Before reading on: which do you think is easier with Durable Functions—running tasks in order or running many tasks at once? Commit to your answer.

Concept: Durable Functions support patterns like chaining tasks, running tasks in parallel, and waiting for external events.

You can chain tasks so one runs after another, run many tasks at the same time to save time, or pause a workflow until an external event happens (like user input). Durable Functions handle all these patterns with simple code.

Result

You can build flexible workflows that match real-world business needs.

Recognizing these patterns shows how Durable Functions simplify complex workflow logic.

6

Advanced

Handling Failures and Retries in Workflows

🤔Before reading on: do you think Durable Functions retry failed tasks automatically or require manual retry code? Commit to your answer.

Concept: Durable Functions provide built-in support for retrying failed tasks and handling errors gracefully.

You can configure activity functions to retry if they fail due to temporary issues, like network problems. The orchestrator can catch errors and decide what to do next, such as compensating actions or alerting users. This makes workflows robust and fault-tolerant.

Result

Workflows continue running smoothly even when some tasks fail temporarily.

Knowing built-in retry and error handling reduces the need for complex error code and improves workflow reliability.

7

Expert

Scaling and Performance Considerations Internally

🤔Before reading on: do you think all orchestrator functions run continuously or are replayed on demand? Commit to your answer.

Concept: Durable Functions orchestrators replay their history to rebuild state, enabling scale and consistency.

The orchestrator function does not run continuously. Instead, it replays past events from storage to reconstruct its state each time it runs. This replay mechanism allows Azure to scale workflows efficiently and ensures deterministic behavior. However, it requires writing orchestrator code without side effects.

Result

You understand why orchestrator code must be deterministic and how Durable Functions scale workflows.

Understanding replay explains why some coding patterns are forbidden and how Durable Functions balance reliability with scalability.

Under the Hood

Durable Functions use an orchestration engine that records every action and event in a durable storage like Azure Storage Tables and Queues. When an orchestrator function runs, it replays these events to rebuild the workflow's state. Activity functions run separately and report results back. This design allows workflows to pause, resume, and recover from failures without losing progress.

Why designed this way?

This design was chosen to solve the problem of managing long-running, stateful workflows in a stateless serverless environment. Traditional functions lose state when they stop, so Durable Functions use event sourcing and replay to maintain state reliably. Alternatives like keeping state in memory or databases were less reliable or more complex.

┌─────────────────────────────┐
│ Orchestrator Function        │
│ (Replays events to restore) │
└─────────────┬───────────────┘
              │ Calls
              ▼
┌─────────────────────────────┐
│ Activity Functions           │
│ (Perform tasks, report back) │
└─────────────┬───────────────┘
              │ Events & Results
              ▼
┌─────────────────────────────┐
│ Durable Storage (Queues,     │
│ Tables) records events/state│
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do Durable Functions keep all workflow data in memory during execution? Commit to yes or no.

Common Belief: Durable Functions keep the entire workflow state in memory while running.

Reality: Durable Functions store workflow state externally in durable storage and replay events to rebuild state, not in memory.

Why it matters: Believing state is in memory leads to writing non-deterministic code that breaks replay and causes workflow failures.

Quick: Do you think orchestrator functions can call any asynchronous code freely? Commit to yes or no.

Common Belief: Orchestrator functions can run any async code like normal functions.

Reality: Orchestrator functions must be deterministic and cannot call arbitrary async code or have side effects during replay.

Why it matters: Ignoring this causes subtle bugs and inconsistent workflow behavior that are hard to debug.

Quick: Do Durable Functions automatically scale all parts of the workflow equally? Commit to yes or no.

Common Belief: All parts of Durable Functions workflows scale automatically and equally.

Reality: Activity functions scale independently, but orchestrator functions have limits due to replay and must be designed carefully.

Why it matters: Misunderstanding scaling can cause performance bottlenecks or unexpected costs.

Quick: Do you think Durable Functions are only useful for short tasks? Commit to yes or no.

Common Belief: Durable Functions are only for short-lived tasks and not suitable for long workflows.

Reality: Durable Functions are designed specifically for long-running workflows that can last hours or days.

Why it matters: Underestimating their use limits leads to building fragile custom solutions for long tasks.

Expert Zone

1

The replay mechanism means orchestrator code must avoid non-deterministic operations like random numbers or current time calls without special handling.

2

Durable Functions support external event waiting, allowing workflows to pause indefinitely until an outside signal resumes them, enabling human interaction or external triggers.

3

Checkpointing state after each activity call balances between performance and reliability, but too frequent checkpoints can increase storage costs.

When NOT to use

Avoid Durable Functions for extremely high-throughput, low-latency tasks where milliseconds matter, as replay adds overhead. Use event-driven microservices or Azure Logic Apps for simpler or visual workflows without custom code.

Production Patterns

In production, Durable Functions are used for order processing pipelines, approval workflows, IoT device management, and data ingestion jobs. Patterns include chaining activities, fan-out/fan-in parallel tasks, and human interaction with external events.

Connections

Event Sourcing

Durable Functions use event sourcing internally to record and replay workflow state.

Understanding event sourcing helps grasp how Durable Functions rebuild state reliably from stored events.

State Machines

Durable Functions orchestrators act like state machines controlling workflow steps and transitions.

Knowing state machines clarifies how workflows move between steps and handle events systematically.

Project Management

Workflows in Durable Functions resemble project task lists with dependencies and progress tracking.

Seeing workflows as project plans helps understand the importance of state, retries, and coordination.

Common Pitfalls

#1 Writing non-deterministic code inside orchestrator functions.

Wrong approach:

var currentTime = Date.now(); // used directly in orchestrator
if (currentTime % 2 === 0) { callActivity(); }

Correct approach:

var currentTime = context.df.currentUtcDateTime; // use Durable Functions API
if (currentTime.getSeconds() % 2 === 0) { callActivity(); }

Root cause:

Orchestrator functions replay events and must produce the same results each time; using real-time calls breaks this.

#2 Calling activity functions directly without the orchestrator.

Wrong approach:

await activityFunction(); // called outside orchestrator

Correct approach:

await context.df.callActivity('activityFunction'); // called inside orchestrator

Root cause:

Activity functions must be invoked through the orchestrator to track state and retries.

#3 Ignoring error handling and retries in workflows.

Wrong approach:

await context.df.callActivity('task'); // no retry or error handling

Correct approach:

await context.df.callActivityWithRetry('task', retryOptions);

Root cause:

Not handling failures leads to workflow crashes and incomplete processes.

Key Takeaways

Durable Functions enable writing cloud workflows that remember their progress and can pause and resume reliably.

They separate workflow logic (orchestrator) from tasks (activities) and store state externally to survive failures.

Orchestrator functions replay events to rebuild state, requiring deterministic code without side effects.

Built-in retry and error handling make workflows robust and easier to maintain.

Understanding Durable Functions unlocks building complex, long-running, and reliable cloud processes with less manual effort.