Common concurrency patterns in Go - Deep Dive

Overview - Common concurrency patterns

What is it?

Common concurrency patterns are ways to organize multiple tasks running at the same time in a program. They help manage how these tasks communicate, share data, and avoid problems like conflicts or waiting forever. In Go, concurrency means running functions or processes simultaneously to make programs faster and more efficient. These patterns provide tested solutions to common challenges when working with many tasks at once.

Why it matters

Without concurrency patterns, programs that do many things at once can become confusing, buggy, or slow. Imagine trying to cook many dishes at the same time without a plan—things get messy and some dishes might burn or get cold. Concurrency patterns give a clear plan to handle multiple tasks safely and efficiently. This makes programs faster, more reliable, and easier to maintain, which is important for real-world apps like web servers or data processors.

Where it fits

Before learning concurrency patterns, you should understand basic Go syntax, functions, and goroutines (Go's way to run tasks concurrently). After mastering these patterns, you can explore advanced topics like synchronization primitives, context cancellation, and designing scalable distributed systems.

Mental Model

Core Idea

Concurrency patterns are like recipes that organize how multiple tasks run and talk to each other safely and efficiently.

Think of it like...

Think of a busy kitchen where several chefs work together. Each chef has a role, and they pass ingredients or dishes in an organized way to avoid bumping into each other or spoiling the food. Concurrency patterns are the kitchen's workflow plans that keep everything running smoothly.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Producer    │──────▶│   Channel     │──────▶│   Consumer    │
└───────────────┘       └───────────────┘       └───────────────┘

This shows a simple pipeline pattern where data flows from a producer through a channel to a consumer.

Build-Up - 7 Steps

1

Foundation

Understanding goroutines basics

Concept: Learn how to start multiple tasks that run at the same time using goroutines.

In Go, you can run a function concurrently by adding the keyword 'go' before it. For example: func sayHello() { fmt.Println("Hello") } func main() { go sayHello() // runs concurrently time.Sleep(time.Second) // wait so goroutine can finish } This starts 'sayHello' in a new goroutine, which runs alongside the main function.

Result

The program prints 'Hello' while the main function waits briefly to let the goroutine finish.

Understanding goroutines is the foundation of concurrency in Go because they let you run many tasks at once without complex thread management.

2

Foundation

Using channels for communication

Concept: Channels let goroutines send and receive data safely between each other.

Channels are like pipes that connect goroutines. One goroutine can send data into a channel, and another can receive it. Example: ch := make(chan string) func sender() { ch <- "data" } func receiver() { msg := <-ch fmt.Println(msg) } func main() { go sender() go receiver() time.Sleep(time.Second) } This passes the string "data" from sender to receiver.

Result

The program prints 'data' showing communication between goroutines.

Channels provide a safe way to share data without conflicts, avoiding common bugs in concurrent programs.

3

Intermediate

Pipeline pattern for task chaining

🤔Before reading on: do you think pipelines can improve performance by breaking tasks into stages? Commit to your answer.

Concept: Pipelines connect multiple stages where each stage processes data and passes it on, improving concurrency and clarity.

A pipeline has several goroutines connected by channels. Each goroutine does part of the work and sends results to the next. Example: func gen(nums ...int) <-chan int { out := make(chan int) go func() { for _, n := range nums { out <- n } close(out) }() return out } func sq(in <-chan int) <-chan int { out := make(chan int) go func() { for n := range in { out <- n * n } close(out) }() return out } func main() { nums := gen(2, 3, 4) squares := sq(nums) for sq := range squares { fmt.Println(sq) } } This pipeline generates numbers and squares them in separate stages.

Result

The program prints 4, 9, 16 each on a new line.

Pipelines help organize complex tasks into simple, concurrent steps that improve performance and code readability.

4

Intermediate

Worker pool pattern for load balancing

🤔Before reading on: do you think worker pools limit the number of concurrent tasks? Commit to your answer.

Concept: Worker pools run a fixed number of goroutines to handle many tasks, balancing load and controlling resource use.

Instead of starting a goroutine per task, a pool has a set number of workers pulling tasks from a shared channel. Example: func worker(id int, jobs <-chan int, results chan<- int) { for j := range jobs { fmt.Printf("worker %d processing job %d\n", id, j) results <- j * 2 } } func main() { jobs := make(chan int, 5) results := make(chan int, 5) for w := 1; w <= 3; w++ { go worker(w, jobs, results) } for j := 1; j <= 5; j++ { jobs <- j } close(jobs) for a := 1; a <= 5; a++ { fmt.Println(<-results) } } This runs 3 workers to process 5 jobs concurrently.

Result

The program prints worker messages and results like 2, 4, 6, 8, 10 in any order.

Worker pools prevent resource overload by limiting concurrency and distributing work evenly.

5

Intermediate

Select statement for multiple channel waits

🤔Before reading on: do you think select waits for all channels or just one? Commit to your answer.

Concept: The select statement lets a goroutine wait on multiple channels and react to whichever is ready first.

Select works like a traffic controller for channels. Example: ch1 := make(chan string) ch2 := make(chan string) go func() { time.Sleep(500 * time.Millisecond) ch1 <- "from ch1" }() go func() { time.Sleep(300 * time.Millisecond) ch2 <- "from ch2" }() select { case msg1 := <-ch1: fmt.Println(msg1) case msg2 := <-ch2: fmt.Println(msg2) } This prints the message from the channel that sends first.

Result

The program prints 'from ch2' because ch2 sends earlier.

Select enables responsive programs that handle multiple events without blocking unnecessarily.

6

Advanced

Context pattern for cancellation control

🤔Before reading on: do you think goroutines stop automatically when main ends? Commit to your answer.

Concept: Context lets you signal multiple goroutines to stop work early, avoiding wasted effort and leaks.

Context carries cancellation signals and deadlines. Example: func worker(ctx context.Context) { for { select { case <-ctx.Done(): fmt.Println("worker stopped") return default: fmt.Println("working") time.Sleep(100 * time.Millisecond) } } } func main() { ctx, cancel := context.WithCancel(context.Background()) go worker(ctx) time.Sleep(300 * time.Millisecond) cancel() // signal worker to stop time.Sleep(100 * time.Millisecond) } This stops the worker goroutine cleanly.

Result

The program prints 'working' a few times then 'worker stopped'.

Using context prevents goroutines from running forever and leaking resources, which is critical in real applications.

7

Expert

Avoiding race conditions with sync primitives

🤔Before reading on: do you think channels alone prevent all data races? Commit to your answer.

Concept: Sometimes you need locks or atomic operations to protect shared data beyond channels.

Channels help communication but don't protect shared variables accessed by multiple goroutines. Example of a race: var counter int func increment(wg *sync.WaitGroup) { defer wg.Done() counter++ // unsafe } func main() { var wg sync.WaitGroup for i := 0; i < 1000; i++ { wg.Add(1) go increment(&wg) } wg.Wait() fmt.Println(counter) // often less than 1000 due to race } Fix with a mutex: var mu sync.Mutex func incrementSafe(wg *sync.WaitGroup) { defer wg.Done() mu.Lock() counter++ mu.Unlock() } This ensures correct counting.

Result

Without mutex, counter is incorrect; with mutex, counter is 1000.

Knowing when to use locks versus channels is key to writing correct and efficient concurrent programs.

Under the Hood

Go's runtime manages goroutines as lightweight threads multiplexed onto OS threads. Channels use internal queues and synchronization to safely pass data between goroutines without explicit locks. The scheduler switches goroutines efficiently to maximize CPU use. Select statements use runtime mechanisms to wait on multiple channels without busy waiting. Context propagates cancellation signals through a tree of goroutines. Sync primitives like mutexes use atomic CPU instructions to protect shared memory.

Why designed this way?

Go was designed to make concurrency simple and safe by providing goroutines and channels as first-class features. This avoids complex thread management and common bugs in other languages. The design favors communication over shared memory to reduce errors. Context was added to handle cancellation cleanly in networked and server applications. Sync primitives exist for cases where shared memory is unavoidable.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Goroutine   │──────▶│   Channel     │──────▶│   Goroutine   │
│  Scheduler   │◀──────│  (buffered)   │◀──────│  Scheduler   │
└───────────────┘       └───────────────┘       └───────────────┘

┌───────────────┐
│   Context     │
│ Cancellation  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Goroutine Tree │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do channels automatically protect shared variables from race conditions? Commit to yes or no.

Common Belief: Channels always prevent race conditions because they handle communication safely.

Reality: Channels only protect data passed through them, not shared variables accessed outside channels.

Why it matters: Assuming channels protect all shared data leads to subtle bugs and unpredictable program behavior.

Quick: Do goroutines stop automatically when the main function ends? Commit to yes or no.

Common Belief: When main ends, all goroutines stop immediately.

Reality: Goroutines keep running unless the program exits or they receive a stop signal like context cancellation.

Why it matters: Ignoring this causes goroutines to leak and waste resources, leading to crashes or slowdowns.

Quick: Does using many goroutines always make a program faster? Commit to yes or no.

Common Belief: More goroutines always mean better performance.

Reality: Too many goroutines can cause overhead, contention, and slowdowns if not managed properly.

Why it matters: Blindly spawning goroutines wastes CPU and memory, hurting performance instead of helping.

Quick: Does the select statement wait for all channels to be ready before proceeding? Commit to yes or no.

Common Belief: Select waits until all channels have data before choosing one.

Reality: Select proceeds as soon as any one channel is ready, picking randomly if multiple are ready.

Why it matters: Misunderstanding select can cause logic errors and missed events in concurrent programs.

Expert Zone

1

Channels can be buffered or unbuffered, and choosing the right type affects blocking behavior and performance subtly.

2

Context cancellation propagates through goroutine trees, but improper use can cause leaks if not handled carefully.

3

Using sync primitives alongside channels requires careful design to avoid deadlocks and race conditions.

When NOT to use

Avoid concurrency patterns when tasks are simple and sequential, as concurrency adds complexity and overhead. For shared state, consider using atomic operations or specialized concurrent data structures instead of channels. In distributed systems, use message queues or event streams rather than local concurrency patterns.

Production Patterns

In real systems, pipelines are used for data processing stages, worker pools handle HTTP requests or jobs, and context manages request lifecycles and cancellations. Select is used for multiplexing IO and timers. Mutexes protect caches or counters. Combining these patterns carefully leads to scalable, maintainable concurrent applications.

Connections

Event-driven programming

Builds-on

Concurrency patterns in Go often implement event-driven ideas where tasks react to events or messages, improving responsiveness.

Operating system threads

Underlying mechanism

Goroutines are multiplexed onto OS threads, so understanding OS threads helps grasp Go's concurrency efficiency and limits.

Project management workflows

Analogous pattern

Just like concurrency patterns organize tasks and communication in code, project workflows organize people and tasks to avoid conflicts and delays.

Common Pitfalls

#1 Starting goroutines without synchronization causes the main program to exit before they finish.

Wrong approach:

func main() {
go func() {
fmt.Println("Hello")
}()
}

Correct approach:

func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
fmt.Println("Hello")
}()
wg.Wait()
}

Root cause:

The main function exits immediately, killing all goroutines; waiting ensures they complete.

#2 Accessing shared variables from multiple goroutines without locks causes race conditions.

Wrong approach:

var counter int
func increment() {
counter++ // unsafe
}

Correct approach:

var mu sync.Mutex
func increment() {
mu.Lock()
counter++
mu.Unlock()
}

Root cause:

Concurrent writes/read without synchronization lead to unpredictable results.

#3 Closing a channel multiple times causes a panic.

Wrong approach:

close(ch)
close(ch) // panic

Correct approach:

close(ch) // close only once

Root cause:

Channels must be closed exactly once to signal no more data; closing again crashes the program.

Key Takeaways

Goroutines are lightweight threads that let you run many tasks concurrently in Go.

Channels provide safe communication between goroutines, avoiding shared memory conflicts.

Common concurrency patterns like pipelines and worker pools organize tasks for better performance and clarity.

Context helps manage cancellation and timeouts, preventing resource leaks in concurrent programs.

Understanding when to use locks versus channels is essential to avoid race conditions and deadlocks.

Practice

1. What is the primary purpose of a WaitGroup in Go concurrency?

easy

A. To limit the number of goroutines running at the same time

B. To send data safely between goroutines

C. To wait for a collection of goroutines to finish before continuing

D. To create new goroutines

Solution

1. Step 1: Understand WaitGroup's role

   A WaitGroup is used to wait for multiple goroutines to complete their work.

2. Step 2: Compare with other concurrency tools

   Channels send data, semaphores limit concurrency, and goroutines run functions concurrently.

3. Final Answer:

   To wait for a collection of goroutines to finish before continuing -> Option C

4. Quick Check:

   WaitGroup = wait for goroutines [OK]

Hint: WaitGroup waits for goroutines to finish, not for data passing [OK]

Common Mistakes:

• Confusing WaitGroup with channels
• Thinking WaitGroup limits goroutines
• Using WaitGroup to send data

2. Which of the following is the correct syntax to start a new goroutine calling function doWork()?

easy

A. run doWork()

B. goroutine doWork()

C. start doWork()

D. go doWork()

Solution

1. Step 1: Recall goroutine syntax

   In Go, the keyword go before a function call starts it as a goroutine.

2. Step 2: Check options

   Only go doWork() matches the correct syntax; others are invalid keywords.

3. Final Answer:

   go doWork() -> Option D

4. Quick Check:

   Start goroutine with 'go' keyword [OK]

Hint: Start goroutine with 'go' keyword before function call [OK]

Common Mistakes:

• Using non-existent keywords like 'goroutine' or 'start'
• Omitting 'go' keyword
• Trying to run goroutine without parentheses

3. What will be the output of this Go program?

package main
import (
  "fmt"
  "sync"
)
func main() {
  var wg sync.WaitGroup
  wg.Add(2)
  go func() {
    fmt.Println("Hello from goroutine 1")
    wg.Done()
  }()
  go func() {
    fmt.Println("Hello from goroutine 2")
    wg.Done()
  }()
  wg.Wait()
  fmt.Println("All goroutines finished")
}

medium

A. Hello from goroutine 2
Hello from goroutine 1

B. Hello from goroutine 1
Hello from goroutine 2
All goroutines finished

C. Hello from goroutine 1
All goroutines finished
Hello from goroutine 2

D. All goroutines finished
Hello from goroutine 1
Hello from goroutine 2

Solution

1. Step 1: Understand WaitGroup usage

   The program adds 2 to the WaitGroup, then starts 2 goroutines that each print a message and call wg.Done().

2. Step 2: Analyze output order

   The main goroutine waits for both goroutines to finish before printing "All goroutines finished". The two goroutine prints can appear in any order but both appear before the final print.

3. Final Answer:

   Hello from goroutine 1 Hello from goroutine 2 All goroutines finished -> Option B

4. Quick Check:

   WaitGroup ensures final print after goroutines [OK]

Hint: WaitGroup.Wait blocks until all Done calls complete [OK]

Common Mistakes:

• Assuming main prints before goroutines
• Ignoring WaitGroup synchronization
• Expecting goroutine prints after final print

4. Identify the error in this Go code snippet using channels:

package main
import "fmt"
func main() {
  ch := make(chan int)
  go func() {
    ch <- 10
  }()
  fmt.Println(<-ch)
  fmt.Println(<-ch)
}

medium

A. Second read from channel blocks forever

B. Channel is not closed before reading

C. Channel should be buffered to send data

D. Goroutine is not started correctly

Solution

1. Step 1: Analyze channel sends and receives

   The goroutine sends one value (10) into the channel. The main goroutine reads twice from the channel.

2. Step 2: Identify blocking behavior

   The first read gets 10 successfully. The second read waits for another value that never comes, causing a deadlock (blocks forever).

3. Final Answer:

   Second read from channel blocks forever -> Option A

4. Quick Check:

   One send, two receives causes blocking [OK]

Hint: Reads must match sends; extra read blocks forever [OK]

Common Mistakes:

• Thinking channel must be closed before reading
• Assuming buffered channel fixes blocking here
• Believing goroutine not started properly

5. You want to create a worker pool in Go to process 5 jobs concurrently but limit to 2 workers at a time. Which pattern correctly implements this?

hard

A. Use a buffered channel of size 2 as a semaphore to limit goroutines, and a WaitGroup to wait for all jobs

B. Start 5 goroutines without limit and use a WaitGroup to wait

C. Use an unbuffered channel to send jobs and no WaitGroup

D. Use a mutex to lock and unlock workers

Solution

1. Step 1: Understand worker pool concurrency limit

   To limit concurrent workers, a buffered channel (semaphore) of size 2 can control how many goroutines run simultaneously.

2. Step 2: Use WaitGroup to wait for all jobs

   A WaitGroup tracks all 5 jobs to ensure main waits until all finish before exiting.

3. Final Answer:

   Use a buffered channel of size 2 as a semaphore to limit goroutines, and a WaitGroup to wait for all jobs -> Option A

4. Quick Check:

   Semaphore + WaitGroup = controlled concurrency [OK]

Hint: Semaphore channel limits workers; WaitGroup waits for all [OK]

Common Mistakes:

• Starting all goroutines without limit
• Not using WaitGroup to wait for completion
• Using mutex incorrectly for concurrency limit