GoHow-ToBeginner · 4 min read

How to Implement Worker Pool in Go: Simple Guide and Example

In Go, a worker pool is implemented by creating a fixed number of goroutines (workers) that receive tasks from a shared channel. You send jobs to this channel, and each worker processes them concurrently, improving efficiency and control over resource usage.

📐

Syntax

A worker pool in Go typically involves these parts:

Job channel: A channel to send tasks to workers.
Worker goroutines: Fixed number of goroutines that receive jobs from the channel.
WaitGroup: To wait for all workers to finish processing.

This pattern controls concurrency by limiting how many workers run at once.

jobs := make(chan int, 100) // channel to send jobs
var wg sync.WaitGroup       // wait group to wait for workers

// Worker function
func worker(id int, jobs <-chan int, wg *sync.WaitGroup) {
    for job := range jobs {
        // process job
    }
    wg.Done()
}

// Start workers
for w := 1; w <= 5; w++ {
    wg.Add(1)
    go worker(w, jobs, &wg)
}

// Send jobs
for j := 1; j <= 10; j++ {
    jobs <- j
}
close(jobs) // close channel to signal no more jobs

wg.Wait() // wait for all workers to finish

💻

Example

This example creates 3 workers that process 5 jobs. Each worker prints when it starts and finishes a job. It shows how to use channels and WaitGroup to coordinate work.

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(id int, jobs <-chan int, wg *sync.WaitGroup) {
    for job := range jobs {
        fmt.Printf("Worker %d started job %d\n", id, job)
        time.Sleep(time.Second) // simulate work
        fmt.Printf("Worker %d finished job %d\n", id, job)
    }
    wg.Done()
}

func main() {
    jobs := make(chan int, 5)
    var wg sync.WaitGroup

    // Start 3 workers
    for w := 1; w <= 3; w++ {
        wg.Add(1)
        go worker(w, jobs, &wg)
    }

    // Send 5 jobs
    for j := 1; j <= 5; j++ {
        jobs <- j
    }
    close(jobs) // no more jobs

    wg.Wait() // wait for all workers
}

Output

Worker 1 started job 1 Worker 2 started job 2 Worker 3 started job 3 Worker 1 finished job 1 Worker 1 started job 4 Worker 2 finished job 2 Worker 2 started job 5 Worker 3 finished job 3 Worker 1 finished job 4 Worker 2 finished job 5

⚠️

Common Pitfalls

Common mistakes when implementing worker pools in Go include:

Not closing the jobs channel, causing workers to wait forever.
Not using sync.WaitGroup properly, leading to premature program exit.
Sending more jobs than the channel buffer without closing it, causing deadlocks.
Not handling errors inside workers, which can silently fail.

Always close the jobs channel after sending all jobs and use WaitGroup to wait for workers.

package main

import (
    "fmt"
    "sync"
)

func worker(id int, jobs <-chan int, wg *sync.WaitGroup) {
    for job := range jobs {
        fmt.Printf("Worker %d processing job %d\n", id, job)
    }
    wg.Done()
}

func main() {
    jobs := make(chan int)
    var wg sync.WaitGroup

    // Start 2 workers
    for w := 1; w <= 2; w++ {
        wg.Add(1)
        go worker(w, jobs, &wg)
    }

    // Mistake: Not closing jobs channel
    for j := 1; j <= 3; j++ {
        jobs <- j
    }
    // close(jobs) // <- missing causes deadlock

    wg.Wait() // program hangs here
}

📊

Quick Reference

jobs channel: Send tasks here.
workers: Fixed goroutines reading from jobs.
WaitGroup: Wait for all workers to finish.
Close channel: Signal no more jobs.
Buffer size: Controls how many jobs can wait in queue.

✅

Key Takeaways

Use a channel to send jobs and fixed goroutines as workers to process them concurrently.

Always close the jobs channel after sending all tasks to avoid deadlocks.

Use sync.WaitGroup to wait for all workers to finish before exiting the program.

Limit the number of workers to control concurrency and resource usage.

Handle errors inside workers to avoid silent failures.