Pipeline Pattern in Go: What It Is and How It Works

The pipeline pattern in Go is a way to connect multiple stages of processing using goroutines and channels, where each stage receives input from the previous one and sends output to the next. It helps organize concurrent tasks as a chain, making data flow clear and efficient.

How It Works

Imagine an assembly line in a factory where each worker does one specific job and then passes the product to the next worker. The pipeline pattern in Go works similarly by connecting several processing steps (called stages) in a sequence. Each stage runs in its own goroutine and communicates with the next stage through channels.

This setup allows data to flow smoothly from one stage to another without waiting for the entire process to finish. Each stage can work independently and concurrently, improving efficiency and making the program easier to understand and maintain.

Example

This example shows a simple pipeline with three stages: generating numbers, squaring them, and printing the results.

go

package main

import (
	"fmt"
)

// gen sends numbers to a channel
func gen(nums ...int) <-chan int {
	out := make(chan int)
	go func() {
		for _, n := range nums {
			out <- n
		}
		close(out)
	}()
	return out
}

// sq reads numbers, squares them, and sends to a channel
func sq(in <-chan int) <-chan int {
	out := make(chan int)
	go func() {
		for n := range in {
			out <- n * n
		}
		close(out)
	}()
	return out
}

func main() {
	// Set up the pipeline
	numbers := gen(2, 3, 4)
	squares := sq(numbers)

	// Receive and print results
	for sq := range squares {
		fmt.Println(sq)
	}
}

Output

4
9
16

When to Use

Use the pipeline pattern when you have multiple steps that process data in order, especially if each step can work independently and in parallel. It is great for tasks like data processing, streaming, or any situation where you want to break a big job into smaller, manageable parts.

For example, you might use it to read data from a file, filter or transform it, and then write the results somewhere else, all running concurrently to improve speed and clarity.

Key Points

• The pipeline pattern connects stages with channels and goroutines.
• Each stage processes data and passes it along, like an assembly line.
• It improves concurrency and code organization.
• Channels handle communication and synchronization between stages.
• It is useful for streaming and multi-step data processing.

Key Takeaways

The pipeline pattern uses goroutines and channels to connect processing stages in Go.

It allows concurrent, step-by-step data processing like an assembly line.

Each stage runs independently and passes results to the next via channels.

Use it to organize complex workflows into clear, manageable parts.

It improves performance by enabling parallel execution of tasks.