Go from scratch, part 13: concurrency patterns (worker pools, fan-in, fan-out and pipelines)

> series: learn go from 0 to 100 | article 13 of 19

---

You’ve got goroutines and channels. Today we combine them into the three Go concurrency patterns that solve 90% of real concurrent problems. These are the same blueprints industry uses, documented on the official Go blog and in Rob Pike’s classic talk Go Concurrency Patterns.

---

Pipeline: the assembly line

A pipeline is a series of stages connected by channels: each stage receives from the previous one, transforms, and sends downstream (Go Concurrency Patterns: Pipelines):

// Stage 1: generate numbers.
func gen(nums ...int) <-chan int {
  out := make(chan int)
  go func() {
    defer close(out) // the sender closes
    for _, n := range nums {
      out <- n
    }
  }()
  return out
}

// Stage 2: square them.
func square(in <-chan int) <-chan int {
  out := make(chan int)
  go func() {
    defer close(out)
    for n := range in {
      out <- n * n
    }
  }()
  return out
}

// Full pipeline: reads like a sentence.
for v := range square(gen(2, 3, 4)) {
  fmt.Println(v) // 4 9 16
}

Note each stage’s pattern: create your output channel, launch your goroutine, return the channel. The defer close(out) honors article 12‘s golden rule: the sender closes. Each stage runs in parallel with the rest, like a real assembly line: while one part gets painted, the next is already being cut.

Worker pool: N workers, one queue

The most profitable pattern in real work: capping how many tasks you process in parallel (API calls, queries, files) without slowing the intake:

func worker(id int, jobs <-chan int, results chan<- int) {
  for j := range jobs { // automatic distribution
    results <- j * 2 // simulated hard work
  }
}

jobs := make(chan int, 100)
results := make(chan int, 100)

// 3 workers compete for the same jobs.
for w := 1; w <= 3; w++ {
  go worker(w, jobs, results)
}

for j := 1; j <= 9; j++ {
  jobs <- j
}
close(jobs) // no more work

for i := 0; i < 9; i++ {
  fmt.Println(<-results)
}

The magic is several workers doing range over the SAME channel: each job gets taken by exactly one, distribution is automatic and free. The Java equivalent? An ExecutorService with its queue and thread factory. Here it’s fifteen lines of stdlib.

Fan-out and fan-in: spread and gather

The worker pool already hides the fan-out: several readers on one channel splitting the work. Its twin is fan-in: merging several sources into a single channel (pipelines blog):

// fan-in: merge several channels into one.
func merge(inputs ...<-chan int) <-chan int {
  out := make(chan int)
  var wg sync.WaitGroup
  wg.Add(len(inputs))
  for _, in := range inputs {
    go func(c <-chan int) {
      defer wg.Done()
      for v := range c {
        out <- v
      }
    }(in)
  }
  go func() {
    wg.Wait() // when all are done...
    close(out) // ...close the output
  }()
  return out
}

Here sync.WaitGroup shows up ahead of its article (14): it’s a counter of live goroutines. The problem it solves is subtle: the output can only close when ALL inputs are done, and no single copying goroutine knows that on its own.

All three, together

A real service chains all three without blinking: generate URLs (pipeline), spread them across N downloading workers (fan-out), merge results (fan-in) and keep processing downstream. Each piece is small, testable, and lock-free on the surface. This is article 11‘s CSP promise delivered by these Go concurrency patterns: the program’s structure IS the concurrency’s structure.

Technical honesty notice: one ingredient is missing. What if the consumer walks away midway and workers stay blocked sending into a channel nobody reads? That’s a goroutine leak, and its official fix (cancellation with context) is next article’s main course. The pipelines blog covers the done channel version if you can’t wait.

---

Weekly challenge: put Go concurrency patterns to work

• Build a three stage pipeline: generate 1-20, filter evens, cube them. Each stage with its goroutine and defer close.
• A 4-worker pool checking which of 50 numbers are prime. Compare timing with 1 worker vs 4 (hint: time.Since).
• Implement merge and wire it to two independent generators. Verify with -race.
• Bonus: break something on purpose. Remove the worker pool’s close(jobs) and watch the deadlock the runtime detects and reports. Knowing the error up close vaccinates.

---

Go concurrency patterns: level complete.

Next article: sync, context and cancellation. How to politely tell a hundred goroutines the party’s over.

> exit