golang parallel assignment

Worker Pool in Go

In high-performance applications, managing tasks concurrently can help maximize efficiency and reduce execution time. One common concurrency pattern for task processing is the Worker Pool . A worker pool is a mechanism where multiple tasks are distributed among a fixed number of workers that process them concurrently. In Go, with its lightweight goroutines, creating a worker pool is straightforward and effective.

This guide will walk you through setting up a worker pool in Go, explaining each component and best practices along the way.

Why Use a Worker Pool?

Worker pools are particularly useful for tasks that:

• Are CPU-bound or I/O-bound and can be parallelized.
• Require a rate limit on concurrent execution (e.g., hitting an external API).
• Need better memory management by limiting the number of active goroutines.

Instead of spawning a new goroutine for each task, a worker pool allows you to control the number of goroutines. This prevents excessive memory usage and ensures your application remains performant under heavy load.

Worker Pool Components

A Go worker pool typically consists of:

• Tasks Queue (Channel) : Holds tasks for workers to pick up.
• Worker Goroutines : Perform the task assigned from the queue.
• Result Channel (Optional) : Collects results if needed.
• Main Goroutine : Orchestrates the creation of tasks, workers, and waits for completion.

Implementing a Worker Pool in Go

Here’s a step-by-step example of implementing a simple worker pool in Go.

Step 1: Define Your Task and Worker Functions

Let’s assume each worker will process an integer task by squaring it.

• worker : Each worker receives tasks from the tasks channel and sends results to the results channel. The id is for logging purposes to show which worker is processing each task.

Step 2: Initialize the Task and Result Channels

The tasks channel will hold tasks, and results will store the squared numbers returned by workers.

• Task Dispatching : Tasks are added to the tasks channel. Closing the tasks channel signals to workers that no more tasks will be provided.

Step 3: Collect and Display Results

To avoid blocking the main goroutine, use a sync.WaitGroup to ensure all tasks are completed before processing results.

• Result Processing : The results are printed in the main goroutine, which waits until all workers have processed their tasks.

Full Example Code

Key takeaways.

• Concurrency Management : Worker pools prevent creating too many goroutines, allowing better control over system resources.
• Task Queue : Channels help decouple task distribution from task processing.
• Graceful Shutdown : By closing the tasks channel, workers exit gracefully after processing all tasks.

Use Cases for Worker Pools

• Web Scraping : Parallelize requests to multiple web pages.
• File Processing : Process multiple files concurrently, limiting resource usage.
• API Requests : Send multiple requests in parallel while respecting rate limits.

By implementing a worker pool in Go, you gain fine-grained control over concurrency, which is crucial in applications with high loads. Worker pools can reduce memory usage and increase efficiency, helping applications scale effectively.

• Is CS 121 a good fit for you?
• Academic Honesty
• Module Readings
• M1: Introduction to Parallel Programming & Golang
• M2: Shared Memory Architectures
• M3: Principles of Mutual Exclusion
• M4: Concurrent Objects (Part 1)
• M5: Concurrent Objects (Part 2)
• M6: Concurrent Execution Models
• M7: Advanced Parallel Scheduling Techniques (Part 1)
• M8: Advanced Parallel Scheduling Techniques (Part 2)
• M9: GPGPU Programming & Parallel Programming in C & Python
• Assignments
• Office Hours
• Asking Questions on Ed
• UChicago CS Student Resource Guide
• Running Go Remotely

M1: Introduction to Parallel Programming & Golang ¶

This first module provides an introduction to the course and parallel programming. During this module you will also get familiar with the language for this course, Golang.

Pre-recorded Lectures ¶

Note: The pre-recorded videos for M1 will be posted after Tuesday’s lecture .

The pre-recorded lectures are available here: M1 Videos . You can also find the videos under the “Panopto” tab on the MPCS 52060 canvas site.

The lectures are a series of approx 20-30 minute videos divided into the following sections:

Golang Code Structure & Tips <- If you already know Go then you can skip this video.

The slides presented in lecture and these videos are accessible on our Canvas Page. Click on the Files link and you then can download the m1.zip file.

Resources/Readings ¶

For this module, your focus should be to learn the basic syntax of the Go language. Look at the Module 1 slides (slide 36) to know what constructs to focus on.

The slides and code presented in this module are accessible on our Canvas Page. Click on the Files link and you then can download the m1.zip file.

See M1 slides to know the links to click on.

Great resource for learning and experimenting with the language syntax

The Go Programming Language (textbook) provides a few chapters on the syntax of the language. This is not required but if you want a more in-depth description about language constructs then you should reference this book.

There will be no readings from the official textbook this week.

Synchronous Session (In-person Lecture) ¶

As a reminder here are the dates and times for the synchronous session for this module:

Section 1: Tuesday September 27th @ 12:30pm-2:20pm

Course Logistics

Introduction to Parallel Programming

Golang Overview [if time permits]

Assignment ¶

Assignments are always due on Wednesday evenings.

Homework #1 , due Wednesday October 5th at 11:59pm CDT

Parallel Function Execution in Go Using Concurrency

Introduction

As part of my exploration of Golang, I came across a popular feature: first-class support for concurrency. I believe we all understand the benefit or importance of concurrency. In the HTTP way, when an endpoint needs to fetch data from multiple upstreams , aggregate the data and produce it as a response, Go concurrency helps to reduce the latency for that API request. Two features in Go, goroutines and channels make concurrency easier when used together.

Goroutines example: Run functions in parallel

Modern computers are equipped with processors, or CPUs , designed to efficiently handle multiple streams of code simultaneously. These processors are built with one or more "cores," each capable of running one code stream at a given time. To fully utilize the speed boost multiple cores offer, programs must be able to split into various streams of code. This division can be challenging, but Go was explicitly developed to simplify this process.

Go achieves this through a feature known as goroutines , special functions that can run alongside other goroutines. When a program is built to execute multiple streams of code simultaneously, it operates concurrently . Unlike traditional foreground operations, in which a function runs to completion before the following code executes, goroutines allow for background processing, enabling the following code to run while the goroutine is still active. This background operation ensures that the code doesn't block other processes from running.

Goroutines provide the advantage of running on separate processor cores simultaneously. For instance, if a computer has four processor cores and a program has four goroutines, all four can run concurrently. This simultaneous execution of multiple code streams on different cores is called parallel processing.

Jumping into the example, create a multifunc directory named go-concurrency-project .

Once you’re in the go-concurrency-project Directory, open a file named main.go using nano , or the editor of your choice:

Add the following code to the main.go file,

Based on the above setup, make and print Functions are structured to run in sequence. make Accepts a number to generate up to and prints only five numbers.

This is how it will look like when we execute main.go ,

If you notice, the function printed the output in sequence based on its execution pattern.

When running two functions synchronously , the program takes the total time for both functions to run. But if the functions are independent, you can speed up the program by running them concurrently using goroutines , potentially cutting the time in half. To run a function as a goroutine, use the go keyword before the function call. However, you need to add a way for the program to wait until both goroutines have finished running to ensure they all complete running.

To synchronize functions and wait for them to finish in Go, you can use a WaitGroup from the sync package. The WaitGroup primitive counts how many things it needs to wait for using the Add , Done , and Wait functions. The Add function increases the count, Done decreases the count, and Wait can be used to wait until the count reaches zero.

To do that update main.go ,

After declaring WaitGroup , specify how many processes to wait for. In the example, the goroutine waits for two Done calls before finishing. If not set before starting the goroutines, things might happen out of order, or the code may panic because wg doesn't know if it should wait for any Done calls.

Each function will use defer to call Done , which decreases the count by one after the function finishes. The main function is updated to include a call to Wait on the WaitGroup . This ensures that the main function waits until both functions call Done before continuing and exiting the program.

After saving your main.go execute the file,

Your output may vary each time you run the program. With both functions running concurrently, the output depends on how much time Go and your operating system allocates to each function. Sometimes, each function runs entirely, and you'll see their complete sequences. Other times, the text will be interspersed.

If you’re interested in learning more about concurrency in Go, the Effective Go document created by the Go team provides much more detail. The Concurrency is not parallelism Go blog post is also an exciting follow-up about the relationship between concurrency and parallelism. These two terms are sometimes mistakenly thought to mean the same thing.

Thank you for reading this article! If you're interested in DevOps, Security, or Leadership for your startup, feel free to reach out at [email protected] or book a slot in my calendar . Don't forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for more posts!

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

parallel-programming

Here are 41 public repositories matching this topic..., workanator / go-floc.

Floc: Orchestrate goroutines with ease.

• Updated Aug 10, 2021

xyctruth / stream

A Stream processing library based on Go 1.18+ Generics (parallel, pipeline, lazy) 🍒

• Updated Jul 3, 2023

zllangct / ecs

A Go-implementation of the ECS (Entity-Component-System), focus on the development of game server.

• Updated Nov 27, 2024

Balun-courses / concurrency_go

Repository for course by Concurrency in Go

• Updated Dec 6, 2024

Prrromanssss / DAEC-fullstack

Distributed arithmetic expression calculator

• Updated May 31, 2024

Deeptiman / go-batch

A Simple Batch Processing library in Go

• Updated Jun 26, 2023

hyp3rd / go-worker

`go-worker` provides a simple way to manage and execute tasks concurrently and prioritized, leveraging a `TaskManager` that spawns a pool of `workers`.

• Updated Jun 4, 2023

simonayzman / chess-openings-analyzer

Analyzes user-provided chess opening positions based on historical grandmaster games.

• Updated Dec 17, 2020

letcgo / tigo

This is a tiny framework for golang.

• Updated Jul 24, 2018

freddiehaddad / justification

Multithreaded Text Justification

• Updated Mar 1, 2024

zevolution / golang-netflix-hexagonal-arch

• Updated Apr 28, 2023

felixgunawan / safe-step

Safe goroutine flow handler

• Updated Jun 13, 2021

jcbritobr / go-concurrency-patterns

Examples on how to implement concurrency patterns in golang, and use the best feature of the language.

• Updated Apr 26, 2023

DhruvSrikanth / PipelineBSPImageEditor

An image editor implemented in Go, utilizing the pipeline and BSP parallelism schemes.

• Updated Dec 23, 2022

harrisonho99 / concurrency-in-go

• Updated Aug 14, 2022

jvrieger / 245ProgrammingLanguages

BMC CMSC245 Principles of Programming Languages Assignments and Labs with Geoffrey Towel

• Updated Jan 11, 2024

pw22232 / game-of-life

parallel and distributed programming on Conway's Game Of Life

• Updated Mar 11, 2024

DhruvSrikanth / WorkBalancingStealingImageEditor

An image editor implemented using the work balancing and work stealing parallelism schemes in Go.

harryosmar / go-playground

Go playground using https://echo.labstack.com/

• Updated May 13, 2020

pouyakary / uniparallel

A set of tools for Go to teach Parallel Programming

• Updated May 21, 2018

Improve this page

Add a description, image, and links to the parallel-programming topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the parallel-programming topic, visit your repo's landing page and select "manage topics."