Te-Yuan Liu
Home Blog
Go
May 29, 2026 · 7300 words · 37 mins read

Go is a statically typed, compiled programming language. It has fast compilation and concurrency support via goroutines and channels. It uses a garbage collector to manage the heap memory.

File

File naming

We use snake_case for Go file names.

Due to compatibility across case-sensitive OS like Linux and case-insensitive OS like Windows and MacOS, it's safer to use lowercase in every name.

Because the Go compiler uses underscores for special file suffixes to control the build process (*_test.go for test file and *_linux.go for Linux-only compiling), it's idiomatic to use underscores to group words together, and we arrive at the conclusion of using snake_case for Go files.

We use kebab-case for everything else for example compiled binaries, such that they are consistent with other shell tools.

Expression and statement

A Go file consists of expressions and statements for the computer to read, parse, and execute instructions. Here is a comparison between the expression and statement.

\ExpressionStatement
PurposeTo produce dataTo execute an instruction
Value returningYesNo
Examples5; a + b; len(array)x = 5; if ... else ...; for ...; return value; import "fmt"; i++

Package

A package is a collection of Go files inside the same directory. Variables, constants, functions, and types defined under the same package are visible across all Go files in the package.

Package naming

We use lowercase single word for package names so the user service package becomes userservice. The same applies to directory names as they have to match package names.

Module

A module is a collection of packages. It is like a project or repository. A module contains all packages inside its root directory (the directory that has the go.mod), including those in the subdirectories, except any subdirectory that contains another go.mod, which therefore defines another module.

Module initialization

We use go init mod <MODULE_PATH> to initialize a module where <MODULE_PATH> is the path to our module on a repository platform like GitHub. We usually use a GitHub repository to host a module so the module path is github.com/<ORGANIZATION>/<REPOSITORY>. The module path also defines the import path prefix for all packages within the module.

Running the command generates the go.mod file that stores the module path and the Go version we are using.

go.mod 

module <MODULE_PATH>
go <VERSION>

Code structure

The goal of this code structure is flexibility with minimum abstraction. It divides the system into 3 layers, handler, service, and infrastructure.

Here is an application for user registration, login, and logout.

/cmd
    /main
        main.go
/internal
    /server
        /rest.go
    /domain
        /user.go
        /port.go
    /database
        /postgresql.go

Here are the 3 layers.

At last, cmd/main/main.go creates infrastructure interface values, service structs, and then the handler, injecting dependencies, and starts the handler.

Variable

Variable declaration

The var statement declares a list of variables. It can be at package or function level.

Go uses the var <VARIABLE_NAME> <VARIABLE_TYPE> declaration format for variables. This makes variable declaration easier to understand when comparing with C, especially for pointers.

// Declare an integer pointer p in C.
int *p

// Declare an integer pointer p in Go.
var p *int

Variables declared without an initial value are given their zero value.

We can declare a variable via the explicit way or its shorter version. Note that the shorter version is only available within a function so we must use the explicit way at the package level.

The shorter version is more commonly used because it is more concise. When using the shorter version, Go can infer the variable type automatically from its initial value. When the variable initial value is an untyped numeric constant, the variable type is decided by Go based on the precision of the constant.

// Explicit declaration
var i int = 10  // The int type here can be omitted as Go can infer it from the initial value.

// Shorter version
i := 10

Variable naming convention

The Go community has the naming convention of PascalCase for exported variables, functions, and camelCase for unexported variables, functions. For example, Pi is exported from the math package and can be accessed via math.Pi.

Primitive variable type

Variable type conversion

We can use the T(v) expression to convert a variable v to the type T.

// Explicit declaration
var i int = 2
var f float64 = float64(i)

// Shorter version
i := 2
f := float64(i)

Byte

Rune

String

String concatenation

Array

An array is a contiguous chunk of memory that has a fixed length, so the length is part of its type.

If the compiler thinks that an array is only accessed within a specific scope and its memory consumption is small (less than 64KB), it is stored on the stack and freed once out-of-scope.

Otherwise, it is allocated on the heap, which is managed by Go's garbage collector. The Go garbage collector reclaims memory when it sees that an object is no longer reachable by any active part of the program.

Array declaration

// Declare an integer array with 2 in length. By default the elements are set to 0.
var a [2]int

// Declare an integer array with an array literal.
var a = [2]int{1, 2}

// Shorter version
a := [2]int{1, 2}

// Declare an integer array with an array literal with length-inferring.
var a = [...]int{1, 2}

// Shorter version
a := [...]int{1, 2}

Slice

A slice is dynamically-sized, flexible view into the elements of an array. When created, it is allocated on the stack.

It contains:

Because a slice is lightweight and super cheap to copy, we usually pass it around by value.

If any slice still holds a pointer to its underlying array, the entire array remains in memory. Therefore, the way to release the array's memory is to set the slice to nil.

If our slice is using a small portion of the underlying array, we can use the below methods to cut down memory cost.

Slice declaration

A slice literal is like an array literal without the length. During the slice literal declaration, Go creates the underlying array and builds a slice that references it.

We can use the built-in println function to learn a slice's information.

// Declare an integer slice. By default its pointer = nil and length = capacity = 0.
var s []int

// Declare via a slice literal.
var s = []int{1, 2}

// Shorter version
s := []int{1, 2}

// Declare a slice with make so it has 2 in length and 4 in capacity.
// Under the hood, it creates an array with 4 in length and sets the slice's pointer to point to the array's first element.
s = make([]int, 2, 4)

// Shorter version
s := make([]int, 2, 4)

// Print the slice.
println(s)          // Print [2/4]0xa00001a120, where 2 for length, 4 for capacity, 0xa00001a120 for the pointer.
fmt.Println(len(s)) // Print 2 for length
fmt.Println(cap(s)) // Print 4 for capacity

// Declare a slice from an existing array
var a [2]int

// The following statements are equivalent.
s := a[:]
s := a[0:]
s := a[:2]
s := a[0:2]

Slice appending

We can append one or multiple elements or even another slice to the end of a slice. If the slice's capacity is exceeded, meaning the underlying array doesn't have the space to accommodate all elements, Go creates another array (usually doubling the array length), copying the elements into it, and updates the slice to use the new array.

If we know the exact capacity needed, or even just an estimate, we can preallocate the slice capacity and save the work for multiple array recreations.

s1 := []int{1}          // s1 = {1}
s1 = append(s1, 2, 3)   // s1 = {1, 2, 3}
s2 := []int{4}          // s2 = {4}
s1 = append(s1, s2...)  // s1 = {1, 2, 3, 4}

Byte slice generation

String slice concatenation

Heap

We can use the container/heap package to implement a min heap. It requires us to implement the Len, Less, Swap, Push, and Pop function for the container/heap package to use.

type IntHeap []int

func (h IntHeap) Len() int {
    return len(h)
}

func (h IntHeap) Less(i, j int) bool {
    return h[i] < h[j]
}

func (h IntHeap) Swap(i, j int) {
    h[i], h[j] = h[j], h[i]
}

func (h *IntHeap) Push(x any) {
    *h = append(*h, x.(int))
}

func (h *IntHeap) Pop() any {
    n := len(*h)
    last := (*h)[n - 1]
    *h = (*h)[: n - 1]
    return last
}

func main() {
    h := &IntHeap{}
    heap.Init(h)
    heap.Push(h, 1)
    s := heap.Pop(h)
}

Map

A map is used to store key-value pairs. Its zero value is nil and has no keys or the ability to accept new keys.

Map declaration

// Declare an integer-to-string map. This is meaningless because it's a nil map and cannot accept new keys.
var m map[int]string

// Declare via a map literal.
var m = map[int]string{1: "one"}

// Shorter version
m := map[int]string{1: "one"}

// Declare a map with make.
m := make(map[int]string)

When using a map literal to declare a map, we can omit the top-level value type if it is a type name.

type Vertex struct {
    X, Y int
}

// Explicit value type
m := map[string]Vertex{
    "Go": Vertex{1, 2},
}

// Shorter version
m := map[string]Vertex{
    "Go": {1, 2},
}

Map operation

// Insert or update a key-value pair.
m[key] = value

// Retrieve the value of a key.
value := m[key]

// Delete a key-value pair
delete(m, key)

// Test a key's existence.
value, ok := m[key]
// If the key is in m, ok is true.
// Otherwise, the value is the value type's zero value, and ok is false.

Pointer

A pointer is a variable that stores a memory address, usually the memory address of another variable. With a pointer, we can work on a large struct directly inside different functions without the need to copy and pass the object around.

Address operator

We can use the address operator & to get the memory address of a variable.

i := 2
p := &i

Dereference operator

We can use the dereference operator * to access the pointed-to variable.

i := 2
p := &i
*p = 4  // Now i has value 4.

Constant

Constant declaration

Go uses the const <CONSTANT_NAME> <CONSTANT_TYPE> declaration format for constants. Note that we cannot use the shorter version := to declare a constant.

Numeric constants are high-precision values. An untyped constant takes the type needed by its context.

Printing

We often use the Print, Println, and Printf function from the built-in fmt package.

Flow control

Loop

For loop

A for loop has 3 components separated by semicolons.

  1. An initial statement is executed before the first iteration.
  2. A condition expression is evaluated before every iteration.
  3. A post statement is executed at the end of every iteration.
for i := 0; i < 10; i++ {
    fmt.Println(i)
}

i := 0
for i < 10 {
    fmt.Println(i)
    i++
}

// Declare a for loop that runs forever.
for {
    fmt.Println("Here we Go!")
}

For-range loop

A for-range loop provides a concise way to iterate over a range, string, array, slice, or map. The basic syntax assigns iteration values to one or two variables, followed by the range keyword and then the collection.

We can even use this to read a channel. Note that if we want to handle different values received from the channel separately, consider to use the goroutine select.

// Loop over the index.
for index := range collection

// Loop over both the index and value.
for index, value := range collection

// Loop over the value.
for _, value := range collection

// Loop over the value received from a channel.
for value := range channel
Collection typeFirst valueSecond value (optional)
IntegerIndex (int)Not applicable
StringIndex (int)Rune (Unicode code point)
Array or SliceIndex (int)Element copy
MapKeyValue copy
ChannelElementNot applicable

If and else

An if statement has 2 components separated by a semicolon.

  1. An initial statement is executed first. Its variables stay in the scope of the if and else.
  2. A condition expression is then evaluated.
if check := true; check == true {
    fmt.Println("Check is true.")
} else if check == false {
    fmt.Println("Check is false.")
} else {
    fmt.Println("Panic: check is not boolean.")
}

Switch

A switch statement is a shorter way for flow control. It evaluates cases from top to bottom, and runs the first case whose value is equal to the condition expression.

import (
    "fmt"
    "runtime"
)

func main() {
    fmt.Print("Go runs on ")
    switch os := runtime.GOOS; os {
    case "linux":
        fmt.Println("Linux.")
    case "darwin":
        fmt.Println("macOS.")
    case "windows":
        fmt.Println("Windows.")
    default:
        fmt.Printf("%s.\n", os)
    }
}

Switch without condition is the same as switch true and is a clean way to write long if-then-else chains.

Defer

A defer statement evaluates its function arguments immediately but postpones the function execution until the defer statement's surrounding function returns.

Deferred function calls are pushed onto a stack and follows the Last-In-First-Out (LIFO) execution order.

We should avoid calling deferred function inside a for loop because all deferred function calls will stack on each other and accumulate resource usage. Instead, we should move the loop body which includes the deferred function call into its own function and call this function inside the for loop.

Function

Function declaration

Go uses the func <FUNCTION_NAME>(<FUNCTION_PARAMETER>) <RETURN_TYPE> declaration format for functions. This is also easier to understand comparing with C.

// Declare a function f that takes an integer parameter and returns an integer in C.
int f(int p)

// Declare a function f that takes an integer parameter and returns an integer in Go.
func f(p int) int

Function parameter

When two or more consecutive named function parameters share a type, we can omit the type from all but the last.

// Declare a function add that takes two integer parameters and returns an integer.
func add(x int, y int) int

// Shorter version
func add(x, y int) int

Function return value

When a function returns a value, it copies the value from the variable within the function and passes it to the outer world.

func main() {
    value := returnValue()
	fmt.Printf("value stored at %p\n", &value) // memory address 2
}

func returnValue() bool {
	value := true
	fmt.Printf("value stored at %p\n", &value) // memory address 1
	return value
}

The same holds for pointer returning. But note that value now lives on the heap instead of stack for access across different frames on the call stack.

func main() {
	pointer := returnPointer()
	fmt.Printf("pointer pointing to %p\n", pointer) // memory address 1
	fmt.Printf("pointer stored at %p\n", &pointer)  // memory address 3
}

func returnPointer() *bool {
	value := true
	pointer := &value
	fmt.Printf("pointer pointing to %p\n", pointer) // memory address 1
	fmt.Printf("pointer stored at %p\n", &pointer) // memory address 2
	return pointer
}

A function usually returns a value when the returned value is small or immutable. On the other hand, it returns a pointer when the returned value is large, to avoid costly copying, or a resource like API client, database connection, or a file, that shouldn't be copied.

A function's return values may be named. If so, they are treated as variables defined at the top of the function. These return value names should be used to document the meaning of the return values. A return statement without arguments returns the named return values.

We generally don't use named return values as they can decrease readability in longer functions.

func getXY() (x, y int) {
    x = 1
    y = 2
    return
}

Function value

Functions are values and can be passed around just like other values.

func compute(fn func(float64, float64) float64) float64 {
    return fn(3, 4)
}

func main() {
    hypot := func(x, y float64) float64 {
        return math.Sqrt(x * x + y * y)
    }
    fmt.Println(compute(hypot))
}

Function closure

A closure is a function value that references variables from outside its body. Each closure is bound to its own external variables.

// fibonacciFactory is a function that returns
// a fibonacci function, which is a closure, that returns
// the next number in the Fibonacci series.
func fibonacciFactory() func() int {
    i, pp, p := -1, 0, 1
    fibonacci := func() int {
        i += 1
        if i == 0 {
            return 0
        } else if i == 1 {
            return 1
        } else {
            c := pp + p
            pp = p
            p = c
            return c
        }
    }
    return fibonacci
}

func main() {
    f := fibonacciFactory()
    for i := 0; i < 10; i++ {
        fmt.Println(f())
    }
}

Method

A method is a function with a special receiver argument. We use the receiver argument to attach the function to a type, usually a struct. Note that the receiver type can't be a pointer and must be defined in the same package so we cannot declare a method with a receiver whose type is int.

There are 2 kinds of receivers, the value receiver and the pointer receiver.

Value receiver method

Pointer receiver method

Generic function

A generic function can handle the same parameter of different types using type parameters.

// Declare a function that takes in a T-typed slice `s` and a T-typed variable `x`. The T type can be any type that fulfills the built-in comparable constraint.
// The comparable constraint ensures that we can use a comparison operator like `==` on values of that type.
func getIndex[T comparable](s []T, x T) int

Function error handling

Go addresses function error with an explicit and imperative approach. In general, a function is expected to return a pair of values. The first one is the result value and the other one is the error value. Go expects us to check and handle the error using if err != nil {} for every function call.

Here are some tips for function error handling tailored for the code structure in this post.

Function panic handling

Unlike an expectable error that can be handled via if err != nil, a panic means something unrecoverable has occurred and the program cannot continue safely.

When a panic happens in any goroutine of the program, Go performs the following steps.

  1. Stop the program execution immediately.
  2. Unwind the call stack from the root cause of the panic all the way to the main function. As each function is exited, any deferred function is still executed normally.
  3. If the panic reaches the main function without being handled, the program crashes and prints the stack trace following a reverse chronological order (error -> main). The concept behind it is giving the most important detail for debugging first.

We can catch the panic during unwinding using defer with recover() and implement panic recovering. If recover() is called during a panic, it captures the panic value and stops the program from crashing, allowing it to resume normal execution. This is often used in web servers to prevent one bad request from taking down the entire service.

func panicFunction() {
    defer func() {
        if r := recover(); r != nil {
            // Recover the function.
        }
    }()
}

Type

Type declaration

Go uses the type <TYPE_NAME> <UNDERLYING_TYPE> declaration format for types.

Struct

A struct is a collection of fields (variables).

Struct declaration

A struct literal denotes a newly allocated struct value by listing the values of its fields.

type Vertex struct {
    X int
    Y int
}

func main() {
    v1 := Vertex{}      // By default X = 0 and Y = 0
    v2 := Vertex{1, 2}  // X = 1 and Y = 2
    v3 := Vertex{X: 3}  // X = 3 and Y = 0
}

Struct field access

Struct fields are accessible via a dot. If we have a struct pointer, using a dot does the dereferencing automatically.

type Vertex struct {
    X int
    Y int
}

func main() {
    v := Vertex{1, 2}
    v.X = 3
    p := &Vertex{4, 5}
    p.Y = 6 // Same as (*p).Y = 6
}

Empty struct type

Empty struct type, struct{}, is a type that takes zero bytes of memory, and is the smallest possible data type in Go. Its type value is struct{}{}.

It allows creating a set from a map with empty struct value type, or a channel purely for signaling.

Interface

An Interface defines a set of method signatures for other types to implement, achieving polymorphism (flexibility). An interface value can hold any concrete type value as long as that concrete type implements those methods.

Under the hood, an interface value is a data structure that contains 2 things.

When calling an interface value method, Go effectively executes the same-named method of the concrete type value through the below steps.

  1. Look up the concrete type.
  2. Look up the concrete type method table.
  3. Call the concrete type method.

This strips away the below optimizations the Go compiler provides.

We can see how the Go compiler applies devirtualization and inlining by building the binary with the -gcflags="-m" flag.

When a method is called many times, the indirection may cause considerable latency overhead. Thus, it may be better to use the concrete type instead of the interface in such case.

If the interface value's data pointer is nil, calling the method will result in a nil pointer dereference runtime error. Therefore, it's a good practice to write code to gracefully handle nil receiver method call.

type I interface {
    M()
}

func describe(i I) {
    fmt.Printf("(%v, %T)\n", i, i)
}

type T1 struct {
    S string
}

func (t T1) M() {
    fmt.Println(t.S)
}

type T2 struct {
    S string
}

func (t *T2) M() {
    fmt.Println(t.S)
}

func main() {
    var i I
    describe(i) // (nil, nil)
    i.M()       // nil pointer dereference runtime error

    i = T1{"Here we Go!"}
    describe(i) // ({"Here we Go!"}, main.T1)
    i.M()       // "Here we Go!"

    i = &T2{"Here we Go!"}
    describe(i) // (&{"Here we Go!"}, *main.T2)
    i.M()       // "Here we Go!"
}

Empty interface type

An interface type that specifies no methods is an empty interface type, interface{}, or any. An empty interface type value, interface{}{}, or any{}, can hold any concrete type value and is used to handle the concrete type value that is unknown at compile time but figured out at runtime.

When we use an any type parameter in a function, Go has to box it and place it on the heap, pressuring Garbage Collection (GC).

Heavier GC load causes higher response latency because GC needs longer execution freezes to clean up the heap memory.

If possible, we should change to use typed parameters or generic type parameters because the types can be resolved at compile time, leading to zero heap allocation per function call. This is especially useful for high-concurrency logging, eventing, or caching.

Interface concrete type assertion

An interface concrete type assertion checks if the interface value holds a value of the specified concrete type. If so, it returns the concrete type value and a true ok value. Otherwise, it returns the zero value of the specified concrete type and a false ok value.

Interface type switch

An interface type switch is a switch statement that uses types as cases, rather than values as cases.

func do(i any) {
    switch v := i.(type) {
    case int:
        fmt.Printf("Twice %v is %v\n", v, v*2)
    case string:
        fmt.Printf("%q is %v bytes long\n", v, len(v))
    default:
        fmt.Printf("I don't know about type %T!\n", v)
    }
}

Error interface

The built-in error type is an interface that requires an Error() string method. we can use errors.New() or fmt.Errorf() to quickly build an error value, or we can develop a custom error type.

When implementing the method for a custom error type, we should use fmt.Sprintf on the error value's internal fields or the type-converted error value, rather than passing the error value back into the fmt function. This makes sure we don't create a recursion that leads to infinite looping.

Any function call, including arithmetic operation, file read/write, and network request, may fail unexpectedly. Therefore, Go makes functions return two values: the result, and an error. This forces the caller to explicitly handle the potential failure immediately.

type MyError struct {
    When time.Time
    What string
}

func (e *MyError) Error() string {
    return fmt.Sprintf("at %v, %s",
        e.When, e.What)
}

func run() error {
    return &MyError{
        time.Now(),
        "it didn't work",
    }
}

func main() {
    if err := run(); err != nil {
        fmt.Println(err)
    }
}

IO Reader interface

The io.Reader interface has a Read method that populates the given byte slice with data and returns the number of bytes populated and an error value. It returns an io.EOF error when the stream ends.

func (T) Read(b []byte) (n int, err error)

Image interface

type Image interface {
    ColorModel() color.Model
    Bounds() Rectangle
    At(x, y int) color.Color
}

Generic type

The generic type helps reduce boilerplate code as one generic value can hold any type value and we can consolidate similar logics into one.

// List represents a singly-linked list that holds any type value.
type List[T any] struct {
    next *List[T]
    val  T
}

Here are some common use cases.

However, the generic type is not suitable for some use cases.

Concurrency

Goroutine

A goroutine is a lightweight (2 KB memory) thread managed by the Go runtime, whereas an OS thread (8 MB memory) is managed by the OS.

Given a small number of OS threads (usually the number of logical cores), the Go runtime can schedule thousands of goroutines to run on these few threads.

If a goroutine blocks because of waiting for I/O, the Go runtime scheduler swaps another runnable goroutine onto the thread.

We can start running a function f(x) inside a goroutine using go f(x), The evaluation of f and x happens in the current goroutine and the execution of f(x) happens in the new goroutine.

When a goroutine panics, it crashes the entire process.

Channel

Goroutines shared the same process memory so memory access must be synchronized. This is why we use channels.

A channel is where goroutines synchronize data with each other.

Channel declaration

We can use make to declare a channel.

ch := make(chan int)

Channel operation

We can send and receive elements through a channel with the channel operator <-.

If we have one sender and that sender has no more elements to send, then it should call close() to shutdown the channel.

If we have multiple senders, we should use a coordinator goroutine to wait for all senders to finish via sync.WaitGroup and then close the channel.

When the passed-in function of sync.WaitGroup.Go() panics, it does not call sync.WaitGroup.Done() but re-panics to prevent the main goroutine from proceeding.

Only a single sender or a coordinator goroutine should close the channel because sending to a closed channel causes a panic but receiving from a closed channel doesn't cause a panic.

A receiver can test whether a channel has been closed by assigning a second parameter ok to the receive expression. If ok is false then the channel is closed.

A receiver can use a for-range loop for i := range c to receive elements from the c channel repeatedly until it is closed. Note that this blocks the receiver goroutine until the channel is closed. This is one of the occasions where we need to close the channel to let the the receiver goroutine be garbage collected. Other than this, we usually don't need to close a channel.

ch <- y         // Send the value of y to channel ch.
x := <-ch       // Receive an element from channel ch and use it to create x.
close(ch)       // A sender closes the ch channel.
z, ok := <-ch   // A receiver checks if the ch channel is closed via the ok variable.

Channel buffer

A buffer is a channel's internal queue. By default the buffer capacity is 0, so a send blocks a goroutine until the element it's sending is received by another goroutine. Same for the read, a read blocks a goroutine until another goroutine sends an element to the channel for it to receive.

We can use the second argument of make to set the buffer capacity. We can get the buffer capacity via cap(ch), and len(ch) for the number of current elements in the buffer.

A send doesn't block a goroutine if the number of current elements in the buffer is less than the buffer capacity. Otherwise, the send blocks the goroutine. A read works in a similar way.

Generally, we don't want the sender to be blocked by a send so we will set the buffer capacity to an exact number or at least an estimate.

Goroutine worker pool

Although goroutine is cheap, it is not free. Creating one goroutine per job can grow the number of concurrent goroutines indefinitely. Therefore, we often adopt the worker pool pattern to limit the number of goroutines.

If the job is CPU-bound, we can set the number of workers to the number of logical CPUs using the default GOMAXPROCS. However, in a containerization environment like Docker or Kubernetes, the Go runtime often sees the host CPU count instead of the container's CPU limit, leading to excessive context switching. Therefore, we should set the worker count to the accurate CPU count using `import _ "go.uber.org/automaxprocs".

If the job is IO-bound, we can use the wait-to-compute ratio worker count = CPU count x (1 + wait time / compute time) as a guidance, and scale the worker count from 2x the CPU count to 10x or even more. When the number gets larger, it starts to get limited by other settings like database connection pool size, file descriptor limit, and upstream service rate limit.

// For containerization uncomment the below line.
// import _ "go.uber.org/automaxprocs
type Result struct {
    JobID int
    WorkerID int
    Value int
    Error error
}

func main() {
    in := make(chan int, 100)
    out := make(chan Result, cap(in))

    numWorkers := runtime.NumCPU() // For containerization use runtime.GOMAXPROCS(0) instead.
    var wg sync.WaitGroup
    for workerID := range numWorkers {
        wg.Go(func() {
            work(workerID, in, out)
        })
    }

    for n := range 100 {
        in <- n
    }
    close(in)

    go func() {
        wg.Wait()
        close(out)
    }()

    for o := range out {
        if o.Error != nil {
            fmt.Println(o.Error.Error())
        }
        fmt.Println(o.Value)
    }
    fmt.Println("finished!")
}

func work(workerID int, in <-chan int, out chan<- int) {
    for i := range in {
        if i % 2 == 0 {
            out <- Result{
                JobID: i,
                WorkerID: workerID,
                Value: i * i,
                Error: nil,
            }
        } else {
            out <- Result{
                JobID: i,
                WorkerID: workerID,
                Error: errors.New("remainder is 1"),
            }
        }
    }
}

We use a coordinator goroutine to close the out channel and thus unblock the main function.

Here we use sync.WaitGroup but we can switch to errgroup from golang.org/x/sync/errgroup if we need the all-or-nothing characteristic.

Goroutine pipeline

We can create a pipeline with goroutines.

func process1(in <-chan int) <-chan int {
    out := make(chan int, cap(in))
    go func () {
        defer close(out)
        for num := range in {
            out <- num
        }
    }()
    return out
}

func process2(in <-chan int) <-chan int {
    out := make(chan int, cap(in))
    go func () {
        defer close(out)
        for num := range in {
            out <- num
        }
    }()
    return out
}

out1 := process1(in)
out2 := process2(out1)

Goroutine select

Goroutine select is used for cases like signal handling alongside receiving values from a channel. The select statement pauses the current goroutine until it can execute one of its communication cases. If multiple are ready, it randomly choose one of them to execute.

We can specify a default case for it to send or receive without blocking when other cases are not ready yet.

If the select statement is in the main function, we can label the outer loop and break that loop when the result channel is closed.

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 4 * time.Second)
	defer cancel()

    results := make(chan int)
    var wg sync.WaitGroup

    for i := range 3 {
        wg.Go(func() {
            multiplyBy10(i, ctx, results)
        })
    }

    go func() {
        wg.Wait()
        close(results)
    }()

    for result := results {
        fmt.Printf("Received result: %d\n", result)
    }
}

func multiplyBy10(i int, ctx context.Context, results chan int) {
    select {
    case <- time.After(2 * time.Second):
        results <- i * 10
    case <- ctx.Done():
        fmt.Printf("Stopped due to context cancellation: %v", ctx.Err())
    default:
        time.Sleep(500 * time.Millisecond)
    }
}

Mutex

We use Mutual-exclusion (Mutex) to ensure the exclusive access of a variable by one goroutine at a time. Go provides the sync.Mutex type and its methods, Lock, and Unlock, to achieve this.

var mu sync.Mutex

go func() {
    mu.Lock()
    defer mu.Unlock()
    process()
}()

Mutex deadlock

A Mutex deadlock is a situation that each one of the two goroutines is waiting for the resource held by the other one.

var (
    mu1 sync.Mutex
    mu2 sync.Mutex
)

go func() {
    mu1.Lock()
    defer mu1.Unlock()
    process()
    mu2.Lock()
    defer mu2.Unlock()
}

go func() {
    mu2.Lock()
    defer mu2.Unlock()
    process()
    mu1.Lock()
    defer mu1.Unlock()
}

One way to prevent the deadlock from happening is to establish a global lock ordering. If every goroutine has to lock mu1 first and then mu2, no deadlock would happen.

Mutex blocking

When our code makes an I/O call and writes the result to a shared resource via mutex, we should only lock the mutex during the write, such that other goroutines are not blocked by the I/O call.

Semaphore

A semaphore is a signaling counter that allows a limited number of goroutines to access a resource. It is useful for rate-limiting API calls or database connections.

Mutex is like a semaphore with size one.

type Semaphore chan struct{}

func NewSemaphore(n int) Semaphore {
    return make(Semaphore, n)
}

func (s Semaphore) Acquire() {
    s <- struct{}{}
}

func (s Semaphore) Release() {
    <-s
}

func main() {
    numConnection := 10
    s := NewSemaphore(numConnection)

    var wg sync.WaitGroup
    for i := range numConnection {
        wg.Go(func() {
            s.Acquire()
            defer s.Release()
            worker(i)
        })
    }
    go func() {
        wg.Wait()
        close(out)
    }()
}

Context

Contexts propagate request-scoped data, cancellation and timeout signals across service calls, preventing resource leaks and allowing for graceful shutdowns.

The creation of a context gives us the context and its cancel function. The cancel function closes the context.Done() channel under the hood and we can use the cancel function to cancel the context.

If we use one of the context with cause types like context.WithCancelCause(), we use context.Cause() for context cancel visibility, otherwise we use ctx.Err() given the context ctx. However, we till have to be aware of nested timeouts and the complexity brought by cross-service propagation.

func work() {
    ctx, cancel := context.WithTimeout(context.Background(), 2 * time.Second) // The timer of ctx starts ticking here.
    defer cancel()

    select {
    case <- time.after(1 * time.Second):
        fmt.Println("Finished work successfully.")
    case <- ctx.Done():
        fmt.Println("Timed out:", ctx.Err())
    }
}

In general we want to handle OS signals like interrupt or terminate via signal.NotifyContext. Note that the cancel function given by signal.NotifyContext can yield back OS signal handling to the OS in addition to its context cancellation functionality. After the OS signal handling is given back to the OS, if we press Ctrl + C the second time, then the OS can kill the process by force. That's why we need to call it manually after <-baseContext.Done() as receiving an OS signal does cancel the context but the OS signal handling is still held by Go instead of the OS.

func main() {
    baseContext, baseCancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
    defer baseCancel()

	mainContext, mainCancel := context.WithCancelCause(baseContext)
	defer mainCancel(errors.New("main context canceled"))

    backgroundService := &domain.BackgroundService{}

    backgroundService.Start(mainContext, mainCancel)

    server := &http.Server{}

    go func() {
        err := server.ListenAndServe()
        if err != nil && !errors.Is(err, http.ErrServerClose) {
            log.Fatal(err)
        }
    }()

    <-baseContext.Done()
    baseCancel()

    shutdownContext, shutdownCancel := context.WithTimeoutCause(context.Background(), 10 * time.Second, errors.New("shutdown context canceled"))
    defer shutdownCancel()
    server.Shutdown(shutdownContext)
}

We use context.WithValue() to pass request-scoped data between middleware. However, multiple layers of middleware, each having a context.WithValue(), can lead to many heap allocations. It's probably better to store data in a request-to-data map inside the pointer receiver handler struct.

Dependency management

Logging

Configuration

Validation

Dependency injection

Command Line Interface (CLI)

Database

Networking

Testing

Downloading remote modules

In each Go file, we use the import keyword to import packages from both local module and remote modules. To download a remote module and record its version in our go.mod. We use the go mod tidy command to add the missing module and also remove unneeded modules.

If two different modules have packages with the same name, we have to use alias for at least one of the package to avoid conflicting import.

import (
    "example.com/module1/x"
    m2x "example.com/module2/x" // Alias example.com/module2/x to m2x.
)

Removing all downloaded modules

Use go clean -modcache to remove all downloaded modules.

Compilation

We can create an example Go program by making a here_we_go.go file. Later on during the compile process, the Go compiler first converts this Go file into an assembly-like internal representation and then compiles the internal representation into a binary.

In order to let the compile succeed, there must be a Go file (here_we_go.go in our case here) that not only belongs to the main package but also includes a main function such that the compiler can locate the entrypoint of the binary.

here_we_go.go 
// Declare this file as part of the main package.
package main

// Import the built-in fmt package for input/output.
import (
    "fmt"
)

// Declare the main function in the main package.
func main() {
    // Print a line saying "Here we Go!".
    fmt.Println("Here we Go!")
}

We can use the go build command to instruct the Go compiler to compile the working directory's Go files into a binary.

# Generate a binary named after the working directory.
go build

# Generate a binary with the specified binary name.
go build -o <BINARY_NAME>

# Generate a binary with the specified main package path.
go build <MAIN_PACKAGE_PATH>

# Generate a binary with garbage collection decisions printed like devirtualization, inlining, and heap escape.
go build -gcflag="-m"

# Generate a binary with garbage collection decisions and reasoning printed like devirtualization, inlining, and heap escape.
go build -gcflag="-m=2"

# Generate a binary such that it is independent from the host's C libraries and can run in a scratch container without the exec format error and no such file or directory error.
CGO_ENABLE=0 go build -o <BINARY_NAME> <MAIN_PACKAGE_PATH>

# Generate a binary for specified OS and hardware.
GOOS=linux GOARCH=amd64 go build -o <BINARY_NAME> <MAIN_PACKAGE_PATH>

Escape

The compiler promotes a value from the stack to the heap if it cannot prove the value's lifetime is limited to the current function scope. Here are some cases where the compiler makes the value escape from the stack to the heap.

Execution

And then we can run the produced binary.

# Run the binary, assuming its name is here-we-go.
./here-we-go

Furthermore, we can combine the compile and run into one command go run.

# Generate an ephemeral binary go-buildxxx inside the temporary directory /tmp, running it, and delete the binary after execution.
go run here_we_go.go

# This also works as long as the working directory contains the Go file that includes the main package's main function.
go run .

If we want to install the binary so we can run it anywhere on the host, export the Go install path and install the binary.

# Locate the Go install path.
go list -f '{{.Target}}'

# Export the path.
export PATH=$PATH:<GO_INSTALL_PATH>

# Compile and install the binary to the Go install path.
go install

Memory management

Common memory bugs

Profile

Go's built-in profiler pprof can start a http server and let we inspect things like heap allocations and how many goroutines are running and which line of code started them. We can set it up with the below steps.

  1. Add import _ "net/http/pprof".
  2. Register the handler function to an existing HTTP ServeMux with http.HandleFunc("/debug/pprof/", pprof.Index). Otherwise, run it in a side server with go func() {http.ListenAndServe("localhost:6060", nil)}().
  3. Visit http://localhost:6060/debug/pprof/goroutine?debug=1.

Benchmark

We can use the built-in benchmark tool to measure performance. First we create the benchmark function.

func BenchmarkBuildURL(b *testing.B) {
    for range b.N {
        buildURL("domain", "/api/users", 8080)
    }
}

Then we run the benchmark.

go test -bench=. -benchmem -count=5 ./...

We can run for heap profile.

go test -bench=. -memprofile=mem.out
go tool pprof mem.out

Or run for CPU profile.

go test -bench=. -cpuprofile=cpu.out
go tool pprof cpu.out

Production best practice

  1. Profile
  2. Identify bottleneck from heap escape and goroutine analysis
  3. Improve and benchmark

Error handling

Asynchronous processing

Memory

Containerization

Build tag

We can use build tags to mark the source file that should only be included during a local build.

//go:build local

// The tag has to be at the top of the file and the second line must be a blank line.
package main

import _ "net/http/pprof"

We can include the tag in the build command.

go build -tags="local" -o <BINARY_NAME> <MAIN_PACKAGE_PATH>

Termination handling

Docker or Kubernetes usually send the SIGTERM signal before killing a container with SIGKILL, so we should handle the SIGTERM signal otherwise in-flight requests will be dropped.

Kubernetes termination grace period

In the application pod manifest, set spec.terminationGracePeriodSeconds to a number slightly greater than the shutdown timeout duration we configure in the application shutdown handling logic.

Health check

We can add both the liveness and readiness health check endpoint to the HTTP server.

func main() {
    mux := http.NewServeMux()

    mux.HandleFunc("GET /healthz/live", liveHandler)
    mux.HandleFunc("GET /healthz/ready", readyHandler)

    httpServer := &http.Server{Addr: ":8080", Handler: mux}

    database := &database.PostgreSQL{}

    restServer := &server.RESTServer{HTTPServer: httpServer, Database: database}
}

func (s *RESTServer) liveHandler(w http.ResponseWriter, r *http.Request){
    w.WriteHeader(http.StatusOK)
}

func (s *RESTServer) readyHandler(w http.ResponseWriter, r *http.Request){
    ctx, cancel := context.WithTimeout(r.Context(), 5 * time.Second)
    defer cancel()

    err := s.Database.PingContext(ctx)
    if err != nil {
        http.Error(w, "database is not ready", http.StatusServiceUnavailable)
        return
    }

    w.WriteHeader(http.StatusOK)
}

Docker health check

Docker only supports single health check so we choose the liveness one. We need to build a health check binary and configure Docker to use it for health checks in the Dockerfile.

func main() {
    client := &http.Client{Timeout: 5 * time.Second}
    response, err := client.Get("http://localhost:8080/health/live")
    if err != nil || response.StatusCode != 200 {
        os.Exit(1)
    }
    os.Exit(0)
}

Kubernetes health check

We configure the liveness and readiness probe endpoint in the deployment manifest such that Kubernetes can scrape them. See more in the Kubernetes post.

Dockerfile

Here is an example Dockerfile.

FROM golang:1.26-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLE=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o server ./cmd/server

FROM scratch
COPY --from=builder /app/server /server
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=30s --start-period=10s retries=3 \
    CMD ["./healthcheck"]
ENTRYPOINT ["/server"]

Kubernetes Deployment

Here is a deployment manifest example.

apiVersion: apps/v1
kind: Deployment
metadata:
    name: myapp
    labels:
        app: myapp
spec:
    selector:
        matchLabels:
            app: myapp
    replicas: 3
    strategy:
        type: RollingUpdate
        rollingUpdate:
            maxUnavailable: 0
            maxSurge: 1
    template:
        metadata:
            labels:
                app: myapp
        spec:
            terminationGracePeriodSeconds: 30
            containers:
                -   name: myapp
                    image: myrepo/myapp@hash
                    env:
                        -   name: LOG_LEVEL
                            valueFrom:
                                configMapKeyRef:
                                    name: myapp
                                    key: LOG_LEVEL
                        -   name: DB_DSN
                            valueFrom:
                                secretKeyRef:
                                    name: myapp
                                    key: db-dsn
                    ports:
                        -   containerPort: 8080
                    resources:
                        requests:
                            cpu: "100m"
                            memory: "64Mi"
                        limits:
                            cpu: "100m"
                            memory: "64Mi"
                    livenessProbe:
                        httpGet:
                            path: /healthz/live
                            port: 8080
                        initialDelaySeconds: 10
                        periodSeconds: 30
                    readinessProbe:
                        httpGet:
                            path: /healthz/ready
                            port: 8080
                        initialDelaySeconds: 20
                        periodSeconds: 30

Kubernetes ConfigMap and Secret

We use Kubernetes ConfigMap and Secret to store the application settings.

apiVersion: v1
kind: ConfigMap
metadata:
    name: myapp
data:
    LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
    name: myapp
type: Opaque
stringData:
    db-dsn: "postgres://user:password@host:5432/myapp"

Inside our code, we use a config struct and os.Getenv() to read from environment variables.

type Config struct {
    LogLevel string
    DBDSN string
}

func LoadConfig() Config {
    return Config{
        LogLevel: os.getEnv("LOG_LEVEL", "info"),
        DBDSN: os.getEnv("DB_DSN"),
    }
}

Kubernetes resource request and limit

See more in the Kubernetes post.

Adoption challenge

References

←Previous Next→
Copyright © Te-Yuan Liu