Go GC glossary

GC CPU limiter

A safeguard added in Go 1.19 that caps the CPU time the collector can take, around 50 percent, when the memory limit is reached.

With a memory limit close to the live heap, the collector could run continuously and starve the program, a death spiral. The limiter tracks the fraction of CPU used by GC and assists in a sliding window and, when it exceeds 50 percent, lets the program allocate over the limit instead.

The memory limit is therefore soft: the program may exceed GOMEMLIMIT temporarily rather than stop making progress. If gctrace shows back-to-back cycles with the heap near the limit, the live heap is too large for the limit: raise the limit or reduce memory use.

Official documentation: A Guide to the Go Garbage Collector

gctrace

A line the Go runtime writes to standard error at each collection when GODEBUG=gctrace=1 is set. GC Doctor reads these lines.

A line starts with the cycle number and the time since program start, then the share of time spent in GC. It gives the wall-clock times of the three phases (stop-the-world sweep termination, concurrent mark, stop-the-world mark termination), the CPU times (assist, background, idle), the heap size at the start, at the end and live, the heap goal, the sizes of stacks and globals, and the number of processors.

The reference for every field is the documentation of the runtime package. GODEBUG=scavtrace=1 adds the scavenger lines.

Official documentation: Package runtime: environment variables (GODEBUG gctrace, scavtrace)

GOGC

The main Go GC tuning knob: the percentage of the live heap that may be allocated before the next cycle starts. The default is 100.

With GOGC=100, the heap goal is twice the live heap (plus stacks and globals, since Go 1.18): a program with 100 MiB of live heap starts a cycle at about 200 MiB. Doubling GOGC doubles the extra memory and roughly halves the CPU cost of GC, and the other way round. Set it with the GOGC environment variable or debug.SetGCPercent. GOGC=off disables the collector, which is only reasonable together with a memory limit.

In gctrace, compare the heap sizes (at start, at end, live) with the goal. A program with a small, stable live heap and plenty of spare memory can raise GOGC to spend less CPU on GC. A program in a container close to its memory limit should rather use GOMEMLIMIT.

Official documentation: A Guide to the Go Garbage Collector

GOMEMLIMIT

A soft limit on the total memory used by the Go runtime, available since Go 1.19. The collector works harder as usage approaches it.

The limit covers the whole runtime: heap, stacks and runtime structures, not memory returned to the OS. When the total approaches it, the collector runs more often than GOGC alone would require, trading CPU for memory. The limit is soft: if the live data really exceeds it, the program keeps running, and the GC CPU limiter prevents the collector from taking over the CPU.

A typical setup in a container is a limit 5 to 10 percent below the container memory, with GOGC unchanged or off. Set it with the GOMEMLIMIT environment variable (for example 900MiB) or debug.SetMemoryLimit. When the limit binds, cycles become frequent and the share of CPU spent in GC rises.

Official documentation: A Guide to the Go Garbage Collector

Goroutine

A lightweight thread managed by the Go runtime, cheap enough to run in the thousands or millions.

Goroutines start with a stack of a few kilobytes that grows and shrinks, and the scheduler multiplexes them onto a small number of OS threads. For the garbage collector they matter in three ways. Their stacks are roots that must be scanned. Their stacks count toward the heap goal (since Go 1.18 the goal includes stacks and globals). And a goroutine that allocates during a cycle may be drafted to help with marking (mark assist).

Leaked goroutines, blocked forever on a channel, retain everything their stacks reference: a common cause of steadily growing memory.

Heap goal

The total heap size at which the Go runtime wants the current collection to have finished.

At the end of each cycle the runtime computes the goal from the live heap, the stacks, the globals and GOGC (goal = live + (live + stacks + globals) x GOGC / 100), or from the memory limit if it is lower. The pacer starts marking early enough to finish as the heap reaches the goal: earlier when the allocation rate is high or marking is slow.

In gctrace the goal appears as the MB goal field. If the heap routinely overshoots it, the pacer is losing the race: expect mark assists and little margin.

Official documentation: A Guide to the Go Garbage Collector

Mark assist

Marking work done by an allocating goroutine itself, when the program allocates faster than the background workers can mark.

During a cycle the runtime tries to finish marking before the heap reaches the goal. Dedicated background workers use 25 percent of the processors. If a goroutine allocates faster than they progress, it is charged and must do a proportional amount of marking before its allocation can proceed: an assist.

Assists show in the gctrace line as the first number of the CPU time triple (assist, background, idle) and as extra latency for requests that allocate heavily. Persistent assists suggest that the heap goal is too close to the live heap (raise GOGC or the memory limit) or that the allocation rate is too high.

Official documentation: A Guide to the Go Garbage Collector

Scavenger

The background task that returns unused heap memory to the operating system.

After a heap peak, the runtime keeps freed memory for reuse. The scavenger releases it gradually to the OS so that the resident memory of the process follows the real use of the program. It paces itself to use about 1 percent of the CPU time and keeps slightly more memory than the heap goal; a memory limit makes it more aggressive.

With GODEBUG=scavtrace=1 the runtime prints scavenger activity, which GC Doctor reads. If the resident memory stays high after the load goes away, look at the scavenger and at how the OS accounts for released memory.

Official documentation: Package runtime: environment variables (GODEBUG gctrace, scavtrace)

Stop the world (Go)

The two short phases of a Go cycle where all goroutines are stopped: sweep termination and mark termination.

A Go cycle has two stop-the-world phases. Sweep termination, at the start, finishes the previous cycle and turns on the write barrier. Mark termination, at the end, finishes marking and turns the barrier off. Everything in between is concurrent. They are the first and third values of the wall-clock triple in gctrace, typically tens to hundreds of microseconds.

A long stop-the-world phase usually means that some goroutine did not reach a safe point promptly (a tight loop without function calls, a classic cause before asynchronous preemption arrived in Go 1.14) or that the machine was overloaded. During the concurrent phase the program keeps running but shares the CPUs with the collector.

Write barrier (Go)

A small piece of code the compiler adds to pointer writes, enabled only while the collector is marking.

Go marks concurrently, while the program runs. If the program stored a pointer to an unmarked object into an object already scanned, the collector would miss it and free a live object. The write barrier, a hybrid of deletion and insertion barriers, shades the objects involved so that this cannot happen.

Outside the marking phase the barrier is off, so its cost applies only during a cycle. This is one reason why CPU use and allocation latency are higher while a cycle runs.