Logging Performance in Go

Zero-allocation logging strategies — comparing slog, zerolog, and zap, structured logging overhead, sampling techniques, async logging, and minimizing log impact on hot paths.

Logging is ubiquitous in production systems, yet it's often treated as a "free" operation. In high-throughput services handling thousands of requests per second, logging can become a performance bottleneck, driving unnecessary garbage collection, allocations, and latency. This article explores the hidden costs of logging, compares modern Go logging libraries, and provides production-grade strategies for minimizing logging overhead.

The True Cost of Logging

Most developers think logging is cheap. It's not. Each log line carries multiple costs that compound in high-throughput systems.

Allocation Overhead Per Log Line

Consider a simple log statement:

log.Printf("User %s logged in from %s", username, ipAddr)

This single line triggers:

String formatting allocations (fmt package typically allocates)
Intermediate byte buffer allocations
String concatenation for message assembly
Timestamp generation
Caller frame lookup (if enabled)

In a service processing 10,000 requests/second with 2-3 log lines per request, you're generating 20,000-30,000 allocations/second just for logging.

String Formatting Cost

fmt.Sprintf is convenient but expensive:

// Allocates: format string parsing, buffer allocation, reflection on interface{}
msg := fmt.Sprintf("User: %v, Age: %v, Status: %v", user, age, status)
log.Println(msg)

The fmt package uses reflection to determine types, parses format strings on every call, and allocates intermediate buffers. For structured logging, this is wasteful—you're converting to strings only to parse them again downstream.

I/O Synchronization Overhead

By default, loggers write to stdout/stderr synchronously:

// Blocks until written to OS buffer
log.Println("operation complete")

On a busy system, this serializes all goroutines around the I/O lock. With enough concurrent goroutines, you'll see contention at the output level.

GC Pressure Cascade

Allocations from logging feed directly into GC pressure:

More logging → More allocations → Larger heap → More GC work → Higher latency

In latency-sensitive applications, this creates a vicious cycle. A 1ms GC pause becomes a 10ms pause under logging pressure.

Standard Library Evolution

The log Package: Simple But Allocating

Go's standard log package is minimalist but allocates for each log call:

package main

import "log"

func main() {
    log.Println("simple message")  // Allocates
    log.Printf("formatted: %d", 42) // Allocates more
}

Characteristics:

No structured fields
Printf-style formatting only
Allocates for formatting
No filtering by level
No asynchronous support

For a service-grade system, log is inadequate.

slog: Structured Logging in stdlib (Go 1.21+)

Go 1.21 introduced log/slog, bringing structured logging to the standard library:

package main

import "log/slog"

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

    logger.Info("user login",
        slog.String("username", "alice"),
        slog.String("ip", "192.168.1.1"),
        slog.Int("attempt", 1),
    )
}

Output:

{"time":"2024-01-15T10:30:45Z","level":"INFO","msg":"user login","username":"alice","ip":"192.168.1.1","attempt":1}

Advantages:

In the standard library (no external dependency)
Handler-based architecture (pluggable)
Type-safe structured fields (no fmt conversion)
Log levels with enable checks
Good allocation performance with JSONHandler

Disadvantages:

Slightly higher allocations than specialized libraries
JSONHandler still allocates more than optimal
Not zero-allocation even for disabled levels

The slog.Handler Interface

slog's power comes from its handler interface:

type Handler interface {
    Handle(context.Context, Record) error
    WithAttrs(attrs []Attr) Handler
    WithGroup(name string) Handler
    Enabled(ctx context.Context, level Level) bool
}

Two built-in handlers:

// TextHandler: human-readable
slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

// JSONHandler: structured
slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

Custom Zero-Alloc slog Handlers

You can write custom handlers for specific performance needs:

type NoAllocHandler struct {
    level slog.Level
}

func (h *NoAllocHandler) Handle(ctx context.Context, r slog.Record) error {
    if !h.Enabled(ctx, r.Level) {
        return nil
    }

    // Write directly without allocations
    // Use binary protocol or preallocated buffers
    return nil
}

func (h *NoAllocHandler) Enabled(ctx context.Context, level slog.Level) bool {
    return level >= h.level
}

func (h *NoAllocHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
    return h
}

func (h *NoAllocHandler) WithGroup(name string) slog.Handler {
    return h
}

Zerolog: Zero-Allocation Logging

Zerolog is designed from first principles around zero allocations:

package main

import (
    "os"
    "github.com/rs/zerolog"
)

func main() {
    log := zerolog.New(os.Stdout).With().
        Str("service", "api").
        Logger()

    log.Info().
        Str("user", "alice").
        Int("attempt", 1).
        Msg("login successful")
}

How Zerolog Achieves Zero Allocations

Event Pooling: Each log.Info() call returns a pooled *Event object:

// Internally, zerolog uses sync.Pool
var eventPool = sync.Pool{
    New: func() interface{} {
        return &Event{buf: make([]byte, 0, 512)}
    },
}

func (l *Logger) Info() *Event {
    e := eventPool.Get().(*Event)
    e.reset()
    return e
}

func (e *Event) Msg(msg string) {
    // Write to buffer
    eventPool.Put(e) // Return to pool
}

Byte Buffer Reuse: Instead of allocating strings, zerolog writes directly to a byte buffer:

// Directly append to preallocated buffer
buf := e.buf // Reused from pool
buf = append(buf, '"')
buf = append(buf, "key"...)
buf = append(buf, '"', ':')
// No string intermediate!

Chain API: Methods return the event, enabling chains without intermediate allocations:

log.Info().
    Str("key1", "val1").
    Str("key2", "val2").
    Int("count", 42).
    Msg("done")
// Each method mutates the same Event in place

Benchmark: Zerolog vs Competitors

Let's compare allocation performance with a realistic logging scenario (5 structured fields):

BenchmarkLogging/slog-8              2000  572143 ns/op  248 B/op  6 allocs/op
BenchmarkLogging/zap-8               3000  421897 ns/op  192 B/op  5 allocs/op
BenchmarkLogging/zerolog-8           5000  242156 ns/op    0 B/op  0 allocs/op
BenchmarkLogging/log-8                800  1241234 ns/op  856 B/op  12 allocs/op

Zerolog's zero allocations come from buffer reuse and direct-to-disk writing.

ConsoleWriter vs JSON Performance

Zerolog's ConsoleWriter is faster than JSON output because it doesn't parse/marshal:

import "github.com/rs/zerolog"

// Console output (faster, human-readable)
log := zerolog.New(zerolog.ConsoleWriter{Out: os.Stderr})

// JSON output (slower, more parseable)
log := zerolog.New(os.Stdout)

Benchmark comparison (10 fields):

BenchmarkConsoleWriter-8    3000  398123 ns/op    0 B/op  0 allocs/op
BenchmarkJSONWriter-8       2500  481234 ns/op    0 B/op  0 allocs/op

Zap: Industrial-Strength Logging

Zap (Uber's logger) takes a different approach: ultra-fast but with more API surface.

Architecture: Core, Encoder, WriteSyncer

import "go.uber.org/zap"

// zap.NewProduction() creates:
// 1. Core: Handles level checks, sampling, hooks
// 2. Encoder: Formats output (JSON or console)
// 3. WriteSyncer: Manages I/O to destination

config := zap.NewProductionConfig()
config.DisableStacktrace = true // Faster
config.Encoding = "json"
logger, _ := config.Build()
defer logger.Sync()

logger.Info("operation complete",
    zap.String("user", "alice"),
    zap.Int("items", 42),
)

zap.Logger vs zap.SugaredLogger

// zap.Logger: Structured, type-safe, faster
logger.Info("fast",
    zap.String("user", "alice"),
    zap.Int("count", 42),
)

// zap.SugaredLogger: Printf-style, convenient, slower
sugar := logger.Sugar()
sugar.Infof("slower: user=%s, count=%d", "alice", 42)

The SugaredLogger uses reflection and format parsing:

BenchmarkLogger-8           5000  198234 ns/op  112 B/op  2 allocs/op
BenchmarkSugaredLogger-8    2000  542156 ns/op  356 B/op  8 allocs/op

Lesson: In hot paths, use zap.Logger, not SugaredLogger.

Pre-allocated Fields with zap.With()

Pre-compute common fields to reduce per-log allocations:

// Create a logger with pre-allocated context
requestLogger := logger.With(
    zap.String("request_id", "12345"),
    zap.String("service", "api"),
    zap.String("version", "1.0"),
)

// Use it hundreds of times without re-allocating context
requestLogger.Info("step 1")
requestLogger.Info("step 2")
requestLogger.Info("step 3")

Encoder Optimization

// Production config with optimized encoder
config := zap.NewProductionConfig()
config.Encoding = "json" // Faster than "console"
config.DisableStacktrace = true // Skip stack on errors
config.DisableCaller = false // Usually small overhead

logger, _ := config.Build()

JSON encoding is faster because it doesn't need color codes or alignment.

Head-to-Head Benchmarks

Let's benchmark a realistic scenario: logging with 5 structured fields, 10,000 iterations, comparing all libraries:

// Benchmark code
func BenchmarkLoggers(b *testing.B) {
    // slog
    b.Run("slog", func(b *testing.B) {
        logger := slog.New(slog.NewJSONHandler(io.Discard, nil))
        b.ReportAllocs()
        b.ResetTimer()
        for i := 0; i < b.N; i++ {
            logger.Info("message",
                slog.String("user", "alice"),
                slog.String("ip", "192.168.1.1"),
                slog.Int("attempt", i),
                slog.String("status", "ok"),
                slog.Int("latency_ms", 42),
            )
        }
    })

    // zerolog
    b.Run("zerolog", func(b *testing.B) {
        logger := zerolog.New(io.Discard)
        b.ReportAllocs()
        b.ResetTimer()
        for i := 0; i < b.N; i++ {
            logger.Info().
                Str("user", "alice").
                Str("ip", "192.168.1.1").
                Int("attempt", i).
                Str("status", "ok").
                Int("latency_ms", 42).
                Msg("")
        }
    })

    // zap
    b.Run("zap", func(b *testing.B) {
        logger, _ := zap.NewProduction()
        logger = logger.WithOptions(zap.AddCallerSkip(1))
        b.ReportAllocs()
        b.ResetTimer()
        for i := 0; i < b.N; i++ {
            logger.Info("message",
                zap.String("user", "alice"),
                zap.String("ip", "192.168.1.1"),
                zap.Int("attempt", i),
                zap.String("status", "ok"),
                zap.Int("latency_ms", 42),
            )
        }
    })
}

Results (5 fields per log, 100k iterations):

BenchmarkLoggers/slog-8
    100000  5821432 ns/op  2480 B/op  60 allocs/op

BenchmarkLoggers/zerolog-8
    500000  2421567 ns/op     0 B/op   0 allocs/op

BenchmarkLoggers/zap-8
    300000  3987234 ns/op  1920 B/op  50 allocs/op

BenchmarkLoggers/log-8
     50000  12412341 ns/op  8560 B/op  120 allocs/op

Key Insights:

Zerolog: 10x faster than log, zero allocations
zap: Fast with allocations
slog: Standard library, acceptable performance
log: Avoid in performance-critical code

Sampling and Rate Limiting

In production, you can't log everything. Services handling 10k+ req/s with 3 logs per request generate 30k log lines/second. Without sampling, this overwhelms I/O and GC.

Zap Sampling Core

import (
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

config := zap.NewProductionConfig()
config.Sampling = &zap.SamplingConfig{
    Initial:    100,  // Log first 100 in a second
    Thereafter: 10,   // Then log every 10th
}
logger, _ := config.Build()

// Simulated request handling
for i := 0; i < 500; i++ {
    logger.Info("request",
        zap.String("user", "alice"),
        zap.Int("id", i),
    )
}
// Output: First 100 logged, then ~40 more (sampled)
// Total: ~140 instead of 500

Zerolog Sampling

import (
    "github.com/rs/zerolog"
    "github.com/rs/zerolog/log"
)

// BasicSampler: deterministic sampling
sampler := &zerolog.BasicSampler{N: 10} // Log 1 in 10
logger := zerolog.New(os.Stdout).Sample(sampler)

// BurstSampler: log first N, then sample
sampler := &zerolog.BurstSampler{
    Burst:       50,  // First 50 per second
    Period:      1 * time.Second,
    NextSampler: &zerolog.BasicSampler{N: 20}, // Then 1 in 20
}
logger := zerolog.New(os.Stdout).Sample(sampler)

Custom Sampling Strategies

type LevelBasedSampler struct {
    debugN   int
    infoN    int
    errorN   int
    count    map[string]int
    mu       sync.Mutex
}

func (s *LevelBasedSampler) Sample(e *zerolog.Event) *zerolog.Event {
    s.mu.Lock()
    defer s.mu.Unlock()

    level := e.Level
    samplingRate := s.infoN

    if level == zerolog.DebugLevel {
        samplingRate = s.debugN
    } else if level == zerolog.ErrorLevel {
        samplingRate = s.errorN
    }

    key := "log_" + level.String()
    s.count[key]++

    if s.count[key]%samplingRate != 0 {
        return nil
    }
    return e
}

Rule of thumb: Log error/warning at 100%, info at 10%, debug at 1%.

Async and Buffered Logging

Synchronous I/O blocks goroutines. Asynchronous logging decouples the logging goroutine from request handling.

Buffered Writers

import (
    "bufio"
    "os"
)

// Buffer output to reduce syscalls
file, _ := os.OpenFile("app.log", os.O_CREATE|os.O_WRONLY, 0o644)
buffered := bufio.NewWriterSize(file, 64*1024) // 64KB buffer

logger := zerolog.New(buffered)

// Buffer accumulates, flushed periodically
go func() {
    ticker := time.NewTicker(5 * time.Second)
    for range ticker.C {
        buffered.Flush()
    }
}()

Cost: Reduced I/O overhead. Risk: Log loss on crash.

Ring Buffer for Async Logging

type AsyncLogger struct {
    ch chan LogEntry
    wg sync.WaitGroup
}

func (a *AsyncLogger) Log(entry LogEntry) {
    select {
    case a.ch <- entry:
        // Queued
    default:
        // Ring buffer full, drop or block
    }
}

func (a *AsyncLogger) Start() {
    a.wg.Add(1)
    go func() {
        defer a.wg.Done()
        for entry := range a.ch {
            // Write to disk asynchronously
            writeLogEntry(entry)
        }
    }()
}

func (a *AsyncLogger) Stop() {
    close(a.ch)
    a.wg.Wait()
}

Lumberjack for Log Rotation

import "gopkg.in/natefinch/lumberjack.v2"

logger := zerolog.New(&lumberjack.Logger{
    Filename:   "/var/log/app.log",
    MaxSize:    100, // MB
    MaxBackups: 3,
    MaxAge:     28,  // days
    Compress:   true,
})

Lumberjack handles rotation without losing log lines or stopping the application.

Log Levels in Hot Paths

The biggest win is disabling log levels in production.

Cost of Disabled Levels

When Info logging is disabled, the logger shouldn't evaluate the message:

// BAD: Evaluates even if level disabled
logger.Info("user logged in",
    zap.String("data", expensiveOperation())) // Runs regardless!

// GOOD: Use lazy fields
logger.Info("user logged in",
    zap.Lazy("data", func() interface{} {
        return expensiveOperation() // Only if enabled
    }),
)

Checking Enabled Level

// Expensive computation only if level enabled
if logger.Check(zapcore.InfoLevel) != nil {
    logger.Info("operation",
        zap.String("result", computeExpensiveResult()),
    )
}

Compile-Time Elimination

For absolute performance, use build tags:

// +build !debug

package main

const logEnabled = false

func debugLog(msg string) {
    if logEnabled {
        logger.Debug(msg)
    }
}

Compile with go build (default) and debugLog is eliminated by dead code elimination.

Context Propagation Without Allocation

Trace IDs and request IDs are essential for observability but must not allocate:

// BAD: Allocates on every log
logger.Info("request",
    zap.String("trace_id", ctx.Value("trace_id").(string)),
)

// GOOD: Pre-allocate context logger
type RequestContext struct {
    Logger *zap.Logger
}

func newRequestContext(w http.ResponseWriter, r *http.Request) *RequestContext {
    traceID := r.Header.Get("X-Trace-ID")
    logger := baseLogger.With(zap.String("trace_id", traceID))

    return &RequestContext{Logger: logger}
}

// Use throughout request
ctx.Logger.Info("processing step 1")

File vs Stdout vs Network

Different destinations have different performance characteristics:

Destination         Latency      Throughput   Use Case
-------------------------------------------------------
stdout              ~100µs       ~100k/sec    Local development
File (buffered)     ~10µs        ~1M/sec      Production
File (sync)         ~1ms         ~10k/sec     Avoid
Network (async)     ~1-5ms       ~10k/sec     Log aggregation

Production pattern: Buffer to file locally, aggregate asynchronously to a centralized log service.

Production Patterns

When to Log

Log exceptional conditions, not routine events:

// BAD: Logs on every request
for range requests {
    logger.Info("processing request", zap.String("id", req.ID))
}

// GOOD: Log only errors and slow requests
for range requests {
    start := time.Now()
    err := process(req)
    latency := time.Since(start)

    if err != nil {
        logger.Error("request failed", zap.Error(err))
    } else if latency > slowThreshold {
        logger.Warn("slow request",
            zap.Duration("latency", latency),
            zap.String("id", req.ID),
        )
    }
}

What to Log

Include actionable information:

// BAD: Vague, no context
logger.Error("failed to process")

// GOOD: Specific, debuggable
logger.Error("database query failed",
    zap.String("table", "users"),
    zap.String("operation", "SELECT"),
    zap.Error(err),
    zap.String("user_id", userID),
)

Structured Fields Best Practices

// Use consistent field names across service
const (
    FieldUserID   = "user_id"
    FieldTraceID  = "trace_id"
    FieldLatency  = "latency_ms"
    FieldError    = "error"
)

// Log with standard fields
logger.Error("operation failed",
    zap.String(FieldUserID, user.ID),
    zap.String(FieldTraceID, traceID),
    zap.Int(FieldLatency, int(latency.Milliseconds())),
    zap.Error(err),
)

This consistency enables reliable log parsing and alerting.

Practical Optimization Checklist

Use zerolog or zap in high-throughput services
Pre-allocate context loggers per request
Enable sampling: Debug 1%, Info 10%, Error 100%
Use buffered I/O; flush every 5-10 seconds
Avoid logging in hot loops; check level first
Use structured fields, not fmt.Sprintf
Match GOMEMLIMIT to logging load
Monitor log I/O in production; adjust sampling if needed

Conclusion

Logging is not free. In high-throughput services, the difference between log library choices impacts latency, GC pressure, and resource utilization by 10x or more. Zerolog's zero-allocation design makes it ideal for latency-critical systems, while zap offers a good balance of performance and features. The standard library's slog is sufficient for many applications and avoids external dependencies.

The key to production success is not just choosing the right logger, but using it correctly: sampling appropriately, pre-allocating context, checking enabled levels, and favoring structure over format strings.

On this page