Debugging · observability

strace — the Linux syscall tracer

strace prints every syscall a process makes, with its arguments, return value, and (optionally) timing. It is the fastest way to answer "what is this program actually doing" — and almost always the first tool to reach for when a binary misbehaves and you don't have its source.

How it works

strace is a thin wrapper around the kernel's ptrace(2) facility. It attaches to the target process, asks the kernel to stop it before and after every syscall (PTRACE_SYSCALL), reads the registers to decode the call, formats them human-readably, and lets the target resume. Modern kernels expose PTRACE_GET_SYSCALL_INFO so strace can do this without needing per-architecture register knowledge.

Because it rides on ptrace, strace inherits its security model: Yama ptrace_scope and CAP_SYS_PTRACE apply. See ptrace(2) for the full mechanism.

Flags you will actually use

-f: Follow forks. Without this, strace only traces the leader; with it, every child via fork()/clone() is traced too. Almost always wanted.
-p PID: Attach to an already-running process by PID instead of spawning a new one.
-e trace=set: Filter by syscall name or set. Examples: -e trace=openat,read,write or -e trace=%file or -e trace=%network.
-e read=set / write=set: Show the contents of reads/writes on the listed fds (e.g. -e write=1,2 dumps stdout/stderr writes).
-s N: Maximum length to print for string buffers. Default 32; bump to -s 4096 to see full payloads.
-c: Skip the line-by-line trace and print a summary table at the end (syscall, count, time, errors) — the fastest way to find where a program spends its time.
-y / -yy: Resolve file descriptors to paths (-y) or to socket endpoints (-yy). Invaluable when reading a noisy trace.
-o file: Write the trace to a file instead of stderr. Combine with -ff to split per-PID files.
-k: Print a userspace stack backtrace for each syscall (requires libdw). Heavy but unmatched for tracking down the call site.
-t / -tt / -ttt: Add timestamps: -t is HH:MM:SS, -tt adds microseconds, -ttt is seconds-since-epoch (machine-parseable).

Recipes

Find the file the program can't open

strace -f -e trace=openat,access -e status=failed ./app 2>&1 | tail

See which syscalls dominate run time

strace -c -f ./app

Attach to nginx and watch network syscalls

strace -f -p $(pidof nginx) -e trace=network -s 512 -t

Dump what the program writes to stderr

strace -e write=2 -s 8192 ./app | grep -A2 'write(2,'

Popular syscalls people trace

Click through for the reference page on each — argument format, common errno values, kernel-history notes.

When strace isn't the right tool

strace incurs a context-switch per syscall, which can be a 10–100× slowdown on syscall-heavy workloads — don't use it on production traffic. For sampling, use perf trace (lighter than strace, same data) or bpftrace tracepoint:syscalls:sys_enter_*. For long-running observability, eBPF-based tools (Falco, Tracee, sysdig) are the modern answer. For library calls rather than syscalls, ltrace is the sibling tool.