Debugging eBPF verifier: Best Practices for High-Performance Kernel Programming

Introduction

eBPF (Extended Berkeley Packet Filter) is a powerful technology that enables safe and efficient execution of custom programs within the Linux kernel. It has been widely adopted for networking, observability, and security use cases. However, eBPF programs are subject to various constraints to ensure performance and security. One of the key limitations is the maximum instruction count, which can pose challenges when developing complex eBPF applications.

The eBPF Instruction Limit

As of recent Linux kernel versions, eBPF programs are limited to 1,000,000 instructions per execution path. This limit is enforced by the eBPF verifier, which ensures that programs are safe and do not lead to unbounded execution, infinite loops, or excessive CPU consumption.

Historically, earlier kernel versions imposed a stricter limit of 4,096 instructions per program. This was later increased to 1,000,000 to accommodate more complex use cases such as advanced tracing, security monitoring, and in-kernel processing.

Why Does This Limitation Exist?

The instruction count limitation exists primarily for the following reasons:

1. Performance and Latency: Since eBPF programs execute within the kernel, excessive computation could lead to increased latency or resource exhaustion.
2. Safety and Termination Guarantee: The kernel must ensure that eBPF programs will always terminate and not enter infinite loops, which could crash or hang the system.
3. Verifier Complexity: The eBPF verifier performs static analysis of all possible execution paths to determine whether the program is safe. The instruction limit helps keep this analysis computationally feasible.

Implications for Developers

For developers working with eBPF, hitting the instruction limit can be a significant challenge, especially when implementing complex logic. Here are some common approaches to mitigate this limitation:

1. Optimize the eBPF Program

• Reduce unnecessary instructions and redundant computations.
• Use compiler optimizations (-O2 or -O3 in Clang) to generate efficient bytecode.
• Avoid deep function call chains that expand into excessive instructions.

2. Use Tail Calls

• eBPF supports tail calls, which allow one program to jump into another without returning. This helps break large programs into multiple smaller ones, avoiding the instruction limit per execution path.
• However, tail calls are limited to 33 chained calls per execution to prevent infinite recursion.

3. Leverage Maps for Complex Logic

• Instead of performing expensive computations in the eBPF program, store data in BPF maps and perform processing in user space.
• Hash maps, array maps, and ring buffers can help offload heavy operations.

4. Minimize Loop Iterations

• eBPF supports bounded loops, but their iteration count is analyzed by the verifier to prevent unbounded execution.
• Reducing loop complexity and iterations helps stay within the instruction limit.

5. Use Helper Functions

• eBPF provides a set of helper functions that execute optimized kernel operations (e.g., accessing packet data, retrieving timestamps, etc.).
• Using these helpers instead of custom logic can save valuable instructions.

6. Measuring eBPF Verifier Instruction Count Using eBPF

• The eBPF verifier only provides feedback when a program reaches the 1,000,000 instruction limit, preventing successful loading. However, developers may still want to track the instruction count of successfully loaded programs to analyze the impact of code changes.
• To achieve this, a simple eBPF program can attach kprobe and kretprobe to the do_check function (static int do_check(struct bpf_verifier_env *env)).
• The kprobe captures the pointer to env, and the kretprobe extracts and prints relevant details, including:
  
  • Instruction count: env->insn_processed
  • Program name: env->prog->aux->name
  • Program length: env->prog->len
• With this approach, developers can continuously measure how changes to their eBPF programs impact the verifier’s instruction count, enabling more precise optimizations, eliminating instruction limit surprises

Conclusion

The eBPF instruction count limitation is a fundamental constraint designed to balance performance, security, and feasibility of verification. While the current 1,000,000 instruction limit is sufficient for most use cases, developers must be mindful of it when writing complex eBPF programs. By leveraging optimizations, tail calls, maps, helper functions and measuring instructions count, developers can work within this constraint to build efficient and scalable eBPF-based solutions.

from bcc import BPF
import signal

# eBPF Program (C)
bpf_prog = """
#include <uapi/linux/ptrace.h>
#include <linux/bpf_verifier.h>

BPF_PERCPU_ARRAY(bpf_verifier_env_ptr, unsigned long, 1);

int do_check_start(struct pt_regs *ctx) {
    int index = 0;
    unsigned long* v = bpf_verifier_env_ptr.lookup(&index);
    if (!v) {
        return -1;
    }
    
    *v = ctx->di;
    return 0;
}

int do_check_end(struct pt_regs *ctx) {
    int index = 0;    
    unsigned long* v = bpf_verifier_env_ptr.lookup(&index);
    if (!v) {
        return -1;
    }
        
    u32 insn_processed = 0;
    struct bpf_prog* prog = NULL;
    struct bpf_prog_aux	*aux = NULL;
    unsigned long len = 0;
    char name[BPF_OBJ_NAME_LEN] = {0};
    
    struct bpf_verifier_env* env = (struct bpf_verifier_env*)(*v);
    int ret = bpf_probe_read_kernel(&insn_processed, sizeof(insn_processed), &env->insn_processed);
    if (ret) {
        bpf_trace_printk("failed to read env->insn_processed");
        return ret;
    }
    
    ret = bpf_probe_read_kernel(&prog, sizeof(prog), &env->prog);
    if (ret) {
        bpf_trace_printk("failed to read env->prog");
        return ret;
    }

    ret = bpf_probe_read_kernel(&len, sizeof(len), &prog->len);
    if (ret) {
        bpf_trace_printk("failed to read env->prog->len");
        return ret;
    }
    
    ret = bpf_probe_read_kernel(&aux, sizeof(aux), &prog->aux);
    if (ret) {
        bpf_trace_printk("failed to read env->prog->aux");
        return ret;
    } 
    
    ret = bpf_probe_read_kernel_str(name, sizeof(name), aux->name);
    if (ret < 0) {
        bpf_trace_printk("failed to read env->prog->aux->name");
        return ret;
    }

    bpf_trace_printk("bpf name = %s, bpf length = %d, pbf processed instructions = %u", name, len, insn_processed);
    return 0;
}
"""

b = BPF(text=bpf_prog)

b.attach_kprobe(event="do_check", fn_name="do_check_start")
b.attach_kretprobe(event="do_check", fn_name="do_check_end")

print("Tracing bpf program loading()... Hit Ctrl+C to stop.")
signal.pause()