FRRmalloc: An Allocator Against UAF — A Look Back

As an undergraduate researcher in Hanyang University's System Security Lab, I assisted the FRRmalloc project — a memory allocator that stops Use-After-Free (UAF) at the allocator level, keeping the strengths of existing approaches while softening their weaknesses. The work was presented at KIPS ASK 2024. My part was mostly analyzing the internals of existing allocators (ptmalloc2, ffmalloc) and benchmarking memory overhead to compare them. Here is a write-up and a look back.

Why UAF is a problem

UAF happens when a program references heap memory after it has been freed. If a pointer to the freed region survives, it becomes a dangling pointer; once another object is reallocated there, an attacker can read or tamper with someone else's data through that pointer — up to arbitrary code execution or privilege escalation. It ranked #4 on the CWE Top 25 (2023), which says a lot about its impact.

Three lines of defense, and OTA

UAF defenses have largely followed three lines:

1. Detect dangling-pointer use / validate access — catch and block access to freed memory at runtime (PTAuth, CETS, MemSafe)
2. Prevent dangling-pointer creation — on free, reset related pointers to a safe value like null (DangSan, FreeSentry)
3. Never reuse unsafe memory — don't reuse freed memory for a while (or ever)

The flagship of the third line is One-Time-Allocation (OTA): always hand out a fresh, higher virtual address and never reuse a spent one. Since virtual addresses never overlap, old pointers become invalid automatically, and no runtime checks are needed — so the performance overhead is nearly zero. FFmalloc is the representative implementation.

The catch is memory. Because spent addresses are never reused, allocation-intensive programs keep growing their real physical footprint (RSS). In short, OTA is fast but wasteful with memory — and that is exactly FRRmalloc's starting point: keep OTA's speed, fix only the memory waste.

FRRmalloc's core — reusing physical memory via remapping

The trick is remapping, using two kinds of pages:

• canonical page: where the real physical memory lives.
• shadow page: a remap of a canonical page — same physical frame, different virtual address.

Outwardly it hands out a fresh virtual address every time (like OTA), but behind the scenes it reuses already-freed physical memory — it hands out a fresh virtual address every time but reuses the freed physical memory.

The idea itself came first from Oscar (USENIX Security 2017). But Oscar does a page-granularity remap for every object created, making system calls far too frequent and driving up overhead. FRRmalloc changes this to batch page remapping — remapping many pages at once instead of per object — to cut the number of system calls.

Design ① — finer, smarter bins

To reclaim a physical page via remapping, every object on that page must be dead. If objects with different lifetimes share a page, one survivor keeps the whole page alive. So it matters to group objects of similar lifetime/context into the same bin.

Traditionally bins were split by size or allocation type only, which was weak for allocation-intensive programs. FRRmalloc adds a new context. Among candidates (Stack Pointer, stack depth, allocation site, …), it found Stack Pointer (SP) and allocation site (callsite) meaningful, and combines size + SP + callsite to subdivide bins. The callsite is obtained by walking up the call stack with the compiler builtin __builtin_return_address(depth).

The intuition: objects allocated from the same code location with a similar call-stack state tend to share a role and lifetime. Group them and they die together, so pages empty out cleanly.

Design ② — batch page remapping

Remapping for virtual-address reuse is done in batches of pages. A large batch means fewer system calls (good for speed) but the actually-reusable fraction of memory may drop somewhat over time; a small batch improves reuse but calls into the kernel more often. Balancing the two, the initial model chose a batch size of 64.

Evaluation — what improved

Using the allocation-intensive Cfrac benchmark (200k+ malloc calls), remapping at the 25% mark of total runtime, compared against FFmalloc:

• Bin subdivision: increasing the added-context depth (0 / 3 / 5 / 10), peak memory usage dropped in proportion to the depth (depth 0: ~80.1 MB → depth 10: ~72.9 MB).
• Remapping: on two sufficiently large allocation cases (20.3M and 44.4M calls), enabling remap cut memory overhead by 73% and 18% respectively.

The authors are explicit about a limitation, though: these numbers don't yet account for the memory overhead of the extra metadata added for remapping. Since finer bins mean more metadata, "deeper is always better" doesn't hold — there's a trade-off. Profiling and managing that metadata overhead was left as future work.

Retrospective — what I took from assisting

My role wasn't the flashy part but the groundwork. I dug through the internals of ptmalloc2 and ffmalloc to see how the heap is actually managed and where overhead comes from, and I measured and compared FRRmalloc's and ffmalloc's memory overhead directly with allocation-heavy workloads like Cfrac.

Two things stuck with me. One is what "memory overhead" actually measures — RSS, peak RSS (maxrss), and how those tangle with fragmentation and metadata — learned hands-on. The other is a feel for how good security defenses are usually a matter of trade-offs. OTA trades memory for speed; remapping trades system calls for memory. FRRmalloc re-found the balance point between them with batching and finer bins — and confirming that balance in numbers was my part of the job.

Presented at KIPS ASK 2024 · Baseline: FFmalloc · Benchmark: Cfrac