A few months ago, a quiet patch was merged into Firecracker that fixed a security issue that impacts aarch64 only. It’s a vulnerability in Firecracker’s jailer process that I’ll describe in a minute. I thought it would be fun to explore what would have happened if this class of vulnerability impacted Firecracker in the future.

The specific PR from Amazon is: #5956

What is the Jailer

The concept of the “Jailer” here is what’s referred to sometimes as the “supervisor”: For Android that was the zygote process, for containers that was the shim, it’s a process that is responsible for setting up the isolation environment before it gets executed.

But the jailer isn’t a privileged supervisor process sitting next to Firecracker forever. It performs things like setting up the chroot, namespaces, cgroups, UID/GID, resource limits, etc., then drops privileges and exec()s the Firecracker binary. At that point what remains is the jailed, unprivileged Firecracker VMM process.

So the defense-in-depth question I’m interested in is Suppose something crosses the virtualization boundary and gives an attacker control of the Firecracker VMM process. What can that process actually do next?

What (would have been) the vulnerability

The jailer has to run as root to perform all of the operations necessary to setup the environment like namespacing, chroots, cgroups, etc. During that setup process the jail wrote a value into a file inside the jail environment and then changed the file’s owner to the unprivileged jail UID. The key here is that it’s two separate calls of a WRITE and a CHOWN and both of them would have followed symlinks and neither verified that it actually was applying it to the file it expected. Just arbitrary paths.

This, as I’ve learned the hard way at previous companies, is a common breakout path. I’ve been burned by these unchecked file operations a few times especially when dealing with extracting tar balls. The shape of this vuln is essentially any file system operation owned by a privileged user, with an opportunity for an attacker to manipulate the file/path/mount or file descriptor.

In this case, other architectures were not vulnerability because they compiled out the copy_cache_info() (and copy_midr_el1_info()) function which was aarch64 specific.

// We now change the permissions.
let dest_path_cstr = to_cstring(&jailer_cache_file)?;
// SAFETY: Safe because dest_path_cstr is null-terminated.
SyscallReturnCode(unsafe {
    libc::chown(dest_path_cstr.as_ptr(), self.uid(), self.gid())
})
.into_empty_result()
.map_err(|err| {
    JailerError::ChangeFileOwner(jailer_cache_file.to_owned(), err)
})?;

This ends up being a TOCTOU bug you can go explore more yourself. This bug isn’t that interesting when you look at all the dependencies.

Seccomp as a post-exploitation defense? No.

Here is my main point, crammed down that the bottom: Seccomp is applied to the critical jailed components to be a backstop for exactly this situation, but it’s not effective. It’s the classic io_uring issue. The seccomp policy that is applied allows io_uring to support a non-default feature, which as many have talked about before, io_uring gives attackers a way to multiplex system calls that would normally have been filtered by a seccomp-bpf filter.

Here’s an example:

symlinkat("/etc/passwd", AT_FDCWD, path)

System call #36 is blocked by the Firecracker seccomp policy.

Using io_uring:

io_uring_setup(8)                        # io_uring_setup syscall not blocked
mmap(ring, ..., MAP_SHARED)              # load a shared memory space
sqe.opcode = IORING_OP_RENAMEAT;         # load in the actual Syscall 38 RENAME_AT hidden from seccomp
sqe.addr = "/sys";                       
sqe.addr2 = "/sys.stashed";
io_uring_enter(ring, 1, 1, GETEVENTS)    
sqe.opcode = IORING_OP_MKDIRAT           # load Syscall 34
sqe.opcode = IORING_OP_SYMLINKAT         # load Syscall 36
io_uring_enter(ring, 1, 1, GETEVENTS)    # Direct symlinkat = SIGSYS

Why io_uring and the real bug

If you were to ask “Why did they allow io_uring for seccomp?” the answer gets more interesting. 4 years ago, in v1.0.0 there were two modes for what they called the “Block IO Engine” – you can do sync and async. Async needed io_uring so it was added in but even 4 years later it’s still marked as “Developer preview.” I have no inside knowledge but I would imagine that this feature is more likely used by AWS who would be more invested in its performance than a normal user.

In any case, the io_uring bypass risk appears to be self inflicted without any practical reason for most users.

Prompts for fun

Let’s try something new because who needs code, here are some prompts to give your rigs

Conclusions

There’s no break out here but in pulling on a thread I learned some new things. I don’t know why you wouldn’t be able to remove io_uring from the allowed system calls because it does give you a measurable improvement to your line of defense. Having it there now effectively makes it very easy to bypass if a vulnerability like this were to come out. This is again why we shouldn’t be measuring the success of seccomp policies based on the number of system calls blocked because even if you blocked all the othe system calls, allowing io_uring and a few others gets you a simple bypass.

There’s one last aspect that I haven’t explored but I know it’s relevant – io_uring now has it’s own ecosystem for securing itself outside of seccomp and Firecracker tries to use it. I’ll breadcumb you to check out restrictions.rs to see what it’s trying to do. But AFAICT, it’s ineffective at protecting against this class of vuln.

• AIL #1: Human written, AI helped with code and links and to double check I’m not a complete idiot