Running Codex (or other applications) in lightweight FreeBSD jails

It is often necessary to execute somewhat untrusted applications in an isolated environment. Keep in mind that you should never run really untrusted applications where malicious intent is possible on the same physical machine as any other application - always expect sidechannels being possible no matter what isolation or virtualization technique you are using. History has repeatedly shown that such side channels exist. But sometimes you want to isolate applications against accidental or unintended side effects.

In the following article we are going to take a look on how to isolate a typical agent orchestrator for large language models in its own enclave while allowing it to interact with the network as well as with local filesystems and local applications. In the context of this blog article this will be OpenAI’s codex, that I am using in this example in --yolo mode (granting it all access without any approval, experimenting with the long-term planning capabilities of such agents (not only for software development). This will allow the orchestrator to write its own software and execute it on the host to solve problems without interaction of the user. I have also developed my own agent orchestrator that uses a similar jail based approach for isolation of its subagents too, but this article will focus on codex.

The first solution that comes to mind is to use a separate user account and just execute the applications there. This is perfectly valid when you don’t want to provide isolation of the visible kernel namespaces (like running processes, physical networking properties, etc.). But sometimes you also need this.

Many people would solve this problem using Docker containers. Docker primarily addresses application deployment and reproducible software environments rather than security isolation. Although Linux namespaces, seccomp and related mechanisms can provide substantial isolation depending on the configuration, they solve a somewhat different problem. For the lightweight userspace virtualization presented here, FreeBSD jails are a particularly elegant fit.

• What are FreeBSD jails?
• Creating the jail
  • Creating Filesystems and Directories
  • Filesystems and fstab
  • Populating the jails /etc
  • Jail configuration
  • Launching and testing the jail
  • The codex start command
• Resource Control
  • rctl
  • cpuset
  • ZFS quotas
• Security Considerations
  • Shared kernel resources
  • Side channels
  • Shared user permissions
  • Networking
  • Possible future hardening
• Conclusion
• References

What are FreeBSD jails?

A better approach to such a problem is actually applying userspace virtualization techniques like FreeBSD’s jails. Long before application containers such as Docker became popular, and even before Linux gained technologies such as cgroups and LXC, FreeBSD already provided operating-system level virtualization through jails.

Jails build on the concept of chroot and expand the model by virtualizing access to the file system, the set of users and especially also the networking subsystem. There is a huge number of fine-grained controls available depending on how one wants to realize this access control. All jails still share the same kernel and thus are very lightweight.

Usually one makes a distinction between thick jails and thin jails. Thick jails provide their own complete copy of the base system. All binaries and all filesystems are separated from the host system. This provides a high degree of isolation and one is able to run completely different versions of software (as long as it’s compatible with any of the currently running ABIs to the kernel) or configurations on the host and in different jails. Since each jail runs its own copy of the base system also leakage of sensitive information is prohibited - but on the downside each jail requiring a separate copy of the base system means one requires either the storage space or a read only copy of the given base system. In addition one has to maintain each base system (upgrading, etc.) separately. This administrative overhead should not be underestimated, since keeping the base of each container upgraded is about the load of administering a separate machine for every container (this is something that people who utilize docker containers often forget - or see as a feature running ancient libraries inside containers and call this a “compatibility feature”).

Thin jails on the other hand share substantial portions (up to nearly all) parts of the base system. They can be created and bootstrapped extremely fast and share management with the base system. Of course this reduces isolation - they may access information on the base system.

The FreeBSD manual then also introduces the concept of service jails which are just chroot on steroids. A service jail shares the same root file system as the host. Usually they are separated from the network by default but can be explicitly configured to have access to restricted resources. This follows the philosophy of denying any access that has not been explicitly granted (a very good approach). Such a jail needs zero administration, is very resource efficient, fast to deploy and provides additional process isolation - though overall the reduced isolation has to be considered for security analysis.

Creating the jail

For the purpose of this article I wanted to have an agent that has read only access to all public resources on the host and shares all executables, python packages and data files. This design intentionally follows the principle of least privilege: every writable location is explicitly created, while the remainder of the filesystem is exposed read-only. Thus we are going to nullfs-mount the root directory in a read only fashion into the jail. We will only add a writable home directory (and optionally also a tmpfs) into the jail. Note that this means the agent orchestrator can read all host files that are available to its user ID. If you don’t want this create a separate base jail!

Let’s take a walk through creating the environment. For this purpose I will use the following placeholders:

• CODEX_HOME is the home directory on the host in which the agent should execute later on.
• JAIL_ROOT is the location where we are going to store the jail root. This will be its own zfs filesystem (as also the CODEX_HOME will be).
  • The JAILNAME variable will be used as placeholder for the name of the jail, that is also part of JAIL_ROOT.
• CODEX_UID and CODEX_GID are the user ID and group ID that are going to be used by the agent inside the jail. For convenience reasons one can reuse the ID from the host if one also uses the same UID there.

Creating Filesystems and Directories

First we are going to create the jail root (I assume the CODEX_HOME is already existing as its own ZFS filesystem) and the jails are actually located under the directory tree /jails/jails/${JAILNAME} and the tank/jails/jails/${JAILNAME} ZFS namespace:

zfs create -o mountpoint=/jails/jails/${JAILNAME} tank/jails/jails/${JAILNAME}
chmod 755 /jails/jails/${JAILNAME}

mkdir -p \
    /jails/jails/${JAILNAME}/bin \
    /jails/jails/${JAILNAME}/sbin \
    /jails/jails/${JAILNAME}/lib \
    /jails/jails/${JAILNAME}/libexec \
    /jails/jails/${JAILNAME}/usr \
    /jails/jails/${JAILNAME}/usr/local \
    /jails/jails/${JAILNAME}/etc \
    /jails/jails/${JAILNAME}/dev \
    /jails/jails/${JAILNAME}/tmp \
    /jails/jails/${JAILNAME}/var \
    /jails/jails/${JAILNAME}/var/tmp \
    /jails/jails/${JAILNAME}/var/run \
    /jails/jails/${JAILNAME}/var/log \
    /jails/jails/${JAILNAME}/home \
    /jails/jails/${JAILNAME}/home/${JAILNAME} \
    /jails/jails/${JAILNAME}/root

chmod 1777 /jails/jails/${JAILNAME}/tmp
chmod 1777 /jails/jails/${JAILNAME}/var/tmp
chmod 0755 /jails/jails/${JAILNAME}/var/run
chmod 0700 /jails/jails/${JAILNAME}/root

Filesystems and fstab

Now that the filesystem is ready one needs to perform the nullfs-mounts. What we are going to do via fstab (don’t execute it manually, this will run automatically later on) are the following mounts:

mount -t nullfs -o ro /bin /jails/jails/${JAILNAME}/bin
mount -t nullfs -o ro /sbin /jails/jails/${JAILNAME}/sbin
mount -t nullfs -o ro /lib /jails/jails/${JAILNAME}/lib
mount -t nullfs -o ro /libexec /jails/jails/${JAILNAME}/libexec
mount -t nullfs -o ro /usr /jails/jails/${JAILNAME}/usr
mount -t nullfs -o rw /usr/home/${JAILNAME} /jails/jails/${JAILNAME}/home/${JAILNAME}

To do this we are going to create a fstab file that will be executed by the jail management script later on. This could be /etc/fstab.${JAILNAME} for example:

cat > /etc/fstab.${JAILNAME} <<EOF
/bin                        /jails/jails/${JAILNAME}/bin                    nullfs  ro  0  0
/sbin                       /jails/jails/${JAILNAME}/sbin                   nullfs  ro  0  0
/lib                        /jails/jails/${JAILNAME}/lib                    nullfs  ro  0  0
/libexec                    /jails/jails/${JAILNAME}/libexec                nullfs  ro  0  0
/usr                        /jails/jails/${JAILNAME}/usr                    nullfs  ro  0  0
/usr/home/${JAILNAME}  /jails/jails/${JAILNAME}/home/${JAILNAME}  nullfs  rw  0  0
EOF

Populating the jails /etc

The next step is to populate the etc directory of the jail. This should not be shared with the host filesystem. We are going to create:

• Our own passwd instance that will only contain the users that should be visible inside the jail
• A group file that contains only the groups visible inside the jail
• Our own hosts and nsswitch.conf (the latter is especially important when using LDAP on the host)

In addition we are going to copy resolv.conf, shells, services and protocols as well as populate our timezone information inside the jail:

cat > /jails/jails/${JAILNAME}/etc/master.passwd <<EOF
root:*:0:0::0:0:Charlie &:/root:/bin/sh
${JAILNAME}:*:1002:1002::0:0:Codex Agent:/home/${JAILNAME}:/bin/sh
nobody:*:65534:65534::0:0:Unprivileged user:/nonexistent:/usr/sbin/nologin
EOF

pwd_mkdb -d /jails/jails/${JAILNAME}/etc \
    /jails/jails/${JAILNAME}/etc/master.passwd

cat > /jails/jails/${JAILNAME}/etc/group <<EOF
wheel:*:0:root
codex:*:1002:codex
nogroup:*:65533:
nobody:*:65534:
EOF

cat > /jails/jails/${JAILNAME}/etc/nsswitch.conf <<EOF
group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
EOF

cat > /jails/jails/${JAILNAME}/etc/hosts <<EOF
127.0.0.1 localhost
::1 localhost
EOF

cp -p /etc/resolv.conf /jails/jails/${JAILNAME}/etc/resolv.conf
cp -p /etc/shells /jails/jails/${JAILNAME}/etc/shells
cp -p /etc/services /jails/jails/${JAILNAME}/etc/services
cp -p /etc/protocols /jails/jails/${JAILNAME}/etc/protocols
ln -s /usr/share/zoneinfo/Europe/Vienna /jails/jails/${JAILNAME}/etc/localtime

echo 'Europe/Vienna' > /jails/jails/${JAILNAME}/etc/timezone

chown root:wheel /jails/jails/${JAILNAME}/tmp /jails/jails/${JAILNAME}/var/tmp
chmod 1777 /jails/jails/${JAILNAME}/tmp /jails/jails/${JAILNAME}/var/tmp

Jail configuration

The next step is the actual jail configuration for the jail management script. For now we are going to inherit the IPv6 and legacy IP network configurations. The jails configuration goes into /etc/jail.conf.d/ and will be consumed by the rc.init script later on.

cat > /etc/jail.conf.d/${JAILNAME}.conf <<EOF
${JAILNAME} {
    path = "/jails/jails/${JAILNAME}";
    host.hostname = "${JAILNAME}";

    persist;

    mount.fstab = "/etc/fstab.${JAILNAME}";
    mount.devfs;
    devfs_ruleset = 4;

    exec.clean;
    exec.start = "/sbin/ldconfig /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat";
    exec.consolelog = "/var/log/jail_${JAILNAME}_console.log";

    allow.raw_sockets = 0;
    allow.mount = 0;
    allow.set_hostname = 0;
    allow.sysvipc = 0;
    allow.chflags = 0;

    children.max = 0;
    enforce_statfs = 2;

    ip4 = inherit;
    ip6 = inherit;
}
EOF

The exec.start command is especially important. Unlike a traditional thick jail, this service jail starts with only a minimal filesystem layout. Although the required libraries are available through the read-only nullfs mounts, the runtime linker cache has not yet been initialized.

Running ldconfig once inside the jail builds the shared library hints used by the dynamic linker (ld-elf.so). Without this step dynamically linked applications may fail to locate shared libraries even though they are physically present inside the mounted filesystem.

Strictly speaking this only has to be executed once after creating the jail. Running it during every startup is inexpensive and guarantees that the cache remains consistent even if libraries on the host are upgraded.

The devfs_ruleset has to match the hosts description - for now I assumed this to be the default jail devfs ruleset shipped with FreeBSD in the /etc/devfs.rules.

Launching and testing the jail

Now it’s time to start the jail. If jail_enabled is not set to TRUE in /etc/rc.conf one has to use onestart in the following command:

/etc/rc.d/jail onestart ${JAILNAME}

Now one can list the available jails using jls:

   JID  IP Address      Hostname                      Path
     5                  ${JAILNAME}              /jails/jails/${JAILNAME}

Now as a quick check one can see if the expected users are available - and the jail is not able to resolve external users (i.e. we are really executing with the /etc/ we would be expecting):

jexec ${JAILNAME} id ${JAILNAME}

This should yield something like

uid=1002(${JAILNAME}) gid=1002(codex) groups=1002(codex)

Testing for any other user that should exist on the host but not inside the jail should yield

id: testuser: no such user

Now we can execute a small test script, that will validate that we can execute Python executables inside the jail:

jexec -U ${JAILNAME} ${JAILNAME} /usr/local/bin/python3 - <<PY
import os
import sys
import site
import ssl
 
print("uid:", os.getuid())
print("cwd:", os.getcwd())
print("python:", sys.executable)
print("site:", site.getsitepackages())
print("ssl:", ssl.get_default_verify_paths())
PY

This should yield output similar to

uid: 1002
cwd: /
python: /usr/local/bin/python3
site: ['/usr/local/lib/python3.11/site-packages']
ssl: DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')

Now it’s a good idea to also check that we are really unable to write into the read only mounted sections of the file system:

jexec -U ${JAILNAME} ${JAILNAME} /bin/sh -c 'touch /usr/local/THIS_MUST_FAIL'

This should yield

touch: /usr/local/THIS_MUST_FAIL: Read-only file system

As a last step we could take a look what we see from the inside of the jail:

jexec -U ${JAILNAME} ${JAILNAME} /bin/sh -c ' \
    echo "Root:" \
    ls -la / \
    echo "Home:" \
    ls -la /home \
    echo "Host root home:" \
    ls -la /root 2>&1 || true \
    echo "Host var:" \
    ls -la /var'

The codex start command

To make life easier we will also create a start command for our codex instance. In the following example it will always run in yolo mode and execute with an ollama backend:

cat > /usr/home/${YOURUSER}/${JAILNAME}-jailed.sh <<EOF
#!/bin/sh
set -eu

WORKSPACE="\$1"
shift

exec jexec -U codex ${JAILNAME} \
    /usr/bin/env -i \
    HOME=/home/${JAILNAME} \
    USER=${JAILNAME} \
    LOGNAME=${JAILNAME} \
    SHELL=/bin/sh \
    PATH=/usr/local/bin:/usr/bin:/bin \
    LANG=C.UTF-8 \
    LC_ALL=C.UTF-8 \
    TMPDIR=/tmp \
    XDG_CACHE_HOME=/tmp/${JAILNAME}-cache \
    XDG_STATE_HOME=/tmp/${JAILNAME}-state \
    PYTHONPYCACHEPREFIX=/tmp/python-pycache \
    PIP_CACHE_DIR=/tmp/pip-cache \
    CODEX_OSS_BASE_URL="http://203.0.113.2:8182/v1" \
    /bin/sh -c '
        mkdir -p \
            "\$XDG_CACHE_HOME" \
            "\$XDG_STATE_HOME" \
            "\$PYTHONPYCACHEPREFIX" \
            "\$PIP_CACHE_DIR" \

        cd "\$HOME"
        umask 077
        cd "\$WORKSPACE"

        exec /usr/local/bin/codex --yolo --oss -p glm52 -m glm-5.2-abliterated:432b "\$@"
    ' sh "\$@"
EOF

chmod 0755 /usr/home/${YOURUSER}/${JAILNAME}-jailed.sh

Now one can invoke the codex instance with

~/${JAILNAME}-jailed.sh WORKSPACE [ARGUMENTS]

The WORKSPACE variable is a convenience method to change directory inside the jail before executing codex.

Resource Control

Filesystem isolation alone does not prevent an agent from exhausting host resources. Depending on the intended workload it may therefore be useful to combine jails with FreeBSD’s resource control mechanisms.

Typical limits include

• maximum number of processes
• maximum resident memory
• CPU utilization
• open file descriptors
• network bandwidth
• filesystem quotas

FreeBSD provides several mechanisms for this. These mechanisms are intentionally orthogonal to jails and therefore compose naturally with the isolation model described above.

rctl

The rctl framework allows fine-grained resource limits to be applied to users, login classes or jails. Examples include limiting the maximum amount of physical memory, preventing excessive process creation or restricting CPU utilization.

This is particularly useful when experimenting with autonomous software that may accidentally fork recursively or allocate excessive amounts of memory.

cpuset

For CPU intensive workloads one may additionally assign a jail to dedicated processor cores using cpuset. This prevents long-running AI workloads from interfering with latency-sensitive services executing on the host.

ZFS quotas

Since the writable portions of the jail reside on their own ZFS filesystem, applying quotas and reservations is straightforward. This prevents accidentally generated datasets from consuming all remaining disk space.

Security Considerations

While the setup shown above provides a significant degree of isolation, it is important to understand exactly what security properties it provides and where its limits lie.

The most important fact is that FreeBSD jails are not virtual machines. Every jail on the system shares the same kernel with the host. A successful kernel exploit therefore compromises every jail as well as the host system. For this reason one should never execute software with a realistic expectation of malicious behaviour on the same physical machine as sensitive workloads. In such situations a dedicated machine with external network isolation remains the correct solution.

The setup described here should instead be viewed as protection against accidental damage, programming errors and unintended side effects of autonomous agents. In other words, the threat model considered here assumes that the agent is buggy or overly enthusiastic rather than actively malicious.

Shared kernel resources

Even though filesystem namespaces, user databases and many networking facilities are isolated, many kernel resources remain shared:

• scheduler
• memory allocator
• CPU caches
• kernel networking stack
• block device drivers

Any kernel vulnerability affects both host and jail.

Side channels

Like all operating-system level virtualization technologies, jails cannot protect against hardware side channels.

Examples include

• CPU cache timing attacks
• speculative execution attacks (Spectre, Meltdown, MDS, etc.)
• branch predictor state
• memory bus contention
• shared filesystem timing
• shared network timing

Although exploiting such channels is considerably harder than simply reading files, they demonstrate why strong isolation between mutually distrustful workloads still requires separate physical machines.

Fortunately these attacks are generally irrelevant when the goal is merely preventing an AI agent from accidentally modifying the host system.

Shared user permissions

In the example above the jail intentionally reuses the host UID. This greatly simplifies access to existing workspaces and avoids maintaining duplicate file ownership.

However, this also means the agent can still read every file that this user could normally access. If this is undesirable, a dedicated service account together with a separate thin or thick jail should be created instead.

Networking

The example configuration inherits the host networking stack. This was chosen because the primary goal is running local AI agents with minimal administrative overhead. Inherited networking also avoids creating virtual interfaces or bridge configurations while still allowing direct access to locally running services such as Ollama or MCP servers.

If stronger network isolation is required, FreeBSD’s VNET infrastructure allows each jail to receive its own complete virtual networking stack, including independent interfaces, routing tables and firewall rules.

Possible future hardening

The setup presented here intentionally balances simplicity and isolation. Depending on the threat model one could further strengthen the environment by introducing additional FreeBSD mechanisms, for example:

• VNET networking instead of inherited networking.
• Dedicated service accounts with minimal filesystem permissions.
• Separate thin or thick jails for applications requiring stronger isolation.
• ZFS snapshots for quickly restoring the writable workspace after experiments.
• Resource limits using rctl and CPU affinity through cpuset.
• Mandatory Access Control (MAC) policies for additional kernel-enforced restrictions.
• Capsicum capability mode for individual helper applications that support it.
• Read-only package repositories and immutable build environments.

None of these change the overall architecture described in this article, but they illustrate how the presented service jail can serve as a foundation for progressively stronger isolation as requirements evolve.

Conclusion

FreeBSD service jails provide a remarkably lightweight mechanism for isolating AI agents without introducing another operating system instance. By sharing only the components that need to be shared while keeping writable state confined to dedicated locations, one can achieve a practical balance between convenience, performance and security. Although this is not a replacement for virtual machines when defending against determined attackers, it is an excellent solution for protecting a workstation from accidental damage caused by autonomous agents.

Although the examples in this article use OpenAI’s Codex, exactly the same approach works for many other autonomous development agents, CI workers, automation services or long-running background applications that benefit from lightweight isolation.

References

• A short story of using QWen3.6:35b-a3b with codex
• Using Codex with Hardware In The Loop for Microcontrollers
• Using Proof Assistants to Ground LLMs in Mathematical Truth
• OpenAIs Codex lightweight coding agent
• Docker, one of the most popular container solutions
• Xen, the most popular hypervisor for large scale virtualization deployments.
• FreeBSD jails
• lxc, Linux version of jails
• cgroups, a method to define fine grained permissions on Linux
• Books:
  • Michael W Lucas, FreeBSD Mastery: Jails (IT Mastery, Band 15)
  • X.Y. Wang, The FreeBSD Admin’s Handbook: Configuration, Tuning, and Maintenance
  • George Neville-Neil, Marshall McKusick, Robert Watson, Design and Implementation of the FreeBSD Operating System
  • Michael W Lucas, Absolute FreeBSD, 3rd Edition: The Complete Guide to FreeBSD

This article is tagged:

• Programming
• Artificial Intelligence
• Tutorial
• Basics
• FreeBSD
• System administration
• Large Language Models
• Machine learning
• Xen
• LLM
• Vibe coding
• OpenAI
• Testing
• Ollama