Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD

Modern frontier scale Large Language Models are usually associated with clusters of expensive GPUs, hundreds of gigabytes of fast memory and correspondingly large electricity bills. This makes running these models locally rather challenging - or prohibitive for a typical hobby or home user. Smaller models have become surprisingly capable, but there are still tasks for which model scale and the capabilities emerging from it make a noticeable difference.

Nevertheless I am currently running the 744 billion parameter GLM-5.2 model on an old dual processor Xeon workstation containing only 20 GB of DDR3 ECC memory. There is no large GPU array hidden in the machine. The model is evaluated on the CPUs while its experts are streamed from NVMe storage using the Colibrì inference runtime.

This is not fast. It is, however, a complete frontier scale model producing useful answers on hardware on which its quantized weights could not possibly fit into memory. This article takes a look at why large models are interesting, how Mixture-of-Experts models make this particular trick possible, how Colibrì turns NVMe storage into another level of the memory hierarchy and how I made the runtime work on FreeBSD.

• Why Run Such a Huge Model Locally?
• The Memory Problem
• Mixture of Experts
• Colibri and NVMe Streaming
• The Test System
• SSD setup
• Activation Vector Steering
  • Does Steering Reduce Performance?
• Running Colibri Behind an API Gateway
• Using the Model with Codex
• Performance Measurements
• What This Does Not Do
• Conclusion
• References

Why Run Such a Huge Model Locally?

Small language models are often entirely sufficient. A model with a few billion parameters can summarize text, extract structured data, answer straightforward questions and perform many constrained automation tasks. Models in the range of 20 to 70 billion parameters can already be highly useful for programming, scientific discussion and agentic applications. This naturally raises the question why anyone would want to operate a model containing hundreds of billions of parameters at home.

The simple answer is that model scale still matters. Increasing scale does not merely add more memorized facts. Larger models tend to become better at combining distant concepts, maintaining constraints through longer reasoning processes, handling unusual problems and recovering when the first attempted solution does not work. They also tend to be more reliable during long running agentic tasks in which small errors accumulate over many individual decisions. Such phase transitions are often described as emergent properties. At the moment of writing this article it is not totally clear why these properties emerge and why they are also present in massive MoE networks.

Of course scale is not the only important factor. Model architecture, training data, post-training and inference methods are also at least as important as the raw parameter count. Nevertheless there is a very real practical difference between a small model that occasionally demonstrates a capability and a large model that can apply it reliably. For long horizon reasoning, difficult programming tasks, scientific problems and autonomous tool use, this reliability may be more important than the ability to produce a single impressive benchmark answer.

Running a capable large model locally additionally provides:

• Control over the complete inference environment
• Independence from an external API and changing usage limits
• The ability to inspect and modify the runtime
• Privacy for local documents and source code
• Reproducible experiments with quantization, routing and activation steering
• The simple satisfaction of operating the entire system oneself

The last point should not be underestimated.

The Memory Problem

GLM-5.2 contains approximately 744 billion parameters. Storing a parameter using a 32 bit floating point value requires four bytes. Even without considering any runtime state, this would result in

Reducing the numerical precision reduces the storage requirement correspondingly:

The converted model used for my Colibrì experiment occupies approximately 383.7 GB in 144 shards. The small difference from the idealized INT4 calculation is caused by additional model data, metadata and components that are stored at different precision.

The workstation currently contains only 20 GB of physical RAM. The model is therefore not just slightly too large. Its weight files are about nineteen times larger than the entire installed memory.

Three different resource limits are often mixed together when discussing local language model inference:

• Storage capacity determines whether the complete model can be kept at all.
• Memory capacity determines how much of the model can remain immediately accessible to the processors.
• Bandwidth determines how quickly weights can be transported from storage or memory to the processing units.

Quantization attacks the first two problems by reducing the number of bytes per parameter. It does not make them disappear. A dense 744B model quantized to four bits would still normally require hundreds of gigabytes of RAM or VRAM during inference, let alone the space required for the KV cache, runtime buffers and temporary activations.

Fortunately GLM-5.2 is not a dense model.

Mixture of Experts

GLM-5.2 uses a sparse Mixture-of-Experts architecture. Instead of passing every token through every feed-forward parameter, the model contains a collection of expert networks. A learned router selects a small subset of them for every token and layer.

In a simplified representation the output of such a layer is

where $r$ is the router, $E_i$ denotes an expert and $g_i$ is the corresponding routing weight. Only the ($\operatorname{top}_k$) experts selected by the router have to be evaluated for the current token. Tuning how many experts are evaluates is another tuning option for the model, available via the --topp parameter.

According to the Colibrì documentation, the 744B model activates approximately 40 billion parameters per token. The routed experts whose selection changes between tokens account for about 11 GB of weight data. The computation required for a token is therefore much smaller than the total parameter count might suggest.

It is important to distinguish computational sparsity from storage sparsity. The runtime does not know in advance which experts will be selected by all future tokens. Consequently all experts still have to be stored somewhere. Conventional runtimes keep them in RAM or VRAM so that any routing decision can be serviced immediately.

This leads to an interesting possibility: If only a small subset is required at any one time, perhaps the remaining experts do not have to reside in fast memory at all.

Colibri and NVMe Streaming

Colibrì is a lightweight inference runtime written by JustVugg in C specifically for GLM-5.2 and similar sparse models. It treats GPU memory, system RAM and storage as levels of a managed memory hierarchy.

The dense model components and other continuously required data remain resident in RAM or VRAM. Routed experts are cached in RAM, while the complete expert collection remains on storage. When a required expert is not present in the cache it is loaded from disk. Frequently used experts may remain cached or become pinned while less useful entries are replaced.

Conceptually this resembles virtual memory, but the runtime understands the structure of the model. It knows which files contain individual experts, which experts the router selected and which data is likely to be useful again. This allows it to make considerably more informed decisions than a generic operating system page cache, like simply mmap of the model with ollama

Insufficient RAM no longer makes inference impossible. It makes inference slower.

This is a rather substantial change. The machine still needs enough storage for the complete model and enough resident memory for the dense components, attention state, runtime buffers and an expert cache. It does not, however, need enough RAM to hold all 383.7 GB of model data simultaneously.

Colibrì also implements several techniques intended to hide or reduce the resulting I/O cost:

• A learned expert cache and the ability to pin frequently used experts
• Automatic sizing of the cache according to the RAM budget
• Direct I/O to avoid unhelpful duplication in the operating system cache (bypassing the operating system cache layer)
• Pipeline workers that overlap expert loading with computation
• Experimental router look-ahead to prefetch experts for the following layer
• Multi-token prediction for speculative decoding
• Optional RAM and GPU tiers for selected experts

None of these techniques can repeal the bandwidth and latency characteristics of the storage device. They can, however, keep it busy more efficiently and avoid some unnecessary reads.

The Test System

My current test system is an older dual processor Xeon workstation:

This is emphatically not the hardware one would normally associate with frontier scale inference. The limited memory is particularly interesting because it leaves relatively little space for the expert cache after the resident model components and operating system are taken into account.

The machine will soon be upgraded to 96 GB RAM. This should allow a much larger fraction of useful experts (from about 12 GB to 88 GB) to remain cached and will provide an interesting comparison on otherwise identical hardware. It may also move the bottleneck away from storage and towards DDR3 memory bandwidth or integer matrix multiplication.

The dual socket ccNUMA arrangement introduces an additional variable. Memory access on such systems is non-uniform, and thread placement as well as the location of allocated memory may influence performance. This is something worth investigating separately after establishing a useful baseline.

SSD setup

The NVMe disk setup is pretty straight forward. Just keep in mind that nvme0 and nvme0ns1 are the controller and namespace device, the storage device is exposed as nvd device. The properties of the SSD that I am using are shown below:

$ sudo nvmecontrol identify nvme0ns1
Size:                        1953525168 blocks
Capacity:                    1953525168 blocks
Utilization:                 1953525168 blocks
Thin Provisioning:           Not Supported
Number of LBA Formats:       1
Current LBA Format:          LBA Format #00
Data Protection Caps:        Not Supported
Data Protection Settings:    Not Enabled
Multi-Path I/O Capabilities: Not Supported
Reservation Capabilities:    Not Supported
Format Progress Indicator:   0% remains
Deallocate Logical Block:    Read 00h
Optimal I/O Boundary:        0 blocks
NVM Capacity:                0 bytes
Preferred Write Granularity: 8 blocks
Preferred Write Alignment:   8 blocks
Preferred Deallocate Granul: 8 blocks
Preferred Deallocate Align:  8 blocks
Optimal Write Size:          8 blocks
Globally Unique Identifier:  000000000000000100a075255449c16e
IEEE EUI64:                  00a075015449c16e
LBA Format #00: Data Size:   512  Metadata Size:     0  Performance: Best

After making sure it’s the right device to overwrite I created a zpool and a zfs filesystem for the model files:

zpool create -o ashift=12 -o mountpoint=/nvme1 llm_nvme1 /dev/nvd0 
mkdir /llm_nvme1/glm52_i4_original
zfs create llm_nvme1/glm52_i4_original

Porting Colibri to FreeBSD

The original runtime primarily targeted Linux, Windows and macOS. Since the actual engine is written in C and has very few dependencies, adapting it to FreeBSD did not require a fundamental port, it was merly some basic single line changes.

My changes are available in my Colibrì fork on GitHub.

The build system contained a phony glm target intended to map the target name to glm.exe on Windows. On Unix systems EXE is empty, which made the phony target collide with the actual output target. Guarding the alias so it is only created when an executable suffix exists fixes the Makefile behaviour.

The engine also uses sys/resource.h, sys/mman.h and mlock related functionality on supported Unix-like systems. Adding __FreeBSD__ to the existing platform condition enables the corresponding implementation:

#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
#include <sys/resource.h>
#include <sys/mman.h>
#endif

With GCC and its OpenMP runtime installed, the runtime can then be built and executed natively on FreeBSD.

cd colibri/c
gmake glm

After the build suceeded the download can be initiated - with the conversion taking place on the fly. This takes around 2 days, especially due to rate limiting of HuggingFace:

./coli convert --model /llm1/glm52_i4_001

On my system I currently supply the GCC runtime library path explicitly:

env LD_LIBRARY_PATH=/usr/local/lib/gcc14 \
    COLI_MODEL=/llm_nvme1/glm52_i4 \
    ./coli plan --ram 19

The plan command is useful before starting the large model. It displays the detected model size, storage availability, selected RAM budget and the planned distribution of model data across the available tiers.

An interactive chat can be started using:

env LD_LIBRARY_PATH=/usr/local/lib/gcc14 \
    COLI_MODEL=/llm_nvme1/glm52_i4 \
    ./coli chat --ram 19

After the model has initialized, prompts can be entered directly. Responses do not arrive at the speed expected from a hosted GPU service, but they do arrive and are produced by the complete local model.

Activation Vector Steering

Since I already had a locally modifiable runtime, I also experimented with porting activation vector steering and refusal-direction ablation to Colibrì, based on a very well known implementation.

The basic idea is to collect hidden-state vectors for two prompt classes, for example prompts that trigger refusal behaviour and corresponding harmless prompts. Their mean difference provides a direction in activation space:

At a selected transformer layer the component of a hidden state along this direction can then be removed:

where $\alpha$ controls the strength of the intervention.

The implementation in my fork can collect activation samples, calculate a normalized direction using a small Python utility and apply the direction in the C inference engine. This is an experimental facility and should not be confused with retraining or fine-tuning the model. It is an intervention in the forward pass.

Does Steering Reduce Performance?

During ordinary inference the additional computation is tiny. At one selected layer the engine:

• Calculates one dot product over the hidden dimension
• Performs one scaled vector subtraction over the same dimension

This requires approximately $2D$ multiplications and $2D$ additions per token for hidden width $D$, together with two linear passes over a single activation vector. This is negligible compared with the matrix multiplications involving roughly 40 billion active parameters per token. On a configuration dominated by loading many gigabytes of experts from NVMe, the difference should be below ordinary run-to-run variation. It does add a small amount of memory traffic through the activation vector and steering direction. This might become measurable only on a highly optimized configuration that is already limited by memory bandwidth rather than storage or expert matrix multiplication. It should still be a very small fraction of the complete forward pass.

Activation collection mode has a different performance characteristic. For every collected prompt it opens the sample file and appends a float32 hidden-state vector. It also deliberately runs short controlled generations while collecting the training examples. Collection mode should therefore not be enabled during inference benchmarks.

To verify the expected result, the final benchmark suite should include otherwise identical runs with steering disabled and enabled. The comparison has to use the same prompt, cache state, decoding settings and generated token count. Since token routing may change when activation steering changes the continuation, wall-clock comparisons across freely generated long answers are otherwise easily confounded.

Running Colibrì Behind an API Gateway

An interactive terminal is useful for testing, but I normally access language models through my own OpenAI compatible API gateway, miniapigw. Placing Colibrì behind the gateway allows existing applications to use the runtime without knowing whether a request is handled by one of the local Ollama instances, a remote cloud service or the NVMe streamed GLM-5.2 backend.

The basic arrangement is:

The gateway can provide a stable model name, authentication, request logging, streaming and backend-specific timeout handling. It also allows applications to switch between a fast smaller model and GLM-5.2 without changing their client implementation.

{
  "type" : "openai",
  "name" : "colibriwssz1",
  "base_url" : "http://198.51.100.0:8123/v1",
  "api_key" : "local",
  "concurrency" : 4,
  "supports" : {
    "chat" : [ "glm-5.2-colibri" ],
    "embeddings" : [],
    "responses" : [],
    "images" : []
  },
  "cost" : {
    "currency" : "eur",
    "unit" : "1k_tokens",
    "models" : { }
  },
  "responses_shim" : {
    "enabled" : true,
    "operation" : "chat",
    "unsupported_tools_policy" : "silent_strip",
    "reorder_system_messages_to_front" : false,
    "coalesce_system_messages" : false
  }
},

For a backend of this type it is important that the gateway does not interpret a period of computation or disk activity as a failed request. Streaming responses and appropriately long idle and total request timeouts are essential.

Using the Model with Codex

Chatting with a slowly streamed model and using it as the reasoning engine of a coding agent are two rather different workloads.

An agent such as Codex repeatedly combines a potentially large repository context with tool results, generates commands, waits for their execution and continues reasoning. This creates several challenges:

• Initial prompt prefill may be much larger than in an ordinary chat
• The context grows after every tool interaction
• A single task may require many separate inference calls
• Client and gateway timeouts may assume GPU-like response times
• Long generated explanations consume time without advancing the task
• Parallel agents can compete for a backend that is most useful when serving one request at a time

For slow local operation it is therefore useful to keep the supplied context deliberate, compact conversations before they become unnecessarily large, limit output lengths and configure generous timeouts. Streaming should be used wherever possible so that progress remains visible.

The expected use case is not rapid interactive autocomplete. It is a patient, long running task (utilizing /goal) for which the capability and local control of the larger model justify waiting. Whether this is practically useful depends not only on tokens per second but also on how many correct tool iterations are required to complete the task. A slower model that follows the goal co herently may still finish before a faster model that repeatedly has to be corrected.

Performance Measurements

My current measurements test combinations of direct I/O, router look-ahead, pipelined loading and different worker counts under a 19 GB runtime RAM budget. A fixed prompt is used to make the runs comparable:

Explain why the sky is blue in 3 short paragraphs.

The benchmark records total wall-clock time and generated output. Since Colibrì does not currently print all desired timing statistics in the tested mode, the external test script measures the complete request and derives comparable throughput values from the produced token count where possible.

All measurements were repeated for 10 times, massive outliers during night time periodic(8) runs have been discarded (they deviated by more than $5\sigma$ in the specific configuration group and where directly correlatable to the night time periodic jobs). Also note that the piping of the output through a Python script also showed slight performance reduction in comparison to direct inference.

As the cache hit ratio was very low in all test cases, it will be particularly interesting to repeat the same test after upgrading the machine from 20 GB to 96 GB. This will show how strongly the larger expert cache affects the hit rate and whether the system transitions from an I/O dominated regime towards a compute or memory-bandwidth dominated regime.

What This Does Not Do

Colibrì does not magically turn an old Xeon workstation into a rack of modern GPUs. There are several rather fundamental limitations:

• Approximately 400 GB of model storage is still required
• Fast NVMe storage matters, maybe even mirror them to counter I/O bottlenecks, depending on the CPUs you are using
• Generation latency is high on old CPUs
• A small RAM cache causes repeated expert loads
• Long prompts require substantial prefill computation
• Agentic clients require adjusted timeout and context handling
• Performance depends on cache history and expert routing

Nor does INT4 quantization necessarily reproduce the full precision model exactly. Quantized kernels can alter close logits, speculative decoding may select a different but still valid continuation and the quality cost of the quantized model has to be measured rather than assumed away.

This is therefore not a replacement for proper high performance inference infrastructure when latency, concurrency or throughput matter. It is a way to make a model accessible on machines that could otherwise not load it at all.

Conclusion

A complete 744 billion parameter language model is currently answering questions on my FreeBSD workstation containing 20 GB of old DDR3 ECC memory.

The model is about nineteen times larger than the installed RAM. Its quantized weights occupy nearly 400 GB. It is evaluated by two old six-core Xeon processors and supplied with routed experts from an NVMe SSD. This is not fast, and every generated answer makes the physical limits of storage bandwidth, memory bandwidth and computation quite visible.

But it works.

This is what makes Colibrì so fascinating. It does not pretend that the model has become small. Instead it exploits the sparse structure of a Mixture-of-Experts model and turns an absolute capacity limit into a continuum of performance. More RAM, faster storage and faster processors improve the result, but insufficient RAM no longer closes the door completely.

A few years ago, running a frontier scale model locally meant owning hardware capable of keeping the entire model resident in expensive memory. Now an old FreeBSD workstation can keep the dense parts alive, fetch experts as they are requested and eventually produce the answer.

One has to be patient, but not wait till the heat death of the universe. One may want another NVMe SSD. One will almost certainly start thinking about memory upgrades. The response to a simple question taking half an hour for sure means that the solution is not usable for a real-time chat task, but it makes those models actually suitable as the long term goal following planning backend of agentic frameworks, which controls smaller scale agents (like llama3.5 or qwen3.6 in their 35 billion or 70 billion parameter variants that happily run on consumer hardware)

But a 744B model is running in 20 GB of RAM - and this is simply tremendously exciting.

What will hopefully follow up is how the RAM upgrade to 96GB influences the performance - which I expect to be massive due to the low cache hit ratio and the available data on the influence of RAM on the generation speed.

References

• The original Colibrì runtime
• My repository containing the FreeBSD port and activation ablation experiments
• zAIs blog post on GLM-5.2: Built for Long-Horizon Tasks
• The accompanying paper about GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• Hardware one may need:
  • A fast NVMe SSD
  • Maybe also an PCIe to M.2 NVMe adapter
• My OpenAI compatible API gateway, that includes shims to support codex in front of different backends and to handle slow backends better
• ollama as very simple inference engine for other smaller models
• HuggingFace, the largest repository for open weight models and datasets
• A reference implementation of refusal removal with transformers
• My own blog articles on
  • Using Codex with a custom API gateway
  • using QWen3.6:35b-a3b with codex
  • Running Codex or other applications in lightweight FreeBSD jails

This article is tagged: Programming, Artificial Intelligence, Tutorial, FreeBSD, System administration, Administration, Large Language Models, Automation, Computational linear algebra, Privacy, LLM, Vibe coding, n8n