<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://www.tspi.at/atom.xml" rel="self" type="application/atom+xml" /><link href="https://www.tspi.at/" rel="alternate" type="text/html" /><updated>2026-08-08T11:34:12+02:00</updated><id>https://www.tspi.at/atom.xml</id><title type="html">tspi.at</title><entry><title type="html">Artificial Intelligence doing CAD: ChatGPT, Codex and FreeCAD</title><link href="https://www.tspi.at/2026/08/06/llmfreecad.html" rel="alternate" type="text/html" title="Artificial Intelligence doing CAD: ChatGPT, Codex and FreeCAD" /><published>2026-08-06T00:00:00+02:00</published><updated>2026-08-08T11:00:37+02:00</updated><id>https://www.tspi.at/2026/08/06/llmfreecad</id><content type="html" xml:base="https://www.tspi.at/2026/08/06/llmfreecad.html"><![CDATA[<p>One of the most interesting ideas in the current wave of AI is not to keep large language models trapped inside text, code, and chat windows, but to give them the ability to act on the physical world. If an LLM can design something that can actually be manufactured, then it stops being merely a conversational or programming tool and starts becoming part of a real engineering toolchain. This is - besides the generation of <em>new</em> ideas and exploration of the unknown - one of the most exciting directions right now: let the model build <em>real objects</em>. Since I had already experience with letting an LLM orchestrator accessing and operating an (older) <a href="https://github.com/tspspi/mcpXL30">XL30 scanning electron microscope</a>, I decided to take it a step further.</p>

<p>The first practical step toward that goal is giving the model access to CAD. Some models can already generate STL meshes directly, and sometimes they do surprisingly well, but in many cases they are essentially using a scripted CAD detour or emitting geometry without a strong editable model behind it. That is useful for quick experiments, but it leaves a lot of power on the table. If instead the model can control a full CAD system, it gains access not just to geometry creation but to the whole surrounding ecosystem: parametric modeling, exact BREP data, custom geometric calculations, rendering, visual inspection, simulation workflows, and the many workbenches that already exist. More importantly, it gains access to another <em>independent representation</em> of the design that can later be compared against mathematical reasoning, rendered images, simulations, and ultimately the manufactured part itself.</p>

<p>This is also a very natural route toward autonomous manufacturing. I touched on some of that fascination in my <a href="/2026/07/03/autonomousmachines.html">The Hum That Never Sleeps</a> article: once tools begin extending themselves into more and more of the real production chain, the step from automation to partial autonomy gets surprisingly small. We are not at fully autonomous factories yet, but one can already see the shape of it. An LLM that can design parts inside CAD, validate them, pass them to a slicer, queue them for printing and eventually hand them to further assembly stages is no longer just generating content. It is participating in manufacturing.</p>

<p>And honestly, I think it is amazing how far this has already gone. Not with local models yet, unfortunately. At least in my experiments, the current locally running models are not there for these complex multi-hour engineering tasks. I hope we get there, because this is exactly the kind of capability I would love to run fully on my own hardware. But with the currently strongest hosted models, this has become very real.</p>

<ul>
  <li><a href="#the-idea">The Idea</a></li>
  <li><a href="#the-freecad-mcp-approach">The FreeCAD MCP Approach</a>
    <ul>
      <li><a href="#engineering-requires-discipline">Engineering Requires Discipline</a></li>
    </ul>
  </li>
  <li><a href="#why-visual-inspection-matters-so-much">Why Visual Inspection Matters So Much</a></li>
  <li><a href="#codex-versus-chatgpt-web">Codex versus ChatGPT Web</a></li>
  <li><a href="#results">Results</a></li>
  <li><a href="#what-comes-next">What comes next</a></li>
</ul>

<p><img src="/assets/images/png/llmcad001.png" alt="" /></p>

<h2 id="the-idea">The Idea</h2>

<p>What I wanted to try was simple in principle: give an LLM agent direct access to a real CAD package and see whether it can create mechanically meaningful geometry on its own. Not toy boxes or decorative shapes, but actual parts that can be printed and used.</p>

<p>For this I used <a href="https://www.freecad.org/">FreeCAD</a>. That choice is important. FreeCAD is not just a mesh generator, it provides parametric modeling, sketch-based workflows, solid modeling on top of exact boundary representations, Python programmability, a wide range of workbenches and optional simulation-oriented extensions - and it is open source so you can tap into every function it offers. If an LLM gets access to that environment, it is suddenly no longer restricted to <em>guessing geometry</em>. It can work with a mature engineering system.</p>

<p>This matters for several reasons:</p>

<ul>
  <li>It can build parametric models instead of one-shot meshes</li>
  <li>It can inspect BREP-level geometry and topology programmatically</li>
  <li>It can use spreadsheets and expressions to keep dimensions editable</li>
  <li>It can render the current state and perform visual inspection</li>
  <li>It can access existing workbenches instead of re-inventing every operation from scratch</li>
  <li>It can use advanced workflows such as finite element analysis, optics-related tooling, and mechanical helper workbenches that are already present in the CAD environment</li>
</ul>

<p>In other words: instead of teaching an LLM a tiny fake CAD language, one can hand it the whole machine shop of a modern CAD system.</p>

<h2 id="the-freecad-mcp-approach">The FreeCAD MCP Approach</h2>

<p>To make this practical I built, with substantial help from <a href="https://openai.com/codex/">Codex</a>, a small bridge service that plugs into a running FreeCAD session and exposes that capability through MCP. MCP was choosen over scripting, GUI automation or custom RPC due to MCP providing a standardized tool interface that multiple different LLM systems can consume without requiring model specific integrations (note that one still has to tune the skills and promps massively to match the given LLM in the end). Because FreeCAD embeds Python, the bridge runs inside FreeCAD’s Python runtime and therefore has full access to the application state, document model, GUI state, geometry objects, workbenches, and Python APIs. This is a huge advantage. The bridge is not screen-scraping or performing UI automation from the outside. It is operating from inside the CAD system.</p>

<p>The setup is split into two pieces:</p>

<ul>
  <li>A bridge module loaded into the running FreeCAD process</li>
  <li>An MCP server that talks to that bridge and exposes tools to an agent orchestrator</li>
</ul>

<p>The MCP server supports both local <code class="language-plaintext highlighter-rouge">stdio</code> transport and a remote HTTP transport. The remote HTTP mode is useful for remote orchestrators and also for attaching the same tooling to the web version of ChatGPT. In that browser-based setup, a reverse proxy in front of the MCP endpoint is practically required, both to expose the service cleanly under HTTPS and to handle the path mapping that the web client expects. One should obviously not expose such a service carelessly; even for experimentation it deserves proper authentication, TLS, and a deliberately constrained deployment.</p>

<p>The project comes with two files that are particularly important:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">LLM_USAGE.md</code>, which describes the operating discipline for the model</li>
  <li>The Codex skill, which encodes the workflow and priorities needed for reliable CAD work</li>
</ul>

<p>That second point turned out to matter much more than one might expect. The raw tools alone are not enough. The model needs a working discipline: inspect before editing, prefer parametric and mutable features, validate sketches before downstream operations, inspect the document tree after changes, and explicitly perform visual checks instead of assuming success from tool output alone.</p>

<p>This discipline is what made the difference between <em>“it sometimes kind of works”</em> and <em>“it can eventually complete real tasks correctly”</em>.</p>

<p><img src="/assets/images/png/llmcad002.png" alt="FreeCAD running side by side with codex" /></p>

<h3 id="engineering-requires-discipline">Engineering Requires Discipline</h3>

<p>The lesson was that giving an LLM access to CAD tools is, as mentioned, only part of the solution. The much larger challenge is teaching it an engineering workflow. Unlike conversational tasks, CAD models evolve over many editing operations, and seemingly minor mistakes can propagate through the feature tree until they become difficult to diagnose. The operating discipline encoded in <code class="language-plaintext highlighter-rouge">LLM_USAGE.md</code> and the project-specific mechanical design rules therefore became almost as important as the tools themselves.</p>

<p>The workflow deliberately treats every modeling step as something that must be verified before building upon it. Sketches are checked for solver consistency and closed profiles before creating downstream features. After each modification, the document tree is re-inspected to ensure the expected feature hierarchy exists. The view is then prepared explicitly, unnecessary objects are hidden, and screenshots are captured for visual validation. Mechanical constraints receive a similar level of scrutiny: holes are checked to ensure they actually pass through their intended targets, movable parts are examined over their complete range of motion instead of only in their neutral position, fasteners are verified for tool access and assembly order and expensive operations such as threaded fasteners or gears are initially represented by simplified proxy geometry to make collision detection and iteration practical.</p>

<p>This layered verification strategy turned out to be crucial. Rather than assuming that a successful API call implies a correct mechanical result, the agent continuously validates the model from several independent perspectives: geometric consistency, feature-tree integrity, visual appearance, manufacturability, and finally assemblability. Individually none of these checks is sufficient, but together they make long autonomous design sessions surprisingly robust.</p>

<p>Looking back, I no longer think the decisive ingredient was giving the model access to FreeCAD. The real breakthrough was giving it access to <strong>multiple independent representations of the same object</strong>. Every representation exposes different classes of mistakes. Mathematical reasoning catches inconsistencies in dimensions and formulas, the CAD model verifies geometric construction, rendered images reveal spatial relationships and collisions that are difficult to infer symbolically, while mechanical reasoning and assembly validation expose problems that are invisible in pure geometry. The model continually moves between these representations until they all describe the same object consistently.</p>

<p>That iterative cross-validation feels surprisingly similar to how human engineers work. Professional engineers almost never trust a single representation. Instead we alternate between equations, drawings, CAD models, prototypes and measurements, using each to verify the others. Rather than replacing this engineering process, the LLM is beginning to participate in it.</p>

<h2 id="why-visual-inspection-matters-so-much">Why Visual Inspection Matters So Much</h2>

<p>The most important lesson in all of this was that visual inspection is not a gimmick. It is crucial.</p>

<p>At first glance it may seem that a CAD agent should be able to rely purely on geometric and topological queries. And yes, those are also extremely important. Being able to inspect object trees, sketch closure, feature dependencies, body structure, constraints, dimensions and topology is already powerful. But it is not sufficient by itself.</p>

<p>There are many errors that are much easier to catch visually, especially since the LLMs tend to make (sign) errors in their code interpretation often:</p>

<ul>
  <li>a feature extruded in the wrong direction</li>
  <li>a missing cutout that was supposed to go through</li>
  <li>a hidden collision that only becomes obvious from a certain view</li>
  <li>a misplaced body or support</li>
  <li>a shape that is technically present but obviously wrong in context</li>
  <li>an operation that succeeded mathematically but produced nonsense mechanically</li>
</ul>

<p>Once the agent can intentionally orient the view, hide clutter, fit the model, capture snapshots, and then reason over the rendered result, the loop becomes dramatically more robust. In my experiments that visual loop was one of the decisive ingredients. It gave the model a way to check whether the CAD state <em>looked</em> like the intended mechanical object instead of only checking whether some API call returned success.</p>

<p>What surprised me most was why the visual loop helped so much. Current LLMs are usually capable of writing the mathematics required for geometric reasoning, but they often make <em>subtle</em> mistakes when interpreting the coordinate systems those calculations operate in. They may accidentally mix local and global coordinates, swap reference frames, compare objects expressed in different coordinate systems or simply assume that an intermediate calculation represents what they intended rather than what actually exists. The resulting formulas are often mathematically self-consistent and look superficially plausible, yet they describe the wrong geometric relationship.</p>

<p>A rendered image largely removes that ambiguity. Once the model can actually <em>see</em> that a hole stops before reaching the opposite face, that two parts intersect, or that a feature protrudes into a space where no material should exist, the mistake becomes immediately obvious. Interestingly, the model can then often work backwards from that visual observation, identify which geometric assumption must have been wrong, and correct its own calculations. In practice the screenshot acts as an <strong>independent verification domain</strong>. Instead of checking mathematics against more mathematics, the model compares its internal geometric reasoning against what the CAD system actually produced.</p>

<p>That is why I would argue that a serious CAD-capable agent should not just have geometry creation and topology inspection. It should also have rendering and deliberate visual review as first-class tools.</p>

<h2 id="codex-versus-chatgpt-web">Codex versus ChatGPT Web</h2>

<p>I used the same MCP server in two ways: with <a href="https://chatgpt.com/codex/">Codex</a> as the main orchestrator, and embedded as a connector into the <a href="https://chat.openai.com">web version of ChatGPT</a>.</p>

<p>In my experience Codex works <em>significantly</em> better for this kind of task. There are several reasons:</p>

<ul>
  <li>It can iterate deeply over a task without collapsing back into chatbot behavior</li>
  <li>It can handle longer refinement loops and more deliberate task execution</li>
  <li>It can work more naturally with planning and extended verification passes</li>
  <li>It can directly process visual inspection artifacts as part of the workflow</li>
  <li>Its goal-oriented execution style fits engineering tasks much better</li>
</ul>

<p>The web version of ChatGPT can still be surprisingly capable, but it is more constrained. It is fundamentally optimized as a chatbot. In my experiments it does not handle long iterative CAD sessions as gracefully, and it does not reliably consume screenshots from arbitrary private URIs produced by the CAD side. Sometimes it tries to work around that limitation, but not in a dependable way. This matters a lot, because as mentioned above, the screenshot-based visual verification loop turned out to be essential.</p>

<p>There is also the more general issue that long, complex CAD runs consume <em>a lot</em> of model budget. For serious engineering iteration that becomes noticeable very quickly. Codex is simply the better fit for prolonged agentic work of this type
.</p>

<p>This leads to the more practical issue of model budget. Unlike many coding tasks, complex CAD work is highly iterative. A single engineering session can easily involve hundreds of tool invocations, repeated visual inspections, validation passes, corrections, and several complete redesign cycles before converging on a satisfactory result. In practice this means that one substantial mechanical design may consume a surprisingly large fraction of the available model quota. During some of my experiments, a single engineering task burned roughly half of the monthly Codex allowance included with a ChatGPT Pro subscription, while more ambitious assemblies can consume considerably more. At the current state of the technology, autonomous engineering is therefore not only limited by model capability but also by available inference budget and account limits.</p>

<h2 id="results">Results</h2>

<p>The really surprising part is that this was not just a proof of concept. It produced useful results.</p>

<p>The strongest example so far was a planetary gearbox generated through a long Codex session. The full raw footage of that run is 1 hour and 6 seconds long and mostly shows the conversation, the intermediate reasoning process, and the geometry gradually taking shape in FreeCAD. There were some interventions while steering the run, but the final result was printed without manual geometry fixes. It worked flawlessly on the first try and ran play-free immediately after printing. For an autonomously generated mechanical assembly, that is a pretty striking outcome.</p>

<iframe width="560" height="315" src="https://www.youtube.com/embed/f-h2nNMLB_4?si=ce8YyN3i4cUe_7g9" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe>

<p>I also recorded a second video in which ChatGPT itself generated a standard 6-way CF40 vacuum cross. That run took 9 minutes and 15 seconds and completed successfully, including the knife edges. The result was standards-conformant and again demonstrated that the model could do more than just decorative geometry. It could generate a piece of actual technical hardware with the right functional details.</p>

<iframe width="560" height="315" src="https://www.youtube.com/embed/GctU1S20pTo?si=mjZTZyb5vxisb6_1" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe>

<p>There were also additional successful ChatGPT web runs that I did not record as videos:</p>

<ul>
  <li>A lawnmower disk carrying eight cutting blades at an angle, including the attachment to a keyed shaft and screw-based retention</li>
</ul>

<p><img src="/assets/images/png/llmcad_result01.png" alt="Lawnmower cutting disk designed by ChatGPT web in FreeCAD" /></p>

<p><img src="/assets/images/png/llmcad_result02.png" alt="Lawnmower cutting disk designed by ChatGPT web in FreeCAD" /></p>

<p><img src="/assets/images/jpg/llmcad001.jpg" alt="3D printed lawnmower cutting disk designed by ChatGPT web in FreeCAD" /></p>

<ul>
  <li>A cap for an air duct opening with louvers for rain protection plus an inner retainer ring clamped with screws and nuts to hold a wire mesh in place</li>
</ul>

<p><img src="/assets/images/png/llmcad_result03.png" alt="Air duct cover designed by ChatGPT web in FreeCAD" /></p>

<p><img src="/assets/images/png/llmcad_result04.png" alt="Air duct cover designed by ChatGPT web in FreeCAD" /></p>

<ul>
  <li>A modular parameterized hydrocyclone for an electrochemical machining process, designed from the ChatGPT web interface, featuring a TPU gasket</li>
</ul>

<p><img src="/assets/images/png/llmcad_result05.png" alt="A 3D printed PLA hydrocyclone fully designed by ChatGPT web" /></p>

<p>What is remarkable here is not just that the model can emit geometry, but that with the correct tooling <em>and skill guidance</em> it can eventually produce mechanically and functionally correct CAD models without manual geometry repair. <em>That is the threshold I was most interested in</em>.</p>

<p>Of course this did not happen by accident. It took some steering, and the operating discipline encoded in the skill and <code class="language-plaintext highlighter-rouge">LLM_USAGE.md</code> mattered a lot. But once that guidance was in place, the agent became capable of iterating until the result was actually right.</p>

<p>This is why I think this line of work is so exciting. It extends LLM agents out of the purely virtual domain and into real-world object generation. Once an agent can create geometry that survives the jump into manufacturing, something qualitatively changes. It stops being only a text machine and becomes part of an engineering pipeline. And I do not particularly doubt that with the same approach, a sufficiently capable agent can design much more complex machines completely on its own, at least in 3D-printable form. The real constraint right now feels less like principle and more like model budget, available iteration time, and how much patience one has for long autonomous runs.</p>

<h2 id="what-comes-next">What comes next</h2>

<p>The obvious next step is to continue extending the toolchain beyond CAD.</p>

<p>If the model can already design parts in FreeCAD, then the next logical additions are:</p>

<ul>
  <li>access to the slicer</li>
  <li>access to printer profiles and process settings</li>
  <li>automated print job preparation</li>
  <li>print queue management</li>
  <li>eventually feedback from the printer and post-processing stages</li>
  <li>expanding to the ECM machine as well as the CNC mill</li>
</ul>

<p>At that point the loop starts to look less like <em>“AI doing CAD”</em> and more like the early skeleton of autonomous fabrication. Not full autonomous industry, not yet, and certainly not without safeguards, supervision and <em>a lot of engineering around the edges</em>. But clearly the direction is there.</p>

<p>To me that is the fascinating part. We are beginning to give LLM agents the first real handles into the physical world, an ability to shape the real world, not mearly measure or perceive it. CAD is a very good first handle, because it is where intent becomes geometry. The next handles are manufacturing preparation, machine execution, inspection, and iteration. Once these links are chained together, one can already glimpse a version of Industry 4.0 in which autonomous software agents do not just optimize dashboards or summarize documents, but actively participate in making things.</p>

<p>More generally, I suspect this principle extends <em>far</em> beyond CAD. Autonomous engineering will probably not emerge from increasingly sophisticated reasoning alone, but from systems that can repeatedly validate their own work across multiple independent domains. CAD geometry, rendered images, simulations, finite element analysis, manufacturing constraints, machine feedback and eventually physical measurements are all different views of the same underlying object. As more of these representations become accessible to an agent, the opportunities for independent verification increase as well. The engineering process becomes less about trusting a single prediction and more about achieving consistency across many different ways of describing reality.</p>

<p>Perhaps that is the real lesson here. Autonomous engineering is probably not about building models that never make mistakes. It is about giving them enough independent ways of looking at the same design that those mistakes become increasingly difficult to survive. In that sense, the future may belong less to ever larger reasoning models than to systems that continually verify their own conclusions across many <em>different</em> representations of reality.</p>

<p>I think that is an amazing development. I also think we are still only at the beginning.</p>

<h2 id="references">References</h2>

<ul>
  <li>OpenAI <a href="https://chatgpt.com/codex">Codex</a></li>
  <li>OpenAI <a href="https://chat.openai.com">ChatGPT</a></li>
  <li><a href="https://www.freecad.org/">FreeCAD</a></li>
  <li>Previous blog articles:
    <ul>
      <li><a href="/2026/07/03/autonomousmachines.html">The Hum That Never Sleeps</a>, a vision about self replicating autonomous machines</li>
      <li><a href="/2026/04/06/tripo3dapi.html">Programmatic 3D Model Generation with the Tripo3D API</a></li>
      <li><a href="/2026/05/01/opensource3dassets.html">A Local Image/Text to 3D Solution: Hunyuan3D</a></li>
      <li><a href="/2026/01/10/chatgptremotemcp.html">Playing with ChatGPT RemoteMCP without OAuth</a>, highlighting a method to authenticate against an MCP without fully implementing OAuth and still being able to use the MCP from ChatGPTs web interface</li>
      <li><a href="/2026/07/12/codexyolojail.html">Running Codex (or other applications) in lightweight FreeBSD jails</a>, a very good approach for long running autonomous agents</li>
      <li><a href="/2026/07/18/glm52colibrifreebsd.html">Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD</a>, a way to run large scale LLMs on consumer hardware (very slow, even with optimiziations presented in <a href="/2026/07/23/glmoptiwest01.html">Optimizing GLM for dual processor Westmere Xeon E5620: 200% gain (3x faster) till now</a>)</li>
    </ul>
  </li>
  <li>3D printers:
    <ul>
      <li><a href="https://amzn.to/4eo6tkN">Anycubic Kobra X</a> (optionally <a href="https://amzn.to/43NVGdK">combination including the external colormanagement device</a>)</li>
      <li><a href="https://amzn.to/4t438LF">Creality Ender 3 V3 KE</a> 3D printer for fast and reliable single color prints, a very good printer especially if you plan custom modifications</li>
    </ul>
  </li>
  <li>Mentioned other projects:
    <ul>
      <li>The <a href="https://github.com/tspspi/mcpXL30">MCP server for the XL30 scanning electron microscope</a></li>
    </ul>
  </li>
  <li>FreeCAD books:
    <ul>
      <li>Alec Sorando, <a href="https://amzn.to/4hRRbqw">FreeCAD 1.1.1 Vom Anfänger Zum Profi: Eine Schritt-für-Schritt-Anleitung zu parametrischer Modellierung, Baugruppen, Zeichnungen, CAM, FEM, und Python-Automatisierung</a></li>
    </ul>
  </li>
</ul>]]></content><author><name>tsp</name></author><category term="Artificial Intelligence" /><category term="Large Language Models" /><category term="LLM" /><category term="CAD" /><category term="FreeCAD" /><category term="Mechanics" /><category term="3D printing" /><category term="Automation" /><category term="Programming" /><category term="Industry" /><category term="Vibe coding" /><category term="Projects" /><category term="Lab automation" /><summary type="html"><![CDATA[Giving large language models access to CAD is one of the most direct ways to let them step out of the purely virtual world and begin creating real objects. This article describes an MCP bridge for FreeCAD, why visual inspection turned out to be crucial, and how ChatGPT and Codex were able to generate manufacturable geometry ranging from vacuum hardware to a printable planetary gearbox.]]></summary></entry><entry><title type="html">Protecting Children Without Turning the Internet into an Identity Checkpoint</title><link href="https://www.tspi.at/2026/07/30/pcwtiic.html" rel="alternate" type="text/html" title="Protecting Children Without Turning the Internet into an Identity Checkpoint" /><published>2026-07-30T00:00:00+02:00</published><updated>2026-07-30T15:56:20+02:00</updated><id>https://www.tspi.at/2026/07/30/pcwtiic</id><content type="html" xml:base="https://www.tspi.at/2026/07/30/pcwtiic.html"><![CDATA[<ul>
  <li><a href="#the-problem-is-real-the-proposed-cure-is-not">The Problem is Real, the Proposed Cure is not</a></li>
  <li><a href="#children-should-not-be-treated-as-tiny-adults-in-a-maximised-attention-market">Children Should not be Treated as Tiny Adults in a Maximised-Attention Market</a></li>
  <li><a href="#age-assurance-is-not-automatically-identity-verification---but-it-can-become-it">Age Assurance is not Automatically Identity Verification - but it can become it</a></li>
  <li><a href="#a-phone-app-is-not-a-passport-to-public-life">A Phone App is not a Passport to Public Life</a></li>
  <li><a href="#identity-gates-change-the-relation-between-people-and-information">Identity Gates Change the Relation Between People and Information</a></li>
  <li><a href="#the-record-of-only-this-purpose-is-not-reassuring">The Record of “Only This Purpose” is not Reassuring</a></li>
  <li><a href="#a-confidential-system-is-still-a-system-that-can-fail">A Confidential System is Still a System That Can Fail</a></li>
  <li><a href="#hate-extremism-and-the-difference-between-response-and-amplification">Hate, Extremism and the Difference Between Response and Amplification</a></li>
  <li><a href="#why-anonymity-tools-are-a-democratic-safety-valve">Why Anonymity Tools are a Democratic Safety Valve</a></li>
  <li><a href="#what-a-proportionate-approach-looks-like">What a proportionate approach looks like</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/internetid001.png" alt="" /></p>

<h2 id="the-problem-is-real-the-proposed-cure-is-not">The Problem is Real, the Proposed Cure is not</h2>

<p>There is a temptation to collapse two questions into one. The first is whether children and teenagers should be left, without guidance or limits, in the largest and most commercially optimised communication environment humanity has built. The second is whether states should respond by making ordinary access to information conditional on a verifiable identity, an age credential tied to identity or a real-name account. The answer to the first question can quite reasonably be <em>no</em>. The answer to the second should still be <em>no</em>.</p>

<p>Children need boundaries. This is neither puritanism nor a claim that the internet is uniquely evil. It is ordinary care. A parent would not normally tell an eleven-year-old to go alone to any bar, remote meeting place, or room full of unknown adults at night, then announce that this is safe. Yet a child with an unrestricted smartphone can be introduced, at any hour, to arbitrary adults, sexual material, violence, gambling, self-harm communities, manipulative advertising, harassment and an endless stream designed to hold attention rather than serve development. The medium is real social space, even when it is encountered through a small glowing rectangle.</p>

<p>It is also important not to make the inverse mistake. Children are not made safe by excluding them from knowledge, from peers, from queer communities, from mental-health support, from political information or from the wider world. Online spaces can provide friendship, creative work, education, help in a crisis and a way out of isolation. The relevant distinction is not <em>“the internet”</em> versus <em>“real life”</em>. It is between a child having structured, age-appropriate, supervised access and a child being placed alone in a global, opaque, incentive-driven environment.</p>

<p>That distinction matters because a genuine problem does not make every response legitimate. The present enthusiasm for mandatory age checks and real-name access risks building a general-purpose checkpoint between a person and information. Its first use may be restricted adult material. Its later uses are easy to imagine: gambling, <em>“harmful”</em> political speech, health information, protest organising, minority communities, discussion of war, labour organising or simply the views of a governments opponents. A system designed to decide who may see a category of information is a system that can later decide which categories count and who qualifies.</p>

<p>The better position is therefore demanding in two directions at once: take child safety seriously and <em>refuse</em> to turn that duty into a population-wide identity-and-surveillance infrastructure.</p>

<h2 id="children-should-not-be-treated-as-tiny-adults-in-a-maximised-attention-market">Children Should not be Treated as Tiny Adults in a Maximised-Attention Market</h2>

<p>Some online content really is not suitable for children. This includes material made to sexually arouse adults; realistic violence; instructions and communities that promote self-harm, eating disorders, coercion or exploitation; gambling-like systems; content that deliberately frightens, humiliates or targets a young person. The fact that an older teenager may be able to discuss some of these subjects in an educational setting does not make a commercially packaged, frictionless, recommendation-driven feed appropriate for a younger child.</p>

<p>The danger is not only particular pages. A platform can convert a passing curiosity, a moment of distress or a single click into a repeated stream of similar material. Recommendation systems optimise measurable engagement - watch time, return visits, clicks, shares - not the long-term wellbeing of the person whose attention they capture. For minors, who are still developing impulse control, social judgement and a stable sense of self, this is a particularly poor environment in which to outsource the setting of boundaries.</p>

<p>The evidence on social media and mental health deserves accuracy rather than alarmist slogans. It does not establish that every hour online causes depression, or that social media has one uniform effect on every young person. Effects tend to be heterogeneous, often modest at population level and are entangled with prior distress, sleep, bullying, family circumstances, the kind of use, and the kind of content. An umbrella review by Valkenburg, Meier and Beyens found that much of the review literature described the association as <em>weak or inconsistent</em>, while also arguing that this does not make risks for particular adolescents unimportant[<a href="#ref23">23</a>]. A recent systematic review similarly found an association between problematic social-media use and symptoms of depression and anxiety, with sleep loss, social comparison and feedback-seeking among the relevant pathways; it explicitly calls for stronger longitudinal evidence[<a href="#ref17">17</a>].</p>

<p>This is the right intellectual posture: take the risk seriously without pretending that correlation has solved every causal question. A 2020 meta-analysis found a small but statistically significant association between adolescent social-media use and depressive symptoms (pooled $r=0.11$); small effects can still matter when exposure is widespread and when a subset of children is especially vulnerable[<a href="#ref14">14</a>]. Reviews also connect excessive social-media use with poorer sleep, which is itself an important route by which unrestricted late-night use can damage wellbeing[<a href="#ef1">1</a>]. The policy implication is not <em>“ban communication”</em>. It is to create time boundaries, reduce persuasive and addictive design, require genuinely safe defaults for minors, provide responsive moderation and reporting and give parents and young people usable controls.</p>

<p><em>“Doomscrolling”</em> is an especially useful word here because it describes a pattern rather than a medium: repetitive, difficult-to-stop consumption of alarming or negative material. It is not only a youth problem; <em>adults do it too</em>. Research is still emerging and much of it is cross-sectional, so it should not be described as proof of a single causal disease. But the signal is not imaginary. Satici et al. found doomscrolling associated with psychological distress and lower wellbeing in their validation studies[<a href="#ref18">18</a>], while a 2024 study in samples from Iran and the United States reported associations with existential anxiety and more pessimistic views of human nature[<a href="#ref19">19</a>]. This supports a mundane conclusion: <em>a feed engineered to keep a person searching for the next threat is not neutral nutrition for a developing mind</em>.</p>

<p>No machine can supply the role of an attentive adult. Age-appropriate devices, bedtime rules, family discussions, trusted contact routes, school support, limited public discoverability and the ability to say <em>“show me what happened”</em> are not glamorous policy, but they are <em>real protection</em>. Where parents are too overworked, absent, unwell, unsupported or themselves captured by the same attention economy, that is a social problem - of care, housing, working time, education, mental-health support and community - not merely an isolated defect of <em>“the media”</em>. A state that avoids those responsibilities but asks every citizen to present credentials before reading is choosing the administratively easy solution, not necessarily the humane or effective one.</p>

<h2 id="age-assurance-is-not-automatically-identity-verification---but-it-can-become-it">Age Assurance is not Automatically Identity Verification - but it can become it</h2>

<p>The technical distinction must be made plainly. It is possible, in principle, for a person to prove a limited statement - <em>“I am over 18”</em> - without telling a website their name, exact birthdate, address or a stable identifier. This can use a credential issuer, cryptographic signatures, unlinkable or limited-use tokens and a zero-knowledge proof. The European Commissions current age-verification blueprint expressly describes this aspiration: proof that a user is above a threshold without revealing other personal information[<a href="#ref8">8</a>].</p>

<p>That design is substantially better than uploading a passport to every adult site. It is not, however, a reason to stop asking hard questions - or to decide that any institution deserves the role it requires. <strong>No government, credential issuer, platform, wallet provider, browser vendor, app-store operator, network operator or contractor is sufficiently trustworthy to become the standing gatekeeper of a persons access to information</strong>. This is not an accusation that every employee is malicious. It is a recognition that institutions change, are pressured, are breached, make mistakes, follow changing laws, outsource functions and eventually acquire new administrators and political masters. <strong>A free society must be designed so that ordinary lawful reading and speech do not depend on continuing confidence in the discretion, competence or goodwill of any one of them</strong>.</p>

<p>Who initially binds the credential to a person? <em>Which issuer is trusted</em>? Can an issuer, wallet provider, browser, network operator, or government observe issuance, renewal, revocation or repeated use? Are proofs genuinely unlinkable across relying parties? What metadata is retained, for how long and under which legal power can it be demanded? Does the implementation match the published design, and can independent researchers audit both code and deployment? Can a teenager safely seek information about sexuality, abuse, mental health or politics without generating a durable record? The cryptographic protocol may reveal little to the final website while the surrounding system still creates a sensitive map. And keep in mind: Laws are changed quickly. Technical reality is not. A system that does not exist cannot be misused - a system that exists can be modified quickly.</p>

<p><em>“Privacy-preserving”</em> is an engineering goal and a property to be demonstrated, not a spell that makes a political risk disappear. A central issuer can be designed not to learn where a credential is used. That is good cryptographic hygiene, <strong>but it does not make an access gate acceptable</strong>. <strong>It still places a state-recognised or state-regulated credential between a person and information, establishes an authority able to issue, deny, revoke or alter admission and makes access contingent on carrying an approved proof</strong>. A system may begin with pornography and gambling; it also creates a ready-made interface for any future law that says access to a category requires an eligibility proof. The dangerous change is not only a database leak. It is the normalisation of asking permission, through an accredited gatekeeper, before receiving information.</p>

<p>The conclusion of this essay is therefore deliberately stronger than a request for better implementation: <strong>a privacy-preserving proof of age must not become a prerequisite for receiving lawful information</strong>. If a proof is voluntary and used by a person to obtain a benefit from a particular private service, that raises a different question. <strong>A compulsory proof, however cleverly designed, changes a basic liberty into a conditional entitlement</strong>. The public should not have to obtain a digital token, persuade a credential issuer, possess an approved device or pass an eligibility check before it may read and participate in ordinary lawful communication. <strong>A state-managed identity requirement or a platform real-name rule is more intrusive still, and should be rejected.</strong></p>

<p>This does not leave child protection empty-handed. It directs regulation toward the party that sells and designs the risky environment: require safer defaults for accounts known or reasonably likely to belong to minors; restrict exploitative recommender systems and advertising; provide device- and family-level controls chosen by parents; remove <em>criminal</em> (not undesired) material; and investigate concrete abuse. The burden should not be shifted from platforms and parents onto every adults right to receive information.</p>

<h2 id="a-phone-app-is-not-a-passport-to-public-life">A Phone App is not a Passport to Public Life</h2>

<p>The proposal becomes even less acceptable when the credential is carried in a compulsory mobile-phone application. <strong>A phone is not an identity document, a democratic qualification or a condition of access to public discourse</strong>. Making an app the route through an age gate silently turns possession of a particular, current, networked consumer device into a prerequisite for reading. It excludes people who do not own a smartphone, cannot afford a supported model or data plan, use a shared device, have a disability that makes the app inaccessible, use a privacy-oriented operating system, are travelling or simply refuse to carry a personal tracking device.</p>

<p><strong>It also concentrates risk</strong>. A mobile application is embedded in an operating system with a device identifier, app-store account, push-notification service, network connection, telemetry, crash reporting, analytics libraries, battery and location-related metadata and a <strong>continuous update channel</strong>. A protocol may promise that the final site sees only <em>“over 18”</em>, while the operating system, application provider, issuer, network <strong>or a future compulsory update can still create a rich correlation surface</strong>. A phone-based gate makes revocation and denial immediate: disable an account, demand a new app version, change a policy, or make the device fail attestation <em>and a person can be excluded from information at once</em>.</p>

<p>There is a deeper objection than security. <strong>A government should not get to establish a de facto rule that one must carry a state-compatible app in ones pocket to enter parts of the public conversation</strong>. Such a rule makes freedom of information dependent on private hardware vendors, app-store policies, battery life, network service <em>and administrative approval</em>. That is a total non-starter for a free society. The right response is to make services safer for children, not to turn the smartphone into a domestic border checkpoint.</p>

<h2 id="identity-gates-change-the-relation-between-people-and-information">Identity Gates Change the Relation Between People and Information</h2>

<p>Anonymous and pseudonymous speech have a legitimate social function. A survivor of abuse may need to ask for help. A teenager may need to explore an identity before they can safely disclose it at home. A whistle-blower, a union organiser, a dissident, a patient, a journalists source or a person living in a small community may need to speak and read without making themselves immediately legible to an employer, local authority, political faction, family member, platform or state.</p>

<p>The argument is sometimes reduced to the straw man that anonymity means impunity. It does not. Serious crimes can be investigated with due process and evidence. Services can moderate conduct, throttle abuse and suspend accounts without collecting a government identity for every reader. <em>Pseudonymity</em> also allows reputation, continuity and community norms without forcing a legal name into every interaction. The relevant question is not whether identity is ever useful. <strong>It is whether everyone should have to disclose or certify identity before participating in ordinary communication. They should not.</strong></p>

<p>The same point applies to companies. A platform does not need a passport scan to apply rate limits, block harassment, maintain an account reputation, provide child settings or comply with a court order directed at a specific serious offence. Giving every platform a verified identity or a reusable age credential that becomes a de facto identifier, expands the number of institutions capable of correlating a persons behaviour. Data that begin as <em>“proof of age”</em> have a predictable gravitational pull toward advertising, profiling, fraud systems, employment screening, insurance, political pressure and law-enforcement requests.</p>

<p><strong>The state should not receive that map either.</strong> To know who reads, searches for, joins or discusses a topic is to know something about private life, association, belief and future action. <em>Freedom to seek and receive information is a basic condition of personal autonomy and democratic self-government</em>; it must begin from a presumption of open, uncredentialed access, not from the question whether a person has passed an app-mediated admission test. The European Convention on Human Rights protects not only speaking but receiving and imparting information and ideas without interference by public authority[<a href="#ref3">3</a>]. Article 19 of the International Covenant on Civil and Political Rights protects opinion without interference and protects the freedom to seek, receive and impart information[<a href="#ref21">21</a>].</p>

<p>These rights are not absolute. The same human-rights instruments permit carefully defined restrictions for interests such as the rights of others, public safety and prevention of crime, subject to <em>legality, necessity and proportionality</em>. That qualification is crucial: nobody has a civil-liberties right to sexually exploit a child, make credible threats, stalk someone, distribute abuse material or coordinate violence. But the qualification is not a blank cheque. A general identity gate imposed on everyone in order to deal with a minority of offenders is a particularly poor fit for necessity and proportionality. It burdens the innocent reader, chills lawful speech, establishes a reusable control point <strong>and still leaves determined offenders able to use stolen credentials, foreign services, automation, encryption, proxies or simply one anothers accounts</strong>.</p>

<h2 id="the-record-of-only-this-purpose-is-not-reassuring">The Record of “Only This Purpose” is not Reassuring</h2>

<p>History does not prove that every child-protection measure becomes tyranny. It does show however why powers built in response to fear should be judged by their structure, not merely by the intention announced at enactment. The recurring pattern is familiar:</p>

<ul>
  <li>a narrow emergency is presented</li>
  <li>a broad technical capability is acquired</li>
  <li>exceptions and data-sharing arrangements follow</li>
  <li>independent courts or investigators later discover that safeguards were weaker than promised</li>
</ul>

<p>After 11 September 2001, the United States expanded surveillance powers in the name of counter-terrorism. Section 215 of the USA PATRIOT Act was used for the NSAs bulk telephone-records programme. The independent Privacy and Civil Liberties Oversight Board later concluded that the programme lacked a viable legal foundation, raised serious constitutional and civil-liberties concerns and had shown only limited value; the USA FREEDOM Act ended that bulk programme in 2015[<a href="#ref16">16</a>]. This is not an argument that terrorism was invented or that intelligence work is never needed. It is an argument that <em>“terrorism”</em> is not a sufficient answer to the question <em>“may the state assemble a record of ordinary peoples associations?”</em></p>

<p>Europe supplies its own lessons. The Court of Justice of the European Union annulled the Data Retention Directive after it required broad retention of communications metadata for serious-crime purposes, finding the interference with privacy and data protection disproportionate[<a href="#ref4">4</a>]. In Big Brother Watch and Others v. United Kingdom, the European Court of Human Rights found violations of Articles 8 and 10 in the United Kingdoms historic bulk interception and communications-data regimes; it identified inadequate independent authorisation, insufficient safeguards and insufficient protection for confidential journalistic material[<a href="#ref10">10</a>]. The Court did not say that all bulk interception is inherently unlawful. It said the power required genuine end-to-end safeguards, necessity and proportionality. That is a high bar for good reason.</p>

<p>Other examples make the same point in different vocabulary. Anti-terror legislation has been used to facilitate broad communications collection; <em>“extremism”</em> definitions in authoritarian states routinely encompass political opposition and minority groups; and proposals framed as stopping child sexual abuse have sometimes been designed in ways that would weaken private communications or introduce client-side scanning. The UN Special Rapporteur and other mandate holders warned in 2020 that the proposed US EARN IT Act, though directed at child sexual-abuse material, raised concerns for privacy, freedom of expression and encryption[<a href="#ref22">22</a>]. The lesson is not to ignore child exploitation. <strong>It is to insist that the investigation of specific offences target offenders and evidence, rather than converting private communication into a permanently inspectable space.</strong></p>

<p>The speed-control point is especially important. <strong>Censorship is not limited to deleting a text</strong>. <strong>A state can make reading slower, more costly, riskier, less private, or conditional on a credential</strong>. It can force a service to identify readers, delay dissemination, lower reach, filter search results, remove circumvention tools or create self-censorship through the possibility of later identification. When people believe that their reading list and associations can be reconstructed, many will avoid lawful but sensitive subjects. <strong>That chilling effect is not a side issue; it is how a formally free information space becomes practically narrower.</strong></p>

<p>The United Kingdom provides a concrete warning about overbroad speech policing without requiring exaggerated claims that <em>“police arrest people for every Facebook post”</em>. In Miller vs College of Policing, a police officer visited Harry Millers workplace over tweets that were <strong>recorded as a non-crime hate incident</strong>, warned him about escalation <strong>and advised him to stop posting gender-critical messages</strong>. The Court of Appeal held that the guidance’s operation unlawfully interfered with Article 10 rights and had a chilling effect in a public-interest debate[<a href="#ref7">7</a>]. This does not mean harassment or incitement should be tolerated; British prosecutorial guidance itself distinguishes communications that are merely offensive, shocking or rude - which are generally protected - from conduct crossing the high threshold for an offence[<a href="#ref5">5</a>]. <strong>It shows instead how a laudable desire to oppose hatred can produce an apparatus that pressures lawful speech when concepts are broad and consequences occur before a court ever considers guilt.</strong></p>

<h2 id="a-confidential-system-is-still-a-system-that-can-fail">A Confidential System is Still a System That Can Fail</h2>

<p>Claims that an identity or age-verification mechanism is anonymous, double-blind or secure should be tested, not ridiculed. Good cryptography can materially reduce disclosure. But no one can guarantee that every implementation, vendor integration, logging setting, network path, support process, lawful-access request, future update, or data-sharing agreement will preserve that ideal. Systems fail through code defects, bad architecture, insider access, misconfigured spreadsheets, phishing, procurement shortcuts and political decisions as well as through broken mathematics.</p>

<p>Governments are not uniquely malicious, <strong>but neither are they uniquely competent custodians of a permanent access map</strong>. In 2015, the US Office of Personnel Management reported breaches affecting 21.5 million peoples sensitive background-investigation records, involving current, former and prospective federal employees and contractors[<a href="#ref12">12</a>]. In 2023, the Police Service of Northern Ireland disclosed a spreadsheet in response to a freedom-of-information request; hidden data exposed the surnames, initials, ranks and roles of all 9.483 officers and staff. The UK Information Commissioner later fined the PSNI 750.000 pounds[<a href="#ref13">13</a>], but this does not erase the leaked data. These events do not imply that a particular European age-proof system will necessarily leak. They establish the more modest and decisive point: <strong>institutions that possess sensitive identity data are fallible and a database or log cannot be compromised if the system never needed to create it.</strong></p>

<p>The case against a general identity gate is therefore not based on distrust of one current government alone. Laws, administrations, contractors and political majorities change. A credential issued under a liberal government can be inherited by an illiberal one. Security guarantees are never stronger than the weakest component and the future legal environment. <strong>The appropriate design principle is data minimisation: do not collect a map of lawful private reading, association or expression merely because it might be convenient later.</strong></p>

<h2 id="hate-extremism-and-the-difference-between-response-and-amplification">Hate, Extremism and the Difference Between Response and Amplification</h2>

<p>There is no need to romanticise hateful speech. National Socialism, racism, antisemitism, misogyny, homophobia, threats and campaigns to hurt people are ugly, destructive and <em>sometimes</em> criminal. People targeted by such speech deserve support and protection. The question is how a free society denies it power without giving the state a general licence to decide what citizens may read, say or discuss.</p>

<p><em>One indispensable mechanism is social</em>: do not give destructive movements a permanent resonator. Do not quote-tweet every provocateur into prominence; do not turn every small groups bait into a national spectacle; build better institutions, local ties, economic security and persuasive counter-speech; and enforce clear laws against threats, stalking, coercion, violence and targeted abuse. A society that treats every odious claim as irresistible content for outrage-driven amplification often helps the claim travel further.</p>

<p>That is not a magic doctrine of <em>“ignore all extremism”</em>. Direct threats, recruitment into violence, coordinated harassment, child exploitation and incitement require serious, lawful response. Nor should vulnerable people be told to endure abuse silently. The point is narrower: broad censorship and mass identification cannot supply the social stability, trust and resilience that make destructive ideas unattractive. <strong>If a society has become so brittle that an anonymous troll or fringe slogan can destabilise it, the long-term remedy is to repair the social conditions, not to record the identity of every reader.</strong></p>

<p>The media itself illustrates why attention should be allocated carefully. Research on European immigration coverage finds recurring patterns in which migrants are under-represented and commonly framed through delinquency, criminality, negativity and conflict; repeated exposure can shape attitudes and even vote choice[<a href="#ref6">6</a>]. In another well-known study, Boykoff and Boykoff showed how the journalistic norm of presenting <em>“both sides”</em> of climate science produced coverage that diverged sharply from the scientific evidentiary balance[<a href="#ref2">2</a>]. <strong>The point is not that all journalists lie or that all coverage is a conspiracy. Framing is often produced by selection pressures, deadlines, editorial habits, commercial incentives, access to official sources and the need for a dramatic conflict.</strong></p>

<p>There are also institutional admissions of error. In 2004, The New York Times publicly reviewed shortcomings in parts of its pre-war reporting on alleged Iraqi weapons of mass destruction[<a href="#ref25">25</a>]. That example is useful precisely because it comes from a major outlet with correction mechanisms: large established media can produce excellent reporting and can still transmit mistaken premises, selective sourcing or political framing at consequential moments.</p>

<p><strong>Artificial intelligence makes source literacy more necessary, not less</strong>. Synthetic images, generated text, coordinated persona networks and cheap translation can make an item look more polished than it is. But the corrective cannot be <em>“trust official media, distrust everything else”</em>, nor <em>“trust a blue tick, an ID card or a government-approved source”</em>. A resilient public compares independent sources, follows primary documents, checks dates and incentives, distinguishes evidence from commentary, asks what is missing and remains able to inspect dissenting accounts. The ability to do that privately matters. A state that controls which sources may be reached, and knows who sought an unpopular source, has made verification dependent on its own permission.</p>

<h2 id="why-anonymity-tools-are-a-democratic-safety-valve">Why Anonymity Tools are a Democratic Safety Valve</h2>

<p>Tor is useful here not because it is mystical or because it prevents all investigation. It is a privacy and censorship-circumvention tool that routes traffic through a network so that a destination does not simply see the users ordinary network address. It can reduce linkability between a persons location and their reading or publication. Its protection has limits: a user can still reveal themselves by logging into an identified account, by sharing distinctive information, by malware compromise or through poorly configured applications. It is not a substitute for careful operational security.</p>

<p>Nevertheless, the reaction of censoring states is revealing. Chinas Great Firewall has long tried to block Tor and discover Tor bridges; Winter and Lindskogs technical study documented the mechanism and potential countermeasures[<a href="#ref24">24</a>]. Russia began blocking access to the Tor network and Tor Project website on some networks in 2021, as measured by the Open Observatory of Network Interference (OONI), and later efforts have targeted bridges and circumvention transports[<a href="#ref15">15</a>]. The Tor Project reported further Russian blocking pressure in 2024 while noting that bridges and pluggable transports remained important means of access[<a href="#ref20">20</a>]. In China, many users have relied on circumvention tools to access and share information during politically sensitive events, despite extensive blocking[<a href="#ref11">11</a>].</p>

<p>This does not prove that every user of Tor is virtuous. Tools that preserve privacy can be used by people with good intentions and by people with bad ones, just as roads, cash, locks and encryption can. The question for a free society is whether the ordinary person may have private access to lawful information and a means to speak without becoming a target. In China and Russia, the answer is visibly important because the state has already made access politically dangerous. In Europe, the same capacity is a safeguard against future excess, against coercive profiling and against the temptation to treat inconvenient citizens as a security problem.</p>

<p>It also exposes a practical weakness in identity-gate policy. A determined criminal, determined abuser or determined political actor can seek technical workarounds, stolen credentials, offshore providers, coercion or private networks. The group most reliably burdened by universal checks is the broad ordinary public: adults who want lawful privacy, small services without compliance departments, young people seeking sensitive help, minorities, researchers and people who simply do not want their reading habits connected to a legal identity. <strong>When a policy is easy to bypass for the determined and highly intrusive for everyone else, it should not be casually called effective child protection.</strong></p>

<h2 id="what-a-proportionate-approach-looks-like">What a proportionate approach looks like</h2>

<p>The choice is not between abandoning children and constructing a universal speech passport. A serious approach starts from the different responsibilities of families, platforms, schools, service providers and the state.</p>

<p>Parents and guardians should provide age-appropriate devices and accounts, establish sleep and screen-time boundaries, talk about sexual material and manipulation before a platform does, use tools that give them visibility without humiliating surveillance and remain a safe person to contact when something goes wrong. Society should make this feasible by supporting parents with time, education, counselling and functioning child-protection services.</p>

<p>Platforms should be expected to design for minors rather than merely write terms of service. That means safe defaults, low discoverability by strangers, no targeted behavioural advertising to children, clear reporting routes, rapid action on grooming and abuse, meaningful researcher access, limits on addictive recommender patterns, age-appropriate privacy settings and transparent moderation procedures. The European Commissions guidance under Article 28 of the Digital Services Act itself identifies risks including grooming, harmful content, addictive behaviour, cyberbullying and harmful commercial practices; age assurance is only one possible measure in a wider set[<a href="#ref9">9</a>].</p>

<p>The state should fund education, mental-health care, child-protection investigation and specialised law enforcement; prosecute actual abuse, threats, stalking and exploitation through due process; protect encryption and private communication; require strong security and transparency from platforms and preserve judicial, parliamentary and independent oversight. It should avoid mandatory real-name rules, compulsory general identity credentials for ordinary online participation, universal scanning, and retention systems that make everyones reading and communication available for later inspection.</p>

<p><strong>No age proof - whether identity-revealing, <em>“anonymous”</em>, double-blind, wallet-based or zero-knowledge - should be compulsory merely to read, search, join a discussion or receive information. Nor should a mobile-phone app be required as the key to any part of ordinary public discourse</strong>. Where adults choose to use voluntary controls for a particular service, they should be free to do so; but <strong>law and platform policy must not convert those controls into a general condition of access</strong>. The burden belongs on service design, targeted enforcement, child care and parental tools, not on every person to authenticate their right to learn.</p>
<h2 id="conclusion">Conclusion</h2>

<p>Children deserve more than an unfiltered encounter with the whole internet. They deserve adults who pay attention, platforms that do not exploit immaturity, schools that teach judgement and a society that provides <em>real</em> support when family care fails. Some content is unsuitable for minors. Excessive and compulsive engagement is harmful for many people, including adults. These are good reasons for boundaries, better design and targeted protection.</p>

<p>They are not good reasons to place a government-approved identity gate between the public and information. A privacy-preserving proof may disclose less than a passport scan, but it still makes the right to receive information conditional on a credential, issuer and technical infrastructure; it is therefore not an acceptable solution. <strong>A mandatory mobile app is worse again: it makes a personal networked device the key to public life and adds exclusion, correlation and remote-control risks. Such gates create a technical and legal lever that can later be reused to sort readers, slow information, profile association and suppress unpopular communities or opposition</strong>. Claims of anonymity and privacy are worthwhile only when the entire system - not only its brochure or protocol- can demonstrate them over time. <strong>The historical record of emergency surveillance, overbroad data retention and public-sector data breaches gives no reason to be casual about that trust</strong>.</p>

<p>A free and stable society does not become safer by making every citizen legible before they may look, learn or speak. It becomes safer when children are cared for, harmful conduct is addressed precisely, platforms are held to account, people learn how to judge information, and privacy remains a normal condition of lawful life rather than a privilege one must justify.</p>

<h2 id="references">References</h2>

<ul>
  <li>[<span id="ref1">1</span>] Alonzo, R. <em>et al.</em> “Interplay Between Social Media Use, Sleep Quality, and Mental Health in Youth: A Systematic Review.” <em>Sleep Medicine Reviews</em> 56 (2021): 101414. <a href="https://doi.org/10.1016/j.smrv.2020.101414">https://doi.org/10.1016/j.smrv.2020.101414</a></li>
  <li>[<span id="ref2">2</span>] Boykoff, M. T., and J. M. Boykoff. “Balance as Bias: Global Warming and the US Prestige Press.” <em>Global Environmental Change</em> 14, no. 2 (2004): 125–136. <a href="https://doi.org/10.1016/j.gloenvcha.2003.10.001">https://doi.org/10.1016/j.gloenvcha.2003.10.001</a></li>
  <li>[<span id="ref3">3</span>] . Council of Europe. <em>European Convention on Human Rights</em>, Article 10. <a href="https://www.echr.coe.int/documents/d/echr/convention_ENG">Official text</a></li>
  <li>[<span id="ref4">4</span>] Court of Justice of the European Union. <em>Digital Rights Ireland and Seitlinger and Others</em>, Joined Cases C-293/12 and C-594/12, judgment of 8 April 2014. <a href="https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:62012CJ0293">Judgment</a></li>
  <li>[<span id="ref5">5</span>] . Crown Prosecution Service. “Responding to Intimidating Behaviour in Elections and Public Office.” <a href="https://www.cps.gov.uk/publication/responding-intimidating-behaviour-elections-and-public-office-cps-guide">Guidance</a></li>
  <li>[<span id="ref6">6</span>] Eberl, J.-M. <em>et al.</em> “The European Media Discourse on Immigration and its Effects: A Literature Review.” <em>Annals of the International Communication Association</em> 42, no. 3 (2018): 207–223. <a href="https://doi.org/10.1093/anncom/wqy002">https://doi.org/10.1093/anncom/wqy002</a></li>
  <li>[<span id="ref7">7</span>] England and Wales Court of Appeal. <em>R (Miller) v College of Policing</em> 2021 EWCA Civ 1926. <a href="https://www.judiciary.uk/wp-content/uploads/2022/07/Miller-v-College-of-Policing-judgment-201221.pdf">Judgment</a></li>
  <li>[<span id="ref8">8</span>] European Commission. “Commission Makes Available Age-Verification Blueprint.” 14 July 2025. <a href="https://digital-strategy.ec.europa.eu/en/news/commission-makes-available-age-verification-blueprint">Official release</a></li>
  <li>[<span id="ref9">9</span>] European Commission. “Commission Publishes Guidelines on the Protection of Minors.” 14 July 2025, updated 22 April 2026. <a href="https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-protection-minors">Guidelines page</a></li>
  <li>[<span id="ref10">10</span>] European Court of Human Rights. <em>Big Brother Watch and Others v. the United Kingdom</em> GC, 25 May 2021. <a href="https://www.echr.coe.int/documents/d/echr/fs_mass_surveillance_eng">Court factsheet and case summary</a></li>
  <li>[<span id="ref11">11</span>] Freedom House. <em>China: Freedom on the Net 2024</em>. <a href="https://freedomhouse.org/country/china/freedom-net/2024">Report</a></li>
  <li>[<span id="ref12">12</span>] Government Accountability Office. <em>OPM Has Improved Controls, but Further Efforts Are Needed</em>. GAO-17-614, 2017. <a href="https://www.gao.gov/products/gao-17-614">Report</a></li>
  <li>[<span id="ref13">13</span>] Information Commissioner’s Office. “What Price Privacy? Poor PSNI Procedures Culminate in £750k Fine.” 3 October 2024. <a href="https://ico.org.uk/about-the-ico/media-centre/news-and-blogs/2024/10/what-price-privacy-poor-psni-procedures-culminate-in-750k-fine/">Decision summary</a></li>
  <li>[<span id="ref14">14</span>] Ivie, E. J. <em>et al.</em> “A Meta-Analysis of the Association Between Adolescent Social Media Use and Depressive Symptoms.” <em>Clinical Psychological Science</em> 8, no. 6 (2020): 1102–1116. <a href="https://doi.org/10.1177/2167702620923842">https://doi.org/10.1177/2167702620923842</a></li>
  <li>[<span id="ref15">15</span>] Open Observatory of Network Interference. “Russia Started Blocking Tor.” 17 December 2021. <a href="https://ooni.org/post/2021-russia-blocks-tor/">Measurement report</a></li>
  <li>[<span id="ref16">16</span>] Privacy and Civil Liberties Oversight Board. <em>Recommendations Assessment Report</em>. 2016. <a href="https://documents.pclob.gov/prod/Documents/OversightReport/b1accb9f-0469-46f1-b660-b66acfbc601a/Recommendations_Assessment_Report_20160205.pdf">Report</a></li>
  <li>[<span id="ref17">17</span>] Saleem, N. <em>et al.</em> “Exploring the Relationship Between Social Media Use and Symptoms of Depression and Anxiety Among Children and Adolescents: A Systematic Narrative Review.” <em>Cyberpsychology, Behavior, and Social Networking</em> 27, no. 11 (2024): 771–797. <a href="https://doi.org/10.1089/cyber.2023.0456">https://doi.org/10.1089/cyber.2023.0456</a></li>
  <li>[<span id="ref18">18</span>] Satici, S. A., E. Gocet Tekin, M. E. Deniz, and B. Satici. “Doomscrolling Scale: Its Association with Personality Traits, Psychological Distress, Social Media Use, and Wellbeing.” <em>Applied Research in Quality of Life</em> 18, no. 2 (2023): 833–847. <a href="https://doi.org/10.1007/s11482-022-10110-7">https://doi.org/10.1007/s11482-022-10110-7</a></li>
  <li>[<span id="ref19">19</span>] Shabahang, R. <em>et al.</em> “Doomscrolling Evokes Existential Anxiety and Fosters Pessimism About Human Nature? Evidence From Iran and the United States.” <em>Computers in Human Behavior Reports</em> 15 (2024): 100438. <a href="https://doi.org/10.1016/j.chbr.2024.100438">https://doi.org/10.1016/j.chbr.2024.100438</a></li>
  <li>[<span id="ref20">20</span>] Tor Project. “Tor in Russia: A Call for More WebTunnel Bridges.” 28 November 2024. <a href="https://blog.torproject.org/call-for-webtunnel-bridges/">Technical update</a></li>
  <li>[<span id="ref21">21</span>] United Nations Office of the High Commissioner for Human Rights. <em>International Covenant on Civil and Political Rights</em>, Article 19. <a href="https://www.ohchr.org/en/instruments-mechanisms/instruments/international-covenant-civil-and-political-rights">Official text</a></li>
  <li>[<span id="ref22">22</span>] United Nations Special Procedures. “Communication Concerning the Proposed EARN IT Act.” 19 March 2020. <a href="https://spcommreports.ohchr.org/TMResultsBase/DownLoadPublicCommunicationFile?gId=25134">Communication</a></li>
  <li>[<span id="ref23">23</span>] Valkenburg, P. M., A. Meier, and I. Beyens. “Social Media Use and its Impact on Adolescent Mental Health: An Umbrella Review of the Evidence.” <em>Current Opinion in Psychology</em> 44 (2022): 58–68. <a href="https://doi.org/10.1016/j.copsyc.2021.08.017">https://doi.org/10.1016/j.copsyc.2021.08.017</a></li>
  <li>[<span id="ref24">24</span>] Winter, P., and S. Lindskog. “How China Is Blocking Tor.” USENIX FOCI, 2012. <a href="https://www.usenix.org/conference/foci12/workshop-program/presentation/winter">Paper</a></li>
  <li>[<span id="ref25">25</span>] PBS <em>NewsHour</em>. “The New York Times WMD Coverage.” 26 May 2004. <a href="https://www.pbs.org/newshour/show/the-new-york-times-wmd-coverage">Report on the newspaper’s self-critique</a></li>
</ul>]]></content><author><name>tsp</name></author><category term="Opinion" /><category term="Internet" /><category term="Society" /><category term="Privacy" /><category term="Experience" /><category term="Social" /><category term="Security" /><summary type="html"><![CDATA[The debate surrounding age verification often presents a false choice: either leave children exposed to the unrestricted internet or require everyone to prove who they are before accessing online information. This article argues that the first concern is entirely legitimate - children do need protection - but that mandatory identity or age-verification systems are the wrong pseudo-solution. Instead, it explores how child safety can be improved through better platform design, parental support and targeted regulation without creating a universal checkpoint between people and information. Drawing on academic research, court decisions and historical examples, the article examines the technical, legal and societal implications of identity-based access systems. It discusses privacy-preserving cryptography, smartphone-based credentials, function creep, surveillance risks and freedom of information, ultimately arguing that protecting minors should never come at the cost of making lawful access to knowledge conditional on government-approved credentials]]></summary></entry><entry><title type="html">Installing n8n on a RaspberryPi 3B+</title><link href="https://www.tspi.at/2026/07/29/n8nrpi3b.html" rel="alternate" type="text/html" title="Installing n8n on a RaspberryPi 3B+" /><published>2026-07-29T00:00:00+02:00</published><updated>2026-07-29T23:41:33+02:00</updated><id>https://www.tspi.at/2026/07/29/n8nrpi3b</id><content type="html" xml:base="https://www.tspi.at/2026/07/29/n8nrpi3b.html"><![CDATA[<p>There is a certain class of advice that is always annoying me. In the case of <a href="https://n8n.io">n8n</a> on a <a href="https://amzn.to/3RKueuG">RaspberryPi 3B+</a>, this usually sounds like: <em>just use Docker</em> (or do not bother at all because <code class="language-plaintext highlighter-rouge">npm install</code> will fail anyway). The problem with that advice is that it leaves a practical gap for the exact kind of setup where one wants to use older low-power boards in the first place but still wants to avoid to pull arbitrary strange binary containers from anywhere on the net. Docker containers are a really great deployment tool. But they are great then when <em>you build them by yourself</em>.</p>

<p>In my case these small machines are not used for highly reliable production infrastructure. They are used for background experiments, data collection from remote locations, light uncritical automation and all the odd jobs for which one does not want to waste a full workstation that produces more heat, more fan noise and of course more power consumption over <code class="language-plaintext highlighter-rouge">24/7</code>. A <a href="https://amzn.to/4pS6Gkw">RaspberryPi 5 with 4 GB of RAM</a> is already sufficient for small local language model tasks such as <code class="language-plaintext highlighter-rouge">llama3.2:3b</code>, <code class="language-plaintext highlighter-rouge">gemma3:1b</code> or <code class="language-plaintext highlighter-rouge">llama3.2-abliterated</code>, and it also works well as a tiny embeddings server for <code class="language-plaintext highlighter-rouge">mxbai-embed-large:latest</code> when all one wants is local vector indexing and similarity search. The old <code class="language-plaintext highlighter-rouge">RaspberryPi 3B+</code> is much weaker, but it is still perfectly usable for the control plane around such experiments.</p>

<p>I also prefer to keep the software stack uniform where possible. On these devices I run <a href="https://www.raspberrypi.com/software/operating-systems/">RaspberryPi OS</a> because I want to manage both the <code class="language-plaintext highlighter-rouge">Pi 5</code> and the <code class="language-plaintext highlighter-rouge">Pi 3B+</code> with the same automation, and on the <code class="language-plaintext highlighter-rouge">Pi 5</code> I currently need that operating system due to hardware support considerations. Otherwise I would very likely have chosen <a href="https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/">FreeBSD</a>. Once one accepts that the <code class="language-plaintext highlighter-rouge">Pi 3B+</code> is an lightweight machine and not a miracle worker, the question is no longer whether it is elegant to run <code class="language-plaintext highlighter-rouge">n8n</code> there. The question is simply whether it works well enough. The answer is <em>yes</em>, provided one handles memory carefully <em>during installation</em>.</p>

<p>As of today, the official <code class="language-plaintext highlighter-rouge">n8n</code> documentation <a href="https://docs.n8n.io/deploy/host-n8n/install-options/install-with-npm/">still supports installation via <code class="language-plaintext highlighter-rouge">npm</code></a> and the documentation states that <code class="language-plaintext highlighter-rouge">n8n</code> requires a <code class="language-plaintext highlighter-rouge">Node.js</code> version between <code class="language-plaintext highlighter-rouge">20.19</code> and <code class="language-plaintext highlighter-rouge">24.x</code> inclusive. So the <code class="language-plaintext highlighter-rouge">npm</code> route is not some unsupported hack. The real obstacle is memory pressure during install, not a fundamental incompatibility.</p>

<ul>
  <li><a href="#why-the-pi-3b-is-annoying-for-npm">Why the Pi 3B+ Is Annoying For NPM</a></li>
  <li><a href="#base-system-preparation">Base System Preparation</a></li>
  <li><a href="#creating-temporary-real-swap">Creating Temporary Real Swap</a></li>
  <li><a href="#installing-nodejs-and-n8n-with-nvm">Installing Node.js and n8n with nvm</a></li>
  <li><a href="#creating-a-small-startup-wrapper">Creating a Small Startup Wrapper</a></li>
  <li><a href="#environment-configuration">Environment Configuration</a></li>
  <li><a href="#systemd-service">Systemd Service</a></li>
  <li><a href="#first-start-and-autostart">First Start and Autostart</a></li>
  <li><a href="#notes-on-data-sd-cards-and-security">Notes on Data, SD Cards and Security</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<p><img src="/assets/images/jpg/raspi3bn8n001.jpg" alt="A nearly empty RaspberryPi Rack" /></p>

<h2 id="why-the-pi-3b-is-annoying-for-npm">Why the Pi 3B+ Is Annoying For NPM</h2>

<p>The <a href="https://amzn.to/3RKueuG">RaspberryPi 3B+</a> has only 1 GB of RAM. That is enough for many long-running low-intensity services, but it is not enough to comfortably install large modern JavaScript applications in the most naive way. The operating system itself already needs some memory, the package manager needs memory, the <a href="https://nodejs.org/en">node</a> JavaScript runtime needs massive loads of memory and the <a href="https://www.npmjs.com/">npm</a> dependency tree of <a href="https://n8n.io/">n8n</a> is not exactly tiny.</p>

<p>The particularly annoying part is that on current <a href="https://www.raspberrypi.com/software/operating-systems/">RaspberryPi OS</a> releases one often begins with <code class="language-plaintext highlighter-rouge">zram</code>-based swap or at least a swap setup that heavily prefers compressed RAM first. That is sensible for many workloads because it reduces wear on SD cards (if you dont do this you will experience them dying periodically), but it does not help enough when the actual problem is a large installation step that briefly needs more memory than the machine physically has. In that case you usually need a temporary real swap file with enough space to let the install finish at all.</p>

<p>This is one of those situations where I would rather accept a somewhat ugly one-time installation procedure than pretend the hardware is unusable. Once <a href="https://n8n.io">n8n</a> is installed and configured conservatively, runtime memory usage is much easier to control than the installation peak.</p>

<h2 id="base-system-preparation">Base System Preparation</h2>

<p>I assume a plain <a href="https://www.raspberrypi.com/software/operating-systems/">RaspberryPi OS</a> installation with SSH access already working.</p>

<p>I first updated the system and installed a few basic tools:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt update
sudo apt full-upgrade -y
sudo apt autoremove -y
sudo apt clean

sudo apt install -y \
    git \
    curl \
    ca-certificates \
    build-essential \
    python3 \
    python3-venv \
    python3-pip \
    sqlite3 \
    rsync
</code></pre></div></div>

<p>For <code class="language-plaintext highlighter-rouge">n8n</code> itself I created a dedicated system user and a dedicated directory tree below <code class="language-plaintext highlighter-rouge">/srv/n8n</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo adduser \
    --system \
    --group \
    --home /srv/n8n \
    --shell /bin/bash \
    n8n

sudo install -d -o n8n -g n8n -m 0750 /srv/n8n
sudo install -d -o n8n -g n8n -m 0750 /srv/n8n/data
sudo install -d -o n8n -g n8n -m 0750 /srv/n8n/bin
sudo install -d -o root -g n8n -m 0750 /etc/n8n
</code></pre></div></div>

<p>If you have a USB SSD or even a decent USB flash device available, it is better to place <code class="language-plaintext highlighter-rouge">/srv/n8n/data</code> there instead of on the SD card. The installation itself is already write-heavy enough; there is no need to additionally burn through the card with workflow history if one can avoid it.</p>

<h2 id="creating-temporary-real-swap">Creating Temporary Real Swap</h2>

<p>This is the step that makes the difference between a frustrating evening and a successful install. I created a temporary 2 GB swap file, enabled it with higher priority, and disabled the in-memory <code class="language-plaintext highlighter-rouge">zram</code> swap device <em>for the duration of the installation</em>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon --priority 10 /swapfile
sudo swapoff /dev/zram0
</code></pre></div></div>

<p>If your system does not currently use <code class="language-plaintext highlighter-rouge">/dev/zram0</code>, simply skip the last line. The point is not the exact device name, the point is that during the installation I wanted the machine to rely on actual swap space instead of first trying to compress itself to death inside already scarce RAM.</p>

<p><strong>This is not something I would necessarily keep forever</strong>. It is a pragmatic installation measure. After the system is running, you can decide whether to restore the previous swap arrangement, keep the swap file, or move swap to more suitable storage.</p>

<h2 id="installing-nodejs-and-n8n-with-nvm">Installing Node.js and n8n with nvm</h2>

<p>The official <a href="https://n8n.io">n8n</a> npm installation path is straightforward. The tricky part on a RaspberryPi 3B+ is doing it in a memory-conscious way. I switched to the <code class="language-plaintext highlighter-rouge">n8n</code> user, installed <code class="language-plaintext highlighter-rouge">nvm</code>, selected <code class="language-plaintext highlighter-rouge">Node.js</code> version 22, reduced concurrency a bit and limited the JavaScript heap during install. Note that the following example shows piping an install script into <code class="language-plaintext highlighter-rouge">bash</code>. <strong>This is bad practice</strong> but follows official documentation. In reality first fetch the script, verify it’s content (read and understand it) and <em>then</em> perform the installation process.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -u n8n -H bash

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash

export NVM_DIR="/srv/n8n/.nvm"
. "${NVM_DIR}/nvm.sh"

nvm install 22
nvm alias default 22
nvm use 22

# Verify installed versions
node --version
npm --version

export NODE_OPTIONS="--max-old-space-size=768"
export UV_THREADPOOL_SIZE=1
npm config set maxsockets 1

nice -n 19 ionice -c3 npm install --no-audit --no-fund --progress=false --global n8n

# Verify installed version
n8n --version
exit
</code></pre></div></div>

<p>There are two important remarks here.</p>

<p>First, the install can take a <em>long time</em> (read: many hours, let it running overnight)  on this hardware. That is normal. This is not the moment to become impatient and interrupt it after half an hour. On my setup it finally succeeded overnight once sufficient real swap was available. Second, this does not contradict the common observation that the installation often fails on a RaspberryPi 3B+. It does fail if one simply starts from the default memory situation and hopes for the best.</p>

<h2 id="creating-a-small-startup-wrapper">Creating a Small Startup Wrapper</h2>

<p>Because <code class="language-plaintext highlighter-rouge">n8n</code> was installed under <code class="language-plaintext highlighter-rouge">nvm</code>, I used a tiny wrapper script that sets the environment and then launches the service:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo tee /srv/n8n/bin/start-n8n &gt;/dev/null &lt;&lt;EOF
#!/bin/bash
set -euo pipefail

export HOME=/srv/n8n
export NVM_DIR=/srv/n8n/.nvm

. "${NVM_DIR}/nvm.sh"
nvm use --silent default

exec n8n start
EOF

sudo chown n8n:n8n /srv/n8n/bin/start-n8n
sudo chmod 0750 /srv/n8n/bin/start-n8n
</code></pre></div></div>

<p>This is intentionally boring. Boring startup scripts are good startup scripts.</p>

<h2 id="environment-configuration">Environment Configuration</h2>

<p>My runtime configuration lives in <code class="language-plaintext highlighter-rouge">/etc/n8n/n8n.env</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>N8N_USER_FOLDER=/srv/n8n/data

N8N_HOST=examplehost.example.net
N8N_PORT=5678
N8N_PROTOCOL=http

N8N_ENCRYPTION_KEY=replace-this-with-a-long-random-string

GENERIC_TIMEZONE=Europe/Vienna
TZ=Europe/Vienna

N8N_DIAGNOSTICS_ENABLED=false
N8N_VERSION_NOTIFICATIONS_ENABLED=true

EXECUTIONS_CONCURRENCY=2

EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=168
EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000

EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_PROGRESS=false

N8N_LOG_LEVEL=info

NODE_OPTIONS=--max-old-space-size=512
</code></pre></div></div>

<p>The two settings that matter most for a small machine are <code class="language-plaintext highlighter-rouge">EXECUTIONS_CONCURRENCY=2</code> and the reduced heap size in <code class="language-plaintext highlighter-rouge">NODE_OPTIONS</code>. They reflect the fact that this is not a large multi-user automation cluster. It is a tiny background worker for experiments and light workflows.</p>

<p>If you migrate an existing <code class="language-plaintext highlighter-rouge">.n8n</code> directory from another machine, remember that the <code class="language-plaintext highlighter-rouge">N8N_ENCRYPTION_KEY</code> has to match the one used previously or existing credentials will not decrypt correctly. That detail is easy to overlook and mildly irritating when one does.</p>

<p>Depending on your workflows, you may also need additional settings:</p>

<ul>
  <li>If you expose <code class="language-plaintext highlighter-rouge">n8n</code> only over plain HTTP inside a trusted internal network segment, <code class="language-plaintext highlighter-rouge">N8N_SECURE_COOKIE=false</code> may be required. If you place it behind proper HTTPS, do not do that.</li>
  <li>If you use file read/write nodes, <code class="language-plaintext highlighter-rouge">N8N_RESTRICT_FILE_ACCESS_TO</code> can be set to a specific allowed directory. According to the <code class="language-plaintext highlighter-rouge">n8n</code> documentation this variable is the intended way to restrict disk access. I would recommend pointing it to a dedicated path rather than disabling restrictions globally.</li>
</ul>

<h2 id="systemd-service">Systemd Service</h2>

<p>The following unit file worked well for me:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=n8n Workflow Automation
Documentation=https://docs.n8n.io/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=n8n
Group=n8n

WorkingDirectory=/srv/n8n
EnvironmentFile=/etc/n8n/n8n.env

ExecStart=/srv/n8n/bin/start-n8n

Restart=on-failure
RestartSec=10
TimeoutStopSec=120
KillSignal=SIGTERM

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/n8n

UMask=0077

[Install]
WantedBy=multi-user.target
</code></pre></div></div>

<p>Save this as <code class="language-plaintext highlighter-rouge">/etc/systemd/system/n8n.service</code>, then reload <code class="language-plaintext highlighter-rouge">systemd</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo systemctl daemon-reload
</code></pre></div></div>

<p>The hardening options above are still simple enough not to become a maintenance burden. At the same time they avoid the all-too-common habit of running everything with broad filesystem access just because it is convenient during the first test.</p>

<h2 id="first-start-and-autostart">First Start and Autostart</h2>

<p>Once the installation has completed, the service can be started and enabled:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo systemctl enable --now n8n
</code></pre></div></div>

<p>If you want to inspect the logs immediately afterwards:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>journalctl -u n8n -n 100 --no-pager
</code></pre></div></div>

<p>If the service is not starting, check the obvious things first:</p>

<ul>
  <li>Does <code class="language-plaintext highlighter-rouge">/srv/n8n/bin/start-n8n</code> exist and belong to the <code class="language-plaintext highlighter-rouge">n8n</code> user?</li>
  <li>Does <code class="language-plaintext highlighter-rouge">/etc/n8n/n8n.env</code> contain valid shell-style assignments?</li>
  <li>Does the <code class="language-plaintext highlighter-rouge">N8N_ENCRYPTION_KEY</code> contain special shell characters that should have been quoted?</li>
  <li>Did the <code class="language-plaintext highlighter-rouge">npm install</code> actually complete, or was it interrupted during the night because somebody got impatient?</li>
</ul>

<p>I also strongly recommend a full reboot test immediately after initial setup:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo reboot
</code></pre></div></div>

<p>Then reconnect via SSH and confirm:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl status n8n
</code></pre></div></div>

<p>If it survives a reboot, the machine is usually good enough for the kind of low-intensity background usage I described above.</p>

<h2 id="notes-on-data-sd-cards-and-security">Notes on Data, SD Cards and Security</h2>

<p>A setup like this is useful, but one should remain honest about its limitations.</p>

<p>The primary weakness is still the storage medium. <strong>SD cards are not ideal for persistent databases and long-running write-heavy services</strong>. <code class="language-plaintext highlighter-rouge">n8n</code> is manageable because one can reduce retained execution data aggressively, move the user folder to external storage and treat the whole machine as somewhat disposable infrastructure. <strong>Regular backups are not optional here</strong>. They are part of the design.</p>

<p>The second weakness is that <code class="language-plaintext highlighter-rouge">SQLite</code> is perfectly fine for a small one-user instance, but this is not the machine on which I would build a bigger queue-based automation platform. The official <code class="language-plaintext highlighter-rouge">n8n</code> documentation still uses <code class="language-plaintext highlighter-rouge">SQLite</code> by default for self-hosted installs, which is entirely reasonable for this sort of tiny deployment. It just means one should keep expectations aligned with the hardware.</p>

<p>The third weakness is operational temptation. Once <code class="language-plaintext highlighter-rouge">n8n</code> is running on a cute little low-power board, it becomes very easy to keep attaching more things to it until it quietly becomes important infrastructure. I would avoid that. My use case is deliberately modest: data mining, automation glue, background experiments, and some integration work around local or remote AI systems. When the machine dies, it should be annoying, not catastrophic.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Installing <code class="language-plaintext highlighter-rouge">n8n</code> on a <a href="https://amzn.to/3RKueuG">RaspberryPi 3B+</a> via <code class="language-plaintext highlighter-rouge">npm</code> is entirely possible. The part that usually fails is not conceptual compatibility but memory pressure during installation. With a temporary real swap file, a dedicated service account, a conservative runtime configuration and a little patience, the old board is still useful.</p>

<p>I would not recommend this as the default path for everyone, and I definitely would not recommend it for serious production use. But if you already have a <code class="language-plaintext highlighter-rouge">RaspberryPi 3B+</code>, want a tiny always-on workflow engine and are comfortable accepting the limits of <code class="language-plaintext highlighter-rouge">1 GB</code> RAM and SD-card-backed storage, there is no need to pretend that only Docker on larger hardware counts as a valid solution.</p>

<p>In my opinion that is exactly where these small machines still shine: not as replacements for proper servers, but as cheap, quiet and sufficiently capable companions for the slightly strange experimental ecosystems many of us accumulate over time.</p>

<h2 id="references">References</h2>

<ul>
  <li>Hardware
    <ul>
      <li><a href="https://amzn.to/3RKueuG">RaspberryPi 3B+</a> with 1GB RAM. These had been very cheap and are still somewhat affordable as of today</li>
      <li><a href="https://amzn.to/4pS6Gkw">RaspberryPi 5, 4GB version</a>, offering PCI Express and allowing for <a href="https://amzn.to/4fL0A0A">NVMe SSDs</a></li>
      <li><a href="https://amzn.to/4bPxmwe">RaspberryPi 5, 16GB version</a>, the currently largest version of the RaspberryPi</li>
    </ul>
  </li>
  <li>Software
    <ul>
      <li><a href="https://n8n.io/">n8n workflow automation</a></li>
      <li><a href="https://nodejs.org/en">NodeJS JavaScript runtime</a></li>
      <li><a href="https://www.npmjs.com/">npm</a>, the node.js package manager</li>
      <li><a href="https://www.raspberrypi.com/software/operating-systems/">RaspberryPi OS</a></li>
      <li><a href="https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/">FreeBSD for aarch64</a>, the RPI images are directly useable</li>
    </ul>
  </li>
  <li>Documentation
    <ul>
      <li><a href="https://docs.n8n.io/deploy/host-n8n/install-options/install-with-npm/">n8n Instlalation manual using npm</a></li>
      <li><a href="https://docs.n8n.io/deploy/host-n8n/">n8n self-hosting overview</a></li>
      <li><a href="https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables/security/">n8n security-related environment variables</a></li>
      <li><a href="https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.readwritefile/">n8n file access restriction variable</a></li>
      <li><a href="https://docs.n8n.io/deploy/host-n8n/configure-n8n/choose-n8ns-database/">n8n database defaults</a></li>
      <li><a href="https://www.raspberrypi.com/news/trixie-the-new-version-of-raspberry-pi-os/">RaspberryPi OS - note on default <code class="language-plaintext highlighter-rouge">zram</code> swap</a></li>
    </ul>
  </li>
</ul>]]></content><author><name>tsp</name></author><category term="RaspberryPi" /><category term="n8n" /><category term="System administration" /><category term="Tutorial" /><category term="Artificial Intelligence" /><category term="Home automation" /><summary type="html"><![CDATA[A practical guide for installing n8n via npm on a RaspberryPi 3B+ with only 1 GB of RAM. This article explains why the usual advice to just use Docker is not always helpful, how to work around the memory bottleneck with temporary swap, and how to run n8n reliably enough for small experimental workflows on low-power hardware.]]></summary></entry><entry><title type="html">Physical Review Research: Ghost imaging with free electron-photon pairs</title><link href="https://www.tspi.at/2026/07/28/ghost.html" rel="alternate" type="text/html" title="Physical Review Research: Ghost imaging with free electron-photon pairs" /><published>2026-07-28T00:00:00+02:00</published><updated>2022-08-21T00:21:52+02:00</updated><id>https://www.tspi.at/2026/07/28/ghost</id><content type="html" xml:base="https://www.tspi.at/2026/07/28/ghost.html"><![CDATA[]]></content><author><name>Sergei Bogdanov, Alexander Preimesberger, Harsh Mishra, Dominik Hornof, Thomas Spielauer, Florian Thajer, Max Maurer, Pia Falb, Leo Stöger, Thomas Schachinger, Friedrich Bleicher, Michael S. Seifner, Isobel C. Bicket, Philipp Haslinger</name></author><summary type="html"><![CDATA[Published in Physical Review Research: Coincidence imaging, also known as ghost imaging, is a technique that exploits correlations between two particles to reconstruct information about a specimen. The particle that relays the spatial information about the object remains completely noninteracting, while the particle used to probe the object is not spatially resolved. While ghost imaging has been primarily implemented on photonic platforms, applying it to particles with fundamentally different properties opens up scientific directions. Mixing massive, charged electrons with massless, neutral photons introduces a hybrid architecture that unites two fundamental microscopic platforms, each serving as a cornerstone of highly advanced imaging systems. In this work, we investigate coincidence imaging using electron–cathodoluminescence photon pairs generated within a transmission electron microscope. Utilizing a custom-built free-space cathodoluminescence setup, we demonstrate two-dimensional ghost imaging of complex patterns. We are able to obtain a spatial resolution down to 2um, paving the way for adaptation of quantum-enhanced imaging techniques from photonic quantum optics to electron microscopy.]]></summary></entry><entry><title type="html">Stability Is Key: How Lasting Systems Create Real Progress</title><link href="https://www.tspi.at/2026/07/26/stabilityiskey.html" rel="alternate" type="text/html" title="Stability Is Key: How Lasting Systems Create Real Progress" /><published>2026-07-26T00:00:00+02:00</published><updated>2026-07-26T11:48:31+02:00</updated><id>https://www.tspi.at/2026/07/26/stabilityiskey</id><content type="html" xml:base="https://www.tspi.at/2026/07/26/stabilityiskey.html"><![CDATA[<blockquote>
  <p>Progress is not the amount of novelty produced. Progress is the amount of useful capability that remains available.</p>
</blockquote>

<p>This article argues for a rather simple idea that is strangely unpopular in modern technical culture: <strong>stability is the basis on which nearly all real progress is built</strong>.</p>

<p>The fashionable picture looks different. Technology is said to move extremely fast. Companies are expected to release one feature after another, research projects are expected to produce breakthroughs inside one or two funding periods and software is expected to replace its interfaces, frameworks and sometimes its entire conceptual foundation every few years. If this creates permanent work for everyone involved it is accepted as the natural condition of a dynamic world. One frequently hears that stagnation is death, that one has to move fast and that anybody who objects simply failed to keep up.</p>

<p>I think this picture is fundamentally wrong.</p>

<p>There is of course change. There is exploration, there are prototypes and sometimes an experiment has to be performed quickly before one even knows whether an idea is useful. None of this is disputed here. But an experiment is not yet infrastructure, a prototype is not yet a product and a published result is not yet a capability society can rely on. Exploration only discovers a possible path. The actual construction of progress begins afterwards, when the useful part is made reproducible, compatible, maintainable and sufficiently stable that other people can build upon it <em>without being forced to rebuild their own work every few months</em>.</p>

<p>This distinction is essential. A rapidly changing prototype may be valuable for a short time because it produces information. If its findings are never consolidated into something stable, however, its long-term contribution approaches zero. It has consumed human lifetime and perhaps produced an interesting demonstration, but it has not increased the stock of dependable tools available to everyone else.</p>

<p><strong>The best technology is therefore not necessarily the technology with the largest number of releases. It is the technology one can still use in thirty years.</strong> It may receive security fixes, performance improvements and additional features throughout this time. Stability does not require abandonment. It requires the opposite: maintenance without betrayal of the existing contract.</p>

<p>This article develops this argument from four directions. First, it gives a more useful definition of progress and a simple model for the cost of instability. Second, it looks at computing systems that are usually described as rapidly moving even though their success depends on interfaces preserved for decades. Third, it examines scientific and industrial breakthroughs whose apparently sudden arrival was built from long, slow chains of work. Finally, it turns the argument into practical rules for engineering and management.</p>

<p>The central thought extends an earlier article about automation and stability[<a href="#ref1">1</a>]. <em>Automation compounds human time only when the automated environment remains stable</em>. The same is true more generally: knowledge, software, machines and institutions compound only when yesterday’s useful work remains usable tomorrow.</p>

<ul>
  <li><a href="#motion-is-not-progress">Motion Is Not Progress</a></li>
  <li><a href="#stability-is-not-stagnation">Stability Is Not Stagnation</a></li>
  <li><a href="#the-hidden-economy-of-breakage">The Hidden Economy of Breakage</a></li>
  <li><a href="#the-fast-moving-computer-that-barely-moves">The Fast-Moving Computer That Barely Moves</a></li>
  <li><a href="#standards-are-machines-for-accumulating-work">Standards Are Machines for Accumulating Work</a></li>
  <li><a href="#apparently-sudden-technology">Apparently Sudden Technology</a></li>
  <li><a href="#science-moves-at-the-speed-of-evidence">Science Moves at the Speed of Evidence</a></li>
  <li><a href="#the-physical-world-is-built-from-frozen-decisions">The Physical World Is Built from Frozen Decisions</a></li>
  <li><a href="#exploration-has-one-job">Exploration Has One Job</a></li>
  <li><a href="#why-organizations-reward-instability">Why Organizations Reward Instability</a></li>
  <li><a href="#measuring-what-actually-matters">Measuring What Actually Matters</a></li>
  <li><a href="#how-to-build-technology-for-thirty-years">How to Build Technology for Thirty Years</a></li>
  <li><a href="#typical-objections">Typical Objections</a></li>
  <li><a href="#a-stable-world-moves-further">A Stable World Moves Further</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/stability001.png" alt="" /></p>

<h2 id="motion-is-not-progress">Motion Is Not Progress</h2>

<h3 id="the-stock-of-retained-capability">The stock of retained capability</h3>

<p>Most discussions about technological progress use quantities that are easy to observe. Features released per quarter, commits per day, papers per year, products launched, patents filed and funding acquired all measure activity. Activity is visible and can be attributed to the people performing it. Stability, in contrast, is visible mostly through the absence of events. A protocol that keeps working does not produce an emergency meeting. A machine that requires no manual intervention does not create an impressive incident report. A program that compiles unchanged after fifteen years rarely becomes a news story.</p>

<p>This creates the first important measurement error. <strong>Activity is a flow, while progress is a stock.</strong></p>

<p>Let $F(t)$ be the rate at which useful new capability is created. Let $L(t)$ be the rate at which existing capability is lost because interfaces disappear, formats become unreadable, dependencies break or knowledge becomes unusable. The retained stock $P(t)$ can then be written in the deliberately simple form</p>

[
P(t) = P(0) + \int_0^t \left(F(\tau)-L(\tau)\right)\ \mathrm{d}\tau
]

<p>An organization that produces ten units of capability and destroys nine has moved a great deal but progressed by only one. Another organization that produces three units and preserves all three has progressed three times as far. If only feature output is measured, the first organization looks more dynamic. If retained capability is measured, the second is clearly better.</p>

<p>This is not merely a metaphor for software. A scientific result that cannot be reproduced has a large publication flow and a small contribution to retained knowledge. A laboratory instrument that becomes unusable because its control software requires an abandoned cloud service loses physical capability even though the hardware still exists. A public database whose format changes without a reliable migration path destroys part of the value previously invested in collecting the data.</p>

<p>The relevant question is therefore not <em>“How much changed?”</em> but <em>“What can be done now that could not be done before, without losing what could already be done?”</em></p>

<h3 id="local-speed-and-global-speed">Local speed and global speed</h3>

<p>The second measurement error is a confusion of system boundaries.</p>

<p>Suppose a library maintainer removes an old interface and replaces it with a cleaner one. The local change may take two days. From the maintainers viewpoint the project moved quickly. If ten thousand users each need twenty minutes to understand the change, update their code, test it and deploy it, the global cost is more than 3300 hours. The local team has saved perhaps a week and consumed roughly two working years elsewhere.</p>

<p>For an upstream component with downstream users $i=1,\ldots,N$, a first approximation of the cost of a breaking change is</p>

[
C_{\mathrm{break}} = C_{\mathrm{upstream}} +
\sum_{i=1}^{N} p_i
\left(C_{\mathrm{understand},i}+
C_{\mathrm{modify},i}+
C_{\mathrm{test},i}+
C_{\mathrm{deploy},i}\right)
]

<p>where $p_i$ is the probability that the changed contract affects user $i$. Even if every individual downstream cost is small, the sum may be enormous. Most organizations do not see this sum because the work appears in other budgets, other companies and the private time of users. Instability is therefore unusually easy to externalize.</p>

<p>Empirical studies of API evolution confirm that breaking changes propagate into real client work[<a href="#ref8">8</a>,<a href="#ref9">9</a>]. The exact measured fraction varies with language, ecosystem and what researchers can observe. Private code is usually invisible, behavioral incompatibilities are harder to detect than syntactic ones and abandoned clients disappear from samples. But the structural fact does not depend on a particular percentage: a single upstream decision can create work at every downstream edge.</p>

<p>The same applies inside organizations. A configuration change may be cheap for the team that owns a service and expensive for every automation pipeline that consumes it. Renaming a field may look like cleanup in the producing repository and become weeks of coordination in the whole system.</p>

<blockquote>
  <p><strong>Local velocity is not global progress.</strong></p>
</blockquote>

<h3 id="rework-is-negative-production">Rework is negative production</h3>

<p>When an engineer spends a day adapting working software to a new interface without gaining a new capability, <em>no progress has been made during that day</em>. Existing capability has merely been restored.</p>

<p>This work is often counted as development because code was changed. In economic terms it is closer to replacing a road that another department dug up yesterday. The replacement may be technically necessary after the damage and the engineers performing it may do excellent work, but it does not follow that the damage was productive.</p>

<p>Technical debt studies repeatedly find substantial developer time lost to avoidable friction. One longitudinal study reported an average of roughly 23% of development time wasted because of technical debt[<a href="#ref3">3</a>]. The exact number should not be treated as a universal constant, but its scale matters. It shows that apparently productive organizations can consume a large share of their capacity merely overcoming conditions created by earlier development.</p>

<p>Googles site reliability terminology calls manual, repetitive, automatable work without enduring value <em>“toil”</em>[<a href="#ref7">7</a>]. This is a useful definition far beyond operations. Compatibility work forced by avoidable churn is dependency toil. Re-entering data after a format change is user toil. Rewriting a build configuration because a tool changed its vocabulary is toolchain toil. <em>None of it is a law of nature</em>.</p>

<p>The phrase <em>“that is life”</em>, that one often hears in this context, is therefore not an explanation. It is an admission that the cost has become culturally invisible.</p>

<h2 id="stability-is-not-stagnation">Stability Is Not Stagnation</h2>

<h3 id="what-has-to-remain-stable">What has to remain stable</h3>

<p>Stability does not mean that every line of source code must be frozen. It does not mean refusing security fixes, preserving known defects forever or rejecting additional features. It means preserving contracts. A contract may be:</p>

<ul>
  <li>a function signature and its documented behavior</li>
  <li>a network message and its interpretation</li>
  <li>a file format</li>
  <li>an instruction set</li>
  <li>a connector geometry</li>
  <li>a unit of measurement</li>
  <li>a command-line invocation</li>
  <li>a physical mounting pattern</li>
  <li>a documented operational procedure</li>
  <li>or a promise about the continued availability of a feature</li>
</ul>

<p>The internal implementation behind such a contract can change radically. A database may replace algorithms, add indexes, improve its query planner and harden its storage layer while continuing to open old files. A processor may use completely different microarchitecture while executing an established instruction set. A web browser may replace its rendering engine while still presenting an old page.</p>

<p>This is the most productive kind of change: freedom inside, stability at the boundary.</p>

<p>The distinction is familiar in systems engineering. NASA describes configuration management as the discipline that preserves product integrity by controlling changes to functional and physical characteristics across the life cycle[<a href="#ref4">4</a>]. The point is not to prevent every modification. It is to ensure that a modification is beneficial, understood and introduced without uncontrolled adverse consequences.</p>

<h3 id="maintenance-is-active-work">Maintenance is active work</h3>

<p>A stable system is not a neglected system.</p>

<p>Real environments change. Security vulnerabilities are discovered. Hardware fails. Regulations change. New scale reveals old assumptions. Lehmans work on software evolution observes that systems embedded in a changing real world have to be adapted or they become progressively less satisfactory, while their complexity tends to increase unless work is explicitly performed to control it[<a href="#ref2">2</a>].</p>

<p>This does not support arbitrary interface churn. It supports maintenance.</p>

<p>Maintenance may require replacing an implementation, removing a vulnerability, extending an address field, adding a negotiation mechanism or writing a compatibility layer. The stable approach asks how the old contract can remain available while these improvements are introduced. Sometimes the answer is an adapter. Sometimes it is a new optional field, an additional operation, a capability query or a parallel protocol. Sometimes the old behavior can be reproduced safely on top of a completely new core.</p>

<p>The work is often less glamorous than a clean break. It may require understanding decisions made by people who left the project years ago. It may preserve awkward details. But this work protects the accumulated investment of every existing user.</p>

<blockquote>
  <p><strong>Stability is a continuing service performed for the future.</strong></p>
</blockquote>

<h3 id="compatibility-has-more-than-one-direction">Compatibility has more than one direction</h3>

<p>Several forms of compatibility are often mixed together:</p>

<ul>
  <li><strong>Backward compatibility</strong> means a newer implementation can consume old inputs or serve old clients.</li>
  <li><strong>Forward compatibility</strong> means an older implementation can tolerate some newer inputs, usually by ignoring what it does not understand.</li>
  <li><strong>Source compatibility</strong> means old source code can still be compiled.</li>
  <li><strong>Binary compatibility</strong> means an old executable or library client still runs.</li>
  <li><strong>Behavioral compatibility</strong> means the same input retains the meaning on which users relied.</li>
  <li><strong>Data compatibility</strong> means stored information remains readable and correctly interpreted.</li>
  <li><strong>Physical compatibility</strong> means components continue to fit and communicate mechanically or electrically.</li>
</ul>

<p>A project can preserve one while breaking another. A library may compile old code yet subtly alter behavior. A file may remain parseable while changing units. A connector may fit physically while supplying an unsafe voltage. Long-term engineering has to state the contract precisely rather than using the word compatibility as decoration.</p>

<p>For the purpose of this article, stability means that established useful behavior remains available unless preserving it would create a demonstrable and disproportionate danger. <em>“We prefer the new design”</em> is not such a danger. <em>“The old path requires maintenance”</em> is not such a danger either; that maintenance is part of the cost accepted when other people were encouraged to build on the path.</p>

<h3 id="major-versions-do-not-remove-the-cost">Major versions do not remove the cost</h3>

<p>Semantic versioning is useful communication. A major version number can warn users that an interface changed. <strong>It does not make the change free.</strong></p>

<p>If a bridge is closed, a sign warning drivers about the closure is better than no sign. The sign does not restore the bridge. In the same way, calling a release <code class="language-plaintext highlighter-rouge">5.0</code> rather than <code class="language-plaintext highlighter-rouge">4.9</code> may prevent surprise, but every downstream migration still consumes human lifetime.</p>

<p>Versioning is therefore a last line of communication, not a license to break. A well-maintained long-lived product should be proud of how rarely it needs a destructive major transition. If a new architecture is truly incompatible, it can often be offered beside the existing one rather than being declared its immediate replacement.</p>

<blockquote>
  <p><strong>The strongest compatibility policy is not “we label breakage correctly”. It is “we avoid requiring it”.</strong></p>
</blockquote>

<h2 id="the-hidden-economy-of-breakage">The Hidden Economy of Breakage</h2>

<h3 id="the-dependency-graph">The dependency graph</h3>

<p>Modern technical systems are graphs, not lists.</p>

<p>An application depends on libraries. Libraries depend on runtimes. Runtimes depend on operating-system interfaces. Services depend on protocols, identity providers, databases, message formats and deployment systems. Scientific software additionally depends on numerical conventions, calibration files, instrument firmware, data schemas and analysis environments.</p>

<p>If node $j$ has downstream dependants $D(j)$, the immediate adaptation work caused by changing its contract is distributed across $D(j)$. But each dependant may itself expose another contract. A rushed adaptation can therefore propagate defects, version constraints and temporary workarounds into further layers.</p>

<blockquote>
  <p><strong>This is why instability grows faster than the number of changed lines suggests. The cost follows graph reachability.</strong></p>
</blockquote>

<p>The designers of large systems have understood this for a long time. The <code class="language-plaintext highlighter-rouge">System/360</code> architecture explicitly separated architecture from implementation so machines with very different performance could present a compatible programming model[<a href="#ref18">18</a>]. Internet architecture hid routing evolution from hosts and required interoperability across diverse networks[<a href="#ref14">14</a>]. These were not aesthetic choices. They were methods for preventing the dependency graph from becoming globally synchronized around every local change.</p>

<h3 id="the-pinning-trap">The pinning trap</h3>

<p>When updates are risky, users pin versions. <strong>This is rational locally and dangerous globally.</strong></p>

<p>A pinned dependency prevents an unexpected breaking change today. It also prevents automatic delivery of security fixes and reliability improvements tomorrow. The user is forced to choose between known stagnation and unknown migration work. Eventually several pinned layers interact, and upgrading one requires upgrading all of them together. The cost grows until it is postponed again.</p>

<blockquote>
  <p><strong>This is sometimes presented as evidence that old software is the problem. Usually it is evidence that the upgrade path is untrustworthy.</strong></p>
</blockquote>

<p>In a stable ecosystem, users can accept routine fixes because the contract remains intact. The new implementation becomes safer and faster while the client remains unchanged. Automatic upgrading becomes possible precisely because compatibility reduces the expected migration cost.</p>

<p>Instability therefore blocks one of the main benefits often claimed for rapid delivery. A release pipeline can publish every hour, but users cannot safely consume those releases if each one may invalidate their systems.</p>

<h3 id="security-is-not-an-argument-for-churn">Security is not an argument for churn</h3>

<p>Security is frequently used as a universal excuse for incompatibility. Sometimes an old behavior really is inseparable from a vulnerability. Cryptographic algorithms can become inadequate, authentication methods can expose credentials and unsafe parsers may need stricter rules. In such cases a transition is necessary. But the existence of necessary exceptions does not justify making breakage routine.</p>

<p>A security transition can be designed with capability negotiation, dual operation, warnings, measured deprecation, migration tools and long overlap periods. Stored data can be rewritten automatically. Old clients can be isolated behind a compatibility gateway. The unsafe part can be removed while preserving unrelated behavior.</p>

<p>More importantly, unstable ecosystems create security problems of their own. If users fear updates, vulnerable versions remain deployed. If upgrades require manual rewrites, patches arrive late. If dependency trees cannot be reproduced, nobody knows which fix is actually running. Compatibility and security are therefore usually allies.</p>

<blockquote>
  <p><strong>The ideal security update is boring: the implementation changes, the vulnerability disappears and the user notices nothing.</strong></p>
</blockquote>

<h3 id="the-maintenance-paradox">The maintenance paradox</h3>

<p>There is a common claim that compatibility creates maintenance burden. This is true in the narrow sense. Supporting an old contract requires tests, documentation and sometimes additional code.</p>

<p>The missing question is: maintenance burden for whom?</p>

<p>Removing a compatibility path transfers work from the small upstream team, which knows the system, to a potentially enormous number of downstream teams, which do not. The total cost usually increases even if the producer’s cost decreases. What appears as simplification in one repository becomes complexity across the ecosystem.</p>

<p>This is especially perverse when the upstream organization is funded, visible and technically specialized while downstream users are researchers, small companies, public institutions or individuals. The party most able to solve the compatibility problem once delegates it to thousands of parties that must each solve a fragment.</p>

<blockquote>
  <p><strong>A stable project accepts an asymmetric responsibility: centralize compatibility work where knowledge is concentrated, so everybody else can keep using the system.</strong></p>
</blockquote>

<h2 id="the-fast-moving-computer-that-barely-moves">The Fast-Moving Computer That Barely Moves</h2>

<h3 id="a-machine-made-from-old-promises">A machine made from old promises</h3>

<p>A modern computer looks like a symbol of rapid technological change. Processor generations, operating-system releases, programming languages and web frameworks arrive continuously. Yet the machine is useful because its lower contracts move extremely slowly.</p>

<p>A program opens a file through operating-system calls whose conceptual ancestry reaches back decades. It sends data through TCP, standardized in 1981[<a href="#ref15">15</a>] and resolves names through DNS specifications published in 1987[<a href="#ref16">16</a>]. It may be written in C, use POSIX interfaces, store images in JPEG and produce PDF documents. <em>None of these foundations is new</em>. Their age is not a weakness. Their age is why they can be trusted as foundations.</p>

<blockquote>
  <p><strong>The visible surface moves because the invisible base does not.</strong></p>
</blockquote>

<h3 id="unix-posix-and-the-value-of-a-boring-interface">Unix, POSIX and the value of a boring interface</h3>

<p>Unix-like systems vary internally. Kernels have been rewritten, schedulers replaced, storage systems added and hardware architectures changed. Still, a large class of programs can be moved between systems because the process, file-descriptor and stream abstractions remain recognizable.</p>

<p>POSIX did not freeze every operating system into one implementation. It standardized a useful boundary[<a href="#ref17">17</a>]. This allowed competition below the boundary and portability above it. A shell script or C program written against a conservative subset can survive hardware generations and vendor changes because the contract is more durable than any one machine. Linux makes the distinction particularly clear. Its internal kernel interfaces are intentionally free to evolve, while userspace interfaces receive explicit stability treatment. The kernel documentation states that most interfaces such as system calls are expected never to change and always to remain available[<a href="#ref11">11</a>].</p>

<p>This policy is not evidence that Linux stopped developing. Linux changes constantly. It supports new processors, filesystems, security mechanisms and devices. Its success demonstrates the opposite relationship: extensive internal evolution is sustainable because established userspace contracts are protected.</p>

<h3 id="system360-and-the-invention-of-architectural-time">System/360 and the invention of architectural time</h3>

<p>IBM announced <code class="language-plaintext highlighter-rouge">System/360</code> in 1964 as a compatible family spanning a wide performance range. The original architecture paper emphasized strict upward and downward machine-language compatibility across models[<a href="#ref18">18</a>]. IBM’s historical account describes the replacement of several incompatible product lines by a unified software-compatible architecture[<a href="#ref19">19</a>].</p>

<p>The important invention was not only a particular instruction set. It was the separation of architecture from implementation.</p>

<p>Once software targeted the architecture rather than the physical details of one machine, processors could be made faster, larger or internally different without invalidating the software investment. Microcode could reproduce instructions. New models could extend capacity. Customers could move through the product family without rewriting everything.</p>

<p>This created architectural time: the lifetime of the interface became longer than the lifetime of each implementation.</p>

<p>Modern instruction-set families use the same principle. An old binary is not forced to understand superscalar execution, speculative pipelines, caches or semiconductor process changes. These are internal matters. The binary sees the established machine model.</p>

<blockquote>
  <p><strong>Compatibility is sometimes mocked as historical baggage. In this case the baggage is the accumulated software value that made the architecture worth buying.</strong></p>
</blockquote>

<h3 id="ethernet-and-the-internet">Ethernet and the Internet</h3>

<p>Ethernet evolved from early megabit networks to standards covering hundreds of gigabits per second while retaining a common media-access layer model[<a href="#ref20">20</a>]. Cables, transceivers and encoding methods changed. Switches replaced shared collision domains. The useful identity of Ethernet survived.</p>

<p>The Internet provides an even stronger example. <a href="https://datatracker.ietf.org/doc/html/rfc1122">RFC 1122</a> described requirements for hosts across diverse and changing networks and explicitly aimed to insulate host software from evolution in the routing architecture [14]. This separation allows applications to communicate without knowing whether packets cross copper, fiber, radio links, tunnels or satellite systems.</p>

<p>The Internet did not become globally useful because every layer was replaced whenever a better local idea appeared. It became useful because new links and applications could be added around a stable packet-delivery core.</p>

<p>Even the incomplete transition from IPv4 to IPv6 illustrates how strong installed contracts become. A theoretically cleaner replacement cannot simply order the world to migrate. Dual stacks, translation mechanisms and long coexistence are required because the network is not a laboratory. Billions of independently operated systems are part of the contract.</p>

<h3 id="the-web-preserves-its-mistakes">The web preserves its mistakes</h3>

<p>HTML is often described as fast-moving web technology. Browsers update frequently and new APIs are added continuously. Yet the webs unusual success depends on the expectation that old pages keep rendering.</p>

<p>The HTML design principles place support for existing content at the beginning of their compatibility discussion. <strong>They require implementations to handle documents that depend on older specifications, proprietary behavior and even historical error handling[<a href="#ref13">13</a>]</strong>. This is not theoretically beautiful. It is operationally brilliant.</p>

<p>The web treats deployed content as part of reality. A standards committee cannot edit every old page, so new browsers absorb much of the compatibility burden. The result is that documents created decades ago remain accessible while the same platform gains video, graphics, accessibility features, cryptography and complex applications.</p>

<p>A competing design that was conceptually cleaner but invalidated the existing web would have been less progressive, because it would have destroyed more retained capability than it created.</p>

<h3 id="sqlite-and-an-explicit-promise-to-the-future">SQLite and an explicit promise to the future</h3>

<p><strong>SQLite states an intention to be supported through 2050</strong> and promises backward compatibility for both its C API and on-disk format[<a href="#ref12">12</a>]. The formulation is unusually direct: applications written today should be able to link against future versions, and data stored today should remain accessible to future generations.</p>

<p>This is what a serious technology promise looks like.</p>

<p>SQLite still gains features and fixes. Query planning improves. Platforms change. The long-term contract constrains how these changes are introduced but does not prevent them. Instead, it gives users confidence to embed the database in systems expected to live longer than the present development team.</p>

<p>The promise also improves engineering decisions today. A format intended to survive decades encourages explicit encoding, careful corruption handling, strong tests and conservative semantics. Long-term intent is not only a support policy. It changes the quality of the design.</p>

<h2 id="standards-are-machines-for-accumulating-work">Standards Are Machines for Accumulating Work</h2>

<h3 id="compatibility-creates-network-value">Compatibility creates network value</h3>

<p>The value of many technologies depends on the number of compatible complements around them. A communication protocol becomes more useful as more systems speak it. A file format becomes more useful as more tools read it. A connector becomes more useful as more devices fit it. Economic work on network externalities and standardization formalized these effects decades ago[<a href="#ref39">39</a>,<a href="#ref40">40</a>].</p>

<p>The important consequence is that compatibility produces value even when no individual component improves.</p>

<p>If a new editor can open all existing documents, it inherits a document ecosystem. If it requires a new format, it begins with an empty world. If a new musical instrument speaks the established protocol, it can interact with old instruments and software immediately. If it speaks only a new protocol, its individual technical superiority may be irrelevant.</p>

<p>Standards are therefore not bureaucratic obstacles placed after innovation. Good standards are machines that allow independently produced work to combine.</p>

<h3 id="midi-four-decades-of-musical-machinery">MIDI: Four decades of musical machinery</h3>

<p>MIDI 1.0 dates from the early 1980s. Its bandwidth and resolution reflect that time, yet it remains embedded in instruments, controllers, studios and software. MIDI 2.0 adds richer messages and bidirectional capabilities while using an architecture designed for backward compatibility and translation[<a href="#ref22">22</a>].</p>

<p>This is the correct response to a successful old interface.</p>

<p>The limitations of MIDI 1.0 are real. They justify an extension. They do not justify making every existing keyboard, synthesizer and composition tool worthless. MIDI 2.0 gains immediate practical relevance precisely because it can coexist with the installed base.</p>

<p>The standard moved slowly, and because it moved slowly an enormous amount of creative and industrial activity could move quickly around it.</p>

<h3 id="jpeg-and-pdf">JPEG and PDF</h3>

<p>JPEG compression was standardized in the early 1990s[<a href="#ref23">23</a>]. The format is far from perfect for every modern imaging task. New codecs achieve better compression, support high dynamic range and handle different content more effectively. Still, almost every relevant system can decode ordinary JPEG images.</p>

<p>This universality is itself a capability. An image can pass through cameras, websites, archives, scientific tools and operating systems without negotiating a vendor-specific world. New formats can be added beside JPEG where their advantages matter. The old images do not have to disappear.</p>

<p>PDF follows a similar pattern. It was designed to preserve document presentation across software and hardware and later moved into ISO standardization[<a href="#ref24">24</a>]. The ecosystem is complex and not every PDF feature is equally archival, but the stable idea of a self-contained portable document has survived platform generations.</p>

<p>The value of these formats cannot be measured only by compression efficiency or feature count. It includes every avoided conversion, every document that remains readable and every implementation that can interoperate without asking the original author for help.</p>

<h3 id="usb-and-physical-continuity">USB and physical continuity</h3>

<p>USB has accumulated several naming mistakes and connector transitions, and it should not be idealized. Nevertheless, its specifications illustrate the deliberate use of layered compatibility. Newer transfer modes and power capabilities have repeatedly been introduced while retaining ways for older devices to communicate[<a href="#ref21">21</a>].</p>

<p>This matters because physical devices have longer and more irregular lives than software releases. A laboratory instrument, keyboard, serial adapter or measurement device may remain perfectly useful for decades. Replacing it because a software company prefers a new abstraction is waste.</p>

<p>Where physical connector changes are unavoidable, adapters and protocol continuity preserve part of the investment. Where they are merely fashionable, they convert functional hardware into electronic waste.</p>

<h3 id="the-lego-test">The LEGO test</h3>

<p>The LEGO System in Play expressed a compatibility promise in unusually tangible form: elements from yesterday, today and tomorrow should fit together[<a href="#ref41">41</a>]. The modern stud-and-tube brick dates to 1958, yet later sets add an enormous variety of shapes, mechanisms and themes without invalidating the elementary geometry.</p>

<p>This is more than a charming toy example. It is almost a perfect demonstration of an extensible interface.</p>

<p>The stable connection rule creates a combinatorial design space. Every new element can combine with old elements, so the value of the whole system grows faster than the catalogue. A hypothetical annual connector redesign might permit locally elegant pieces, but it would destroy the system.</p>

<blockquote>
  <p><strong>Software APIs should be judged by the same simple question: will the new piece still connect to what users already own?</strong></p>
</blockquote>

<h2 id="apparently-sudden-technology">Apparently Sudden Technology</h2>

<h3 id="the-launch-date-illusion">The launch-date illusion</h3>

<p>Products have launch dates. Foundations usually do not.</p>

<p>A product is announced on a stage, photographed and sold. The date becomes a convenient point in history. The enabling work is distributed across earlier decades, organizations and disciplines, so it is harder to narrate. <strong>This produces the illusion that technology arrives suddenly</strong>.</p>

<p>The first iPhone was announced in January 2007 as a phone, media player and Internet communicator using a multi-touch interface[<a href="#ref25">25</a>]. As a product integration it was highly significant. It was not created from technologies invented during the months before the announcement.</p>

<p>It depended on mature cellular networks, semiconductor manufacturing, battery chemistry, capacitive sensing, radio engineering, operating systems, web standards, audio and video codecs, flash storage and global manufacturing. Each layer represented many decades of work. More importantly, each layer was usable because it exposed sufficiently stable properties to the designers above it.</p>

<p>The product <em>appeared</em> fast because the foundations had become boring.</p>

<h3 id="integration-is-real-innovation">Integration is real innovation</h3>

<p>Pointing out the age of the foundations does not diminish the work of integration. Combining mature components into a coherent product is difficult. Interfaces have to be reconciled, constraints balanced and manufacturing made reliable.</p>

<p>But integration reinforces rather than weakens the stability thesis.</p>

<p>An integrator cannot build a complex product if every lower layer changes unpredictably during development. Mechanical dimensions, radio behavior, processor interfaces, toolchains and protocols have to remain stable long enough for the whole to be tested. After launch, application developers need the product platform to preserve their investments as well.</p>

<blockquote>
  <p><strong>The apparently fast product is therefore a temporary convergence of slow-moving layers.</strong></p>
</blockquote>

<h3 id="artificial-intelligence-did-not-appear-in-one-year">Artificial intelligence did not appear in one year</h3>

<p>Modern artificial intelligence is another strong example of compressed public history.</p>

<p>Back-propagation was famously demonstrated for learning internal representations in 1986[<a href="#ref26">26</a>], building on earlier work in neural computation and optimization. Convolutional networks developed over decades. Large labeled datasets, better hardware and GPU implementations made much larger experiments practical; the 2012 ImageNet result was a prominent turning point[<a href="#ref27">27</a>]. The transformer architecture was published in 2017[<a href="#ref28">28</a>]. Reviews of deep learning already described the long convergence of representation learning, compute and data before the present wave of generative products[<a href="#ref29">29</a>].</p>

<p>From a media viewpoint, a chatbot may seem to transform the world during a single release season. From a scientific viewpoint, it is one visible output of a long accumulation.</p>

<p>Even after the visible breakthrough, a durable capability is not finished. Models have to become reproducible, efficient, inspectable and supportable. Data and evaluation practices have to mature. Interfaces need to stop changing arbitrarily. Applications require predictable behavior, versioned semantics and local fallbacks. Hardware and runtimes need stable execution paths.</p>

<blockquote>
  <p><strong>Without this consolidation, an impressive model remains an expensive experiment surrounded by permanent integration work.</strong></p>
</blockquote>

<h3 id="the-danger-of-confusing-a-demonstration-with-a-foundation">The danger of confusing a demonstration with a foundation</h3>

<p>A demonstration answers <em>“Can this be done at least once under these conditions?”</em> Infrastructure answers a different question: <em>“Can other people depend on this without the original team standing beside it?”</em></p>

<p>The gap includes:</p>

<ul>
  <li>reproducibility</li>
  <li>failure handling</li>
  <li>documentation</li>
  <li>compatibility</li>
  <li>performance bounds</li>
  <li>security maintenance</li>
  <li>supply chains</li>
  <li>calibration</li>
  <li>test coverage</li>
  <li>data preservation</li>
</ul>

<p>Media reports often collapse this gap. Management plans then inherit the compressed timeline. A research demonstration is expected to become a production system during the next quarter. When the missing engineering consumes years, it is treated as delay rather than as the main body of the work.</p>

<p>The prototype was not ninety percent complete. It answered a smaller question.</p>

<h2 id="science-moves-at-the-speed-of-evidence">Science Moves at the Speed of Evidence</h2>

<h3 id="projects-are-slices-through-longer-work">Projects are slices through longer work</h3>

<p>Scientific funding is organized into projects because budgets seem to require boundaries. <strong>Nature, and thus reality, does not respect these boundaries.</strong></p>

<p>A two-year project may test one hypothesis, build one instrument or collect one dataset. The idea often began way before the proposal, relies on methods developed by others and produces questions that continue after the final report. Publication may require additional controls, repeated measurements, analysis, peer review and revision. Adoption by other groups requires independent reproduction and usable methods.</p>

<p>For a medium-sized scientific idea, <em>half a decade from first serious work to a mature publication is entirely unsurprising</em>. <strong>For large ideas, the relevant time scale may be a whole career</strong>.</p>

<p>This does not mean scientists work slowly in the everyday sense. Individual experiments may be performed quickly. Code may be written in days. The long duration comes from the number of uncertain steps and the fact that reality can reject every convenient schedule.</p>

<blockquote>
  <p><strong>Science is not a feature factory. Its output is justified belief, and justification takes time.</strong></p>
</blockquote>

<h3 id="mrna-vaccines">mRNA vaccines</h3>

<p>The COVID-19 mRNA vaccines were developed and deployed with exceptional urgency. Their visible success in 2020 is often used as an example of extremely fast science.</p>

<p>The platform was not invented in 2020.</p>

<p>Research into mRNA, delivery and immune response had been progressing for decades. Kariko, Weissman and colleagues published the effect of nucleoside modification on innate immune recognition in 2005[<a href="#ref31">31</a>]. The Nobel scientific background describes how these discoveries enabled effective mRNA vaccines when the pandemic created an urgent application[<a href="#ref30">30</a>].</p>

<p>The rapid final response was possible because the slow foundation already existed. Sequencing, lipid nanoparticles, manufacturing processes, trial methods, regulatory institutions and global logistics were also available. The last stage moved quickly because many earlier stages had been stabilized.</p>

<p>Calling only the final year <em>“the development”</em> erases the work that made speed possible.</p>

<h3 id="lithium-ion-batteries">Lithium-ion batteries</h3>

<p>Lithium-ion batteries are associated with fast-moving consumer electronics and electric vehicles. Their development story spans decades.</p>

<p>Whittinghams work on intercalation cathodes emerged in the 1970s. Goodenough developed a higher-potential cobalt-oxide cathode around 1980. Yoshino developed a safer carbon-based anode and a commercially viable cell in the 1980s. Commercial introduction followed in the early 1990s. The Nobel background traces this sequence and the different technical obstacles solved along the way[<a href="#ref32">32</a>].</p>

<p>The mobile products of the 2000s depended on electrochemistry consolidated over the preceding decades. Continuing improvements in materials, manufacturing, safety and battery management then accumulated without requiring every device engineer to rediscover the cell.</p>

<blockquote>
  <p><strong>The progress was not one breakthrough. It was a stable chain of solved constraints.</strong></p>
</blockquote>

<h3 id="crispr">CRISPR</h3>

<p>CRISPR gene editing also looked sudden when programmable Cas9 systems became widely usable. The underlying biological patterns had been observed long before their function was understood. Work across microbiology, genetics and biochemistry gradually established CRISPR as an adaptive immune system. Jinek and colleagues demonstrated a programmable RNA-guided DNA endonuclease in 2012[<a href="#ref34">34</a>], and the Nobel scientific background places this result within the longer discovery chain[<a href="#ref33">33</a>].</p>

<p>After the programmable mechanism came another long chain: delivery, off-target analysis, organism-specific methods, clinical trials, manufacturing, ethics and regulation.</p>

<p>Again the distinction is important. A method that edits DNA in a controlled laboratory experiment is a major result. It is not yet a routine, safe and broadly available therapy. Treating these as the same stage creates unrealistic promises and bad policy.</p>

<h3 id="gravitational-waves">Gravitational waves</h3>

<p>Einsteins theory predicted gravitational waves roughly a century before their first direct observation. The LIGO collaborations 2016 publication reported a signal observed in September 2015 and explicitly described the detection as occurring a century after the fundamental theoretical predictions[<a href="#ref35">35</a>].</p>

<p><strong>The delay was not intellectual inactivity. It was the time required to make an almost impossibly precise measurement reliable.</strong></p>

<p>Laser stability, mirror coatings, vibration isolation, vacuum systems, control theory, numerical relativity, data analysis and large-scale scientific coordination all had to advance. Early detector generations could operate correctly and still observe no event. Their value included learning how to build the next generation.</p>

<p>Calling the final detection a sudden breakthrough is understandable as storytelling. As an engineering description it is incomplete. The signal became visible because decades of stable intermediate capabilities were assembled.</p>

<h3 id="discovery-publication-and-use-are-different-clocks">Discovery, publication and use are different clocks</h3>

<p><strong>There is no universal translation time from research to practice</strong>. Different fields have different constraints. A mathematical result can be distributed as a paper; a medical treatment requires safety evidence, production and clinical systems; a new material may require an industrial supply chain.</p>

<p>Still, empirical work on medical translation found that time lags commonly discussed under the famous <em>“17 years”</em> summary arise from several distinct stages and definitions[<a href="#ref36">36</a>]. The number should not be turned into another simplistic rule. Its value is that it destroys the fiction that discovery and routine use are nearly simultaneous.</p>

<p>Three clocks should be kept separate:</p>

<ul>
  <li><strong>Discovery time:</strong> the work required to establish that an effect or method exists.</li>
  <li><strong>Consolidation time:</strong> the work required to reproduce, explain and delimit it.</li>
  <li><strong>Deployment time:</strong> the work required to make it safe, scalable, affordable and maintainable.</li>
</ul>

<blockquote>
  <p><strong>Only media time is instantaneous.</strong></p>
</blockquote>

<h2 id="the-physical-world-is-built-from-frozen-decisions">The Physical World Is Built from Frozen Decisions</h2>

<h3 id="a-box-that-changed-trade">A box that changed trade</h3>

<p>The standardized freight container is mechanically unexciting. It is a box with carefully controlled dimensions, ratings and handling points. ISO 668 still defines the classification and principal dimensions of the Series 1 containers around which ships, cranes, trucks, rail wagons, ports and warehouses are designed[<a href="#ref37">37</a>].</p>

<p>The economic effect was anything but unexciting. Empirical analysis associates container adoption with major growth in international trade and emphasizes that containerization required complementary technological and organizational changes[<a href="#ref38">38</a>].</p>

<blockquote>
  <p><strong>The crucial object was not merely the box. It was the stable interface between transport systems.</strong></p>
</blockquote>

<p>Before standardization, cargo handling was repeatedly adapted to ships, ports and local practices. After standardization, an independently packed unit could move through a global intermodal network. Crane manufacturers could design for the same geometry as shipyards. Truck builders could design for the same unit as ports. Warehouses could arrange storage around known dimensions.</p>

<p>If the container interface had been improved incompatibly every two years, this network could never have accumulated. A slightly better box would have produced a much worse transport system.</p>

<h3 id="units-are-infrastructure">Units are infrastructure</h3>

<p>The International System of Units is another foundation that works by refusing arbitrary local invention. The SI has been refined - for example by defining units through physical constants - but the practical continuity of measurements is treated as essential[<a href="#ref42">42</a>].</p>

<p>Imagine an engineering culture that renamed units, changed scale factors or altered sign conventions at the pace of web frameworks. Every drawing, calibration, material specification and instrument would become a migration project. Bridges would not become innovative. They would become dangerous.</p>

<p>Measurement standards show that stability does not oppose scientific improvement. Metrology advances continuously. Realizations become more precise. The external quantity remains sufficiently continuous that earlier measurements retain meaning.</p>

<blockquote>
  <p><strong>This is the physical equivalent of replacing an implementation behind a stable API.</strong></p>
</blockquote>

<h3 id="long-lived-equipment-and-the-software-problem">Long-lived equipment and the software problem</h3>

<p>Physical equipment frequently outlives the software fashion surrounding it.</p>

<p>A microscope, lathe, power supply, spectrometer, environmental chamber or industrial controller may remain useful for more than hundred years. Its bearings can be replaced, electronics repaired and calibration renewed. Increasingly, however, the machine is made dependent on short-lived software: a proprietary operating-system driver, an online activation service, an undocumented file format or a vendor cloud endpoint.</p>

<p>The result is a reversal of expected lifetimes. A mechanically sound instrument becomes waste because a much cheaper software layer was not designed to survive.</p>

<blockquote>
  <p><strong>This is not progress. It is the destruction of capital through interface negligence.</strong></p>
</blockquote>

<p>A serious instrument should therefore provide documented protocols, stable command sets, local operation, exportable calibration data and replaceable control software. Network features may be added, but the machine should not require a remote service for basic function. The goal is not to preserve every original computer. It is to make the control boundary sufficiently open and stable that a future computer can take its place.</p>

<h3 id="repairability-is-compatibility-across-time">Repairability is compatibility across time</h3>

<p>Repair is usually discussed as a question of spare parts and access. At a deeper level it is temporal compatibility.</p>

<p>A replacement part must fit the old mechanical interface. A diagnostic tool must understand the old protocol. Documentation must still describe the deployed revision. Firmware must remain obtainable. A new component may improve efficiency or reliability, but it should reproduce the relevant contract of the old component.</p>

<p>Products designed this way can evolve by substitution. Products designed as sealed historical moments fail as soon as one layer ages.</p>

<p>The same principle applies to organizations. If knowledge exists only in the memory of one employee, the organization has an undocumented interface. If a process works only through one person’s manual intervention, it has no stable operational contract. Succession, documentation and automation are forms of repairability.</p>

<h3 id="the-environment-also-pays">The environment also pays</h3>

<p>Breaking software is often treated as immaterial because no physical object is visibly destroyed. Yet software instability causes hardware replacement, repeated computation, travel, emergency work and duplicated development. It consumes energy and materials through secondary effects.</p>

<blockquote>
  <p><strong>The cleanest environmental product is often not the device with the most impressive new efficiency figure. It is the device that remains in service.</strong></p>
</blockquote>

<p>Long-lived products distribute their manufacturing cost over more useful years. Stable protocols allow controllers and computers to be replaced without replacing sensors, motors and structures. Additive upgrades preserve embodied energy. A compatibility layer that extends the life of ten thousand instruments may have greater environmental value than a new <em>green</em> product line.</p>

<blockquote>
  <p><strong>Durability is not nostalgic. It is resource efficiency.</strong></p>
</blockquote>

<h2 id="exploration-has-one-job">Exploration Has One Job</h2>

<h3 id="learning-under-uncertainty">Learning under uncertainty</h3>

<p>Exploration is necessary because the correct design is not always known in advance. At the beginning of a project it may be rational to write disposable code, change data structures repeatedly and test several incompatible approaches. Scientific instruments are often assembled from improvised parts before the relevant signal is even known to exist. A new algorithm may need synthetic experiments before its useful parameters are understood.</p>

<p>This is the legitimate domain of moving fast.</p>

<p>But the output of exploration is information, not infrastructure.</p>

<p>The prototype should answer questions:</p>

<ul>
  <li>Does the effect exist?</li>
  <li>Which constraints dominate?</li>
  <li>Which interface is actually useful?</li>
  <li>What fails?</li>
  <li>Which parts need to remain flexible?</li>
  <li>What can be specified?</li>
</ul>

<blockquote>
  <p><strong>Once those answers exist, the engineering mode has to change.</strong></p>
</blockquote>

<h3 id="the-consolidation-boundary">The consolidation boundary</h3>

<p>The most damaging projects never declare the point at which experimentation ends and dependence begins.</p>

<p>A prototype acquires users. Its accidental data format becomes a public format. A temporary API is copied into production. A research script becomes a service. Because the system is still culturally labeled experimental, maintainers feel free to change it. Because other people now depend on it, every change creates real downstream cost.</p>

<p>The project has crossed the consolidation boundary without acknowledging it.</p>

<p>This boundary should be explicit. Before crossing it, interfaces may be unstable and users must not be encouraged to build durable systems on them. After crossing it, contracts are versioned, documented, tested and treated as obligations. Internal experimentation can continue behind them.</p>

<p>A useful rule is:</p>

<blockquote>
  <p><strong>If another independent person is expected to invest substantial work against an interface, the interface is no longer an experiment.</strong></p>
</blockquote>

<p><strong>Beta written in a header for five years does not change this reality.</strong></p>

<h3 id="throw-prototypes-away-or-finish-them">Throw prototypes away-or finish them</h3>

<p>Prototype code has accumulated assumptions optimized for learning speed. It may omit error handling, portability and migration. Turning it directly into infrastructure often creates the technical debt originally described by Cunningham: the apparent speed gained by an incomplete design produces interest in later work[<a href="#ref5">5</a>].</p>

<p>There are two honest choices:</p>

<ul>
  <li>discard the prototype after extracting what was learned and implement a stable system</li>
  <li>stop treating it as disposable and perform the work required to make it durable</li>
</ul>

<blockquote>
  <p><strong>The dishonest choice is to deploy the prototype, call it production, retain experimental instability and make users pay the resulting interest.</strong></p>
</blockquote>

<p>This is especially important in machine learning. Sculley and colleagues described how small amounts of ML code can sit inside large systems of data dependencies, configuration, monitoring and glue, creating hidden technical debt[<a href="#ref6">6</a>]. A model demonstration may be quick. A reliable system that knows when the model is wrong is not.</p>

<h3 id="exploration-and-exploitation-are-not-symmetric">Exploration and exploitation are not symmetric</h3>

<p>Organizational research distinguishes exploration of new possibilities from exploitation of established knowledge[<a href="#ref43">43</a>]. Both require resources and a system that performs only one becomes unhealthy.</p>

<p>For the argument here, however, they do not have equal final status.</p>

<p><strong>Exploration creates options. Consolidation and stable exploitation turn selected options into retained capability</strong>. Ten experiments may reasonably fail. The useful eleventh must then be made dependable. If all resources are moved immediately to the next experiment, the organization accumulates demonstrations rather than progress.</p>

<p>This explains a common paradox in research and technology companies. <em>They appear highly innovative because they announce many prototypes</em>, yet their users solve the same integration problems year after year. Novelty is produced, but it never settles into a foundation.</p>

<p>Exploration is valuable only because something may eventually stop exploring.</p>

<h3 id="two-lanes-not-one-compromise">Two lanes, not one compromise</h3>

<p>The best organization does not force the same stability policy onto every activity. It maintains two lanes.</p>

<p>The <em>exploratory lane</em> is explicitly temporary:</p>

<ul>
  <li>no compatibility promise</li>
  <li>small user population</li>
  <li>easy reset</li>
  <li>strong recording of results</li>
  <li>and a defined question being tested</li>
</ul>

<p>The <em>foundation lane</em> is explicitly durable:</p>

<ul>
  <li>documented contracts</li>
  <li>compatibility tests</li>
  <li>migration planning</li>
  <li>operational ownership</li>
  <li>security support</li>
  <li>and a long time horizon</li>
</ul>

<p>Ideas can move from the first lane to the second, but they pass through a consolidation step. Production users are not used as involuntary experimental instrumentation. This separation permits fast learning without normalizing a fragile world.</p>

<h2 id="why-organizations-reward-instability">Why Organizations Reward Instability</h2>

<h3 id="visible-work-wins">Visible work wins</h3>

<blockquote>
  <p><strong>Stable systems create a political problem: their value is counterfactual.</strong></p>
</blockquote>

<p>One cannot easily photograph the outage that did not happen. The manager who funds compatibility testing cannot present a dramatic rescue. An engineer who removes a recurring manual task may make their future workload appear smaller. In contrast, a feature launch has screenshots, deadlines and a release announcement. A crisis has meetings, urgency and identifiable heroes.</p>

<p><strong>Organizations therefore tend to reward visible production and visible recovery more than invisible prevention.</strong></p>

<p>This does not require malicious management, only incapable one. It follows from short reporting periods and poor metrics. <strong>If work is counted rather than eliminated work measured, instability can create its own evidence of importance.</strong></p>

<p>The team that causes repeated migrations may look busy. The team whose interface has required no migration for fifteen years may look inactive.</p>

<h3 id="feature-accounting">Feature accounting</h3>

<p>Feature counts treat every addition as positive and almost never subtract the complexity imposed on users.</p>

<p>Suppose a release adds five features, changes three workflows and removes two established capabilities. A launch presentation can count five. The lost habits, updated documentation, retraining, automation changes and accessibility regressions are dispersed and usually absent from the ledger.</p>

<p>A better release balance is</p>

[
\Delta V =
V_{\mathrm{new}}
- C_{\mathrm{migration}}
- C_{\mathrm{lost\ capability}}
- C_{\mathrm{future\ maintenance}}
- C_{\mathrm{cognitive\ change}}.
]

<p>The new value $V_{\mathrm{new}}$ should include actual user benefit rather than the existence of a new control. The costs should be counted across all affected users, not only inside the producing team.</p>

<blockquote>
  <p><strong>Under this accounting, many celebrated releases become negative.</strong></p>
</blockquote>

<h3 id="the-deadline-creates-a-biased-experiment">The deadline creates a biased experiment</h3>

<p><strong>Short deadlines systematically favor visible incomplete work.</strong></p>

<p>Compatibility design, documentation, long-duration testing and migration tooling have costs today and benefits later. Cutting them helps the present milestone. The downstream failure occurs after the launch and may be assigned to support, operations or the user.</p>

<blockquote>
  <p><strong>This is a temporal externality. Present management spends future engineering time.</strong></p>
</blockquote>

<p>The same distortion appears in science when funding rewards papers within a project period but not the maintenance of datasets, instruments and software afterwards. A new prototype is attributable to the current grant. Ten years of stable operation is distributed across later users and may have no obvious owner.</p>

<p>Organizations obtain exactly the time horizon they reward.</p>

<h3 id="firefighting-as-organizational-theater">Firefighting as organizational theater</h3>

<p>Incident response is necessary after an incident. It should not be confused with the creation of value.</p>

<p>If engineers repeatedly repair the same category of failure, the system is consuming them. The correct goal is to eliminate the failure class, automate recovery or redesign the unstable boundary. Googles description of toil emphasizes that work which scales linearly with service growth has no enduring value[<a href="#ref7">7</a>]. A system that needs proportionally more human attention as it succeeds is not successfully automated.</p>

<blockquote>
  <p><strong>The cultural difficulty is that permanent elimination also eliminates future opportunities for heroism.</strong></p>
</blockquote>

<p>Management has to reward the person who makes the alarm silent, not only the person who answers it at three in the morning.</p>

<h3 id="speed-metrics-already-know-they-need-stability">Speed metrics already know they need stability</h3>

<p>Modern software delivery research is sometimes invoked to demand more releases. This is an incomplete reading.</p>

<p>DORA evaluates delivery performance using measures that include change failure and deployment rework, not frequency alone[<a href="#ref10">10</a>]. A deployment that requires a rollback, hotfix or unplanned corrective deployment is evidence of instability. Fast delivery and low failure can coexist when changes are small, automated and well tested. Nothing in this requires breaking public interfaces.</p>

<p>An organization may deploy internal implementations many times per day while keeping client contracts stable for decades. In fact, stable boundaries make frequent internal deployment safer because fewer independent systems have to move in lockstep.</p>

<blockquote>
  <p><strong>The real opposition is not slow versus fast. It is accumulated progress versus accumulated rework.</strong></p>
</blockquote>

<h2 id="measuring-what-actually-matters">Measuring What Actually Matters</h2>

<h3 id="retained-capability">Retained capability</h3>

<p>The first metric should be brutally simple:</p>

<blockquote>
  <p>What useful capability from last ten years is still available without mandatory user work?</p>
</blockquote>

<p>This can be measured through compatibility suites, old input corpora, archived client binaries and repeatable workflows. A project should regularly test:</p>

<ul>
  <li>whether old files still open</li>
  <li>whether old clients still communicate</li>
  <li>whether documented commands retain meaning</li>
  <li>whether previous automation still runs</li>
  <li>whether users can upgrade without migration</li>
  <li>and whether data can be exported without the original service</li>
</ul>

<p>These tests turn stability from an aspiration into an observable property.</p>

<h3 id="human-intervention-rate">Human intervention rate</h3>

<p><strong>Automation is valuable because it removes repeated human work[<a href="#ref1">1</a>]. Its failure can be measured by intervention frequency</strong>.</p>

<p>For an operational system, record:</p>

[
I = \frac{\text{unplanned human interventions}}{\text{unit of useful operation}}
]

<p>The denominator may be transactions, instrument hours, deployments, samples, users or months. A maturing system should drive $I$ downward. If feature growth drives it upward, the system is becoming less advanced even if its feature list grows.</p>

<blockquote>
  <p><strong>Long unattended operation is a more meaningful achievement than frequent dashboard activity.</strong></p>
</blockquote>

<h3 id="downstream-hours-caused">Downstream hours caused</h3>

<p>Every breaking change should carry an estimate of downstream work:</p>

<ul>
  <li>number of affected clients</li>
  <li>expected understanding time</li>
  <li>modification time</li>
  <li>validation time</li>
  <li>deployment time</li>
  <li>training and documentation changes</li>
  <li>and probability of missed or abandoned migrations</li>
</ul>

<p>The estimate will be uncertain. Uncertainty is not a reason to record zero.</p>

<p>The producing team should own this cost in its decision process even when another organization performs the work. If the estimated global migration cost exceeds the cost of maintaining a compatibility path, the compatibility path is the economically rational choice.</p>

<h3 id="age-is-not-a-defect-count">Age is not a defect count</h3>

<p>Projects often use dependency age as a proxy for danger. Age can correlate with missing security fixes, but it can also indicate maturity.</p>

<p>A better inventory distinguishes:</p>

<ul>
  <li>known vulnerabilities</li>
  <li>unsupported implementations</li>
  <li>untested contracts</li>
  <li>environmental incompatibilities</li>
  <li>merely old but functioning components</li>
</ul>

<p>Replacing an old component without a concrete defect introduces change risk and discards accumulated operational knowledge. The replacement needs evidence of positive net value, not youth.</p>

<p>Old and unmaintained is a problem. Old and maintained is infrastructure.</p>

<h3 id="stability-budget">Stability budget</h3>

<p>Projects routinely allocate performance budgets, memory budgets and financial budgets. They should also allocate a stability budget.</p>

<p>The budget constrains:</p>

<ul>
  <li>number of mandatory migrations</li>
  <li>allowed incompatibilities</li>
  <li>user-visible workflow changes</li>
  <li>format conversions</li>
  <li>removals of established behavior</li>
</ul>

<p>Spending the budget requires justification and mitigation. Unused budget is not failure to innovate. It is preserved trust.</p>

<p>For foundational components, the target budget may be zero for ordinary releases.</p>

<h3 id="thirty-year-questions">Thirty-year questions</h3>

<p>Before adopting a technology, ask:</p>

<ul>
  <li>Can the data be read without the original vendor?</li>
  <li>Is the interface documented?</li>
  <li>Can the implementation be replaced?</li>
  <li>Is local operation possible?</li>
  <li>What happens if updates stop?</li>
  <li>Can security fixes be applied without changing user workflows?</li>
  <li>Are old inputs part of automated tests?</li>
  <li>Is there an explicit compatibility policy?</li>
  <li>Does the business model benefit from forced replacement?</li>
  <li>Would this still be usable if the present development team disappeared?</li>
</ul>

<p><strong>These questions are not pessimistic. They determine whether present work can become future capital.</strong></p>

<h2 id="how-to-build-technology-for-thirty-years">How to Build Technology for Thirty Years</h2>

<h3 id="freeze-the-boundary-late-enough-and-then-mean-it">Freeze the boundary late enough-and then mean it</h3>

<p>An interface frozen before its problem is understood may preserve the wrong abstraction. An interface left unstable after adoption prevents accumulation. Timing matters.</p>

<p>Before publication:</p>

<ul>
  <li>build several real clients</li>
  <li>test the smallest useful operations</li>
  <li>observe failure modes</li>
  <li>separate policy from mechanism</li>
  <li>identify which dimensions are likely to grow</li>
</ul>

<p><strong>Then define the contract and treat it as an obligation.</strong></p>

<p>The cost of careful interface design is paid once. The cost of careless design is paid by every client.</p>

<h3 id="prefer-additive-evolution">Prefer additive evolution</h3>

<p>New capabilities should normally be added rather than substituted.</p>

<p>Useful mechanisms include:</p>

<ul>
  <li>optional fields with defined defaults</li>
  <li>new operations beside old operations</li>
  <li>capability discovery</li>
  <li>extension namespaces</li>
  <li>reserved bits and identifiers</li>
  <li>length-prefixed structures</li>
  <li>tolerant reading with strict writing</li>
  <li>adapters</li>
  <li>explicit negotiation</li>
</ul>

<blockquote>
  <p><strong>An additive interface lets old clients continue using the subset they understand. New clients gain new capability without demanding synchronized migration.</strong></p>
</blockquote>

<p>Care is still required. Optional behavior can become ambiguous, and permissive parsing can create security problems. The goal is not to accept arbitrary input. It is to design extension points deliberately.</p>

<h3 id="make-the-data-outlive-the-program">Make the data outlive the program</h3>

<p>Data often matters longer than the software that created it.</p>

<p>A durable format should have:</p>

<ul>
  <li>a public specification</li>
  <li>explicit byte order and numeric representation</li>
  <li>units and coordinate systems</li>
  <li>checksums where corruption matters</li>
  <li>version and extension mechanisms</li>
  <li>test vectors</li>
  <li>more than one implementation where practical</li>
  <li>export into simpler archival representations</li>
</ul>

<p>Schema migrations should be automated, reversible where possible and tested against real historical data. Never require users to reconstruct a chain of obsolete application versions merely to reach the current format.</p>

<p>SQLites long-term promise for its on-disk format is a model because it recognizes that stored information is the user’s asset, not an implementation detail[<a href="#ref12">12</a>].</p>

<h3 id="preserve-behavior-not-only-syntax">Preserve behavior, not only syntax</h3>

<p>Compilation tests catch only part of compatibility.</p>

<p>A stable function can retain its name and change rounding, ordering, timeout behavior, units or error semantics. For scientific and control software, such changes can be worse than an obvious compile failure because they silently alter results.</p>

<p>Compatibility testing should therefore include:</p>

<ul>
  <li>golden outputs</li>
  <li>numerical tolerances justified by the domain</li>
  <li>property tests</li>
  <li>historical bug cases</li>
  <li>timing and resource bounds where clients depend on them</li>
  <li>error behavior</li>
  <li>end-to-end tests with old clients</li>
</ul>

<p>If a historical behavior is a genuine bug, first determine whether users may rely on it. A corrected operation can be added while the old one remains reproducible, or a transition can be made explicit and detectable. Silent reinterpretation is the worst option.</p>

<h3 id="keep-specifications-independent-of-implementations">Keep specifications independent of implementations</h3>

<p>An interface documented only by its current source code is not a durable interface.</p>

<p>The specification should describe observable behavior independently. Reference implementations and conformance suites can then test replacements. This allows internals, languages, operating systems and maintainers to change without destroying the contract.</p>

<p>Open standards such as Internet RFCs demonstrate the value of this separation[<a href="#ref14">14</a>,<a href="#ref15">15</a>,<a href="#ref16">16</a>]. Independent implementations are not duplication for its own sake. They reveal hidden assumptions and reduce the risk that one codebase becomes the only executable specification.</p>

<h3 id="own-adapters-centrally">Own adapters centrally</h3>

<p>When an internal redesign is beneficial, the team introducing it should normally provide the compatibility adapter.</p>

<p>This centralizes knowledge and allows one well-tested translation instead of thousands of improvised migrations. The adapter may eventually become the permanent public boundary while internals continue to evolve behind it.</p>

<p>Performance arguments should be measured. Many compatibility paths are cheap compared with network, storage or user interaction costs. Where overhead is real, provide a new optimized path without removing the dependable path by default.</p>

<h3 id="maintain-simple-escape-routes">Maintain simple escape routes</h3>

<p>Long-lived systems need ways out.</p>

<p>Users should be able to:</p>

<ul>
  <li>export their data</li>
  <li>run core functions locally</li>
  <li>replace a service endpoint</li>
  <li>inspect configuration</li>
  <li>recover without a vendor account</li>
  <li>operate a reduced feature set when optional infrastructure disappears</li>
</ul>

<blockquote>
  <p><strong>An escape route reduces the probability that one failed organization destroys an otherwise useful technical system.</strong></p>
</blockquote>

<p>This is particularly important for research. A published result should not depend exclusively on a temporary commercial API or an unarchived model. The complete original environment may be impossible to preserve, but the data, method and relevant contract can be.</p>

<h3 id="budget-for-maintenance-before-launch">Budget for maintenance before launch</h3>

<p>A product without a maintenance plan is a deferred disposal decision.</p>

<p>Before users are recruited, decide:</p>

<ul>
  <li>who owns security fixes</li>
  <li>how long the contract will be supported</li>
  <li>how archival releases are stored</li>
  <li>what happens when maintainers leave</li>
  <li>how keys and signing infrastructure are transferred</li>
  <li>and how users are informed if support genuinely must end</li>
</ul>

<p><strong>The support horizon should match the expected life of the systems built on top</strong>. A library used in embedded equipment cannot honestly offer the same horizon as an experimental visualization package.</p>

<blockquote>
  <p><strong>Trust begins before version 1.0.</strong></p>
</blockquote>

<h2 id="typical-objections">Typical Objections</h2>

<h3 id="stagnation-is-death">Stagnation is death</h3>

<p>For a living system, refusing to respond to a changed environment can indeed be fatal. But stability is not refusal to respond.</p>

<p>A stable system fixes faults, improves internals, adds capabilities and adapts its implementation. It simply preserves useful contracts while doing so. The biological analogy actually supports this: an organism replaces molecules continuously while preserving enough organization to remain the same organism.</p>

<blockquote>
  <p><strong>Breaking existing usage is not proof of adaptation. Often it is proof that adaptation was delegated to the user.</strong></p>
</blockquote>

<h3 id="we-would-be-trapped-by-bad-early-decisions">We would be trapped by bad early decisions</h3>

<blockquote>
  <p><strong>Compatibility creates constraints. That is the price of accepting dependence from others.</strong></p>
</blockquote>

<p>The answer is careful pre-release exploration, narrow public contracts and strong internal abstraction. If a poor contract has already escaped, adapters and parallel interfaces are usually preferable to forced migration. Sometimes a clean break is unavoidable, but “clean” describes the new design, not the global transition.</p>

<p>A new system can be offered beside the old one. It earns adoption by providing enough value to justify voluntary migration. If it cannot do that, destroying the old system is not innovation; it is coercion.</p>

<h3 id="compatibility-code-is-ugly">Compatibility code is ugly</h3>

<p>It can be.</p>

<p>Bridges, electrical substations and sewage systems also contain details that are not aesthetically pure. Infrastructure is judged by service, not by how closely reality matches a diagram. The ugliness can be isolated at boundaries. Internals remain clean. Tests ensure the adapter does not contaminate new logic. The extra code represents encoded respect for existing users.</p>

<p>Deleting it makes the producer’s repository prettier by making the world messier.</p>

<h3 id="security-requires-everyone-to-upgrade">Security requires everyone to upgrade</h3>

<p>Security requires vulnerable behavior to stop. <strong>It does not automatically require unrelated interfaces to change.</strong></p>

<p>Where an old contract is intrinsically unsafe, a transition must be made. The burden of proof should identify the specific danger, the least destructive correction and the longest safe overlap. Migration tooling and compatibility gateways are part of the security work, not optional customer service.</p>

<p>Routine breakage can itself prevent upgrades and leave vulnerable systems deployed. A trusted compatible update channel is one of the strongest security mechanisms available.</p>

<h3 id="users-demand-new-features">Users demand new features</h3>

<p>Users often ask for new capability. They rarely ask for their existing workflows to stop working.</p>

<p>Features can be additive, optional and discoverable. Defaults can remain stable. New interfaces can coexist with old ones. If a user explicitly chooses a new workflow, that is different from forcing every user to migrate because the producer wants a uniform code path.</p>

<p>The request for a new room does not imply permission to demolish the house.</p>

<h3 id="hardware-changes-too-quickly">Hardware changes too quickly</h3>

<p>Hardware changes are exactly why stable abstractions matter.</p>

<p>Instruction sets, operating-system interfaces, protocols and portable languages allow software to survive physical replacement. Mechanical standards allow new components to fit old systems. A compatibility layer converts hardware change from a global rewrite into a local implementation problem.</p>

<p>Without stable boundaries hardware progress would slow, because every new processor or device would begin with no usable ecosystem.</p>

<p>And by the way - hardware changes way slower than people suggest. The major instruction set on consumer computers is x86 and it’s in place since 1978. This is only fast on a vampires scale of time.</p>

<h3 id="legacy-support-prevents-optimization">Legacy support prevents optimization</h3>

<p>Sometimes an old path limits performance. The correct response is to measure where the limit matters and add a new path for users who need it.</p>

<p>Most users do not require every theoretical optimization. They require their system to continue working. Forcing a migration onto all users to improve a minority case is a poor global optimization. System/360, Ethernet and the web all evolved through substantial performance changes while preserving architectural continuity[<a href="#ref18">18</a>,<a href="#ref20">20</a>,<a href="#ref13">13</a>]. The engineering may be harder. The accumulated result is larger.</p>

<h3 id="a-startup-cannot-promise-thirty-years">A startup cannot promise thirty years</h3>

<p>A young organization cannot guarantee its own survival. <strong>It can avoid making its survival a technical prerequisite.</strong></p>

<p>It can publish formats, document protocols, provide export, support local operation, release reference code and use standard components. It can design a graceful failure mode in which customers retain their data and basic capability.</p>

<blockquote>
  <p><strong>Long-term engineering is not a prediction that the company will exist forever. It is a refusal to make users lose everything if it does not.</strong></p>
</blockquote>

<h3 id="sometimes-replacement-really-is-better">Sometimes replacement really is better</h3>

<p>Yes. Stability is not a claim that every old system deserves infinite operation. A replacement may offer overwhelming safety, capability or efficiency benefits. The stability principle determines how the transition should be evaluated:</p>

<ul>
  <li>count global migration cost</li>
  <li>preserve data</li>
  <li>provide adapters</li>
  <li>allow overlap</li>
  <li>avoid unrelated breakage</li>
  <li>make migration voluntary until continued operation creates a concrete harm</li>
</ul>

<p>Replacement becomes responsible when it adds more retained capability than it destroys.</p>

<h2 id="a-stable-world-moves-further">A Stable World Moves Further</h2>

<h3 id="progress-compounds-only-when-it-remains">Progress compounds only when it remains</h3>

<p>Economic accounts of technical change and learning by doing describe growth that emerges from accumulated knowledge, improved methods and repeated production[<a href="#ref44">44</a>,<a href="#ref45">45</a>]. Accumulation is the operative word.</p>

<p>If every generation must repair or recreate the tools of the previous one, knowledge does not compound efficiently. Human lifetime is spent restoring the starting point. <strong>The society may look busy and still advance slowly.</strong> Stable foundations change the arithmetic. Once a problem is solved well enough, it remains solved. The next person begins above it. Their work can address a genuinely new problem rather than an avoidable incompatibility.</p>

<p>This is how a small amount of reliable infrastructure supports an enormous amount of visible activity.</p>

<h3 id="the-hierarchy-of-useful-work">The hierarchy of useful work</h3>

<p>Technical work can be arranged in a rough hierarchy:</p>

<ol>
  <li>repeated manual execution</li>
  <li>repair of preventable failures</li>
  <li>migration caused by avoidable change</li>
  <li>automation of repetition</li>
  <li>stabilization of the automated system</li>
  <li>creation of genuinely new capability</li>
  <li>consolidation of that capability into a foundation for others</li>
</ol>

<p>Modern culture often celebrates level 6 and ignores level 7. It also mistakes levels 2 and 3 for evidence of a vibrant productive technical organization.</p>

<blockquote>
  <p><strong>The highest leverage comes from completing the cycle. A new capability becomes civilizational progress only when it can be inherited.</strong></p>
</blockquote>

<h3 id="what-should-move-fast">What should move fast</h3>

<p>Ideas may move fast inside notebooks, laboratories and private branches. Measurements should move fast through automated pipelines. Bug fixes should move fast through trusted compatible update channels. Internal implementations may move fast behind stable boundaries.</p>

<blockquote>
  <p><strong>Public obligations should move slowly.</strong></p>
</blockquote>

<p>Data meanings, protocols, file formats, physical interfaces and user workflows should change only with evidence that the global benefit exceeds the global cost. The more downstream dependence an interface has attracted, the stronger this presumption should become.</p>

<p>This is not conservatism about goals. It is ambition about timescale.</p>

<h3 id="fifty-years-is-not-an-eternity">Fifty years is not an eternity</h3>

<p>Fifty years sounds long in a software planning meeting. It is short for a building, an instrument, a scientific field, an archive, a railway, an industrial plant or a human career.</p>

<p>A person beginning research today may still need the resulting data in fifty years. A control system installed today may still operate machinery then. A child creating a document today may reasonably want to read it later. A public institution has obligations longer than a venture-capital cycle.</p>

<blockquote>
  <p><strong>Technology that cannot imagine fifty years is not modern. It is temporally narrow.</strong></p>
</blockquote>

<h3 id="the-final-criterion">The final criterion</h3>

<p>When evaluating a new system, one can ignore much of the marketing and ask one question:</p>

<blockquote>
  <p>Does this allow future people to build on present work, or does it require them to rebuild present work first?</p>
</blockquote>

<p>The first kind of technology creates progress. The second creates motion.</p>

<p>Stable systems are not dead systems. They are systems whose useful past remains present while their internals and capabilities improve. They reduce human supervision, permit automation, preserve knowledge, protect physical investment and make independent work composable.</p>

<blockquote>
  <p><strong>The world does not improve because everything changes. It improves because things stop breaking.</strong></p>
</blockquote>

<h2 id="references">References</h2>

<ul>
  <li>Foundations, software evolution and maintenance
    <ul>
      <li>[<span id="ref1">1</span>] T. Spielauer, <a href="https://www.tspi.at/various/AutomationStability.html">Automation and Stability as Foundations of Sustainable Growth</a></li>
      <li>[<span id="ref2">2</span>] M. M. Lehman, <a href="https://doi.org/10.1109/PROC.1980.11805">Programs, Life Cycles, and Laws of Software Evolution</a>, <em>Proceedings of the IEEE</em>, vol. 68, no. 9, pp. 1060–1076, 1980.</li>
      <li>[<span id="ref3">3</span>] T. Besker, A. Martini, and J. Bosch, <a href="https://doi.org/10.1016/j.jss.2019.06.004">Software Developer Productivity Loss Due to Technical Debt - A Replication and Extension Study Examining Developers’ Development Work</a>, <em>Journal of Systems and Software</em>, vol. 156, pp. 41–61, 2019.</li>
      <li>[<span id="ref4">4</span>] NASA, <a href="https://www.nasa.gov/wp-content/uploads/2018/09/nasa_systems_engineering_handbook_0.pdf">NASA Systems Engineering Handbook</a>, NASA/SP-2016-6105 Rev. 2, 2016.</li>
      <li>[<span id="ref5">5</span>] V. Cunningham, <a href="https://doi.org/10.1145/157709.157715">The WyCash Portfolio Management System</a> in <em>Addendum to the Proceedings on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA ’92)</em>, 1992, pp. 29–30.</li>
      <li>[<span id="ref6">6</span>] D. Sculley et al., <a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems">Hidden Technical Debt in Machine Learning Systems</a>, in <em>Advances in Neural Information Processing Systems 28</em>, 2015.</li>
      <li>[<span id="ref7">7</span>] Google, <a href="https://sre.google/sre-book/eliminating-toil/">Eliminating Toil</a>, in <em>Site Reliability Engineering: How Google Runs Production Systems</em>, O’Reilly Media, 2016.</li>
      <li>[<span id="ref8">8</span>] L. Xavier, A. Brito, A. Hora, and M. T. Valente, <a href="https://doi.org/10.1109/SANER.2017.7884616">Historical and Impact Analysis of API Breaking Changes: A Large-Scale Study</a>, in <em>2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER)</em>, 2017, pp. 138–147.</li>
      <li>[<span id="ref9">9</span>] D. Jayasuriya et al., <a href="">An Extended Study of Syntactic Breaking Changes in the Wild</a>https://doi.org/10.1007/s10664-024-10563-4
, <em>Empirical Software Engineering</em>, vol. 30, 2025.</li>
      <li>[<span id="ref10">10</span>] DORA, <a href="https://dora.dev/guides/dora-metrics/">DORA’s Software Delivery Performance Metrics</a>, 2026.</li>
    </ul>
  </li>
  <li>Long-lived computing foundations
    <ul>
      <li>[<span id="ref11">11</span>] Linux kernel project, <a href="https://docs.kernel.org/admin-guide/abi.html">Linux ABI Description</a>, <em>The Linux Kernel Documentation</em>.</li>
      <li>[<span id="ref12">12</span>] SQLite project, <a href="https://sqlite.org/lts.html">Long Term Support</a>, updated 31 May 2025.</li>
      <li>[<span id="ref13">13</span>] W3C HTML Working Group, <a href="https://www.w3.org/TR/html-design-principles/">HTML Design Principles</a>, W3C Working Draft, 26 November 2007.</li>
      <li>[<span id="ref14">14</span>] R. Braden, ed., <a href="https://www.rfc-editor.org/rfc/rfc1122.html">Requirements for Internet Hosts—Communication Layers</a>, RFC 1122, Internet Engineering Task Force, October 1989.</li>
      <li>[<span id="ref15">15</span>] J. Postel, ed., <a href="https://www.rfc-editor.org/rfc/rfc793.html">Transmission Control Protocol</a>, RFC 793, Internet Engineering Task Force, September 1981.</li>
      <li>[<span id="ref16">16</span>] P. Mockapetris, <a href="https://www.rfc-editor.org/rfc/rfc1034.html">Domain Names—Concepts and Facilities</a>, RFC 1034, Internet Engineering Task Force, November 1987.</li>
      <li>[<span id="ref17">17</span>] The Open Group and IEEE, <a href="https://pubs.opengroup.org/onlinepubs/9799919799/">POSIX.1-2024 / IEEE Std 1003.1-2024: Base Specifications, Issue 8</a>, 2024.</li>
      <li>[<span id="ref18">18</span>] G. M. Amdahl, G. A. Blaauw, and F. P. Brooks, Jr., <a href="https://doi.org/10.1147/rd.82.0087">Architecture of the IBM System/360</a>, <em>IBM Journal of Research and Development</em>, vol. 8, no. 2, pp. 87–101, 1964.</li>
      <li>[<span id="ref19">19</span>] IBM, <a href="https://www.ibm.com/history/system-360">The IBM System/360</a>, <em>IBM History</em>.</li>
      <li>[<span id="ref20">20</span>] IEEE Standards Association, <a href="https://standards.ieee.org/ieee/802.3/10422/">IEEE 802.3-2022: IEEE Standard for Ethernet</a>, 2022.</li>
      <li>[<span id="ref21">21</span>] USB Implementers Forum, <a href="https://www.usb.org/document-library/usb-32-revision-11-june-2022">Universal Serial Bus 3.2 Specification</a>, Revision 1.1, June 2022.</li>
      <li>[<span id="ref22">22</span>] MIDI Association, <a href="https://midi.org/details-about-midi-2-0-midi-ci-profiles-and-property-exchange-updated-june-2023">Details about MIDI 2.0, MIDI-CI, Profiles and Property Exchange</a>, updated June 2023.</li>
      <li>[<span id="ref23">23</span>] ISO/IEC and ITU-T, <a href="https://www.w3.org/Graphics/JPEG/itu-t81.pdf">Digital Compression and Coding of Continuous-Tone Still Images: Requirements and Guidelines</a>, ISO/IEC 10918-1 / ITU-T Recommendation T.81, 1992.</li>
      <li>[<span id="ref24">24</span>] ISO, <a href="https://www.iso.org/standard/75839.html">ISO 32000-2:2020—Document Management—Portable Document Format—Part 2: PDF 2.0</a>, 2020.</li>
    </ul>
  </li>
  <li>Apparently sudden products built from slow foundations
    <ul>
      <li>[<span id="ref25">25</span>] Apple, <a href="https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the-Phone-with-iPhone/">Apple Reinvents the Phone with iPhone</a>, press release, 9 January 2007.</li>
      <li>[<span id="ref26">26</span>] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, <a href="https://doi.org/10.1038/323533a0">Learning Representations by Back-Propagating Errors</a>, <em>Nature</em>, vol. 323, pp. 533–536, 1986.</li>
      <li>[<span id="ref27">27</span>] A. Krizhevsky, I. Sutskever, and G. E. Hinton, <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks">ImageNet Classification with Deep Convolutional Neural Networks</a>, in <em>Advances in Neural Information Processing Systems 25</em>, 2012.</li>
      <li>[<span id="ref28">28</span>] A. Vaswani et al., <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>, in <em>Advances in Neural Information Processing Systems 30</em>, 2017.</li>
      <li>[<span id="ref29">29</span>] Y. LeCun, Y. Bengio, and G. Hinton, <a href="https://doi.org/10.1038/nature14539">Deep Learning</a>, <em>Nature</em>, vol. 521, pp. 436–444, 2015.</li>
    </ul>
  </li>
  <li>Scientific and technological time scales
    <ul>
      <li>[<span id="ref30">30</span>] Nobel Assembly at Karolinska Institutet, <a href="https://www.nobelprize.org/prizes/medicine/2023/advanced-information/">The Nobel Prize in Physiology or Medicine 2023—Advanced Information: Discoveries Concerning Nucleoside Base Modifications that Enabled the Development of Effective mRNA Vaccines Against COVID-19</a>, 2023.</li>
      <li>[<span id="ref31">31</span>] K. Karikó, M. Buckstein, H. Ni, and D. Weissman, <a href="https://doi.org/10.1016/j.immuni.2005.06.008">Suppression of RNA Recognition by Toll-like Receptors: The Impact of Nucleoside Modification and the Evolutionary Origin of RNA</a>, <em>Immunity</em>, vol. 23, no. 2, pp. 165–175, 2005.</li>
      <li>[<span id="ref32">32</span>] Royal Swedish Academy of Sciences, <a href="https://www.nobelprize.org/uploads/2019/10/advanced-chemistryprize2019-2.pdf">The Nobel Prize in Chemistry 2019—Advanced Information: Lithium-Ion Batteries</a>, 2019.</li>
      <li>[<span id="ref33">33</span>] Royal Swedish Academy of Sciences, <a href="https://www.nobelprize.org/uploads/2020/10/advanced-chemistryprize2020.pdf">The Nobel Prize in Chemistry 2020—Advanced Information: A Tool for Genome Editing</a>, 2020.</li>
      <li>[<span id="ref34">34</span>] M. Jinek et al., <a href="https://doi.org/10.1126/science.1225829">A Programmable Dual-RNA-Guided DNA Endonuclease in Adaptive Bacterial Immunity</a>, <em>Science</em>, vol. 337, no. 6096, pp. 816–821, 2012.</li>
      <li>[<span id="ref35">35</span>] B. P. Abbott et al. (LIGO Scientific Collaboration and Virgo Collaboration), <a href="https://doi.org/10.1103/PhysRevLett.116.061102">Observation of Gravitational Waves from a Binary Black Hole Merger</a>, <em>Physical Review Letters</em>, vol. 116, 061102, 2016.</li>
      <li>[<span id="ref36">36</span>] Z. S. Morris, S. Wooding, and J. Grant, <a href="https://doi.org/10.1258/jrsm.2011.110180">The Answer Is 17 Years, What Is the Question: Understanding Time Lags in Translational Research</a>, <em>Journal of the Royal Society of Medicine</em>, vol. 104, no. 12, pp. 510–520, 2011.</li>
    </ul>
  </li>
  <li>Standards, infrastructure and network value
    <ul>
      <li>[<span id="ref37">37</span>] ISO, <a href="https://www.iso.org/standard/76912.html">ISO 668:2020—Series 1 Freight Containers—Classification, Dimensions and Ratings</a>, 2020.</li>
      <li>[<span id="ref38">38</span>] D. M. Bernhofen, Z. El-Sahli, and R. Kneller, <a href="https://doi.org/10.1016/j.jinteco.2015.09.001">Estimating the Effects of the Container Revolution on World Trade</a>, <em>Journal of International Economics</em>, vol. 98, pp. 36–50, 2016.</li>
      <li>[<span id="ref39">39</span>] M. L. Katz and C. Shapiro, <a href="https://www.jstor.org/stable/1814809">Network Externalities, Competition, and Compatibility</a>, <em>American Economic Review</em>, vol. 75, no. 3, pp. 424–440, 1985.</li>
      <li>[<span id="ref40">40</span>] J. Farrell and G. Saloner, <a href="https://doi.org/10.2307/2555589">Standardization, Compatibility, and Innovation</a>, <em>RAND Journal of Economics</em>, vol. 16, no. 1, pp. 70–83, 1985.</li>
      <li>[<span id="ref41">41</span>] LEGO Group, <a href="https://www.lego.com/en-us/history/articles/lego-system-in-play">LEGO System in Play</a>, <em>LEGO History</em>.</li>
      <li>[<span id="ref42">42</span>] Bureau International des Poids et Mesures, <a href="https://www.bipm.org/en/publications/si-brochure">The International System of Units (SI)</a>, 9th ed., 2019, updated 2022.</li>
    </ul>
  </li>
  <li>Organization, exploration and long-term productivity
    <ul>
      <li>[<span id="ref43">43</span>] J. G. March, <a href="https://doi.org/10.1287/orsc.2.1.71">Exploration and Exploitation in Organizational Learning</a>, <em>Organization Science</em>, vol. 2, no. 1, pp. 71–87, 1991.</li>
      <li>[<span id="ref44">44</span>] R. M. Solow, <a href="https://doi.org/10.2307/1926047">Technical Change and the Aggregate Production Function</a>, <em>Review of Economics and Statistics</em>, vol. 39, no. 3, pp. 312–320, 1957.</li>
      <li>[<span id="ref45">45</span>] K. J. Arrow, <a href="https://doi.org/10.2307/2295952">The Economic Implications of Learning by Doing</a>, <em>Review of Economic Studies</em>, vol. 29, no. 3, pp. 155–173, 1962.</li>
    </ul>
  </li>
</ul>]]></content><author><name>tsp</name></author><category term="Programming" /><category term="Opinion" /><category term="Basics" /><category term="How stuff works" /><category term="Internet" /><category term="Society" /><summary type="html"><![CDATA[Technology is often described as a race where success belongs to those who move the fastest. New frameworks, new interfaces and constant reinvention are frequently celebrated as progress in themselves. But is that really how civilization advances? Or is the true foundation of innovation something far less glamorous: the ability to preserve yesterdays achievements so tomorrows work can build upon them? In this article we take a journey through software engineering, science, industry and history to argue that stability - not perpetual disruption - is the real engine of progress. From POSIX and TCP/IP to container shipping, the SI system and modern research, we explore why the most valuable technologies are often the ones that quietly keep working for decades while the world around them evolves. Stable foundations do not slow progress - they allow it to compound.]]></summary></entry><entry><title type="html">Optimizing GLM for dual processor Westmere Xeon E5620: 200% gain (3x faster) till now</title><link href="https://www.tspi.at/2026/07/23/glmoptiwest01.html" rel="alternate" type="text/html" title="Optimizing GLM for dual processor Westmere Xeon E5620: 200% gain (3x faster) till now" /><published>2026-07-23T00:00:00+02:00</published><updated>2026-07-23T23:55:47+02:00</updated><id>https://www.tspi.at/2026/07/23/glmoptiwest01</id><content type="html" xml:base="https://www.tspi.at/2026/07/23/glmoptiwest01.html"><![CDATA[<p><a href="/2026/07/18/glm52colibrifreebsd.html">Running GLM-5.2</a> on an older dual-socket <a href="https://www.intel.de/content/www/de/de/products/sku/47925/intel-xeon-processor-e5620-12m-cache-2-40-ghz-5-86-gts-intel-qpi/specifications.html">Xeon E5620</a> workstation is mainly a performance engineering problem. The model does run, but it does so on a machine with limited memory bandwidth, limited cache capacity, no modern wide vector units and a runtime that has to combine dense transformer work, expert routing and NVMe-backed expert streaming. This means one cannot simply optimize <em>the model</em> in some abstract sense. One has to identify which concrete paths are expensive on the actual machine and then validate every change on the real workload.</p>

<p>This article summarizes the optimization work that has been done on <a href="https://github.com/tspspi/colibri">my Colibri fork</a> so far. The focus is on the actual inference path with the model on NVMe storage, measured sequentially, with a fixed prompt and without parallel GLM runs. I am intentionally not using the original repositories tiny test models or synthetic-only benchmarks as the main source of truth, because several apparently reasonable optimizations do not survive contact with the real inference path.</p>

<p><a href="/2026/07/18/glm52colibrifreebsd.html">Compared with the older benchmark figures</a>, the current rough result is that the system now reaches about <code class="language-plaintext highlighter-rouge">0.0600 tok/s</code> instead of roughly <code class="language-plaintext highlighter-rouge">0.0200 tok/s</code> in the presently best stable configurations. This corresponds to an overall speedup of about 3 times (approximately <strong>200% gain till now</strong>). On modern GPU systems this would still be a slow configuration, but on CPU-only machines it is large enough to make a practical difference.</p>

<p>Not every attempted optimization helped. Some were neutral, some were slower and some overlapped with the still unresolved token-repeat problem that can lead to responses that do not terminate cleanly. I am including those failed attempts as well, because they help to explain which parts of the runtime are actually worth touching and which ideas are probably not promising on this system.</p>

<p>Before discussing the individual changes it makes sense to briefly summarize how the inference flow looks from the perspective of optimization, because names such as <code class="language-plaintext highlighter-rouge">qrow_i8</code>, <code class="language-plaintext highlighter-rouge">gate/up</code> or <em>attention projection</em> are otherwise not very informative.</p>

<p>The machine that this optimization has been performed on was an dual processor Intel Xeon E5620 (8 physical cores, 16 logical threads) with 20 GByte or DDR3 unbuffered RAM operating on <code class="language-plaintext highlighter-rouge">FreeBSD 13.5-RELEASE (p6)</code>. The models were stored on an <a href="https://amzn.to/3RFcYHm">PCIx4 NVMe SSD</a>, the runtime was compiled with <code class="language-plaintext highlighter-rouge">clang 19.1.7</code> using it’s internal <code class="language-plaintext highlighter-rouge">-O3</code> optimizations.</p>

<ul>
  <li><a href="#how-the-numbers-were-taken">How The Numbers Were Taken</a></li>
  <li><a href="#what-the-runtime-is-actually-doing">What The Runtime Is Actually Doing</a></li>
  <li><a href="#where-we-stand-right-now">Where We Stand Right Now</a></li>
  <li>The optimizations
    <ul>
      <li><a href="#freebsd-ram-detection-and-planning-success-0-gain">FreeBSD RAM Detection And Planning (Success: 0% gain)</a></li>
      <li><a href="#freebsd-numa-slab-interleave-support-failed">FreeBSD NUMA Slab Interleave Support (Failed)</a></li>
      <li><a href="#ssse3-idot-kernels-for-non-avx2-x86-success-26-gain">SSSE3 IDOT Kernels For Non-AVX2 x86 (Success: 26% gain)</a></li>
      <li><a href="#sse-activation-quantization-for-qrow_i8-success-4-gain">SSE Activation Quantization For qrow_i8 (Success: 4% gain)</a></li>
      <li><a href="#rope-inverse-frequency-cache-success-7-gain">RoPE Inverse-Frequency Cache (Success: 7% gain)</a></li>
      <li><a href="#sse41-rmsnorm-output-pass-success-4-gain">SSE4.1 RMSNorm Output Pass (Success: 4% gain)</a></li>
      <li><a href="#sse41-exact-int4-attention-projection-success-13-gain">SSE4.1 Exact Int4 Attention Projection (Success: 13% gain)</a></li>
      <li><a href="#small-n-expert-reuse-for-int8-gateup-success-4-gain">Small-n Expert Reuse For Int8 gate/up (Success: 4% gain)</a></li>
      <li><a href="#small-n-shared-output-pass-for-int8-gateup-success-3-gain">Small-n Shared Output Pass For Int8 gate/up (Success: 3% gain)</a></li>
      <li><a href="#small-n-direct-row-gather-removal-success-2-gain">Small-n Direct-Row Gather Removal (Success: 2% gain)</a></li>
      <li><a href="#ssse3-4-way-unroll-experiment-failed">SSSE3 4-Way Unroll Experiment (Failed)</a></li>
      <li><a href="#forcing-8-openmp-threads-by-default-failed">Forcing 8 OpenMP Threads By Default (Failed)</a></li>
      <li><a href="#ssse3-output-pair-idot-driver-failed">SSSE3 Output-Pair IDOT Driver (Failed)</a></li>
      <li><a href="#the-first-exact-int4-projection-attempt-failed">The First Exact Int4 Projection Attempt (Failed)</a></li>
      <li><a href="#multi-row-gateup-fusion-failed">Multi-Row gate/up Fusion (Failed)</a></li>
      <li><a href="#thresholded-openmp-for-silugate--up-failed">Thresholded OpenMP For silu(gate) * up (Failed)</a></li>
      <li><a href="#broad-int8-gateup-reuse-failed">Broad Int8 gate/up Reuse (Failed)</a></li>
      <li><a href="#broad-int8-gateup-pair-fusion-failed">Broad Int8 gate/up Pair Fusion (Failed)</a></li>
      <li><a href="#parallel-plain-routing-rows-failed">Parallel Plain Routing Rows (Failed)</a></li>
      <li><a href="#sincosf-for-rope-failed">sincosf For RoPE (Failed)</a></li>
      <li><a href="#sse41-qt_addrow-on-decode-failed">SSE4.1 qt_addrow On Decode (Failed)</a></li>
      <li><a href="#sse41-exact-int8-attention-projection-failed">SSE4.1 Exact Int8 Attention Projection (Failed)</a></li>
      <li><a href="#sse41-expert-accumulation-helpers-failed">SSE4.1 Expert Accumulation Helpers (Failed)</a></li>
      <li><a href="#ssse3-2x2-int8-idot-driver-failed">SSSE3 2x2 Int8 IDOT Driver (Failed)</a></li>
      <li><a href="#small-n-shared-x-dual-dot-helper-failed">Small-n Shared-x Dual-Dot Helper (Failed)</a></li>
    </ul>
  </li>
  <li><a href="#conclusion-what-actually-mattered">Conclusion: What Actually Mattered</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/colibri003.png" alt="" /></p>

<h2 id="how-the-numbers-were-taken">How The Numbers Were Taken</h2>

<p>The important part is not only what was changed, but also how the measurements were taken. Optimization without measurement is worthless.</p>

<p>All runtime measurements were done against the <em>real model</em> on <a href="https://amzn.to/3RFcYHm">NVMe storage</a>, using the actual <code class="language-plaintext highlighter-rouge">coli chat</code> execution path and a fixed prompt. I explicitly avoided parallel GLM runs because the machine simply does not have the headroom for that. Where possible, changes were tested in alternating <code class="language-plaintext highlighter-rouge">baseline -&gt; modified -&gt; baseline -&gt; modified</code> sequences so that one-time penalties, cache warmup artifacts, and especially random background noise would not immediately fool the measurement.</p>

<p>For most of the deeper tuning work the prompt was held constant and the benchmark was stopped at a fixed prefill checkpoint (inference was intentionally terminated before full completion to reduce benchmark duration while still gathering meaningful data), which made comparisons slow but trustworthy. Exactness checks were also added for the low-level SIMD paths so that a speedup did not silently change the math.</p>

<p>This distinction turned out to matter quite a lot. Several ideas looked respectable in isolation and still lost once the real model and the real chat path were involved.</p>

<h2 id="what-the-runtime-is-actually-doing">What The Runtime Is Actually Doing</h2>

<p>At a very high level, each token passes through a sequence of transformer layers. Inside those layers one can roughly separate four kinds of work that matter for this optimization story:</p>

<ul>
  <li>dense projections for <a href="https://arxiv.org/abs/1706.03762">attention</a> and other always-active parts</li>
  <li>positional work such as <a href="https://arxiv.org/abs/2104.09864">RoPE</a></li>
  <li>normalization and elementwise vector operations</li>
  <li>and the <a href="https://arxiv.org/abs/2507.11181">Mixture-of-Experts</a> feed-forward path with routing, expert selection and expert matrix multiplications</li>
</ul>

<p>The last point is the unusual one. GLM-5.2 is not a dense model in which every token activates every feed-forward weight. Instead, a router decides which experts are needed for the current token and only those experts are evaluated. Colibri then tries to keep useful experts in RAM while loading missing ones from NVMe storage when necessary.</p>

<p>This means the forward pass is not just <em>multiplication of a lot of matrices</em>. It is a pipeline that repeatedly does something like the following:</p>

<ul>
  <li>take the current activations</li>
  <li>normalize and project them for attention</li>
  <li>apply positional rotation</li>
  <li>run routing to decide which experts are needed</li>
  <li>gather or quantize the routed activations</li>
  <li>evaluate the selected expert <code class="language-plaintext highlighter-rouge">gate</code>, <code class="language-plaintext highlighter-rouge">up</code> and related projections</li>
  <li>combine those results back into the residual stream</li>
  <li>continue to the next layer</li>
</ul>

<p>Several of the optimized paths sit directly in that sequence. <code class="language-plaintext highlighter-rouge">qrow_i8</code> is part of step 5, where floating-point activations are compressed into <code class="language-plaintext highlighter-rouge">int8</code> rows so that the following integer kernels can work more efficiently. The <code class="language-plaintext highlighter-rouge">gate/up</code> paths belong to step 6 inside the selected experts. <code class="language-plaintext highlighter-rouge">RoPE</code> belongs to step 3. <code class="language-plaintext highlighter-rouge">RMSNorm</code> belongs to step 2 and appears repeatedly across layers. The exact <code class="language-plaintext highlighter-rouge">int4</code> attention projection sits in step 2 as well, in the dense attention-side work that remains active regardless of which experts the router chooses.</p>

<p>There is also an important distinction between <strong>prefill</strong> and <strong>decode</strong>. During <em>prefill</em> the model consumes the whole prompt and therefore processes many tokens through many layers at once. During <em>decode</em> it usually extends the answer one token at a time with a warm <em>KV cache</em>. Some optimizations help prefill far more than decode, and some helpers that look interesting on decode barely matter for the larger prompt-ingestion cost. Since the original problem was that the runtime was already very slow while prefilling the first layers, most of the tuning effort naturally focused there.</p>

<p>Once viewed this way, the logic behind the optimization choices becomes less mysterious. One either tries to reduce repeated scalar work in steps that run constantly or one tries to simplify the expert path that dominates large parts of the CPU time on this machine.</p>

<h2 id="where-we-stand-right-now">Where We Stand Right Now</h2>

<p>The latest long running benchmark is the cleanest high-level summary at the moment. The currently stable best runs sit at about <code class="language-plaintext highlighter-rouge">0.0600 tok/s</code>. Note that <code class="language-plaintext highlighter-rouge">baseline</code> in the table below is already measured after the optimizations, it is the baseline with respect to the different runtime options that also influence speed massivly.</p>

<p>For the overall historical comparison one instead has to look back to the <a href="/2026/07/18/glm52colibrifreebsd.html">older benchmark numbers</a>, where all listed configurations were still around <code class="language-plaintext highlighter-rouge">0.0200 tok/s</code>. Relative to that earlier state, the present <code class="language-plaintext highlighter-rouge">0.0600 tok/s</code> result corresponds to roughly a <strong>threefold throughput increase</strong>.</p>

<table>
  <thead>
    <tr>
      <th>Configuration</th>
      <th style="text-align: right">n</th>
      <th style="text-align: right">tok/s</th>
      <th style="text-align: right">gen [s]</th>
      <th style="text-align: right">wall [s]</th>
      <th style="text-align: right">hit %</th>
      <th style="text-align: right">RSS [GB]</th>
      <th style="text-align: right">Speedup</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">pipe2</code></td>
      <td style="text-align: right">16</td>
      <td style="text-align: right">0.0600</td>
      <td style="text-align: right">487.0</td>
      <td style="text-align: right">526.0</td>
      <td style="text-align: right">16.0</td>
      <td style="text-align: right">24.6</td>
      <td style="text-align: right">1.200x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct_pipe2</code></td>
      <td style="text-align: right">14</td>
      <td style="text-align: right">0.0600</td>
      <td style="text-align: right">491.0</td>
      <td style="text-align: right">531.6</td>
      <td style="text-align: right">15.5</td>
      <td style="text-align: right">24.6</td>
      <td style="text-align: right">1.200x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">pilot_pipe2</code></td>
      <td style="text-align: right">16</td>
      <td style="text-align: right">0.0600</td>
      <td style="text-align: right">484.5</td>
      <td style="text-align: right">523.0</td>
      <td style="text-align: right">15.5</td>
      <td style="text-align: right">24.6</td>
      <td style="text-align: right">1.200x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">baseline</code></td>
      <td style="text-align: right">15</td>
      <td style="text-align: right">0.0500</td>
      <td style="text-align: right">570.0</td>
      <td style="text-align: right">608.6</td>
      <td style="text-align: right">15.0</td>
      <td style="text-align: right">24.7</td>
      <td style="text-align: right">1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct</code></td>
      <td style="text-align: right">1</td>
      <td style="text-align: right">0.0500</td>
      <td style="text-align: right">627.0</td>
      <td style="text-align: right">669.6</td>
      <td style="text-align: right">14.0</td>
      <td style="text-align: right">24.9</td>
      <td style="text-align: right">1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct_pilot_pipe2</code></td>
      <td style="text-align: right">1</td>
      <td style="text-align: right">0.0400</td>
      <td style="text-align: right">515.0</td>
      <td style="text-align: right">553.7</td>
      <td style="text-align: right">14.0</td>
      <td style="text-align: right">24.8</td>
      <td style="text-align: right">0.800x</td>
    </tr>
  </tbody>
</table>

<p>The small-<code class="language-plaintext highlighter-rouge">n</code> configurations should be treated with care. Most of those runs were terminated because of token repetition and responses that did not stop on their own.</p>

<h2 id="freebsd-ram-detection-and-planning-success-0-gain">FreeBSD RAM Detection And Planning (Success: 0% gain)</h2>

<p>This is not a glamorous optimization, but it had to be fixed.</p>

<p>Before this change, it was necessary to force <code class="language-plaintext highlighter-rouge">--ram 19</code> manually because automatic RAM detection on FreeBSD was not describing the machine correctly. This is not merely inconvenient, it also means the planner may make poor resource decisions and one may end up optimizing around a false view of the available memory. The first step was therefore simply to make the runtime understand the actual memory budget of the operating system it is running on.</p>

<p>There is no direct throughput win to put into a graph here, so I am calling this a <code class="language-plaintext highlighter-rouge">0%</code> speedup. But in practice it was a prerequisite for sane testing, and it removes one of the more annoying pieces of manual babysitting from normal runs.</p>

<h2 id="freebsd-numa-slab-interleave-support-failed">FreeBSD NUMA Slab Interleave Support (Failed)</h2>

<p>This was one of those ideas that made enough architectural sense that it was worth implementing even without a guaranteed speedup.</p>

<p>The machine is NUMA, memory pressure is real and the large slabs are exactly the sort of thing where bad placement can become expensive. So support was added to first-touch large allocations with interleave policy across both NUMA domains. In other words, the runtime can now spread those pages in a controlled way instead of leaving placement entirely to chance.</p>

<p>On the bounded real-model prefill check it did not improve performance. In fact, on that specific run it was slower. So strictly speaking this was a failed speed optimization. Still, I would not call it wasted work. Once the memory configuration changes (much more memory is added) or once larger runs become less memory-starved having the switch already there may become useful.</p>

<h2 id="ssse3-idot-kernels-for-non-avx2-x86-success-26-gain">SSSE3 IDOT Kernels For Non-AVX2 x86 (Success: 26% gain)</h2>

<p>This was the first clearly meaningful speedup and one of the easiest to justify.</p>

<p>The local box is old enough that one cannot simply assume newer vector instruction sets, but it still has enough SIMD to do better than the previous scalar-heavy path. So SSSE3 IDOT kernels were added specifically for x86 CPUs that do not have AVX2. The important part was not only to make them fast, but to keep them numerically exact against the existing code.</p>

<p>In practice this path matters because the quantized chat runtime spends a great deal of its time in integer dot products. Once activations have been quantized into <code class="language-plaintext highlighter-rouge">int8</code> rows and the corresponding weights are already stored in low precision, the forward pass repeatedly evaluates a large number of dot products to perform the projection through many quantized weights. If that inner loop is too scalar, the whole model becomes compute-bound very quickly even before storage latency dominates.</p>

<p>SSSE3 is useful here because it gives just enough vector machinery to process several packed small integers at once and to reduce them efficiently without requiring the much newer AVX2 instruction set. On an older Xeon this is the kind of low-level change one would expect to help: it attacks a path that is called constantly and does so using features the CPU already has but the code did not fully exploit before.</p>

<p>That paid off very clearly. On the bounded real-model prefill path this alone improved things by about <code class="language-plaintext highlighter-rouge">1.26x</code>.</p>

<h2 id="sse-activation-quantization-for-qrow_i8-success-4-gain">SSE Activation Quantization For qrow_i8 (Success: 4% gain)</h2>

<p>After the main <code class="language-plaintext highlighter-rouge">IDOT</code> work, the next obvious waste was the repeated scalar activation quantization feeding those kernels.</p>

<p>The old path spent too much time turning floating-point activations into the packed <code class="language-plaintext highlighter-rouge">int8</code> rows used later on. That is not flashy work, but when a kernel sits on the critical path often enough, even the boring helper becomes expensive. The change here was therefore simple: use SSE for the <code class="language-plaintext highlighter-rouge">max-abs</code> scan and for the packing work instead of leaving the whole thing scalar.</p>

<p>What this means in less implementation-heavy terms is that the runtime first has to compress a normal floating-point activation vector into a quantized form before the cheaper integer arithmetic can begin. If preparing that quantized row is itself slow, one gives back a noticeable fraction of the gain the integer kernels were supposed to provide. In other words, a fast quantized <code class="language-plaintext highlighter-rouge">matmul</code> (matrix multiplication) still suffers if its input preparation is lagging behind.</p>

<p>This was therefore a very natural follow-up optimization. The expectation was not that it would revolutionize end-to-end throughput, but rather that it would remove a visible bottleneck sitting directly in front of the newly accelerated <code class="language-plaintext highlighter-rouge">IDOT</code> path. That is exactly the kind of surrounding helper one should revisit after improving a core kernel.</p>

<p>This was not a huge change, but it was stable enough to keep. On the real chat path it came out at roughly <code class="language-plaintext highlighter-rouge">1.04x</code>.</p>

<h2 id="rope-inverse-frequency-cache-success-7-gain">RoPE Inverse-Frequency Cache (Success: 7% gain)</h2>

<p>This was a nice example of a small structural change beating a more heroic low-level one.</p>

<p>The RoPE code kept recomputing inverse-frequency terms for the same <code class="language-plaintext highlighter-rouge">qk_rope/theta</code> combinations over and over again. That is wasteful, especially on a machine where every repeated transcendental operation is more noticeable than it would be on a modern desktop CPU. So rather than trying to be clever inside the trigonometric functions themselves, the better answer was to stop asking for the same values repeatedly.</p>

<p>For readers not living inside transformer kernels every day, this path sits in the positional encoding part of attention. Before the model can compare or mix token states, it rotates parts of the query and key vectors according to their position. Some pieces of that calculation depend only on the model geometry and not on the individual token content. If those pieces are recomputed over and over again, the runtime burns cycles on repeated setup work instead of on the actual per-token math.</p>

<p>This was a good example of why one should first look for repeated work before chasing more exotic tricks. The expectation was straightforward: if a quantity is reused many times and does not change, caching it should cut overhead with relatively little risk. On older CPUs, avoiding unnecessary <code class="language-plaintext highlighter-rouge">powf</code>-style work is often a better bargain than trying to force a tiny gain out of already well-tuned arithmetic.</p>

<p>Caching those inverse frequencies per thread turned out to be a clean win. It was not earth-shattering, but a measured <code class="language-plaintext highlighter-rouge">1.07x</code> speedup. And as we will see small gain sums up quickly.</p>

<h2 id="sse41-rmsnorm-output-pass-success-4-gain">SSE4.1 RMSNorm Output Pass (Success: 4% gain)</h2>

<p>The full <code class="language-plaintext highlighter-rouge">RMSNorm</code> reduction was not touched here. Only the output pass was.</p>

<p>That matters because it kept the change relatively low-risk. The scalar accumulation stayed as it was, while the final <code class="language-plaintext highlighter-rouge">out[i] = x[i] * r * w[i]</code> work was vectorized with SSE4.1. On older x86 hardware that sort of targeted cleanup is often preferable to a grand rewrite because it yields a gain without exploding the complexity or the validation burden.</p>

<p><code class="language-plaintext highlighter-rouge">RMSNorm</code> itself appears all over the model. It is one of those deceptively simple operations that does not look expensive until one sees how often it runs and how wide the vectors are. The reduction part calculates a normalization factor and the output pass then applies that factor together with learned weights across the whole vector. Even if each individual pass is simple the total amount of elementwise work becomes significant across many layers.</p>

<p>The reason this particular optimization was plausible is that the output pass is regular. It is essentially the same formula applied element by element, which maps very naturally to SIMD. Leaving the reduction scalar avoided the more delicate exactness questions while still letting the CPU chew through the wide output loop more efficiently.</p>

<p>The result was modest but real: about <code class="language-plaintext highlighter-rouge">1.04x</code>.</p>

<h2 id="sse41-exact-int4-attention-projection-success-13-gain">SSE4.1 Exact Int4 Attention Projection (Success: 13% gain)</h2>

<p>This one deserves a little more attention because it was not the first attempt at the same general idea.</p>

<p>The attention projection path with exact <code class="language-plaintext highlighter-rouge">int4</code> weights was clearly expensive enough to deserve scrutiny, but the first SIMD version did not help. That could have been the end of the story. Instead, the path was reworked so that the packed int4 decode and accumulation order stayed aligned with the scalar reference while still cutting down the hot loop cost.</p>

<p>This path is easier to understand if one thinks of it as one of the dense projection steps inside attention, except the weights are stored in 4-bit packed form. That compact storage is good for memory traffic but it also means the CPU cannot simply read a convenient array of full integers and multiply away. It first has to unpack nibbles, interpret them correctly, apply scales, and then accumulate results. On an older machine that <em>unpacking cost is very visible</em>.</p>

<p>One would therefore expect a carefully written SIMD path to help, provided it does not subtly change the arithmetic order and thereby the result. That second condition turned out to be the difficult part. The first attempt was exact but not faster. The kept version only became convincing once the packed decode and reduction order were reworked so that the SIMD path matched the scalar behavior closely enough while still reducing the number of wasted operations.</p>

<p>The second try was much more convincing. On the real-model prefill checkpoint it reduced the median from <code class="language-plaintext highlighter-rouge">77</code> seconds to <code class="language-plaintext highlighter-rouge">68</code> seconds in the alternating run, which is about <code class="language-plaintext highlighter-rouge">1.13x</code>. In practical terms, this became one of the more useful wins in the session.</p>

<h2 id="small-n-expert-reuse-for-int8-gateup-success-4-gain">Small-n Expert Reuse For Int8 gate/up (Success: 4% gain)</h2>

<p>One of the more important observations from profiling was that the routed expert work was not dominated by large row counts. Quite the opposite.</p>

<p>At a deeper prefill checkpoint, roughly <code class="language-plaintext highlighter-rouge">79%</code> of the routed experts were running with <code class="language-plaintext highlighter-rouge">nr &lt;= 3</code>. That changed the whole tuning strategy. Instead of optimizing for a broad and elegant general case, it suddenly made much more sense to focus on the tiny-row majority case that the machine actually sees most of the time.</p>

<p>The first kept step based on that was to quantize the activation rows once and reuse them for both <code class="language-plaintext highlighter-rouge">gate</code> and <code class="language-plaintext highlighter-rouge">up</code>, but only for the small-<code class="language-plaintext highlighter-rouge">n</code> case. That avoided some duplicated work without overcomplicating the wider path. The result was about <code class="language-plaintext highlighter-rouge">1.04x</code>.</p>

<p>In terms of model structure, <code class="language-plaintext highlighter-rouge">gate</code> and <code class="language-plaintext highlighter-rouge">up</code> are two projections used inside the feed-forward expert blocks. If both of them consume the same routed activation rows, quantizing those rows twice is simply duplicated work. The reason the first broad reuse attempt failed, but the small-<code class="language-plaintext highlighter-rouge">n</code> one survived, is likely that overhead matters differently depending on how many rows are being handled. For the tiny-row dominant case, the reused preparation work is a larger fraction of the total cost, so saving it is actually noticeable.</p>

<h2 id="small-n-shared-output-pass-for-int8-gateup-success-3-gain">Small-n Shared Output Pass For Int8 gate/up (Success: 3% gain)</h2>

<p>Once the small-<code class="language-plaintext highlighter-rouge">n</code> reuse path was in place, another inefficiency became obvious: even in that case, <code class="language-plaintext highlighter-rouge">gate</code> and <code class="language-plaintext highlighter-rouge">up</code> still launched separate output passes.</p>

<p>That is the kind of thing one often misses before looking closely, because the code still <em>works</em> and the algorithm is unchanged. But on a machine like this, launching the same style of work twice for a tiny-row dominant workload is the sort of overhead that starts to matter. So the next change merged those passes for the already-specialized small-<code class="language-plaintext highlighter-rouge">n</code> path.</p>

<p>The expectation here was fairly direct. Once the quantized activation rows are already shared, the runtime is still doing two very similar sweeps over the outputs, one for <code class="language-plaintext highlighter-rouge">gate</code> and one for <code class="language-plaintext highlighter-rouge">up</code>. For large batches this extra structure may be hidden beneath the much larger compute load. For very small routed batches, however, those framework costs stand out much more sharply. Merging the output-side handling was therefore an attempt to reduce scheduling and loop overhead in the exact regime this machine sees most often.</p>

<p>Again, this was not dramatic, but it was consistent enough to keep at roughly <code class="language-plaintext highlighter-rouge">1.03x</code>.</p>

<h2 id="small-n-direct-row-gather-removal-success-2-gain">Small-n Direct-Row Gather Removal (Success: 2% gain)</h2>

<p>After that, there was still one last bit of unnecessary movement left in the same area.</p>

<p>The CPU MoE path copied <code class="language-plaintext highlighter-rouge">nr * hidden</code> floats into a temporary buffer and then immediately quantized those copied rows for the int8 <code class="language-plaintext highlighter-rouge">gate/up</code> path. Given that the hot case was mostly <code class="language-plaintext highlighter-rouge">nr &lt;= 3</code>, it made more sense to stop copying and quantize directly from the original routed rows.</p>

<p>This is one of the less glamorous kinds of optimization. No fancy new instruction set is involved. The code simply <strong>stopped moving data into a temporary place when the next step could just as well operate on the original rows</strong>. On a wide vector machine that might be a footnote. On an older memory-constrained dual-socket system, unnecessary copies are harder to hide.</p>

<p>One would not expect a huge improvement from this alone, and indeed it is the smallest kept gain in the list. But it fits the general theme very well: once the workload shape is understood, the little pieces of avoidable work around the dominant small-<code class="language-plaintext highlighter-rouge">n</code> expert case become worth trimming.</p>

<p>That only saved about <code class="language-plaintext highlighter-rouge">1.02x</code>, so this is the smallest of the kept wins. But as we see small wins sum up surprisingly quick.</p>

<h2 id="ssse3-4-way-unroll-experiment-failed">SSSE3 4-Way Unroll Experiment (Failed)</h2>

<p>This was a rather classical microbenchmark trap.</p>

<p>A wider SSSE3 unroll for <code class="language-plaintext highlighter-rouge">dot_i8i8</code> and <code class="language-plaintext highlighter-rouge">dot_i4i8</code> looked appealing. Unfortunately, once the real-model prefill path was measured, it came out slower than the simpler implementation. That was enough reason to drop it.</p>

<h2 id="forcing-8-openmp-threads-by-default-failed">Forcing 8 OpenMP Threads By Default (Failed)</h2>

<p>This was motivated by the observation that the machine often did not look fully busy at every instant. Since the machine has 8 physical cores but 16 logical threads limiting to the cores may actually reduce resource sharing and thus increase efficiency. On the real chat path the opposite happened: the <code class="language-plaintext highlighter-rouge">16</code>-thread behavior was faster than the patched <code class="language-plaintext highlighter-rouge">8</code>-thread variant. So the idea was discarded.</p>

<h2 id="ssse3-output-pair-idot-driver-failed">SSSE3 Output-Pair IDOT Driver (Failed)</h2>

<p>The thought here was reasonable enough: if the output side is expensive, maybe pair outputs and share more work there. In practice that version lost on the real chat path.</p>

<h2 id="the-first-exact-int4-projection-attempt-failed">The First Exact Int4 Projection Attempt (Failed)</h2>

<p>The first crack at an SSE4.1 exact <code class="language-plaintext highlighter-rouge">int4</code> attention projection was technically correct and still not worth keeping.</p>

<p>That was mildly frustrating, but also educational. Packed low-bit arithmetic tends to punish premature optimism. The first version preserved exactness and still ended up slightly slower on the real workload. Only the later reworked version actually earned its place.</p>

<h2 id="multi-row-gateup-fusion-failed">Multi-Row gate/up Fusion (Failed)</h2>

<p>At first glance this looked like the sort of optimization that should obviously help.</p>

<p>Fuse more work, reduce overhead, share intermediate state and surely the machine gets happier. Except here it did not. The real comparison was <em>clearly</em> worse, not marginally noisy-worse but properly worse, so this idea was dropped.</p>

<h2 id="thresholded-openmp-for-silugate--up-failed">Thresholded OpenMP For silu(gate) * up (Failed)</h2>

<p>This was another attempt to improve the glue logic around the expert path rather than the heavy dot products themselves.</p>

<p>A thresholded OpenMP helper for the <code class="language-plaintext highlighter-rouge">silu(gate) * up</code> stage sounded plausible, especially given the machine’s intermittent parallelism profile. But once tested on the real model, the median was slightly slower. That is not enough to justify more code and another special case.</p>

<h2 id="broad-int8-gateup-reuse-failed">Broad Int8 gate/up Reuse (Failed)</h2>

<p>The first version of reuse for expert <code class="language-plaintext highlighter-rouge">gate/up</code> was too broad.</p>

<p>Conceptually it was fine: the path quantized the same activation rows twice, so why not quantize once and share the result? The answer, at least on this host, was that the general version did not pay for itself. Only after narrowing the idea down to the small-<code class="language-plaintext highlighter-rouge">n</code> majority case did it become useful.</p>

<p>The likely explanation is that the general case drags in bookkeeping and structure that are not free. If one tries to optimize every case with the same machinery, the overhead can easily cancel the intended gain. This is precisely why the histogram of routed expert row counts turned out to be so important: it showed that the machine does not need a universally elegant answer here. It needs a very efficient answer for the overwhelmingly common tiny-row case.</p>

<p>This was one of the more useful failures because it directly led to the narrower version that did survive.</p>

<h2 id="broad-int8-gateup-pair-fusion-failed">Broad Int8 gate/up Pair Fusion (Failed)</h2>

<p>The same story repeated for the broader fused <code class="language-plaintext highlighter-rouge">gate</code> and <code class="language-plaintext highlighter-rouge">up</code> output pass.</p>

<p>As a general optimization it was slightly slower on the deeper prefill checkpoint. Once again the lesson was that the real workload here is lopsided. Broad elegant machinery was not what this machine wanted. Narrow special handling for the common tiny-row case worked better.</p>

<h2 id="parallel-plain-routing-rows-failed">Parallel Plain Routing Rows (Failed)</h2>

<p>Because thread-state sampling showed periods of low visible parallelism, it was natural to look for serial windows between the heavy kernels.</p>

<p>One such candidate was the plain routing row selection stage in <code class="language-plaintext highlighter-rouge">moe()</code>. That work was parallelized experimentally - the idea made a certain amount of sense. Unfortunately the longer alternating run showed the modified path was slower. So even though the motivation was sound, the machine did not reward it.</p>

<p><strong>Not every serial-looking region deserves immediate parallelization</strong>. Some phases are simply too small, too synchronization-heavy or too memory-irregular for additional OpenMP structure to help. On a dual-socket system the cost of waking workers, synchronizing them and touching data across domains can easily eat the hoped-for gain.</p>

<h2 id="sincosf-for-rope-failed">sincosf For RoPE (Failed)</h2>

<p>This was one of the more annoying almost-wins.</p>

<p>After adding the inverse-frequency cache, each new position still paid for one <code class="language-plaintext highlighter-rouge">cosf</code> and one <code class="language-plaintext highlighter-rouge">sinf</code> per lane. On paper, using <code class="language-plaintext highlighter-rouge">sincosf</code> looked attractive and even local microbench behavior was encouraging. On the real benchmark, however, the result was in the noise and leaned slightly slower. That was not enough to keep it.</p>

<p>This also confirmed that <strong>the cache mattered much more than the trigonometric call style</strong>.</p>

<h2 id="sse41-qt_addrow-on-decode-failed">SSE4.1 qt_addrow On Decode (Failed)</h2>

<p>This looked good in a model-shaped local microbenchmark and then refused to matter where it counted.</p>

<p><code class="language-plaintext highlighter-rouge">qt_addrow()</code> participates in the exact absorbed-attention helper used on decode-sized batches, so it was not an unreasonable target. But the real-model warm-KV decode test did not improve. If a helper gets faster and the application does not, then the only honest conclusion is that one optimized the wrong thing or optimized too little of it.</p>

<p><strong>Decode-sized work behaves differently from long prefill work</strong>. A helper can be visibly faster in isolation and still be too small a fraction of the full decode step to matter. This is why real-model validation remained essential throughout the session.</p>

<h2 id="sse41-exact-int8-attention-projection-failed">SSE4.1 Exact Int8 Attention Projection (Failed)</h2>

<p>There was a small apparent gain here, but not one I would trust.</p>

<p>The measured improvement vanished into the noise floor. Given the added complexity, that was not good enough. So the exact <code class="language-plaintext highlighter-rouge">int8</code> attention path experiment was discarded.</p>

<h2 id="sse41-expert-accumulation-helpers-failed">SSE4.1 Expert Accumulation Helpers (Failed)</h2>

<p>The expert path still had large scalar accumulation loops, so accelerating those looked sensible. The pairwise result leaned the wrong way and the apparent raw median win was not stable. <strong>Too much uncertainty, too little payoff</strong>. Out it went.</p>

<h2 id="ssse3-2x2-int8-idot-driver-failed">SSSE3 2x2 Int8 IDOT Driver (Failed)</h2>

<p>This was a more ambitious follow-up to the earlier output-pair idea.</p>

<p>The plan was to share both weight and activation loads across two outputs and two rows, which certainly sounds more substantial than merely pairing outputs. Unfortunately, the measured result was effectively neutral to slightly worse. <strong>Neutral code is not free code, so it did not stay</strong>.</p>

<h2 id="small-n-shared-x-dual-dot-helper-failed">Small-n Shared-x Dual-Dot Helper (Failed)</h2>

<p>After the kept small-<code class="language-plaintext highlighter-rouge">n</code> changes there was still the temptation to push the specialization one level further and reuse the loaded activation row inside a dual-dot helper.</p>

<p>That turned out to be one specialization too far. The effect leaned slightly slower and never justified the extra path. So this was the point where it made sense to stop <em>being clever</em> and accept that the simpler small-<code class="language-plaintext highlighter-rouge">n</code> optimizations were already the better trade-off.</p>

<h2 id="conclusion-what-actually-mattered">Conclusion: What Actually Mattered</h2>

<p>Looking back over the whole session, the pattern is fairly consistent.</p>

<p>The biggest wins did not come from broad theoretical cleanups. They came from identifying the specific CPU features this old x86 machine still has, enabling them carefully and then focusing on the workload shape the real model actually produces. In practice that meant:</p>

<ul>
  <li>Using SSSE3 and SSE4.1 where the existing path was still too scalar</li>
  <li>Removing repeated helper work around quantization and RoPE</li>
  <li>Specializing the routed expert path for the tiny <code class="language-plaintext highlighter-rouge">nr</code> values that dominate on this host</li>
</ul>

<p>Just as importantly, <strong>several obvious ideas did not help at all</strong>. More fusion was not automatically better. More OpenMP was not automatically better. A microbenchmark win was definitely not automatically a real-model win. And some things that are architecturally useful - such as the NUMA support - still do not yet translate into a direct speedup on the current memory setup. Maybe they will when I get a substantial memory upgrade, which in turn will also massively reduce QPI traffic between the NUMA domains.</p>

<p>So there is visible progress: about <strong>200% overall gain till now</strong> relative to the earlier benchmark state - roughly <strong>a <code class="language-plaintext highlighter-rouge">3x</code> throughput increase</strong>. But just as valuable is the narrower search space we are left with. The machine has told us rather clearly which classes of ideas it likes and which ones it does not.</p>

<p>That is the result of an proper optimization session: not only a faster program, but also a much clearer understanding of where not to spend time next.</p>

<p>Of course this is not the end of the story: In future there will be more tries to optimize critical paths and as well as reduce QPI traffic when more RAM is available. Leaving the disk bound operation regime will also show if the discarded optimizations may actually have an effect on a non-disk and memory bound machine.</p>

<h2 id="references">References</h2>

<ul>
  <li>Fast <a href="https://amzn.to/3RFcYHm">PCIe x4 NVMe SSD</a></li>
  <li><a href="/2026/07/18/glm52colibrifreebsd.html">Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD</a></li>
  <li>Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a></li>
  <li>Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu; <a href="https://arxiv.org/abs/2104.09864">RoFormer: Enhanced Transformer with Rotary Position Embedding</a></li>
  <li>Danyang Zhang, Junhao Song, Ziqian Bi, Xinyuan Song, Yingfang Yuan, Tianyang Wang, Joe Yeong, Junfeng Hao, <a href="https://arxiv.org/abs/2507.11181">Mixture of Experts in Large Language Models</a></li>
</ul>]]></content><author><name>tsp</name></author><category term="Programming" /><category term="Artificial Intelligence" /><category term="FreeBSD" /><category term="Large Language Models" /><category term="Computational linear algebra" /><category term="Optimization" /><summary type="html"><![CDATA[In this article I am summarizing a bunch of optimizations I performed on the colibri runtime for GLM-5.2 specifically for an Westmere Xeon E5620 dual-socket Xeon machine. It is a summary of the ideas, what worked and what did not - and it shows why some seemingly cleaver ideas were simply not worth keeping.]]></summary></entry><entry><title type="html">Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD</title><link href="https://www.tspi.at/2026/07/18/glm52colibrifreebsd.html" rel="alternate" type="text/html" title="Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD" /><published>2026-07-18T00:00:00+02:00</published><updated>2026-07-18T13:00:01+02:00</updated><id>https://www.tspi.at/2026/07/18/glm52colibrifreebsd</id><content type="html" xml:base="https://www.tspi.at/2026/07/18/glm52colibrifreebsd.html"><![CDATA[<p>Modern frontier scale Large Language Models are usually associated with clusters of expensive GPUs, hundreds of gigabytes of fast memory and correspondingly large electricity bills. This makes running these models locally rather challenging - or prohibitive for a typical hobby or home user. Smaller models have become surprisingly capable, but there are still tasks for which model scale and the capabilities emerging from it make a noticeable difference.</p>

<p>Nevertheless I am currently running the 744 billion parameter <a href="https://z.ai/blog/glm-5.2">GLM-5.2</a> model on an old dual processor Xeon workstation containing only 20 GB of DDR3 ECC memory. There is no large GPU array hidden in the machine. The model is evaluated on the CPUs while its experts are streamed from NVMe storage using the <a href="https://github.com/JustVugg/colibri">Colibrì inference runtime</a>.</p>

<p>This is not fast. It is, however, a complete frontier scale model producing useful answers on hardware on which its quantized weights could not possibly fit into memory. This article takes a look at why large models are interesting, how Mixture-of-Experts models make this particular trick possible, how Colibrì turns NVMe storage into another level of the memory hierarchy and how I made the runtime work on FreeBSD.</p>

<ul>
  <li><a href="#why-run-such-a-huge-model-locally">Why Run Such a Huge Model Locally?</a></li>
  <li><a href="#the-memory-problem">The Memory Problem</a></li>
  <li><a href="#mixture-of-experts">Mixture of Experts</a></li>
  <li><a href="#colibri-and-nvme-streaming">Colibri and NVMe Streaming</a></li>
  <li><a href="#the-test-system">The Test System</a></li>
  <li><a href="#ssd-setup">SSD setup</a></li>
  <li><a href="#activation-vector-steering">Activation Vector Steering</a>
    <ul>
      <li><a href="#does-steering-reduce-performance">Does Steering Reduce Performance?</a></li>
    </ul>
  </li>
  <li><a href="#running-colibri-behind-an-api-gateway">Running Colibri Behind an API Gateway</a></li>
  <li><a href="#using-the-model-with-codex">Using the Model with Codex</a></li>
  <li><a href="#performance-measurements">Performance Measurements</a></li>
  <li><a href="#what-this-does-not-do">What This Does Not Do</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/colibri001.png" alt="" /></p>

<h2 id="why-run-such-a-huge-model-locally">Why Run Such a Huge Model Locally?</h2>

<p>Small language models are often entirely sufficient. A model with a few billion parameters can summarize text, extract structured data, answer straightforward questions and perform many constrained automation tasks. Models in the range of 20 to 70 billion parameters can already be highly useful for programming, scientific discussion and agentic applications. This naturally raises the question why anyone would want to operate a model containing hundreds of billions of parameters at home.</p>

<p>The simple answer is that <em>model scale still matters</em>. Increasing scale does not merely add more memorized facts. Larger models tend to become better at combining distant concepts, maintaining constraints through longer reasoning processes, handling unusual problems and recovering when the first attempted solution does not work. They also tend to be more reliable during long running agentic tasks in which small errors accumulate over many individual decisions. Such phase transitions are often described as <strong>emergent properties</strong>. At the moment of writing this article it is not totally clear why these properties emerge and why they are also present in massive MoE networks.</p>

<p>Of course scale is not the only important factor. Model architecture, training data, post-training and inference methods are also at least as important as the raw parameter count. Nevertheless there is a very real practical difference between a small model that occasionally demonstrates a capability and a large model that can apply it reliably. For long horizon reasoning, difficult programming tasks, scientific problems and autonomous tool use, this reliability may be more important than the ability to produce a single impressive benchmark answer.</p>

<p>Running a capable large model locally additionally provides:</p>

<ul>
  <li>Control over the complete inference environment</li>
  <li>Independence from an external API and changing usage limits</li>
  <li>The ability to inspect and modify the runtime</li>
  <li>Privacy for local documents and source code</li>
  <li>Reproducible experiments with quantization, routing and activation steering</li>
  <li>The simple satisfaction of operating the entire system oneself</li>
</ul>

<p>The last point should not be underestimated.</p>

<h2 id="the-memory-problem">The Memory Problem</h2>

<p>GLM-5.2 contains approximately 744 billion parameters. Storing a parameter using a 32 bit floating point value requires four bytes. Even without considering any runtime state, this would result in</p>

[
744 \cdot 10^9 \cdot 4\ \mathrm{bytes} \approx 2.98\ \mathrm{TB}
]

<p>Reducing the numerical precision reduces the storage requirement correspondingly:</p>

<table>
  <thead>
    <tr>
      <th>Weight representation</th>
      <th>Approximate storage for 744B parameters</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>FP32</td>
      <td>2.98 TB</td>
    </tr>
    <tr>
      <td>FP16 or BF16</td>
      <td>1.49 TB</td>
    </tr>
    <tr>
      <td>INT8</td>
      <td>744 GB</td>
    </tr>
    <tr>
      <td>INT4</td>
      <td>372 GB</td>
    </tr>
  </tbody>
</table>

<p>The converted model used for my Colibrì experiment occupies approximately 383.7 GB in 144 shards. The small difference from the idealized INT4 calculation is caused by additional model data, metadata and components that are stored at different precision.</p>

<p>The workstation currently contains only 20 GB of physical RAM. The model is therefore not just slightly too large. Its weight files are about nineteen times larger than the entire installed memory.</p>

<p>Three different resource limits are often mixed together when discussing local language model inference:</p>

<ul>
  <li><strong>Storage capacity</strong> determines whether the complete model can be kept at all.</li>
  <li><strong>Memory capacity</strong> determines how much of the model can remain immediately accessible to the processors.</li>
  <li><strong>Bandwidth</strong> determines how quickly weights can be transported from storage or memory to the processing units.</li>
</ul>

<p>Quantization attacks the first two problems by reducing the number of bytes per parameter. It does not make them disappear. A dense 744B model quantized to four bits would still normally require hundreds of gigabytes of RAM or VRAM during inference, let alone the space required for the KV cache, runtime buffers and temporary activations.</p>

<p><em>Fortunately GLM-5.2 is not a dense model.</em></p>

<h2 id="mixture-of-experts">Mixture of Experts</h2>

<p>GLM-5.2 uses a sparse <strong>Mixture-of-Experts</strong> architecture. Instead of passing every token through every feed-forward parameter, the model contains a collection of expert networks. A learned router selects a small subset of them for every token and layer.</p>

<p>In a simplified representation the output of such a layer is</p>

[
\mathbf y =
\sum_{i \in \operatorname{TopK}(r(\mathbf x))} g_i(\mathbf x) E_i(\mathbf x)
]

<p>where $r$ is the router, $E_i$ denotes an expert and $g_i$ is the corresponding routing weight. Only the ($\operatorname{top}_k$) experts selected by the router have to be evaluated for the current token. Tuning how many experts are evaluates is another tuning option for the model, available via the <code class="language-plaintext highlighter-rouge">--topp</code> parameter.</p>

<p>According to the Colibrì documentation, the 744B model activates approximately 40 billion parameters per token. The routed experts whose selection changes between tokens account for about 11 GB of weight data. The computation required for a token is therefore much smaller than the total parameter count might suggest.</p>

<p>It is important to distinguish computational sparsity from storage sparsity. The runtime does not know in advance which experts will be selected by all future tokens. Consequently all experts still have to be stored somewhere. Conventional runtimes keep them in RAM or VRAM so that any routing decision can be serviced immediately.</p>

<p>This leads to an interesting possibility: If only a small subset is required at any one time, perhaps the remaining experts do not have to reside in fast memory at all.</p>

<h2 id="colibri-and-nvme-streaming">Colibri and NVMe Streaming</h2>

<p><a href="https://github.com/JustVugg/colibri">Colibrì</a> is a lightweight inference runtime written by <a href="https://github.com/JustVugg">JustVugg</a> in C <em>specifically</em> for GLM-5.2 and similar sparse models. It treats GPU memory, system RAM and storage as levels of a managed memory hierarchy.</p>

<p>The dense model components and other continuously required data remain resident in RAM or VRAM. Routed experts are cached in RAM, while the complete expert collection remains on storage. When a required expert is not present in the cache it is loaded from disk. Frequently used experts may remain cached or become pinned while less useful entries are replaced.</p>

<p>Conceptually this resembles virtual memory, but the runtime <em>understands</em> the structure of the model. It knows which files contain individual experts, which experts the router selected and which data is likely to be useful again. This allows it to make considerably more informed decisions than a generic operating system page cache, like simply <code class="language-plaintext highlighter-rouge">mmap</code> of the model with <a href="https://ollama.com/">ollama</a></p>

<blockquote>
  <p><strong>Insufficient RAM no longer makes inference impossible. It makes inference slower.</strong></p>
</blockquote>

<p>This is a rather substantial change. The machine still needs enough storage for the complete model and enough resident memory for the dense components, attention state, runtime buffers and an expert cache. It does not, however, need enough RAM to hold all 383.7 GB of model data simultaneously.</p>

<p>Colibrì also implements several techniques intended to hide or reduce the resulting I/O cost:</p>

<ul>
  <li>A learned expert cache and the ability to pin frequently used experts</li>
  <li>Automatic sizing of the cache according to the RAM budget</li>
  <li>Direct I/O to avoid unhelpful duplication in the operating system cache (bypassing the operating system cache layer)</li>
  <li>Pipeline workers that overlap expert loading with computation</li>
  <li>Experimental router look-ahead to prefetch experts for the following layer</li>
  <li>Multi-token prediction for speculative decoding</li>
  <li>Optional RAM and GPU tiers for selected experts</li>
</ul>

<p>None of these techniques can repeal the bandwidth and latency characteristics of the storage device. They can, however, keep it busy more efficiently and avoid some unnecessary reads.</p>

<h2 id="the-test-system">The Test System</h2>

<p>My current test system is an older dual processor Xeon workstation:</p>

<table>
  <thead>
    <tr>
      <th>Component</th>
      <th>Configuration</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>CPU</td>
      <td>Dual Intel Xeon E3, six cores per processor</td>
    </tr>
    <tr>
      <td>Memory</td>
      <td>20 GB DDR3 ECC UDIMM</td>
    </tr>
    <tr>
      <td>Operating system</td>
      <td>FreeBSD (as usual for all of my machines)</td>
    </tr>
    <tr>
      <td>Model storage</td>
      <td><a href="https://amzn.to/3RFcYHm">PCIe attached NVMe SSD</a> via an <a href="https://amzn.to/4wh83vu">NVMe to PCIe adapter</a></td>
    </tr>
    <tr>
      <td>Model</td>
      <td><a href="https://huggingface.co/zai-org/GLM-5.2">GLM-5.2, 744B MoE</a>, INT4 converted locally</td>
    </tr>
    <tr>
      <td>Model size</td>
      <td>Approximately 383.7 GB in 144 shards</td>
    </tr>
    <tr>
      <td>Inference</td>
      <td>CPU with NVMe streamed experts</td>
    </tr>
  </tbody>
</table>

<p>This is emphatically not the hardware one would normally associate with frontier scale inference. The limited memory is particularly interesting because it leaves relatively little space for the expert cache after the resident model components and operating system are taken into account.</p>

<p>The machine will soon be upgraded to <em>96 GB RAM</em>. This should allow a much larger fraction of useful experts (from about 12 GB to 88 GB) to remain cached and will provide an interesting comparison on otherwise identical hardware. It may also move the bottleneck away from storage and towards DDR3 memory bandwidth or integer matrix multiplication.</p>

<p>The dual socket ccNUMA arrangement introduces an additional variable. Memory access on such systems is non-uniform, and thread placement as well as the location of allocated memory may influence performance. This is something worth investigating separately after establishing a useful baseline.</p>

<h2 id="ssd-setup">SSD setup</h2>

<p>The NVMe disk setup is pretty straight forward. Just keep in mind that <code class="language-plaintext highlighter-rouge">nvme0</code> and <code class="language-plaintext highlighter-rouge">nvme0ns1</code> are the controller and namespace device, the storage device is exposed as <code class="language-plaintext highlighter-rouge">nvd</code> device. The properties of the <a href="https://amzn.to/3RFcYHm">SSD that I am using</a> are shown below:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo nvmecontrol identify nvme0ns1
Size:                        1953525168 blocks
Capacity:                    1953525168 blocks
Utilization:                 1953525168 blocks
Thin Provisioning:           Not Supported
Number of LBA Formats:       1
Current LBA Format:          LBA Format #00
Data Protection Caps:        Not Supported
Data Protection Settings:    Not Enabled
Multi-Path I/O Capabilities: Not Supported
Reservation Capabilities:    Not Supported
Format Progress Indicator:   0% remains
Deallocate Logical Block:    Read 00h
Optimal I/O Boundary:        0 blocks
NVM Capacity:                0 bytes
Preferred Write Granularity: 8 blocks
Preferred Write Alignment:   8 blocks
Preferred Deallocate Granul: 8 blocks
Preferred Deallocate Align:  8 blocks
Optimal Write Size:          8 blocks
Globally Unique Identifier:  000000000000000100a075255449c16e
IEEE EUI64:                  00a075015449c16e
LBA Format #00: Data Size:   512  Metadata Size:     0  Performance: Best
</code></pre></div></div>

<p>After making sure it’s the right device to overwrite I created a <code class="language-plaintext highlighter-rouge">zpool</code> and a <code class="language-plaintext highlighter-rouge">zfs</code> filesystem for the model files:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zpool create -o ashift=12 -o mountpoint=/nvme1 llm_nvme1 /dev/nvd0 
mkdir /llm_nvme1/glm52_i4_original
zfs create llm_nvme1/glm52_i4_original
</code></pre></div></div>

<h2 id="porting-colibri-to-freebsd">Porting Colibri to FreeBSD</h2>

<p>The original runtime primarily targeted Linux, Windows and macOS. Since the actual engine is written in C and has very few dependencies, adapting it to FreeBSD did not require a fundamental port, it was merly some basic single line changes.</p>

<p>My changes are available in my <a href="https://github.com/tspspi/colibri">Colibrì fork on GitHub</a>.</p>

<p>The build system contained a phony <code class="language-plaintext highlighter-rouge">glm</code> target intended to map the target name to <code class="language-plaintext highlighter-rouge">glm.exe</code> on Windows. On Unix systems <code class="language-plaintext highlighter-rouge">EXE</code> is empty, which made the phony target collide with the actual output target. Guarding the alias so it is only created when an executable suffix exists fixes the <code class="language-plaintext highlighter-rouge">Makefile</code> behaviour.</p>

<p>The engine also uses <code class="language-plaintext highlighter-rouge">sys/resource.h</code>, <code class="language-plaintext highlighter-rouge">sys/mman.h</code> and <code class="language-plaintext highlighter-rouge">mlock</code> related functionality on supported Unix-like systems. Adding <code class="language-plaintext highlighter-rouge">__FreeBSD__</code> to the existing platform condition enables the corresponding implementation:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
#include</span> <span class="cpf">&lt;sys/resource.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;sys/mman.h&gt;</span><span class="cp">
#endif
</span></code></pre></div></div>

<p>With GCC and its OpenMP runtime installed, the runtime can then be built and executed natively on FreeBSD.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd colibri/c
gmake glm
</code></pre></div></div>

<p>After the build suceeded the download can be initiated - with the conversion taking place on the fly. This takes around 2 days, especially due to rate limiting of <a href="https://huggingface.co/">HuggingFace</a>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./coli convert --model /llm1/glm52_i4_001
</code></pre></div></div>

<p>On my system I currently supply the GCC runtime library path explicitly:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">env </span><span class="nv">LD_LIBRARY_PATH</span><span class="o">=</span>/usr/local/lib/gcc14 <span class="se">\</span>
    <span class="nv">COLI_MODEL</span><span class="o">=</span>/llm_nvme1/glm52_i4 <span class="se">\</span>
    ./coli plan <span class="nt">--ram</span> 19
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">plan</code> command is useful before starting the large model. It displays the detected model size, storage availability, selected RAM budget and the planned distribution of model data across the available tiers.</p>

<p>An interactive chat can be started using:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">env </span><span class="nv">LD_LIBRARY_PATH</span><span class="o">=</span>/usr/local/lib/gcc14 <span class="se">\</span>
    <span class="nv">COLI_MODEL</span><span class="o">=</span>/llm_nvme1/glm52_i4 <span class="se">\</span>
    ./coli chat <span class="nt">--ram</span> 19
</code></pre></div></div>

<p>After the model has initialized, prompts can be entered directly. Responses do not arrive at the speed expected from a hosted GPU service, but they do arrive and are produced by the complete local model.</p>

<h2 id="activation-vector-steering">Activation Vector Steering</h2>

<p>Since I already had a locally modifiable runtime, I also experimented with porting activation vector steering and refusal-direction ablation to Colibrì, based on a <a href="https://github.com/Sumandora/remove-refusals-with-transformers">very well known implementation</a>.</p>

<p>The basic idea is to collect hidden-state vectors for two prompt classes, for example prompts that trigger refusal behaviour and corresponding harmless prompts. Their mean difference provides a direction in activation space:</p>

[
\mathbf d = \frac{    \langle \mathbf h \rangle_{\mathrm{refusal}} -    \langle \mathbf h \rangle_{\mathrm{harmless}} }{     \left\|     \langle \mathbf h \rangle_{\mathrm{refusal}} -     \langle \mathbf h \rangle_{\mathrm{harmless}}     \right\| }
]

<p>At a selected transformer layer the component of a hidden state along this direction can then be removed:</p>

[
\mathbf h' = \mathbf h - \alpha (\mathbf h \cdot \mathbf d)\mathbf d
]

<p>where $\alpha$ controls the strength of the intervention.</p>

<p>The implementation in my fork can collect activation samples, calculate a normalized direction using a small Python utility and apply the direction in the C inference engine. This is an experimental facility and should not be confused with retraining or fine-tuning the model. It is an intervention in the forward pass.</p>

<h3 id="does-steering-reduce-performance">Does Steering Reduce Performance?</h3>

<p>During ordinary inference the additional computation is tiny. At one selected layer the engine:</p>

<ul>
  <li>Calculates one dot product over the hidden dimension</li>
  <li>Performs one scaled vector subtraction over the same dimension</li>
</ul>

<p>This requires approximately $2D$ multiplications and $2D$ additions per token for hidden width $D$, together with two linear passes over a single activation vector. This is negligible compared with the matrix multiplications involving roughly 40 billion active parameters per token. On a configuration dominated by loading many gigabytes of experts from NVMe, the difference should be below ordinary run-to-run variation. It does add a small amount of memory traffic through the activation vector and steering direction. This might become measurable only on a highly optimized configuration that is already limited by memory bandwidth rather than storage or expert matrix multiplication. It should still be a very small fraction of the complete forward pass.</p>

<p>Activation <em>collection mode</em> has a different performance characteristic. For every collected prompt it opens the sample file and appends a <code class="language-plaintext highlighter-rouge">float32</code> hidden-state vector. It also deliberately runs short controlled generations while collecting the training examples. Collection mode should therefore <em>not be enabled during inference benchmarks</em>.</p>

<p>To verify the expected result, the final benchmark suite should include otherwise identical runs with steering disabled and enabled. The comparison has to use the same prompt, cache state, decoding settings and generated token count. Since token routing may change when activation steering changes the continuation, wall-clock comparisons across freely generated long answers are otherwise easily confounded.</p>

<h2 id="running-colibrì-behind-an-api-gateway">Running Colibrì Behind an API Gateway</h2>

<p>An interactive terminal is useful for testing, but I normally access language models through my own <a href="/2025/10/25/miniapigw.html">OpenAI compatible API gateway</a>, <code class="language-plaintext highlighter-rouge">miniapigw</code>. Placing Colibrì behind the gateway allows existing applications to use the runtime without knowing whether a request is handled by one of the local <a href="https://ollama.com/">Ollama</a> instances, a remote cloud service or the NVMe streamed GLM-5.2 backend.</p>

<p>The basic arrangement is:</p>

<p><img src="/assets/images/png/colibri002.png" alt="Multiple applications including codex can attach via the responses API or chat API to mini-apigw, which provides shims to translate to the colibri backend which then in turn runs glm" /></p>

<p>The gateway can provide a stable model name, authentication, request logging, streaming and backend-specific timeout handling. It also allows applications to switch between a fast smaller model and GLM-5.2 without changing their client implementation.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
  "type" : "openai",
  "name" : "colibriwssz1",
  "base_url" : "http://198.51.100.0:8123/v1",
  "api_key" : "local",
  "concurrency" : 4,
  "supports" : {
    "chat" : [ "glm-5.2-colibri" ],
    "embeddings" : [],
    "responses" : [],
    "images" : []
  },
  "cost" : {
    "currency" : "eur",
    "unit" : "1k_tokens",
    "models" : { }
  },
  "responses_shim" : {
    "enabled" : true,
    "operation" : "chat",
    "unsupported_tools_policy" : "silent_strip",
    "reorder_system_messages_to_front" : false,
    "coalesce_system_messages" : false
  }
},
</code></pre></div></div>

<p>For a backend of this type it is important that the gateway does not interpret a period of computation or disk activity as a failed request. Streaming responses and appropriately long idle and total request timeouts are essential.</p>

<h2 id="using-the-model-with-codex">Using the Model with Codex</h2>

<p>Chatting with a slowly streamed model and using it as the reasoning engine of a coding agent are two rather different workloads.</p>

<p>An agent such as Codex repeatedly combines a potentially large repository context with tool results, generates commands, waits for their execution and continues reasoning. This creates several challenges:</p>

<ul>
  <li>Initial prompt prefill may be <em>much larger</em> than in an ordinary chat</li>
  <li>The context grows after every tool interaction</li>
  <li>A single task may require many separate inference calls</li>
  <li>Client and gateway timeouts may assume GPU-like response times</li>
  <li>Long generated explanations consume time without advancing the task</li>
  <li>Parallel agents can compete for a backend that is most useful when serving one request at a time</li>
</ul>

<p>For slow local operation it is therefore useful to keep the supplied context deliberate, compact conversations before they become unnecessarily large, limit output lengths and configure generous timeouts. Streaming should be used wherever possible so that progress remains visible.</p>

<p><strong>The expected use case is not rapid interactive autocomplete</strong>. It is a patient, long running task (utilizing <code class="language-plaintext highlighter-rouge">/goal</code>) for which the capability and local control of the larger model justify waiting. Whether this is practically useful depends not only on tokens per second but also on how many correct tool iterations are required to complete the task. A slower model that follows the goal co herently may still finish before a faster model that repeatedly has to be corrected.</p>

<h2 id="performance-measurements">Performance Measurements</h2>

<p>My current measurements test combinations of direct I/O, router look-ahead, pipelined loading and different worker counts under a 19 GB runtime RAM budget. A fixed prompt is used to make the runs comparable:</p>

<blockquote>
  <p>Explain why the sky is blue in 3 short paragraphs.</p>
</blockquote>

<p>The benchmark records total wall-clock time and generated output. Since Colibrì does not currently print all desired timing statistics in the tested mode, the external test script measures the complete request and derives comparable throughput values from the produced token count where possible.</p>

<table>
  <thead>
    <tr>
      <th>Configuration</th>
      <th>Direct I/O</th>
      <th>Pilot</th>
      <th>Pipeline workers</th>
      <th>Token/sec</th>
      <th>Generation time (s)</th>
      <th>Wall time (s)</th>
      <th>Cache hit ratio (%)</th>
      <th>RSS[GB]</th>
      <th>Speedup</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">baseline</code></td>
      <td>0</td>
      <td>0</td>
      <td> </td>
      <td>0.0200</td>
      <td>1850.0</td>
      <td>1903.2</td>
      <td>3.0</td>
      <td>13.8</td>
      <td>1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct</code></td>
      <td>1</td>
      <td>0</td>
      <td> </td>
      <td>0.0200</td>
      <td>1895.0</td>
      <td>1947.1</td>
      <td>3.0</td>
      <td>14.2</td>
      <td>1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">pipe2</code></td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0.0200</td>
      <td>1558.0</td>
      <td>1608.9</td>
      <td>3.0</td>
      <td>14.0</td>
      <td>1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct_pipe2</code></td>
      <td>1</td>
      <td>0</td>
      <td>2</td>
      <td>0.0200</td>
      <td>1547.0</td>
      <td>1597.4</td>
      <td>3.0</td>
      <td>14.2</td>
      <td>1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">pilot_pipe2</code></td>
      <td>0</td>
      <td>1</td>
      <td>2</td>
      <td>0.0200</td>
      <td>1548.5</td>
      <td>1597.0</td>
      <td>3.0</td>
      <td>14.1</td>
      <td>1.000x</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">direct_pilot_pipe2</code></td>
      <td>1</td>
      <td>1</td>
      <td>2</td>
      <td>0.0200</td>
      <td>1555.0</td>
      <td>1604.4</td>
      <td>3.0</td>
      <td>13.9</td>
      <td>1.000x</td>
    </tr>
  </tbody>
</table>

<p>All measurements were repeated for 10 times, massive outliers during night time <a href="https://man.freebsd.org/cgi/man.cgi?periodic">periodic(8)</a> runs have been discarded (they deviated by more than $5\sigma$ in the specific configuration group and where directly correlatable to the night time <code class="language-plaintext highlighter-rouge">periodic</code> jobs). Also note that the piping of the output through a Python script also showed slight performance reduction in comparison to direct inference.</p>

<p>As the cache hit ratio was <em>very low</em> in all test cases, it will be particularly interesting to repeat the same test after upgrading the machine from 20 GB to 96 GB. This will show how strongly the larger expert cache affects the hit rate and whether the system transitions from an I/O dominated regime towards a compute or memory-bandwidth dominated regime.</p>

<h2 id="what-this-does-not-do">What This Does Not Do</h2>

<p>Colibrì does not magically turn an old Xeon workstation into a rack of modern GPUs. There are several rather fundamental limitations:</p>

<ul>
  <li>Approximately 400 GB of model storage is still required</li>
  <li>Fast <a href="https://amzn.to/3RFcYHm">NVMe storage</a> matters, maybe even mirror them to counter I/O bottlenecks, depending on the CPUs you are using</li>
  <li>Generation latency is high on old CPUs</li>
  <li>A small RAM cache causes repeated expert loads</li>
  <li>Long prompts require substantial prefill computation</li>
  <li>Agentic clients require adjusted timeout and context handling</li>
  <li>Performance depends on cache history and expert routing</li>
</ul>

<p>Nor does INT4 quantization necessarily reproduce the full precision model exactly. Quantized kernels can alter close logits, speculative decoding may select a different but still valid continuation and the quality cost of the quantized model has to be measured rather than assumed away.</p>

<p>This is therefore not a replacement for proper high performance inference infrastructure when latency, concurrency or throughput matter. It is a way to make a model accessible on machines that could otherwise not load it at all.</p>

<h2 id="conclusion">Conclusion</h2>

<p>A complete 744 billion parameter language model is currently answering questions on my FreeBSD workstation containing 20 GB of old DDR3 ECC memory.</p>

<p>The model is about nineteen times larger than the installed RAM. Its quantized weights occupy nearly 400 GB. It is evaluated by two old six-core Xeon processors and supplied with routed experts from an NVMe SSD. This is not fast, and every generated answer makes the physical limits of storage bandwidth, memory bandwidth and computation quite visible.</p>

<blockquote>
  <p><strong>But it works.</strong></p>
</blockquote>

<p>This is what makes Colibrì so fascinating. It does not pretend that the model has become small. Instead it exploits the sparse structure of a Mixture-of-Experts model and turns an absolute capacity limit into a continuum of performance. More RAM, faster storage and faster processors improve the result, but insufficient RAM no longer closes the door completely.</p>

<p>A few years ago, running a frontier scale model locally meant owning hardware capable of keeping the entire model resident in expensive memory. Now an old FreeBSD workstation can keep the dense parts alive, fetch experts as they are requested and eventually produce the answer.</p>

<p>One has to be patient, but not wait till the heat death of the universe. One may want <a href="https://amzn.to/3RFcYHm">another NVMe SSD</a>. One will almost certainly start thinking about memory upgrades. The response to a simple question taking half an hour for sure means that the solution is not usable for a real-time chat task, but it makes those models actually suitable as the long term goal following planning backend of agentic frameworks, which controls smaller scale agents (like <code class="language-plaintext highlighter-rouge">llama3.5</code> or <code class="language-plaintext highlighter-rouge">qwen3.6</code> in their 35 billion or 70 billion parameter variants that happily run on consumer hardware)</p>

<p>But a 744B model is running in 20 GB of RAM - and this is simply tremendously exciting.</p>

<p>What will hopefully follow up is how the RAM upgrade to 96GB influences the performance - which I expect to be massive due to the low cache hit ratio and the available data on the influence of RAM on the generation speed.</p>

<h2 id="references">References</h2>

<ul>
  <li>The original <a href="https://github.com/JustVugg/colibri">Colibrì runtime</a></li>
  <li>My repository containing the <a href="https://github.com/tspspi/colibri">FreeBSD port and activation ablation experiments</a></li>
  <li>zAIs blog post on <a href="https://z.ai/blog/glm-5.2">GLM-5.2: Built for Long-Horizon Tasks</a></li>
  <li>The accompanying paper about <a href="https://arxiv.org/abs/2112.06905">GLaM: Efficient Scaling of Language Models with Mixture-of-Experts</a></li>
  <li>Hardware one may need:
    <ul>
      <li>A fast <a href="https://amzn.to/3RFcYHm">NVMe SSD</a></li>
      <li>Maybe also an <a href="https://amzn.to/4wh83vu">PCIe to M.2 NVMe adapter</a></li>
    </ul>
  </li>
  <li>My <a href="/2025/10/25/miniapigw.html">OpenAI compatible API gateway</a>, that includes shims to support codex in front of different backends and to handle slow backends better</li>
  <li><a href="https://ollama.com/">ollama</a> as very simple inference engine for other smaller models</li>
  <li><a href="https://huggingface.co/">HuggingFace</a>, the largest repository for open weight models and datasets</li>
  <li>A reference implementation of <a href="https://github.com/Sumandora/remove-refusals-with-transformers">refusal removal with transformers</a></li>
  <li>My own blog articles on
    <ul>
      <li><a href="/2026/07/15/codexopenaigateway.html">Using Codex with a custom API gateway</a></li>
      <li><a href="/2026/07/05/qwencodex.html">using QWen3.6:35b-a3b with codex</a></li>
      <li><a href="/2026/07/12/codexyolojail.html">Running Codex or other applications in lightweight FreeBSD jails</a></li>
    </ul>
  </li>
</ul>

<p><img src="/assets/images/png/colibri003.png" alt="" /></p>]]></content><author><name>tsp</name></author><category term="Programming" /><category term="Artificial Intelligence" /><category term="Tutorial" /><category term="FreeBSD" /><category term="System administration" /><category term="Administration" /><category term="Large Language Models" /><category term="Automation" /><category term="Computational linear algebra" /><category term="Privacy" /><category term="LLM" /><category term="Vibe coding" /><category term="n8n" /><summary type="html"><![CDATA[Modern frontier scale large language models (LLMs) are normally associated with clusters of very expensive GPUs and hundreds of gigabytes of memory. Yet recent advances in Mixture-of-Experts architectures and intelligent inference runtimes challenge this assumption. In this short article we explore how the 744 billion parameter GLM-5.2 model can be executed on an old dual-Xeon FreeBSD workstation equipped with just 20 GB of RAM by streaming expert weights directly from NVMe storage. Beyond documenting the necessary FreeBSD port of the Colibrì runtime, the article explains the underlying memory hierarchy, Mixture-of-Experts routing, activation vector steering and practical performance characteristics of such an unconventional setup. While the resulting system is certainly no replacement for modern GPU servers, it demonstrates that frontier scale language models are becoming accessible on surprisingly modest hardware, provided one is willing to trade latency for capability.]]></summary></entry><entry><title type="html">GPL or BSD? A Personal View on Licensing Hobby and Academic Software</title><link href="https://www.tspi.at/2026/07/16/gplvsbsd.html" rel="alternate" type="text/html" title="GPL or BSD? A Personal View on Licensing Hobby and Academic Software" /><published>2026-07-16T00:00:00+02:00</published><updated>2026-07-16T23:47:17+02:00</updated><id>https://www.tspi.at/2026/07/16/gplvsbsd</id><content type="html" xml:base="https://www.tspi.at/2026/07/16/gplvsbsd.html"><![CDATA[<p>Software licenses are sometimes discussed as if one of them represented <em>freedom</em> while another did not. This is not particularly helpful. Both the <a href="https://de.wikipedia.org/wiki/GNU_General_Public_License">GNU General Public License</a> and the <a href="https://en.wikipedia.org/wiki/BSD_licenses">BSD licenses</a> grant freedoms which ordinary proprietary licenses withhold. They differ mainly in <em>which freedom is protected when the software is passed on</em>.</p>

<p>The GPL attempts to preserve a commons. A recipient may use, inspect, modify and redistribute the program, but a distributed derivative <em>work must remain available under the corresponding GPL conditions</em> (copyleft, virality). A permissive BSD license makes a different decision: the recipient may also incorporate the code into a closed product, provided that the rather small set of conditions in the license is respected (<em>“do whatever you want, just cite it”</em>).</p>

<p>My own <em>personal</em> preference, especially for hobby projects and research software, is usually the BSD approach. This is not based on the belief that the GPL has failed. Quite the opposite: it has produced some very visible successes. It is based on a different answer to the question of what it means to give software away <em>“for free”</em>.</p>

<blockquote class="disclaimer">
  <p>⚠️ <strong>Disclaimer</strong>: I am not a lawyer, and this article does not constitute legal advice. It presents a technical and personal interpretation of common software-licensing principles. For concrete licensing decisions or disputes, qualified legal advice should be obtained.</p>
</blockquote>

<ul>
  <li><a href="#two-different-meanings-of-freedom">Two different meanings of freedom</a></li>
  <li><a href="#what-the-gpl-has-demonstrably-achieved">What the GPL has demonstrably achieved</a></li>
  <li><a href="#when-copyleft-prevents-otherwise-useful-combinations">When copyleft prevents otherwise useful combinations</a></li>
  <li><a href="#why-a-permissive-license-often-fits-hobby-software">Why a permissive license often fits hobby software</a></li>
  <li><a href="#academic-software-has-an-additional-purpose">Academic software has an additional purpose</a></li>
  <li><a href="#the-advertising-clause-confusion">The advertising-clause confusion</a>
    <ul>
      <li><a href="#why-gpl-projects-ask-for-the-advertising-clause-to-be-removed">Why GPL projects ask for the advertising clause to be removed</a></li>
    </ul>
  </li>
  <li><a href="#do-what-you-want-but-cite-correctly">“Do what you want, but cite correctly”</a></li>
  <li><a href="#some-qualifications">Some qualifications</a></li>
  <li><a href="#a-preference-not-a-universal-rule">A preference, not a universal rule</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/bsdgpl001.png" alt="" /></p>

<h2 id="two-different-meanings-of-freedom">Two different meanings of freedom</h2>

<p>The word <em>freedom</em> hides the central disagreement.</p>

<p>Under a permissive license, freedom is primarily granted to the next recipient. The code may be studied, modified, redistributed, sold, embedded into a device or combined with a proprietary program. Few demands are made about what the recipient must do with a larger work constructed around it.</p>

<p>Under the GPL, freedom is treated more collectively and over time. The next recipient is given extensive rights, but is not permitted to distribute a covered derivative under terms which deny the same essential rights to later recipients. The restriction imposed on one distributor is intended to preserve freedom for everyone farther down the chain.</p>

<p>Consequently, it is possible to describe either license family as the freer one, depending on what is being measured:</p>

<table>
  <thead>
    <tr>
      <th>Question</th>
      <th>GPL</th>
      <th>BSD-2/BSD-3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>May the software be used privately or commercially?</td>
      <td>Yes</td>
      <td>Yes</td>
    </tr>
    <tr>
      <td>May the source be modified privately?</td>
      <td>Yes; private changes need not generally be published</td>
      <td>Yes</td>
    </tr>
    <tr>
      <td>May modified versions be redistributed?</td>
      <td>Yes, subject to the GPL’s copyleft conditions</td>
      <td>Yes, with notices and disclaimers retained as required</td>
    </tr>
    <tr>
      <td>May the code become part of a proprietary derivative?</td>
      <td>Generally not when the combined work is covered by the GPL</td>
      <td>Yes</td>
    </tr>
    <tr>
      <td>Is source availability preserved for downstream recipients of covered distributions?</td>
      <td>Yes</td>
      <td>No</td>
    </tr>
  </tbody>
</table>

<p>Calling GPL software <em>“restricting freedom”</em> would therefore be inaccurate. Calling the GPL viral is also rather imprecise, even though the term captures the practical concern many developers have. The GPL does not somehow spread merely because GPL software is installed beside another program, used as a development tool or communicates with independent software in an ordinary way. The difficult questions concern whether code has been copied, adapted, linked or combined into a single derivative work and whether that work is being conveyed to someone else. The exact boundary depends on the technical arrangement, the applicable GPL version and ultimately copyright law.</p>

<p>Nevertheless, <a href="https://de.wikipedia.org/wiki/Copyleft">copyleft</a> is deliberately reciprocal. That reciprocity is exactly what supporters value and exactly what others may not want to impose.</p>

<h2 id="what-the-gpl-has-demonstrably-achieved">What the GPL has demonstrably achieved</h2>

<p>It would be difficult to discuss this subject without acknowledging what the GPL has accomplished.</p>

<p>The <a href="https://de.wikipedia.org/wiki/Linksys_WRT54G">Linksys WRT54G</a> is the conventional and still useful example. Its firmware contained GPL-licensed software (Linux). Once the corresponding licensing obligations were enforced, source code became available and helped enable an unusually productive ecosystem of alternative firmware. The <code class="language-plaintext highlighter-rouge">WRT54G</code> and later <code class="language-plaintext highlighter-rouge">WRT54GL</code> became useful far beyond the intentions normally associated with a consumer router. <a href="https://openwrt.org/">OpenWrt</a>, <a href="https://dd-wrt.com/">DD-WRT</a>, <a href="https://freshtomato.org/index.html">Tomato</a> and many experimental installations benefited from the opportunity.</p>

<p>This was not an accidental side effect of an open-source license. It was exactly the type of result for which <em>copyleft</em> had been designed. Had the relevant components been available only under a permissive license, publication of the vendor’s changes could not have been demanded merely on the basis of that license.</p>

<p>Similar reasoning applies whenever there is a realistic danger that improvements to a shared foundation will disappear into appliances or proprietary distributions. A compiler, operating-system component, infrastructure platform or other work intended to remain a public commons can be very well served by the GPL. The license is not merely an expression of distrust toward companies, it is a mechanism which prevents any distributor from accepting the common source while withholding covered improvements from its recipients.</p>

<p><strong>For such a goal, the GPL is both coherent and effective.</strong></p>

<h2 id="when-copyleft-prevents-otherwise-useful-combinations">When copyleft prevents otherwise useful combinations</h2>

<p>The same mechanism can also obstruct integration even when all of the software involved is open source. <a href="https://en.wikipedia.org/wiki/ZFS">ZFS</a> provides a particularly instructive example.</p>

<p>ZFS originated at Sun and was released as part of <a href="https://docs.oracle.com/cd/E19253-01/817-6295/gkcec/index.html">OpenSolaris</a> under the <a href="https://en.wikipedia.org/wiki/Common_Development_and_Distribution_License">Common Development and Distribution License (CDDL)</a>. The <code class="language-plaintext highlighter-rouge">CDDL</code> is itself an open-source license and a form of file-level copyleft. It requires covered source files and modifications to them to remain available under the <code class="language-plaintext highlighter-rouge">CDDL</code>, while permitting them to be used as part of a larger work under different terms.</p>

<p>The Linux kernel, by contrast, is distributed under GPL version 2. The prevailing interpretation is that CDDL-covered ZFS code and the GPLv2-covered Linux kernel <em>cannot simply be distributed as one combined kernel work</em>: the CDDL does not allow its covered code to be relicensed under GPLv2, while GPLv2 does not permit a covered combined work to retain additional incompatible licensing conditions. Neither license supplies a set of terms which satisfies the other.</p>

<p>This is sometimes shortened to the statement that <em>“the GPL would require ZFS to be relicensed.”</em> That conveys the practical obstacle, but it is not quite the legal mechanism. The GPL does not acquire the copyright in ZFS or automatically change its license. Rather, if integration produces a work governed as a whole by GPLv2, distribution would require permissions which the existing CDDL license does not provide. Relicensing the ZFS code under GPL-compatible terms could resolve the conflict, but only the relevant copyright holders could grant that permission. In a mature project with code from many contributors, this is not a trivial option.</p>

<p><a href="https://www.freebsd.org">FreeBSD</a> could take a different path. Although the FreeBSD base system is not literally composed only of BSD-licensed files, its predominantly permissive licensing does not demand that an integrated kernel component be relicensed under BSD terms. FreeBSDs licensing policy explicitly permits CDDL-licensed ZFS kernel and userland code. ZFS could therefore become a deeply integrated part of FreeBSD while remaining under the CDDL.</p>

<p>ZFS is available on Linux today through <a href="https://openzfs.org/wiki/Main_Page">OpenZFS</a>, normally as a separately maintained, out-of-tree kernel module rather than code merged into the mainline Linux kernel. Whether particular ways of combining and distributing such a module with Linux are lawful has been debated for years and different distributors have reached different practical conclusions. This makes it unsuitable as a simple claim that <em>“ZFS cannot run on Linux.”</em> What the licensing conflict prevented was the straightforward inclusion and distribution of the CDDL implementation as an ordinary part of the GPLv2 kernel source tree.</p>

<p>This example is useful because no proprietary vendor needs to be introduced. Both sides consist of open-source software, both licenses were intended to protect openness and yet the reciprocal conditions do not compose. The result is duplicated integration work, an out-of-tree compatibility layer and continuing uncertainty for distributors.</p>

<p>It would also be unfair to attribute the entire conflict to one side. The CDDL is not a permissive BSD license; it deliberately keeps covered files under the CDDL. The incompatibility arises from the interaction of two <em>copyleft</em> regimes. Nevertheless, the GPLs whole-work reciprocity makes it considerably harder for a GPL-covered kernel to absorb independently developed code under another open-source license. A permissively licensed host such as FreeBSD can accept such a component without demanding that the component adopt the hosts license.</p>

<p>This is the corresponding cost of the protection illustrated by the <code class="language-plaintext highlighter-rouge">WRT54G</code>. Copyleft can force improvements to remain available when compatible code is distributed as part of a covered work. It can also prevent useful open-source components from being combined when their authors selected different, mutually incompatible conditions. Whether the protection is worth that cost depends on the purpose of the project.</p>

<h2 id="why-a-permissive-license-often-fits-hobby-software">Why a permissive license often fits hobby software</h2>

<p>A hobby project is frequently published for a simpler reason: something useful was created, and there is no wish to prevent someone else from using it.</p>

<p>In that setting, a reciprocal obligation may exceed the author’s actual intention. If a small numerical routine, device driver, parser, visualization component or microcontroller library is useful inside somebody elses larger project, it may not matter whether that project is open or closed. Use by a small company, inclusion in a commercial instrument or adaptation for an unexpected platform may be welcomed rather than treated as a loss.</p>

<p>The <code class="language-plaintext highlighter-rouge">BSD</code> licenses express this position with unusual clarity. Permission is broad. The copyright notice, license conditions and disclaimer must be preserved in the ways specified by the license. BSD-3-Clause additionally prevents the names of the copyright holder and contributors from being used to endorse derived products without permission. Beyond this, the recipient is largely left alone.</p>

<p>This also reduces license-integration problems. Permissively licensed code can be accepted by GPL projects and, in most cases, by proprietary projects. GPL code cannot simply be incorporated into every project whose distribution model is incompatible with copyleft. For a small library whose purpose is maximum adoption, the asymmetry matters.</p>

<p>There is, of course, a downside. A company may take the code, improve it and distribute the result without publishing those improvements. The original author may never receive a patch. This is not an unforeseen loophole in the BSD model; it is the permission which the license intentionally grants.</p>

<p>For my own hobby software, that possibility is usually acceptable. Once the code has been released, another person’s use of it does not remove the original public version. A proprietary derivative may be disappointing, but it does not retroactively make the published code less useful. If preventing closed derivatives is an essential project goal, the GPL should be selected instead. If it is not, imposing that goal merely as a precaution appears unnecessary.</p>

<h2 id="academic-software-has-an-additional-purpose">Academic software has an additional purpose</h2>

<p>Research software occupies a somewhat different position. It is not only a program but often also part of the method by which a scientific result was obtained. Several goals then become relevant at once:</p>

<ul>
  <li>results should be reproducible</li>
  <li>methods should be inspectable</li>
  <li>other groups should be able to adapt the implementation</li>
  <li>the origin of ideas and implementations should remain visible</li>
  <li>researchers should receive appropriate scholarly credit</li>
  <li>transfer into teaching, instrumentation and industry should not be obstructed without a reason</li>
</ul>

<p>A permissive license is a very natural default for these goals. It permits replication and modification without making the licensing of an entire surrounding research system dependent on the chosen component. This matters because academic code is often combined with vendor libraries, laboratory-control software, institutional frameworks and pre-existing tools under many different terms. A small permissively licensed module can travel through this environment more easily than a strongly copyleft component.</p>

<p>It can also move into practice. If an algorithm developed at a university becomes useful in a microscope, a medical-analysis pipeline or an industrial controller, adoption should not necessarily depend on whether the complete product can be released under the GPL. There are cases in which reciprocity is more important than adoption, but this should be an intentional decision rather than an automatic one. This also reflects the idea that products of Universities should be available to the general public since the universities are usually publicly funded.</p>

<p>The BSD model also resembles an important academic convention: work may be taken further, including for purposes the original author did not anticipate, while its origin should not be erased.</p>

<h2 id="the-advertising-clause-confusion">The advertising-clause confusion</h2>

<p>The commonly used BSD variants are easy to confuse:</p>

<ul>
  <li><strong>BSD-2-Clause</strong> requires preservation of the copyright notice, license conditions and disclaimer in source redistributions and their reproduction in documentation or other materials accompanying binary redistributions.</li>
  <li><strong>BSD-3-Clause</strong> adds the non-endorsement condition: the names of the copyright holder and contributors may not be used to promote a derived product without written permission.</li>
  <li><strong>BSD-4-Clause</strong>, historically called the original BSD license, additionally contained the advertising clause. Advertising materials mentioning features or use of the software had to display an acknowledgement.</li>
</ul>

<p>Thus, the license with the advertising clause is the <em>four-clause</em> BSD license, although the advertising requirement itself was numbered clause 3. This is probably the source of much of the confusion. UC Berkeley rescinded that clause for its BSD-licensed code, and the revised three-clause license became the familiar form.</p>

<h3 id="why-gpl-projects-ask-for-the-advertising-clause-to-be-removed">Why GPL projects ask for the advertising clause to be removed</h3>

<p>Removal of the advertising clause was not requested only because acknowledgements became inconvenient. It also made BSD-licensed code substantially easier to combine with GPL-covered software.</p>

<p>The important concept is <em>license compatibility</em>. Two programs may each be free and open-source software while their licenses still make it impossible to distribute a derivative work containing both. Every condition of both licenses must be satisfiable at the same time. If one license requires a condition which the other forbids distributors from imposing, broad permission in each license considered separately does not solve the conflict.</p>

<p>GPLv2 illustrates this directly. Section 6 says that a distributor may not impose further restrictions on a recipient’s exercise of the rights granted by the GPL. A combined derivative distributed under GPLv2 must therefore reach its recipients under the GPL’s conditions without an additional obligation inherited from some incorporated component.</p>

<p>BSD-4-Clause, however, requires advertising materials which mention the software’s features or use to display a specified acknowledgement. This obligation is not present in GPLv2. If BSD-4-Clause code were incorporated into a GPLv2-covered program, a distributor would face incompatible instructions:</p>

<ul>
  <li>BSD-4-Clause would require the additional advertising acknowledgement;</li>
  <li>GPLv2 would prohibit imposing that additional restriction on downstream redistribution.</li>
</ul>

<p>Dropping the acknowledgement would violate the BSD license. Retaining it as a mandatory condition would conflict with the GPL. Consequently, the original four-clause BSD license is generally treated as GPL-incompatible even though it is a free-software license.</p>

<p>This explains why developers wishing to reuse such code in GPL projects have often asked copyright holders to remove the advertising clause or grant an appropriate exception. A downstream developer cannot simply delete it from an existing license notice: only the holder of the relevant copyright can offer that code under altered terms. UC Berkeleys rescission resolved the issue for the code to which Berkeley held the applicable rights, but it did not automatically rewrite similar four-clause licenses adopted by unrelated authors.</p>

<p>BSD-3-Clause avoids this conflict. Its ordinary notice-preservation requirements and non-endorsement clause are regarded as compatible with the GPL. Code under BSD-3-Clause can therefore normally be incorporated into a GPL-covered derivative and the resulting combined work distributed under the applicable GPL terms, while the original BSD copyright and license notices remain preserved for the BSD-derived portions.</p>

<p>This is sometimes described loosely as the BSD code <em>becoming GPL</em>. More precisely, the original BSD-licensed code remains available from its copyright holder under BSD terms. The distributor applies the GPL to the combined derivative work and must also preserve the BSD notices associated with the incorporated material. Other people may still obtain the original component under BSD-3-Clause and use it in non-GPL projects. Copyleft governs the distributed combination; it does not retroactively cancel the permissive license already granted by the original author.</p>

<p>GPLv3 contains a more elaborate treatment of certain permitted additional terms, including some forms of attribution and legal-notice preservation. Compatibility analysis can therefore differ between GPL versions and between exact license wordings. The conventional and safest general statement remains that BSD-2-Clause and BSD-3-Clause are GPL-compatible, whereas the original BSD-4-Clause advertising requirement is treated as GPL-incompatible. This is another reason not to revive the historical clause merely to encourage academic citation.</p>

<p>At first sight, the old requirement appears attractive for academic software: do what you want, but acknowledge where it came from. I support that idea.</p>

<h2 id="do-what-you-want-but-cite-correctly">“Do what you want, but cite correctly”</h2>

<p>The desired arrangement can be expressed more effectively as a combination rather than as an unusual software license:</p>

<ul>
  <li>Release the source under BSD-3-Clause.</li>
  <li>Preserve a clear copyright statement and identify the contributors.</li>
  <li>Add a <code class="language-plaintext highlighter-rouge">CITATION.cff</code> file containing the preferred software citation if desired.</li>
  <li>Archive significant releases in a repository which issues a persistent identifier, such as a DOI (<a href="https://zenodo.org/">Zenodo</a> for example).</li>
  <li>If there is a paper describing the method, state clearly when that paper is the appropriate scholarly reference.</li>
  <li>Keep the citation request separate from the license unless there is a specific legal reason to do otherwise.</li>
</ul>

<p>This says, in effect: the legal permission to use the software does not depend on performing an academic ritual, but ordinary standards of research integrity still apply. A person who redistributes the code must comply with the BSD license. A person who relies on the software scientifically should cite it because the contribution is relevant to the work, just as methods, datasets and prior results should be cited.</p>

<p>This distinction may initially look weaker because the citation request is not converted into an additional copyright condition. In practice, it is also more accurate. Citations belong in papers because they explain provenance and give credit, not because an advertising clause happened to be triggered. When a use is too trivial or remote to warrant scholarly citation, the license notice may still need to be retained in a redistribution. When a scientific result depends critically on the software without redistributing it, citation may be appropriate even though the BSD conditions were never triggered. The two systems cover different relations.</p>

<h2 id="some-qualifications">Some qualifications</h2>

<p>No short license comparison can settle every concrete case. In particular, <em>GPL</em> is a family of licenses and versions and linking or combination questions can become legally and technically complicated. The <em>Affero GPL</em> adds network-use provisions which ordinary GPL versions do not contain. Merely operating a modified GPL program as a service does not generally create the same source-provision obligation as distributing it; this is one reason the <em>AGPL</em> exists.</p>

<p>The BSD licenses also do not solve every problem. Their short texts contain no express patent grant comparable to that in Apache License 2.0. For projects in patent-sensitive fields, Apache-2.0 may therefore be a better permissive choice. University ownership policies, employment agreements, third-party dependencies, export controls, personal data and ethical restrictions can impose additional constraints which cannot be removed by selecting a license template.</p>

<p>Finally, an author can license only rights which the author actually controls. Code copied or derived from other projects remains subject to their licenses.</p>

<p>These are reasons to inspect the concrete project, not reasons to leave it without a license. <em>_Without a license, publication of source code does not ordinarily give everybody a general right to modify and redistribute it. “Available on the Internet” and “open source” are not equivalent.</em></p>

<h2 id="a-preference-not-a-universal-rule">A preference, not a universal rule</h2>

<p>For most hobby applications and much academic software, I prefer BSD-3-Clause. It grants the broadest practical scope to whoever receives the work, creates few compatibility obstacles and leaves later applications - open, commercial or otherwise - to the people building them. Together with explicit citation metadata, it also fits the academic principle that work may be reused freely while its origin remains properly documented.</p>

<p>This preference does not require the GPL to be portrayed as hostile or obsolete. The GPL protects a different and legitimate objective: a body of software which cannot be made proprietary by its distributors. The history of the WRT54G demonstrates why that mechanism can matter. If preserving an open commons is the primary purpose of a project, copyleft may be the best fitting license choice.</p>

<p>If the primary purpose is instead to place a tool into the world with as few restrictions as reasonably possible, a BSD license is closer to the intended gift.</p>

<p>The relevant question is therefore not simply which license is <em>freer</em>. It is <em>which freedom</em> the author intends to preserve: the recipients freedom to do almost anything with the code, or the later recipients freedom to receive covered derivatives under the same open terms.</p>

<h2 id="references">References</h2>

<ul>
  <li><a href="https://opensource.org/license/bsd-3-clause">The 3-Clause BSD License</a></li>
  <li><a href="https://www.gnu.org/licenses/gpl-2.0.html">GNU General Public License, version 2</a></li>
  <li><a href="https://www.gnu.org/licenses/gpl-3.0.html">GNU General Public License, version 3</a></li>
  <li><a href="https://www.gnu.org/licenses/license-list.html">Various Licenses and Comments about Them</a></li>
  <li><a href="https://ipira.berkeley.edu/sites/default/files/amendment_of_4-clause_bsd_software_license.pdf">Amendment of the 4-Clause BSD Software License</a></li>
  <li><a href="https://opensource.org/license/cddl-1-1">Common Development and Distribution License 1.1</a></li>
  <li><a href="https://docs.freebsd.org/en/articles/license-guide/">FreeBSD Licensing Policy</a></li>
  <li><a href="https://openzfs.org/wiki/Main_Page">OpenZFS</a></li>
  <li>Andrew Morin, <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC3406002/">A Quick Guide to Software Licensing for the Scientist-Programmer</a></li>
  <li>University of Cambridge Open Research: <a href="https://www.openresearch.cam.ac.uk/copyright-licensing-citation/software-code">Licensing software and code</a></li>
</ul>]]></content><author><name>tsp</name></author><summary type="html"><![CDATA[One of the oldest debates in open-source software asks a deceptively simple question: should code remain free forever through copyleft or should recipients be free to use it however they wish? The GNU GPL and the BSD licenses embody two different answers, each protecting a different notion of freedom. This article explores both philosophies, illustrating where each approach excels and where practical trade-offs emerge. Using well-known examples such as the Linksys WRT54G and the ZFS licensing situation, the short article examines how licensing choices affect hobby projects, academic research and generic software engineering. Rather than arguing that one license is universally superior, it explains the goals each was designed to achieve and why for many hobby and research projects a permissive BSD license may be the authors preferred choice]]></summary></entry><entry><title type="html">Using Codex with a custom API gateway</title><link href="https://www.tspi.at/2026/07/15/codexopenaigateway.html" rel="alternate" type="text/html" title="Using Codex with a custom API gateway" /><published>2026-07-15T00:00:00+02:00</published><updated>2026-07-15T00:07:09+02:00</updated><id>https://www.tspi.at/2026/07/15/codexopenaigateway</id><content type="html" xml:base="https://www.tspi.at/2026/07/15/codexopenaigateway.html"><![CDATA[<p>So we all know that OpenAIs <a href="https://openai.com/en/codex/">codex</a> is, at the time of writing this short note, one of the best coding agents that are currently available. But then one often does not want to use it against ones <a href="https://chat.openai.com">ChatGPT</a> subscription due to limited credits - and maybe also not against the <a href="https://platform.openai.com/">platform API from OpenAI</a> for various reasons:</p>

<ul>
  <li>One may want to like to exploit agentic behaviour of local running models and just play with them.</li>
  <li>One may need to guarantee privacy - not sending data to an external cloud service and trace privacy for performance and capabilities.</li>
  <li>One may try to use local hardware to avoid rate limits or exploding bills.</li>
  <li>One may have a local LLM service, like for example at <a href="https://www.tuwien.at/">TU Wien</a>, that one may want to use for all of the reasons above.</li>
</ul>

<p>Note that for consumer hardware it is <em>very</em> unlikely that you will achieve a result that you are really satisfied with in the same way as with the frontier cloud hosted models like <a href="https://openai.com/de-DE/index/introducing-gpt-5-5/">GPT 5.5</a> or <a href="https://openai.com/en/index/gpt-5-6/">GPT 5.6</a> because today’s strongest coding agents rely not only on model quality but also on very large context windows, strong reasoning capabilities and reliable tool use. When your institution operates the necessary hardware and provides for example <a href="https://huggingface.co/zai-org/GLM-5.2">GLM 5.2</a> you may be satisfied with the outcome even for complex tasks though.</p>

<ul>
  <li><a href="#arbitration-and-gateway">Arbitration and Gateway</a></li>
  <li><a href="#executing-codex">Executing Codex</a>
    <ul>
      <li><a href="#profile-configuration">Profile Configuration</a></li>
      <li><a href="#model-metadata">Model Metadata</a></li>
      <li><a href="#launching-codex">Launching Codex</a></li>
    </ul>
  </li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/qwencodex001.png" alt="" /></p>

<h2 id="arbitration-and-gateway">Arbitration and Gateway</h2>

<p>For pure local inference using the <a href="https://ollama.com/">ollama</a> runtime one may directly use the <code class="language-plaintext highlighter-rouge">--oss</code> feature of <code class="language-plaintext highlighter-rouge">codex</code>, as has also been explained <a href="/2026/07/05/qwencodex.html">previously</a>. But especially when you are operating with services distributed over the whole Internet or want to use different frontends for your self-hosted LLMs there is need for arbitration</p>

<p>Previously <a href="/2025/10/25/miniapigw.html">I have introduced my mini-apigw</a>. There are also solutions like the cloud service <a href="https://openrouter.ai/">OpenRouter</a> or the gateway <a href="https://github.com/TU-Wien-dataLAB/aqueduct">Aqueduct</a> that has been developed at TU Wien on top of <a href="https://docs.litellm.ai/docs/routing">LiteLLM Router</a>. The problem with many backends is - they do not support the same feature set, especially over the new and (still) rapidly evolving <a href="https://developers.openai.com/api/reference/responses/overview">Responses API</a>. And since <code class="language-plaintext highlighter-rouge">codex</code> is a product by OpenAI it uses nearly every feature from responses. This lead to the client not being able to perform its operation against services as our locally hosted GLM 5.2 - or against my older <code class="language-plaintext highlighter-rouge">ollama</code> deployments.</p>

<p><img src="/assets/images/png/miniapigw002.png" alt="Structure of my mini-apigw" /></p>

<p>To resolve that problem, especially since I am also playing with more experimental backends as running <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> or <a href="https://github.com/JustVugg/colibri">colibri</a>, I decided to resolve the problem once and for all: <a href="https://github.com/tspspi/mini-apigw">mini-apigw</a> now supports a <code class="language-plaintext highlighter-rouge">responses</code> shim that speaks the same language as advanced tools as codex while supporting <em>any</em> legacy backend via the <a href="https://developers.openai.com/api/reference/chat-completions/overview">chat completions API</a> or the <code class="language-plaintext highlighter-rouge">ollama</code> client. This is a typical case where a simple API gateway helps to resolve compatibility issues - hopefully on the long run. The idea is to never change the application facing API even when the backend gets exchanged or updated - with all features like scalability and accounting being there <em>“for free”</em>. Especially due to the rapidly evolving nature of the <code class="language-plaintext highlighter-rouge">Responses</code> API the shim that provides compatibility needs regular updates - but putting this into a central place like a gateway means not having to keep track of such changes for each and every application individually.</p>

<p>In addition the application of an API gateway allows to achieve <em>vendor independence</em>. Even in case one is using cloud services one can easily swap different backends by different vendors without a single change on the application facing API or configuration. Another gain from using a gateway is separation of concerns with respect to API tokens (though it also introduces a central spot where tokens for different backends can be stolen) - every application gets an individual API token, the gateway itself is responsible to authenticate against a backend. Accounting and monitoring allows one to inspect the requests and responses and keep track of usage of the different backends.</p>

<h2 id="executing-codex">Executing Codex</h2>

<h3 id="profile-configuration">Profile Configuration</h3>

<p>Now the interesting part - how do we run <code class="language-plaintext highlighter-rouge">codex</code> with a custom backend? Usually one needs to create a new profile, which is done by creating a <code class="language-plaintext highlighter-rouge">{PROFILENAME}.config.toml</code> file in the local <code class="language-plaintext highlighter-rouge">~/.codex</code> directory.</p>

<p>Here is an example that may be called <code class="language-plaintext highlighter-rouge">example.config.toml</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>model_provider = "exampleprovider"
model_context_window = 256144
model = "glm-5.2-744b-preview"
model_reasoning_effort = "medium"
model_catalog_json = "/usr/home/exampleuser/.codex/example-models.json"

[features]
standalone_web_search = false
imagegenext = false
terminal_resize_reflow = false

[model_providers.exampleprovider]
name = "Example gateway"
base_url = "https://example.com/v1"
wire_api = "responses"
env_key = "EXAMPLE_API_KEY"
timeout = 600
requires_openai_auth = false
</code></pre></div></div>

<p>As one can see the file has multiple sections. In the generic section one selects the model provider and can set model properties that can also be overridden later on. The <code class="language-plaintext highlighter-rouge">model_catalog_json</code> though is important. It contains the base system prompt as well as information about context size and compactification for the given models. I personally think its a good idea to create a single JSON file for all models available on ones own gateway.</p>

<p>The <code class="language-plaintext highlighter-rouge">[features]</code> section behaves like the <code class="language-plaintext highlighter-rouge">config.toml</code> of <code class="language-plaintext highlighter-rouge">codex</code> itself.</p>

<p>Then a <code class="language-plaintext highlighter-rouge">provider</code> is specified. This has to match the name of the <code class="language-plaintext highlighter-rouge">model_provider</code>. Here one defines a name and a base URI. The <code class="language-plaintext highlighter-rouge">wire_api</code> always has to be <code class="language-plaintext highlighter-rouge">responses</code>, this field exists for legacy reasons. And since one does not want to leak the OpenAI authentication token one can use <code class="language-plaintext highlighter-rouge">requires_openai_auth=false</code>. When supplying the <code class="language-plaintext highlighter-rouge">env_key</code> a bearer token is added to each requests <code class="language-plaintext highlighter-rouge">Authentication</code> header. The token itself is fetched from an environment variable with the name specified in <code class="language-plaintext highlighter-rouge">env_key</code></p>

<h3 id="model-metadata">Model Metadata</h3>

<p>The second important file is the <code class="language-plaintext highlighter-rouge">model_catalog_json</code>. This contains a JSON list of all known models.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
  "models" : [
    {
      "slug": "glm-5.2-744b-preview",
      "display_name": "GLM 5.2 744B Preview",
      "description": "GLM 5.2 744B Preview via Example Gateway",
      "provider": "exampleprovider",
      "visibility": "list",
      "supported_in_api": true,
      "priority": 100,

      "default_reasoning_level": "medium",
      "supported_reasoning_levels": [
        {
          "effort": "low",
          "description": "Fast reasoning"
        },
        {
          "effort": "medium",
          "description": "Balanced reasoning"
        },
        {
          "effort": "high",
          "description": "Deeper reasoning"
        }
      ],
      "supports_reasoning_summaries": false,
      "default_reasoning_summary": "none",
      "support_verbosity": true,
      "default_verbosity": "low",

      "shell_type": "shell_command",
      "apply_patch_tool_type": "freeform",
      "web_search_tool_type": "text_and_image",
      "supports_parallel_tool_calls": false,
      "supports_image_detail_original": false,

      "context_window": 262144,
      "max_context_window": 262144,
      "effective_context_window_percent": 75,

      "truncation_policy": {
        "mode": "tokens",
        "limit": 20000
      },

      "experimental_supported_tools": [],
      "input_modalities": ["text"],
      "supports_search_tool": false,

      "base_instructions": "You are Codex, a coding agent. Work in short, precise steps. Use shell and patch tools carefully. Do not repeat yourself. When tool calls are needed, emit valid tool calls only. If you are stuck, summarize the blocker and stop instead of looping."
    },
    {
      "slug": "glm-4.7-flash:q4_K_M",
      "display_name": "GLM 4.7 Flash",
      "description": "GLM 4.7 Flash via Example gateway",
      "provider": "example",
      "visibility": "list",
      "supported_in_api": true,
      "priority": 100,

      "default_reasoning_level": "medium",
      "supported_reasoning_levels": [
        {
          "effort": "low",
          "description": "Fast reasoning"
        },
        {
          "effort": "medium",
          "description": "Balanced reasoning"
        },
        {
          "effort": "high",
          "description": "Deeper reasoning"
        }
      ],

      "supports_reasoning_summaries": false,
      "default_reasoning_summary": "none",
      "support_verbosity": true,
      "default_verbosity": "low",

      "shell_type": "shell_command",
      "apply_patch_tool_type": "freeform",
      "web_search_tool_type": "text_and_image",
      "supports_parallel_tool_calls": false,
      "supports_image_detail_original": false,

      "context_window": 262144,
      "max_context_window": 262144,
      "effective_context_window_percent": 75,

      "truncation_policy": {
        "mode": "tokens",
        "limit": 20000
      },

      "experimental_supported_tools": [],
      "input_modalities": ["text"],
      "supports_search_tool": false,

      "base_instructions": "You are Codex, a coding agent. Work in short, precise steps. Use shell and patch tools carefully. Do not repeat yourself. When tool calls are needed, emit valid tool calls only. If you are stuck, summarize the blocker and stop instead of looping."
    }
  ]
}
</code></pre></div></div>

<p>The models in the list are addressed via the <code class="language-plaintext highlighter-rouge">slug</code>, which is also displayed in the TUI of <code class="language-plaintext highlighter-rouge">codex</code>. The provider has to be specified in the <code class="language-plaintext highlighter-rouge">example.config.toml</code> that we created earlier on. In addition only supported reasoning levels should be listed, this is model dependent. As one can see I also disabled <code class="language-plaintext highlighter-rouge">reasoning_summaries</code> due to buggy implementations on some of my backends. The <code class="language-plaintext highlighter-rouge">shell_type</code> and <code class="language-plaintext highlighter-rouge">apply_patch_tool_type</code> depend on the training and expectations of the model one is using. I also disabled parallel tool calls and other features that are not widely supported over all APIs to reduce friction in the compatibility layer. In the end I also disabled the <em>search tool</em>, which would have to be implemented in the gateway (maybe this will happen in future). The <code class="language-plaintext highlighter-rouge">base_instructions</code> are then used to specify the system prompt to which the complex prompts by <code class="language-plaintext highlighter-rouge">codex</code> get appended.</p>

<p>Note that the parameter <code class="language-plaintext highlighter-rouge">effective_context_window_percent</code> is not a model parameter - but a hint to codex how large of the context window one should utilize. Whenever the context
goes above the specified threshold, compactification should happen - it is very undesirable to fill up the context window of a model completely due to weird behaviour happening for such edge cases. In addition longer running loops may exhaust an already nearly full context window.</p>

<p>If you need ideas on how to phrase the <code class="language-plaintext highlighter-rouge">base_instructions</code> you may take a look at <code class="language-plaintext highlighter-rouge">~/.codex/models_cache.json</code>. The one presented in this article is just a placeholder.</p>

<h3 id="launching-codex">Launching Codex</h3>

<p>With the files in place one can launch <code class="language-plaintext highlighter-rouge">codex</code> - for this example they are:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">~/.codex/example.config.toml</code></li>
  <li><code class="language-plaintext highlighter-rouge">~/.codex/example-models.json</code></li>
</ul>

<p>To launch codex with the specified OpenAI compatible API backend:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>env EXAMPLE_API_KEY=sk-example codex -p example -m glm-4.7-flash:q4_K_M
</code></pre></div></div>

<p>That’s it. If one is lucky codex works - though I experienced that especially for smaller models longer running <code class="language-plaintext highlighter-rouge">/goal</code> loops yield corrupted context.</p>

<h2 id="conclusion">Conclusion</h2>

<p>With this setup Codex becomes independent of any particular inference backend. Existing applications continue to use the same <code class="language-plaintext highlighter-rouge">Responses API</code> while the gateway translates requests to whatever local or remote model is available. This also makes experimenting with new inference engines considerably easier.</p>

<h2 id="references">References</h2>

<ul>
  <li>Commercial Products
    <ul>
      <li>OpenAIs <a href="https://openai.com/en/codex/">codex</a></li>
      <li>OpenAIs <a href="https://chat.openai.com">ChatGPT</a></li>
      <li>OpenAIs <a href="https://platform.openai.com/">platform API</a></li>
    </ul>
  </li>
  <li>Frontier Models (Commercial):
    <ul>
      <li>OpenAIs <a href="https://openai.com/de-DE/index/introducing-gpt-5-5/">GPT 5.5</a></li>
      <li>OpenAIs <a href="https://openai.com/en/index/gpt-5-6/">GPT 5.6</a></li>
    </ul>
  </li>
  <li>Frontier Models (Open):
    <ul>
      <li>zAIs <a href="https://huggingface.co/zai-org/GLM-5.2">GLM 5.2</a></li>
    </ul>
  </li>
  <li><a href="/2025/10/25/miniapigw.html">mini-apigw: A Lightweight Gateway for Multi-Model AI Infrastructure</a>
    <ul>
      <li>Including the <a href="https://pypi.org/project/mini-apigw/">PyPi project</a> and <a href="https://github.com/tspspi/mini-apigw">GitHub repository</a></li>
    </ul>
  </li>
  <li>The <a href="https://ollama.com/">ollama</a> runtime, that makes executing models locally very easy</li>
  <li><a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> as LLM runtime</li>
  <li><a href="https://github.com/JustVugg/colibri">colibri</a> as runtime for the GLM-5.2 744B MoE system on a 25GB RAM consumer machine</li>
  <li>TU-Wiens <a href="https://github.com/TU-Wien-dataLAB/aqueduct">Aqueduct gateway</a></li>
  <li><a href="/2026/07/05/qwencodex.html">A short story of using QWen3.6:35b-a3b with codex</a></li>
</ul>]]></content><author><name>tsp</name></author><category term="Programming" /><category term="Artificial Intelligence" /><category term="System administration" /><category term="Large Language Models" /><category term="Machine learning" /><category term="Vibe coding" /><category term="Case study" /><summary type="html"><![CDATA[Modern coding agents such as Codex are designed around OpenAIs Responses API and most of the time around a specific large language model, making it difficult to use them with self-hosted models or alternative inference backends. Fortunately, this limitation is mostly architectural rather than fundamental. By introducing a lightweight API gateway that translates between the Responses API and more widely supported interfaces such as Chat Completions or Ollamas API, Codex can be connected to a wide range of local and institutional LLM deployments. This short article demonstrates how to configure Codex for a custom OpenAI compatible gateway, explains the required profile and model metadata files, and discusses the motivation behind introducing a compatibility layer. Whether you are running local models with llama.cpp, Ollama, Colibri, or a shared university inference service, the same approach allows existing applications to remain vendor-independent while presenting a stable interface to rapidly evolving AI backends.]]></summary></entry><entry><title type="html">Running Codex (or other applications) in lightweight FreeBSD jails</title><link href="https://www.tspi.at/2026/07/12/codexyolojail.html" rel="alternate" type="text/html" title="Running Codex (or other applications) in lightweight FreeBSD jails" /><published>2026-07-12T00:00:00+02:00</published><updated>2026-07-12T04:44:48+02:00</updated><id>https://www.tspi.at/2026/07/12/codexyolojail</id><content type="html" xml:base="https://www.tspi.at/2026/07/12/codexyolojail.html"><![CDATA[<p>It is often necessary to execute somewhat untrusted applications in an isolated environment. Keep in mind that you should never run <em>really</em> untrusted applications where malicious intent is possible on the same physical machine as any other application - always expect sidechannels being possible no matter what isolation or virtualization technique you are using. History has repeatedly shown that such side channels exist. But sometimes you want to isolate applications against accidental or unintended side effects.</p>

<p>In the following article we are going to take a look on how to isolate a typical agent orchestrator for large language models in its own enclave while allowing it to interact with the network as well as with local filesystems and local applications. In the context of this blog article this will be OpenAI’s <a href="https://github.com/openai/codex">codex</a>, that I am using in this example in <code class="language-plaintext highlighter-rouge">--yolo</code> mode (granting it all access without any approval, experimenting with the long-term planning capabilities of such agents (not only for software development). This will allow the orchestrator to write its own software and execute it on the host to solve problems without interaction of the user. I have also developed my own agent orchestrator that uses a similar jail based approach for isolation of its subagents too, but this article will focus on <code class="language-plaintext highlighter-rouge">codex</code>.</p>

<p>The first solution that comes to mind is to use a separate user account and just execute the applications there. This is perfectly valid when you don’t want to provide isolation of the visible kernel namespaces (like running processes, physical networking properties, etc.). But sometimes you also need this.</p>

<p>Many people would solve this problem using <a href="https://www.docker.com">Docker containers</a>. Docker primarily addresses application deployment and reproducible software environments rather than security isolation. Although Linux namespaces, seccomp and related mechanisms can provide substantial isolation depending on the configuration, they solve a somewhat different problem. For the lightweight userspace virtualization presented here, FreeBSD jails are a particularly elegant fit.</p>

<ul>
  <li><a href="#what-are-freebsd-jails&quot;">What are FreeBSD jails?</a></li>
  <li><a href="#creating-the-jail">Creating the jail</a>
    <ul>
      <li><a href="#creating-filesystems-and-directories">Creating Filesystems and Directories</a></li>
      <li><a href="#filesystems-and-fstab">Filesystems and <code class="language-plaintext highlighter-rouge">fstab</code></a></li>
      <li><a href="#populating-the-jails-etc">Populating the jails <code class="language-plaintext highlighter-rouge">/etc</code></a></li>
      <li><a href="#jail-configuration">Jail configuration</a></li>
      <li><a href="#launching-and-testing-the-jail">Launching and testing the jail</a></li>
      <li><a href="#the-codex-start-command">The codex start command</a></li>
    </ul>
  </li>
  <li><a href="#resource-control">Resource Control</a>
    <ul>
      <li><a href="#rctl">rctl</a></li>
      <li><a href="#cpuset">cpuset</a></li>
      <li><a href="#zfs-quotas">ZFS quotas</a></li>
    </ul>
  </li>
  <li><a href="#security-considerations">Security Considerations</a>
    <ul>
      <li><a href="#shared-kernel-resources">Shared kernel resources</a></li>
      <li><a href="#side-channels">Side channels</a></li>
      <li><a href="#shared-user-permissions">Shared user permissions</a></li>
      <li><a href="#networking">Networking</a></li>
      <li><a href="#possible-future-hardening">Possible future hardening</a></li>
    </ul>
  </li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#references">References</a></li>
</ul>

<p><img src="/assets/images/png/sandbox001.png" alt="" /></p>

<h2 id="what-are-freebsd-jails">What are FreeBSD jails?</h2>

<p>A better approach to such a problem is actually applying <em>userspace virtualization</em> techniques like FreeBSD’s <a href="https://docs.freebsd.org/en/books/handbook/jails/">jails</a>.  Long before application containers such as Docker became popular, and even before Linux gained technologies such as <a href="https://man7.org/linux/man-pages/man7/cgroups.7.html">cgroups</a> and <a href="https://linuxcontainers.org">LXC</a>, FreeBSD already provided operating-system level virtualization through jails.</p>

<p>Jails build on the concept of <code class="language-plaintext highlighter-rouge">chroot</code> and expand the model by virtualizing access to the file system, the set of users and especially also the networking subsystem. There is a huge number of fine-grained controls available depending on how one wants to realize this access control. All jails still share the same kernel and thus are <em>very</em> lightweight.</p>

<p>Usually one makes a distinction between <em>thick</em> jails and <em>thin</em> jails. <em>Thick</em> jails provide their own complete copy of the base system. All binaries and all filesystems are separated from the host system. This provides a high degree of isolation and one is able to run completely different versions of software (as long as it’s compatible with any of the currently running ABIs to the kernel) or configurations on the host and in different jails. Since each jail runs its own copy of the base system also leakage of sensitive information is prohibited - but on the downside each jail requiring a separate copy of the base system means one requires either the storage space or a read only copy of the given base system. In addition one has to maintain each base system (upgrading, etc.) separately. This administrative overhead should not be underestimated, since keeping the base of <em>each container upgraded</em> is about the load of administering a separate machine for every container (this is something that people who utilize docker containers often forget - or see as a feature running ancient libraries inside containers and call this a “compatibility feature”).</p>

<p><em>Thin</em> jails on the other hand share substantial portions (up to nearly <em>all</em>) parts of the base system. They can be created and bootstrapped <em>extremely</em> fast and share management with the base system. Of course this reduces isolation - they may access information on the base system.</p>

<p>The FreeBSD manual then also introduces the concept of <em>service jails</em> which are just <code class="language-plaintext highlighter-rouge">chroot</code> on steroids. A service jail shares the same root file system as the host. Usually they are separated from the network by default but can be explicitly configured to have access to restricted resources. This follows the philosophy of denying any access that has not been explicitly granted (a very good approach). Such a jail needs zero administration, is very resource efficient, fast to deploy and provides additional process isolation - though overall the reduced isolation has to be considered for security analysis.</p>

<h2 id="creating-the-jail">Creating the jail</h2>

<p>For the purpose of this article I wanted to have an agent that has read only access to all public resources on the host and shares all executables, python packages and data files. This design intentionally follows the principle of least privilege: every writable location is explicitly created, while the remainder of the filesystem is exposed read-only. Thus we are going to <code class="language-plaintext highlighter-rouge">nullfs</code>-mount the root directory in a <em>read only</em> fashion into the jail. We will only add a writable home directory (and optionally also a <code class="language-plaintext highlighter-rouge">tmpfs</code>) into the jail. Note that this means the agent orchestrator can <em>read all host files</em> that are available to its user ID. If you don’t want this create a separate base jail!</p>

<p>Let’s take a walk through creating the environment. For this purpose I will use the following placeholders:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">CODEX_HOME</code> is the home directory <em>on the host</em> in which the agent should execute later on.</li>
  <li><code class="language-plaintext highlighter-rouge">JAIL_ROOT</code> is the location where we are going to store the jail root. This will be its own <code class="language-plaintext highlighter-rouge">zfs</code> filesystem (as also the <code class="language-plaintext highlighter-rouge">CODEX_HOME</code> will be).
    <ul>
      <li>The <code class="language-plaintext highlighter-rouge">JAILNAME</code> variable will be used as placeholder for the name of the jail, that is also part of <code class="language-plaintext highlighter-rouge">JAIL_ROOT</code>.</li>
    </ul>
  </li>
  <li><code class="language-plaintext highlighter-rouge">CODEX_UID</code> and <code class="language-plaintext highlighter-rouge">CODEX_GID</code> are the user ID and group ID that are going to be used by the agent inside the jail. For convenience reasons one can reuse the ID from the host if one also uses the same UID there.</li>
</ul>

<h3 id="creating-filesystems-and-directories">Creating Filesystems and Directories</h3>

<p>First we are going to create the jail root (I assume the <code class="language-plaintext highlighter-rouge">CODEX_HOME</code> is already existing as its own ZFS filesystem) and the jails are actually located under the directory tree <code class="language-plaintext highlighter-rouge">/jails/jails/${JAILNAME}</code> and the <code class="language-plaintext highlighter-rouge">tank/jails/jails/${JAILNAME}</code> ZFS namespace:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zfs create -o mountpoint=/jails/jails/${JAILNAME} tank/jails/jails/${JAILNAME}
chmod 755 /jails/jails/${JAILNAME}

mkdir -p \
    /jails/jails/${JAILNAME}/bin \
    /jails/jails/${JAILNAME}/sbin \
    /jails/jails/${JAILNAME}/lib \
    /jails/jails/${JAILNAME}/libexec \
    /jails/jails/${JAILNAME}/usr \
    /jails/jails/${JAILNAME}/usr/local \
    /jails/jails/${JAILNAME}/etc \
    /jails/jails/${JAILNAME}/dev \
    /jails/jails/${JAILNAME}/tmp \
    /jails/jails/${JAILNAME}/var \
    /jails/jails/${JAILNAME}/var/tmp \
    /jails/jails/${JAILNAME}/var/run \
    /jails/jails/${JAILNAME}/var/log \
    /jails/jails/${JAILNAME}/home \
    /jails/jails/${JAILNAME}/home/${JAILNAME} \
    /jails/jails/${JAILNAME}/root

chmod 1777 /jails/jails/${JAILNAME}/tmp
chmod 1777 /jails/jails/${JAILNAME}/var/tmp
chmod 0755 /jails/jails/${JAILNAME}/var/run
chmod 0700 /jails/jails/${JAILNAME}/root
</code></pre></div></div>

<h3 id="filesystems-and-fstab">Filesystems and <code class="language-plaintext highlighter-rouge">fstab</code></h3>

<p>Now that the filesystem is ready one needs to perform the <code class="language-plaintext highlighter-rouge">nullfs</code>-mounts. What we are going to do via <code class="language-plaintext highlighter-rouge">fstab</code> (<strong>don’t execute it manually</strong>, this will run automatically later on) are the following mounts:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount -t nullfs -o ro /bin /jails/jails/${JAILNAME}/bin
mount -t nullfs -o ro /sbin /jails/jails/${JAILNAME}/sbin
mount -t nullfs -o ro /lib /jails/jails/${JAILNAME}/lib
mount -t nullfs -o ro /libexec /jails/jails/${JAILNAME}/libexec
mount -t nullfs -o ro /usr /jails/jails/${JAILNAME}/usr
mount -t nullfs -o rw /usr/home/${JAILNAME} /jails/jails/${JAILNAME}/home/${JAILNAME}
</code></pre></div></div>

<p>To do this we are going to create a <code class="language-plaintext highlighter-rouge">fstab</code> file that will be executed by the jail management script later on. This could be <code class="language-plaintext highlighter-rouge">/etc/fstab.${JAILNAME}</code> for example:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &gt; /etc/fstab.${JAILNAME} &lt;&lt;EOF
/bin                        /jails/jails/${JAILNAME}/bin                    nullfs  ro  0  0
/sbin                       /jails/jails/${JAILNAME}/sbin                   nullfs  ro  0  0
/lib                        /jails/jails/${JAILNAME}/lib                    nullfs  ro  0  0
/libexec                    /jails/jails/${JAILNAME}/libexec                nullfs  ro  0  0
/usr                        /jails/jails/${JAILNAME}/usr                    nullfs  ro  0  0
/usr/home/${JAILNAME}  /jails/jails/${JAILNAME}/home/${JAILNAME}  nullfs  rw  0  0
EOF
</code></pre></div></div>

<h3 id="populating-the-jails-etc">Populating the jails <code class="language-plaintext highlighter-rouge">/etc</code></h3>

<p>The next step is to populate the <code class="language-plaintext highlighter-rouge">etc</code> directory of the jail. This should not be shared with the host filesystem. We are going to create:</p>

<ul>
  <li>Our own <code class="language-plaintext highlighter-rouge">passwd</code> instance that will only contain the users that should be visible inside the jail</li>
  <li>A <code class="language-plaintext highlighter-rouge">group</code> file that contains only the groups visible inside the jail</li>
  <li>Our own <code class="language-plaintext highlighter-rouge">hosts</code> and <code class="language-plaintext highlighter-rouge">nsswitch.conf</code> (the latter is especially important when using LDAP on the host)</li>
</ul>

<p>In addition we are going to copy <code class="language-plaintext highlighter-rouge">resolv.conf</code>, <code class="language-plaintext highlighter-rouge">shells</code>, <code class="language-plaintext highlighter-rouge">services</code> and <code class="language-plaintext highlighter-rouge">protocols</code> as well as populate our timezone information inside the jail:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &gt; /jails/jails/${JAILNAME}/etc/master.passwd &lt;&lt;EOF
root:*:0:0::0:0:Charlie &amp;:/root:/bin/sh
${JAILNAME}:*:1002:1002::0:0:Codex Agent:/home/${JAILNAME}:/bin/sh
nobody:*:65534:65534::0:0:Unprivileged user:/nonexistent:/usr/sbin/nologin
EOF

pwd_mkdb -d /jails/jails/${JAILNAME}/etc \
    /jails/jails/${JAILNAME}/etc/master.passwd

cat &gt; /jails/jails/${JAILNAME}/etc/group &lt;&lt;EOF
wheel:*:0:root
codex:*:1002:codex
nogroup:*:65533:
nobody:*:65534:
EOF

cat &gt; /jails/jails/${JAILNAME}/etc/nsswitch.conf &lt;&lt;EOF
group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
EOF

cat &gt; /jails/jails/${JAILNAME}/etc/hosts &lt;&lt;EOF
127.0.0.1 localhost
::1 localhost
EOF

cp -p /etc/resolv.conf /jails/jails/${JAILNAME}/etc/resolv.conf
cp -p /etc/shells /jails/jails/${JAILNAME}/etc/shells
cp -p /etc/services /jails/jails/${JAILNAME}/etc/services
cp -p /etc/protocols /jails/jails/${JAILNAME}/etc/protocols
ln -s /usr/share/zoneinfo/Europe/Vienna /jails/jails/${JAILNAME}/etc/localtime

echo 'Europe/Vienna' &gt; /jails/jails/${JAILNAME}/etc/timezone

chown root:wheel /jails/jails/${JAILNAME}/tmp /jails/jails/${JAILNAME}/var/tmp
chmod 1777 /jails/jails/${JAILNAME}/tmp /jails/jails/${JAILNAME}/var/tmp
</code></pre></div></div>

<h3 id="jail-configuration">Jail configuration</h3>

<p>The next step is the actual jail configuration for the jail management script. For now we are going to <em>inherit</em> the IPv6 and legacy IP network configurations. The jails configuration goes into <code class="language-plaintext highlighter-rouge">/etc/jail.conf.d/</code> and will be consumed by the <code class="language-plaintext highlighter-rouge">rc.init</code> script later on.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &gt; /etc/jail.conf.d/${JAILNAME}.conf &lt;&lt;EOF
${JAILNAME} {
    path = "/jails/jails/${JAILNAME}";
    host.hostname = "${JAILNAME}";

    persist;

    mount.fstab = "/etc/fstab.${JAILNAME}";
    mount.devfs;
    devfs_ruleset = 4;

    exec.clean;
    exec.start = "/sbin/ldconfig /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat";
    exec.consolelog = "/var/log/jail_${JAILNAME}_console.log";

    allow.raw_sockets = 0;
    allow.mount = 0;
    allow.set_hostname = 0;
    allow.sysvipc = 0;
    allow.chflags = 0;

    children.max = 0;
    enforce_statfs = 2;

    ip4 = inherit;
    ip6 = inherit;
}
EOF
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">exec.start</code> command is especially important. Unlike a traditional thick jail, this service jail starts with only a minimal filesystem layout. Although the required libraries are available through the read-only <code class="language-plaintext highlighter-rouge">nullfs</code> mounts, the runtime linker cache has not yet been initialized.</p>

<p>Running <code class="language-plaintext highlighter-rouge">ldconfig</code> once inside the jail builds the shared library hints used by the dynamic linker (<code class="language-plaintext highlighter-rouge">ld-elf.so</code>). Without this step dynamically linked applications may fail to locate shared libraries even though they are physically present inside the mounted filesystem.</p>

<p>Strictly speaking this only has to be executed once after creating the jail. Running it during every startup is inexpensive and guarantees that the cache remains consistent even if libraries on the host are upgraded.</p>

<p>The <code class="language-plaintext highlighter-rouge">devfs_ruleset</code> has to match the hosts description - for now I assumed this to be the default jail <code class="language-plaintext highlighter-rouge">devfs</code> ruleset shipped with FreeBSD in the <code class="language-plaintext highlighter-rouge">/etc/devfs.rules</code>.</p>

<h3 id="launching-and-testing-the-jail">Launching and testing the jail</h3>

<p>Now it’s time to start the jail. If <code class="language-plaintext highlighter-rouge">jail_enabled</code> is not set to <code class="language-plaintext highlighter-rouge">TRUE</code> in <code class="language-plaintext highlighter-rouge">/etc/rc.conf</code> one has to use <code class="language-plaintext highlighter-rouge">onestart</code> in the following command:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/rc.d/jail onestart ${JAILNAME}
</code></pre></div></div>

<p>Now one can list the available jails using <code class="language-plaintext highlighter-rouge">jls</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>   JID  IP Address      Hostname                      Path
     5                  ${JAILNAME}              /jails/jails/${JAILNAME}
</code></pre></div></div>

<p>Now as a quick check one can see if the expected users are available - and the jail is not able to resolve external users (i.e. we are really executing with the <code class="language-plaintext highlighter-rouge">/etc/</code> we would be expecting):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jexec ${JAILNAME} id ${JAILNAME}
</code></pre></div></div>

<p>This should yield something like</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uid=1002(${JAILNAME}) gid=1002(codex) groups=1002(codex)
</code></pre></div></div>

<p>Testing for any other user that should exist on the host but not inside the jail should yield</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>id: testuser: no such user
</code></pre></div></div>

<p>Now we can execute a small test script, that will validate that we can execute Python executables inside the jail:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jexec -U ${JAILNAME} ${JAILNAME} /usr/local/bin/python3 - &lt;&lt;PY
import os
import sys
import site
import ssl
 
print("uid:", os.getuid())
print("cwd:", os.getcwd())
print("python:", sys.executable)
print("site:", site.getsitepackages())
print("ssl:", ssl.get_default_verify_paths())
PY
</code></pre></div></div>

<p>This should yield output similar to</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uid: 1002
cwd: /
python: /usr/local/bin/python3
site: ['/usr/local/lib/python3.11/site-packages']
ssl: DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')
</code></pre></div></div>

<p>Now it’s a good idea to also check that we are really unable to write into the read only mounted sections of the file system:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jexec -U ${JAILNAME} ${JAILNAME} /bin/sh -c 'touch /usr/local/THIS_MUST_FAIL'
</code></pre></div></div>

<p>This should yield</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>touch: /usr/local/THIS_MUST_FAIL: Read-only file system
</code></pre></div></div>

<p>As a last step we could take a look what we see from the inside of the jail:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jexec -U ${JAILNAME} ${JAILNAME} /bin/sh -c ' \
    echo "Root:" \
    ls -la / \
    echo "Home:" \
    ls -la /home \
    echo "Host root home:" \
    ls -la /root 2&gt;&amp;1 || true \
    echo "Host var:" \
    ls -la /var'
</code></pre></div></div>

<h3 id="the-codex-start-command">The codex start command</h3>

<p>To make life easier we will also create a start command for our <code class="language-plaintext highlighter-rouge">codex</code> instance. In the following example it will always run in <code class="language-plaintext highlighter-rouge">yolo</code> mode and execute with an <a href="https://ollama.com/">ollama</a> backend:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &gt; /usr/home/${YOURUSER}/${JAILNAME}-jailed.sh &lt;&lt;EOF
#!/bin/sh
set -eu

WORKSPACE="\$1"
shift

exec jexec -U codex ${JAILNAME} \
    /usr/bin/env -i \
    HOME=/home/${JAILNAME} \
    USER=${JAILNAME} \
    LOGNAME=${JAILNAME} \
    SHELL=/bin/sh \
    PATH=/usr/local/bin:/usr/bin:/bin \
    LANG=C.UTF-8 \
    LC_ALL=C.UTF-8 \
    TMPDIR=/tmp \
    XDG_CACHE_HOME=/tmp/${JAILNAME}-cache \
    XDG_STATE_HOME=/tmp/${JAILNAME}-state \
    PYTHONPYCACHEPREFIX=/tmp/python-pycache \
    PIP_CACHE_DIR=/tmp/pip-cache \
    CODEX_OSS_BASE_URL="http://203.0.113.2:8182/v1" \
    /bin/sh -c '
        mkdir -p \
            "\$XDG_CACHE_HOME" \
            "\$XDG_STATE_HOME" \
            "\$PYTHONPYCACHEPREFIX" \
            "\$PIP_CACHE_DIR" \

        cd "\$HOME"
        umask 077
        cd "\$WORKSPACE"

        exec /usr/local/bin/codex --yolo --oss -p glm52 -m glm-5.2-abliterated:432b "\$@"
    ' sh "\$@"
EOF

chmod 0755 /usr/home/${YOURUSER}/${JAILNAME}-jailed.sh
</code></pre></div></div>

<p>Now one can invoke the codex instance with</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/${JAILNAME}-jailed.sh WORKSPACE [ARGUMENTS]
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">WORKSPACE</code> variable is a convenience method to change directory inside the jail before executing codex.</p>

<h2 id="resource-control">Resource Control</h2>

<p>Filesystem isolation alone does not prevent an agent from exhausting host resources. Depending on the intended workload it may therefore be useful to combine jails with FreeBSD’s resource control mechanisms.</p>

<p>Typical limits include</p>

<ul>
  <li>maximum number of processes</li>
  <li>maximum resident memory</li>
  <li>CPU utilization</li>
  <li>open file descriptors</li>
  <li>network bandwidth</li>
  <li>filesystem quotas</li>
</ul>

<p>FreeBSD provides several mechanisms for this. These mechanisms are intentionally orthogonal to jails and therefore compose naturally with the isolation model described above.</p>

<h3 id="rctl">rctl</h3>

<p>The <a href="https://man.freebsd.org/cgi/man.cgi?query=rctl">rctl framework</a> allows fine-grained resource limits to be applied to users, login classes or jails. Examples include limiting the maximum amount of physical memory, preventing excessive process creation or restricting CPU utilization.</p>

<p>This is particularly useful when experimenting with autonomous software that may accidentally fork recursively or allocate excessive amounts of memory.</p>

<h3 id="cpuset">cpuset</h3>

<p>For CPU intensive workloads one may additionally assign a jail to dedicated processor cores using <a href="https://man.freebsd.org/cgi/man.cgi?cpuset">cpuset</a>. This prevents long-running AI workloads from interfering with latency-sensitive services executing on the host.</p>

<h3 id="zfs-quotas">ZFS quotas</h3>

<p>Since the writable portions of the jail reside on their own ZFS filesystem, applying quotas and reservations is straightforward. This prevents accidentally generated datasets from consuming all remaining disk space.</p>

<h2 id="security-considerations">Security Considerations</h2>

<p>While the setup shown above provides a significant degree of isolation, it is important to understand exactly what security properties it provides and where its limits lie.</p>

<p>The most important fact is that FreeBSD jails are not virtual machines. Every jail on the system shares the same kernel with the host. A successful kernel exploit therefore compromises every jail as well as the host system. For this reason one should never execute software with a realistic expectation of malicious behaviour on the same physical machine as sensitive workloads. In such situations a dedicated machine with external network isolation remains the correct solution.</p>

<p>The setup described here should instead be viewed as protection against accidental damage, programming errors and unintended side effects of autonomous agents. In other words, the threat model considered here assumes that the agent is buggy or overly enthusiastic rather than actively malicious.</p>

<h3 id="shared-kernel-resources">Shared kernel resources</h3>

<p>Even though filesystem namespaces, user databases and many networking facilities are isolated, many kernel resources remain shared:</p>

<ul>
  <li>scheduler</li>
  <li>memory allocator</li>
  <li>CPU caches</li>
  <li>kernel networking stack</li>
  <li>block device drivers</li>
</ul>

<p>Any kernel vulnerability affects both host and jail.</p>

<h3 id="side-channels">Side channels</h3>

<p>Like all operating-system level virtualization technologies, jails cannot protect against hardware side channels.</p>

<p>Examples include</p>

<ul>
  <li>CPU cache timing attacks</li>
  <li>speculative execution attacks (Spectre, Meltdown, MDS, etc.)</li>
  <li>branch predictor state</li>
  <li>memory bus contention</li>
  <li>shared filesystem timing</li>
  <li>shared network timing</li>
</ul>

<p>Although exploiting such channels is considerably harder than simply reading files, they demonstrate why strong isolation between mutually distrustful workloads still requires separate physical machines.</p>

<p>Fortunately these attacks are generally irrelevant when the goal is merely preventing an AI agent from accidentally modifying the host system.</p>

<h3 id="shared-user-permissions">Shared user permissions</h3>

<p>In the example above the jail intentionally reuses the host UID. This greatly simplifies access to existing workspaces and avoids maintaining duplicate file ownership.</p>

<p>However, this also means the agent can still read every file that this user could normally access. If this is undesirable, a dedicated service account together with a separate thin or thick jail should be created instead.</p>

<h3 id="networking">Networking</h3>

<p>The example configuration inherits the host networking stack. This was chosen because the primary goal is running local AI agents with minimal administrative overhead. Inherited networking also avoids creating virtual interfaces or bridge configurations while still allowing direct access to locally running services such as Ollama or MCP servers.</p>

<p>If stronger network isolation is required, <a href="https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf">FreeBSD’s VNET</a> infrastructure allows each jail to receive its own complete virtual networking stack, including independent interfaces, routing tables and firewall rules.</p>

<h3 id="possible-future-hardening">Possible future hardening</h3>

<p>The setup presented here intentionally balances simplicity and isolation. Depending on the threat model one could further strengthen the environment by introducing additional FreeBSD mechanisms, for example:</p>

<ul>
  <li>VNET networking instead of inherited networking.</li>
  <li>Dedicated service accounts with minimal filesystem permissions.</li>
  <li>Separate thin or thick jails for applications requiring stronger isolation.</li>
  <li>ZFS snapshots for quickly restoring the writable workspace after experiments.</li>
  <li>Resource limits using rctl and CPU affinity through cpuset.</li>
  <li>Mandatory Access Control (MAC) policies for additional kernel-enforced restrictions.</li>
  <li>Capsicum capability mode for individual helper applications that support it.</li>
  <li>Read-only package repositories and immutable build environments.</li>
</ul>

<p>None of these change the overall architecture described in this article, but they illustrate how the presented service jail can serve as a foundation for progressively stronger isolation as requirements evolve.</p>

<h2 id="conclusion">Conclusion</h2>

<p>FreeBSD service jails provide a remarkably lightweight mechanism for isolating AI agents without introducing another operating system instance. By sharing only the components that need to be shared while keeping writable state confined to dedicated locations, one can achieve a practical balance between convenience, performance and security. Although this is not a replacement for virtual machines when defending against determined attackers, it is an excellent solution for protecting a workstation from accidental damage caused by autonomous agents.</p>

<p>Although the examples in this article use OpenAI’s Codex, exactly the same approach works for many other autonomous development agents, CI workers, automation services or long-running background applications that benefit from lightweight isolation.</p>

<h2 id="references">References</h2>

<ul>
  <li><a href="/2026/07/05/qwencodex.html">A short story of using QWen3.6:35b-a3b with codex</a></li>
  <li><a href="/2026/03/24/hardwareloop.html">Using Codex with Hardware In The Loop for Microcontrollers</a></li>
  <li><a href="/2026/01/08/proofllm.html">Using Proof Assistants to Ground LLMs in Mathematical Truth</a></li>
  <li>OpenAIs <a href="https://github.com/openai/codex">Codex</a> lightweight coding agent</li>
  <li><a href="https://www.docker.com/">Docker</a>, one of the most popular container solutions</li>
  <li><a href="https://xenproject.org/">Xen</a>, the most popular hypervisor for large scale virtualization deployments.</li>
  <li><a href="https://docs.freebsd.org/en/books/handbook/jails/">FreeBSD jails</a></li>
  <li><a href="https://linuxcontainers.org/">lxc</a>, Linux version of jails</li>
  <li><a href="https://man7.org/linux/man-pages/man7/cgroups.7.html">cgroups</a>, a method to define fine grained permissions on Linux</li>
  <li>Books:
    <ul>
      <li>Michael W Lucas, <a href="https://amzn.to/4ydMyNi">FreeBSD Mastery: Jails (IT Mastery, Band 15)</a></li>
      <li>X.Y. Wang, <a href="https://amzn.to/4w3SpDq">The FreeBSD Admin’s Handbook: Configuration, Tuning, and Maintenance</a></li>
      <li>George Neville-Neil, Marshall McKusick, Robert Watson, <a href="https://amzn.to/3SR3cCl">Design and Implementation of the FreeBSD Operating System</a></li>
      <li>Michael W Lucas, <a href="https://amzn.to/3SV0SdC">Absolute FreeBSD, 3rd Edition: The Complete Guide to FreeBSD</a></li>
    </ul>
  </li>
</ul>]]></content><author><name>tsp</name></author><category term="Programming" /><category term="Artificial Intelligence" /><category term="Tutorial" /><category term="Basics" /><category term="FreeBSD" /><category term="System administration" /><category term="Large Language Models" /><category term="Machine learning" /><category term="Xen" /><category term="LLM" /><category term="Vibe coding" /><category term="OpenAI" /><category term="Testing" /><category term="Ollama" /><summary type="html"><![CDATA[Modern AI agents and automation tools are increasingly capable of writing code, executing commands and interacting with local services. While this opens exciting possibilities, it also raises an important question: how can we let such applications work autonomously without granting unrestricted access to our entire workstation? This article demonstrates how FreeBSDs lightweight service jails provide an elegant answer, isolating autonomous applications while keeping them fast and convenient to use. Using OpenAIs Codex as a practical example, we build a minimal service jail that shares only the resources an agent actually requires. Along the way we explore nullfs, custom user databases, runtime linker initialization, resource control, security considerations and the limitations of operating-system level virtualization. Although the example focuses on Codex, the same architecture is equally applicable to CI workers, automation services and many other long-running applications that benefit from lightweight isolation.]]></summary></entry></feed>