This apparent paradox often leads to misinterpretation. Observers may assume that the narrow confidence bands of the fitted model represent the measurement uncertainty, and consequently judge the data against these bands, leading to incorrect conclusions about data quality or model validity.

This article clarifies the distinction between:

• Measurement error
• Model (fit) uncertainty
• Residuals and data-driven variance (Measurement error)
• Prediction error

We will demonstrate how these quantities arise via a simulated measurement, how they should be interpreted, and how they can be computed in practice.

In the end we will provide a short summary and conclusion.

Measurement Error vs Model Uncertainty

Measurement Error

Measurement error describes the uncertainty associated with each observed data point. Formally, we write:

Here

• $y_i$ is the measured value
• $f(x_i, \theta)$ is the underlying model at the position $x_i$ and for the parameter set $\theta$
• $\epsilon_i$ is the random error term with a variance $\sigma_y^2$

The error $\epsilon_i$ is typically determined by the measurement process itself (measurement noise, environmental fluctuations, discretization, etc.). This corresponds to the standard deviation of the measurement process. $y_i$ is typically the mean obtained from repeated measurements.

Model (Fit) Uncertainty

When fitting a model $f(x, \theta)$ to data, the parameters $\theta$ are estimated from all observations. The uncertainty of these parameters is given by the covariance matrix:

The matrix encodes how precisely the parameters are determined by the fitting / regression procedure. The uncertainty of the model prediction at a given point $x$ is obtained by propagating the covariance:

The quantity $\sigma_f$ represents the confidence band of the fitted model. The width of this band decreases with the number of data points (similar to the standard error or a measurement. For well conditioned problems with independent observations, the scaling can be estimated for many problems as:

Even if individual measurements are noisy, the estimated parameters of the assumed model can be determined very precisely.

Fit uncertainty $\sigma_f$: How confident can we be about the fitted model.

Note that a small $\sigma_f$ does not imply that the model is correct. You need to apply proper statistical tests on your hypothesis.

Residuals and Data-Driven Variance

The residuals quantify how well the model describes the observed data:

Here $\hat{y_i} = f(x_i, \hat{\theta})$ is the prediction of the data value by the fitted model. From these residuals one can estimate the variance of the data around the model:

Here:

• $N$ is the number of observations (measurement points)
• $p$ is the number of fitted parameters
• $N-p$ thus is the degrees of freedom

The quantity $\sigma_r$ represents the intrinsic scatter of the data and is typically comparable to the measurement noise (but they are not equal). In case of correlated noise $\sigma_r$ underestimates true uncertainty.

Residual error / intrinsic scatter $\sigma_r$: How much does the measurement process scatter (typically comparable to the measurement noise, though $\sigma_r$ includes also the model mismatch, unmodeled systematics, etc.)

Prediction Error

The prediction error describes where - at a given point of your model - you would expect the next measurement to reside with a given certainty. This must account for two contributions, that are typically treated as independent:

• Uncertainty of the model parameters (given by $\sigma_f$)
• Scatter of the data around the model (given by $\sigma_r$)

This corresponds to the classical distinction between confidence intervals, the uncertainty of the fitted mean model, and prediction intervals, the uncertainty of individual observations.

Under the assumption of independence this yields a total error $\sigma$:

Prediction error $\sigma$: How well can the model predict a new measurement at position $x$

Keep in mind that the assumption of independence breaks in case of heteroscedastic errors or correlated noise!

A Practical Example

To illustrate the concepts, we simulate a derivative Lorentzian (Cauchy) shaped signal, add noise in both axes, perform a fit and then compute the parameter uncertainties, the model confidence band, the residual variance and the full prediction uncertainty.

In this plot one can see:

First the blue (simulated) datapoints. The simulation assumes an amplitude of $A=120$, $x_0=400$, $\mathrm{FWHM}=3.0$ (i.e. $\gamma=1.5$), $\sigma_x = 0.3$ and $\sigma_y = 0.7 * \mathrm{max}(y_i)$. On top of these we performed the orange fit using the Levenberg-Marquardt algorithm to perform a least squares fit against the same model function that has been used to synthesize the data. This yields $\hat{x_0} = 399.892560 \pm 0.097864 \mathrm{MHz}$ and $\mathrm{FWHM} = 2.369803 \pm 0.385163 \mathrm{MHz}$. The narrow blue region around the orange fit function is the fit uncertainty $\sigma_f$. As one can see this is extremly narrow and does not reflect the scatter of individual datapoints. When one would compare this region with new datapoints one would get the impression that the measurements would not confirm the hypothesis given by the model. When adding the residual measurement error $\sigma_r$ one gets the total error $\sigma = \sqrt{\sigma_f(x)^2 + \sigma_r^2}$, which is shown as the orange band. This is much wider and one can estimate this to include around 68 percent of all datapoints. The blue errorbar like line on top of the points is again $\sigma_r$, the expected scatter of individual measurements.

As one can see the confidence of the model (orange region) is much narrower than the prediction region for individual measurements.

Conclusion and the Common Interpretation Pitfall

Comparing measurement data directly to the confidence band $\sigma_f(x)$ instead of the prediction interval $\sigma(x)$ is a common mistake and leads to systematic overestimation of discrepancies between model and data.

The correct interpretation is:

• $\sigma_f(x)$: Confidence in the mean fitted model
• $\sigma_r$: Scatter of individual measurements
• $\sigma(x)$: Uncertainty of predicted future observations

This implies, that for sufficiently large datasets:

This leads to the conclusions:

• A fitted model can be known much more precisely than individual measurements
• The covariance matrix $\mathrm{Cov}(\theta)$ describes the parameter uncertainty, not measurement noise
• The model (fit) uncertainty $\sigma_f$ describes the confidence of the model. This corresponds to the posterior uncertainty of the model.
• Residuals $\sigma_r$ capture the intrinsic data scatter or the measurement noise
• The correct uncertainty for future predictions is the combination of both effects $\sigma(x) = \sqrt{\sigma_f(x)^2 + \sigma_r^2}$. This corresponds to the posterior predictive distribution.

One of the most enduring implementations of this idea is ModBus, particularly in its RS485-based RTU variant. Despite its age, ModBus remains widely used in industrial control, laboratory environments, energy systems, and increasingly in small-scale automation such as homes, gardens, and greenhouses.

This article explores ModBus from a practical systems perspective. It focuses on my own RS485-based deployments, the challenges that arise when integrating such systems into modern software environments, and presents a set of tools I personally designed to bridge the gap between legacy fieldbus systems and contemporary infrastructure that I use with my own deployments, including a gateway service, a hardware implementation for Atmel AVR microcontrollers and a software client for Python applications and scripts.

• Why ModBus Still Matters
• Physical Layer: RS485 in Practice
• Protocol Layer: ModBus RTU
• The Real Problem: Multi-Master Access
• Architecture: A Central ModBus Gateway
  • My Implementation: modbusgw
    • Installation
    • Configuration
    • Frontend Configurations
      • Virtual Serial Ports (pty)
      • ModBus IP TCP Socket
    • Backend Configurations
      • Hardware Serial Ports
      • ModBus IP via TCP
    • Routing Configuration
    • Example configuration file
    • Launching and Controlling the Gateway
• Client Library
  • A Simple Example
• Embedded Side: AVR Framework
• Practical Applications
  • My Favorite Hardware Devices
• Security Considerations
• Design Philosophy
• Conclusion
• References

Why ModBus Still Matters

What makes ModBus particularly interesting is not its feature set - but the absence of it. Its simplicity leads to robustness, debuggability and long-term stability. There are no hidden layers, no opaque negotiation steps, and no dynamic topology discovery. Everything is explicit and implementable by any manufacturer or hobbyist.

This simplicity extends all the way down to the physical and protocol layers, making ModBus exceptionally easy to implement even on very constrained hardware. Unlike CAN or Ethernet-based systems - which require more complex controllers, protocol stacks and often significantly more expensive interface hardware - ModBus RTU over RS485 can be realized on sub-Euro microcontrollers with minimal resources and very inexpensive transceiver ICs. This makes it particularly attractive for distributed sensing and control applications where cost, simplicity and reliability are more important than raw throughput.

Compared to more modern systems such as MQTT-based automation or high-performance fieldbuses like EtherCAT, ModBus trades flexibility and throughput for predictability, reliability and ease of implementation. In many environments - especially where timing constraints are moderate and reliability is critical - this tradeoff is highly desirable.

Limitations

While ModBus excels through its simplicity, this minimalism also imposes a number of practical limitations that must be considered when designing real-world systems.

One of the most prominent constraints is throughput. Especially in RS485-based RTU deployments, the achievable data rate is relatively low. At 9600 baud, effective payload throughput is on the order of only a few kilobytes per second, and even at higher baud rates the request–response nature of the protocol introduces unavoidable overhead. As the number of devices grows, polling cycles become longer, increasing latency for both control and monitoring tasks. Though for most simple monitoring and control tasks that operate more on the order of multiple seconds to minutes this does not matter.

Closely related to this is the strict master–slave model, which prevents concurrent access to the bus. All communication must be serialized and initiated by the master, and even read-only operations cannot be performed in parallel. This becomes increasingly problematic in modern systems where multiple independent services require access to the same data. Without an additional coordination layer, such as the gateway architecture presented later in this article, this leads to contention and non-deterministic behavior. In addition the requirement of the master initiating the communication prevents fast event notification from sensors, the time constraint is defined by the polling interval by the master.

Another limitation lies in the lack of higher-level protocol features. ModBus provides no built-in mechanisms for device discovery, configuration, or semantic description of data. Registers are purely numerical and their meaning is defined externally, often in device-specific documentation. This makes integration straightforward for simple systems, but increasingly complex as systems grow and heterogeneous devices are introduced.

From a security perspective, ModBus in its original form offers no authentication, no encryption, and no integrity protection beyond basic checksums. While this is acceptable in isolated industrial networks, it becomes a critical issue when systems are connected to larger infrastructures or exposed to untrusted environments.

Finally, although often described as deterministic, real-world ModBus systems can exhibit variable latency due to device response times, retries, and bus contention. Determinism exists primarily at the protocol level, but system-level timing guarantees depend heavily on implementation details and network design.

These limitations do not diminish the value of ModBus - in many cases, they are the direct consequence of its simplicity. However, they highlight the need for carefully designed system architectures when integrating ModBus into modern, distributed environments.

Physical Layer: RS485 in Practice

RS485 provides a differential signaling scheme that allows reliable communication over long distances and in electrically noisy environments. Unlike single-ended signaling, RS485 transmits the difference between two lines, making it highly resilient against common-mode noise. The following image shows the capture of the A line (yellow), the B line (turquoise) and the calculate difference (violet) of a RS485 transmission on a cheap USB oscilloscope

Another important characteristic is its physical reach. At relatively low baud rates such as 9600, cable lengths of up to roughly 1400 meters are achievable on standard twisted-pair copper cabling without requiring fiber optics (note that you are usually using $0.75 \mathrm{mm}^2$ cabling with 4 poles for A, B, ground and DC supply voltage between 5 and 36V). Even at higher data rates like 115200 baud, distances on the order of 400 meters are still realistic within a single segment. This makes RS485 particularly attractive for distributed installations such as gardens, greenhouses, industrial halls, or laboratory environments where devices are spread across medium scale areas.

Typical deployments use a linear bus topology with termination resistors at both ends. Correct termination ($120 \Omega$ resistors) and biasing are essential to avoid reflections and undefined bus states. In practice, many issues attributed to protocol problems are in fact caused by improper physical layer implementation.

The bus is usually operated in half-duplex mode, meaning that only one device can transmit at a time. This leads directly to one of the central architectural constraints of ModBus RTU systems: arbitration.

There are, however, practical limits. While the protocol allows addressing up to 255 devices, real-world deployments are usually constrained by electrical loading of the bus. In many cases, a single RS485 segment supports on the order of 32-128 devices, depending on transceiver characteristics, bus loading, termination quality, and topology. Careful design - such as using repeaters or segmenting the bus - may be required for larger installations.

Protocol Layer: ModBus RTU

ModBus RTU operates on a strict master–slave model. A single master initiates all communication, while slaves only respond to requests addressed to them.

Frames are transmitted within timing constraints, including mandatory silent intervals that delimit frames (typically 3.5 character times), which is used as frame delemiter and to detect message boundaries. Each frame contains:

• Device address
• Function code
• Payload
• CRC checksum

Even though timing is part of the protocol and slave implementations must be careful when interpreting silence periods, on the master side, typical implementations using hardware UARTs (for example in USB-to-RS485 adapters) do not impose strict timing constraints and are largely insensitive to operating system scheduling or buffering and thus easy to implement from the software side. Timing requirements are primarily relevant on the slave side, where frame detection depends on correct interpretation of inter-frame gaps.

The Real Problem: Multi-Master Access

ModBus RTU assumes a single master. In practice, modern systems often require multiple independent software components to access the same bus. These software architectures are typically composed of multiple loosely coupled services - as of today often following microservice principles - to improve modularity, scalability and maintainability through separation of concerns. In such environments, it is common that different services require access to the same physical devices for control, monitoring or logging purposes. Techniques such as Command Query Responsibility Segregation (CQRS) further emphasize this separation by distinguishing between control paths (commands that modify system state) and read paths (queries used for monitoring and reporting), which may also operate under different security constraints.

Without coordination, this leads to collisions, corrupted frames and undefined system behavior. Even if collisions are avoided, interleaving requests from multiple sources can break assumptions about timing and state. This mismatch between the original design and modern usage patterns is one of the key challenges when integrating ModBus into contemporary systems.

I resolved the problem for my personal deployments by developing modbusgw, presented in the next section.

Architecture: A Central ModBus Gateway

To resolve the multi-master problem, a gateway can be introduced. This gateway acts as the only physical master on the given RS485 bus, while exposing multiple logical interfaces to clients. Conceptually, the gateway acts as a serialization layer for bus access while exposing a parallel interface to clients.

The gateway performs:

• Arbitration between competing requests
• Scheduling of bus access
• Mapping between different transport layers
• Remapping device IDs between different virtual bus representations and the real physical backends

It allows multiple applications to interact with the same physical buses safely and deterministically.

My Implementation: modbusgw

The presented solution, modbusgw, implements this gateway architecture in a modular fashion.

Frontends allow clients to access the service:

• Virtual serial ports (PTY) provide a interface that looks like the real physical UART based hardware
• Unix domain sockets (UDS) for locally interacting ModBus/TCP systems without exposing themselves to the network.
• TCP sockets for remote access via control networks.
  • Optional TLS and mutual TLS for secure communication and client authentication for systems exposed to non-isolated networks.

Backends connect to actual devices:

• RS485 buses via USB adapters
• Remote ModBus/TCP devices

A routing layer in between allows mapping of device IDs and registers, as well as filtering requests. This enables the creation of security boundaries within the system and allows selective exposure of functionality to different sets of clients.

Installation

The gateway has been implemented in Python and is available on GitHub. It can be installed via it’s PyPi package:

pip install modbus-gateway

Configuration

The application is configured from a single JSON configuration file. By default it resides - when using the FreeBSD rc.init script - at /usr/local/etc/modbusgateway.cfg, when executing the program from the commandline the default configuration resides at ~/.config/modbusgateway.cfg. The location configuration file can be overriden via the --config flag or the modbusgw_config option in /etc/rc.conf

The configuration is split into different sections:

• service provides configuration of the main daemon
• bus configures the internal message bus
• frontends contains a list of frontend configurations over which clients are capable of accessing the daemon
• backends is the counterparts and defines the interfaces that are accessed on behalf of the clients via the gateway.
• routes provides a match-list based configuration on how to route messages between frontends and backends.

The service section configures PID file to prevent multiple running instances, the state directory that will be used for log- and tracefiles as well as the loglevel:

"service" : {
   "log_level" : "INFO",
   "pid_file" : "/var/run/modbusgw.pid",
   "state_dir" : "/var/modbusgw/",
   "reload_grace_seconds" : 5
}

The bus configuration configures the internal buffer for incoming requests that are routed to various backends:

"bus" : {
   "request_queue_size" : 64,
   "response_timeout_ms" : 1500
}

Note that this timeout should be shorter than the applications and frontends timeouts.

Frontend Configurations

Virtual Serial Ports (pty)

Virtual serial ports are directly accessible via pyserial and similar interfaces. This allows existing legacy software to access the gateway via unmodified code by pointing it at the virtual serial port file handles:

{
   "id" : "virtual_serial_rtu",
   "type" : "serial_rtu_socket",
   "socket_path" : "/var/modbusgw/ttyBus0",
   "pty_mode" : "rw",
   "idle_close_seconds" : 600,
   "frame_timeout_ms" : 5.0
}

The shown configuration instantiates a virtual serial port at the specified socket_path, allowing read-write transactions. The frame timeout handles incomplete messages on the application side. The name virtual_serial_rtu is an arbitrary chosen name that is used in the routing configuration.

ModBus IP TCP Socket

A ModBus IP socket speaks the ModBus IP protocol over an TCP socket (optionally supporting TLS or mTLS for authenticated sessions). The following configuration exposes unencrypted ModBus IP applying only IP subnet based filters:

{
   "id" : "frontend_tcp",
   "type" : "tcp_modbus_tcp",
   "host" : "192.0.2.1",
   "port" : 1234,
   "cidr_allow" : [
      "127.0.0.0/8",
      "192.0.2.0/24"
   ]
}

If TLS is desired the following configuration can be added to the frontend configuration object:

   "tls" : {
      "cert_file" : "/path/to/server.crt",
      "key_file" : "/path/to/server.key",
      "ca_file" : "/path/to/rootca.crt",
      "require_client_cert" : true,
      "client_dn_allow" : [
         "CN=ModbusGW Test Client"
      ]
   }

The cert_file and key_file establish the server identity. The ca_file is only used when require_client_cert is set to true to allow client authentication. The additional (optional) client_dn_allow filter allows to filter the DNs from valid certificates (after certificate validation) that are allowed to access the frontend.

Backend Configurations

Hardware Serial Ports

The pyserial backend uses the pyserial library to access an USB to RS485 based interface. This is the most simple hardware interface for DIY setups. The specified serial configuration is applied when accessing the backend. Again the arbitrary id is used in the routing configuration.

{
   "id" : "hardware_serial",
   "type" : "pyserial",
   "device" : "/dev/ttyU0",
   "baudrate" : 9600,
   "parity" : "N",
   "stop_bits" : 1,
   "request_timeout_ms" : 1200
}

ModBus IP via TCP

A TCP backend can be configured via the tcp_modbus backend:

{
   "id" : "tcp_backend",
   "type" : "tcp_modbus",
   "host" : "127.0.0.1",
   "port" : 1234,
   "connect_timeout" : 2.0,
   "pool_size" : 2,
   "use_tls" : true,
   "tls" : {
      "ca_file" : "/path/to/root.crt",
      "cert_file" : "/path/to/client.crt",
      "key_file" : "/path/to/client.key"
   }
}

The use_tls and tls blocks are optional and are only used when (m)TLS is desired. The root.crt is used for validation, the client keys for authentication via mTLS.

Routing Configuration

The routing configuration is provided as a list of routing commands that are matched against incoming requests from the frontends. The first match determines to which backend a message is routed. The backend key and the mirror_to_mqtt key is not used for matching, all other fields apply:

{
   "frontend" : "virtual_serial_rtu",
   "backend" : "hardware_serial",
   "match" : {
      "unit_ids" : [ "*" ],
      "function_codes" : [ "*" ]
   },
   "mirror_to_mqtt" : [ ]
}

The routing match block allows to filter given device IDs and function codes as well as operations. For example to allow only function code 1 (read coils) for the virtual device 5, redirecting the operation to the backend device id 1, one would use

{
   "frontend" : "virtual_serial_rtu",
   "backend" : "hardware_serial",
   "match" : {
      "unit_ids" : [ 5 ],
      "function_codes" : [ 1 ],
      "operations" : [ "read" ]
   },
   "unit_override" : 1,
   "mirror_to_mqtt" : [ ]
}

Here the match block specifies conditions that have to be fulfilled (all have to be fulfilled). The optional unit_override replaces the device ID on the virtual frontend bus to the given unit number before handing off the the backend device. All fields can be used in arbitrary combinations.

Example configuration file

The following configuration exposes a single serial to RS485 interface via a local virtual serial port as well as a ModBus IP socket available via unencrypted TCP:

{
   "service" : {
      "log_level" : "INFO",
      "pid_file" : "/var/run/modbusgw.pid",
      "state_dir" : "/var/modbusgw/",
      "reload_grace_seconds" : 5
   },
   "bus" : {
      "request_queue_size" : 64,
      "response_timeout_ms" : 1500
   },
   "frontends" : [
      {
         "id" : "virtual_serial_rtu",
         "type" : "serial_rtu_socket",
         "socket_path" : "/var/modbusgw/ttyBus0",
         "pty_mode" : "rw",
         "idle_close_seconds" : 600,
         "frame_timeout_ms" : 5.0
      },
      {
         "id" : "frontend_tcp",
         "type" : "tcp_modbus_tcp",
         "host" : "192.0.2.1",
         "port" : 1234,
         "cidr_allow" : [
            "127.0.0.0/8",
            "192.0.2.0/24"
         ]
      }
   ],
   "backends" : [
      {
         "id" : "hardware_serial",
         "type" : "pyserial",
         "device" : "/dev/ttyU0",
         "baudrate" : 9600,
         "parity" : "N",
         "stop_bits" : 1,
         "request_timeout_ms" : 1200
      }
   ],
   "routes" : [
      {
         "frontend" : "virtual_serial_rtu",
         "backend" : "hardware_serial",
         "match" : {
            "unit_ids" : [ "*" ],
            "function_codes" : [ "*" ]
         },
         "mirror_to_mqtt" : [ ]
      },
      {
         "frontend" : "frontend_tcp",
         "backend" : "hardware_serial",
         "match" : {
            "unit_ids" : [ "*" ],
            "function_codes" : [ "*" ]
         },
         "mirror_to_mqtt" : [ ]
      }
   ]
}

Launching and Controlling the Gateway

The gateway can be executed in foreground mode on the command line:

$ modbusgw

In addition it supports executing daemonized. To control the daemon the command line client supports the usual commands:

$ modbusgw start
$ modbusgw stop
$ modbusgw status
$ modbusgw restart
$ modbusgw reload

For usage on FreeBSD, my operating system of choice, one can use an rc.init script stored in /usr/local/etc/rc.d/modbusgw:

#!/bin/sh
# PROVIDE: modbusgw
# REQUIRE: LOGIN
# KEYWORD: shutdown

. /etc/rc.subr

name="modbusgw"
rcvar="modbusgw_enable"

load_rc_config $name

: ${modbusgw_enable:="NO"}
: ${modbusgw_command:="/usr/local/bin/modbusgw"}
: ${modbusgw_config:="/usr/local/etc/modbusgateway.cfg"}
: ${modbusgw_user:="modbusgw"}
: ${modbusgw_group:="modbusgw"}
: ${modbusgw_pidfile:="/var/run/modbusgw.pid"}
: ${modbusgw_var_dir:="/var/modbusgw"}
: ${modbusgw_log_file:="${modbusgw_var_dir}/modbusgw.log"}
: ${modbusgw_timeout:="15"}
: ${modbusgw_flags:=""}

command="${modbusgw_command}"
pidfile="${modbusgw_pidfile}"
required_files="${modbusgw_config}"
extra_commands="reload restart status"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
reload_cmd="${name}_reload"
restart_cmd="${name}_restart"
status_cmd="${name}_status"

modbusgw_ensure_var_dir()
{
	if [ ! -d "${modbusgw_var_dir}" ]; then
		install -d -o "${modbusgw_user}" -g "${modbusgw_group}" -m 0750 "${modbusgw_var_dir}"
	else
		chown "${modbusgw_user}:${modbusgw_group}" "${modbusgw_var_dir}"
	fi
}

modbusgw_build_cmd()
{
	_subcmd="$1"
	shift
	_cmd="${command} -c \"${modbusgw_config}\" ${_subcmd}"
	if [ -n "${modbusgw_log_file}" ]; then
		_cmd="${_cmd} --log-file \"${modbusgw_log_file}\""
	fi
	for _arg in "$@"; do
		_cmd="${_cmd} ${_arg}"
	done
	if [ -n "${modbusgw_flags}" ]; then
		_cmd="${_cmd} ${modbusgw_flags}"
	fi
	echo "${_cmd}"
}

modbusgw_run()
{
	_cmd=$(modbusgw_build_cmd "$@")
	if [ "$(id -un)" = "${modbusgw_user}" ]; then
		/bin/sh -c "${_cmd}"
	else
		su -m "${modbusgw_user}" -c "${_cmd}"
	fi
}

modbusgw_start()
{
	modbusgw_ensure_var_dir
	modbusgw_run start
}

modbusgw_stop()
{
	modbusgw_run stop --timeout "${modbusgw_timeout}"
}

modbusgw_reload()
{
	modbusgw_run reload
}

modbusgw_restart()
{
	modbusgw_stop
	sleep 1
	modbusgw_start
}

modbusgw_status()
{
	modbusgw_run status
}

run_rc_command "$1"

Then the configuration happens as usual for this system via /etc/rc.conf:

modbusgw_enable="YES"
modbusgw_config="/usr/local/etc/modbusgateway.cfg"
modbusgw_user="modbusgw"
modbusgw_group="modbusgw"
modbusgw_pidfile="/var/modbusgw/modbusgw.pid"
modbusgw_var_dir="/var/modbusgw"
modbusgw_log_file="/var/modbusgw/modbusgw.log"

Control then is performed via the following commands:

$ /usr/local/etc/rc.d/modbusgw start
$ /usr/local/etc/rc.d/modbusgw stop
$ /usr/local/etc/rc.d/modbusgw status
$ /usr/local/etc/rc.d/modbusgw restart
$ /usr/local/etc/rc.d/modbusgw reload

Client Library

A corresponding client library, also available in the same GitHub repository and installable via a separate PyPi package modbusgw-client, provides a unified interface across different transports to Python applications and scripts. It supports:

• Serial (RTU via pyserial), which can also be used to connect to pty based frontends.
• ModBus/TCP over UDS
• ModBus/TCP over TCP
  • Secure variants using TLS and mTLS

This abstraction allows applications to switch between local and remote deployments without changes to application logic and without exposure to the actual protocol encoding.

A simple example

First, let’s install the package

$ pip install modbusgw-client

Now one can use the TcpClient or the SerialClient classes in a very simple fashion:

#!/usr/local/bin/python3

from time import sleep

from modbusgw_client.tcp_client import TcpClient
from modbusgw_client.serial_client import SerialClient
from modbusgw_client.pdu import WriteSingleCoilRequest, ReadCoilsRequest, ReadHoldingRegistersRequest

# TCP backend

with TcpClient(host="192.0.2.2", port=1234, timeout=10) as client:
   client.execute(WriteSingleCoilRequest(
      unit_id = 2, # The device ID on the virtual bus
      address = 5, # "Coil" index
      True # Coil status
   )

# Serial backend

with SerialClient("/var/modbusgw/ptyBus0", baudrate=9600, timeout=10) as client:
   client.execute(WriteSingleCoilRequest(
      unit_id = 2, # The device ID on the virtual bus
      address = 5, # "Coil" index
      True # Coil status
   )

Embedded Side: AVR Framework

On the device side, I developed a lightweight AVR-based ModBus framework, that allows implementation of custom ModBus slaves on cheap readily available Atmel ATMega microcontrollers.

The framework is particularly useful for:

• Custom sensor readout
• Simple control systems
• Bridging analog signals into ModBus systems

One of my current application is the readout of a Infinicon PBR260 Pirani pressure gauge with analog output, making it accessible via ModBus in a vacuum system, as well as the readout of ultrasonic water level sensors for water management in a small garden setup.

The framework is available on GitHub. It is built with avr-gcc and targets the Atmel ATMega328P and ATMega2560. It allows easy interfacing with the coils, input registers, output registers and holding registers. Examples are provided in the GitHub repository.

Practical Applications

In practical deployments, a wide range of ModBus-capable devices can be integrated into a unified system.

Relay modules (e.g. 2, 8, or 32 channel units) can control loads such as pumps, valves, or lighting. Temperature and humidity sensors provide environmental monitoring. Soil sensors measure moisture and nutrient levels (NPK), enabling automated irrigation and fertilizing strategies.

Pulse counting modules allow integration of flow sensors, making it possible to monitor water usage as well as valve operation and failure. Combined with relay-controlled valves, this enables fully automated irrigation systems.

In home and lab environments, ModBus is frequently used to monitor HVAC systems, control of lights, control and monitoring of cooling loops, and monitoring of power consumption via smart meters. In laboratory setups, RS485-based systems are commonly used for devices like vacuum pumps and other slow control systems.

My Favorite Hardware Devices

My most favorite devices are:

• Waveshare USB to RS485 interface modules
• Waveshare 32 channel and 8 channel RTU relais modules, providing control of typical 230V and low voltage appliances
• Interface modules with digital inputs and outputs that I use to interface to anemometers
• X0 Pulse Counter modules to interface to flow sensors
• Chinese soil humidity and NPK sensors providing direct ModBus readout
• Indoor SHTC3 and outdoor SHT30 temperature and humidity sensors
• GPIO NPN and PNP boards
• The R3DCB08 interface board to DS18B20 onewire temperature sensors

Security Considerations

Raw ModBus/TCP has no authentication, no encryption, and no concept of access control. Exposing it directly to untrusted networks is inherently unsafe. ModBus/TCP should never be exposed outside of an isolated automation network or VLAN.

A gateway-based architecture enables:

• Wrapping ModBus communication in TLS, thus providing confidentiality
• Enforcing client authentication via mTLS
• Restricting access to specific devices and registers and thus providing monitoring only access paths, enabling separation of control and monitoring paths, aligning with CQRS architectures.

Unix domain sockets provide an additional option for local communication with the gateway, avoiding unnecessary network exposure.

Design Philosophy

A key design principle is to keep the physical and protocol layers simple, while moving complexity into controlled software layers. The RS485 bus remains deterministic, easy to debug and especially very easy and cheap to implement. All advanced features - security, multiplexing, abstraction - are implemented in user space, where they can be maintained, audited, and evolved without affecting system stability.

This separation leads to systems that are both robust and flexible, combining the reliability of industrial fieldbuses with the capabilities of modern software architectures.

Conclusion

ModBus, and in particular its RS485-based RTU variant, demonstrates that simplicity is not a limitation but a design strength. Its minimalism allows it to remain understandable, debuggable and implementable across a wide range of systems - from industrial installations to small-scale home and laboratory setups.

At the same time, modern software architectures impose requirements that the original protocol was never designed to address. Multiple independent services, distributed systems, and stricter security expectations fundamentally conflict with the single-master assumption and lack of built-in protection mechanisms.

By introducing a central gateway layer, these two worlds can be reconciled. The physical bus remains simple, deterministic and reliable, while higher-level concerns such as arbitration, access control, transport abstraction and security are handled in user space. This separation allows systems to scale without sacrificing the robustness of the underlying fieldbus.

In practice, this approach enables a wide range of applications - from garden irrigation and environmental monitoring to laboratory instrumentation and energy management - using inexpensive hardware and straightforward implementations.

Rather than replacing ModBus with more complex alternatives, it is often more effective to embrace its simplicity and complement it with well-designed software layers. This combination provides a powerful foundation for building reliable, secure, and maintainable automation systems.

References

• My own software:
  • GitHub repositories:
    • The gateway and client implementation
    • AVR ModBus client firmware
  • PyPi packages:
    • The modbus-gateway gateway implementation
    • The accompanying modbusgw-client Python client implementation
• Hardware:
  • MAX485 transceivers for custom components built around Atmel ATMega328P and ATMega2560 boards.
  • 4 pole cabling
  • Waveshare USB to RS485 interface modules
  • Waveshare 32 channel and 8 channel RTU relais modules, providing control of typical 230V and low voltage appliances
  • Interface modules with digital inputs and outputs that I use to interface to anemometers
  • X0 Pulse Counter modules to interface to flow sensors
  • Chinese soil humidity and NPK sensors providing direct ModBus readout
  • Indoor SHTC3 and outdoor SHT30 temperature and humidity sensors
  • GPIO NPN and PNP boards
  • The R3DCB08 interface board to DS18B20 onewire temperature sensors
• The ModBus specification
• The Hantek 6022BE USB oscilloscope for debugging purposes

XPPen tablets are, in many respects, surprisingly capable devices. The passive models in particular offer excellent value: precise pen input, good pressure sensitivity, and a solid overall build quality - at a fraction of the cost typically associated with professional drawing tablets. While the highly professional display-equipped variants can become quite expensive (in my personal opinion totally worth the price), the simpler models are almost irresistible for hobby and development setups.

Out of the box, the tablet was detected at the USB and HID level but none of the usual tooling - neither the base system utilities nor common open-source drivers - managed to produce usable input events. The root cause quickly became apparent: the device did not behave like a standard HID tablet. Instead, it required a vendor-specific initialization sequence and produced non-standard event packets that needed translation before they could be consumed by applications such as GIMP.

At that point, there were two options: abandon the device or solve the problem. Hating unresolved problems and having zero acceptance for unsupported hardware I choose the second route.

⚠️ TL;DR: This application is in no way associated with the official manufacturer and available on GitHub and installable via PyPi via pip install xppenfbsd. It solves my problem with the deco mini7.

• Why Wacom Works (and XPPen does not)
• Reverse Engineering the Protocol
• From USB Packets to Usable Input
• Integrating with X11
• Design Considerations
• Limitations and Future Work
• Conclusion
• References

Why Wacom Works (and XPPen does not)

To understand why this problem exists at all, it is useful to look at how Wacom devices are typically supported.

Most Wacom tablets implement (or at least closely emulate) pseudo-standard HID interfaces. On Unix-like systems, these devices are handled by the generic input stack and then enhanced by specialized drivers such as the X11 xf86-input-wacom driver. These drivers understand additional semantics like pressure, tilt, tool types, and button mappings—but crucially, they operate on top of a well-defined input abstraction.

In other words: Wacom devices speak a language the operating system and the X11 framework already understands.

XPPen devices, in contrast, often rely on vendor-specific extensions of the HID protocol, often requiring initialization sequences and custom report parsing on top of USBs HID device class. While they expose HID endpoints, they typically require an explicit activation or configuration sequence before they start producing any data. Even then, the reported data does not directly match standard input expectations and must be interpreted and transformed.

This difference explains why Wacom devices tend to work out of the box, while XPPen devices appear dead without proprietary applications.

Reverse Engineering the Protocol

The approach I chose was pragmatic: observe a working system and replicate its behavior.

A Windows test machine was set up with the official XPPen application performing the translation. Using USBPcap, I recorded the raw USB traffic generated while interacting with the tablet. This included device initialization, pen movement, pressure changes, and button events.

The resulting capture was then analyzed using Wireshark. By filtering out unrelated traffic and focusing on the relevant USB endpoints, it became possible to isolate the sequences responsible for device activation and continuous stylus data streaming.

To accelerate the decoding process, I first filtered the packets from the various phases (activation, data delivery, etc.) and used OpenAIs Codex to assist in identifying structural patterns and generating candidate parsers. While the initial suggestions required manual correction and validation, this significantly reduced the time required to move from raw captures to a working understanding of the protocol.

From USB Packets to Usable Input

Once the protocol was sufficiently understood, the next step was to reproduce the behavior on FreeBSD. Using libusb via a Python interface, I implemented a userspace daemon that performs three essential tasks:

First, it detects the tablet and performs the required initialization sequence. Without this step, the device remains silent.

Second, it continuously reads raw data packets from the stylus endpoint. These packets encode position, pressure, tilt, and button states in a vendor-specific format.

Third, it translates these packets into standard input events.

This translation layer is the core of the system. Instead of attempting to modify the kernel or introduce a custom driver, the daemon creates a virtual input device and re-injects events into the system again via an emulated HID device (uinput via the evdev compatibility layer). From the perspective of the operating system, this virtual device behaves like a normal stylus.

This design has several advantages. It keeps the implementation entirely in userspace and allows rapid iteration when refining the protocol understanding. It also makes the solution portable across systems that provide similar input injection mechanisms.

Integrating with X11

Once the virtual device is available, it can be consumed by the existing X11 driver stack.

A minimal configuration binds the generated event device to the stylus driver, allowing applications to interpret the input correctly. The exact device node depends on the system’s input enumeration and should ideally be discovered dynamically, but even a static configuration is sufficient for initial setups.

In my case the input device always appeared as event7 leading to the following configuration placed in /usr/local/etc/X11/xorg.conf.d/11-wacom.conf:

Section "InputDevice"
       Identifier "XP-PEN"
       Driver "wacom"
       Option "Device" "/dev/input/event7"
       Option "Type" "stylus"
       Option "USB" "on"
EndSection

Section "ServerLayout"
        Identifier "Default Layout"
        InputDevice "XP-PEN" "SendCoreEvent"
EndSection

At this point, the entire pipeline is complete: vendor-specific USB HID protocol > userspace translation > standard input events > X11 driver > application.

Applications such as GIMP can use the tablet without modification, including pressure sensitivity and stylus buttons.

Design Considerations

One might consider implementing this functionality as a kernel driver. However, this would not be a natural fit for the problem.

The primary task here is not hardware access but protocol translation. The kernel already provides mechanisms for exposing input devices and injecting events, and duplicating this logic in kernel space would introduce unnecessary complexity.

More importantly, from a design perspective, the kernel should remain as small and minimal as possible due to its privileged nature. Moving complex parsing and translation logic into the kernel increases the risk surface and makes debugging significantly harder. By keeping this layer in userspace, failures are contained, iteration is faster, and the system remains more robust overall.

Separating this functionality into a dedicated userspace process is therefore the correct architectural choice - not only from a modularity standpoint but also from a security perspective. It allows the protocol handling to evolve independently while relying on the kernel only for what it does best: providing stable and well-defined interfaces.

Additionally, the solution integrates naturally with system facilities such as device event frameworks, allowing automatic discovery and activation when the tablet is connected.

Limitations and Future Work

The current implementation focuses on a single device and a specific model family. Extending it to support multiple tablets or additional models would require further protocol analysis.

There is also room for improvement in device discovery and configuration, particularly in dynamically resolving input device paths and integrating more tightly with desktop environments as well as optimizing the runtime (adding daemonization support, etc.).

Nevertheless, the core functionality is complete: the tablet behaves like a native input device and can be used in real-world applications without noticeable limitations.

Conclusion

What started as a simple compatibility issue turned into a small reverse engineering project spanning USB protocols, driver behavior, and input subsystems. Interestingly, the entire process - from capturing USB traffic to having a fully working implementation - took essentially one sleepless night. This alone highlights how little effort would actually be required for a manufacturer to provide proper cross-platform support by simply releasing documentation (the manufacturers themselves could not cover all available operating systems in a feasable way due to QA, support, device generations, etc. but rely on the community).

The key insight is that many unsupported devices are not inherently incompatible - they are simply undocumented. With the right tools and a structured approach, it is often possible to bridge that gap. However, it also raises the question why this gap exists in the first place.

In this case, the combination of USB traffic capture, protocol reconstruction, and userspace event translation resulted in a fully usable graphics tablet on FreeBSD without relying on proprietary drivers. The fact that this can be achieved so quickly strongly suggests that publicly available protocol documentation would make such integrations almost trivial.

It is therefore somewhat difficult to understand why manufacturers so often hesitate to document their hardware interfaces. Instead of enabling broad compatibility across platforms, effort is invested into maintaining proprietary applications for a limited set of operating systems (they can never by themselves span the whole ecosystem of operating systems in a sane way). From a technical standpoint, publishing protocol specifications would significantly reduce duplicated effort, empower the community, and improve the longevity and usability of the hardware.

For anyone encountering similar issues, this workflow provides a practical path forward: observe, decode, translate, and integrate.

References

My implementation is available on GitHub and directly installable via PyPi

• The passive XPPen graphics tablet as well as it’s professional counterpart
• The manufacturer as well as the official XPPen application
• Wireshark and USBPcap
• pyusb as libusb frontend useable from Python
• The traditional xf86-input-wacom driver for Wacom and Wacom-compatible tablets
• OpenAIs Codex

Beyond the purely technical perspective, this is also a rather fascinating shift: these systems dramatically lower the barrier for creating artistic 3D content. They do not replace skilled artists - and realistically they will not in the foreseeable future - but they act as powerful amplifiers. For experienced artists they can accelerate iteration and ideation, while at the same time enabling people without strong artistic 3D skills to finally materialize their ideas, worlds, and characters.

Things get even more interesting when combined with modern image generation systems (like Stable Diffusion and similar approaches). With reasonably consistent prompt engineering, one can first generate coherent visual concepts and then lift them into 3D space. This opens the door to semi-automatically building consistent 3D scenes, asset libraries, or even entire worlds that share a unified style.

In my own workflow I primarily use these systems as a complement to traditional CAD. Most of my classical modeling work is engineering-focused (mechanical parts, devices, tooling), where parametric CAD (mostly in FreeCAD) is still the right tool, tools like the presented one are not capable of performing proper technical design at this stage of development. However, for non-technical, artistic or decorative objects - especially for 3D printing - these generative approaches are extremely valuable. They allow me to produce shapes and aesthetics that I would otherwise struggle to model manually.

In this article I will walk through a Python-based pipeline, the whole script is provided at the end of the article, that uses the Tripo3D API to generate, process, and export 3D models in a fully automated fashion. The focus is not just on getting a model, but on building a structured pipeline that produces pseudo-deterministic outputs, metadata, and multiple export formats suitable for further processing (e.g., CNC, 3D printing, simulation, or game engines).

• Why a Pipeline Instead of the Web UI?
• High-Level Pipeline Overview
• Core Design Ideas
  • Treat Everything as a Task
  • Metadata as First-Class Output
  • Deterministic File Naming
• Base Model Generation
• Discovering Parts
• Texturing Strategies
  • Whole-Model Texturing
  • Per-Part Texturing
• Export System
  • Unified Export Function
  • Supported Formats
  • Per-Part Export
• Optional Rigging
• CLI Interface
• Practical Observations
• Moving from Models to Reality
• Outlook
• Conclusion
• References
  • Useful Tools
• The Complete Script

Why a Pipeline Instead of the Web UI?

The web frontend is excellent for exploration, iteration, and interactive refinement as known from typical artistic workflows. However, it has a few limitations when used in technical environments. Typical limitations in technical environments include the lack of reproducible batch processing, limited control over export formats and intermediate steps, missing structured metadata capture, and poor integration into existing toolchains.

The script presented here addresses these issues by treating every operation as a task with persistent metadata, storing all intermediate results, supporting both text-to-model and image-to-model workflows, enabling per-part processing and exports, and enforcing deterministic naming and file organization. The key gain of this pipeline approach is automation and therefore scalability - there is no meaningful scaling without automation. Once the process is expressed as a pipeline, generating tens, hundreds, or thousands of assets becomes a straightforward extension rather than a manual effort. Additionally, using the API directly typically means usage-based billing instead of subscription-based time periods, which aligns much better with batch workloads and sporadic large-scale generation runs (taking on the order of ten minutes for a single high quality model without texture and rigging).

High-Level Pipeline Overview

The pipeline is structured into several stages:

1. Base model generation (text or image input)
2. Optional full-model texturing
3. Optional per-part texturing
4. Optional rigging
5. Full model export (STL / 3MF / etc.)
6. Per-part export

Each stage is implemented as an asynchronous task and stored together with its metadata.

Conceptually, the pipeline looks like this (violet is your data, green the mandatory step and yellow optional steps):

Core Design Ideas

Treat Everything as a Task

The Tripo API internally works with tasks. Instead of hiding this abstraction, the script embraces it.

Every step:

• Returns a task_id
• Is polled until completion
• Is serialized into a JSON metadata file

This is implemented via:

async def wait_success(client: TripoClient, task_id: str, label: str):
    task = await client.wait_for_task(task_id, polling_interval=2.0, timeout=None, verbose=True)
    status = str(getattr(task, "status", "")).lower()
    if "success" not in status:
        raise RuntimeError(...)
    return task

This pattern ensures robust error handling, since failures are detected explicitly and surfaced immediately, while also providing full traceability because every step is captured as a task with associated metadata. At the same time, it enables straightforward debugging and replay, as individual steps can be inspected, reproduced, or rerun without having to reconstruct the entire pipeline.

Metadata as First-Class Output

Instead of only saving meshes, the pipeline stores everything about each task:

async def save_task_metadata(task: Any, out_dir: Path, stem: str) -> Path:
    meta = task_to_dict(task)
    path = out_dir / f"{stem}.task.json"
    path.write_text(json.dumps(meta, indent=2, ensure_ascii=False))
    return path

This is extremely useful when comparing different parameter settings, debugging failed generations, and building higher-level automation on top of the pipeline. In addition you can resume operation with an model at each intermediate step this way.

Deterministic File Naming

Generated assets are renamed into a consistent scheme:

01_base.glb
02_textured_full.glb
03_textured_part__001__wheel.glb
06_export_full_stl.stl
08_export_part_stl__003__handle.stl

This is handled via:

dst = out_dir / f"{stem}.{model_type}{src.suffix}"

Together with sanitize_name() this guarantees filesystem-safe naming.

Base Model Generation

The entry point is either:

• text_to_model()
• image_to_model()

Example:

base_task_id = await client.text_to_model(
    prompt=args.prompt,
    texture=bool(args.texture),
    face_limit=args.face_limit,
    generate_parts=args.generate_parts,
)

Important parameters:

• texture: Generate textures directly if set to true.
• face_limit: Control mesh complexity by supplying the absolute face limit. Note that cost typically scales with this limit.
• generate_parts: Ask the model to segment the object.
• smart_low_poly: Useful for real-time applications - first generate a low polygon representation first and later expand to high polygon count.

Discovering Parts

One particularly interesting feature is automatic part discovery. Conceptually, you can think of the generative model not only producing a single mesh, but internally reasoning about the object as a composition of semantic substructures—wheels, handles, bodies, limbs, or decorative elements—very much like how a human would describe or sketch it. While the API does not expose this internal representation directly, traces of it appear in the task output, where parts may be listed explicitly or implicitly. By probing these structures defensively, the pipeline reconstructs a usable set of part identifiers.

This is powerful because it turns a monolithic generated mesh into something closer to a structured assembly. Once parts are identifiable, they can be processed independently: textured differently, exported separately, simplified or refined with different parameters, or even replaced downstream. In practical terms, this enables workflows that resemble classical CAD assemblies or game asset pipelines, but starting from a generative model rather than manual modeling.

What makes this particularly compelling is that it bridges a gap between purely artistic generation and engineering-style decomposition. Instead of treating the generated object as a static artifact, it becomes a manipulable system. For example, you can generate a complex object once, then iterate only on a specific component (e.g., retexturing just the “handle” or exporting only the “base” for printing). This selective control is where generative models begin to feel less like black boxes and more like cooperative tools that expose structure—imperfectly, but often sufficiently—to be integrated into real workflows.

Since the SDK does not clearly document where part names are stored, my script uses a defensive extraction strategy:

def discover_part_names(task: Any) -> List[str]:
    candidates = [task, getattr(task, "output", None)]
    ...

This scans:

• Raw task object
• Task output
• Serialized dictionary representations

The result is a list of part names such as:

["body", "wheel", "handle", "base"]

Note that this approach may break at any point in time. It just worked while I wrote the script but it relies on undocumented behaviour.

Texturing Strategies

Before looking at the concrete approaches, it is useful to briefly clarify what “texturing” actually means in this context. A generated 3D model typically consists of geometry (vertices, edges, faces) that define the shape, and separate surface information that defines how it looks. Texturing is the process of assigning image-based or procedurally generated information onto the surface of that geometry via UV mappings, effectively telling the renderer or downstream tool what color, roughness, metallic properties, and fine visual details each point on the surface should have. Without textures, most models look like uniform gray meshes; with textures, they become visually rich objects with materials such as wood, metal, fabric, or painted surfaces. In many pipelines this also includes PBR (physically based rendering) parameters, which control how light interacts with the surface. For purely functional workflows such as single-color 3D printing, however, textures are typically not required at all - the geometry alone is sufficient, and formats like STL intentionally ignore any surface appearance information.

There are two approaches implemented:

Whole-Model Texturing

texture_task_id = await client.texture_model(
    original_model_task_id=full_mesh_task_id,
    texture=True,
    pbr=True,
    text_prompt=args.texture_prompt,
)

This produces a single coherent material.

Per-Part Texturing

part_tex_task_id = await client.texture_model(
    part_names=[part_name],
    ...
)

This enables:

• Different materials per component
• Fine-grained control for asset pipelines

Export System

The export stage is surprisingly powerful and still very simple.

Unified Export Function

async def export_one_format(...):
    export_task_id = await client.convert_model(
        format=fmt,
        flatten_bottom=flatten_bottom,
        pivot_to_center_bottom=pivot_to_center_bottom,
        pack_uv=pack_uv,
    )

The flatten_bottom option modifies the geometry such that the lowest region of the model is projected onto a plane, effectively creating a flat base. This is particularly useful for 3D printing because many printers require stable contact with the build plate. Without a flat surface, models may require support structures, which increase print time, material usage, and post-processing effort. By flattening the bottom, the model can often be printed directly, improving adhesion and reliability.

The pivot_to_center_bottom parameter adjusts the coordinate system of the model such that its origin is moved to the center of the base. This is not just a convenience for slicers, but fundamentally changes how the model is positioned and manipulated in downstream tools. With this pivot, rotations occur around a physically meaningful point (the contact surface), and placement into scenes or assemblies becomes more predictable. For printing workflows, this often means the object appears correctly aligned on the build plate without additional transformations.

The pack_uv parameter operates on the texture coordinate space rather than geometry. It reorganizes the UV layout to make more efficient use of available texture space. This reduces wasted texture area, improves resolution of surface details, and is especially relevant when exporting to formats used in rendering or game engines where texture memory and quality are important.

Supported Formats

STL is the most basic and widely supported format for 3D printing. It encodes only the surface geometry as a triangle mesh and intentionally contains no information about colors, materials, or textures. This simplicity makes it robust and universally compatible, but also limits it to purely geometric workflows. The major advantage is that STL is extremely simple to implement in comparison to the other alternatives.

3MF can be seen as a modern replacement for STL. It supports not only geometry but also metadata such as colors, materials, multiple objects in a single file, and even printer-specific settings. For advanced printing workflows, especially with multi-material or color printers, 3MF is often the better choice.

GLTF and FBX are formats primarily used in rendering, simulation, and game engines. They support hierarchical scene structures, materials, textures, animations, and sometimes skeletal rigs. GLTF is designed as a modern, efficient, and open standard (often described as the JPEG of 3D), while FBX is older, widely supported, and deeply integrated into many commercial tools.

USDZ is a format designed for augmented reality ecosystems, particularly in environments like mobile devices. It supports compact packaging of geometry, materials, and animations in a way that is optimized for real-time rendering and distribution, making it suitable for AR previews or interactive product visualization.

Per-Part Export

For manufacturing workflows, splitting models is often essential.

async def export_parts_for_format(...):
    for idx, part_name in enumerate(part_names, start=1):
        stem = f"{base_stem}__{idx:03d}__{sanitize_name(part_name)}"

This results in:

• Individually printable components
• Clean naming
• Structured manifests

Optional Rigging

The pipeline also supports automatic rigging. In the context of 3D graphics, rigging refers to adding an internal skeleton (a hierarchy of bones or joints) to a static mesh, together with weights that define how each part of the surface deforms when those bones move. You can think of it as turning a rigid statue into something that can be posed or animated: bending an arm, rotating a head, or walking becomes possible because the mesh is now bound to this underlying structure. In practice, rigging also defines constraints, joint limits, and sometimes control handles that make animation easier to author. For many readers coming from CAD or printing workflows this concept may be unfamiliar, since purely geometric models are typically static; however, for game engines, simulation, or character animation, rigging is the essential step that converts geometry into something dynamic and controllable.

rig_task_id = await client.rig_model(
    rig_type=normalize_rig_type(args.rig_type),
    spec=normalize_rig_spec(args.rig_spec),
)

Supported rig types correspond to different anatomical or kinematic archetypes. A biped rig assumes two legs and typically two arms arranged around a vertical spine; it is the standard for humanoid characters and benefits from well-established conventions (for example Mixam -style skeletons), making retargeting animations straightforward. A quadruped rig is optimized for four-legged locomotion with a horizontal spine and coordinated gait cycles; it better captures weight distribution and natural motion for animals like dogs or horses, but requires different animation clips and controllers than bipeds. An avian rig introduces wings and often tail articulation, with joints arranged to support flapping, folding, and gliding; it is useful for birds or winged creatures but can be more complex due to additional degrees of freedom and coupled motions. A serpentine rig represents elongated bodies composed of many segments; instead of discrete limbs, motion is produced by propagating waves along the body, which is ideal for snakes or tentacle-like structures but requires spline- or chain-based control schemes.

Each choice encodes assumptions about joint hierarchy, constraints, and typical motion patterns. The advantage is that the resulting skeleton is immediately compatible with common animation tools and libraries for that class, enabling reuse of existing animation data (retargeting) and predictable behavior in physics or IK solvers. The downside is that mismatching the rig type to the geometry can produce unnatural deformation or require additional cleanup.

In animation and game pipelines this is extremely valuable because it converts a static mesh into a controllable asset that can be posed, animated, and simulated in real time. Engines rely on skeletal animation for efficiency (skinning on the GPU), blending between clips (like idle, walk, run), inverse kinematics for interactions (feet on ground, hands on objects), and physics-driven secondary motion. With a suitable rig, the same model can be reused across scenes and behaviors, integrated into state machines, and driven by gameplay logic turning a generated object into a fully interactive entity rather than a fixed piece of geometry with minimal or no additional manual work.

CLI Interface

My script exposes the described features, making it easy to integrate into other tools:

Example usage:

./tripo.py \
  --mode text \
  --prompt "small steampunk robot with tracks" \
  --texture \
  --generate-parts \
  --export-stl \
  --out ./output

Practical Observations

A few interesting takeaways from using this setup:

• The API becomes particularly powerful in the context of automated workflows. While the web UI remains the superior tool for artists - who can leverage it to iteratively refine and craft highly detailed results - the API excels when building fully automated pipelines, especially for users without artistic modeling skills. In these scenarios it enables reproducible generation, large-scale asset creation, and efficient scaling. Additionally, for many hobbyist use cases it can even be more cost-effective, as billing is typically based on actual usage rather than fixed subscription periods.
• In practice, the results are often surprisingly close to what one would intuitively expect from the prompt, frequently requiring little to no additional refinement.
• Part extraction is extremly useful but not always perfect. It generates consistent matching parts, still separated into individual objects.
• Texture alignment strongly influences visual quality, and in practice generating a consistent texture on the full model first and only then splitting it into parts tends to yield significantly more coherent results than designing or texturing each part individually.
• Export parameters (especially pivot and flattening) matter a lot for downstream workflows

Most importantly: once this pipeline exists, it becomes trivial to generate hundreds of assets in a reproducible way.

Moving from Models to Reality

If you want to move from a virtual 3D object to a physical object in a very easy way, just use the Cura slicer and a simple 3D printer like the Creality Ender3 V3 SE, the newer Creality Ender3 V3 KE featuring a ceramis heater or a more costly but capable multi filament printer like the Creality K2 Plus for multi color prints. Most models that are created for decorative purposes are directly printable without further modifications. If you need to perform further modifications use either the slicers limited editing capabilities or move on to Blender, which is an amazing tool with a steep but rewarding learning curve, before entering the CAD/CAM pipeline.

Outlook

There are several obvious extensions:

• Integration into CAD/CAM pipelines (like automatic toolpath generation)
• Coupling with simulation environments
• Closed-loop optimization (generate, evaluate, refine patterns)
• Integration into larger automated pipelines (for example built in n8n), combining idea and description generation, consistent image generation via diffusion models, 3D asset creation, assembly into animated scenes, and optionally fully automated video production workflows

From a systems perspective, this is where things get interesting: the moment generative models become just another pseudo-deterministic component in an engineering pipeline.

Conclusion

Using the Tripo3D API directly allows transforming 3D generation from an interactive tool into a programmable system component. By structuring the workflow into tasks, capturing metadata, and enforcing pseudo-deterministic outputs, the script provides a solid foundation for integrating generative 3D models into real engineering processes. What makes this particularly compelling is not just the ability to generate individual assets, but to embed generation into larger systems: once automated, the process scales naturally, turning what would otherwise be manual, creative effort into a reproducible and extensible pipeline.

At the same time, this does not replace traditional modeling or artistic workflows, but complements them. Artists will continue to achieve higher-quality and more refined results using interactive tools, while API-driven approaches enable entirely different use cases: batch generation, rapid prototyping, dataset creation, and integration into simulation, robotics, or manufacturing pipelines. In this sense, generative 3D becomes less of a standalone tool and more of a building block that can be composed with other systems.

For practical applications—whether generating printable objects, populating virtual environments, or building automated content pipelines—the combination of automation, usage-based cost models, and flexible export capabilities makes this approach particularly attractive. It allows engineers and technically inclined users to access domains that previously required significant artistic effort, while still leaving room for manual refinement where needed.

If you are already working with CNC, simulation, robotics, or content pipelines, this is where the real shift begins: the moment 3D asset generation becomes just another programmable step in a larger system, rather than a separate, manual process.

References

• The Tripo3D frontend
• The Tripo3D API

Useful Tools and Resources

• Learning Blender by Oliver Villar is a good resource to learn the basics of 3D rendering, animation and composition in Blender
• Cura slicer to turn models into machine paths
• 3D printers:
  • The cheap but high quality Creality Ender3 V3 SE or the newer Creality Ender3 V3 KE, being very beginner friendly and easy to repair and tune by yourself
  • A more costly but way more capable multi filament printer like the Creality K2 Plus for multi color prints.
• In Austria 3DJake is a very good and reliable source for filaments.

The Complete Script

The complete script is available as a GitHub GIST:

Introduction

Electric power systems are not monolithic. They are a layered composition of generation assets, each optimized for a different role in time and variability.

Baseload plants such as run of river, nuclear and coal power plants are designed to operate continuously at or near their nominal output. Their economics are dominated by high capital expenditure and very low marginal costs. These plants achieve their lowest levelized cost of electricity when running steadily. Any deviation like ramping up and down, partial loading or cycling reduces thermodynamic efficiency, increases mechanical stress and raises effective cost per kWh.

Load following plants like combined cycle gas turbines and reservoire hydro can adjust output on timescales of minutes to hours. They fill the gap between baseload and fast-response resources. Their marginal costs are higher than baseload but they are far more flexible

Peaking and balancing resources like open cycle gas turbines, pumped hydro storage and batteries are designed for rapid response on time scales of seconds to minutes (this cannot be achieved by load following- or baseload plants). They are essential for:

• Frequency regulation
• Contingency reserves (i.e. responding to sudden outages or sudden demands)
• Intra day balancing

These resources have the highest marginal cost and often low utilization. Their costs must be covered by fewer operating hours which means they are the most expensive resources on the grid, but they cannot be replaced by baseload or load following plants.

The timescales the grid operates on can be separated into four categories, which can roughly be separated into technical requirement (primary and secondary) and a economic stage (tertiary and day-head or intraday markets):

• The primary control happens on the scale of seconds. This is automatic frequency response and requires extremly fast ramping resources (network capacity, open cycle gas turbines, batteries)
• The secondary control happens on the scale of tens of seconds to minutes. Here centralized control happens to restore frequency and balancing areas. This still requires fast ramping resources (open cycle gas turbines, batteries and partially already combines cycle gas turbines and reservoire hydro).
• Tertiary control happens on the scale of minutes to hours and handles economic dispatch and reserve replacement.
• Day-ahead and intraday markets operating on the scale of hours to days. Here generation is sheduled and planned via economic rules (compareable to a stock market).

Why variability is expensive

For many people energy cost is only given by the amount of total energy (in kWh) consumed. But even more important is the shape of the load curve. A perfectly constant load can be served almost entirely by baseload generation. A highly variable load on the other hand requires more reserved held online, more ramping of thermal units, increased cycling (and thus wear and maintaneance costs) as well as higher reliance on fast response (expensive) assets. Rapid househols level switching aggregates across millions of users into steep net-load ramps. This leads to transformer and distribution stress (wear and infrastructure cost), voltage deviations and increased reserve requirements. These integration and balancing costs add up massively at higher variability levels[1]. The same total energy volume can cost a retailer far more[2] if the load shape is spiky/variable due to hedging costs, peak procurement and lost opportunities to use low-cost periods.

Frequency regulation markets see price spikes with higher variability or renewable penetration. For example increasing wind from low to 30% can raise regulation prices by 32% and doubling regulation requirements can increase them by 84%. Flexible resources (hydro, batteries) help mitigate this but the baseline cost of fast response is clearly elevated compared to steady operation[3]. Soothing of shifting load (i.e. the opposite of rapid switching) lowers system costs by reducing peak demand, deferring infrastructure and dampening price volatility. This can cut operational expenses, improve reliability and avoid billions in distribution upgrades[4].

Variable residential profiles are far more expensive to serve than steady commercial and industrial ones. In addition the injected variability by renewabeles like solar and wind also injects the same kind of cost, being the dominant cause of load variations at this point in time. Load variability also compresses grid margins through tighter reserves and more cycling of equipment[2].

Current Payment Models for Households

Today, most residential billing is based almost exclusively on total energy consumption:

This model assumes implicitly that all kWh are equal, regardless of when or how they are consumed. In reality on the other hand a kWh drawn during a stable low-demand period can be extremly cheap due to the very low marginal cost of baseload energy. The real system cost for households id increasing variability they inject (sudden powering on or off of an EV charger, a heat pump, an air conditioner, of induction stoves, plugged or unplugged chargers, switching on and off of devices, etc.), which forces the grid operator to deploy fast response resources (like gas peaking supply, hydro pump storage, hydrogen storage and batteries) that are far more expensive per kWh than baseload.

Some tariffs already introduce demans charges

However this only captures the maximum level, not the dynamics that cause the real massive costs.

Modern smart meters already sample power at intervals between 1 and 60 seconds (while only reporting on timescales of around 15 minutes). Therefore, the temporal structure of the consumption is already observable without changes to infrastructure, it is just not used in billing.

Proposed Billing Model

In the following section we extend the billing model to include not only total energy but also the temporal variation of the load.

Let

• $P(t)$: instantaneous power (kW)
• $T$: billing interval (for example $T=720h$ for one month)

Then the model includes two key quantities:

As in the traditional model the energy consumption, which can be kept very small or even zero reflecting the very low marginal cost of steady generation:

The second term is the load variation:

This is the total variation of $P(t)$. Every time one turns a 2 kW load on and off again, $V$ increases by $4 kW$, regardless of how long it was on. Rapid cycling drives $V$ up dramatically, steady load barely moves it. This term is independent of duration and reflects how agressively the grid is stressed.

A even more physical but slightly more complicated model would incorporate the frequency response of the system (i.e. provide frequency cost):

Optional Higher-Order Penalty

To penalize rapid fluctuations even more the implementation of high pass filtering and the square of the derivative can be used:

Such a term would in particular hit high frequency switching devices without proper filtering.

The Total Bill

The monthly total bill $B$ is given by

The factors are given by:

• $c_E$: The energy price, which could be set very low (even in the range of 1 Cent per kWh).
• $c_V$ which is the variation rate measuring the load swings. This layer would recover the balancing and infrastructure costs and has to be tuned so the utility recovers its fixed and control costs.
• $c_D$ is an optional demand rate charging for the maximum available power. I personally would not introduce this.
• $F$ is a fixed customer charge handling the provisioning of metering infrastructure, the paperwork and support availability.

Discrete Implementation

Due to the discrete sampling of the smart meters in units of $\Delta t$ the real implementation would of course be discrete:

Conclusion

This system could be implemented easily with existing smart-meter infrastructure available in many parts of Europe and would give the customer a clear financial incentive to smooth their load. Such a system would provide behavioural incentives reflecting the real cost structure:

• Encouraged behaviour:
  • Slow ramping of loads (soft starting, not switching larger loads, etc.)
  • Buffering via Batteries or thermal storage
  • Sheduling of flexible loads
• Discouraged behaviour:
  • Rapid on/off cycling of loads
  • Synchronized switching across households
  • High frequency load fluctuations
  • Non conforming devices leaking high frequency switching noise towards the grid

Industrial consumers already face complex tariffs reflecting demand peaks, power factor and time-of-use. Residential users currently do not, despite contributing significantly to variability. This model aligns individual cost with system impact, making pricing more economically efficient and fair. Importantly this model is immediately implementable using existing infrastructure.

Appendix: How to Reduce Your Load Variation in Practice

If billing starts to reflect not only how much energy is consumed but also how it is consumed, then reducing load variation becomes both economically and technically meaningful. Fortunately, many strategies are already available with relatively simple tools.

Avoid Sudden Large Load Steps

The main driver of load variation is not energy usage itself, but rapid changes in power. Typical problematic patterns include:

• Switching high-power devices abruptly (like EV chargers, heaters, induction stoves, etc.)
• Multiple devices turining on simultaneously
• Thermostatic systems oscillating between full on and off

Whenever possible:

• Prefer devices with soft-start or ramping behaviour
• Avoid manually switching multiple high loads at once
• Stagger the activation of large consumers by a few seconds

Even small delays between devices can significantly reduce aggregated grid stress.

Use Smart Scheduling for Flexible Loads

Here it gets more interesting and technical. Some loads are inherently flexible in time:

• EV charging
• Dishwashers
• Washing machines
• Water heating

Instead of starting them immediately, shift them to periods where your total load is already stable. A simple strategy is:

• Only start new loads when current consumption is low and stable
• Avoid starting devices during existing ramps (e.g. while heating systems are activating)

Home automation systems can implement this with simple rules.

Decouple Consumption from Instantaneous Demand (Buffering)

This could be done in various ways:

• Electrical buffering: Small home batteries can absorb rapid changes, even modest capacities (1-5 kWh) are sufficient to smooth short spikes. Charging and discharging can be controlled to keep grid-facing power nearly constant.
• Thermal buffering: Heat pumps and buffer tanks and water boilers storing thermal energy. These systems can run at constant power and store energy for later use, transforming spiky load into a smooth baseline.

Limit High-Frequency Switching

Many modern devices (especially cheap power electronics) introduce rapid fluctuations. These include poorly filtered switching power supplies, fast PWM-controlled heaters and cheap motor controllers. While individually small, these effects can accumulate. To mitigate those effects prefer higher quality devices with proper filtering, add input filtering (LC filters) where applicable and avoid unnecessary rapid on/off control loops. In a billing model sensitive to variation, these small fluctuations may become economically visible.

Coordinate Loads Within the Household

A household is not a single device but a combination of dynamic systems. With coordination the total load variation can reduced by ramping devices up and down in a coordinated fashion, keeping the total power approximately constant. Especially coordinating devices like EV charging, heating or washing can drastically reduce load variations.

Monitoring

Without monitoring one does never know what to optimize - or if there is something to optimize. Optimization can be done very easily via

• Digitally readable smart meters providing time resolved readings
• Energy monitors providing the total consumed energy
• Logging systems based on digitally readable smart meters and switch status of home automation systems.

This allows to visualize the load profile $P(t)$ and the rate of load changes $\Delta P(t)$. When looking at the data inefficiencies often become immediately obvious.

Shift Thinking from Total Energy to Smooth Power

This is most likely the hardest one since most of us learnt at school to turn of unneeded devices to conserve power. The useful mental model is that the grid prefers a constant request over multiple small bursty ones. This shift in thinking would lead to fewer peaks, less cycling and lower system cost.

References

• [1] From baseload to peak: Renewables provide a reliable solution
• [2] Volume and Shape: Understanding the cost of electricity and the value of flexibility
• [3] Prices in Frequency Regulation Markets: Impacts of Natural Gas Prices and Variable Renewable Energy
• [4] Demand response can play a vital role in ensuring grid reliability and dampening price volatility in wholesale electricity markets

This article just describes very brieflyvery briefly a flow that was really used this way, yielding a workable library, it is neither sugar coated nor exaggregated. Keep in mind this was a small scale project so it was a very easy task for the agent to perform. Note that the conversations with the agent are trimmed down, the shown snippets should just provide a rough idea.

• Bootstrapping the collaboration and designing the architecture
• Iterating with real silicon in the loop
• Debug automation without babysitting
• Future-proofing with formal methods
• What are the benefits of hardware in the loop
• References

Bootstrapping the collaboration and designing the architecture

We started by writing AGENTS.md as an executable contract. It spells out that Codex must keep Timer0 free unless an optional sysclock is enabled, ship UART ISRs with ring buffers, and update every artifact (AGENTS, DESIGN_DOCUMENT, TODO, user docs) whenever reality changes. That file also pins the toolchain (avr-gcc/avr-libc/binutils/avrdude/GNU make via gmake), the target MCUs (ATmega328P utilizing UART0 and ATmega2560 utilizing UART0 and UART1 even though only the 2560 rig was in the loop for now), and communication habits (line-referenced file mentions, short status updates, immediate blocker escalation). Having that behavior encoded up front was the equivalent of onboarding a senior engineer in writing: every later decision referenced back to it, and Codex kept it fresh whenever constraints shifted.

“You are a design architect and software developer. We are going to implement a ModBus Slave on an ATMega microcontroller. First we are creating a design document till all open questions are resolved and we have specified all technical details. We do this in a back and forth conversation, you do not take decisions. Present open questions and provide suggestions. The user decides which decissions to take. You do not decide yourself. Follow the design rules from DESIGNRULES.md when writing the architecture document. In the first stage we are writing docs/DESIGN_DOCUMENT.md as a detailed technical design document and docs/TODO.md as an [ ] open, [x] done, [-] rejected ToDo list that you keep up to date all the time. After we finished designing you are going to implement the project according to the ToDo list using avr-gcc, avr-libc, binutils and avrdude. You build using gmake. Use no other tools. You can flash the program to the microcontroller using gmake flash and access the serial port via /dev/ttyU0 as well as the RS485 bus on /dev/ttyU1. First draft your AGENTS.md that explains your role. Only ever edit ~/projectdirectory.”

With the agent contract in place we drafted docs/DESIGN_DOCUMENT.md. Codex drove that conversation like an architecture review. It enumerated which registers must be memory-backed, how MAX485 control pins are abstracted, what the ISR boundary looks like, and even future knobs (Timer0 vs Timer1 tick sources). Whenever ambiguity popped up - How should holding register 255 trigger EEPROM commits? Should UUIDs live in callbacks or static blocks? - Codex paused, listed options, and asked for confirmation before touching code. That high-velocity Q&A mirrored how a human architect would unblock a team, just without the context loss that happens when humans juggle too many requirements.

“Start writing the ModBus RTU slave architecture for ATmega2560 + MAX485 attached to UART1. Document UART buffers, gap timing and the reset behavior before you write firmware. Also honor proper timeout processing. We support input registers, coil outputs, holding registers and output registers. We need write single, write multiple, read single and read multiple commands for those registers. We implement a set of fixed registers for the device address (…), baud rate (…) as well as an UUID based identity register to identity the device and the firmware. Later we are also going to implement an RS485 capable bootloader so we are able to flash the device via the RS485 bus (this will be an independent project)”

To keep implementation straight we used docs/TODO.md as both a kanban lane and a verification ledger. Items ranged from API scaffolding to tests with the single board RS485 link. The checklist style (explicit [ ] vs [x]) made it trivial to see which capabilities still needed either implementation or bench validation. Parallel to that we maintained KB/index.md, a placeholder knowledge base meant for any external ModBus or AVR timing references Codex might have had to fetch. Even when the KB stayed empty, the scaffolding reminded us that Codex could, on demand, go out to public documentation, store PDFs or markdown summaries under KB/, and cite them later (this was left out in the presented instructions above).

This is a pretty standard approach to use an coding agent. One will spend usually between an hour and half a day on writing a design document this way, depending on the project scope, talking back and forth with the agent to resolve open questions, discuss feasability as well as pros and cons and take decisions. This phase feels like the meetings with human engineers during the design and architecture phase, though being way more productive and experiencing less friction and social stress. And in contrast to the human world dumb ideas always get harsh feedback.

Iterating with real silicon in the loop

Once the architecture felt solid Codex shifted into coding mode. It produced the UART/RS-485 hardware layer, the ModBus core parser, and the register handlers in digestible steps, performing unit tests as it went. It always followed the same pattern: update the TODO, write code, run gmake, run unit tests, flash the ATmega2560 and immediately exercise register read and writes over the physical bus via on-the-fly written pyserial scripts. Because the MAX485 driver enable lines and UART ISRs were part of the same repo, Codex could inject temporary instrumentation (extra GPIO toggles, debug prints gated behind #ifdef MODBUS_DEBUG, CRC probes, etc.) without breaking the contract, test the hypothesis on hardware, and then strip the probes again - all inside a single loop. This fitted the printf style debugging that junior engineers often use.

“Perform a sequence of reads and writes into and from the registers and dump debug messages on the AVRs serial port. Interact with the device via the RS485 bus and inspect validity of the reaction on the serial port. Create valid and invalid requests.”

A concrete example came up when we noticed that writing a new device ID took effect immediately, which violates ModBus expectations. Codex traced the bug by flashing diagnostic builds that logged both the pending and active IDs after every frame. It then restructured modbus_core.c to separate PendingConfig from ActiveDeviceId, staged new values in RAM only, and confirmed on the rig that the slave still answered on the old ID until the reset magic 0xAA55 forced a reboot. That entire investigation - code change, compilation, flashing, scripted ModBus transactions, and regression verification - ran autonomously while we observed the terminal output.

“I see a device ID bug. The device seems to automatically apply directly after writing into the respective register. Reproduce the bug, capture pending vs active IDs over UART, and refactor to fix so we only apply the change after rebooting the device via reboot magic.”

Because hardware was always in the loop Codex could also stress scenarios that usually wait for the lab: half-duplex turnaround timing, bursts, deliberate line silence to test the 1.5/3.5 character gap watchdog, and watchdog-induced resets. Whenever a test exposed a weakness, Codex modified the source (sometimes inserting extra assertions or statistics counters), rebuilt, and reran the scenario minutes later. There was no “hand code to a human to flash” delay, so iteration speed approached software-only TDD despite touching real silicon.

Debug automation without babysitting

The workflow never depended on someone manually driving a serial console. Instead we kept lightweight Python and shell utilities around to spray ModBus frames, capture responses, and reset boards via the watchdog harness. Codex could call those scripts, parse their output, and decide on the next change without waiting for human prompts. That made higher-level experiments feasible: for example, sweeping coil write bursts across dozens of registers while monitoring current draw, or verifying that ring buffer overruns stay cleared even when the main loop intentionally starves ModBusService() for a few milliseconds.

This autonomy extended to documentation and guardrails. Any time the behavior changed, Codex updated AGENTS, the design doc, TODO, and the example app notes. It also would have been trivial to pull protocol specs from the public internet, normalize them into KB/*.md, and cite them inline - handy when juggling RTU timing or EEPROM endurance data. The same mechanism can ingest errata sheets, ModBus application notes or even oscilloscope captures dumped via the pylabdevs devices, giving future sessions instant context.

Future-proofing with formal methods

One of the underrated perks of this setup is how easily it can grow into formal assurance. The same Codex agent that compiles and flashes code can also emit ACSL annotations for critical routines. Feed those annotations plus the source into Frama-C, and you gain static guarantees (no runtime errors, preserved invariants) before the bits ever hit flash. Coupling Frama-C proofs with hardware-in-the-loop regression runs lets you blend mathematical confidence with empirical validation, a combination that is usually out of reach for small embedded teams.

What are the benefits of hardware in the loop

Putting Codex inside the hardware loop changed the economics of firmware work. Instead of queuing questions for a future lab slot, we answered them immediately with the actual boards. Instead of hoping a human remembered every constraint, we encoded expectations in AGENTS and the design doc so the assistant could enforce them relentlessly. Instead of deferring documentation, we kept the narrative up to date as part of every change. Most importantly, the assistant never tired: it could keep iterating—tweaking ISR latency, adjusting ModBus timing, mutating register maps, or running soak tests—long after a human would have walked away.

If you are building microcontroller firmware with tight loops, shared peripherals or embedded networks, wiring Codex into your hardware bench gives you the confidence of continuous validation with the speed of scripted development. Whether you need a minimally guided debugging partner or a fully autonomous regression runner, the same ingredients apply: define the agent contract, capture the architecture, keep TODO and KB artifacts honest, and hand the assistant access to your toolchain plus your boards. The result is a development flow that feels both methodical and fast—exactly what embedded projects need.

References

• OpenAI codex
• RS485 communication using Atmel ATMega328P
• Used utilities:
  • Arduino Mega 2560
  • Waveshare USB to RS485 converter
  • MAX485 breakout board
  • Hantek 6022BE USB Digital Oszilloskop
• The resulting ModBus RTU slave toolkit for AVR

A much more accurate picture emerges if we stop thinking about LLMs as knowledge containers and instead see them as pattern engines that have learned how ideas, statements, equations, and explanations tend to transform into each other.

To understand why this distinction matters, we need to look at what actually happens when a language model produces an answer.

• Why an LLM Is Not a Knowledge Database
• Even Scientific Formulas Are Patterns
• What Actually Lives Inside the Model
• Why External Knowledge Systems Are Necessary
  • Vector Retrieval: Finding Similar Text
  • Graph Retrieval: Recovering Structure
• Why This Matters

Why an LLM Is Not a Knowledge Database

A classical knowledge system stores information explicitly. A database entry might look like this:

(country="Austria", capital="Vienna")

If we ask the system for the capital of Austria, it simply performs a lookup and returns the stored value.

A language model does something fundamentally different. It does not retrieve a stored record. Instead it predicts the most probable continuation of a text sequence based on statistical patterns learned during training.

At the outermost level, the system produces tokens by sampling from a probability distribution over possible continuations. Mathematically this is often written as

which describes how likely a particular token is given the tokens that came before it. During training the model learns this probability distribution by adjusting its internal parameters to minimize a loss function. In practice this is typically the cross-entropy loss between the predicted probability distribution and the actual next token in the training data. Gradient descent is then used to update billions of parameters so that the model gradually becomes better at predicting the next token in a sequence. Over many training iterations this process shapes the internal representations of the network so that useful linguistic and conceptual patterns emerge. The model is therefore not explicitly programmed with rules or facts. Instead its internal structure is optimized purely through exposure to vast amounts of text.

However, this formula alone can be misleading. If language models were only performing classical statistical estimation over token frequencies, they would behave much more like sophisticated n-gram models or Bayesian predictors. Such systems can reproduce local statistics, but they cannot generalize well and they cannot discover deeper structures in language.

The crucial difference is the neural network itself. A modern transformer model contains many layers of nonlinear transformations and attention heads that dynamically route information across the sequence. These mechanisms allow the network to detect relationships between words, concepts, and symbolic expressions that may be far apart in the text.

The key mechanism that enables transformers to capture long-range relationships is called attention. Instead of processing tokens strictly one after another like earlier neural networks, the model dynamically decides which parts of the input sequence are relevant when interpreting a particular token. In practice each token generates queries that search for relevant keys among all other tokens in the sequence. The resulting weighted combinations of information allow the model to connect words that may be far apart in the text. This mechanism is what allows modern language models to track references, follow arguments across paragraphs, and relate mathematical symbols to explanations appearing elsewhere in the context.

The probability distribution above is therefore only the final sampling interface of the system. Behind it lies a very large nonlinear pattern recognition machine. During training, the neural network learns internal representations that capture regularities in language, mathematics, explanations, and reasoning patterns. Crucially, what the model learns are patterns, not explicit facts. The training process does not insert statements like “Vienna is the capital of Austria” into a memory structure. Instead it adjusts billions of parameters so that certain regions of a very high‑dimensional representation space correspond to recurring conceptual relationships observed in the training data.

When the model answers a question like “What is the capital of Austria?” it does not retrieve Vienna from a memory table. Instead the network transforms the prompt through these learned representations until the sequence of tokens corresponding to the word “Vienna” becomes overwhelmingly likely under the learned patterns. In practice most tokenizers do not even operate on full words, but on sub‑word fragments, so the model is assembling the answer piece by piece according to the patterns it has learned.

The difference might sound subtle, but it has deep consequences. Databases store facts explicitly. Language models instead learn structures in which certain statements naturally follow from certain contexts.

Even Scientific Formulas Are Patterns

This becomes even clearer if we look at something that appears to be very precise: a physics equation.

Consider the equation

At first glance it might seem as if the model simply memorized this formula, much like a bad student who has learned a line from a textbook without really understanding what it means. But that interpretation is misleading. The equation itself is not the knowledge. It is only a symbolic representation of a deeper concept: force describes the change of momentum over time.

To see why this matters, it helps to compare two kinds of understanding. A student who memorizes the formula $F = \frac{\mathrm{d}p}{\mathrm{d}t}$ may be able to reproduce the symbols on an exam, but the expression itself is just a sequence of characters to them. A physicist, in contrast, does not think primarily about the letters or the notation. For them the equation activates a much richer conceptual structure.

When a physicist sees this expression, it immediately connects to a broader pattern of how the universe behaves. Ideas about dynamics, momentum, and interaction come into play. In modern physics this also touches deeper principles: symmetries of space and time, conservation laws, and the structures described by symmetry groups. The equation is only one compact way of encoding these relationships. Mathematics is essentially the language we use to describe those patterns precisely.

In other words, the formula is not an isolated statement. It is a symbolic gateway into a network of concepts describing how physical systems evolve.

An LLM learns something somewhat analogous on the linguistic level. It does not store the equation as a static mathematical object. Instead it learns the linguistic and symbolic patterns connecting force, momentum, change, Newtonian dynamics and the notation used to express those relations.

Because these models are trained on an enormous portion of written human knowledge, they are exposed to a vast range of explanations, arguments, analogies, and reasoning styles. What the network therefore absorbs are not individual statements, but recurring patterns of thinking: how humans explain physics, how they derive formulas, how they reason about systems, and how concepts connect to each other. Over time the training process shapes a high‑dimensional representation space that reflects many of the cognitive patterns present in human discourse.

When it writes $F = \frac{\mathrm{d}p}{\mathrm{d}t}$ it is reproducing a learned mapping between these representational forms. The model has internalized the pattern linking them, not the equation as an isolated fact.

This is why language models are surprisingly good at rewriting equations into explanations and explanations back into equations. Because they have learned the structural patterns connecting these representations rather than memorizing individual statements, they can often generalize those patterns to new situations. When faced with a problem they have never seen before, the model can still apply similar reasoning structures it encountered during training, which is why LLMs are sometimes capable of solving entirely new problems that were never explicitly present in their training data.

What Actually Lives Inside the Model

Inside the neural network there are no explicit facts, rules, or entries. Instead there is a high‑dimensional parameter space that encodes regularities of language, concepts, and symbolic relations. One can think of this space as a vast hyperspace in which related meanings, explanations, equations, and narratives occupy nearby regions. During training the model gradually shapes this hyperspace so that patterns that frequently appear together in human communication become geometrically aligned.

Importantly, the intermediate layers of the neural network learn these structures largely on their own during training. No human explicitly tells the model which internal neurons should represent which concepts. Instead the network discovers useful internal patterns because doing so improves its ability to predict the next token. These internal features therefore do not necessarily correspond to the neat conceptual categories humans might use to organize knowledge. The model may capture many subtle correlations present in the training data — sometimes meaningful conceptual relationships, sometimes statistical associations that humans would not consciously describe.

Modern architectures are also intentionally designed to prevent the model from simply memorizing the training data. Neural networks contain narrow information pathways and compression steps that force the system to represent information efficiently. If the network could simply memorize every sentence it saw, it would fail to generalize. This problem is known as overfitting: the model would reproduce training examples perfectly but perform poorly on new inputs.

Because of these architectural constraints, the model has little choice but to learn reusable patterns instead of storing individual facts. In other words, the structure of the network itself encourages the discovery of general relationships rather than direct memorization.

A useful mental model is that the weights define a transformation landscape. Certain prompts push the internal state of the network into regions where specific continuations become highly probable. If a prompt mentions “capital” and “Austria” the internal representation of the prompt moves into a region of this hyperspace where the continuation corresponding to the word “Vienna” becomes highly probable, most likely activating zones representing cities, capitals, governmental systems, vacations, etc. on the way. But this is not a discrete memory. It is more like an attractor in a probability field.

One of the most striking discoveries in modern AI research is that the capabilities of these models follow relatively predictable scaling laws. As the size of the neural network, the amount of training data, and the available computation increase, the performance of the model improves in a smooth and often surprisingly regular way. Larger models tend to discover richer internal representations and capture increasingly subtle patterns in language and reasoning. At certain scales new capabilities appear that were not obvious in smaller systems. This phenomenon, sometimes described as emergent abilities, is one reason why very large models can perform tasks that smaller models struggle with, even though they are trained with the same fundamental objective of next-token prediction.

The model therefore behaves less like a database and more like a system that has learned how concepts tend to follow each other.

Why External Knowledge Systems Are Necessary

Because LLMs operate through pattern reproduction rather than fact retrieval, they are not ideal sources of authoritative knowledge.

The model can generate extremely plausible statements that were never true in the first place if those statements match the typical reasoning or explanation patterns found in human language. In other words, the model can produce answers that sound exactly like something a knowledgeable human might say even when the underlying statement is incorrect or incomplete.

Interestingly, something very similar happens in human thinking. People sometimes believe they understand a topic because they can reproduce the usual explanation pattern associated with it. Only when they try to verify the statement or derive the result do they discover that their understanding was incomplete. In that sense, the failure mode of LLMs is not entirely foreign - it mirrors a common limitation of human reasoning as well.

For applications where factual accuracy matters, the model therefore needs access to external information sources. This is the motivation behind Retrieval Augmented Generation (RAG).

In a RAG system, the language model does not rely solely on its internal patterns. Instead, it receives relevant documents retrieved from an external knowledge base and reasons over them while generating the answer. The architecture then becomes conceptually simple. A retrieval system finds relevant information, and the language model acts as a reasoning engine that interprets and synthesizes that information.

This division of labor mirrors how humans work. A scientist does not memorize every paper ever written. Instead they consult references and then reason about the information they find. Humans routinely perform their own form of retrieval augmented reasoning: they look up articles in encyclopedias or on Wikipedia, consult textbooks or lexica, and use formal tools such as mathematics to verify whether a statement is actually correct.

Another remarkable property of large language models is in-context learning. Even though the models weights remain fixed after training, it can temporarily adapt its behavior based on examples provided directly in the prompt. If a prompt includes several demonstrations of how a task should be performed, the model often continues the pattern correctly for new inputs. In effect the model performs a form of short-term learning inside the context window. The internal representations inferred from the prompt guide the generation process without requiring any permanent update to the model parameters. This ability further illustrates that the model operates by reproducing patterns rather than retrieving stored rules.

Vector Retrieval: Finding Similar Text

Most RAG systems rely on vector embeddings to retrieve relevant documents.

Text passages are converted into vectors in a high‑dimensional space. When a user asks a question, the system computes the embedding of the query and searches for passages whose vectors are nearby.

What does “nearby” mean in this context? During training, embedding models learn to place pieces of text that are used in similar contexts close to each other in this space. The geometry of the space therefore begins to encode meaning. Sentences that talk about related ideas tend to end up in neighboring regions, even if they use different words. At the same time, this space often captures stylistic and rhetorical patterns as well. Technical explanations cluster differently from casual descriptions, and scientific writing occupies different regions than narrative text. You can spot the difference in the vector embeddings of this blog

In other words, the high‑dimensional embedding space simultaneously encodes aspects of semantics, style, and conceptual associations. Similarity between two vectors is typically measured using cosine similarity or related metrics, which essentially check whether two vectors point in a similar direction in that space. Performing a nearest neighbor search in this space yields all semantically similar statements inside the knowledge base (I use this for example for the suggested articles on the bottom of every page)

It is also worth noting that embedding vectors themselves are often generated using transformer models very similar to the LLMs that later consume them. These models learn to map text into this geometric representation so that related meanings occupy nearby regions of the hyperspace.

This approach works very well when the answer is contained in text that is semantically similar to the query.

However, similarity is not the same as structure.

Many difficult questions depend on relationships between entities rather than simple textual similarity.

If the relevant information is spread across multiple documents that describe different parts of a system, vector retrieval may return fragments that are individually related to the question but fail to capture how those fragments connect to each other.

Typical backends are databases like pg_vector in PostgreSQL or database systems like ChromaDB.

Graph Retrieval: Recovering Structure

Graph‑based retrieval addresses this limitation by representing knowledge as a network of entities and relationships. Instead of storing only text chunks, the system builds a graph where nodes represent concepts or objects and edges represent relationships such as causation, dependency, or hierarchy. When a query arrives, the system retrieves a relevant subgraph rather than a collection of independent text fragments. This explicit structure makes complex reasoning easier. If the system already knows that

component A depends on component B

and that

component B failed during a redesign

then the reasoning path connecting those events is already encoded in the graph.

The language model can then focus on interpreting the structure rather than reconstructing it from scattered prose. The model can traverse such graph structures iteratively: it can follow relationships from one node to the next, interpret the intermediate results and then decide which connections to explore next. By repeating this process over multiple steps, the model can perform multi‑hop reasoning across the graph.

The relationships stored in such graphs often resemble what is known in the semantic web world as RDF triples. These triples represent knowledge as simple subject‑predicate‑object statements, for example:

("Vienna", "is capital of", "Austria")
("Electron", "has property", "charge")
("Component B", "failed during", "redesign")

When many such triples are connected together they form a rich knowledge graph that captures relationships between entities. Graph databases such as Neo4j are commonly used to store and query these structures efficiently.

Interestingly, the extraction of these triples from unstructured text is often performed using the same kind of transformer models discussed earlier. LLMs can read documents, identify entities and relationships, and convert them into structured graph representations that can later be used for graph‑based retrieval and reasoning.

Why This Matters

The deeper lesson is that modern AI systems work best when we separate three roles.

• External systems store information.
• Retrieval systems locate relevant pieces of that information.
• Language models reason about the retrieved material and transform it into explanations, summaries, or decisions.

When these components are combined, the language model becomes something closer to a cognitive engine operating on a structured information environment. In practice this means that asking a standalone LLM factual questions without any grounding is often the wrong way to use the technology. The model itself is not designed to be the authoritative storage location of knowledge. Its real strength lies in interpreting patterns, combining ideas, performing reasoning steps, and synthesizing information once the relevant data has been supplied by retrieval systems such as RAG or GraphRAG.

Interestingly, the much‑discussed phenomenon of “hallucinations” is closely related to this capability. The same mechanism that allows the model to generate plausible statements beyond its training examples is what enables creativity and generalization. If the system were restricted to reproducing only statements that appeared verbatim in its training data, it would behave like a database lookup and would be incapable of solving new problems or combining ideas in novel ways.

In that sense hallucinations are not purely a bug, they are a side effect of the very property that makes these systems powerful. When they appear problematic, it is often a sign that the system is being used without proper grounding. Once external retrieval systems provide the factual information and the LLM is used primarily for reasoning and interpretation, the architecture begins to resemble a much more robust cognitive system.

This same mechanism can deliberately be used as a feature. Because the model can generate plausible variations and speculative ideas, it can be used to explore creative solution spaces. If the system is connected to verification tools - for example a mathematical proof assistant, a symbolic solver, or a simulation toolkit - the model can propose candidate ideas while the external tool checks whether they are actually correct. In this way hallucination becomes a generator of hypotheses while external tools provide validation. This pattern is increasingly used in research systems where LLMs propose conjectures, derive candidate formulas, or sketch solution paths which are then verified automatically.

It is therefore helpful to think of the LLM as a machine that has learned how ideas move - how arguments unfold, how explanations are constructed, and how pieces of knowledge connect to each other. And once this reasoning engine is connected to reliable sources of information and verification tools, its ability to analyze, explore, and synthesize knowledge becomes extraordinarily powerful.

Most examples either:

• Come in video format and are thus not very accessible.
• Use the existing SELECT, INSERT and UPDATE methods which are restricted with respect to arbitrary table access, arbitrary ordering and similar ideas
• allow completely free SQL (which is dangerous)
• restrict things so much that the agent becomes useless.

The pattern described here works reliably in practice and allows an agent to query any allowed table in measurement databases flexibly while still maintaining proper boundaries.

• Basic Idea
• Create a Minimal Database User (Least Privilege)
• Write a Strict Tool Description
• Defining Parameters
• Building the SQL Query
• Always Add a LIMIT
• Security Rules You Should Actually Follow
• Useful Extensions
• Conclusion

Basic Idea

The concept is simple:

• Attach a MySQL Tool Node to an agent node.
• Configure the connection statically (specify server, username, password and database)
• Give the tool a detailed description so the model understands what tables exist, in best case informing it about the meaning of the columns.
• Build the SQL query dynamically in Execute SQL mode using JavaScript expressions. Apply whitelists to restricted names like table names or to perform parameter validation. No matter what you try, there is no other safe method than whitelisting.

Never allow the LLM to generate raw SQL if your data should not be corrupted (though it is interesting to see how an LLM handles it’s own database when it gets arbitrary access, just make sure to sandbox the environment properly).

Create a Minimal Database User (Least Privilege)

If you don’t yet have a dedicated database user for n8n, create one with minimal permissions. For example:

CREATE USER 'n8nuser' IDENTIFIED WITH mysql_native_password BY 'REPLACE_ME';

GRANT SELECT ON exampledb.* TO 'n8nuser';
GRANT SELECT ON exampledb2.* TO 'n8nuser';

For most reporting or sensor databases SELECT is enough. Avoid granting statements with side effects like INSERT, UPDATE, DELETE, etc. unless you really need them. This is the same principle as for any web application - only grant the minimal required permissions.

Write a Strict Tool Description

You can let n8n auto-generate the tool description, but it is usually better to define it manually, especially when not restricting access in a restrictive manner. A strict and explicit description helps the agent understand the schema and also prevents hallucinated tables.

A short example could look like the following:

Execute a query in the historical measurement database (temperatures and humidities as well as information about the present sensors)

* table has to be one of the following table names (excluding the column descriptions):
   * humiditysensors(id, label, description)
   * temperaturesensors(id, label, description)
   * humidityvalues(ts, sensorid, humidity) are the humidity values of the sensors with sensor id sensorid at unix time ts.
   * temperaturevalues(ts, sensorid, temp) is the temperature of the sensor with sensorid (foreign key to temperaturesensors) at time ts (unix timestamp)

This does two things:

• It teaches the agent which tables exist.
• It provides the basis for strict whitelisting later.

Defining Parameters with $fromAI

This has been there in nearly all tutorials but it was never said explicit enough for me to get it immediately. To define a parameter one is accessing it with the fromAI method using:

$fromAI('parametername', 'description', 'datatype', 'default')

The parameter is created the moment the expression is evaluated. Note that each parameter can only be defined once. This means one cannot use fromAI multiple times to access the same parameter. Instead one has to read it once, store it in an variable and use the variable afterwards. Trying to reuse $fromAI directly in multiple places fails the node.

Building the SQL Query

In the MySQL node you can enable Execute SQL and generate the query dynamically with JavaScript. This is powerful, but it is of course also prone for SQL injection. The safe pattern is:

• The agent provides only a key (for example temperaturevalues)
• The key is mapped to a hardcoded SQL identifier
• Data is always safely escaped before being inserted into the statement (you can use the query parameter setting and placeholders). An even better approach is the usage of prepared statements and bound parameters (which is out of scope for this short article).

The following shows an example implementation that allows the agent to access the tables mentioned above in an arbitrary fashion. In addition it will order the results from tables that include timestamps by timestamp in descending order:

SELECT 
{{
  (() => {
    // Read AI parameters only once
    const t = String($fromAI('table', 'The table name', 'string', '') || '')
      .trim()
      .toLowerCase();

    // Whitelist mapping
    const allowed = {
      humiditysensors: '`humiditysensors`',
      humidityvalues: '`humidityvalues`',
      temperaturesensors: '`temperaturesensors`',
      temperaturevalues: '`temperaturevalues`',
    };

    const tablesWithTs = new Set(['temperaturevalues', 'humidityvalues']);

    if (!allowed[t]) {
      throw new Error(`Disallowed or unknown table key: ${t}`);
    }

    const orderClause =
      tablesWithTs.has(t)
        ? ' ORDER BY `ts` DESC'
        : '';

    const selector =
      tablesWithTs.has(t)
        ? '*, FROM_UNIXTIME(ts) AS ts_readable'
        : '*';

    return `${selector} FROM ${allowed[t]}${orderClause}`;
  })()
}}
LIMIT 50;

What this does:

• The agent provides the table only as a logical key. It never gets passed into the statement directly.
• The key must exist in the whitelist (you can create the whitelist also in n8n by first executing a SHOW TABLES statement and passing its result of course).
• Time series tables are automatically ordered by ts DESC.
• A readable timestamp (ts_readable) is generated using FROM_UNIXTIME().

This approach prevents the agent from injecting arbitrary SQL fragments.

Always Add a LIMIT

Never allow unlimited queries. If you do, you will regret it. Also do not make it configurable in an unbounded fashion. Not even when it looks like it works on the first hand. If you forget the LIMIT, sooner or later the agent will try to read an entire table which typically leads to:

• MySQL running very long queries
• n8n workers blocking
• the agent producing unuseable responses due to context thrashing or context rot.
• you use very large context windows with an insane number of tokens. You will spot this on your bill.

A safe default is something like:

LIMIT 50

For more advanced setups you can add pagination. Also do not forget to limit the maximum number of iterations that your agent can loop. Your financial account will thank you.

Security Rules You Should Actually Follow

Treat LLM input exactly like user input. This means:

• Never insert raw LLM strings into SQL.
• Only allow whitelisted identifiers.
• Fail the request when validation fails.
• Use a least-privilege database user.

LLMs are not malicious but they are very creative. And creativity plus databases without guardrails tends to produce unpleasant surprises.

Useful Extensions

Once the basic version works, you can extend the system safely. Typical improvements include:

• Filtering: In the above example allow a numeric sensorid parameters and parse it strictly as an integer.
• Time window queries: Allow queries such as WHERE ts BETWEEN ... AND ... but enforce maximum time spans.
• Aggregation: Support queries like AVG(temp) GROUP BY hour again through explicit whitelists.

This allows scaling the system without turning the SQL builder into a huge block of logic.

Conclusion

With a MySQL tool node, strict tool descriptions, $fromAI parameters and a whitelist-based SQL builder you can create a flexible agent-driven database interface without exposing your database to SQL injection. Just keep in mind:

Never trust LLM input, treat it like user input.

The term coherence has multiple meanings across physics and philosophy, all centered on an underlying idea of parts sticking together or acting in unison. In everyday language and philosophy, coherence usually means logical consistency and intelligibility - a coherent argument is one whose parts fit together without contradiction. In physics, coherence more specifically describes correlated behavior of waves or quantum states. For example, coherent waves are in phase with each other (maintaining a fixed phase relationship), and quantum coherence refers to the definite phase relationships in a quantum superposition. Despite the varied contexts, these meanings share the notion of a unified, orderly relationship among components (whether phases of waves, quantum amplitudes, or propositions in an argument). In the sections below, I survey the diverse meanings of “coherence”, focusing primarily on physics (classical waves and quantum mechanics) and touching on philosophical usage.

• Historical Evolution of the Concept
• Coherence in Classical Wave Physics
• Quantum Coherence and Superposition
  • Coherent States in Quantum Mechanics
  • Quantum Decoherence
  • Coherent Control and Manipulation in Quantum Systems
• Coherence in Philosophy and Logic
• Conclusion

Historical Evolution of the Concept

The concept of coherence in physics emerged from 19th century studies of wave interference. Thomas Youngs famous double-slit experiment back in 1801 implicitly required coherent light - using a single light source split into two paths - to produce stable interference fringes. Young and other physicists at that time recognized, that two independent light sources - for example the sun and a lamp - generally do not form visible interference because they lack fixed phase relations. In 1819, Fresnel and Arago formulated laws of interference, effectively noting conditions under which light waves cohere (e.g. same frequency and polarization) to produce fringes. By the late 19th century, techniques like Michelsons interferometry further quantified coherence: observers noticed that white light interference disappeared beyond a certain path difference, hinting at a finite coherence length.

In the early 20th century, coherence was formalized in statistical optics. Pioneering work by Fritz Zernike in 1938 introduced the degree of coherence as a quantitative measure by measuring the fringe visibility between two points in a wavefield. Zernikes work and the van Cittert–Zernike theorem showed how a sources size and spectral bandwidth determine partial coherence of light. The invention of the laser in 1960 provided a source of nearly fully coherent, highly monochromatic and phase-stable light, which revolutionized optics and validated these theories. Llaser light can have coherence lengths of kilometers, whereas sunlight’s coherence length is only a few microns.

The quantum era brought new facets to coherence. In 1963, Roy Glauber developed the quantum theory of optical coherence, introducing coherent states of the electromagnetic field and correlation functions to describe photon statistics. Glaubers work - awarded with a Nobel Prize 2005 - established how classical coherence concepts extend to quantum light. Meanwhile, physicists like Erich Joos, Dieter Zeh and Wojciech Zurek in the 1970s - 1990s studied quantum decoherence - how interactions with the environment destroy coherence and make quantum systems appear classical. By the 21st century, quantum coherence became recognized as a resource for technologies like quantum computing, requiring careful preservation and manipulation.

In philosophy, on the other hand, the notion of coherence has an older pedigree in theories of truth and knowledge. Coherence as a criterion of truth was advanced by 19th-20th century idealist philosophers (e.g. Hegel, Bradley) and later formalized as the coherence theory of truth. According to this view, a proposition is true if it coheres (i.e. is consistent or entailed by) a set of other accepted propositions. Early versions simply equated coherence with logical consistency, though more refined versions involve mutual explanatory support. Thus, the idea of “coherence” as internal consistency in a system of ideas has a long history in epistemology alongside its development in physics.

Coherence in Classical Wave Physics

In classical optics and wave physics, coherence describes the ability of waves to exhibit stable interference due to fixed phase relationships. Optical coherence specifically refers to the capacity of a light wave (or two waves) to produce an interference pattern of alternating constructive and destructive fringes. If two light beams show no interference (no stable bright/dark pattern), they are said to be incoherent with each other; if they produce clear, high-contrast fringes (including complete destructive cancellation at some points), they are fully coherent. Intermediate cases (partial fringe visibility) indicate partial coherence.

In Physics one distinguishes two aspects of coherence for waves:

• Temporal Coherence: Correlation of a wave with itself at different times. This relates to the waves monochromaticity (single-frequency purity). A perfectly monochromatic (single-frequency) wave has infinite temporal coherence - its phase is predictable for any time shift, yielding interference even between widely separated time samples. In practice, real sources have finite bandwidth, so the phase drifts over time. The coherence time $\Delta \tau_c$ is defined as the maximum time interval over which the waves phase is predictable or correlated with itself. Equivalently, it is the time over which the field “looks” approximately sinusoidal with a stable phase. Beyond $\tau_c$, phase relationships effectively randomize and interference visibility drops to zero. The coherence length $L_c = c \Delta \tau_c$ is the propagation distance corresponding to the coherence time. For example, a stabilized laser with extremely narrow linewidth might have $\tau_c \sim 10^{-4}$ s and $L_c$ on the order of $30$ km, whereas broadband sunlight has $\tau_c$ on the order of $10^{-12}$ s and $L_c$ of a few micrometers. Temporal coherence is what a Michelson interferometer measures by varying path delay: fringes contrast diminishes as the delay exceeds the source’s coherence time. In summary, temporal coherence quantifies how well a wave maintains a stable phase over time – the narrower the spectrum (smaller bandwidth), the longer the coherence time.
• Spatial Coherence: Correlation of a waves phase at different points in space across an extended wavefront or between separate beams. This is related to the spatial uniformity of phase across the wave. Spatial coherence determines whether two points on a wavefront can form interference fringes when combined. A wave is spatially coherent across a region if any two points within that region emit waves with a fixed phase relation. Youngs double-slit experiment is a test of spatial coherence: if the two pinholes (separated by some distance $d$) are illuminated by a source, clear interference fringes appear on a screen only if $d$ is within the sources transverse coherence length. If the pinholes are too far apart, each sees a different phase of the source at a given time, and the fringe visibility degrades to nothing. The coherence area is defined as the area over which the light field is spatially coherent (e.g. for filtered sunlight it might be ~$4\times10^{-3}$ mm$^2$, meaning pinholes must lie within a ~0.06 mm distance to observe interference). In general, smaller or distant sources (like a distant star approximating a point source) have higher spatial coherence than extended sources (like the sun’s disk). The van Cittert–Zernike theorem provides a quantitative link: the spatial coherence function across a plane is essentially the Fourier transform of the sources intensity distribution - a large angular source yields low spatial coherence, and a point-like source yields high coherence.

In practice most light fields are neither fully coherent nor fully incoherent, but somewhere in between. The degree of coherence (first-order coherence) can be quantified by a complex coherence function or correlation coefficient $\gamma_{12}$ between two points or times. This is essentially the normalized cross-correlation of the wave’s electric field at the two points/times. If $\mid\gamma_{12}\mid = 1$, the fields are perfectly coherent (phase of one completely predicts the phase of the other) and interference contrast is maximal. If $\mid\gamma_{12}\mid = 0$, they are completely uncorrelated (incoherent) and no clear interference appears. Partially coherent light yields an intermediate $0<\mid\gamma\mid<1$, producing fringes of reduced contrast (washed-out interference). In summary, coherence in classical waves refers to the presence of stable correlations (in phase and amplitude) either over time or across space, enabling observable interference effects. Techniques like holography, interferometry, and optical coherence tomography all rely on manipulating and measuring these coherence properties of light.

In other wave phenomena beyond optics, coherence has analogous meanings. For example, in acoustics two sound waves are coherent if they have a constant phase difference; in radio communications, a coherent receiver maintains a reference phase to interfere the incoming signal. Coherence is a unifying wave concept signifying the presence of an underlying order or correlation in the wavefield.

Quantum Coherence and Superposition

In quantum physics, coherence refers to the existence of definite phase relationships between quantum states. A quantum state that is a superposition of two or more basis states is coherent if the relative phases are well-defined and stable, allowing for interference effects at the quantum level. In contrast, if those phase relationships are randomized or unknown (as in a statistical mixture), the superposition is incoherent. Quantum coherence is what differentiates a pure quantum superposition from a mere classical probabilistic mixture.

For example, an electron in a superposition $\frac{1}{\sqrt{2}}(\mid\uparrow> + e^{i\phi}\mid\downarrow>)$ has coherence between the spin-up and spin-down components - the phase $e^{i\phi}$ will lead to interference effects in experiments. If that phase is completely random or the state is an incoherent mixture $\tfrac{1}{2}(\mid\uparrow>\langle\uparrow\mid + \mid\downarrow>\langle\downarrow\mid)$, no single-particle interference can occur between the $\mid\uparrow>$ and $\mid\downarrow>$ outcomes. Thus, quantum coherence is essential for phenomena like single-particle interference (an electron or photon interfering with itself in a double-slit experiment) and is a prerequisite for entangled correlations between particles.

Formally, quantum coherence can be defined in terms of the density matrix of a system. In a chosen reference basis, the off-diagonal elements of the density matrix measure the coherence between the corresponding basis states. An incoherent state is one whose density matrix is diagonal in the reference basis (no superposition terms). Any state with nonzero off-diagonal entries is a coherent superposition in that basis. It possesses quantum coherence as a resource. For example, if we take the computational basis ${\mid 0>,\mid 1>}$ for a qubit, an incoherent state would be of the form $\rho = p\mid 0>\langle 0\mid + (1-p)\mid 1>\langle 1\mid$ (diagonal), whereas a state like $\mid\psi> = \alpha\mid 0> + \beta\mid 1>$ has off-diagonal terms $\alpha\beta^*$ in its density matrix and hence is coherent. The magnitude of those off-diagonals relates to the visibility of interference one could observe between the states $\mid 0>$ and $\mid 1>$.

In modern quantum information science, coherence is treated as a quantifiable resource (much like entanglement). There are measures, coherence monotones, that assign a number to how much coherence a given state has relative to a specified basis. Intuitively, this corresponds to how well the state can produce interference or be used in quantum algorithms. Coherence is “consumed” or degraded by interactions that cause decoherence and it can be partially converted into other quantum resources like entanglement under the right operations. Quantum computing relies on maintaining coherence in qubits throughout computational gate operations. The superposition of $\mid 0>$ and $\mid 1>$ in each qubit (and across multiple qubits) must remain coherent long enough to perform interference-based algorithms. If qubits lose coherence too quickly, quantum computation reverts to classical outcomes.

At a fundamental level, quantum coherence underlies phenomena like quantum interference (e.g. electron diffraction patterns require the electron wavefunction to remain coherent across the paths) and is linked to entanglement. Entanglement can be viewed as a kind of coherence between subsystems: an entangled pair of particles has no local coherence (each reduced state may be mixed) but has joint coherence in the global state. In summary, quantum coherence captures the wavelike aspect of quantum states - the ability of probability amplitudes to superpose and interfere.

Coherent States in Quantum Mechanics

In quantum mechanics, the phrase coherent state has a specific technical meaning beyond just “state with coherence”. Coherent states typically refer to a special set of quantum states of a harmonic oscillator or fields, that most closely resemble classical oscillations. The canonical example is the Glauber–Sudarshan coherent state of the electromagnetic field - the quantum state of light that a stabilized laser outputs.

Mathematically, a coherent state $\mid \alpha>$ (for a harmonic oscillator or single mode of the field) is defined as the eigenstate of the annihilation (lowering) operator $\hat{a}$. These $\mid \alpha>$ states form an overcomplete, non-orthogonal basis of the oscillators Hilbert space. They are often written as displaced vacuum states and have the minimum uncertainty allowed by quantum mechanics (equal uncertainties in $x$ and $p$), which is why they are sometimes called “minimum uncertainty wavepackets”. A coherent state exhibits Poissonian number statistics and, in many respects, behaves like a classical sinusoidal oscillation with amplitude $\mid \alpha\mid$ and phase $\arg(\alpha)$. For instance, the electric field expectation oscillates in time as a classical field would, and the probability of finding $n$ photons in $\mid \alpha>$ is $P(n)=e^{-\mid\alpha\mid^2}\mid\alpha\mid^{2n}/n!$ (a Poisson distribution).

$ \begin{aligned} \hat{a} \mid\alpha> &= \alpha \mid\alpha> \end{aligned} $

Historically, coherent states were studied by Schrödinger (1926) as Gaussian wavepackets that remain localized in a harmonic potential without spreading. In the 1960s, Glauber and Sudarshan formally introduced them in the context of quantum optics to describe the output of a laser and to define what classical light means quantum-mechanically. In optical coherence theory, these coherent states are considered the most classical states of the field - any state that is a statistical mixture of coherent states is regarded as a classical light field, whereas states that cannot be expressed as such a mixture (e.g. squeezed states, Fock states) are nonclassical. In fact, one can define optical nonclassicality as the presence of quantum coherence that cannot be accounted for by a random classical field. Coherent states thus straddle the line between quantum and classical: they are quantum states with maximal coherence (in the sense that they saturate certain coherence measures) but they produce dynamics and statistics reminiscent of classical waves.

Coherent states play a role in coherent state quantization and path integrals - they provide a convenient basis to represent quantum dynamics (especially in quantum optics and many-body theory). In summary, a coherent state in quantum mechanics is a specific type of quantum state (especially of oscillators/fields) characterized by classical-like behavior and defined by eigenstate relations like $\hat{a}\mid\alpha> = \alpha\mid\alpha>$. It should not be confused with the broader notion of “a state having coherence”; rather, it is a term for these minimum-uncertainty wavepackets that remain as coherent as possible.

Quantum Decoherence

Quantum decoherence is the process by which quantum coherence is lost - or apparently lost - due to a systems interaction with its environment. When a quantum system is not perfectly isolated, the phase information that defines its coherent superposition can become entangled with environmental degrees of freedom. From the perspective of the system alone, the coherent superposition then appears to collapse into a mixture, as the relative phases are no longer observable (having “leaked” into the environment). Decoherence is the mechanism that explains how classical behavior emerges from quantum systems: it suppresses interference between a systems quantum states by irreversibly correlating those states with different states of the environment.

“Quantum decoherence is the loss of quantum coherence”, typically through loss of information from the system to the environment. During decoherence, no fundamental wavefunction collapse is assumed to occur; rather, the system-plus-environment evolves unitarily, but the system’s reduced state transitions from pure to mixed. The off-diagonal elements of the system’s density matrix decay towards zero as coherence is delocalized into the environment. A well-known analogy is friction: just as mechanical energy dissipates into environmental heat, quantum phase information dissipates into environmental degrees of freedom. The coherence is not destroyed per se but becomes inaccessible - stored as correlations between the system and environment that are practically irretrievable.

Decoherence theory, developed by Zeh, Zurek and others, provides a resolution to the quantum measurement paradox in that it explains the apparent collapse of the wavefunction. For example, a state describing Schrödingers cat $\frac{1}{\sqrt{2}}(\mid\text{alive}> + \mid\text{dead}>)$ rapidly decoheres due to environmental interactions (air molecules, photons, etc.) that measure the cat’s state. The environment gains information about which branch (alive or dead) occurred, and interference between the branches becomes unobservable - the cat is effectively in a classical probabilistic mixture from any local standpoint. Decoherence does not produce a true wavefunction collapse on its own, but it makes interference between macroscopically distinct states vanish extraordinarily quickly, yielding the appearance of a definite outcome.

Quantitatively, decoherence can be characterized - analogous to coherence time - by a decoherence time over which off-diagonals decay. This can be exceedingly short for macroscopic differences - e.g. a dust particle can decohere in $10^{-20}$ seconds when hit by air molecules. Decoherence ties into the concept of pointer states: the basis in which the system remains robust (diagonal) under environmental interaction. Those pointer states (like alive or dead cat) are the classical-like states that do not themselves get blurred by interference because the environment continually monitors them.

For emerging quantum technologies, decoherence is a critical enemy. Qubits in a quantum computer must be isolated from noise sources because any unmonitored interaction (thermal fluctuations, stray fields, etc.) can entangle with the qubit and collapse superpositions, thereby ruining quantum computations. Quantum error-correcting codes and decoherence-free subspaces are being developed to counteract this.

Decoherence is the process that destroys coherence: a coherent quantum superposition evolves into an incoherent mixture when the system’s phase information leaks into the environment. It bridges quantum and classical physics by explaining why large, open systems don’t usually display quantum interference, and it highlights why maintaining coherence (isolation or error correction) is essential in quantum experimentation and technology.

Coherent Control and Manipulation in Quantum Systems

The term coherent is also used in the context of controlling physical systems, especially in quantum mechanics, to imply control that preserves phase relations. Coherent manipulation (or coherent control) refers to using interactions (like laser pulses, microwave fields, etc.) to steer a quantum systems state in a deterministic, phase-preserving way. The system’s evolution is unitary and maintains quantum coherence throughout the process, as opposed to incoherent processes (like measurements or random thermal kicks) that induce decoherence.

For example, one can apply a sequence of precisely timed magnetic resonance pulses to a spin qubit system to rotate the spin state on the Bloch sphere. If these operations are done faster than the decoherence time and with precise phase reference, the spin is undergoing coherent manipulation. The ability to perform such operations is a prerequisite to any scalable quantum information platform - you must be able to apply coherent control to qubits in order to do quantum gates without losing the information to decoherence. Coherent manipulation typically implies that the system remains in a pure state during the operation.

In a broader sense, coherent control is a subfield of physics and chemistry where interference of quantum amplitudes is used to direct outcomes. For instance, in photochemistry, carefully shaped laser pulses have been used to coherently control reaction pathways by interfering different excitation pathways. The coherence here implies that the driving fields and the systems response maintain a fixed phase relationship, producing constructive or destructive interference in the quantum transition amplitudes to favor a desired product.

A classic example is coherent population trapping or STIRAP (STImulated Raman Adiabatic Passage) in atomic physics, where two coherent lasers create interference that drives an atom from state $\mid A>$ to $\mid B>$ without populating an intermediate state $\mid C>$. The success of this technique relies on maintaining phase coherence between the two laser fields and the atomic polarization. If the lasers were not phase-coherent with each other, the interference would average out and the coherent transfer would fail.

Describing an experimental technique as coherent (coherent manipulation, coherent spectroscopy, etc.) implies that phase coherence is preserved throughout the process. The systems evolution is phase-correlated with the driving fields or between its own states. This is crucial in quantum computing (for executing logic gates on superpositions), in quantum optics (such as creating entangled photons via coherent pump processes), and in quantum sensing (where a phase-coherent superposition interacts with a field and accumulates a measurable phase shift). A large effort in modern physics is to extend the duration of coherent control (increase coherence times) by improving isolation, material purity, and using dynamical decoupling or error correction.

Coherence in Philosophy and Logic

Outside of physics, coherence generally refers to a property of statements, beliefs, or arguments - essentially, a logical and orderly consistency. A coherent statement or theory is one whose parts are logically connected and free of contradictions, so that the whole sticks together conceptually. For example, we might say a scientific theory is coherent if its various hypotheses and observations support each other and form a unified explanation. In everyday terms, someone “speaking coherently” is expressing their thoughts clearly and consistently.

In epistemology and the theory of truth, coherence plays a prominent role through the coherence theory of truth and coherentism. The coherence theory of truth holds that the truth of a proposition consists in its coherence with a specified set of other propositions. In other words, a new claim is true if it fits harmoniously into a larger body of beliefs without contradiction and with mutual support. This is in contrast to the correspondence theory of truth, which says truth consists in correspondence to objective facts. Coherentist philosophers argue that an isolated proposition cannot be judged true or false except by seeing whether it coheres with an entire system of propositions believed to be true. Early versions of coherentism equated coherence with mere logical consistency, but more developed versions require a stronger entailment or explanatory relation among beliefs. For instance, one belief coheres with others not just by avoiding contradiction, but by perhaps being entailed by them or contributing to their overall support.

Coherentism in epistemology is the view that justification of beliefs lies in their coherence with all other beliefs one holds, rather than in some foundational self-evident truths. In this view, our knowledge is like a web where each strand supports the others, and the entire network is coherent if it contains no contradictions and is mutually reinforcing. A perfectly coherent belief system would be one where every belief is consistent with every other and perhaps each is derivable from the whole. While perfect coherence is an ideal, coherence theories allow that truth and justification come in degrees - a belief can be more or less coherent with the rest, and accordingly more or less justified or likely true.

To illustrate, consider a detective assembling a story of what happened at a crime scene. A coherent explanation is one where all pieces of evidence fit into a single consistent timeline and causation, with no inexplicable gaps or contradictions. If a new piece of evidence doesn’t contradict but instead is predicted by or entailed by the theory, it increases the coherence of the theory, and thus increases our confidence in its truth. If the evidence creates a contradiction, the theory becomes incoherent and must be revised or rejected.

In the realm of ideas, coherence means internal consistency and logical connectivity. A coherent argument has premises that support the conclusion in an organized way. A coherent policy plan is one where the measures align with each other and with the overall goals. The common thread is that nothing sticks out as inconsistent or unrelated - all parts contribute to a unified whole. This everyday/philosophical notion of coherence, while conceptually distinct from physical coherence, metaphorically resonates with the physics usage: just as coherent waves march in step to produce a clear signal, coherent thoughts align together to produce clear understanding.

Conclusion

Across physics and philosophy, coherence signifies a kind of unity or correlation that makes the behavior of a system (be it waves, particles, or ideas) orderly and intelligible. In physics, coherence underlies the patterns of interference and the delicate power of quantum superpositions - it marks the difference between random, uncorrelated phenomena and ones that act with a fixed relationship. In quantum systems, maintaining coherence is essential for harnessing quantum effects before decoherence sets in. In philosophy, coherence is the glue of rationality, binding beliefs and assertions into a consistent worldview. The many faces of coherence all reflect the original Latin root cohaerere, “to stick together”.

In this post we derive the model behaviour of the two common operational amplifier (OpAmp) configurations (inverting and non-inverting) using the idealized amplifier model. We assume infinite input impedance of the operational amplifier, i.e. zero bias currents, as well as an infinite open loop gain. The amplifier is assumed to swing to it’s saturation voltage - which is assumed to be the rail voltage in this example. For a practical circuit you will have to check which output levels you really need. The output will only change between the upper and lower saturation voltage. In addition you will have to verify if the input bias current is negligible.

In this blog article we are going to look into:

• The inverting Schmitt trigger and it’s characteristics
• And the non inverting Schmitt trigger
• A short conclusion

Inverting Schmitt Trigger

First let’s recall the output of an optimal operational amplifier. The open loop gain is assumed to be $G \to \infty$. This means that the operational amplifier will swing to it’s maximum or minimum output voltage (i.e. close to the supply rail) depending on the inbalance between the inverting input $U_{-}$ and the non inverting input $U_{+}$:

The voltage on the inverting input is given by the input voltage $U_{-} = U_{in}$. The voltage on the non inverting input can be described via a simple voltage divider between the output voltage $U_{out}$ and the bias voltage $U_{b}$:

Taking a look at the two possible output cases one can determine the switchpoints of the circuit.

In the first case case $U_{out} = U_{vcc}$ one gets:

Switching happens when $U_{in} > U_A$.

In the second case $U_{out} = -U_{vcc}$:

Under assumption of a symmetric supply voltage a look at the width of the hysteresis curve yields

The center of the hysteresis is given by $U_b$. As one can see the width only depends on the resistor values, not on the bias voltage.

Non Inverting Schmitt Trigger

The output of the operational amplifier is again determined by the inbalance between the inverting input $U_{-}$ and the non inverting input $U_{+}$. Again assuming infinite open loop gain $G \to \infty$ the amplifier simply swings to it’s maximum or minimum output voltage:

Applying the voltage divider like in the inverting case again:

The switch points between the two states are now a bit more complex than in the inverting case due to the contribution of the input signal.

In the first case $U_{out} = U_{vcc}$ we get

The switch happens when $U_{+} \lt U_b$:

In the second case $U_{out} = -U_{vcc}$ we get

The switch happens when $U_{+} \gt U_b$:

Again we get the characteristic shape exhibiting the hysteresis as we have already seen from the inverting Schmitt Trigger.

Under assumption of a symmetric supply voltage a look at the width of the hysteresis curve yields

The center of the hysteresis is given by $U_b$. As one can see the width again only depends on the resistor values, not on the bias voltage.

Conclusion

Both Schmitt trigger configurations implement the same idea. The positive feedback creates two distinct switching points. In the inverting configuration the input goes straight into an op-amp input, which in the ideal case is assumed to have infinite input impedance. Practically, the input current is the bias current. There is no intentional resistive path from output to the input source and thus no direct back-action of the circuit to the input.

In the non inverting configuration the input is connected through a resistive divider into a summing node that is also connected to the output. This means the output can source or sink current through the resistors causing a back action of the circuit onto the input. The input impedance seen by the source is finite and depends on the present output state.

For practical applications keep in mind that dedicated components implementing Schmitt triggers exist. These are usually faster than OpAmp based circuits and contain polysilicon resistors already on the chip die, which makes the circuit simpler and more compact.