tu-ops-agent

The agent installed on every ToldUntold server. It ships logs, host metrics and host state to TU Ops, which stores them in its ClickHouse database.

It is not a daemon of its own: it is Vector (pinned version) with our configuration, a small state script, a systemd unit with resource limits and an installer.

tu-ops-agent (each server) ──HTTPS + token──► TU Ops (/api/agent/*) ──► ClickHouse

What it sends

Stream	Source	Content
metrics	Vector host_metrics, every 30 s	CPU, memory, load, filesystems, network, uptime
state	scripts/host-state.sh, every 60 s	pending apt updates (cached 1 h), reboot flag, systemd unit states, Docker containers
logs	journald	warnings and worse (priorities 0-4), firewall rejections (UFW BLOCK) dropped
logs	Laravel files (--with=laravel)	warnings and worse, stack traces kept as one entry
logs	Caddy JSON logs (--with=caddy)	5xx responses and Caddy warnings/errors
logs	Docker (--with=docker)	stdout/stderr of every container

The agent only sends its token: TU Ops derives the server, its environment and the destination from it. A token copied to another host still writes under its own server name.

Install

The easy way: in TU Ops, Agents → New agent (or Rotate key) gives a one-line command to paste on the server as root. It downloads this repository and runs install.sh with the options derived from the inventory (units, Laravel logs, Docker). The command is valid once, for 15 minutes.

By hand:

git clone https://git.webrocks.net/ToldUntold/ops-agent.git && cd ops-agent
sudo TU_OPS_AGENT_TOKEN='tuops_…' ./install.sh \
  --ops-url=https://ops.tolduntold.io \
  --units=caddy,php8.4-fpm,supervisor \
  --with=laravel,caddy \
  --laravel-logs='/home/tolduntold/tolduntold.net/api/shared/storage/logs/**/*.log'

Re-run it with different options to change the configuration (./install.sh --help lists them; the token can also be given with --token= or --token-file=). sudo ./uninstall.sh removes the agent (--purge also deletes its buffers and user).

What the installer does: installs the pinned Vector .deb from packages.timber.io (over HTTPS, no checksum verification) and disables its stock service, creates the tu-ops-agent user, writes /etc/tu-ops-agent/ (configs and agent.env, mode 0640), validates the configuration and starts tu-ops-agent.service.

Footprint on the host

• systemd limits: CPUQuota=25%, MemoryMax=256M, Nice=10, lowest I/O priority, 2 threads.
• Filtering happens on the host, so most log lines never leave it.
• A throttle caps each service at 6000 log lines per minute.
• Disk buffers for logs and metrics are capped at 256 MiB each; when one is full, new events are dropped rather than filling the disk.

Access the agent user gets

• systemd-journal and adm groups, to read the journal.
• The group owning the Laravel and Caddy log directories, to read those files.
• With --with=docker, the docker group. This is root-equivalent on the host; only enable it where container logs and states are needed.

Layout

config/10-base.yaml        metrics, state, throttle, sinks (always installed)
config/20-journald.yaml    journal warnings (always installed)
config/optional/*.yaml     laravel, caddy, docker (installed by --with)
scripts/host-state.sh      state probe, POSIX sh
systemd/                   service unit
install.sh, uninstall.sh

A new log source is one file in config/optional/ whose final transform is named parsed_* and emits timestamp, source, service, level, message, fields (fields is a JSON string).

Known limits

• journald is filtered by priority: a service that logs errors at info priority on stdout is not captured yet.
• Laravel timestamps carry no timezone: they are read in the host's timezone, so the application and the host must use the same one.
• Disk metrics cover ext2/3/4, xfs, btrfs, zfs and f2fs filesystems only.
• Vector 0.58 needs VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true to read ${VAR} in configs; the unit sets it. The configs only reference the TU_OPS_* values of agent.env.
• Debian/Ubuntu with systemd only.