> ## Content Index
> Fetch the complete content index at: https://linuxhandbook.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# 4 GB of Logs a Minute: A Kafka Monitoring Incident With 20 Minutes to Spare
- URL: https://linuxhandbook.com/blog/kafka-monitoring-incident/
- Published: 2026-11-20T09:34:13.000Z
- Updated: 2026-11-20T09:46:51.000Z
- Description: A mixed JMX Exporter fleet, rapidly growing Jetty warnings, and the difference between buying time and fixing the cause.
- Author: Yash Kiran Patil
- Tags: blog

About twenty minutes before production traffic was due to begin, we upgraded the JMX Exporter to 1.6.0 on some Kafka brokers. The rollout was only partial: other brokers still had 0.9.0\. 

When the updated scrape configuration began reaching this mixed fleet, it worked against the newer agents, while requests to older instances were accompanied by Jetty warnings. As scrape requests overlapped, warning volume rose quickly. The incident account estimated system logs were growing by 3-4 GB per minute. What began as a partial metrics rollout had become a disk-capacity risk before production traffic started.

That rate changes the priority. Missing metrics is an observability problem; a log stream that can fill a broker’s filesystem is a host-availability problem. The immediate task was to stop the disk from becoming the next failure, while avoiding an unplanned [Kafka](https://kafka.apache.org/?ref=linuxhandbook.com) restart just before traffic. The durable task, making every JVM load the intended exporter, could wait for a controlled maintenance window.

![Prometheus Scrape Rollout](https://linuxhandbook.com/content/images/2026/11/prometheus-scrape-rollout.webp)

## What the version difference did and did not tell us

The JMX Exporter Java agent runs inside the application JVM. It reads selected Java Management Extensions (JMX) MBean values and makes them available in Prometheus exposition format over HTTP. 

Prometheus initiates scrapes at configured intervals and requests a metrics path; for the Java agent, the HTTP endpoint shares the broker’s process and host resources. A bad endpoint or expensive collection can therefore affect more than a dashboard.

The version split is a strong lead, but it is not a complete root-cause analysis. JMX Exporter 1.0.1 changed the default endpoint from `/` to `/metrics` and changed some JVM metric names. Prometheus itself defaults its `metrics_path` to `/metrics`. A configuration that crosses versions can consequently produce different responses, missing metrics, or errors depending on the actual agent and configured path. 

The 1.6.0 release also introduced a customizable metrics path; its HTTP server documentation describes a configurable worker thread pool with defaults of one minimum and ten maximum threads, using a blocking rejection handler. None of these documented changes proves that a particular Jetty warning was caused by endpoint incompatibility or a concurrency defect. The warning text, request path, response code, agent configuration, and stack trace are needed to make that claim.

The incident establishes that older agents emitted warnings under the new scrape configuration; it does not establish a Jetty concurrency bug. Preserve representative logs, compare the same request across versions, check for duplicate scrapers, and correlate request rate and latency with warning volume before naming the cause.

## Contain the resource risk before changing Kafka

First establish whether the filesystem is at risk and which process and file are responsible. Useful checks include:

```bash
df -h
sudo du -xhd1 /var/log
sudo lsof +L1

```

`df` reports filesystem allocation; `du` walks reachable files. Totals can differ because deleted files held open by a process still occupy blocks. `lsof +L1` helps find these unlinked files. Removing a pathname does not reclaim the blocks until the process closes its file descriptor. [Truncating a reachable file](https://linuxhandbook.com/empty-file-linux/) can reclaim blocks while leaving the inode available to the writer. Confirm the path, owner, mount, and log writer; preserve a sample of warnings and timestamps before discarding evidence.

![Scrape status](https://linuxhandbook.com/content/images/2026/11/green-scrape-full-disk.png)

The incident’s cron attempt did not work, and there was too little time to troubleshoot it safely. A narrowly scoped Bash loop, supervised by a temporary systemd service, was used as pressure relief. 

This is a last-resort example, not a logging design: it destroys contents and can conceal the fault. [Log rotation](https://linuxhandbook.com/journalctl-vacuum/), bounded retention, rate limiting, and correcting the request or logger are the proper follow-up.

## Generic emergency example: exact file, explicit scope

Adapt the path only after confirming it is the specific active, oversized log. This example deliberately refuses directories and symlinks. Save evidence first, and do not use it on a directory or wildcard.

```bash
sudo install -d -m 0755 /usr/local/sbin
sudo tee /usr/local/sbin/contain-noisy-log.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Replace this with the one verified, active log file.
readonly LOG_FILE="/var/log/kafka/noisy-jetty.log"

if [[ -L "$LOG_FILE" || ! -f "$LOG_FILE" ]]; then
  echo "Refusing: expected a regular, non-symlink file: $LOG_FILE" >&2
  exit 1
fi

while true; do
  /usr/bin/truncate --size=0 -- "$LOG_FILE"
  /usr/bin/sleep 30
done
EOF
sudo chmod 0750 /usr/local/sbin/contain-noisy-log.sh

```

Run the loop in the foreground under systemd, not with `nohup` or `&`. The [systemd unit below](https://linuxhandbook.com/create-systemd-services/) uses `Type=simple` process tracking and limits writes to the configured log directory. Replace `/var/log/kafka` in both places if the verified log lives elsewhere. Root is used here because log ownership varies; the write path is constrained with `ReadWritePaths`, but operators should still review the unit against their distribution’s systemd version and hardening policy.

```ini
# /etc/systemd/system/contain-noisy-log.service
[Unit]
Description=Temporary containment for one verified noisy log
After=local-fs.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/contain-noisy-log.sh
Restart=on-failure
RestartSec=2s
User=root
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/log/kafka

[Install]
WantedBy=multi-user.target

```

Start it for the current incident without enabling it at boot:

```bash
sudo systemctl daemon-reload
sudo systemctl start contain-noisy-log.service
sudo systemctl status contain-noisy-log.service
sudo journalctl -u contain-noisy-log.service -f

```

When logging is stable, stop and remove the temporary mechanism:

```bash
sudo systemctl stop contain-noisy-log.service
sudo systemctl disable contain-noisy-log.service  # harmless if it was never enabled
sudo rm /etc/systemd/system/contain-noisy-log.service
sudo rm /usr/local/sbin/contain-noisy-log.sh
sudo systemctl daemon-reload

```

Because this loop truncates the file it names, it will not repair disk usage from a deleted-but-open file whose pathname no longer exists. In that case identify the owning process and use its supported reopen or restart procedure during a safe window. Keep checking `df -h`; apparent success in the unit status does not prove that the filesystem pressure is resolved.

## Replace the workaround with a controlled rollout

Once log growth was contained, the JARs were made consistent across approximately 10–20 brokers. But replacing an agent JAR on disk does not hot-swap classes already loaded into a running JVM. A Java agent is supplied at JVM startup (`-javaagent:...`), so the running Kafka process continued using the previously loaded agent until it restarted. This explains why warnings could persist after the files had been updated; it does not independently prove what triggered them.

The restart was deferred until a maintenance window, then performed one broker at a time. Before each operation, check the cluster’s health, under-replicated partitions, in-sync replicas, leadership distribution, and whether another broker is already unavailable. 

Kafka’s controlled shutdown can move partition leadership away from a broker, but it depends on replicas being available; the Kafka operations guide notes that controlled shutdown cannot succeed if a hosted partition has no other live replica. Allow the broker to leave and rejoin, verify its replica state and client health, and stop the sequence if recovery does not meet the team’s criteria. Do not treat “systemd says active” as proof that the Kafka cluster has recovered.

After every restart, test the exporter endpoint from the host or monitoring network, confirm Prometheus target health and expected metric names, and verify broker health before proceeding. A useful canary compares old and new versions against the same configuration and checks path, status code, scrape duration, sample count, and important dashboards. Also inventory versions per host and make deployment automation report success or failure for each broker rather than only a fleet-level result.

![JMX Exporter Kafka incident](https://linuxhandbook.com/content/images/2026/11/jmx-kafka-incident-flow.webp)

As a precautionary measure, we can redirect the logs of Kafka to a specific location and can apply log rotation based on size and age of the logs or an hourly and weekly cleanup.

## What the incident leaves us with

The containment loop reduced immediate pressure long enough to avoid an unsafe pre-traffic restart. The permanent correction was to standardize the agent and restart brokers safely so the JVMs loaded it. Those are separate outcomes. The first protected the host in the moment; it did not fix the exporter behavior, and it discarded evidence. The second activated the rollout, but the version change alone does not tell us which request or code path produced every warning.

The durable work is specific: pin and inventory versions, test scrape paths and metric names before rollout, monitor filesystem and log growth, inspect duplicate scrapers and timeouts, and keep a broker-by-broker restart procedure with health gates. Tune the 1.6.0 HTTP thread pool only if measurements point to concurrency or latency; more threads can also increase resource use.

Production readiness is not just whether the new scrape succeeds on the upgraded node. It is whether mixed versions behave predictably, whether a failure has a bounded footprint, whether operators can tell a symptom from a cause, and whether the repair can be activated without putting the data plane at unnecessary risk. 

In this incident, the prudent choice was to buy time before traffic and perform the permanent change during a controlled window. The script was a temporary guardrail, not the fix.