BetterDB
Back to blog

Valkey/Redis Monitoring Overhead: Interleaved Benchmark Results

Kristiyan Ivanov•

Most monitoring tools claim low overhead but never show their work. Here's the interleaved benchmark methodology we use to measure BetterDB's monitoring overhead on Valkey and Redis.

Every monitoring tool claims "minimal overhead." Few show their work. Here's how we actually measure ours, and why the methodology matters more than the number.

If you're running Valkey or Redis in production, monitoring overhead isn't a nice-to-have detail. It directly affects tail latency, throughput, and how much headroom you need to keep SLAs safe. That's why BetterDB focuses on low-overhead observability built specifically for these workloads—see how it works in the BetterDB features.

The Problem with Naive Benchmarking

The most common approach to measuring monitoring overhead is sequential: run a benchmark without the monitor, then run it again with the monitor, and compare. This sounds reasonable but produces unreliable results.

Sequential benchmarks suffer from confounding variables that have nothing to do with your monitoring tool. CPU thermal throttling changes performance over time as your processor heats up. Background processes compete for resources unpredictably. Memory fragmentation accumulates. Garbage collection pauses hit at random intervals. The operating system's I/O scheduler makes different decisions based on cache state.

The result is that you can run the same benchmark twice with no changes at all and see 2-5% variance between runs. If your monitoring overhead is genuinely under 1%, you can't distinguish it from noise using a sequential approach.

Valkey/Redis Monitoring Overhead Benchmark Methodology

Why Interleaving Works

Interleaved benchmarking eliminates systematic bias by randomizing the order of measurements. Instead of "all baseline first, then all monitored," you create pairs of runs where each pair contains one baseline and one monitored run in random order.

Here's the core idea from our benchmark runner:

schedule = []
for i in range(n_runs):
    pair = [
        {"condition": "baseline", "run_id": f"baseline_{i:02d}"},
        {"condition": "monitored", "run_id": f"monitored_{i:02d}"},
    ]
    random.shuffle(pair)
    schedule.extend(pair)

By shuffling within pairs, any environmental drift (thermal throttling, background load, cache warming) affects both conditions equally. If run 3 happens to coincide with a system backup, it corrupts one measurement—not an entire condition.

What We Actually Measure

Each benchmark run uses valkey-perf-benchmark against a real Valkey instance. Between conditions, we start or stop the BetterDB Docker container and wait for stabilization:

def set_condition(condition):
    if condition == "baseline":
        if is_betterdb_running():
            stop_betterdb()
    else:
        if not is_betterdb_running():
            start_betterdb()

The stabilization period (10 seconds by default) ensures the system settles after starting or stopping the monitor.

We test across multiple configurations:

  • Quick profile: SET/GET operations, 64-256 byte payloads, pipelines of 1 and 16, 100K requests per test
  • Full profile: SET/GET/HSET/LPUSH, 64-1024 byte payloads, pipelines of 1/10/50, 500K requests per test

Both use 50 concurrent clients, which represents a realistic production-like connection count.

Reading the Results

For each command type, we compute mean throughput, standard deviation, and coefficient of variation (CV) across all runs of that condition. Overhead is the percent drop in mean throughput when monitoring is enabled. CV (coefficient of variation) is the standard deviation divided by the mean, which tells you how noisy the runs are.

overhead = ((baseline_mean - monitored_mean) / baseline_mean) * 100

Our interpretation thresholds:

  • < 1%: Statistical noise, indistinguishable from measurement variance
  • 1-5%: Acceptable overhead
  • > 5%: Worth investigating

If the CV exceeds 15% for either condition, we flag the results as unstable—meaning you should increase run count or check for system interference.

Optional: Eliminating Hardware Variance

For maximum precision, our system-prep.sh script disables CPU turbo boost and sets the performance governor:

sudo scripts/benchmark/system-prep.sh

Turbo boost dynamically adjusts CPU frequency based on thermal headroom, which introduces measurement variance that has nothing to do with software. Disabling it gives you a fixed clock rate so differences between runs come from the software, not the hardware. These settings reset on reboot, so there's no risk to your system.

Our Results

Across hundreds of interleaved runs, BetterDB consistently measures below 1% overhead—often indistinguishable from noise. In many runs, the overhead shows as negative—meaning the monitored runs were slightly faster, which tells you the real impact is at the noise floor.

This makes sense when you understand what BetterDB actually does on the monitored instance: it maintains a single dedicated connection (named BetterDB-Monitor) and issues read-only commands like INFO, SLOWLOG GET, CLIENT LIST, and LATENCY LATEST at configurable intervals. That's 1-5 KB/s of network traffic on an instance handling hundreds of thousands of operations per second.

Running It Yourself

The benchmark suite is included in the repository. To run it:

cd benchmark
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Preflight checks
../scripts/benchmark/preflight-interleaved.sh

# Quick run (~5 min)
python3 interleaved_benchmark.py --runs 5 --config configs/betterdb-quick.json

# Full run (~15 min)
python3 interleaved_benchmark.py --runs 10 --config configs/betterdb-full.json

Results land in betterdb-results/ with a comparison.md report and raw CSV data for each run.

Why This Matters

Enterprise teams evaluating monitoring tools need to trust the overhead claims. "We benchmarked it and it's fine" isn't sufficient when you're running Valkey at scale with latency SLAs measured in microseconds.

By open-sourcing the benchmark methodology alongside the tool, we're inviting scrutiny. Run it on your hardware, with your workload profile, and see the numbers yourself. The interleaved approach means you'll get reliable results even on a noisy development machine.

We'd rather show our work and let the methodology speak for itself than claim a number you can't verify.

BetterDB is an open-core observability platform for Valkey and Redis. The benchmark suite is MIT-licensed and included in the main repository.

Back to blog