The Illusion: Traffic appears to bypass Toxiproxy completely after the first successful metadata fetch.
The Reality: Kafka clusters hand out routing instructions. When Spring Boot asks the broker (via Toxiproxy at 19092) for partition leaders, the broker replies with its default EXTERNAL address (localhost:9092). Spring Boot caches this and sends all subsequent payloads directly to the broker, effectively circumventing the chaos proxy.
The Fix: The broker's KAFKA_ADVERTISED_LISTENERS must be hardcoded to tell external clients to communicate strictly over the Toxiproxy port (EXTERNAL://localhost:19092).
The Illusion: A 10-second delivery.timeout.ms kill switch hangs for 60 seconds.
The Reality: The delivery.timeout.ms overarching timer does not start until a payload successfully enters the RecordAccumulator buffer. If the producer does not have topic metadata cached, it must fetch it over the network first. If the network drops during this metadata fetch, the payload is stuck outside the buffer on the application thread. It waits for the max.block.ms timer (default 60s) to expire before throwing a TimeoutException: Topic not present in metadata.
The Rule: To accurately test network flight timeouts, you must "prime the pump" by sending one successful message to cache the metadata before activating network blackholes.
The Illusion: Payloads larger than batch.size but smaller than buffer.memory cause no errors and are sent successfully.
The Reality: Kafka bypasses the standard, highly-efficient pre-allocated memory pool for these messages. It dynamically allocates a one-off chunk of unpooled RAM specifically for the giant payload, sends it, and instantly orphans the memory chunk. Under high throughput, this triggers severe JVM Garbage Collection spikes and CPU thrashing.
The Mechanism: BufferExhaustedException is a strict, edge-case error. It only occurs if buffer.memory is 100% full AND max.block.ms is explicitly set to 0. If max.block.ms is greater than zero, the producer politely pauses the calling thread to wait for space. If the stopwatch hits zero, it throws a generic TimeoutException instead.
The Toxiproxy CLI utilizes a consistent syntax pattern for injecting "toxics" (network degradation conditions) into active proxy routes.
ack_blackhole) used to target and delete this specific toxic later.latency, bandwidth, timeout, limit_data, slicer).latency=15000).-d applies the effect to downstream traffic (Broker to Client), while -u applies to upstream traffic (Client to Broker).kafka-proxy).| Scenario to Test | Target Configs | CLI Command | Expected Behavior |
|---|---|---|---|
| The Blackhole (Hold ACKs Hostage) |
request.timeout.msdelivery.timeout.ms |
./toxiproxy-cli toxic add -n latency_spike -t latency -a latency=15000 -d kafka-proxy |
Producer sends message instantly but waits 15s for the ACK. Fails via the 10s overarching delivery umbrella timer. |
| Buffer Exhaustion (Severe Network Choke) |
buffer.memorymax.block.ms |
./toxiproxy-cli toxic add -n choke_pipe -t bandwidth -a rate=15 -u kafka-proxy |
Producer writes to RAM faster than the network can clear it. RAM fills to 100%, thread blocks for 60s, then throws TimeoutException. |
| Idempotence Test (Silent Drops) |
enable.idempotenceacks=all |
./toxiproxy-cli toxic add -n drop_acks -t timeout -a timeout=0 -d kafka-proxy |
Broker commits data to disk but network drops the ACK. Producer retries. Broker recognizes PID/Sequence, avoids duplicating data, and re-ACKs. |
| Remove Toxic (Restore Network) |
N/A | ./toxiproxy-cli toxic remove -n latency_spike kafka-proxy |
Instantly removes the specified toxic and restores 100% healthy traffic flow to the proxy connection. |
| Inspect State | N/A | ./toxiproxy-cli inspect kafka-proxy |
Displays the active upstream/downstream mappings and lists all currently running toxics on the proxy. |