Beer reviews and other ramblings

Technology

Running Out Of Memory-Mapped Files with Apache Kafka

Running Out Of Memory-Mapped Files with Apache Kafka

This was a fun one to track down. Long story short, because of the large amount of partitions (and topics) we had, the number of Kafka segment files exceeded the number of memory-mapped files available to the JVM, which then crashed. However, it crashed only […]

Monitoring Kafka Consumer Lag in Seconds

Monitoring Kafka Consumer Lag in Seconds

In this post I’ll talk about how we’re monitoring Kafka consumer lag using the same metric as our Service Level Objective: lag in seconds, and why that’s important. The logging stack we built at work (ZipRecruiter) uses Apache Kafka as a buffer for logs, which […]

Niceties missing from Apache Kafka

Niceties missing from Apache Kafka

Kafka is awesome, but it’s not new. It’s a technology that has been gaining traction for some time, and has been adopted by many of the biggest names in the tech world. But it doesn’t have all the bells and whistles of newer “modern” technologies. […]