Running Out Of Memory-Mapped Files with Apache Kafka
This was a fun one to track down. Long story short, because of the large amount of partitions (and topics) we had, the number of Kafka segment files exceeded the number of memory-mapped files available to the JVM, which then crashed. However, it crashed only after checksumming and mapping terabytes of data, which takes more than an hour. It’s always fun to try to fix things which take over an hour to test.
The error the JVM returns when this happens is a general one: “There is insufficient memory for the Java Runtime Environment to continue.” Of course I tried increasing the available heap space as the error suggests (multiple times), but that didn’t fix the problem. In the end, a coworker was looking at the mmap man page, and it clicked that we’re probably running out of available mapped files (thanks, Tye).
So, this is here with the hope that someone in need will find it, and they won’t have to spend hours tracking it down.
[2018-02-02 17:47:09,328] INFO [Transaction Coordinator 2]: Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
[thread 140444402059008 also had an error]
# There is insufficient memory for the Java Runtime Environment to continue.
# An error report file with more information is saved as:
Allow the JVM to use more mmap pointers by increasing the value of
$ sysctl -w vm.max_map_count=262144
Add the above option to
/etc/sysctl.conf to set the value permanently. You can verify the setting is correct after a reboot with:
$ sysctl vm.max_map_count
Did this fix your problem?
Please leave a comment about your experience. Confirmation about problems like this can go a long way toward helping other people in the future.