Menu

1 / 2

I built a log compressor you can grep without decompressing

DEV Community: rust·Zayd Mulani·9 days ago
#RALpt7ai
#devtorust#devto#dev#logjar#line
Reading 0:00
15s threshold

Logs pile up. You compress them to keep them, and then the one time you need an answer, you sit there while zgrep inflates gigabytes. The other option is a search cluster that indexes everything and ends up bigger than the logs.

I wanted something in between: archive-sized on disk, searchable in place, nothing to run. So I built logjar , a Rust library and single binary.

$ logjar pack -o hdfs.ljar HDFS.log # 1.5 GiB -> 52 MiB $ logjar grep -c "blk_-1608999687919862906 " hdfs.ljar 268 # 0.9s

Enter fullscreen mode

Exit fullscreen mode

The same search with gzip -dc | grep takes 7.9 seconds.

The idea: most of a log line was written by printf

Look at a line from Hadoop's HDFS log:

081109 203615 148 INFO dfs.DataNode$PacketResponder: Received block blk_-1608999687919862906 of size 91178 from /10.250.10.6

Enter fullscreen mode

Exit fullscreen mode

Most of that is fixed text from one print statement. Only a few values change.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.