Logs pile up. You compress them to keep them, and then the one time you need an answer, you sit there while zgrep inflates gigabytes. The other option is a search cluster that indexes everything and ends up bigger than the logs.
I wanted something in between: archive-sized on disk, searchable in place, nothing to run. So I built logjar , a Rust library and single binary.
$ logjar pack -o hdfs.ljar HDFS.log # 1.5 GiB -> 52 MiB $ logjar grep -c "blk_-1608999687919862906 " hdfs.ljar 268 # 0.9s
Enter fullscreen mode
Exit fullscreen mode
The same search with gzip -dc | grep takes 7.9 seconds.
The idea: most of a log line was written by printf
Look at a line from Hadoop's HDFS log:
081109 203615 148 INFO dfs.DataNode$PacketResponder: Received block blk_-1608999687919862906 of size 91178 from /10.250.10.6
Enter fullscreen mode
Exit fullscreen mode
Most of that is fixed text from one print statement. Only a few values change.…