Menu

1 / 3

HDF5 in Pure Go, No CGO: a 5 KB File That Allocated 40 GB, and Round-Trip Tests That Lied

DEV Community: go·Andrey Kolkov·5 days ago
#3e3UJ8TM
#devtogo#devto#dev#hdf5#library
Reading 0:00
15s threshold

In June 2025 I was porting SURD , a causal-discovery method from a Nature Communications paper, from its Python reference to Go ( causalgo ). The reference data came as MATLAB .mat files. In Python that's one scipy.io.loadmat call. In Go I found nothing I could rely on.

Those files happened to be the older v5 format. But a v7.3 .mat file, the format MATLAB needs for any variable of 2 GB or more, is an HDF5 file, so a Go reader that only handled v5 would fail on the first large dataset. And HDF5 itself is how a large part of science stores its data: climate models, telescopes, particle physics. In Go there were cgo bindings to the C library and a few pure-Go attempts, abandoned years ago and far from complete.

If I was going to write these libraries at all, it made sense to make them universal from the start. So four days later I started two repositories instead of one narrow reader: scigolib/matlab and scigolib/hdf5 , a full HDF5 library, read and write, in pure Go with no cgo.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.