Menu

Post image 1
Post image 2
1 / 2
0

Mounting tar archives as a filesystem in WebAssembly

ʕ•ᴥ•ʔ Notes from Jeroen·@HashtagPLUS·6 months ago
#cOg6qCeq
#wasm#emscripten#tar#vfs#id2#file
Reading 0:00
15s threshold

TLDR : instead of extracting a .tar.gz archive, we can generate a small index file which lists the size and offset of each file in the tar, and use this metadata to mount the tar blob directly via Emscripten’s WORKERFS without any copying. For details see: https://github.com/jeroen/tar-vfs-index The struggle with tarballs Lots of data on the internet lives in tarballs, often distributed as gzipped .tar.gz files. To get to this data, we have to download the entire .tar.gz file, decompress it, and then iterate through the blob from beginning to end to make copies of the files we need. This is expensive and painful in memory constrained environments. A while ago we came up with a cool optimization for WebR (the wasm port of R) that lets us mount contents from a .tar.gz archive without copying by using a metadata file which indexes the size and offset of each file within the tar blob.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More