Menu

Parallel reads made my Rust file transfer tool 4x slower on hard drives
📰

Parallel reads made my Rust file transfer tool 4x slower on hard drives

DEV Community: rust·YourNeelsOG·about 9 hours ago
#cYVX1uoX
#devtorust#devto#dev#disk#hopsync
Reading 0:00
15s threshold

A 1.18 TB copy between two seedboxes in the same data centre took 14.5 hours . The network between them did 8 Gbit/s on a single iperf3 stream. The ping was 0.29 ms. My tool averaged 21.6 MiB/s.

The tool is Hopsync , an open-source Rust service I'm building for moving files between SSH servers from a web UI or a CLI. The other side only needs a normal SSH account with SFTP. Its main trick is parallel "lanes": several SSH connections, with big files split across them. Over a 200 ms link that roughly doubles scp's speed, and on 300 small files it finishes in 39 s where scp takes 249 s.

On two hard drives sitting next to each other, the same trick made it slower than rclone. Here's why, and what fixed it.

Measure the ceiling first

Before touching code I wanted to know what "fast" meant on these boxes.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.