BitSpeed Transfer 0.5 Sends Whole Folders and Several Chunks at Once

Until now Transfer moved exactly one file, and it sent that file one chunk at a time. Both were deliberate. One file kept the recovery rules small enough to test properly, and one chunk at a time meant the receiver never had to think about two writes at once. But they were also the two best reasons to reach for rsync or rclone instead. Transfer 0.5.0 drops both limits without loosening what a chunk acknowledgement means.

Point --source at a folder and Transfer packs it into one payload, which we call a tree: every file’s bytes in path order, then an index listing each path with its size and SHA-256, then a short footer. To the protocol it’s just a file. Resume, the per-chunk checks and the final receipt all work the same, and because the receipt’s digest covers the index, one receipt now vouches for every file in the folder.

On the receiving side there’s a new command, unpack. It takes a finished transfer and writes the folder wherever you point it. Before the folder appears, it checks each file against its index entry and the whole payload against the receipt. It writes into a hidden folder first and renames it into place at the end, so you never get half a folder. It won’t write over an existing folder or into the receiver’s own storage, and it only reads that storage, so you can run it while the receiver keeps serving.

A real folder, start to finish

This is the published 0.5.0 Linux binary, a folder of six raw frames and a notes file, 18.9 MB in 4 MiB chunks over loopback:

$ bitspeed-transfer send --prepare-only --source=photos-2026 --spool=sender
"tree": {"files": 7, "folders": 3, "bytes": 18895379}

$ bitspeed-transfer send --spool=sender --reuse=ID --url=http://127.0.0.1:18080 --token-file=token
receiver has 0 of 5 chunks verified; sending 5 (18896527 bytes)
sent 1 of 5 chunks, 4194304/18896527 bytes (22%)
...
all 5 chunks verified; receiver is checking and publishing the file

$ bitspeed-transfer unpack --root=receiver --id=ID --out=received
{"unpacked":true, ... "files":7,"folders":3,"bytes":18895379}

diff -r between the original and the received folder came back empty. The 1,148 bytes between the file total and the payload size are the index and the footer.

Some things trees don’t do, on purpose. Symbolic links, devices and other special files stop preparation with an error naming the path, rather than being skipped quietly. Permissions, owners and timestamps aren’t carried. Transfer watches the folder while it copies, and a file that changes size, a file that appears or one that disappears fails the snapshot. Like any copy tool, it can’t catch a program that rewrites a file in place and puts the old timestamp back, so prepare a folder nobody is writing to.

Several chunks in flight

Send now keeps four chunk uploads going at once by default, and --parallel sets anything from 1 to 8. It matters most when the round trip is long. With one chunk at a time, the line sits idle every time the sender waits for the receiver to say “stored”.

To measure that, we ran the downloaded binary through a local proxy that waits 100 ms before passing on each request, standing in for a long link. A 32 MiB file in 1 MiB chunks, three runs each:

--parallel=1   4.44 s   4.45 s   4.49 s
--parallel=4   1.81 s   1.82 s   1.78 s
--parallel=8   1.46 s   1.96 s   1.35 s

That’s one machine and a simulated delay, not a WAN benchmark, and eight in flight was noisier than four. The full numbers are in the project’s recorded results.

What didn’t change is the order of operations for each chunk. The receiver still writes it to a temporary file, checks its length and SHA-256, syncs it, reads it back, links it into place and syncs the folder. Only then does it answer. The difference is that the slow part, the bytes crossing the network, no longer holds a lock, so several chunks can arrive at once. Progress still counts only chunks the receiver has confirmed. A receiver older than 0.5.0 turns away the extra uploads as busy, and the new sender takes the hint and finishes one chunk at a time.

The independent review before release earned its keep again. It found one real hole in the first version. A re-sent chunk was checked against the copy already on disk when it arrived, but not again when it finished. If a status check moved that stored copy aside as corrupt in between, the re-send could be acknowledged while nothing valid sat on disk. Every re-sent chunk now goes to its own temporary file, and the receiver looks again just before it answers. A second review round found nothing to fix.

The Transfer guide and Quick Start have the new commands, and the download page has the 0.5.0 archives. Inspect is unchanged at 0.3.0.