All work

BitTorrent Client

A command-line BitTorrent downloader built without a networking framework: raw epoll and eventfd for the peer connections, libcurl for HTTP trackers, and OpenSSL only for SHA-1. Given a .torrent file, btdl announces to the trackers, connects to peers, downloads pieces rarest first, verifies each one, and writes it into the right place across one or more files.

Role
Independent - design & implementation
Stack
C++20, Linux epoll, libcurl, OpenSSL, Catch2, CMake
Status
Personal project · download only, no seeding yet

How it works

One event loop owns the swarm

A single reactor thread runs an epoll loop over every peer socket. It owns the peer table, the piece bookkeeping, and the decision of whom to dial, so none of that needs a lock. Other threads reach it only by posting a task, which wakes the loop through an eventfd.

The slow work runs elsewhere. A tracker thread makes the blocking HTTP and UDP announces and hands new peers to the loop. Two storage workers take whole pieces off a queue, hash them, and write them with pwrite. Each result is posted back to the loop.

Peers are asked for 16 KiB blocks, up to 10 in flight per peer. Pieces are chosen rarest first, and pieces already started are finished first.

Architecture

BitTorrent client architecturebtdl parses the .torrent file with bencode and computes the info-hash as SHA-1 over the exact info bytes, then blocks in run() until the download is done. A tracker thread runs TrackerManager, which announces to HTTP trackers through libcurl and UDP trackers per BEP 15 across BEP 12 tiers, and hands new peers to the reactor through Reactor::post, a task queue that wakes the loop with an eventfd. The reactor thread runs one epoll loop with a 500 ms tick and owns every PeerConnection, up to 30, each with up to 10 requests of 16 KiB in flight to remote peers over outgoing TCP, and the PieceManager, which picks pieces rarest first with an endgame fallback. Finished pieces go to a mutex and condition variable queue, where two storage workers verify each piece with SHA-1 and write it with pwrite across the output files, then post the result back to the reactor. The reactor tells the caller thread when every piece is verified..torrent filebencode · SHA-1 info-hashCaller threadbtdlBlocks in run() until doneTracker threadTrackerManagerHTTP · UDP · tiersBackoff, jitterTrackersHTTP, UDPReactor::postTask queue + eventfdReactor threadepoll loop500 ms tickPeerConnectionsUp to 30 · 10 in flightPieceManagerRarest first · endgamePeersTCP outPiece queuemutex + condvarStorage workers × 2SHA-1 + pwriteVerify, then write across filesOutput filesSized up front1234
Data and control pathResult posted back to the loop
  1. 1One owner. The reactor thread owns every peer, the piece bookkeeping and the dial policy. Other threads can only post tasks to it.
  2. 2Endgame. When nothing fresh is left, an in-flight block can go to up to 3 peers, and the others get a cancel on the next 500 ms tick.
  3. 3Verify, then write. Workers hash each full piece against the torrent before writing it. A mismatch resets the piece so it is fetched again.
  4. 4Trackers. Announces run on their own thread with jitter, capped backoff and BEP 12 tiers, so a slow tracker doesn’t stall the loop.
btdl on the official Debian 13.7 netinst torrent, recorded in real time for 40 seconds. It reached 1,166 of 3,024 verified pieces. Speed and peer count depend on the swarm and the connection.

Protocol

What it speaks

Each item below is implemented and covered by tests.

  • The peer wire protocol (BEP 3): handshake and all nine standard messages, decoded incrementally so fragmented reads are handled
  • HTTP and HTTPS tracker announces and scrapes (BEP 3, BEP 48), with compact IPv4 and IPv6 peer lists (BEP 23)
  • UDP trackers (BEP 15): connect, announce, and scrape, with transaction-id checks and a cached connection id
  • Multi-tracker tiers (BEP 12), with announce jitter and capped backoff when a tracker fails
  • .torrent files, single and multi-file, with the info-hash computed over the exact original bytes

Decisions

Three choices that shape the client

Hashing the bytes as they arrived

A torrent’s identity is the SHA-1 of its bencoded info dictionary. The parser records the exact byte range of that dictionary and hashes those original bytes, never a re-encoding, so a torrent with non-canonical encoding still gets the hash its swarm uses.

Lock-free by ownership

Swarm state never leaves the reactor thread, so it needs no locks, and the piece manager is deliberately not thread-safe. The CPU-heavy hashing and the disk writes run on a worker pool that reports back by posting to the loop. A peer that closes inside its own callback is parked and destroyed on the next loop iteration, so nothing is freed while it’s still on the stack.

A bounded endgame

When no fresh blocks are left, a block still in flight can be requested from up to 3 peers at once. Once one delivers it, the others get a cancel on the next 500 ms tick. One slow peer can’t hold the last pieces back, and the wasted bandwidth is capped.

The tradeoff

The client only downloads. It never unchokes anyone, doesn’t accept incoming connections, and can’t resume a partial download. Size caps keep the parsers small: 4 MiB per peer message and 2 MiB per tracker response, which would reject an oversized but legal edge case. Every cross-thread interaction has to be marshalled through the loop.

Technical deep dive

Storage

Files are created at full size up front. Each piece is mapped onto the file spans it covers, including pieces that straddle a file boundary, and written with positional writes, so the workers never share a file offset.

A piece whose hash doesn’t match is reset and requested again from the peers that have it.

Trackers

Announces respect the tracker’s minimum interval, add ±20% jitter, and back off up to a cap after failures. UDP replies are accepted only with a matching transaction id, with one 15-second deadline per call, and the connection id is reused for 60 seconds.

Timeouts

Peers have connect and handshake timeouts of 10 seconds, an idle timeout, and a 20-second stall timeout. A 500 ms timer on the loop drives timeouts, cancels, and the progress line.

Testing

16 test targets: 116 Catch2 test cases plus 67 data-driven bencode cases. They pass under AddressSanitizer and UBSan, and the build has no warnings with -Wall -Wextra -Wpedantic. A ThreadSanitizer build runs the same suite with no data-race reports.

The end-to-end tests build a real torrent in memory and download it with the full client from fake seeders over localhost. They cover multi-file downloads from three seeders, a corrupt piece being re-fetched, a seeder disconnecting mid-piece, and a clean stop.

Status

What works and what doesn’t yet

btdl has downloaded and verified a real torrent from a public test tracker. The parts below are split by what exists today.

Working

  • Tracker announces over HTTP, HTTPS and UDP
  • Downloading from many peers at once, rarest first, with an endgame
  • SHA-1 verification and re-download of bad pieces
  • Single and multi-file torrents written to disk
  • btdl, with a live progress line

Not built yet

  • Seeding and incoming connections
  • Resuming a partial download
  • Magnet link downloads (links are parsed, but metadata exchange isn’t implemented)
  • DHT and peer exchange

Source

    GitHub repository