Worth 100 points. There might be slight changes to this file in the next 7 days as inevitably bugs and typos are reported. The final version will be version 1.0.
Project 1 hopefully have familiarized you with basic socket programming. Please check out my implementation (source) of project 1. I hope you can build on it or learn something from it. Project 2 is no longer "toyish." You will implement a basic subset of the BitTorrent protocol, which is one of the (two) most popular P2P file sharing protocols on the Internet. BitTorrent traffic constitutes a large fraction of modern Internet traffic. If you implement the protocol correctly, your "client" should be able to download files from other BitTorrent clients "in the wild," and there are millions of those peers.
The assignment is
to be done in groups of size at
most 2. You are responsible for finding your partner (if you need one),
for finishing off the assignment in case your partner drops the class
or backs out from completing the assignment for whatever reason. The
program
is to be written in C under a Unix platform such as Linux, SunOS,
Solaris, or FreeBSD, or even Mac OS X. Under Windows, it is possible to
develop your program with cygwin
but you must make sure that:
Makefile I
provided in my Project 1 implementation, so
that we can compile your program under these two platforms.
At least, if you don't use my Makefile
please have a Makefile that works.
Let's refer to the subset of the BitTorrent protocol that we (i.e. you
and I) will implement the UBTorrent
protocol.
It is BitTorrent with some advanced features stripped off. For
simplicity, we will also name
the program to be implemented ubtorrent, each
instance of which is called a UBTorrent peer, or
just a peer
for short. For convenience, when we talk about the
behavior of a particular peer, we will call it a (UBTorrent) client,
to distinguish it from other peers.
The "official"
BitTorrent protocol specification written by Bram Cohen is
vague is some areas. Thus, I will copy materials from
a more
detailed specification of BitTorrent 1.0. You do not need to
read the official specifications, because UBTorrent is
(hopefully) simpler.
Peers collaborate to download files. Let's say some client
has a file to be shared named "Simon
and Garfunkel - Sound of Silence.mp3". Actually,
let's
rename it "SS.mp3"
for short. The client first needs to create a metainfo
file
containing information about SS.mp3.
The metainfo file has the .torrent
extension,
and is often referred to as the "dot torrent" file of SS.mp3. The precise
format of the metainfo file will be described below. Roughly, it
contains the file name, keywords (if any), the URL of a program called
a tracker,
which is responsible for keeping track of all the peers who are
collaborating in downloading SS.mp3.
2.1 Creating the metainfo
file
In the wild, you will typically have to upload the metainfo file to
some webserver, allowing others to search for it, and download it. But
in Project 2, we will assume that every peer has the metainfo file
already, bypassing the search step. (Feel free to implement a web
search
engine, in which case you should drop out of UB right about now.) In
particular, we'll just create the metainfo file
for SS.mp3
and "give" it to each participating peer.
You can create the metainfo file using a command line
utility that Bram Cohen wrote in Python a while back, named btmakemetafile.
Here's a directory
containing the utility (and supporting files). Download, gunzip and
untar it,
you
will get a directory named MMT containing the
utility. To create a metafile for SS.mp3,
you can type something like this:
UBTorrent/MMT/btmakemetafile SS.mp3 http://127.0.0.1:6969/announceThe first argument is the file name (or path to file name if you're not running the utility in the directory where the file resides). The second argument
http://127.0.0.1:6969/announce
is called the URL
of the tracker, which basically consists of the IP
address and the port number that the tracker is running and
listening on. Note that the tracker will run the HTTP protocol, and
hence there's the http:// in the beginning.btmakemetainfo
to point to the right path of python for btmakemetainfo
to work. Type which python to get the
path.libowfat
and opentracker. You MUST
put the two directories under the same parent directory for the
compilation to work.
Then, cd into libowfat
and type make, and cd
into opentracker and type make.
Alternatively, you can just follow Dirk Engling direction on his
opentracker website:cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 coThere are two versions,
libowfat
cd libowfat
make
cd ..
cvs -d:pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co opentracker
cd opentracker
make
opentracker and opentracker.debug,
and you can run either of them. For instance,
hungngo@MYMACHINE:~/UBTorrent/opentracker$ ./opentracker.debug -i 127.0.0.1will tell it to listen to port
Binding socket type TCP to address [127.0.0.1]:6969... success.
Binding socket type UDP to address [127.0.0.1]:6969... success.
6969 (default
port) on the local interface.make reports some error, please try gmake.
A large part of this section is copied (selectively) from the
BitTorrent protocol specification. However, I have also
stripped off many requirements. When in doubt, ask me (via the blog),
don't consult the official BitTorrent protocol specification. After
all, we are implementing UBTorrent, not BitTorrent.
3.1 Bencoding
Bencoding is a way to specify and organize data in a terse format. It supports the following types: byte strings, integers, lists, and dictionaries.
3.1.1 Byte stringsByte strings are encoded as follows: <string
length encoded in base ten ASCII>:<string
data>
Note that there is no constant beginning delimiter, and no ending
delimiter.
Integers are encoded as follows: i<integer
encoded in base ten ASCII>e
The initial i and trailing e
are beginning and ending delimiters. You can have negative numbers such
as i-3e. You
cannot prefix the number with a zero such as i04e.
However, i0e
is valid.
Lists are encoded as follows: l<bencoded
values>e
The initial l and trailing e
are beginning and ending delimiters. Lists may contain any bencoded
type, including integers, strings, dictionaries, and other lists.
Dictionaries are encoded as follows: d<bencoded
string><bencoded element>e
The initial d and trailing e
are the beginning and ending delimiters. Note that the keys must be
bencoded strings. The values may be any bencoded type, including
integers, strings, lists, and other dictionaries. Keys must be strings
and appear in sorted order (sorted as raw strings, not alphanumerics).
The strings should be compared using a binary comparison, not a
culture-specific "natural" comparison.
3.1.5 You don't have to implement a bendecoder
You can use a bendecoder
written in C by Mike Frysinger here. Feel free to write your own.
After downloading the entire directory, you can test the bendecoder by
make and then try
hungngo@MYMACHINE:~/tmp/bencode$ ./test i1234e
DECODING: i1234e
int = 1234
hungngo@MYMACHINE:~/tmp/bencode$ ./test l4:spam4:eggse
DECODING: l4:spam4:eggse
list [
str = spam (len = 4)
str = eggs (len = 4)
]
hungngo@MYMACHINE:~/tmp/bencode$ ./test d3:cow3:moo4:spam4:eggse
DECODING: d3:cow3:moo4:spam4:eggse
dict {
cow => str = moo (len = 3)
spam => str = eggs (len = 4)
}
Obviously, you should look into the code to see how to use the functions.
But the decoder is there, no coding needed. This decoder is the bulk of
the metainfo command that you're supposed to implement.
3.2. Metainfo file structure
All data in a metainfo file is bencoded. The content of a metainfo file (the file ending in ".torrent") is a bencoded dictionary, containing the keys listed below. All character string values are UTF-8 encoded.
The info dictionary contains the following fields:
int err;
SHA1Context sha;
uint8_t message_digest[20]; /* 160-bit SHA1 hash value */
err = SHA1Reset(&sha);
if (err) app_error("SHA1Reset error %d\n", err);
/* 'seed' is the string we want to compute the hash of */
err = SHA1Input(&sha, (const unsigned char *) seed, seed_length);
if (err) app_error("SHA1Input Error %d.\n", err );
err = SHA1Result(&sha, message_digest);
if (err) {
app_error("SHA1Result Error %d, could not compute message digest.\n", err);
}
The tracker is an HTTP service which responds to HTTP GET requests. The requests include metrics from clients that help the tracker keep overall statistics about the torrent. The response includes a peer list that helps the client participate in the torrent. The base URL consists of the "announce URL" as defined in the metainfo (.torrent) file. The parameters are then added to this URL, using standard CGI methods (i.e. a '?' after the announce URL, followed by 'param=value' sequences separated by '&').
Note that all binary data in the URL (particularly info_hash and peer_id) must be properly escaped. This means any byte not in the set 0-9, a-z, A-Z, '.', '-', '_' and '~', must be encoded using the "%nn" format, where nn is the hexadecimal value of the byte. (See RFC1738 for details.) This is called the urlencoding of the data.
For a 20-byte hash of
\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a
the right encoded form
is
%124Vx%9A%BC%DE%F1%23Eg%89%AB%CD%EF%124Vx%9A
The parameters used in the client->tracker GET request are as follows:
GET /announce?info_hash=_tWL%26%BD%C4%BDsEn%FD%7E1%2CJ3%40s%1B& peer_id=M3-4-2--5ffd511f4079&port=6881&key=585b8345&uploaded=0&downloaded=0& left=0&compact=1&event=started HTTP/1.1A request from a leecher looks something like this (again, all on one line):
GET /announce?info_hash=_tWL%26%BD%C4%BDsEn%FD%7E1%2CJ3%40s%1B& peer_id=M3-4-2--f8c7df8ee1b9&port=6882&key=a6fa9e7e&uploaded=0&downloaded=0 &left=90112&compact=1&event=started HTTP/1.1Note: 90112 is the size of SS.mp3. A GET request typically consists of the GET request line (as shown above) and an empty line. The last two characters of an HTTP request line must be
\r\n.
The "empty" line must consist of two characters \r\n.
Those are "carriage-return" and "line-feed" characters.
3.3.2 Tracker Response
The tracker responds with "text/plain" document consisting of a
bencoded dictionary with the following keys:
The list of peers is length 50 by default. If there are fewer peers in the torrent, then the list will be smaller. Otherwise, the tracker randomly selects peers to include in the response. The tracker may choose to implement a more intelligent mechanism for peer selection when responding to a request. For instance, reporting seeds to other seeders could be avoided. Since we are not implementing the tracker, we just take whatever the tracker gives.
Clients may send a request to the tracker more often than the specified interval, if an event occurs (i.e. stopped or completed) or if the client needs to learn about more peers. However, it is considered bad practice to "hammer" on a tracker to get multiple peers.
A reply from the tracker may look something like this:HTTP/1.0 200 OK Content-Length: 27 Content-Type: text/plain Pragma: no-cache d8:intervali1800e5:peers0:eIn general, the reply from the tracker follows the format specified here. You should print out the first line of the response. If the status code (e.g. the number 200 above) is anything other than 2xx, then something is wrong, don't read and parse the content. The content of the reply is everything after the blank line. Note: if you didn't do anything special (like having a persistent-connection request along with the GET line), the tracker will close the connection right after sending the reply. Don't be surprise! You should try
telnet localhost 6969 after running opentracker,
and then cut-and-paste a sample request shown in the previous section
to see the tracker's response.
Important note:
The readline() function provided in the sample codes
can be used, but it has to be used with extreme care, because it can read
past the EOL character (for efficiency). Please read the code before
using it!
3.4. Peer Wire Protocol
(over TCP)
3.4.1
Overview
The peer protocol facilitates the exchange of pieces as described in the metainfo file. To request a piece, a client will actually send multiple requests for parts of the piece, called blocks. A block size is typically divisible by the piece size. It is recommended to set the default block size to be 16KB = 214 = 16384 bytes. Feel free to set the default block size to be equal to the piece size if you want. However, since the last piece may not have full size, some block requests will inevitably have to be of small sizes.
A client must maintain state information for each connection that it has with a remote peer:
Note that this also implies that the client will also need to keep track of whether or not it is interested in the remote peer, and if it has the remote peer choked or unchoked. So, the real list looks something like this:
Client connections start out as "choked" and "not interested". In other words:
A block is downloaded by the client when the client is interested in a peer, and that peer is not choking the client. A block is uploaded by a client when the client is not choking a peer, and that peer is interested in the client. It is important for the client to keep its peers informed as to whether or not it is interested in them. This state information should be kept up-to-date with each peer even when the client is choked. This will allow peers to know if the client will begin downloading when it is unchoked (and vice-versa).
3.4.2 Data typeUnless specified otherwise, all integers in the peer wire protocol are encoded as four byte big-endian values. This includes the length prefix on all messages that come after the handshake.
3.4.3 Message flowThe peer wire protocol consists of an initial 2-way handshake. After that, peers communicate via an exchange of length-prefixed messages. The length-prefix is an integer as described above.
3.4.4 HandshakeThe handshake is a required message and must be the first message transmitted by the client. It is (49+len(pstr)) bytes long.
handshake: <pstrlen><pstr><reserved><info_hash><peer_id>
The initiator of a connection is expected to transmit their handshake immediately. The recipient may wait for the initiator's handshake, if it is capable of serving multiple torrents simultaneously (torrents are uniquely identified by their info_hash). However, the recipient must respond as soon as it sees the info_hash part of the handshake. If a client receives a handshake with an info_hash that it is not currently serving, then the client must drop the connection.
If the initiator of the connection receives a handshake in which the peer_id does not match the expected peer_id, then the initiator is expected to drop the connection. Note that the initiator presumably received the peer information from the tracker, which includes the peer_id that was registered by the peer. The peer_id from the tracker and in the handshake are expected to match.
3.4.5 MessagesAll of the remaining messages in the protocol take the form of <length prefix><message ID><payload>. The length prefix is a four byte big-endian value. The message ID is a single decimal byte. The payload is message dependent.
The keep-alive message is a message with zero bytes, specified with the length prefix set to zero. There is no message ID and no payload. Peers may close a connection if they receive no messages (keep-alive or any other message) for a certain period of time, so a keep-alive message must be sent to maintain the connection alive if no command have been sent for a given amount of time. This amount of time is generally two minutes.
The choke message is fixed-length and has no payload.
The unchoke message is fixed-length and has no payload.
The interested message is fixed-length and has no payload.
The not interested message is fixed-length and has no payload.
The have message is fixed length. The payload is the zero-based index of a piece that has just been successfully downloaded and verified via the SHA1 hash.
Implementer's Note: That is the strict definition, in reality some games may be played. In particular because peers are extremely unlikely to download pieces that they already have, a peer may choose not to advertise having a piece to a peer that already has that piece. At a minimum "HAVE suppression" will result in a 50% reduction in the number of HAVE messages, this translates to around a 25-35% reduction in protocol overhead. At the same time, it may be worthwhile to send a HAVE message to a peer that has that piece already since it will be useful in determining which piece is rare.
A malicious peer might also choose to advertise having pieces that it knows the peer will never download. Due to this attempting to model peers using this information is a bad idea.
The bitfield message may only be sent immediately after the handshaking sequence is completed, and before any other messages are sent. The bitfield needs not be sent if a client has no pieces.
The bitfield message is variable length, where X is the length of the bitfield. The payload is a bitfield representing the pieces that have been successfully downloaded. The high bit in the first byte corresponds to piece index 0. Bits that are cleared indicated a missing piece, and set bits indicate a valid and available piece. Spare bits at the end are set to zero. A bitfield of the wrong length is considered an error. Clients should drop the connection if they receive bitfields that are not of the correct size, or if the bitfield has any of the spare bits set.
The request message is fixed length, and is used to request a block. The payload contains the following information:
The piece message is variable length, where X is the length of the block. The payload contains the following information:
The cancel message is fixed length, and is used to cancel block requests. The payload is identical to that of the "request" message. It is typically used during "End Game" (see the Algorithms section below).
3.5 Algorithms
3.5.1 Piece downloading strategy
You're probably overwhelmed by the above description. However, keep in
mind that you can use existing codes for many important functionalities
have been implemented
already: the SHA1 hash, the bendecoder, the tracker, etc. I will
simplify things further by requiring you to implement clients which
support only one single .torrent
file, given as an input to the program ubtorrent.
It is not difficult
to extend the implementation to support multiple torrents, and you are
encourage
to do so in your spare time after this course is over. In fact, run
your implementation
"in the wild" to see how it measures up to other professional ones.
Your ubtorrent is supposed to implement all
of the above requirements though.
The sequence of tasks looks something like this:
opentracker
program. Run it.
ubtorrent, you probably
should write pieces down
to disk, don't keep everything in memory.
1,
2, 3, 4, 5 and so on.
Then put SS.mp3 in one of them. And, put a copy
of ubtorrent in each
of them. Thus, the copy of ubtorrent running
inside directory 1
will be the original seeder. Other copies of ubtorrent
can write temporary files
inside that local directory. Clean up the temp files after the
downloading is done, or
if the user types quit
ubtorrent takes two parameters:
the first argument is the .torrent file name, and
the second argument
is the port number that other peers can connect to. For example,
ubtorrent SS.mp3.torrent 6881
ubtorrent
takes commands from users. I will be more precise on the output formats of these commands in the next few days.
ubtorrent> metainfo
my IP/port : 127.0.1.1/9999
my ID : bcd914c766d969a772823815fdc2737b2c8384bf
metainfo file : Dan Hill & Rique Franks - Sometimes When We Touch.mp3.torrent
info hash : 4a060f199e5dc28ff2c3294f34561e2da423bf0b
file name : Dan Hill & Rique Franks - Sometimes When We Touch.mp3
piece length : 262144
file size : 1007616 (3 * [piece length] + 221184)
announce URL : http://192.168.0.1:9999/announce
pieces' hashes:
0 064b493d90b6811f22e0457aa7f50e9c70b84285
1 d17cb90e50ca06a651a84f88fde0ecfb22a90cca
2 20e82d045341032645ebe27eed38103329281175
3 568c8a0599a7c1e2b3c70d8b8c960134653d497a
ubtorrent>
Everything is in the metainfo file or available locally (IP/port),
except for the info hash
value which you have to compute. (This has to be computed anyhow in order
to send tracker requests.) The value of info hash is the SHA1
hash of the value of the info key in the metainfo dictionary.
Like in Project 1, your program should handle errors graciously. If the input .torrent file does not exist or if there were some errors in reading the file, for example, please do not crash. You can assume that the metainfo file (that we will test) is not greater than 8KB = 8192 bytes. If the given metainfo file is larger than that, you can quit the program and complain (to the screen). In the wild, metainfo files are often kept small, not more than 50KB.
Now, sometimes for debugging purposes
you might want to display the metainfo file to the screen.
However, as there are hash values in it, many characters cannot be displayed,
causing the terminal to behave in unpredictable manner (e.g., change to Korean
or Chinese fonts or something). To be safe, you should use hexdump
to display a metainfo file:
hungngo@MYMACHINE:~/tmp$ hexdump test.torrent -c 0000000 d 8 : a n n o u n c e 1 5 : h t 0000010 t p : / / t e s t / a b c 1 3 : ...I cut the rest of the output. You may find the functions
strtol()
helpful when trying to get the value of the info key out.
4.2. announce
This command asks the client to send a GET request to the tracker. When
the response comes back, update the information to be used later.
It also prints out the status line of the response (the first line which
looks something like HTTP1.0 200 OK).
The output may look like this:
ubtorrent> announce ++++ Tracker responded: HTTP/1.0 200 OK complete | downloaded | incomplete | interval | min interval | --------------------------------------------------------------- 1 | 0 | 0 | 1686 | 843 | --------------------------------------------------------------- ++++ Peer List : IP | Port ------------------------------- 127.0.0.1 | 9999 ... Tracker closed connection(I suggest you store the dictionary in some
be_node.
Remember to be_free the node the next time
an announce is called. The be_node
is used in processing the command trackerinfo.)
4.3. trackerinfo
Prints the information contained in the latest tracker response.
This is a subroutine which should have been called when
announce is typed. So, the output is just
part of the (last sucessful)
announce output, without the status line:
ubtorrent>trackerinfo complete | downloaded | incomplete | interval | min interval | --------------------------------------------------------------- 1 | 0 | 0 | 1686 | 843 | --------------------------------------------------------------- ++++ Peer List (self included): IP | Port ------------------------------- 127.0.0.1 | 9999 127.0.0.1 | 9998 127.0.0.1 | 9997
4.4. show
Prints the status of all current peer connections, including the choked/interested states and upload/download rates. The format looks something like this:
ubtorrent> show ID | IP address | Status | Bitfield | Down/s | Up/s | -------------------------------------------------------------------- 0 | 127.0.0.1 | 0101 | 1111111111111111 | 0 | 59271 1 | 127.0.0.1 | 0101 | 1111111111111111 | 0 | 59271
Down/s is the number of bytes downloaded per second on that
connection.
The status field has 4 bits, corresponding to the status
of that connection: iam_choking (peer),
iam_interested (in peer),
peer_choking (me),
peer_interested (in me).
4.5. status
Prints the status of the current download: which pieces have been
downloaded, which pieces are missing, the number of downloaded, uploaded,
and remaining bytes. The format is something like this:
ubtorrent> status Downloaded | Uploaded | Left | My bit field ------------------------------------------------ 1007616 | 0 | 0 | 1111111111111111or
ubtorrent> status Downloaded | Uploaded | Left | My bit field --------------------------------------------------- 0 | 0 | 1007616 | 0000000000000000