Let’s explore how a file-format design exercise turned into reinventing the wheel, and let’s all appreciate how lucky we are that the protocols that the internet are built on have fantastic public documents called RFCs.
I was idly thinking about designing a good binary file format for a side project of mine called qrxfil. The project aims at QR-code-based file export: split a file in many pieces (based on content limit of QR codes) before reassembling it on the other side.
The more I thought about it, the more I realized that I’ve seen this pattern before: fragmentation of a “message” to push it into “packets” of small, limited, size, sent over an unreliable network, for reassembly on the other side. That’s a foundational feature of the Internet Protocol!
It turns out the internet’s protocols were documented publically, with published specifications for protocols called RFCs. These are organised by a group called the Internet Engineering Task Force (IETF), since about the 1970s.
RFCs are defined by their number, and are written in plain text1, originally because of relying on typewriters. These documents cover a broad number of topics, from recommendations like “Choosing a Name for Your Computer” (RFC 1178), specifications like “Date and Time on the Internet: Timestamps” (RFC 3339), to the downright silly “IP Datagrams over Avian Carriers” April fools’ jokes (RFC 1149, later extended by RFC 2549 and RFC 6214).
As technical documents, these are very important, as they are the authoritative source specificying many protocols. Surprisingly for such foundational papers, though, these are also very accessible explanations, useable by both technically-minded users, of course, but also for many curious minds. In fact, many times, answers to questions on StackOverflow end up being direct quotes of entire sections the RFC of the relevant protocol.
Back to our file format design, I wanted to look up how the internet (as a concept) deals with packet fragmentation, which is covered by the Internet Protocol, codified in RFC 791, published in September 19812.
You may notice that being such an old document (like the FTP specification of RFC 959, from 1985) its layout isn’t as clear as the newer RFCs. That’s because the tooling for writing up RFCs nowadays deals with layout and metadata in a standard way, whereas these older protocols are pre-dating the tooling, so these pages are forged from digital copies of the original typewriter texts. I personally find this amazing: a tangible link to the history of the internet!
Opening RFC 791: INTERNET PROTOCOL, the following motivation awaits us:
The Internet Protocol is designed for use in interconnected systems of packet-switched computer communication networks. Such a system has been called a “catenet” [1]. The internet protocol provides for transmitting blocks of data called datagrams from sources to destinations, where sources and destinations are hosts identified by fixed length addresses. The internet protocol also provides for fragmentation and reassembly of long datagrams, if necessary, for transmission through “small packet” networks.
Indeed, it talks of fragmentation, “for transmission through small packet networks”. That’s exactly what I’m interested in, only for files.
Skimming a bit more, we get to section 2.4: Functional Description, which is split into two concerns: Addressing and Fragmentation. Looking exclusively at Fragmentation, I learned that IP packet don’t use “page numbers”, like I was expecting (where one would send both current page number, and total number of pages to expect, with overall content as a book split by page, regardless of page size). Instead, the IP datagrams define an “identification field”. This field marks all fragments to be from the same original message (distinguishing this overall message from others that may be in the air at the same time. This can be seen as a book identifier. Other dedicated fields gives byte offset and fragment size combo to mark which byte-ranges of the original message is included in current message. This would allow for dynamically sized fragment! Fascinating3!
But if we can keep sending more fragments covering new byte ranges, how do we know the message is complete?
Turns out, there’s a flag called more-fragments, which every subsequent message fragments after the first should set, and the last message should re-set, marking the end of the datagram.
On receiving the last fragment (determined by unset more-fragments flag), the message can be reassembled.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+For the purposes of a non-networked system, like out-of-order scanning of QR codes containing file fragments (my real goal), the idea of a “more-fragments flag” doesn’t work out, but it’s an interesting idea to consider that, for instance, there doesn’t seem to be a “reassembled datagram size” field anywhere in the messages, which I expected to find repeated over and over in each fragment. Instead, there’s per-fragment field called “Total Length”, a “Fragment Offset”, and the rest is computed once receiving the last fragment, knowing its offset + length. This allows for more dynamic messages, good idea!
This document is interesting beyond just fragmentation. I recommend the explanation of the concept of TTL (Time To Live), which is described helpfully as a sort of “self destruct time limit”. Tons of fun to find in these.
Without going deeper into the design of my pet project’s weird file format, I’m amazed at the tremendous amount of teaching value these “old papers” can provide. Not only they specify the protocols, but they also document motivations, reasoning, decisions and tradeoffs that engineers before us wrestled with to make a good system.
I feel that to level up our collective software design ability, we must learn all we can from existing attempts at building systems, instead of risking re-inventing the wheel like I almost did. This way, we can, like in Science, “Stand on the shoulder of giants”. RFCs are an underappreciated piece of our field’s history, as well as monumental technical achievements on their own, and we would all gain from using them more.
For modern usecases, the RFC documents are also published as HTML and PDF, but I’m personally very fond of plain-text files. They may be the most long-lasting file format we’ve ever come up with: Simple characters in a file, perfect for humans. In that spirit, all the links to RFCs in this post are to the text file renderings. Just note that you can also get them as HTML at https://datatracker.ietf.org/
[return]The specific date is muddled, because a lot of these original documents were more of a post-facto documentation of already implemented systems, rather than pre-implementation proposals, so the date is of publication not invention. The scope of RFCs is also usually very constrained, so in case of large specs, there are usually multiple documents on each subtopic, covering different aspects, all published at different dates, like for extensions to a protocol for new usecases.
[return]Networking experts may laugh at this being quite obvious, but coming from high level programming background, it can sometimes be hard to find a handhold in the lower levels of programming, such as protocol design. I believe any opportunity to reveal forgotten Knowledge like this one should be praised, a chance learn!
[return]