Communications of the ACM
Cerf's up
In Praise of Under-Specification?
Eric Schmidt, executive chairman of Alphabet (Google's parent company), recently drew my attention to the notion of "under-specification." He reminded me that the Internet had benefited strongly from this concept. Several specific examples came to mind. The Internet Protocol (IP) specification does not contain any information about routing. It specifies what packets look like as they emerge from or arrive at the hosts at the edge of the Internet, but routing is entirely outside of that specificationa partly because it was not entirely clear what procedures would be used for Internet routing at the time the specification was developed and, indeed, a number of them have been developed over time. There is nothing in the specification that describes the underlying transmission technology nor is there anything in the specification that speaks to how the packet's payload (a string of bits) is to be interpreted. These matters are open to instantiation independent of the specification of packet formats.
Some of the under-specification can be a manifestation of layering that figured strongly in the ARPANET host-host protocols and was carried over in the Internet Protocol suite. The idea is that while there is a well-defined interface between the layers that specifies how information crosses the layer boundary, the details of the layer above or below are hidden. This feature allows for changes in the implementation of and even the characteristics of the upper or lower layer. For example, above the IP layer, one finds a number of different protocols such as User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) or Real-Time Protocol (RTP) that all send and receive Internet packets but they use and interpret the IP packet payloads in different ways. Below the IP layer one finds a variety of different transmission technologies including Ethernet, Multi-protocol Label Switching, Frame Relay, Asynchronous Transfer Mode, Dense Wave-Length Division Multiplexing, and many others. The IP layer doesn't really care how the packets are transported.
What is interesting to contemplate is whether the notion of under-specification that induces flexibility and anticipates new but unknown developments can be codified in a concrete way beyond the purely conceptual. Is there a way to measure the degree of specification in the way that Claude Shannon found to specify information as entropy independent of semantics? Can something be fully specified, partly specified, or completely unspecified and how would these be described or measured more precisely? In circuit design, for instance, there is the notion of "don't care" for some values in a Boolean representation. Can this notion be applied to program specification as well as to protocol specifications? Are there design principles that one can derive from this notion of under-specification?
I am reminded of an anecdote told about doing business with Chinese manufacturers. American companies produced very detailed specifications of what was to be fabricated down to the last detail and the Chinese companies produced exactly what was asked, at a price. But a Chinese company produced a less specific specification, leaving room for the manufacturer to innovate, leading to a design that was less expensive, easier to manufacture, and to maintain.
One of my oldest friends, Jonathan Postel, was the Internet Assigned Numbers Authority for many years and was often quoted: "Be liberal in what you accept and conservative in what you send," in reference to the implementation of protocols. His dictum was aimed at improving interoperability. Of course, people who are particularly concerned about security might take issue with this particular nostrum (and some have!).
As may be apparent to readers who have gotten this far, I am not yet sure there is a there there, but I am fascinated by the possibility that it might be possible to extract some design principles from this notion that would lead to potentially more robust and adaptable designs. Think about what makes a chair a chair. It's a thing to sit on, has legs and usually a back and maybe some arms. But there are so many things we recognize as chairs that are quite varied in their specifics. Flexible design suggests to me that under-specification has something to do with essence or core concepts. I hope interested readers will take a moment to share their thoughts, particularly if they see more deeply into this idea than I have at the present.
Footnotes
a. This is not precisely correct since the notion of "source routing" is part of the specification and allows a host to force packets to flow along a path specified by intermediate IP addresses, but the general route generation and selection process is independent of the IP specification itself.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2017 ACM, Inc.
Comments
Jay Bayne
Agree that layering (i.e., compartmentalized specifications) in the ARPANET, OSI and Internet protocols have been a net benefit in realizing both new functionality and scalability. The "don't care" metaphor might have a role to play going forward. By their nature, learning systems (neural networks, fuzzy logic, etc.) introduce a form of reduced specifications in order to facilitate adaptation. It remains to be seen how correctness, verification and validation proceed in emerging AI-enabled systems.
Mark Christensen
This reminds me of the progression from fixed format text files, to tab delimited, to XML. Where the specification gets less demanding and the allows greater flexibly when dealing with multiple partners.
Jeremy Gibbons
You ask: "Is there a way to measure the degree of specification in the way that Claude Shannon found to specify information as entropy independent of semantics? Can something be fully specified, partly specified, or completely unspecified and how would these be described or measured more precisely? In circuit design, for instance, there is the notion of "don't care" for some values in a Boolean representation. Can this notion be applied to program specification as well as to protocol specifications? Are there design principles that one can derive from this notion of under-specification?"
Absolutely, there are program design principles for underspecification, in the form of program refinement. Programs and specifications are partially ordered by refinement, and there is a fully developed calculus for deriving concrete programs from more abstract specifications. These principles have been explored for over 40 years, and are taught to undergraduates (at least in certain universities). See for example Dijkstra's "A Discipline of Programming" (1976), Gries's "The Science of Programming" (1981), Morgan's "Programming from Specifications" (1990), Hehner's "A Practical Theory of Programming" (1993), and Bird & de Moor's "Algebra of Programming" (1996). Morgan's and Hehner's books are now freely available online, and Hehner even has a free online course called "Formal Methods of Software Design".
Valerio Zanetti
Very inspiring view on the topic - thank you! It reminded me of E. Raymond's book on The Art of Unix Programming: compactness, constraint and orthogonality as guiding principles for a specification might help to find a way out of the maze with too many (technical) options. The creation of ARPA NET or HTTP was also formed by very strong resource and functional constraints which fosters simplicity.
Good design certainly depends on formal and non-formal skills - it's hard to quantize it. That's why there are concepts like Louis Sullivans "form ever follows function".
Another aspect is ubiquity versus control - either of which aspect prevails the design phase changes the outcome in a sustainable way. Maybe HTTP was invented with really just one simple idea: interoperability - everything else is left open to the users and applications.
The absence of single-cored api's for services with a higher aggregation level forces industries to buy brands instead of doing co-innovation. Http most probably was never intended to be a brand but to be a protocol for interoperability.
The quest for higher level under-specified api's is certainly worth the hassle.
Displaying all 4 comments