Data transmission for machines or programmers?

When we are working on building or maintaining an interface to a system for use by another computer program, do we really think whom we are building the interface for?

Pick anyone involved in enterprise application development working on a (relatively) new application – the chances are that he or she is working with some form of SOA implementation; either developing services or consuming them. An overwhelming number of SOA implementations today are built on mostly XML or YAML (be it SOAP, REST, etc.) based services; though SOA does not have to be synonymous to a web service. Now, XML (and YAML) is a machine readable format and is easy for human users (read programmers, operations folks) to understand; but is it the best format for data interchange between two computer programs?

Most of us writing code now deal with high level computer languages further abstracted by many frameworks and pre-built libraries and decorated by the wonderful IDEs around. Not that I am saying we need to, but we hardly think about the machine instructions our statements get converted into. We go as far as shrugging responsibility below the application layer our code runs in. However, an appreciation of the layers and abstractions below does make for a better programmer.

Back to track, if we just concentrate on XML, there are libraries available for parsing XML (using multiple methodologies – SAX, DOM, pull parsing to name a few) in almost every language out there. And for an object oriented language, there are data binding frameworks which paired with IDEs, build tools, reflection and configuration converts between XML and an applications data objects with very little effort from the programmer. So, anyone developing an API for consumption from disparate systems, very easily goes into defining that API in XML. Now what is wrong with that picture? After all, it is easy to develop, easy to document, easy to consume, is a standard, there are more tools and libraries available to support it than most people care to count.

Well, it brings me to the title of this post, are we building the API for data transmission for a machine (software program) or the human programmer? Let’s see a simple XML snippet representing a person’s name:

<PersonName>
    <GivenName>Frederick</GivenName>
    <MiddleName>Joseph</MiddleName>
    <Surname>Flintstone</Surname>
</PersonName>

Without new lines and formatting, the snippet above is 117 characters long. And if I wanted, specially when at the two ends of the communication are two programs, I could have transmitted the data (Fred Flintstone’s name) in 37 characters as:

PNG09FrederickM06JosephS10Flintstone

Now, why would I want to do that? It negates almost all the benefits of using XML as the interface to the program. Well, if you think about it there are multiple advantages: The message is almost 70% shorter than the XML structure, easier on the network and easier on IO (in the program, OS and every other layer). The second message clearly transmits the intended message and its parts. And believe it or not, is much easier for a machine to parse, unlike most XML parsing technologies this does not involve building a tree or stack, and can usually be parsed just using a counter and arrays. Will this work between multiple technologies (like a .net consumer and a Java producer) – it absolutely will, we are only dealing with characters, abstracting out byte handling. Will this scale? designed well, it absolutely can. Is it easy to debug? With the proper tooling (which you might have to write) it is, maybe even easier than XML.

Is the format above a solution to replace XML in all inter-machine interfaces – it is not. The format above is just an example, it solves the little problem we were trying to solve, and will scale to handle even larger problems. But is not the answer to all problems.

What was the last few paragraphs all about? To show that sometimes (or most times) there are better and usually more performing ways to define a (non-human) interface to a system; though it may not always be the easiest. It might take a little longer to develop, need custom tooling and developer training. But, if you are talking about enterprise systems handling multiple thousands of transactions every hour, the extra cost would for most cases pay off. And any tooling and training investment made is portable across applications. Almost all such solutions do not involve rocket science, and aren’t programmers not supposed to be the dumbest lot around?

Do we always have to take the easiest (and well trodden) path? Just a thought.