you're reading...
Musings, Vocem meam

Data transmission for machines or programmers?

When we are working on building or maintaining an interface to a system for use by another computer program, do we really think whom we are building the interface for?

Pick anyone involved in enterprise application development working on a (relatively) new application – the chances are that he or she is working with some form of SOA implementation; either developing services or consuming them. An overwhelming number of SOA implementations today are built on mostly XML or YAML (be it SOAP, REST, etc.) based services; though SOA does not have to be synonymous to a web service. Now, XML (and YAML) is a machine readable format and is easy for human users (read programmers, operations folks) to understand; but is it the best format for data interchange between two computer programs?

Most of us writing code now deal with high level computer languages further abstracted by many frameworks and pre-built libraries and decorated by the wonderful IDEs around. Not that I am saying we need to, but we hardly think about the machine instructions our statements get converted into. We go as far as shrugging responsibility below the application layer our code runs in. However, an appreciation of the layers and abstractions below does make for a better programmer.

Back to track, if we just concentrate on XML, there are libraries available for parsing XML (using multiple methodologies – SAX, DOM, pull parsing to name a few) in almost every language out there. And for an object oriented language, there are data binding frameworks which paired with IDEs, build tools, reflection and configuration converts between XML and an applications data objects with very little effort from the programmer. So, anyone developing an API for consumption from disparate systems, very easily goes into defining that API in XML. Now what is wrong with that picture? After all, it is easy to develop, easy to document, easy to consume, is a standard, there are more tools and libraries available to support it than most people care to count.

Well, it brings me to the title of this post, are we building the API for data transmission for a machine (software program) or the human programmer? Let’s see a simple XML snippet representing a person’s name:


Without new lines and formatting, the snippet above is 117 characters long. And if I wanted, specially when at the two ends of the communication are two programs, I could have transmitted the data (Fred Flintstone’s name) in 37 characters as:


Now, why would I want to do that? It negates almost all the benefits of using XML as the interface to the program. Well, if you think about it there are multiple advantages: The message is almost 70% shorter than the XML structure, easier on the network and easier on IO (in the program, OS and every other layer). The second message clearly transmits the intended message and its parts. And believe it or not, is much easier for a machine to parse, unlike most XML parsing technologies this does not involve building a tree or stack, and can usually be parsed just using a counter and arrays. Will this work between multiple technologies (like a .net consumer and a Java producer) – it absolutely will, we are only dealing with characters, abstracting out byte handling. Will this scale? designed well, it absolutely can. Is it easy to debug? With the proper tooling (which you might have to write) it is, maybe even easier than XML.

Is the format above a solution to replace XML in all inter-machine interfaces – it is not. The format above is just an example, it solves the little problem we were trying to solve, and will scale to handle even larger problems. But is not the answer to all problems.

What was the last few paragraphs all about? To show that sometimes (or most times) there are better and usually more performing ways to define a (non-human) interface to a system; though it may not always be the easiest. It might take a little longer to develop, need custom tooling and developer training. But, if you are talking about enterprise systems handling multiple thousands of transactions every hour, the extra cost would for most cases pay off. And any tooling and training investment made is portable across applications. Almost all such solutions do not involve rocket science, and aren’t programmers not supposed to be the dumbest lot around?

Do we always have to take the easiest (and well trodden) path? Just a thought.



3 thoughts on “Data transmission for machines or programmers?

  1. Let represent this XML for only for machine.

    opss still it is 50% more character than yours.
    But then I will prefer 50% more character than yours for XML is not stable technology and lot of parsers are available. (see I dont have to reinvent wheel again).
    Second point is that in your approach, initially it may seem fool proof, but as, if at all, it is used in practical world, there will be lot of issue with delimiters.

    So I will prefer human readable XML or machine readable XML, but XML 🙂

    Posted by Ravish Kumar Singh | May 11, 2011, 03:27
    • I have actually seen implementations of non XML interfaces with large and complicated data sets. Properly designed, and aided with the right tools they are not hard to work with.

      I do not disagree that there are a lot of parsers (some quite efficient) available for XML. The point I am trying to make in this post is developer efficiency / ease versus machine efficiency at runtime. It is valid that XML is easier and cheaper (man-hours, dollars) to develop in, but I believe that advantage goes away on a system designed to handle millions of transactions an hour – just in the savings you have on the number of servers.

      Posted by Bindul | May 11, 2011, 12:21
  2. I did it again .. (triangular brackets are removed)

    >>Let represent this XML for only for machine.


    Posted by Ravish Kumar Singh | May 11, 2011, 03:29

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

About Random Musings

Random Musings is a collection of my personal thoughts, opinions, views and anything else that I find interesting. The views posted here are mine and may not necessarily reflect the views of MindTree Limited (my employer).
%d bloggers like this: