Intelligence failure: get the right IT system thinking

[Presley Cannady]

I don't know much about information technology but I'm game.

Let's see. Large scale information data warehousing has been around since the early 1960s. Some of the original database warehouses were built to handle fun things like payroll and finance. So, we've got a mature set of technologies. SAP and other companies have been building enterprise level data-warehouse systems with multiple levels of access that exist between physical and logical locations for quite awhile. Anybody though who has tried to roll out People Soft or similar can tell you this is not an easy task.

Setting aside the time granularity of the data, the problem's scale has increased by at least nine orders of magnitude (megabyte to petabyte) over the sum total of hard and electronic storage in that era. Even then, you're only talking about snapshots over several months to several year intervals per target. Also, you'd probably like a taxonomy and tools to crunch this data in a reasonable period of time; the architectures of the early-2000s are woefully inadequate. Anyone want to take a guess how relevant technology from forty years earlier is?

From an enterprise point of view, things that worked well twenty, thirty or forty years ago form a solid foundation for evolving technology. That makes sense. Businesses aren't employing orders of magnitude more people than they were before. Accounts payable still goes out mostly per diem, weekly, bi-monthly, or monthly, and an hour is still and hour. That whole area is largely concerned with tweaking around the edges.

Business analysis--which we seriously need to think of separately from the rest of the enterprise--is a whole other animal. It's concerns itself not with data pertaining to the operation of the business, but the far larger, far finer set necessary to answer arbitrary questions in an arbitrarily short amount of time.

The issue is not in the technology of delivery. The information technology exists to deliver the correct information to the correct individuals for analysis and/or action.

Information technology is no Oracle (no pun intended). It's perennially immature hardware and software that works precisely as badly as it's implemented. We've got maybe twenty years experience with the sort of data centers we need for this kind of work, and less than ten in learning how to federate them properly. In fact, you could say we still don't know how to do it, because hardware and software are still catching up. We can't simply hand wave in the technology if it doesn't exist or perform as advertised.

Visual systems exist that do this within life critical systems currently. The Aviation Administration handles thousands of targets that have to be analyzed constantly with a good amount of accuracy.

That speaks more to the power of billions of dollars in cost overruns thrown at making software, hardware, and people multiply redundant than the technology itself. On paper, the problem isn't terribly difficult, but the FAA's trials and tribulations with implementing good tracking systems are well documented. You can taste the disappointment if you think about how few life critical systems--expensive as they are--come even close to approaching the scale of the air traffic control challenge. Launching the Space Shuttle twice a year costs almost about as much as the FAA's annual operations budget.

This is done over a large area with multiple sensors and tied into a backbone utilizing rule sets for delivery. Not close enough to the target use? Mail systems like HotMail, Gmail, and others do the same basic task but like the FAA example these are limited and the rule sets are finite.

Federating data is not easy when not designed for from the start. It took ten years to get to the point where you could manage the logins for several webmail services seamlessly, and even today it's only the major players that have gone ahead and done so. Only recently are we starting to see consumer tools for going the next step and providing people with a common inbox for all their accounts. Once again, the problem isn't hard on paper (or even in prototyping). It gets real hard when you start thinking about how many people are going to use this tool, how many cycles and how much storage will you need to support the demand, how much pipe will you need to move data from God knows how many places to God knows how many more, etc. We're starting to work through the problem, but that doesn't mean we're there yet. Amazon S3--as dirt cheap a way as you can get to never having to worry about back ups again--has been around for a couple of years now. Is your employer using it?

Federating data in business analysis, intelligence, or the like by definition precludes a priori design. Doing it in realtime...? It's a hard problem. I don't think it's intractable, but at some point IT needs to come clean and let the business know exactly where we're at.