Break the Rules to Make a Big Leap Forward

Close on the heals of our success with the Hotmill Sprays system in the mid-90’s, Alcan’s management gave us the task of building a scheduling system for the Coldmill.  This was a bit of a coup, since we were the Hotmill automation group, not the Coldmill automation group.  But we were thrilled, and got to work right away.

One of the great things about being a systems developer is that you get to name stuff.  I always found it kind of odd when people around you start using the name you invented for that new system in the shower yesterday morning.  A cheap kind of immortality for the mildly self-important, I suppose, but I always got a kick out of it.

The acronym for the Alcan Scheduling System was clearly going to be an issue (ASS?), so in a moment of pre-presentation clarity, I renamed it to the Alcan Scheduling Engine, and the presentations proceeded without tomato-throwing or other assorted ridicule.

For this system, we’d need another OpenVMS Alpha cluster, and some ETL (extract/transform/load) processes copy data from the MIS (business) system that contained the customer orders.  The MIS system was an ancient Supra V1 / Mantis application.  I’m pretty sure none of you readers has ever heard these products.  If you have (are you one my ex-colleagues?), please leave your email address and I’ll send you the appropriate condolences.  

The ETL process downloaded customer orders from the MIS system to our shiny new ASE system.  But the data was crap, because the Supra database didn’t have atomic transactions (i.e. not ACID compliant), and we were getting half-complete transactions mixed into our data.  Dirty reads (as this is sometimes referred to as),  suck.

So the MIS crew dragged back one of their ex-patriots, who re-wrote the ETL program in a way that dirty reads were avoided (thanks, Mike F.!).  I felt it was a hack at an obscene level, but couldn’t help respect the fact Mike got us out of a nasty bind.  I did, however, swear that if I ever met the person who ported DBase I to OpenVMS, apparently resulting in Supra V1, I’d boil him in solder.

Recently I had the privilege of building (along with a talented team of developers), the web-part of the shomi.com video service for Rogers and Shaw.  As part of the application infrastructure, we used the ElasticSearch database to hold our content catalog, and provide high-speed search capabilities. But ElasticSearch is not your daddy’s relational database.

ElasticSearch has some amazing features, but doesn’t include joins or transactions – which is the stuff from which we built applications for decades.  Dirty reads are a problem if your most recent write operation hasn’t propagated to all cluster members.  It’s a fabulous product that fills the high-speed search niche, but to achieve that speed, it sacrifices joins, transactions, and a few other goodies.

Lesson:  Yesterday’s Dirty Reads are the foundation for today’s web-scale components like ElasticSEarch.  And Mantis was probably a precursor to Python (whitespace counts in Mantis too).  So, never assume today’s system design “rules” aren’t going to get broken by tomorrow’s hip, next-gen ideas.   Sometimes breaking the rules is exactly what it takes to make a big leap forward.