About 35 years ago, I was sent from the UK to Canada to troubleshoot a production plant for the first time during my fledgling software engineering career. My job was to complete commissioning of a batch processing plant whose control system consisted almost entirely of a single DEC LSI-11 (PDP-11/03) with less memory than you’d now find in a cheap watch, running a long-forgotten multitasking operating system and deploying a modified version of the BASIC language.

By the time I arrived, the initial commissioning effort was complete, and the plant had a full production schedule to work around. Although problems with the control system meant that it was running quite slowly, the plant needed to provide animal feed for the cold Canadian winter months. My opportunities to address the issues on the live system carried considerable pressure. It was a rare occasion when a slot was given to me to actually do something.

The customer’s ever-patient and supportive engineering team helped me lash up a 240V supply for my UK spec development machine, and I spent hours working on solutions offline, waiting for my window of opportunity to try them online. There were frequent ups and downs along the way, and one especially memorable look from the customer in response to a menu option whose only function was to display the message “Yet to be written”!

The production scheduler software was proven-in-use on countless previous systems. It was designed to ready the production plant to make the next scheduled item, ensure that the bagging machines were readied, and monitor production before starting the cycle again.

But in one particular configuration, it displayed a very elusive bug that occurred once every few weeks. When the glitch occurred, the system place product mixed for one batch into packaging labelled and sized for the next. Given that this was medicated feed to treat animals ranging from chickens to race horses, the potential for disaster and litigation was all too real.

Many hours were burned trying to find the timing issue. It couldn’t be provoked on a simulated system and left no trace when it raised its head on the production system. But something had to be done.

The solution was as effective as it was ugly. Immediately before a new production run began, I added a final check to ensure that the scheduler entry, the allocated production batch, and the packaging all matched. If it didn’t, the scheduler was forced to try again until it did. Pragmatic? Certainly, and the customer was very happy that their plant could now be trusted. Best practice? Hardly. Would you want flight control software written like that? Definitely not!

These days, I work with functional safety standards, cybersecurity standards, and coding standards, and it’s easy to forget that they aren’t just painful overhead! No, in fact, they help us write software that works properly, that doesn’t feed medicated chicken feed to race horses. So, if I start to wonder whether functional safety and security standards are really necessary, I only have to think back to a very cold Canadian winter to find my answer.

Acknowledgements

This content was first published as part of EDN’s “Tales from the cube” feature