Office Speak: Single Source of Truth – Gold Source: Where do you go to find the correct information? What does Conway’s Law teach us?

Getting data is one thing, getting good information is another. Good information is vital to decision making but most organisations are riddled with bad information.

There is a whole myriad of ways in which information can be bad, it can be out-of-date, inaccessible or difficult to access, incomplete, duplicated or just wrong.

What people want is one place where they know the information is good – that is what a Gold Source is, also known as a Single Source of Truth.

(This is one of those phrases where people don’t seem to have settled on an acronym, thankfully. Perhaps it’s because the acronym is a bit of a mouthful, however you say it, S.S.O.T. or S. SOT, or even SSOT? Perhaps that’s why people also use Gold Source😀)

A Single Source of Truth should be simple enough to achieve, shouldn’t it? Surely? It’s what everyone wants after all? Not in my experience.

There is a common pattern in IT systems that goes something like this:

• Team A creates a data store for something with the data they have in it.
• Team B creates a data store for the same type of thing but with similar, but different data. They talked to Team A about adding the data to their store, but Team A would have taken too long to do what they needed so they created their own. 😃
• Team C creates a data store for the same thing, again with similarities to the other two but using a different technology because the technology Team A used is far too expensive for what they need.
• Corporate IT finds out about the three data stores, so sets up a project to create a Single Source of Truth for the information and does so by creating another data store using “enterprise class” technology and a data warehouse. This synchronises data from the other three systems. 🤦‍♂️
• The Corporate IT system doesn’t do what Team D requires so they set up another data store using a different technology.
• Team E takes a daily extract of the data from the Corporate IT system and load it into a spreadsheets and links it with an extract of the data from Team D’s data store. They use the workbook for reporting, except when Mary is on holiday because no-one else knows how to do what Mary does. 🤦‍♀️

I could continue but I think you’ve got the trajectory by now.

As technologists, the answer to this problem is, of course, more technology (and more acronyms). There are four common patterns for trying to resolve this problem (with Graham’s overly simple descriptions):

• Enterprise Service Bus (ESB) – link the systems together via another system.
• Master Data Management (MDM) – create a system for the data that links the other systems together.
• Data Warehouse – put all of the data into a big bucket then sort it out.
• Service Oriented Architecture (SOA) – build the systems in a way that allows other systems to use them.

What we as technologists miss is that the primary reasons for the creation of multiple data sources aren’t technology ones, they are organisational ones, or more specifically, they are communication ones.

Conway’s Law gives a great frame-working for understanding why this might be:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Melvin E. Conway

Using our example to explain, different teams create separate system because their communication structures are different, and those separate systems have different data stores. Whether it knows it or not, the organisation is designed to preclude the creation of a Gold Source.

Fortunately, a Single Source of Truth isn’t always that the right answer anyway. Going back to our example, the approach taken by each team is the right answer from that team’s perspective, and their perspective may be right for the whole business. Each team has undertaken an approach that gives them a significant advantage – autonomy. The rate of change that can be implemented across the disparate systems is much higher than could be achieved by the one system being used by everyone. When systems are used by multiple teams they move at the pace of the slowest team (at best). Coordinating across teams requires a lot of effort and a lot of thinking. There is, however, duplication between the various systems, but perhaps that’s a price worth paying for the increased velocity that autonomy gives.

There are some things for which businesses need a Single Source of Truth, but they aren’t as many as people think.

Header Image: This is Alcock Tarn which sits above Grasmere in the Lake District. It’s a great place to sit and watch the weather go by.