Document Driven v Data Driven

I’ve recently been thinking a lot about forms. Why forms? Forms give us fascinating insights into that way that organisations work.

A Life of Forms

We are surrounded, some would say inundated, by forms:

  • Banking runs on forms.
  • Insurance wouldn’t survive without forms.
  • Most organisations have thousands of ad hoc forms for various diverse purposes.
  • One of the worst things to happen in some organisations is that a situation arises for which there is no form.
  • Visit a medical professional and somewhere within the dialogue a form will become necessary.
  • Subscribe to any service and forms will be used as part of the contracting process.
  • Start a new employment and you are likely to spend much of your first day completing forms.
  • Our birth and our death are accompanied by forms.
  • How many times a day do you complete a two-field form in order to gain access to some technology.
  • Interact with a government organisation and a form will be required.

Sometimes these forms are online, web page, or even forms on mobile devices. There are still, however, many situations where forms are completed with a pen. How many hours have you spent trying to complete a pseudo form that was sent to you as a Word or PDF document.

Document Driven Business

There are many PowerPoint decks, Excel spreadsheets and Word documents that are in essence forms. They are created from a template that sets the titles and contents of each slide/worksheet/section. The person completing them is expected to say certain things in certain ways, just like a form.

  • This first slide has the title on including the reference number, person presenting and target date.
  • The next slide has the required content on and only this content.
  • The following slide will explain what it is you are going to do.
  • The penultimate slide will outline the business case in the supplied table.
  • The final slide will contain the risk register, using the supplied table headings.
  • No other slides may be added.

It’s a form, isn’t it?

A Form to Transact

Each of these form-types exist to support a transaction:

“Once you have completed sections 1 to 5 and 8 of the loan application form we will proceed to the next phase of you application.”

“You are required to complete a tax return of which sections a to e are mandatory.”

“We’ll proceed with your project once you have provided the project initiation template document.”

The boundary of the transaction is defined by the form, without the form nothing moves forward, or backward.

This way of working produces a number of effects:

  • Over preparation – in order to make sure that a transaction can complete documents tend to be over-worked. Many hours are spent making sure that every detail in a form/document are correct to a level of detail that is not required to move onto the next phase, but everyone strives for perfection to avoid rework at all costs. A small amount of over-work is compounded as a process is worked end-to-end. Imagine how much work goes into producing a set of 40 document? Add a little bit of over-preparation to each of them and the amount of effort being expended is huge.
  • Over-stating – The over-preparation of documents often includes over-stating, where things that aren’t required in the document are stated in the document “just-in-case”. The problem with this superfluous information is that it becomes part of the record and is then used by people who make decisions despite its heritage and trustworthiness.
  • Point-in-time perspectives – The information in the form/document was mostly correct at a particular time on a particular day, but that’s all that can be said about it. Any perspective that is taken on that document is locked into the context at that time. The information in the document isn’t being refreshed, it was completed, a transaction took place and now everything within the document is, at best, history. Yet, people will continue to refer back to it as information way beyond the valid life of the data contained within it. The reality is, even before the document is concluded the data within it will be out of date.
  • Action blocking – A form/document represents the end of one activity and the start of another – a phase-shift. The next phase can’t start until it has received the information from the previous phase. Even if an element of the next phases has all that it needs to proceed it can’t until the transaction has been agreed. Consider how many actions are expected to be undertaken following the transaction of a 100 page document? How many of those actions could have safely been undertaken way before the transacting of the document?
  • Phases based on documents – The definition of a document as the point of transaction means that production of the document often becomes the definition of the phases/stages of an activity. This way of planning has little to do with the amount of effort involved, or the value being produced, it just represents a transaction. An activity that only exists to produce a document is a bad activity.

Data Drive Business

Let’s turn our attention to data-driven activities.

The document has been with us for thousands of years, but we no longer need to work at such a coarse level. The information that is placed into a form was not generated by the form. A form is just a place to consolidate information that already exists elsewhere. When you are asked about your date-of-birth in a form you are simply recording information that has existed, for some of us, for many years. So why not link the data directly with the intent for which it is needed. Why bother placing a date of birth on a piece of paper when one system could ask another system whether I’m older than 18 and get the correct answer back.

There are situations where data isn’t enough and a set of information may need to be brought together to tell a particular story. Imagine a design for a network topology, the design may be the first time that it’s been outlined. This isn’t to say that in this situation a document is required, it’s just to highlight that an intermediate step from current state to future state may be required to fill gaps in the data. Even in this network topology example a diagram with meta-data is probably sufficient to communicate the change being proposed and for people to agree to transact. Once the change has been implemented the diagram is no longer required because the current state information becomes the record.

Taking the network topology example even further, the need for a human-readable design demonstrates a gap in policy and understanding. If the change could be codified in a way that a policy mechanism could understand and assess, then the change could have taken place without the need for a diagram. If, as an example, an application needs to add more resources to the network, the network would respond on the basis of the data provided and the policy defined. Likewise, once those resources are no longer required the policy engine would turn the resources off. All of this would happen before someone has filled in half of the “necessary” paperwork.

Our job, as humans, should be to assess and define the required actions for the exception, for those situations where data and policy is not sufficient for a decision to be made.

Time for Transformation

For much of the life of IT may applications have been little more than form replacements and that has given us some productivity gains. In many ways we are only just at the beginning of a transformation from a world driven by documents to one driven by data  This will require a profound change in the way that we think and act.

Organisations that continue to rely upon forms (including apps that are replacing forms) will be overtaken by the machines.

Header Image: This is one from a recent morning walk along the lanes near my house. I’ve always loved the shapes of tree skeletons in the winter.

The Suboptimal Kitchen – The 10 Steps to Getting a Cup of Tea

Process for getting a cup-of-tea in a certain kitchen:

  1. Enter the room and make your way to the sink from south to north. Try to avoid the crossing traffic on your way. Wash cup with provided cleaning equipment. Don’t bother trying to dry your cup because that means trying to get to the paper towels which are the other side of the hot water area and would involve more cross-step navigation than is good for anyone’s health.
  2. Turn around and proceed to collect a tea bag from the receptacle on the western side. Special attention should be given to the fridge door which may be open causing a hazard. There may also be individuals stood behind you awaiting access to the sink, another hazard.
  3. Traverse back across the room towards the hot water geyser. Add water and tea bag into cup. Be sure to maintain awareness of traffic crossing that is seeking to access the fridge on a east-west direction and also people accessing the sink in a south-north direction.
  4. Make the small easterly traverse to collect a stirring stick. There are a few hazards on this trip, but it can sometimes be affected by erratic behaviours around the hot water geyser with people approaching direction from the entrance in a south-north direction. People mistakenly seeking to access the paper towels may also be traversing in the same direction, these people should be regarded with suspicion.
  5. Head, slowly, in a southerly direction whilst stirring the tea bag in the cup. Deposit tea bag into the bin whilst retaining the stirrer once the required level of tea strength has been achieved. You will encounter others on your journey to the bin who prefer a stronger brew than you and are not yet willing to surrender their tea bag. There is also a likelihood of milk foragers intersecting this journey in an east-west direction. There may also be individuals heading directly towards you on their return trip from the bin in step 6.
  6. The location of the milk is dependent upon the time of day and other parameters that are too complex to describe here. To be sure to cover all of the requirements partially retrace your steps back towards the stirrers in order to access whether any milk has been left out of the fridge on the eastern side.
  7. If milk is located on the eastern side it will need to be assessed for its freshness as it has been out of the fridge for an unknown length of time. If fresh, add to tea, stir and exit via step 10. More normally though the milk on the side is just an empty carton, in this situation deposit the empty carton in the appropriate bin, because you know where the bin is and others clearly don’t.
  8. From the bin you will need to navigate across the room again in a westerly direction to the fridge. This involves crossing several north-south and even some west-east traffic. Once safely on the western side remove milk from the fridge, add to tea and stir.
  9. As you have taken some fresh milk out of the fridge you will need to make a complex assessment using several other undefined parameters as to where to place the milk. If someone else is actively looking for milk on the eastern fringes it is expected that you will pass it to them. If the kitchen is empty then it’s normally more appropriate to place it back into the fridge. Other than that it’s almost impossible to know what the right thing to do with the milk is, placing it in the fridge is often the safest choice. You may have to make an assessment of what to do with the milk whilst also traversing from west to east and crossing several other task routes.
  10. It’s now time to leave, assess the required cross-traffic situation and proceed with caution. If you’ve been following closely you will have noticed that you are still in possession of a stirrer. This should be deposited in the bin just outside the kitchen, because it’s just too difficult to get access to the bin in the kitchen.

This is based on a real kitchen, no attempt has been made to hide details in this post. If you’ve been in this kitchen you will know exactly where it is.

Please do not try to optimise these steps, it has been tried before and it’s just too politically difficult.

YouTube is now your Mum/Dad/Practical Friend

One of the things that fascinates me is the social change that is driven by the internet and internet services.

Once upon a time we would answer practical problems in one of two ways:

1. Ask someone we trusted

The question would normally be to our mum or dad or to that a practical friend who knows how to do anything. Their proximity would allow them to show us how to do something in person, or talk us through it over the phone. Sometimes their answer would be to talk to someone else that they know who is practical in a particular way: “Talk to your grandma she’s really good at buttonholes.”; “Ask Eddie he knows how to protect a Koi pond from herons.”; “Ask Mary she’s good for advice on home automation systems.”

As a result our wisdom was limited by their knowledge, or the knowledge of the people that they know. What’s more we only knew if their knowledge was any good when we tried what they suggested. We had to decide whether to try what the suggested by judging their level of confidence in their knowledge. I suspect we’ve all had friends who’ve confidently told us to do something that has later turned out to be the last thing we should have done.

This was the normal way of finding out how to do something.

2. Go to the library or take a course

If we needed to know something outside the knowledge of the individuals we trust we may go as far as to do some formal research. This research would have mandated a trip to the local library and wading through reference manuals and the like. In extreme cases we may even take a course on how to do something, but this was only for the truly dedicated.

This was not the normal way of finding out how to do something, it was only used in exceptional circumstances.

Along comes YouTube (other video sources are available)

For many YouTube has now replaced your mum, dad and practical friend. it’s even replaced the library and training courses for some.

I’ve had two situations recently where this was the case:

Windscreen Washer Failure

It’s been an interesting winter here in the UK with different whether each day, switching from warm and wet to bitterly cold. Windscreen washers have, therefore, become a vital part of road travel, when the washer in the car that my wife drives failed it was important that it was fixed.

My first instinct was that it was just a fuse problem so opened up the in-car manual to see which one, only to discover that the windscreen washer wasn’t listed. Fortunately YouTube had most of the answer – someone called Andy Robertson had experienced exactly the same problem and posted a video. I say most of the answer because the fuse box that Andy shows isn’t quite the same as the one that’s in our Polo, but it did allow me to know that it was a 7.5 amp fuse and following a short process of illumination to find the one that had blown.

iPhone Charging Problem

I’ve been struggling to charge my iPhone recently – I’d plug a lightening cable into it and leave it, when I came back to it later the cable would be slightly out of the socket and no charging will have taken place. Having tried a number of different cables I realised that the problem was with the socket in the iPhone itself, not the cables. Going to the Apple Store to get it fixed sounded like an expensive proposition so I took to YouTube for help. It wasn’t long before I found a set of videos from people all telling me that it was likely to be dust and/or lint in the mechanism and simply to get a pin and dig it out. Putting a metal thing into a charging point didn’t sound like a good idea, but the basic idea worked a treat and now my phone stays plugged in.

I’m not sure which of my practical friends would have known to do that, mu parents certainly wouldn’t.

The New Normal

These are a couple of personal examples of what I think is the new normal way of working out how to do something, but it’s not just me. The car fuse video has been watched over 27,000 times, the iPhone one nearly 700,000 times. A friend recently used another YouTube video to work out how to get a broken headphone jack out of an iPad. Another friend gives overviews of his allotment that people use to get advice on the technicalities of an allotment and allotment life.

I wonder how many of the 1 billion hours of YouTube video that is watched every day is so helping people with their how do I questions?

Co-location – the Super Food of Collaboration

In the western world we have a huge choice of foods that we can eat. We know which ones are good for us, and which ones aren’t, yet many of us are overweight, getting fatter and suffering from the health consequences of poor dietary choices.

The obesity problem is worse in areas of low income – why? One of the  main reasons is that good, healthy food is expensive and for the most part cheap food is unhealthy. This cheap food is normally processed, has travelled a long way from areas of low cost production and is purchased on a whim without the burden of preparation. It makes us feel good because it’s full of sugars and fats that our addicted brains crave but it doesn’t provide a healthy nutritional diet.

Teleconferences are the fast-food of collaboration. We set them up without consideration because they are cheap and immediately accessible. They allow us to use resources from wherever they are in the planet without consideration for those resources and, sometimes, with little consideration for the quality of those resources. They help us to believe that we are collaborating, which we are, but only in the same way as a fast-food burger is food.

Co-located collaboration is different, like organic wholefood it’s more expensive but it’s significantly better for us. The extra expense makes us respect it more so we make sure that we get all of the value out of it that we can. Organic wholefood requires extra preparation to get the right ingredients together but the results are amazing and the same is true for co-located collaboration. Co-located collaboration feeds all of our collaboration dietary needs in a way that the fast-food teleconferences never can, we never really get to know people on teleconferences, they’re just voices. There is marked difference in the level of collaboration that we achieve with people that we have physically met compared to those we haven’t. The most healthy collaboration is achieved when we are co-located with people we know well, this is the super-food of collaboration.

The occasional fast-food teleconference collaboration isn’t going to kills us, but it’s not healthy to live on it.

Repeating Under-Performance – We All Do It

You are an under-performer.

You are performing below your optimum, I can guarantee it.

There are many things that you do every day that you could do more efficiently, fast and better.

Some inefficiencies are seen, others you are unaware of.

One of my repeated inefficiencies goes like this.

  1. A remind pops up on my device to tell me that I’m due to join a meeting.
  2. I double-click on the meeting, opening up the invitation
  3. I scroll down to find the link for joining the meeting which are somewhere in the text of the meeting:join-skype-meeting
  4. I then click on the link to Join Skype Meeting.

It’s a simple four step process, but it’s a 25% inefficient process if I use steps as the simplistic measure of process flow. Step 3 is nugatory.

That’s not all though, my simple measure of efficiency is missing the fact that the scrolling down activity takes the longest of all of the steps to complete.

I must have repeated this set of tasks thousands of times. They are so deeply ingrained in my process memory that I don’t even challenge them.

So why is step 3 nugatory?

The efficient process goes like this:

  1. A remind pops up on my device to tell me that I’m due to join a meeting.
  2. I click on the meeting reminder.
  3. I then click on the Join Online button that is shown in the reminder and join the meeting: join-online

It’s a trivial example, but these are the things the we do every day that make us under-performers. Or do they? Is this type of efficiency a good measure of performance? I wonder.

Why do we complain about free things? I can think of some reasons.

Facebook is free.

Twitter is free.

Instagram is free.

Google is free.

WhatsApp is free.

Skype is free (for most of us)

Yet, when they don’t do what we expect them to do we complain bitterly, it’s as if they were part of our monthly utility bill.

Why do we do that?

I think that there are several reasons.

  1. We are invested in them – Even though we haven’t paid cash for these services we have invested in them. We’ve invested our time and energy into the content that we’ve placed on them. We’ve invested time in understanding how to use them. We’ve changed our life to fit them in. That investment gives us a right to complain when things go wrong.
  2. They are charging us – There is a charge for each of these services and the charge is attention. Most of these services are subsidised by advertising which takes our attention. However good you think you are at ignoring these adverts you are kidding yourself if you think that they aren’t taking some of your time away. If you could put a value on that time what would it be? We secretly know we are being charged and that gives us a right to complain when things go wrong.
  3. They are selling you – As well as charging your attention these services are also selling your information to someone. We make a contract with them that enables them to do this on the understanding that the service stays free. We know they are selling us and that gives us the right to complain when things go wrong.
  4. Entitlement – The cost of something rarely defines our feeling of entitlement to it. When someone promises to do something for you, even if it’s for free, we get upset when they don’t deliver. These free applications have promised to do something for us and now they aren’t. If you are entitled to something you have a right to complain when things go wrong.

I’m pretty sure that there are more reasons, but I think those are the main ones.

I apologise if this free service didn’t live up to your expectations, please feel free to complain. Please be assured that reason 2 does not apply to this blog; I don’t think reason 3 applies to this blog, I don’t sell your information to anyone, but I can’t be sure that others in the supply chain don’t; I’d be amazed if reasons 1 and 4 applied to your use of this blog :-).

Your data in their hands | When was the last time you read a privacy policy?

Last week Evernote got themselves into a public relations storm by updating their terms and conditions relating to privacy of data. They then had to hastily update the policy, stating that they would no longer be making the changes as planned.

The other month I wrote about digital exhaust, but there’s a lot of data that we place into others hands deliberately. When you type an email, upload a file, fill in an online form do you think about who may have access to that data? I’m not sure we often give it the consideration it deserves.

You should assume that the data is going to live forever, so our actions have lasting consequences, and so do the actions of those people who have access to our data.

Each of us have signed up to many terms and conditions that have included privacy statements, but few of us have read any of them.

Those privacy policies were mostly written for a relatively static world but we are entering a new era of data privacy concerns as more of our data gets given to artificial intelligence and machine learning to assess and give value on. That was one of the aspects of the Evernote situation:

“Human beings don’t read notes without people’s permission. Full stop. We just don’t do that,” says O’Neill, noting that there’s an exception for court-mandated requests. “Where we were ham-fisted in communicating is this notion of taking advantage of machine learning and other technologies, which frankly are commonplace anywhere in the valley or anywhere you look in any tech company today.”

Evernote CEO Explains Why He Reversed Its New Privacy Policy: “We Screwed Up”

The reality is that Google, Microsoft, Facebook and Apple have both been using machine learning for a long time, that’s how they know to tells us interesting things like pre-warning us about traffic problems on our journey home when we haven’t told them where home is.

Most of the time we don’t even give the privacy of our data a thought, and we should. Did you know:

  • Many site reserve the right to change the terms without telling you.
  • Many services claim copyright over parts, or all of your data.
  • Some sites don’t let you delete your account.
  • Many sites track you on other sites.

It’s terms like these that enable adverts for an item I searched for just a few minutes ago to now be showing in my Facebook.

When was the last time you checked the PrivacyGrade of an app before you downloaded it? Or check Terms of Service: Didn’t Read before agreeing to the terms on a site? I suspect that for most of my readers they’ve never visited these sites.

Ultimately the only lever that we have over these services is the commercial one and most of them aren’t going to do anything to jeopardize that, but that won’t stop them pushing up against the edges of what we regard as acceptable. What we regard as acceptable is greatly influenced by whether we feel like we are getting something for free.

This constant pushing against the barriers will then influence what the next generation regard as acceptable. The Facebook privacy policy runs to 2719 words and was last updated on the 29th September 2016. Even if I had read the privacy policy in when I started using it I couldn’t tell you how many iterations it had been through or what changes had been made.

We are trading our privacy for access and I’m not sure we really understand the cost.