Customs and Practices

On Mars (as everyone knows) there are rocks. There are also two kinds of lifeforms. There are Martians, who eat voles, and luckily for the Martians, there are voles. Voles hide behind rocks and eat them.

Not much happens on Mars. Martians mainly spend their time sitting in the desert and watching for voles darting between rocks. Because there are rocks as far as the eye can see on Mars, in all directions, a Martian needs to be able to see well in all directions at once. That is why Martians evolved their characteristic four large eyes, each mounted on stalks and pointing in different directions.

Not much happens on Mars, so Martian evolution has progressed entirely in the direction of vole spotting. Each huge eye has an enormous visual cortex behind it, that can spot a vole miles away in all manner of light conditions. Most of a Martian's brain consists of visual cortex, and these four sub-brains are richly cross-connected to allow for weird light conditions to be compensated for. Martians do a great deal of processing up front, in the semi-autonomous sub-brains, so they don't really have `attention' like humans - they focus on the play of input between their `attentions' instead.

When they spot a vole, the Martians have to sneak up on it. This means keeping the vole's rock between themselves and the Vole. It requires Intelligence. Soon after the Martians evolved Intelligence, they invented Great Literature. This is scratched on big rocks using little rocks. It obeys the rules of Martian grammar, and uses the North voice for emotion, the South voice for action, the East voice for speech, and the West voice for circumstance. Not much happens on Mars, so the closest Martian equivalent to our own Crime and Punishment is called Who Stole My Vole?:

Breathtaking, isn't it? That sudden void in the East voice as the South voice swoops - a view into the very mouth of an undeniable damnation, real albeit of and by the self! I'm told it helps to have the right brain structure...

What is the point of this Weird Tale? Well, imagine what a Martian programmer might make of C.A.R. Hoare's Communicating Sequential Processes (CSP) theory. Its brain (easy to avoid gender pronoun here - Martians have seventeen sexes, and a one night stand can take ten years to arrange) is already hardwired to enable it to apprehend complex relationships between independent activities, so the processes Hoare renders linear by series of symbolical transforms, and thus intelligible to a human are already obvious to a Martian's inspection. On the other tentacle, by squeezing all the work into a huge tangle that only one sub-brain can see, the human readable version is made unintelligible to the Martian.

A joint human Martian spaceship management system design effort, with lots of communicating sequential processes controlling warp drives, blasters, ansibles and so on would face problems bleaker than the mapper/packer communication barrier, even though theorem provers using CSP could perform automated translations of many of the ideas.

The point is, complexity is in the eye of the beholder. We don't need an alien physiology to notice the difference between how individuals rate complexity - it's the whole point of mental maps. When we discover a structure that lets us understand what is happening, we can apprehend the antics of many more entities in a single glance. Think of any complex situation that you understand, perhaps the deck of a yacht or the stage of an amateur dramatics company. When you first saw it, it would have looked like a chaos of ropes, pulleys, vast sheets of canvas, boxes, walkways, and odd metal fitments of totally undiscernible purpose. When you had acquired a pattern by finding out what all that stuff was for, the deck, stage or whatever seemed emptier, tidier than when you first saw it. But it hasn't changed, you have.

No sane skipper or director would attempt to operate in such a way that the greenest novice could understand what is going at first glance. Just sailing around the harbour or raising the curtain take some training.

In the software industry the leverage issue, mapper/packer communication barrier and language specialisation have all acted to mask this point. The leverage issue says that the benefit we get from moving numbers around instead of bricks means that we can afford the investment to ensure that the simple numbers describing bricks are presented in a form accessible to all. The industrial and commercial context of many programming operations has the notion that the competent just have their information organised, that complexity is handled by not having any, and that in that way, the progress of the bricks is assured. This is all true, and is the correct attitude to information about bricks. But when we substitute information for bricks, we get a problem. We can't abstract from a big ugly brick to an informational "1". We can abstract of course, but every time we do, we lose important information which may bite us later. We can't just say that the competent just have their data organised, because the job is now organising the huge pile of data that has just been dumped on us. We no longer need the fork lift driver's skills, but we need new ones. And we can't handle the representation of complexity just by requiring that the representation be simple. The mapper/packer communication barrier makes the situation with this inappropriate analogy between information and bricks harder to discuss, because just about every step of the brick logic is there in the information argument, but instead of 1% control and 99% payload, the management problem is more like 90% control and 10% payload. This relationship is what makes the difference in all the brick management heuristics, and it's relationships that packers score badly on. They think they recognise the situation, trot out the knowledge packet about being neat and documenting everything, and off they go. The idea that they may be creating a stage rocket that will never get off the ground because the engines are too inefficient, and exponentially greater management of the management will is needed to compensate for lack of subtlety is hard for someone to grasp if they aren't trained to draw mental models and see patterns in them. Finally, the existence of specialist languages increases the appearance of the possible. If only everything could be as easy as SQL. One must remember:

1. It's taken 30 years
2. The kind of things it does are very restricted.
3. It's very processor intensive

SQL isn't easy. It's exactly what we've described - control through familiarity with idioms - everybody understands horrors like outer joins.

The Knight's Fork appears again here. Is it better to make a really sophisticated code, send it, and then a short message, or to send a simple code, and a longer message. What is the expected calibre and experience of the maintainer? How much can we extend their understanding in the documentation, so we can use more `complex' idioms? A long switch() statement is usually a pretty grim way to do control flow, unless you're writing a GUI event loop, where we all expect it and go looking for it.

There is no absolute measure of `complexity'. This must be born in mind when talking of complexity of algorithms and code style, and whatever it is that complexity analysis tools produce after the pictorial representations of the system, which can be very valuable. Complexity in these pictures (after the system has been reduced to necessary sufficiency) is not A Bad Thing - it is the essence of your problem laid bare. We should not be trying to drive out the inherent complexity of problems. It is a futile strategy that leads us away from the development of abstract understanding that will let us organise it.

Reviews and Previews

A fundamental element of the packer view of work is control by threat. Perform action so-and-so, or else. To make the threat effective, the rule must be policed, so we must look, after the fact, to ensure that the rule has been followed. Then the idle and untrustworthy workers will know they will get caught out and will perform the action as specified, because of their fear of retribution. In fact, an important aspect of denying the nature of mapper work is supporting the falsehood that mapper jobs can be microspecified, and the only reason why that must be done is so that rules can be specified. Only with rules written on pieces of paper can the programmers be caught breaking them, and it's the catching out that is central to the whole model!

Of course, there is an additional purpose, in that a review can also spot small oversights than must be corrected before the work is allowed to go to the customer. This is like inspecting a Rolls Royce and polishing off a tiny smear on the bonnet with a soft cloth - it can't turn a soap-box cart into a Rolls Royce.

In the software industry, we have processes that put the conventional emphasis on review, which is either trivial or a police action, but do nothing to pull the team's focus and group experience into finding the best way to do the job. This leads to reviews where individuals or subgroups produce their best efforts, which frankly are often not very good, especially if the programmers concerned are inexperienced. The time for doing the work has elapsed, so if a baroque collection of application layer processes with complex instantiation rules have been used to kludge up something the OS does for free, it is too late to redesign. The review group members have to collude to not notice the ropey logic, and concentrate their efforts onto ritualised objections to trivial matters of style. None of this does anything for quality.

The solution of course, is a mapper one. We have to accept that most programmers would love to do a really good job, given half a chance, and invest in helping them rather than threatening them. Instead of spending our half-day or whatever in a review when it is too late, we should spend it in a preview, where we can evaluate the options and agree the general direction, before the work is done. Then the situation where the reviewers pass the work with regret can be avoided, because with the workpiece already Rolls Royce shaped, the final review just need to check for trivial smears on the bonnet.

Code Inspections and Step Checks

Code inspections form an important part of many organisations processes. The primary reason they are done is to fulfill the common sense, mapper, TQM dictum: `When you have done the job, look at what you have done, and check that bit is OK.' But there is something funny about code inspections, that often deflects their purpose. This is because code inspections are like Christmas. They are an old festival, older than the structure they currently have a place in. And like Christmas, there's still a lot of holly and mistletoe hanging around from the Old Religion.

Once upon a time, programs had to b written down on coding sheets, which were given to typists to key onto punch cards. These were rekeyed for check, and then the cards were fed into the computer, which would be used for an incredibly expensive compiler run. It wasn't just the labour and capitalisation costs - the cycle time for a single attempt could be a week. So the wise developed the habit of sitting down together and examining each others' coding sheets in minute detail before they were sent off for punching.

Today we have spiffy editors, and processors in our toasters, so the original motives no longer apply, but we need to continue to inspect our code so that logical errors can be identified before they cause a fault in service. This is where the confusion sets in. Few organisations are as confused as the IT manager who recently said that the staff should perform code inspections on their listings before compiling them. For pities sake, we have our design phase to get the big picture right, and the compiler to check syntax errors. Manually poring through the code looking for syntax errors is not going to impose any useful discipline, and is a very high cost and unreliable method of finding syntax errors, even though it is the way it has always been done! The balance has changed, and we need to consult our mental maps, as with everything we do in this information intensive game.

Although most organisations let people compile and test the code before inspecting it, the holly is still up over the fireplace. Why do half a dozen people suddenly get up, abandon their whizz-bang class browsing, abstract modelled, GUI development environments, and huddle away for an afternoon with paper listing, in organisation after organisation, day after day? At least get a workstation in there, so that people can ask searching questions and get hard answers to questions like, will that value always be positive by searching the source.

Code inspections are very costly, and we should aim to get the best out of them. A very good way to do this where there is a symbolic graphical debugger available is to break the job into two parts. First the individual author, who is intimately familiar with the structure and intentionality of the code uses the debugger to single step every single line and test in the program. This may sound labour intensive and low benefit, but the effects are wonderful. The program itself naturally keeps track of exactly what the designer is checking at any time, and the designer's eye is in turn focussed on each step of the logic. One doesn't have to traverse the buggy side of a test to see it, because the mind runs more quickly than the finger on the mouse button, and picks things up so long as it is pointed in the right direction. It can be useful to print a listing, chunk it with horizontal lines between functions and code blocks, and draw verticals down the side as the sections are verified. This uses the designer's knowledge, machine assist, and takes one person, while picking up a lot of problems. The full group code inspection can then focus on things that a fresh viewpoint can bring, like identifying implicit assumptions that may not be valid, as the designer explains the logic.

Code inspections structured in this way in conjunction with individual step checks have a purpose, and are less likely to degenerate into holier than thou religious wars over comment style, as too many expensive programmers currently spend time.

Coding Standards and Style Guides

Coding standards and style guides have come up several times in this course, and what has been said may well be at variance with what is commonly believed. It will be useful to explore these differences, and understand exactly what we are talking about.

We have argued that the software industry often finds itself trying to sit between two very different views of the world. The packers have been trained to structure reasoning and discourse around identifying the situation in terms of learned responses, and then applying the action that is indicated. One's conduct in the world is basically about doing the right thing. At work, one is told what to do. The mappers seek to obtain a general understanding of the situation, reusing known patterns where they are appropriate, and making up new ones where they are not. Mappers expect to act guided by their map, given an objective. At work, one understands the problem and finds an optimal solution. We have also seen that the mapping approach is what making new software is all about, and packing can't do it.

So coding standards are part mapper motivated, part packer motivated, and have their purposes confused. The mapper/packer communication barrier applies, so mappers tear their hair out as packers smear the complexity around until it is invisible on any one page of listing, citing Knowledge Packet 47684 - Clarity Is Good, while removing all clarity from the work.

If we accept that mapping and TQM are at the heart of good programming and that mapping and TQM are all about understanding and control, we can look at the goals, and see what we can do to make better coding standards and style guides.

The first point is about clarity. There is an idea around that using the syntactic richness of one's language is A Bad Thing, because it is `complex'. Not when the compound syntax is an idiom. Not even when the idiom has been introduced and discussed in documentation. And not necessarily in any circumstances. When Newton wrote Principia, he wrote it all out in words, even though he could have used algebraic symbols, being the co-discoverer of the fluctions or calculus. Today we have decided that it is better to do maths with algebraic notation, even in exposition. Now it takes several pages of text waffle to say what can be said succinctly in algebraic notation in one page, and although the reading speed per page goes down, the total reading time goes up, because it's much harder to keep track of the text waffle. So should our programs be more like prose in their density of complexity per page or expression, or should they be more like maths? We suggest that if a person is new to a language, it is better that they can see the overall structure succinctly and struggle with each page for a few minutes than that they can read each idiot line perfectly and not have the faintest idea what it's intentionality is! How often have we seen keen people, sitting in front of reams of code, not having the faintest idea where to start, and feeling it must be their fault.

So as we walk the tree, we start by making calls that lead us left, left, left, left... until the bottom. We then wander about in the tree, visiting every combination of left, left, right, right, left... and so on, until we finish our walk by doing a whole bunch of returns from our final visit that was right, right, right, right...

Every step left or right entails opening a new stack frame, copying the argument into the stack frame, performing the call and returning. On some jobs with a lot of navigation but not a lot of per node processing, this overhead can mount up. But look at this powerful idiom, known as tail recursion elimination:

void Walk(NODE *Node)
{
Label:

   // Do whatever we came here to do...

   // Shall we recurse left?

   if(Node->Left) Walk(Node->Left);

   // Shall we recurse right?

   if(Node->Right)
   {
      // Tail recursion elimination used for efficiency

      Node = Node->Right;

      goto Label;
   }
}

We use the stack to keep track of where we are to the left, but after we have explored left, the right walk doesn't need to keep position. So we eliminate a full 50% of the call and return overhead. This can make the difference between staying in C and having to add an assembly language module to the source. For an awesome example of this kind of thing, see Duff's Device in Stroustrup's The C++ Programming Language.

The second example is a pure style issue - no need to invoke assembly language at all. Remember that when Djikstra considered goto harmful, he was referring to the habitual use of goto for flow control in unstructured 1960s code. The idea was that by using goto less, we would improve clarity. The idea was not to sacrifice clarity to avoid goto at all costs. Imagine a program that needs to open a port, initialise it, initialise a modem, set up a connection, logon and perform a download. If anything goes up, anywhere in the logic, we have to go right back to the beginning. A self-styled structuralist might say:

BOOL Done = FALSE;

  while(!Done)
  {
    if(OpenPort())
    {
      if(InitPort())
      {
        if(InitModem())
        {
          if(SetupConnection())
          {
            if(Logon())
            {
              if(Fetch())
              {
                Done = TRUE; // Ouch! Hit the right hand side!
              }
            }
          }
        }
      }
    }
  }

Which we think is just silly. There is a cleaner alternative that makes use of the way the && operator stops as soon as the statement it is in is rendered FALSE - `misuse' of the language normally deprecated in most coding standards:

while(!(OpenPort()&&
           InitPort()&&
           InitModem()&&
           SetupConnection()&&
           Logon()&&
           Fetch()));

This is clear, and neat provided we can encapsulate each step into a function. The trouble with this kind of job though, is that there are a bunch of horrid things that have to be gotten right, like initialisation strings and so forth, and to work with the code one needs it laid out like a script. In this case, we can do this:

 Start:  if(!OpenPort())goto Start;
            if(!InitPort())goto Start;
            if(!InitModem())goto Start;
            if(!SetupConnection())goto Start;
            if(!Logon())goto Start;
            if(!Fetch())goto Start;

Which is exactly what specialist scripting languages designed for this kind of job allow us to do!

Don't forget, if you want your SO to understand your love letter, you won't allow pedantries of spelling and grammar to distort the letter, and if you want your colleague to understand your program, don't twist the structure out of all recognition in the name of `clarity'.

Meaningful Metrics

We can turn the lens of practical understanding of purpose on the collection and interpretation of metrics, which many sites spend a lot of money on, so it behoves us to get them right.

There are three kinds of motives for going out and collecting numbers. All are valuable, but it is always important to understand what our motive is. The three motives are:

Descriptive Science

This involves going out and collecting data about an area to see if one can find any interesting features in the data. One needn't know what one is expecting to find; that is the point. Uncritically collected raw source data are the roots of everything. Modern entymology owes a great debt to the Victorian and Edwardian ladies that spent their time producing perfectly detailed watercolours of every butterfly and stick insect they could find. It has become something of a tradition that really interesting comets are simultaneously discovered by a professional and an amateur astronomer. Our discipline has suffered from a crude transfer of `metrics' from mass production to an intellectual, labour intensive activity. If we want to draw analogies with factories, we need to ask about what optimises the complicated human elements of our production facility. We need to spend more time on our descriptive roots. For example, do test case fails go up with code that was written in summer, when there are so many other things one could be doing instead of taking the time to check one's logic? One could brick up the windows, or factor seasonality into the site's work plan to maximise the chance of quality. What are useful indicators of quality? Internal and external fault reports per function point? KLOCS per function point?

Experimental Science

This involves making a change to an otherwise controlled environment, and seeing if the result is what we expected. It enables us to validate and improve our mental map of the workplace. It's fairly easy to do in a mass production environment, very hard in software engineering, where cycle times can be months, team members change, and no two jobs are exactly alike. One can either hire a good statistician with a real understanding of the art of programming, or look for really big wins that drown out the noise. We know there are big wins out there, because the hackers exist. This course is designed to point professionals towards under-exploited areas where big wins lurk.

Cybernetic Technology

This is where we really know what we are doing. Before we take a measurement, we know how we will interpret it, and what variable we will adjust given the value recorded. If we really had software enginering down pat, then this is what we would do. Unfortunately we don't. The field is so complex that we probably never will, but we can develop some very good heuristics. We must take care that a packer culture's need to pretend that we already have total control does not prevent us from achieving better partial control by mystifying our actions and the interpretation of such statistics as we can get.

The pattern that emerges is, don't put the cart before the horse. If we run around collecting statistics without a clear understanding of what we are doing, an important tool is distorted into a bean counting exercise. Without wise interpretation, people become more concerned with creating rewardable artifacts in the statistics than getting the job done well and letting the statistics reflect this. This is not a vice of machine tools. Without a clear cybernetic model, `bad' statistics become a stick to beat people with. they are bad people, and should consider their sins. This will surely produce improvement. People start having meetings where they try to count the bugs in different ways, to `improve the situation', but the customer's program still doesn't work.

With metrics, like everything else, we can do nothing by abdicating our responsibility to a procedure.

Attitude to Tools

And when someone enthuses about what a new product can tell you, always pause to check if the right response should be, `So what?'

Software Structures are Problem Structures

Can you imagine what Margot Fonteyn would have looked like with her arms and legs in plaster casts, and a pair of chain-gang style leg irons on her ankles? The result would not be very graceful.

One of the saddest things that able young designers do is start out with a step by step approach that enables them to start building software, and while they are doing it they get familiar with the idioms that enable them to map problem to language and approach. The whole point is that the languages and approaches they use are supposed to map to their problem domains. It is hardly surprising when they start to see and describe their problems in terms of software structures. The dividing line between a good way to describe a problem and a good way to solve it is blurred, and with object approaches and languages this intent is to maximise this blurring.

But just at the point where they could become skilled and insightful, these designers start to feel guilty, because they feel they should `see the problem, not the solution'. So they start a peculiar performance where they pretend they can't see the very thing they are talking about. If they have a specific goal to fulfill, like maintaining implementational independence, this kind of manoeuvre can be accomplished adroitly, because they know exactly what they don't know and it becomes an exercise in rigour, but if they are just pretending to be dumber than they are, where are they supposed to stop?

If you are good, you are an asset to your organisation, because you can succinctly state that solution Y is good for problem X and why, next question please!

It is not a crime to be skilled and experienced.

Root Cause Analysis

Root cause analysis is formalised as a part of many organisations process. In it, the organisation recognises situations where things have got screwed up, and looks at what happened to understand what happened and ensure that it does not happen again. Mappers do not have a problem with this - it's business as usual. But for packers its a pretty unusual thing to do at work.

To understand what is important about root cause analysis, we can look at how a mapper does the same thing, by a different name, in requirements elicitation.

Imagine a transport firm that because of its marshalling policy has traditionally categorised all jobs as rural or urban. The rural and urban split might easily work its way into every aspect of the business process. When the software engineer turns up to identify the needs for the new system, it is important that the engineer looks at the patterns of data flow, and does not get confused by the customer's continual talk of rural and urban, which has no bearing on most of the requirements, and will need big pairs of complexity piles to import into the design.

The lesson is to see what is there, not what you are told to see. This means that you must step outside the system seen by the customer.

When performing root cause analysis in the workplace, it is important to see what actually happened, rather than expressing the events in the language of the process. In a car factory, adding a rubber stop to a conveyor belt might stop damage to workpieces, but we rarely have all the elements of the situation in front of us like the parts of an assembly line. If the events are always expressed in terms of the process, the most likely conclusion is that a scapegoat has failed to follow the process, which serves the twin packer goals of allocating blame and celebrating the perfection of the process.

In fact, causes of problems can be classified in terms of the involvement of the process as:

Unconnected

The whole team went down with chicken pox for a month. We can't do anything about this by changing the process, but maybe we can gather some interesting observations to pass to senior management about the tradeoffs in risk management.

Operational

The order should have been entered into the system but wasn't. This is often taken as the only kind of problem, but actually is the most rare. Even when it occurs, the packer tendency to then assert that the sales clerk is morally wanting and regard the problem as solved is not good enough, because we have not in fact established a root cause. Perhaps the sales clerk has been poorly trained. Perhaps the process is ambiguous. The question is why the order didn't get entered. This kind of problem can sometimes be solved by messing with the process definition, but it usually comes down to issues of morale, or identification with the job. That is, sociological effects that are powerful in the workplace but outside the domain of the process.

Ergonomic

The process is OK in principle, but the implementation is not viable. The sales clerk has to do cash sales too, and the continual interruptions interfere with the accuracy of the data input. Leave the process definition as it is but apply a little common sense to the implementation.

Procedural

The process is badly defined. Customers are invoiced for parts according to the delivery notes we get from our supplier, so customers are charged for parts that haven't even been delivered to us yet when our suppliers make an error. Change the process.

Complexity Matching and Incremental Boildown

All interesting systems from a game of backgammon to calculus, have operations that make the state more complex, and others that simplify it. Most of the formal process and study of software engineering is focussed on growing the corpus. Opportunities to shrink it, with all the benefits that brings, we must take on our own initiative. We can be tasked to write a function, but we can't be tasked to realise that it can be generalised and wipe out three others.

Great lumps of obfuscation always come in pairs, one to convolute things and one to unconvolute them. This extends to requirements elicitation, where it is common for users to request the functionality of their existing system be reproduced, including procedural consequences of the existing systems limitations, and their workarounds! These are sometimes called `degrading practices'.

A problem that is emerging with object libraries is that objects really do a very good job of encapsulating their implementation. This allows us to use class hierarchies that we really know nothing about the insides of. So we can end up calling through several layers of classes, each of which (to take an example from Geographical Information Systems) mess with the co-ordinate system on the way, just to place a marker on a map. The difficulty doesn't emerge until we attempt to set up several hundred markers for the first time, and we discover that the `simple assignment' we are so pleased with is so computationally intensive it takes half an hour to draw one screen.

A project that does not regularly examine its class hierarchies to ensure that the internal currencies are natural and standard, and that the design is appropriate for the use cases found in practice, can suddenly find itself in very deep water with no warning because of this hidden cost of reuse.

As ever, the best weapon against bloat is conceptual integrity, achieved by comparing mental models of the project.

The Infinite Regress of `Software Architectures'

There was a team whose customer had asked them to build a runtime environment which could support little processes being glued together by their I/O, like a kind of graphical shell which could specify complicated pipelines. If we must classify it, it was a systems programming job. This team decided to be Professional, and Use Proper Methods. So they went and produced an object oriented model of the stuff in the URD, using a tool that automates the Shlear-Mellor approach. There was a bit of difficulty with this, because the problem was wrong in places (\fIsic), but eventually they kludged it into the Proper Methodology. Then they could use the code generator! They pressed the button, and what came out was a whole bunch of glue classes. Each one just called down to an API routine from the `Software Architecture', a layer that the approach requires, that allows the application to be mounted on a particular type of computer system. In this case the Software Architecture would be a system facility providing a graphical interface to a shell supporting complicated pipelines!