Microsoft has done research on some popular conceptions about software engineering and come up with hard numbers on some factors affecting code quality.  Here are the main findings reported in the artice, with links to the research papers, in case the original is lost forever:

• More test coverage does not equal better code quality, as measured by number of post-release fixes.  Usage patterns and code complexity are the main reason test coverage is a poor predictor of quality.
• Test Driven Development increases quality, creating code that was up to 40% to 90% better in the Microsoft research. The cost is increased development time, 15% to 35% in the same studies. (Realizing quality improvement through test driven development: results and experiences of four industrial teams)
• Use of Assertions decrease program faults. However, this is also correlated with programmer experience, meaning more experienced programmers generally write more assertions and have less program faults. (Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation). And here is more about Assertions.
• Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. Not only that, but organizational structure was by far the best predictor of code quality, and was at least 8% percent better than the best predictor the researchers could get from code-based measurements. (The influence of organizational structure on software quality: an empirical case study)
• Organizational closeness is more important than geographical closeness, which has no effect on quality (Does Distributed Development Affect Software Quality? An Empirical Case Study of Windows Vista)

Here’s Microsoft’s article: Exploding Software-Engineering Myths (or if that doesn’t work, a Google Cache link).

One drawback with this research is that this is primarily based on case studies, which is a generally poor research method for drawing general conclusions. How valid are these observations for other organizations outside Microsoft? Is the organizational structure of your project or company actually more decisive than your programming methodology?

Also, how transferable is this to other programming frameworks. In dynamic typed languages like Perl, is test coverage more important? I often find that a sub-set of my tests do what a compiler could have done in a statically typed language, or even for Perl if I just had a more automatic testing tool. So maybe coverage would be more predictive of bugs if the compiler catches fewer mistakes? That would be a good candidate for further research.

However, this is excellent as a step towards more evidence-based software engineering.

Found through John Tells All.

Meet SEED: The Software Engineering Evidence Database. This is a project from California Polytechnic trying to make empirical evidence-based research on software engineering easily accessible. Acknowledging that “software developers are known for adopting new technologies and practices based solely on their novelty, promise, or anecdotal evidence“, the university researchers have tried to put together a database of experiment summaries on topics of interest to software developers.

The database covers subject such as OO Metrics, Design Patterns, Testing and more, and the 200+ current summaries are written by graduate students and are provided with quality ratings. The goals of the project can be summarized as:

The concept of a community-driven Web database was proposed to engage Net generation students and softwareprofessionals with evidence-based software engineering. We deliberately chose the social networking approach of user-generated and reviewed content as a way to implement SEEDS since we thought that students would more easily relate to the course project and be more enthusiastic about it.

Also have a look at the project summary: “Engaging the net generation with evidence-based software engineering through a community-driven web database” by David S. Janzena  and Jungwoo Ryoob.

Neither approach is particularly good -I can’t imagine code written for violent psychopaths  would really be that great to maintain. And on the other end, a lot of coders optimize for their own readability, and will argue the finer points of formatting from that point of view, not that of the actual readers – missing the point that readability is ultimately decided by the reader, not the writer.

I certainly catch myself doing this quite often.

In the Perl world, this is probably an even bigger issue than in other languages.  There-Is-More-Than-One-Way-To-Do-It is still one of Perls big strenghts, but certainly not all the ways are equally good – which is why Perl Best Practices is now a staple of any Perl-wielding office. The contrasting Python, with it’s “there should be one — and preferably only one — obvious way to do it”, seems to be able to get away with 19 simple statements to define the pythonic best practice.

Now, anyone can have an opinion, particularly since research on code readability it still quite lacking. We just don’t know for certain yet what makes people get code. However, cognitive psychologists have been interested for several decades in how people generally organize large data structures in their brains, and some of this has given some neat practical applications for coding, such as Furnas’ paper on ‘Fish-eye’ views (1986). (This is basically about IDEs with collapsing code branches, but now you know why that’s nice, how to do it right and who thought of it first.)

Research on general language comprehension, however, is a massive field. In the ACM Journal of Computer Documentation‘s August 2000 issue, George R. Klare provides some clear-minded and research-based adviced for communicators that might be enlightening and is a good starting point for thinking about readability of normal text – that might apply to programming.

He specifies four purposes of readability:

• Reading speed and efficiency.
• Reader judgment.
• Readership.
• Comprehension, Learning and Retention.

Of these, only the last is of any big interest for programmers; you don’t typically care about the speed the maintainer needs to read your code. You might care about the Reader  judgment if you code to impress your fellow programmers, but that is another chapter. Readership applies to how the size of the readership may be a function of  the simplicity of the text – a consideration on the skill level of your maintainers, perhaps, but few people code with the intention of their code to be read by a large audience.

However, the biggest issue with text and code is comprehension. Even more so in programming code, as it is essential that the maintainer is able to create a precise representation of the code in his own mind. Also he must be able to understand both what it actually does and what the intention is, since these don’t always add up.  For debugging purposes, it is actually completely essential that the reader can separate the soft human intentions from the hard computer operations, so any code must be understood on two levels simultaneously.

Now, how does readability affect comprehension? First, keep in mind that readability in natural language usually refers to choice of words and sentence length, and is typically measured by level of education necessary to read the text. This might not be appropriate for code readability – level of programming skill might not map to level of code understanding in the same way, and readability in code is often just a matter of indentation and syntactics.

But it probably maps somewhat close. This is where research is lacking again, so it is hard to tell.

Klare’s big advice, however, is that higher readability does not always convert into higher comprehension, but is modulated by situation and traits in the readers. And that is the important part. He describes four conditions when it does not work as you would think:

1. A reader can understand at higher levels than expected if his motivation is high. Also, skill level is a fuzzy concept.
2. If time is not limited, increase in readability might not make a difference on comprehension. The more time is limited, the larger the effect of readability
3. The greater the readers background knowledge on the topic, the less effect does readability have. On the other hand, even an expert on one field may prefer higher levels of readability in texts outside his field.
4. Type and level of motivation might affect comprehension.

or to quote his summary:

[..] more readable, written material is likely to produce greater comprehension, learning and retention than less readable only when one or more of the following factors are present: the less readable is much harder than the more readable, and clearly beyond the reader’s usual level; reading time is limited; the reader does not have a large amount of background or experience with the topic being covered; and, the reader has a relatively strong set to learn.

Now does this particulary increase understanding of readability of programming code?

Perhaps not.

However, if we think of readability not as just the reading of code as natural language, but rather of understanding of the semantic concepts, there is an interesting observation to be made.  If we consider using more basic programming to make the program easier to understand, while avoiding more advanced concepts, Klare’s summary of readability shows something that also adds up with other research, namely that this approach to readability doesn’t always increase comprehension.

In a very recent piece of research using Scala, Gilles Dubouchet found that using more compact and advanced functional programming methods rather than basic, typical loop-constructs increased comprehension.  Although one single piece of research such as Dubouchet’s is too limited to base decisions on, it becomes more interesting when it actually adds up with prior research on language comprehension.

Together it indicates that using more advanced methodology can increase comprehension for both original programmers and maintainers, unless they are pressed on time or motivation.

Unlike many other languages, Perl allows you to increase the complexity of your programming across methodology quite freely. You can start with the simple baby-Perl, go through procedural programming, add objects or start playing with functional approaches. You can use Aspect Oriented Programming or add on your own crazy, homespun programming methodology, if you so please.  For comprehension and readability purposes, the above research indicates that if you consider your audience and their situation well, going for a higher and more advanced level might not be a disadvantage.

But do keep in mind that the research is still a bit patchy, and this is mostly an argument without empirical data. But I’ll make sure I report what I find, as this is just the first in many articles about readability…

This is based on a finding by Finnish researcher Jorma Sajaniemi (published work), who discovered that 99% of variables in novice’s code can be categorized in to 11 different roles. These roles are variable uses every programmer will recognize: iterators, constants, flags and so on – although in role-terminology the names somewhat differ (steppers, fixed-values and one-way flags, for example).  Mr. Sajaniemi has also found that these roles matches tacit knowledge in expert programmers – i.e. the 11 roles are also typically intuitively recognized by expert programmers even if it is not explicit or active knowledge.

Whether or not the same 11 variable roles are enough to categorize expert variable use seems harder to pin down, but there is a master’s thesis from their lab finding that they are sufficient. And, not surprisingly, in expert-written programs, the roles have a significantly different distribution. Judge for yourself, is this enough to describe your own variables?

The list is from An introduction to the role of variables, but there is also a more extensive description available.

Sajaniemi’s aim with the research and role concept appears to be teaching programming. For Perl-programmers, it can be interesting to see an article about Teaching Python using Roles, which might tell on how interesting it is for teaching Perl. Otherwise, his research in variable roles seems to revolve around a Java world.

What I find most exciting with this is the approach to studying and extending programming. Instead of going the computer science route, it looks at how people program, identifies interesting patterns and puts forward numbers and testable hypotheses.

Now can this be used to actually extend programming and help expert programmers?

My first observation is that at least five of the variables seem to all play parts in very typical loop patterns, namely stepper, most-recent-holder, most-wanted-holder, gatherer and follower. If this is such a typical way of organizing code, perhaps language design can help this – or perhaps it already does in the immutable states and for-comprehensions of functional languages such as Scala, or the map, grep and higher-order functions in Perl. Or if nothing else it may explain why expert programmers often tend more towards those constructs (or ?).

But if we know these are the typical ways variables are used, how about implementing variable roles (instead of types) with special functionality that simplifies and enhances what they are used for: most-wanted-holders that triggers events, gatherers and followers with history, walkers with an implicit track, organizers and containers optimized for moving elements or not and so on.

But if that is a good idea is hard to tell. At least in Perl, some things can be patched on with a little magical module, so it would be simple to test. I’m playing around with it, and I’ll keep you updated if something meaningful comes out of it. If you know of any other language that implements something similar, please leave a comment!