And it makes me think, to what degree are organizations’ code bases shaped by their formal or informal organization structure? Are core modules and root objects often the domain of senior developers and objects lower in the hierarchy the domain of juniors? My experience is that it often tends to be, and it seems a reasonable overlap: after all, you want your more trusted developers fiddling where the damage can be greatest.

But how about other attributes of the code base? In the world of Perl, are CPAN authors often hired as external consultants? Are the most communicative programmers the ones that will write network services? Are the modules most used also written by the programmers that are most in contact with others?

And does organization structure also shape general code base structure? Will a more hierarchical organization tend more towards hierarchical object structures, while more chaotic or flat organizations tend towards more chaotic or flat code organization?

A lot of questions, but no answers… But one thing that comes to mind is, if organization and code structure follows each other, is this a good idea? I think few people designing a data model or object hierarchy starts with the organization structure as a blueprint, but speaking from my own limited experience, you can often at least see a reflection of either in both. Is this a good or bad thing? What can the consequences be?

Microsoft has done research on some popular conceptions about software engineering and come up with hard numbers on some factors affecting code quality.  Here are the main findings reported in the artice, with links to the research papers, in case the original is lost forever:

• More test coverage does not equal better code quality, as measured by number of post-release fixes.  Usage patterns and code complexity are the main reason test coverage is a poor predictor of quality.
• Test Driven Development increases quality, creating code that was up to 40% to 90% better in the Microsoft research. The cost is increased development time, 15% to 35% in the same studies. (Realizing quality improvement through test driven development: results and experiences of four industrial teams)
• Use of Assertions decrease program faults. However, this is also correlated with programmer experience, meaning more experienced programmers generally write more assertions and have less program faults. (Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation). And here is more about Assertions.
• Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. Not only that, but organizational structure was by far the best predictor of code quality, and was at least 8% percent better than the best predictor the researchers could get from code-based measurements. (The influence of organizational structure on software quality: an empirical case study)
• Organizational closeness is more important than geographical closeness, which has no effect on quality (Does Distributed Development Affect Software Quality? An Empirical Case Study of Windows Vista)

Here’s Microsoft’s article: Exploding Software-Engineering Myths (or if that doesn’t work, a Google Cache link).

One drawback with this research is that this is primarily based on case studies, which is a generally poor research method for drawing general conclusions. How valid are these observations for other organizations outside Microsoft? Is the organizational structure of your project or company actually more decisive than your programming methodology?

Also, how transferable is this to other programming frameworks. In dynamic typed languages like Perl, is test coverage more important? I often find that a sub-set of my tests do what a compiler could have done in a statically typed language, or even for Perl if I just had a more automatic testing tool. So maybe coverage would be more predictive of bugs if the compiler catches fewer mistakes? That would be a good candidate for further research.

However, this is excellent as a step towards more evidence-based software engineering.

Found through John Tells All.

THE SYSTEMATIC DEVELOPER: Writes code defensively. Does everything he or she can to protect code from unstable and untrustworthy processes running in parallel with their code. Develops a deep understanding of a technology before using it. Prides himself or herself on building elegant solutions.
THE PRAGMATIC DEVELOPER: Writes code methodically. Develops sufficient understanding of a technology to enable competent use of it. Prides himself or herself on building robust applications.
THE OPPORTUNISTIC DEVELOPER: Writes code in an exploratory fashion. Develops a sufficient understanding of a technology to understand how it can solve a business problem. Prides himself/herself on solving business problems.

Now why should you care?

Almost every mention I’ve seen of this online – or of any other personality type categorization system – is usually followed by a “which type are you?”. This misses the point utterly and completely. Psychological research like this first becomes really valueable when you stop thinking about yourself and start asking how this can help you understand other people. If you design API’s and base your design on what makes most sense to your own coding style, you will create something that two thirds of your audience will find difficult to use. Even if you don’t like or agree with their style.

Granted, that makes the assumption that programmers are always equally distributed among styles, which is a pretty wild assumption. The point is that other people are more likely to think differently than similarly to you.

That is also a good thing to keep in mind when formatting code for readability: if your coding style differs from standard Perl Tidy or your company’s coding standard, keep in mind that you are not formatting for yourself, but a colleague, maintainer or anonymous CPAN downloader. They are more likely to understand a common standard than your standard.  It sounds obvious, don’t it? I don’t think many (any) programmers think like this even so.

Now, Clarke, in an article to Dr. Dobbs Journal, has an example of a cognitive mapping of programmer types and API traits which is quite illustrative.  In Figure 1, thick blue lines shows the expectations of a particular programmer type, while the dark lines shows the score of a particular API. As you can see in this case, the match is bad. Now the good thing is that Clarke’s research gives you a framework to discuss how and why.

That’s important because programming style isn’t related to experience level or educational background. An programmer with an opportunistic style will not necessarily grow into a systematic programmer with more experience, neither will a systematic programmer become more pragmatic with age. Or perhaps they will – but you can’t assume that.

Finally, the dimensions which Clarke suggests APIs can be understood on at a cognitive level:

• Abstraction level. The minimum and maximum levels of abstraction exposed by the API, and the minimum and maximum levels usable by a targeted developer.

• Learning style. The learning requirements posed by the API, and the learning styles available to a targeted developer.

• Working framework. The size of the conceptual chunk (developer working set) needed to work effectively.

• Work-step unit. How much of a programming task must/can be completed in a single step.

• Progressive evaluation. To what extent partially completed code can be executed to obtain feedback on code behavior.

• Premature commitment. The amount of decisions that developers have to make when writing code for a given scenario and the consequences of those decisions.

• Penetrability. How the API facilitates exploration, analysis, and understanding of its components, and how targeted de- velopers go about retrieving what is needed.

• API elaboration. The extent to which the API must be adapted to meet the needs of targeted developers.

• API viscosity. The barriers to change inherent in the API, and how much effort a targeted developer needs to expend to make a change.

• Consistency. How much of the rest of an API can be inferred once part of it is learned.

• Role expressiveness. How apparent the relationship is between each component exposed by an API and the program as a whole.

• Domain correspondence. How clearly the API components map to the domain and any special tricks that the developer needs to be aware of to accomplish some functionality.

Steven Clarke seems to have discontinued his blog, but his scientific work is chronicled at Scentific Commons.

(Found, among other places, assertTrue() ).

The first observation anyone with some experience in programming will see, is that the majority of the roles are loop control variables (check the post, but basically they mention “stepper”, “most-recent-holder”, “most-wanted-holder”, “gatherer” and “follower” as distinct roles, as well as “fixed-value” and “temporary”, who also could be said to be typical loop control roles.

My first idea was that if all these are necessary to traverse loops, perhaps a historized scalar variable would be helpful. I.e. the “follower” could just be a ->prev() method, there could be a sum of the history and so on. And it turns out there is a variable history module: Tie::History (just remember to turn on AutoCommit).

Now maybe that wasn’t such a great idea. Or maybe it just isn’t thought properly through. Because what happens then is that you just start doing list operations on the history. Hence, maybe the list operations are just the way to go, and a more functional programming style will get rid off the elaborate loop control variables. Or so I suggested in the post.

Here, I’m trying out some attempts at bypassing the loop control variable roles by using a more declarative/functional programming style. I’ve made some examples of typical imperative style loops, and have tried to see I could get rid of the traditional variable roles by doing it in a declarative style. Have a look and see if it seems meaningful.

(Also, it gives me an opportunity to try out the awesome CodeColorer code viewer).

In my alternative take, I realized I ended up with both sort of a “most-wanted-holder” and TWO temporary variables. However, that I want to store my value somewhere, I can’t help, and at least there are no iterations using it. The temporary variables are worse, however. The point was to get rid of them after all.

This version looks a bit sleeker, but sorting could be slower than a single iteration. Another approach that should have the same profile as the intial loop is reduce from the core module List::Util:

And now we have even more internal variables in use..

Ok, the “stepper” role should be easier to remove. Here is a simple loop with a “stepper” and a “gatherer”:

It breaks my heart to write code like this. But it is also typical imperative code! This one lends itself easier to list operations too:

One construct I vividly recognize is the “follower”, described as “A data item that gets its new value always from the old value of some other data item”. I am probably not the only one having written code balancing a pair of previous/this type variables. Here is a slightly contrived example that reports every adjacent pair of the same length:

This did not turn out as easy as I expected. If you try to do functional operations in Perl using core libraries, reduce is basically the only method that let you operate on list items in a non-independent fashion. And the only way I found to do this with reduce is even more contrived than my imperative example:

Now, except from the last example, I find the declarative approach to be generally far more easy to read. While that could just be because they are shorter, I don’t think that is sufficient as an explanation for that – there is plenty of compact code that would be easier to understand if it was more explicitly stated.

Now, cognitive psychology actually has an explanation of why declarative list operations like this are easier to understand. And I’ll present the evidence for that in my next post.

Until then, there is an article about functional programming on Joel on Software that is worth reading.

Computer programming perhaps more than any other manufacturing endeavor begins with a thought and through skilled application of knowledge yields an intrinsically proven object that is itself almost mental (encoded electrical information).

It’s a good argument for why cognitive psychology is relevant for computer programming, but even more important, it points out the almost mental nature of computer programs.

Physiologically, the way our brains operate is mainly through bursts of electrical current called Action Potentials, that propagate information down neurons in a on-off fashion. Some people will call it binary, but the information conveyed is typically frequencies rather than binary patterns  (But this is a somewhat contentious question).

Here’s a beautiful drawing of neural communication from Wikipedia:

So you have a mental construction in a human brain contained in electrical signals, and this is transferred over to a construction of electrical signals in a computer.  Add that you usually want these two representations to be identical, you have a good argument why programming language design should be based strongly on cognitive science!

He makes a point about user interfaces, but it is good as a general observation:  A good-enough solution that improves life for many people is better than a perfect solution that can’t possibly be made. Also, he points out that classic programming theory will teach you a lot of computer-sciency methods for problem solving, while in real-life are heuristic methods trying to understand user input as good as possible. That, however, I think is because he is a good programmer, not something all programmers come across: I bet the majority still will approach unexpected input with a dying program..

Or, as an otherwise very good programmer of the old-school I had the honour of knowing once said to a customer who complained about a particularly harshly crashing program: “Considering what you tried to do, that was a mild reaction”.

Also, there is this cartoon.