• The Risk of Being Undone
• The Risk of Maintenance
• The Risk of Failure
• Pronouns in Perl
• Functional versus Structural Code
• The Act of Naming
• On Answers to Smart Questions
• Embrace the Copious Documentation
• Embracing Idioms
• Cleaning Up Bad Code
• Declarations and Scope
• Scalar Context and Arrays
  

And a bonus: Essential Skills For Perl 5 Programmers.

Are there any other good resources out there expicitly aiming at taking novices to the next level – or is that just when general “documentation” takes over as learning instruments? What’s it like in other languages?

Also, while updating with Perl news, scruffy old perl.org has become dashing new perl.org!

Updated with new articles March 8, 2010.

It may seem like the recent efforts to market Perl, as well as the Perl Ironman blogging drive, is paying off big time in terms of online attention! The graph certainly sends a clear message that Perl is alive and kicking as never before.

Note: I tried to create comparative graphs for Ruby, Python and Java, but was left with enough noise from fake gems, snake attacks and earthquakes to fill several Hollywood movies. Any suggestions for good searches for comparison are welcome.

Note 2: Maybe this is just caused by some Google indexing algorithm gone bad, but a quick visual inspection of the first 100 hits indicated that the August and September hits are real Perl mentions. Is this a real empirical indication that the recent efforts are really paying off?

The first observation anyone with some experience in programming will see, is that the majority of the roles are loop control variables (check the post, but basically they mention “stepper”, “most-recent-holder”, “most-wanted-holder”, “gatherer” and “follower” as distinct roles, as well as “fixed-value” and “temporary”, who also could be said to be typical loop control roles.

My first idea was that if all these are necessary to traverse loops, perhaps a historized scalar variable would be helpful. I.e. the “follower” could just be a ->prev() method, there could be a sum of the history and so on. And it turns out there is a variable history module: Tie::History (just remember to turn on AutoCommit).

Now maybe that wasn’t such a great idea. Or maybe it just isn’t thought properly through. Because what happens then is that you just start doing list operations on the history. Hence, maybe the list operations are just the way to go, and a more functional programming style will get rid off the elaborate loop control variables. Or so I suggested in the post.

Here, I’m trying out some attempts at bypassing the loop control variable roles by using a more declarative/functional programming style. I’ve made some examples of typical imperative style loops, and have tried to see I could get rid of the traditional variable roles by doing it in a declarative style. Have a look and see if it seems meaningful.

(Also, it gives me an opportunity to try out the awesome CodeColorer code viewer).

In my alternative take, I realized I ended up with both sort of a “most-wanted-holder” and TWO temporary variables. However, that I want to store my value somewhere, I can’t help, and at least there are no iterations using it. The temporary variables are worse, however. The point was to get rid of them after all.

This version looks a bit sleeker, but sorting could be slower than a single iteration. Another approach that should have the same profile as the intial loop is reduce from the core module List::Util:

And now we have even more internal variables in use..

Ok, the “stepper” role should be easier to remove. Here is a simple loop with a “stepper” and a “gatherer”:

It breaks my heart to write code like this. But it is also typical imperative code! This one lends itself easier to list operations too:

One construct I vividly recognize is the “follower”, described as “A data item that gets its new value always from the old value of some other data item”. I am probably not the only one having written code balancing a pair of previous/this type variables. Here is a slightly contrived example that reports every adjacent pair of the same length:

This did not turn out as easy as I expected. If you try to do functional operations in Perl using core libraries, reduce is basically the only method that let you operate on list items in a non-independent fashion. And the only way I found to do this with reduce is even more contrived than my imperative example:

Now, except from the last example, I find the declarative approach to be generally far more easy to read. While that could just be because they are shorter, I don’t think that is sufficient as an explanation for that – there is plenty of compact code that would be easier to understand if it was more explicitly stated.

Now, cognitive psychology actually has an explanation of why declarative list operations like this are easier to understand. And I’ll present the evidence for that in my next post.

Until then, there is an article about functional programming on Joel on Software that is worth reading.

Computer programming perhaps more than any other manufacturing endeavor begins with a thought and through skilled application of knowledge yields an intrinsically proven object that is itself almost mental (encoded electrical information).

It’s a good argument for why cognitive psychology is relevant for computer programming, but even more important, it points out the almost mental nature of computer programs.

Physiologically, the way our brains operate is mainly through bursts of electrical current called Action Potentials, that propagate information down neurons in a on-off fashion. Some people will call it binary, but the information conveyed is typically frequencies rather than binary patterns  (But this is a somewhat contentious question).

Here’s a beautiful drawing of neural communication from Wikipedia:

So you have a mental construction in a human brain contained in electrical signals, and this is transferred over to a construction of electrical signals in a computer.  Add that you usually want these two representations to be identical, you have a good argument why programming language design should be based strongly on cognitive science!

Neither approach is particularly good -I can’t imagine code written for violent psychopaths  would really be that great to maintain. And on the other end, a lot of coders optimize for their own readability, and will argue the finer points of formatting from that point of view, not that of the actual readers – missing the point that readability is ultimately decided by the reader, not the writer.

I certainly catch myself doing this quite often.

In the Perl world, this is probably an even bigger issue than in other languages.  There-Is-More-Than-One-Way-To-Do-It is still one of Perls big strenghts, but certainly not all the ways are equally good – which is why Perl Best Practices is now a staple of any Perl-wielding office. The contrasting Python, with it’s “there should be one — and preferably only one — obvious way to do it”, seems to be able to get away with 19 simple statements to define the pythonic best practice.

Now, anyone can have an opinion, particularly since research on code readability it still quite lacking. We just don’t know for certain yet what makes people get code. However, cognitive psychologists have been interested for several decades in how people generally organize large data structures in their brains, and some of this has given some neat practical applications for coding, such as Furnas’ paper on ‘Fish-eye’ views (1986). (This is basically about IDEs with collapsing code branches, but now you know why that’s nice, how to do it right and who thought of it first.)

Research on general language comprehension, however, is a massive field. In the ACM Journal of Computer Documentation‘s August 2000 issue, George R. Klare provides some clear-minded and research-based adviced for communicators that might be enlightening and is a good starting point for thinking about readability of normal text – that might apply to programming.

He specifies four purposes of readability:

• Reading speed and efficiency.
• Reader judgment.
• Readership.
• Comprehension, Learning and Retention.

Of these, only the last is of any big interest for programmers; you don’t typically care about the speed the maintainer needs to read your code. You might care about the Reader  judgment if you code to impress your fellow programmers, but that is another chapter. Readership applies to how the size of the readership may be a function of  the simplicity of the text – a consideration on the skill level of your maintainers, perhaps, but few people code with the intention of their code to be read by a large audience.

However, the biggest issue with text and code is comprehension. Even more so in programming code, as it is essential that the maintainer is able to create a precise representation of the code in his own mind. Also he must be able to understand both what it actually does and what the intention is, since these don’t always add up.  For debugging purposes, it is actually completely essential that the reader can separate the soft human intentions from the hard computer operations, so any code must be understood on two levels simultaneously.

Now, how does readability affect comprehension? First, keep in mind that readability in natural language usually refers to choice of words and sentence length, and is typically measured by level of education necessary to read the text. This might not be appropriate for code readability – level of programming skill might not map to level of code understanding in the same way, and readability in code is often just a matter of indentation and syntactics.

But it probably maps somewhat close. This is where research is lacking again, so it is hard to tell.

Klare’s big advice, however, is that higher readability does not always convert into higher comprehension, but is modulated by situation and traits in the readers. And that is the important part. He describes four conditions when it does not work as you would think:

1. A reader can understand at higher levels than expected if his motivation is high. Also, skill level is a fuzzy concept.
2. If time is not limited, increase in readability might not make a difference on comprehension. The more time is limited, the larger the effect of readability
3. The greater the readers background knowledge on the topic, the less effect does readability have. On the other hand, even an expert on one field may prefer higher levels of readability in texts outside his field.
4. Type and level of motivation might affect comprehension.

or to quote his summary:

[..] more readable, written material is likely to produce greater comprehension, learning and retention than less readable only when one or more of the following factors are present: the less readable is much harder than the more readable, and clearly beyond the reader’s usual level; reading time is limited; the reader does not have a large amount of background or experience with the topic being covered; and, the reader has a relatively strong set to learn.

Now does this particulary increase understanding of readability of programming code?

Perhaps not.

However, if we think of readability not as just the reading of code as natural language, but rather of understanding of the semantic concepts, there is an interesting observation to be made.  If we consider using more basic programming to make the program easier to understand, while avoiding more advanced concepts, Klare’s summary of readability shows something that also adds up with other research, namely that this approach to readability doesn’t always increase comprehension.

In a very recent piece of research using Scala, Gilles Dubouchet found that using more compact and advanced functional programming methods rather than basic, typical loop-constructs increased comprehension.  Although one single piece of research such as Dubouchet’s is too limited to base decisions on, it becomes more interesting when it actually adds up with prior research on language comprehension.

Together it indicates that using more advanced methodology can increase comprehension for both original programmers and maintainers, unless they are pressed on time or motivation.

Unlike many other languages, Perl allows you to increase the complexity of your programming across methodology quite freely. You can start with the simple baby-Perl, go through procedural programming, add objects or start playing with functional approaches. You can use Aspect Oriented Programming or add on your own crazy, homespun programming methodology, if you so please.  For comprehension and readability purposes, the above research indicates that if you consider your audience and their situation well, going for a higher and more advanced level might not be a disadvantage.

But do keep in mind that the research is still a bit patchy, and this is mostly an argument without empirical data. But I’ll make sure I report what I find, as this is just the first in many articles about readability…

What’s so good is the fairly rigorous and natural approach – instead of leaning exclusively on local students or one company’s employees performing some unnatural task, the researchers solicited for solutions online for the scripting languages (Perl, Python, Rexx and Tcl) and got 80 different implementations of a simple dictionary task. With these tasks they ran a lot of metrics and generated solid, comparable empirical data. Read it for the details (there is also a prettified version available).

I liked this observation, however:

For all program aspects investigated, the performance variability due to different programmers (as described by the bad/good ratios) is on average about as large or even larger than the variability due to different languages.

It’s quite profound – and even if the study itself is a bit dated and has some minor flaws – this observation probably still holds regardless!

Also, for Perl programmers, this statistic is quite neat:

The variability in execution speed for Perl programs were far lower than for the other languages except Tcl. It might be over-interpreting, but it indicates Perl is a bit more predictable – or Perl programmers are – than most of the other languages. Also, speed-wise is holding up quite good, and even the higher end of the range of execution speed is better than the higher end for all other languages.

The curious thing is that this indicates that the variability between programmers matters less with Perl (and Tcl) than the other languages. Or at least speed-wise, but the low variability also seems to be a tendency in the other metrics in the paper.

What’s so interesting, though, is how this demonstrates in hard numbers the importance of human variables in programming. It’s almost like framework matters less than the people using it.. funny, eh?

Also see this page of language comparison links.

This is based on a finding by Finnish researcher Jorma Sajaniemi (published work), who discovered that 99% of variables in novice’s code can be categorized in to 11 different roles. These roles are variable uses every programmer will recognize: iterators, constants, flags and so on – although in role-terminology the names somewhat differ (steppers, fixed-values and one-way flags, for example).  Mr. Sajaniemi has also found that these roles matches tacit knowledge in expert programmers – i.e. the 11 roles are also typically intuitively recognized by expert programmers even if it is not explicit or active knowledge.

Whether or not the same 11 variable roles are enough to categorize expert variable use seems harder to pin down, but there is a master’s thesis from their lab finding that they are sufficient. And, not surprisingly, in expert-written programs, the roles have a significantly different distribution. Judge for yourself, is this enough to describe your own variables?

The list is from An introduction to the role of variables, but there is also a more extensive description available.

Sajaniemi’s aim with the research and role concept appears to be teaching programming. For Perl-programmers, it can be interesting to see an article about Teaching Python using Roles, which might tell on how interesting it is for teaching Perl. Otherwise, his research in variable roles seems to revolve around a Java world.

What I find most exciting with this is the approach to studying and extending programming. Instead of going the computer science route, it looks at how people program, identifies interesting patterns and puts forward numbers and testable hypotheses.

Now can this be used to actually extend programming and help expert programmers?

My first observation is that at least five of the variables seem to all play parts in very typical loop patterns, namely stepper, most-recent-holder, most-wanted-holder, gatherer and follower. If this is such a typical way of organizing code, perhaps language design can help this – or perhaps it already does in the immutable states and for-comprehensions of functional languages such as Scala, or the map, grep and higher-order functions in Perl. Or if nothing else it may explain why expert programmers often tend more towards those constructs (or ?).

But if we know these are the typical ways variables are used, how about implementing variable roles (instead of types) with special functionality that simplifies and enhances what they are used for: most-wanted-holders that triggers events, gatherers and followers with history, walkers with an implicit track, organizers and containers optimized for moving elements or not and so on.

But if that is a good idea is hard to tell. At least in Perl, some things can be patched on with a little magical module, so it would be simple to test. I’m playing around with it, and I’ll keep you updated if something meaningful comes out of it. If you know of any other language that implements something similar, please leave a comment!

My post was not a criticism of Perl 6. I am quite skeptical and apprehensive about P6, which shone through in my article, but criticising a project that started in 2000 for not implementing ideas published in 2008 would be rather unfair. However, Perl seems to be in search of a purpose nowadays, with lessening interest and corresponding calls for better marketing.  I wanted to present a large challenge to programming in general, and to show that this is an opportunity for Perl and the Perl community.

Particularly since I don’t think better marketing is such a good idea. I think a programming ecosystem that makes you go “wow, this is really going to make my programming great” markets itself.

So this was actually more picking up the challenge presented by chromatic himself to come up with a vision for Perl.

Now, that he didn’t understand that, I can’t help. But I am quite impressed by the Perl 6 development team that they actually pick up and notice the talk in the community that quick. That’s promising for the “community rewrite of Perl and the community” that Perl 6 development originally promised.

(Actually I just wanted to show a cool debugging tool but got carried away.)

The researchers working on NPP certainly does approach programming with the aim of making it easier for laypeople, which seems like a turn-off. However, the output of the project is surprisingly interesting to the professional:  Lab studies scientifically testing the utility of design patterns (they are not all great, it turns out), examining how time is spent during programming (35% on navigating code, 22% on reading!) and how we think during debugging (why did this happen?) – to mention some of the research from their department at CMU.

And then they have actually implementation advice and real software to help utilize the findings, like the Java debugging tool WhyLine. You really have to see the 90 second introduction to WhyLine to get the point, but to me it really opens up a whole new world of possible IDE addons intelligently interpreting your code to you.

This is a presentation from the Google Tech Talks which summarizes a lot of the work they do. If you think like me on programming for novices, tolerate the first 7 minutes on making programming accessible – after that it gets good:

A final note relating this to my current programming language of choice, Perl: When Perl was first designed, it was with a lot of thought from Larry Wall on implementing natural language conventions that actually makes sense, like implicit variables, rather than blindly making it like English. Currently, however, few new things in this humanist regard has come out in the Perl world. The largest modernizing effort in Perl is the current, long-running effort to recreate Perl with Perl 6. This, however, is rather playing catch-up with other dynamic languages such as Ruby and Python on technical merits, losing what sets Perl aside as a particularly expressive and truly useful language, namely the natural approach.

At the same time, thorough, scientific study of programming such as the NPP has made serious progress past the “make programming like writing English” idea that usually plague humanist and psychological studies of programming.  If Perl wanted to build on it’s strengths, it would be better reinvented by looking at this kind of approach -how can it fulfill the virtues of a programmer better for experts who are also human?

Not surprisingly, I have some suggestions on what I would have liked to see for a modern Perl, that differs a wee bit from what is being done now:

• An IDE that uses an idea such as WhyLine.
• An IDE with better code navigation tools – watch the video to see some ideas.
• A compiler that compiles your bytecode into human natural language (English) – that is only meant for assisting reading and understanding code!
• A
  use warnings;

  that warns about constructions that are difficult to understand: do you use ! with precedence that readers understand? Do you manipulate more variables in a block than human brains can keep track of? Is your indentation following your blocks? And so on.

• Systematic testing of all the design patterns out there. Can we get quantitative data on number of bugs, development time, maintenance effort and reading time that you can expect when using a specific pattern? Then we would be able to sort out which patterns are worth using and which will be detrimental to your product – a very helpful addition to the current pattern collections and books.
• Variables that even closer follow their semantic use – how about having specialized iterator variables, temporary variables, variables with history and more. (I see that I have to explain this idea further in a later post… Until then, have a look at the idea of variable roles)

Those are some ideas of the top of my head – and not all of them of equal merit..

As the NPP provides statistics on, and as most programmers know by experience, most of your time does not go into writing fresh, new code. For an expert programmer, I dare suggest the act of expressing what you want the computer to do is rarely where the challenge lies in your 9 to 5 job. A programming language and environment that starts taking that seriously would be a truly modern and enlightened language – and winning support would be like shooting fish in a barrel.

In the Perl world, a common criticism of the Perl 6 development is that it is a solution to a non-existing problem. The above, I suggest, is the problem that is just waiting to be solved by a modern Perl.