Isn't the idea of doing things just your way a great one?

A screenshot of a portion of the Pylons home page, showing a plethora of choices.

It sure sounds great, but it's too simplistic a view.

Chapter 4 of my copy of The Mythical Man-Month has a permanant bookmark now: it is the chapter about conceptual integrity, and after rereading it this weekend, I've finally figured out that conceptual integrity is exactly what continues to frustrate me about Pylons. When there is no conceptual integrity, a product is unusable as the basis of further programming, and a product with no conceptual integrity is fundamentally incomparable to one that does. Brooks says as much in the very first chapter - in fact, on the first page:

One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales.... Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.

A lot of things have changed since Brooks wrote those words, but I'm continually amazed at how accurate they remain. Today, the URL mapper, the template engine, the request object - these are things where a garage-group project will more than suffice. And their work will quickly converge to similar, swappable components.

But these things are, as trivializing as it sounds to say it, commodities. They take time and effort to make, but that is all.

Brooks says that moving something from being a program to a programming product takes three times as much effort as writing the program itself. This effort is put into documentation, maintenance, refactoring, and stability. This effort has not been put into Pylons.

A partial screenshot of mislabeled, incorrect, and difficult-to-read Pylons documentation

It manifests itself in everything from the font size (too small) to the section header (mislabeled) to the content itself (which is out of date in claiming that the other documentation is out of date) 1.

And, of course, there is the sheer incredulity of the reader that authentication cannot be handled by the framework, but must be home-grown. As if merely there to puntuate my point, the "solution" for authorization:

A partial screenshot of unhelpful Pylons documentation

These are not trivial issues. If a project lacks conceptual integrity, you will forever be plagued by changing interfaces, continually deprecating APIs, and most importantly, you will always lack consistency in how things are done.

There are a million excuses that can be told a million ways: the libraries and the framework are not 1.0. They are in development. They are improving. "Patches welcome." I don't buy these arguments, for one simple reason:

The bazaar is not a catch-all. If you are going to release software with the intent of other people using it, take the time to document it, take the time to test it, to use it, so that other programmers aren't left out in the cold:

A partial screenshot of the ToscaWidget homepage, showing a warning that the documentation is out of date

(I first noticed the message on the 21st of November. I pointed it out to my Pylons-content2 coworkers later that week, and they weren't worried - it hadn't even been a week since the notice was put up! Well, it's almost a month later, and I, at least, am not surprised to still find it there. I just hope no one tried using it, only to now find themselves stranded.)

I wish I could say that what I have written so far is unfair. I wish it were overly harsh, or even in bad faith. But it's not: projects which are not documented are not useful for anyone other than the authors, and Pylons lacks useful documentation. It gets old, fast, for everyone else.

How To Have Conceptual Integrity

Every once in a while, programming system products - as Brooks calls them - are made which are joys to program for and in. Django is not a complete product: it is just one piece of an ecosystem which must include databases, webservers, operating systems, and even, to some extent, client-side browsers. But it is Django that fills a rather large piece, spanning from persisting data - the mechanics of which are dealt with by databases - to generating markup for users, at which point webservers take over. And Django fits this difficult role splendidly.

The root cause of this (so far as I can tell, as an outsider) is that Django was made for an already-existing need. As such, the programmers who made it had a very clear sense of what needed to be accomplished, and in this way, they acheived conceptual integrity. Pylons, in contrast, has been made and marketed as a Rails for Python. Unfortunately, even Rails has conceptual integrity where Pylons lacks it. Rails was the first in its domain, and as such mostly had to be written from scratch; the same was true of Django. Pylons, on the other hand, tried and tries to use preexisting modules wherever possible, without regard to how well they fit within the framework.

Worse, it was developed in expectation of, rather than in tandem with, applications. Surely, someone writing applications would have realized how awkward it was to use a class with a __before__ method instead of a function with a decorator, but somehow, Pylons has missed this, and now it's too late to break backward compatibility in such a dramatic way.

Unfortunately, that's not a very useful observation. What's done is done.

Fortunately, I think there are a few other things to point out that can be done to improve the conceptual integrity of a project.

Opinionation is Not Bad

E.B. White's The Elements of Style is chock-full of writing gems. In most situations, what he writes isn't very controversial, but on issues where English leaves some ambiguity - and there are a lot of those - White does something refreshing and extremely effective: he takes a side.

Singular possessive with an ending s? Doesn't matter, the trailing S should not be removed (Charles's objects, etc). There is no particular reason to follow that convention; it is, largely, a matter of preference. In the long run, it is as insignificant as your choice of text editor, templating engine, login shell or intentation style. There is only one important thing: that you decide. And then you must be consistent.

Reuse is Not an End

Software reuse is a programming holy grail - or, perhaps, a siren song. I think it's far more useful to say that we should not reimplement without reason, than to say that we should avoid reimplementation. The latter is certainly in line with conventional thinking, but it's missing the point. It's the same thinking behind calling the request argument to each Django view repetitive. Well, is it not repeated for every controller? Yes - as it should be. The exact wrong "solution" is to therefore make the request object a global variable, as Pylons does. The goal here is not to avoid typing "request" many times - the point is to say exactly what you mean. And in a controller, you mean to accept a request and return a response, and that is exactly what Django makes clear, and what Pylons obscures behind multiple global singletons.

The misguided quest for reuse at all costs can be seen in how Pylons approaches libraries. A choice of multiple ORMs, templating languages, URL dispatchers? Why? So that every programmer's Pylons installation will be just different enough to be unmaintainable by other Pylons programmers? Or perhaps so that any attempt to keep your app generic results in going directly to the lowest common denominator?

I have no idea what the theory behind making everything a plugin layer was. I only know what happens in practice. What happens in practice is that development time which should be spent on writing solutions that work was instead spent porting Myghty - which was used because it was once the default - to Mako (the new default), and then yet more development time spent wrangling over whether to use Genshi or Mako. And in the end, other merits aside, we ended up using Mako because it was the default, and in most situations, that will be the case: the default will be used for maintainability purposes (and quite rightly so). The "best tool for the job" almost always consists of "the most easily maintained tool for the job," which itself correlates strongly with "what comes out the box." The fewer pluggable, add-on, additional-setup-required components there are, the better. Two templating languages is not a feature, it's a mistake. Four? That's a nightmare.

But Pylons seems to have missed this point. They hold reusability on too high of a pedestal. This is the same misguided attitude that brought us Java and Zope; wrapping everything in endless wrappers does not bring code reuse. Making all templating languages work through Buffet does not mean that suddenly templating languages are interoperable; it just means that templating language implementers have yet another entry point to code for.

Buffet deserves special mention. Unlike, say, a JSON decoder and encoder, you are never - quite literally, never - going to change templating engines without also changing the templates themselves. You are going to go through all of your controllers to change the template filename. There's no way around that. So why, then, are the Pylons developers so intent on making the templating engine something that can be changed completely from a config file?

It's just plain missing the point, that's all.

Software reuse is not an end. Software reuse is a means, and if the available means don't meet your ends, then find other means - write it yourself, if need be. Touting software reuse as a feature is missing the point: it's like a database implementer touting that they use a particular parser. That's wonderful. Now how fast can you deal with my queries?

Make Decisions So Users Won't Have To

Don't Make Me Think is technically a book (and a good one!) about web usability. But the framework user is the programmer, so many of the same principles apply. One of the points that the author makes is that mindless decisions - usually about where next to click - are not nearly as "expensive", usability-wise, as difficult, conscious decisions. (My copy is at work, so I'm going from memory.) Likewise, asking your users to choose between multiple templating languages, ORMs, URL dispatchers and Javascript libraries is not doing them a favor.

When someone uses a framework - or, for that matter, a programming language - what they are really doing is delegating decision-making to someone who they think will have better judgement. Consider the disagreements over the decorator syntax in Python: people trusted GvR's judgement as to how to handle such a large project, even if they individually had different tastes. With Django, I have no problem putting the same trust in Django's authors. I don't like some pieces of their ORM, and I sometimes miss Mako, but I know that they're not going to introduce, barely document, and then deprecate something like SAContext. (This is one instance where I appreciate Ben Bangert's stance on Elixir, even if it's not the one I would have made. A decision is better than change.) And even then, quite often I find that, six months later, I can still edit my Django templates without a problem, but my Mako ones tend to have accumulated a fair amount of cruft, and I can't help but wonder if they weren't right after all. I can't analyze and agonize over every decision; instead, I leave those things to people who are better at it.

A partial screenshot of the Django homepage

Pylons is not doing this for me. They ask the programmer to make too many decisons, and their components are disparate enough to make them lack a fundamental consistency.

What does Pylons value? What are its fundamental concepts? Where is its conceptual integrity?

I want to see Pylons take on these things, and I think it'll be better for it. Until then, I will be using Django, when the choice is mine.

  1. It could, of course, be said that using this example is unfair, since the document is very out of date. Yet, when I click "Docs" on the Pylons homepage and search for "authorization", this is the first entry. I know, because I was bit by it. I don't care about what technical reasons there are behind it (it's Confluence's fault, you should search for "authkit" instead, etc etc) - just get it fixed, because there are people out there today who are struggling with this. ↩