Unfortunately, all is not wine and roses (or scotch and bacon, or whatever). The major dynamic webapp frameworks push you by convention into doing the bulk of your application work syncronously in the request-processing loop, rather than asynchronously in a background thread. All of the accumulated wisdom about building responsive graphical user interfaces gets thrown out and re-discovered by each framework’s user community, resulting in a multitude of solutions for the basic problem of pushing work into a queue and dealing with it later.

As the fine folks at Twitter so famously discovered, synchronous processing puts a hard upper limit on how much (and how quickly) you can scale an application. Even at the much more modest loads my current project at work receives, there are quite a few performance problems that can’t be solved by simply throwing more stuff in memcached and hoping for the best.

Some folks are starting to catch on, and bake asynchronous processing into their frameworks by default, but the solutions tend to either be limited to very particular deployment and application models, or esoteric in the extreme. Meanwhile, desktop application authors continue to politely chuckle at all of our bumbling, and old-skool enterprise developers look at our hackish background-worker implementations and (rightly) consider them to be toys compared to the classic “big boy” message queueing solutions, or even the newer open source alternatives.

The next generation of web application frameworks should be designed around the idea that work is done asynchronously by default, with a fallback to syncronous jobs only in cases where a user needs to see the result immediately. Since applications also need to scale across a potentially large and heterogenous set of CPUs and servers, those delayed jobs also may not be running in the same memory space as the web application itself. That means machine and language-agnostic serialization, fast network IPC, and callback and event-driven programming.

Developers who grok these concepts now will have a leg up on the competition when building tomorrow’s crop of web applications.

Then came Subversion. It improved on many of the failings of CVS — notably, Windows support was dramatically better, repositories could be shared over HTTP, and many operations that just didn’t work in CVS (renames, binary diffs, etc.) performed reasonably well out of the box. To this day, Subversion is a reasonable choice for many projects, especially given the advanced level of support for it in IDEs, graphical repository browsers, and the like.

Much of the reason for that diversity of useful tooling built atop Subversion, of course, is that it was written in C, and built with an eye towards allowing high-level languages to use bindings into the same runtime libraries upon which the ’svn’ command itself relied. In fact, Python, Perl, Java, and Ruby are all supported by the core Subversion maintainers, and additional bindings using those same underlying libraries are available for a number of other languages.

Enter the distributed version control systems: Git, Mercurial, Bazaar, Darcs, and their ilk. The basic workflow they offer is in some ways more like RCS than it is Subversion: each developer works locally against their own copy of a repository, and they share their work via patch files and periodic synchronization. (This is of course a gross over-simplifaction, as all of them offer much more sophisicated change-tracking under the hood than RCS did, but the user-visible behavior is still reminiscent.) However, their ability to maintain change history across many developers and systems without forcing everyone to eventually squash their work down into a single source tree makes a number of new modes of project management possible, or at least much easier than before.

All of the above DVCS systems potentially offer a huge gain of productivity for many developers, since you can easily experiment with changes locally, selectively share only those modifications which you wish to, and continue working without being connected to the central repository. (This is especially significant for those whose employers maintain draconian firewall rules and disallow off-site access to their source control.)

Unfortunately, none of the popular DVCS systems have anything resembling the level of cross-language API support that Subversion does. Mercurial and Bazaar are both implemented in Python, making access from other Python code quite fast, and that from any other language painfully slow. Git is implemented in C, but without a supported and documented core library of functions designed to be used to facilitate access from other languages. Darcs is written in Haskell, which means only crazy mathematicians and CS majors have any ability or interest in using it. (I’m kidding here, but the point remains that Haskell isn’t exactly the most useful substrate for scripting language bindings.)

The fallout from all of this is that we’re left using wrapper libraries which fork out to the command-line tools for each DVCS. Such wrappers have a number of problems: the performance sucks, the internal APIs are usually only as robust as the set of regular expressions you write to parse the output of the commands, and almost no work is shared between the various wrapper implementations.

Don’t get me wrong: as a simple version control tool, I’ve found Git in particular (and distributed version control in general) to be a big step up from the old centralized-repository model. However, the very eighties-esque fork-and-regexp-scrape model for IPC — coupled with the lack of an obvious “best of breed” leader in the DVCS space — means that I (along with anyone else trying to support DVCS in a general-purpose way) end up doing a lot of low-level grunt work when we could be building real value for users.

Even something as simple as a standard dump format for a common subset of the information available from the popular DVCS types would be a start. I do know that, for the time being, I’m stuck supporting a bunch of very brittle code which relies on the various idiosyncratic console output formats of each version-control system.

Playing prognosticator, I would even go so far as to suggest that the first DVCS system to provide supported, documented interfaces in a number of popular programming languages could climb to the top of the dogpile that exists currently and emerge as a clear standard.

use CGI;
use LWP;

my $ua = new LWP;
my $req = new CGI;

my $res_id = $req->param('rid');
my $img = $req->param('img');
my $url = "http://somehost/cgi-bin/fetch.cgi?id=$res_id";

$req->get($url, :content_file => $img);

open FH, $img;
unlink $img;

print $req->header(-type=>'application/octet-stream');

while (<FH>) {
        print $_;
}
close FH;

How horribly bad is this script? Well, it allows no less than the deletion/overwriting of any file writable by the web server user. While that won’t allow injection of shellcode under most configurations, it would allow an attacker to delete logfiles, insert malicious replacements to files in upload directories, and generally mess with your system in all kinds of ways.

Even better, it completely misuses the Content-Type HTTP header to force download instead of inline view, instead of using the semantically-appropriate Content-Disposition: attachment route to force a download dialog box to appear on the client.

There are doubtless millions of lines of code like this out there in the world, and (at least in Perl-land) almost all of them could be caught with the simple addition of the -T (”taint check”) flag to the #!/usr/bin/perl line at the top of the script.

Unfortunately, I’m still chafing a bit at a number of little annoyances:

• The REPL is basically crippled since you can’t define functions. Being forced to think in terms of compilation units (rather than simple expressions) pisses me off.
• Why oh why do I need to explicitly list the module name in my file header if I’m also bound by the restriction that filenames and module names have to be the same? The old Java package/file path ties were always a big annoyance when I was stuck in that environment.
• For a functional language, there’s an awful lot of syntactic vinegar for basic operations like map and fold. I appreciate having a concise syntax for lambdas, but writing fun my_function/2 smells a bit.
• Records (as syntactic sugar for tuples) are a poor substitute for a real type system. Both tutorial and real-world Erlang code I’ve seen is basically full of tagged tuples, which means you get the verbosity of a strongly-typed language without any of the ability of real type checking to catch errors at compilation time.

I want to stick with it long enough to find the real gems underneath all this noise. I mean, if I can sit through extended sessions reading and writing Perl, I should be able to find something to love about Erlang. Furthermore, most of the complaints I make above are inapplicable to mainstream languages — i.e., C and Java dont have an REPL or lambdas, and Ruby and Perl don’t have anything resembling a traditional compiler — not miraculously better.

I definitely think that learning a new language should make you feel a little bit uncomfortable. Unfortunately, right now Erlang leaves me feeling uncomfortable in all the wrong ways: I understand everything that’s going on with the language, and just don’t like it.

I’m going to keep plugging away for at least a little bit longer, though. Next up: reading the source to EJabberD to (hopefully) get a sense for idiomatic language use in a context where its unique features (lightweight concurrency + distributed computing) are a real advantage.

I’m unlikely to work as a proprietary desktop software developer any time soon, given that I haven’t tried to write a desktop GUI application in about five years, but I do appreciate the work that Panic puts into polishing their user experience, and have a great deal of respect for them as a development team. And so, it was extremely refreshing to me to see many of the fears and frustrations I’ve expressed about the iPhone/App Store platform lock-in as an outside being echoed by someone who is working within that ecosystem.

One realization I had early on in my IT career was that almost any two firms could easily become competitors. Given the rate at which a competent team can develop most any class of application on the planet, and the generality of the platforms on which our code runs, it is entirely conceivable that your platform vendor of choice today (or your client buying your tools or OS) could become a direct competitor tomorrow. For that very reason, trusting the future of your business to the continuing benevolence of a vendor is a very risky decision.

Every developer working on applications for the iPhone/iTouch platform is in exactly that precarious position right now. Those innovative applications that drive sales of the platform could easily be incorporated into the next release of the OS, since Apple has more than enough resources to re-implement any 3rd-party app without breaking a sweat. (There is certainly precedent for this: just ask the developers of Watson what it’s like to one-up Apple in the system utility space.) Even worse, truly disruptive applications that offer capabilities beyond Apple’s comfort zone (or that of one or more of their mobile network providers) could find themselves summarily dropped from the App Store, effectively strangling any business the developer hoped to build around the product.

This is, to me at least, an unacceptable bargain, and I’m surprised that so many other developers seem to have no problem with the arrangement. (Have I mentioned how much I can’t wait for Android?)

Podcast Producer does to the production of podcasts what the assembly line did to automobile production.

ORLY?

The parents, in particular, seemed to have basically given up on their kids being able to find work, due to the classic “you need experience to get experience” Catch-22. I personally found myself desperately wanting to yell back at them to stop whining, and start helping their kids out with something other than rides to the mall.

Going to school absolutely does not prepare kids for the workforce. Nothing short of work prepares people for work. There is an endemic assumption right now that the only reasonable course for a young person is through the K-12 system, then straight on to college, perhaps with a brief detour into a volunteering stint along the way to help pad the college application. This track produces exactly the kind of clueless, over-privileged 22-year-old that Baby Boomer managers love to complain about.

Admittedly, my own perspective on this is a bit skewed, compared even to a lot of my coworkers and friends. I started working in the summers when I was 14, and year-round by the time I was 16. My first job was running a summer reading program at the local county library branch, and I locked it in before the interview even started by being the only applicant to show up wearing a tie, resumé in hand. I continued working for the year I was in college, and have been supporting myself (and other people as well, occasionally) since I was 18.

Not every one of those jobs has been pleasant — packing boxes for shipping at a Mailboxes, Etc. during the Christmas season, for example, or making salads at a pizza shop — but I’ve learned something useful from each and every customer, project, and boss. 

Of course, no parent would want to listen to the advice of a college dropout who has no kids of his own, right? Obviously, college and a pure white-collar background are the only real path to success.

That being said, I am quite experienced as a customer of other firms, and so have this one simple tip to offer anyone selling high-ticket consumer goods:

If a customer comes to you, ready to buy, do not send them away unsatisfied. Even more importantly, do not tell them that, while you can no longer sell them the product they evaluated 48 hours prior and are now itching to buy, you will happily accept 20% of the purchase price in order to motivate you, the seller, to acquire another such item so that they might buy it later.

This is an especially egregious offense if the product is neither particularly rare, nor difficult to acquire from one of your competitors.

Failure to remember this simple tip may cost you a long-term customer.

We use MIT Kerberos, OpenLDAP, and Cyrus SASL to provide single-sign-on across our network services at Reed. I’ve been working for a while to port legacy Perl-based systems over to Ruby, and using ActiveLdap for much of that work, but have hit a snag that limits its utility to use in many cases: namely, that GSSAPI-mech SASL binds fail if there is no corresponding LDAP directory entry for the principal under which you’re doing the bind.

Let me explain a bit of background:

OpenLDAP ACLs allow you to specify something like the following:

authz-regexp
uid=(.*),cn=gssapi,cn=auth

This basically says, “for any principal that our authentication backend accepts, treat them as a valid LDAP entity with the DN ‘uid=[principal-name],cn=gssapi,cn=auth’”. This is incredibly handy when coupled with Kerberos keytab files, as it lets us get the same basic benefit as certificate-based authentication without maintaining a certificate authority. (As an aside, we do actually maintain a small CA setup, but since the CA data lives on an external drive in a locked cabinet, rolling a new cert is a huge pain compared to creating a new keytab from kadmin.)

This is all well and good, and lets us set up background processes which have keytab-based privileges to selectively read and edit protected attributes on our directory, without putting passwords into configuration files or source code. It also means we don’t have to create a full-fledged directory entry for every little background service which may need to authenticate.

Unfortunately, ActiveLdap is unable to determine that it has successfully bound to the directory if the SASL GSSAPI principal it uses to bind doesn’t have a corresponding entry. It just loops over the reconnect method until the configured number of attempts is reached, then falls back to an anonymous bind.

If all of that was gibberish to you, rest easy; like I said, our particular mixture of infrastructure and directory-maintenance practices is pretty rare, especially outside academia, so you’ll probably never have to worry about any of this.