Author Archive for lennon

A Tale of Two Conferences

This summer, I attended (and gave presentations at) two conferences here in Portland: Open Source Bridge and OSCON. Broadly, the subject matter and target audience for these two events are similar (web developers, open source programmers, sysadmins, and mobile developers) but their approach, culture, and (for lack of a better word) “personalities” are quite different.

Overall, I think that OSBridge more effectively captured the spirit of the community from which it springs. The lack of the usual sponsored keynotes, expo hall stuffed with vendors, and “big shot” community leaders made the software itself, and the community building it, the main attraction. OSCON, by contrast, is much more focused on products in the OSS space, and talks are often a sales presentation wrapped in a technical overview.

On the downside, the quality of talks at OSBridge was highly variable. With fewer proposals to choose from, it was inevitable that some weaker or less-experienced speakers would end up on the program. (Hell, they even let me speak.) OSCON attracts a pretty stellar level of presentation, with rare exceptions (again, yours truly) and has an air of legitimacy that helps convince employers and sponsors to support the event.

I also don’t think that the commercial focus of OSCON is inherently a bad thing. I for one am happy to get paid to work with and on open source software, and I don’t begrudge those attempting to derive their livelihood entirely from it a few minutes to pitch their wares. If anything, I think the Portland tech community should be more focused on commercializing their projects, the better to sustain and improve the pool of talented developers who choose to live and work here.

Unfortunately, OSCON has two critical flaws. First and foremost, it’s too expensive. While the catered lunches, free coffee, and after-hours parties are all lovely in their own right, the total cost of the conference (around $1200 for the normal sessions only, and closer to $1800 with a full set of tutorials, plus airfare and lodging) puts it out of reach for many developers who simply want to get together and hack and share tips with their peers.

Secondly, many of the “keynote” presentations are simply apologia from proprietary OS and platform vendors, and really shouldn’t have a place in a conference focused on open source. I don’t mind hearing a pitch from a talented speaker, but market-speak-centric presentations on “cloud computing” and “open standards” are anathema to the DIY spirit of a true hacker’s conference.

Sadly, addressing both of these issues concurrently is a serious challenge, as I can only presume that Microsoft, et. al., paid handsomely for the opportunity to present themselves. Other events manage to keep costs down, however, so I assume there’s some room for cost savings that wouldn’t require moving the event to a spare O’Reilly warehouse and catering it entirely with Folger’s coffee and stale PB&J sandwiches.

Personally, I find I have more fun (and often learn more) at smaller, more focused events like JSConf, Ruby Future, DjangoCon. Emerging Languages Camp was another perfect example of this — by stripping down the focus of the event, the organizers and speakers crammed an incredible amount of high-quality content into two days.

I find myself wondering if OSCON might not work better if it were split into an “executive” event focused on platform and service offerings, and the “hacker” gathering which narrowed in on code, tooling, and collaboration. I still very much enjoy getting to see community leaders like Rob Pike and Larry Wall speak, and will happily continue attending OSCON if I can do so safe in the knowledge that I don’t be inundated with salesdroids throughout the week.

Mobile Roundup

As a compulsive gadget hound, I have now owned and used mobile phones (and one tablet) running Android, WebOS, and iPhone OS/iOS. This is my brief and totally unscientific take on the strengths and weaknesses of each platform:

Android has an excellent mix of app availability, features, and flexibility. It also demolishes any other platform in terms of variety of available hardware; sadly, most of that hardware is crap that is hardly worth the “subsidized” carrier price.

I used an HTC G1 (for which I paid the full, no-contract price) as my primary phone for more than a year before finally getting tired of waiting for upgrades to make their way down the value chain to us early-adopters, and left the platform in a huff about six months ago.

I’ve been playing with the Nexus One handset I got at OSCON last week for a few days now, though, and have since discovered t that Android can deliver a quality user experience on sufficiently high-end hardware.

Prior to receiving the N1, I had switched to a Palm Pre Plus on Verizon’s CDMA network. I was lured over by the promise of a more reasonable UX than was available on Android, along with the quality of Verizon’s 3G coverable. (I was also swayed in large part by the offer of free WiFi tethering, which has proven useful and reliable, if a bit of a battery hog.)

Since then, I’ve been alternately thrilled and disappointed by the Pre. The overall design of the core OS and apps are excellent, and the hardware is a very satisfying size and shape, despite feeling somewhat plastic-y. Unfortunately, the few 3rd-party apps available tend to be buggy (I’m looking at you, Foursquare!) and the seemingly-contradictory pairing of horrid battery life and s-l-o-o-w performance have left me lukewarm-to-negative, about the whole experience.

Which leaves, of course, the 800 lb. gorilla that is the iOS ecosystem. I resisted ownership of any such device for many years, but finally caved when the iPad was released. Since then, I’ve clocked a fair number of hours (and spent quite a bit of cash on apps) to put the iPad through its paces. I rather like the hardware, love the handful of core apps (Instapaper, Reeder, Twitteriffic) that typify good mobile UI design, and still can’t help feeling dirty for being involved at all.

Even after developing an app or two, I can’t shake the sense that I’ve given up something truly precious in exchange for some shiny trinkets. That being said, Apple absolutely owns the market for mobile customers who will actually, y’know, pay for content and add-ons. Android has a ways to come in that respect, and neither HP/Palm, Microsoft, or RIM appear to be anywhere near a realistic app/media store solution.

In the near-term, I expect to shift over to the Nexus One as my default mobile handset, if only because it’s the newest and fastest hardware available to me. I also still believe strongly in the potential of WebOS, but only if HP gets off their respective asses and delivers some hardware that can deliver on the system’s potential.

It seems safe to predict that the iOS platform will be an important piece of the software ecosystem, especially for shops like mine, but I can’t say that I’m particularly excited about helping Apple continue to steamroll over the protests of developers. Dealing with their byzantine and controlling approach to software development and distribution in a professional capacity has made me seriously wistful about the good ol’ days of writing Ruby and administering Debian boxen.*

* – this should not be construed as dissatisfaction with my current job, or a criticism of our choice to publish on Apple’s platform. To the contrary, I lobbied for the latter out of my desire for the former to continue in a mutually-fruitful way for a long time to come. I don’t have to like Apple to do business with or through them.

Brewing

I’m not going to apologize for not blogging in such a long time. Between Twitter, face-to-face conversations, and actually doing stuff, I just haven’t had as much to talk about here. That being said, there have also been opportunities lost for sharing ideas and recipes (of both the delicious and software kind), so I’m going to try to get back into the habit of posting medium-length reports on stuff I’ve made.

As a teaser, here’s a picture of yours truly midway through construction of a wort chiller, which I have since put to use on a couple of batches of beer:

DSC_0040

I’ll write up some more substantial observations about brewing beer later; for now, suffice to say that I consider it among the essential skills for modern man, right up there with bacon curing and being able to throw a baseball.

* for those who don’t grok brewer jargon, a wort chiller is a heat exchanger used to cool the “wort”, which is the hot-tea-like mixture of sweet grains and bittering hops that gets fermented into beer.

2009 Roll-up

Since Jan. 1 2009, I have:

  • Changed jobs – and not just jobs, but entire industry sectors, switching from services for software developers to publishing and good-old-fashioned e-commerce.
  • Bought a house (my first!) with my lovely girlfriend Hannah
  • Bought a car (not my first, but the first in ~5 years) and driven to Montana, Idaho, and Washington for bike rides and other general fun
  • Joined the Citizen Campaign Commission, in order to help oversee Portland’s publicly-funded elections

It’s not an earth-shattering list, but there’s at least some sign of positive movement there, and I’m optimistic about what 2010 should have in store.

Code Reading

We do full code reviews at my shop — no code goes into production without at least two pairs of eyes on every line of the change.

As I switch between projects, I find myself willing to be absolutely ruthless in my code reviews when reading other people’s Python code. When I’m looking at PHP or Ruby source, I expect a certain amount of license to be taken with indentation, naming, and even encapsulation. Python? No dice. If you’re gonna use the language, use it right.

Use a less-than-descriptive variable name? Rejected.

Call an internal implementation method (ex.: _do_stuff()) outside the class that defined it (even in a unit test)? Rejected.

Let a line go over 120 characters in width? Rejected.

Forget to provide a useful comment for the test used in an if: block? Rejected.

I probably should be just as strict with PHP, Ruby, etc., but the culture doesn’t seem to be as forgiving of hard-and-fast style guidelines in those communities.

Django for Rails devs

I’ve recently made the transition from full-time Rails development to a mix of technologies including, in large part, Django. Since I was a Python guy before I ever started using Ruby, the transition has mostly been an easy one, but there are some fairly significant differences in design and philosophy between the two frameworks that are worth thinking about if you’re picking a platform for a new project. Given that most of the comparisons out there on the Intertubes seem to be woefully out of date — the first page of Google results is mostly populated by articles that are 3-4 years old — I thought I might toss out some of my own highly-subjective observations out there anyone else trying to evaluate both stacks.

Similarities

Generally, the two projects are more alike than they are different, at least from the POV of a working web developer. Either one will give you a nice boost in productivity when building non-trivial applications where time-to-market trumps hard performance, runtime platform, or office politics. Much as Ruby and Python offer similar competitive advantage to the teams using them, neither Rails and Django will leave you struggling to keep up with other agile web development teams (or conversely, easily coasting past them).

Both Django and Rails have fairly powerful object-relational layers baked-in, and have good support for popular open source databases, including MySQL, PostgreSQL, and SQLite3. They both offer flexible URL mapping/routing tools, fairly easy-to-learn standard template formats, and (at least for Rails >= 2.3) the ability to insert middleware into the fast path of your request/response cycle, either to manipulate the request data, or to short-circuit loading in cases where you don’t need the whole framework for a particular client request.

In addition, both frameworks benefit from an active, supportive community that will help you get up to speed and answer questions. The online documentation for each project is fairly extensive, although I personally feel that the Django folks have done a better job of pulling the 80% of the docs that most developers need when they’re getting started into one place, with a consistent style and voice.

Django wins

There are a couple of big ones: form classes and idiomatic use of Python modules.

Form classes

This is, without a doubt, my favorite feature of Django that simply has no real equivalent in Rails. Basically, a Django form lets you de-couple the HTML editing UI for a model from its storage and business logic. The big win here is due to the fact that form validation != domain model validation. Depending on your application, you want to allow users to populate forms with more or less information than would be stored in a single model class instance, and validate those forms using their own internal validation logic, rather than delegating all validation decisions to the model class.

As a case in point, consider a simple comment form. If you add a CAPTCHA to the form, you could make your Django form class perform checking of that field, and display errors in CAPTCHA solving alongside those affecting the other comment fields, without forcing the comment model to even be aware that its views relied on such protections. Furthermore, once that form class was implemented, you could re-use it in any number of views without duplicating the display or validation logic. This is simply a better way to build views than the Rails model of helpers and shared controller methods, and I would encourage the Rails community to find some way to provide more support for structured views, especially in the areas of form handling.

Python modules

Ruby and Python both support modules as a namespace construct. However, much as Ruby classes default to having private instance variables and public methods, the Ruby module type is largely opaque except when used as a mixin to a class implementation. Few Ruby libraries would be written to use instances of the Module type directly; rather, they would expect either an instance of some class, or a hash mapping symbols to objects. In Python, however, modules are “open” by default, with any defined names visible both inside and outside the module — just as Python objects default to being public bags of attributes.

This leads to one very natural means to connecting applications and components in a Django project: passing modules (or module names) as first-class constructs. Want to “mount” all the URLs in a pluggable Django app into your project? Use include('myapp.urls'). Want to override the content model classes used in a CMS workflow app? Parameterize the application with an optional model namespace module, and look up the needed model classes in that module at runtime. Dependency injection, “duck typed” polymorphism, etc., can all happen at the module level, and the entire Django framework (and well-written reusable applications that use it) capitalize quite effectively on this capability.

Rails wins

There are two major areas where Rails stomps Django: testing tools and database schema migrations.

Testing tools

This one should be obvious to anyone who compares recent conference talk or blog post titles from the Rails and Django worlds. For those who don’t want to click through: RailsConf had 3-4x as many testing-focused sessions, and it was mentioned in many if not most other talks. The highly-scientific Google Fight also shows a much higher amount of online discussion of Rails testing, adding further anecdotal evidence to support the argument that the Rails community is at least talking about testing a lot more than their Django counterparts.

Rails developers, as a community, have been thoroughly bitten by the testing bug, and are always on the lookup for better ways to write more copious and useful collections of tests for their applications. This has led to the development of great tools like RSpec, Shoulda, WebRat, and Cucumber for authoring tests, along with a huge supporting cast of libraries, reporting tools, and howto guides to make testing as easy as possible for Rails developers. Django has TestCase and TestClient, with a smattering of support from other Python tools like Windmill and python-nose to speed things along.

I recognize that most of the Ruby modules I linked to above are not part of the Rails core, and than there are lots of similarities in the features available for testing in either Ruby or Python. The difference is that most Rails developers I’ve talked to use the full breadth of testing tools available to them, and extol their use to others, while the Django community takes a much more lackadaisical attitude about testing outside of the Django core. (Even the bundled “contrib” apps in Django often have weak test coverage — as an example, there are zero included unit or doctests for the django.contrib.admin in my local django trunk checkout at r11578).

Schema migrations

ActiveRecord migrations are not the solution for all possible database changes in real-world applications, but they cover most cases in a consistent, easy-to-learn way. Once they learn a handful of migration library methods, Rails developers can happily write clear, lightweight database manipulation routines that allow their application database to evolve as requirements change without having to resort to non-portable low-level SQL queries. This is a Good Thing ™, and worth emulating in other frameworks.

South is an entirely-reasonable implementation of a very similar model. The Django core team should adopt it as part of the platform, or implement their own simplified version. This is an obvious case where the perfect is the enemy of the good.

Conclusions

Generally, I’m pretty happy about working with Django instead of Rails these days. Whereas I spent days struggling with obscure classloading issues trying to trivially extend the Rails framework, I’ve been able to make use of the module-driven pluggability of Django to swap out cache backends, template libraries, and entire domain model namespaces in my Django apps without much more than a brief foray into the source for an external library or two. Modulo the testing issues I raised above, and the lack of a really good equivalent to ActiveMerchant, I would call myself a fairly satisfied Django user, at least until the next big thing comes along.

There are some less technical reasons to consider Django over Rails, as well. First, who doesn’t love any typically male-dominated developer community which adopts such a ridiculous mascot? Also, I attended both RailsConf and DjangoCon this year, and have to say that I enjoyed the latter quite a bit more. The difference in location for the two conferences may have altered my perceptions a bit, but I personally had more fun at DjangoCon. I also didn’t overhear anyone describe themselves as a “rockstar” at there, which was just fine with me.

For all our sakes

I’ve switched jobs twice in the last 12 months. It’s certainly not unheard of in my trade to bounce around a bit, and it’s not the first time I’ve had the experience. It has, however, reminded me of many of the unique challenges associated with trying to quickly get up to speed with a body of existing code, and especially those idioms and misfeatures which most complicate the ramp-up process.

Since most of what I work on these days is web application code, the issues below will be focused there, but most of the basic concepts should hold true for most any type of programming.

So, here are my top three recommendations for anyone who expects other people to have to eventually read or maintain their webapp code:

Logic/template separation

Web developers need to be willing to switch between 4-5 languages from moment to moment: HTML, CSS, JS, SQL, and a general-purpose language for business logic. That being said, for the sake of all who will read your code after you write it (including your future self), avoid interleaving languages arbitrarily within a single logical block of code (method, source file, or module).

I still routinely see markup, Javascript, and Python/PHP/Ruby code mixed in the same source file, usually with one language nested inside a loop defined in a another, emitting yet another syntax for consumption by the browser.

If you’re generating dynamic Javascript, create a JSON array which can be iterated over by plain, static JS code, rather than interpolating values directly into JS method calls. Similarly, when producing HTML, minimize the logic in the “template” sections of code. If you’re interleaving database queries, ‘foreach’ loops, and emission of <tr> and <li> elements, it will be nigh-impossible to change the business logic being used without also breaking the layout, and visa versa.

Commented code blocks

If you’re using version control, there should be no need to leave large blocks of code commented out or disabled. (If you’re not using version control, stop reading this immediately and go buy a book on Subversion, Git, or Mercurial. Come back when you have your version control workflow established.) Commit messages, revision diffs, branches, and supplementary documentation (such as a team wiki, another tool in the “must have” category) should provide a sufficient amount of sideband communication about proposed or unfinished code.

If you leave large amounts of inactive code lying around, on the other hand, you’re encouraging bit-rot and cargo-cult design. Your commented-out code will not have test coverage, or be kept up to date with internal API and schema changes.

None of this applies to example code — usage tips provided in comment blocks for documentation purposes are handy, as long as they’re kept up to date. I object specifically to operational code which is disabled en masse rather than removed.

Conventional coding

Even after settling on a programming language and framework, most teams still have a lot of leeway in terms of how to structure their code. Naming conventions, whitespace, inline documentation, and module layout are usually left largely up to you. However, there are some major benefits you can realize simply by imposing some basic rules for consistent coding style across your entire project.

You should start with basic syntactic conventions: 4-space tabstops, braces in K&R style, class names in StudlyCaps, etc. At some point, you may want to make a full cleanup pass across your codebase that does nothing but enforce these standards to avoid polluting your working patches with simple readability cleanups.

From there you can move on to more semantically-meaningful rules: no mutable global variables, JavaDoc/PHPDoc/Python ReST Docstrings for all public API entry points, use bind params instead of string interpolation when building SQL queries, etc.

The payoff

As you move through the code making these changes, you’ll find certain regular and repeated patterns emerge as the line noise of differing style and sloppy coding evaporates. Now you’re ready to start refactoring. Lift duplicated code into utility functions, bundle those functions into classes with shared state or domain knowledge, and then arrange those classes into useful packages.

Most of the responsibility for these tasks falls squarely on the shoulder of the development team. However, management has a critical role to play as well: when your developers begin grumbling about unmaintainable code, before giving them leave to start over or abandon existing working applications, press gently to see if a bit of housekeeping like what I’ve outlined above might let them work a little longer with the current implementation.

There are any number of reasons why an old implementation can and should be abandoned — obsolete technology, dramatically changed requirements, heavy turnover — but “ugly” or “messy” code shouldn’t be sufficient justification on its own.

Open Source Bridge presentation

In case anyone stumbles here looking for the notes and examples from my Open Source Bridge talk, here they are:

osbridge_2009.zip

Note: this is a ~30MB download, since it contains (amongst other things) a full copy of JRuby 1.3.1 and the ActiveMQ runtime. The actual presentation and example code are very light.

You can also just view the talk slides, though they aren’t terribly informative without the code.

Update: video is available on blip.tv now. My apologies for the long delay while everyone downloaded the demo archive in minutes 3:00-8:15 or so.

Better late than never

I was reading a brief but interesting post surveying the current state of the art in security as programming language features, and realized that a lot of the links overlapped with the material from the paper I wrote for my security theory course a while back. Rather than re-post all of those as a blog entry, I thought I should probably just put a link to the finished PDF.

Given that this was a school paper, I hope that folks will forgive the somewhat stilted grammar and obviously-academic format. If you get nothing else from it, though, the bibliography may at least be of interest.

Never do today what you can put off ’til tomorrow

In many ways, this is a golden age for web developers: we have a bunch of good, high-level frameworks for writing apps in highly-productive dynamic languages and a solid corpus of best practices for testing, service API design, and data serialization. We don’t have to deal with dog-slow CGI scripts, complicated J2EE stacks, or proprietary ColdFusion code that only runs atop expensive application servers.

Unfortunately, all is not wine and roses (or scotch and bacon, or whatever). The major dynamic webapp frameworks push you by convention into doing the bulk of your application work syncronously in the request-processing loop, rather than asynchronously in a background thread. All of the accumulated wisdom about building responsive graphical user interfaces gets thrown out and re-discovered by each framework’s user community, resulting in a multitude of solutions for the basic problem of pushing work into a queue and dealing with it later.

As the fine folks at Twitter so famously discovered, synchronous processing puts a hard upper limit on how much (and how quickly) you can scale an application. Even at the much more modest loads my current project at work receives, there are quite a few performance problems that can’t be solved by simply throwing more stuff in memcached and hoping for the best.

Some folks are starting to catch on, and bake asynchronous processing into their frameworks by default, but the solutions tend to either be limited to very particular deployment and application models, or esoteric in the extreme. Meanwhile, desktop application authors continue to politely chuckle at all of our bumbling, and old-skool enterprise developers look at our hackish background-worker implementations and (rightly) consider them to be toys compared to the classic “big boy” message queueing solutions, or even the newer open source alternatives.

The next generation of web application frameworks should be designed around the idea that work is done asynchronously by default, with a fallback to syncronous jobs only in cases where a user needs to see the result immediately. Since applications also need to scale across a potentially large and heterogenous set of CPUs and servers, those delayed jobs also may not be running in the same memory space as the web application itself. That means machine and language-agnostic serialization, fast network IPC, and callback and event-driven programming.

Developers who grok these concepts now will have a leg up on the competition when building tomorrow’s crop of web applications.