tommorris.org

Discussing software, the web, politics, sexuality and the unending supply of human stupidity.


software


Fault tolerance on the Web and the dark side of Postel's Law

I’ve been reading @adactio’s new book. Pretty much all I have read is great, and I highly recommend reading it.

But in the grand spirit of pedantic blogging, I take issue with this:

“Imperative languages tend to be harder to learn than declarative languages”

That tend to is doing a lot of work.

I think Jeremy is conflating declarative with fault tolerant here. HTML and CSS are declarative and fault tolerant. SQL is declarative and decidedly fault intolerant (and quite hard to master to boot). PHP’s type system is a lot more permissive and tolerant (“weak”, one might say) than a language like Python. The former is great for newbies and non-programmers, because adding 1 + "1" (that is, adding a string and an integer together) will give you 2, or at least something that when printed to screen looks vaguely like a two, though under the covers it may be Cthulhu.1 And the behaviour of something like Python is great for jaded old gits like me who don’t want the system to magically convert strings into integers but to blow up as quickly as possible so that it can be fixed before all this nastiness gets stitched together with the rest of the code and causes some real major bugs. The same principle applies on a grander scale with stricter type systems like Java or the broad ML family (including things like Scala, F#, Haskell etc.).

“A misspelt tag in HTML or a missing curly brace in CSS can also cause headaches, but imperative programs must be well‐formed or they won’t run at all.”

Again, depends on the language. PHP is pretty permissive. If you make a very minor mistake, you can often just carry on and keep going. If you are feeling particularly careless about your errors, you prefix your expression with an @ sign and then security people get to laugh at you. I hesitate to say this was “designed” but it was at the very least chosen as a behaviour in PHP.

This may all seem rather pedantic, but it is actually quite important. HTML and CSS are declarative and permissively fault-tolerant. That’s a key strength. In an environment like the web, it creates a certain robustness. If your JavaScript fails to load entirely, you can get some very strange behaviour if, say, a function that is expected to be there isn’t. (I use a site frequently that makes multiple Ajax calls but if one fails, say due to a bad connection, the site is unusable and must be reloaded from scratch. It is also contained in an iOS app, which must be manually restarted.) But if some of your CSS doesn’t load, that’s not the end of the world: you still get something you can read. If some of your HTML has weird new stuff in it, as Jeremy points out elsewhere in the book, that’s still backwards compatible–the browser simply ignores that element and renders its content normally.

This error handling model, this permissiveness of web technologies, isn’t a side-effect of being declarative. It’s actually a property of them being designed for the web, of human choice by the creators. There is a cost to this. It has been incredibly hard for us to secure the web. Permissive error handling can and has enabled a whole class of security vulnerabilities.

If Postel’s Law gives you the ability to use border-radius in your CSS or aside in your HTML and some terrible old version of Internet Explorer happily ignoring it without vomiting XML warnings all across your screen, then Postel’s Law also comes with the cost of having to worry about downgrade attacks. We collectively left SSLv2 running long after it should have been dead and we got DROWN. We did the same with SSLv3 and we got POODLE. These are examples of ungraceful degradation and the sad cost is your server being vulnerable to being pwned.2

With the last few attacks on SSL/TLS, it wasn’t just nasty old versions of Internet Explorer on Windows XP getting shut out of the web, it was non-browser apps that talked to HTTPS-based APIs. The Logjam attack meant that a lot of people upgraded their servers to not serve DH keypairs that are below 1024-bit. For most current day browsers, this was not an issue. Apple, Mozilla, Google, Microsoft and others released patches for their current browsers. Java 6 didn’t get a patch for a very long time. If you had a Java desktop app that consumed an HTTPS-based RESTful API which had patched Logjam, that broke with no graceful degradation, and the general solution was to upgrade to a new version of Java. On OS X, Java used to be distributed by Apple, albeit as something of a reluctant afterthought. Upgrading every desktop Java user on OS X was a bit of a faff. (My particular Java Logjam issue was with… JOSM, the Java OpenStreetMap editor.)

Postel’s Law giveth and it taketh away. One could give this kind of rather pedantic counterexample to Jeremy’s account of Postel’s Law and then conclude hastily “given how badly the web is at solving these security problems, the grass on the native apps side of the fence might just be a little bit greener and yummier”. Oh dear. Just you wait. When every app on a platform is basically reimplementing a site-specific client, you might actually get more fragility. Consider our recent vulnerabilities with SSL/TLS. After something like Logjam, the bat signal went out: fix your servers, fix your clients. On the server side, generally it meant changing a config file for Apache or Nginx in a fairly trivial way and then restarting the web server process. On the client side, it meant downloading a patch for Chrome or Firefox or Safari or whatever. That may have just been rolled into the general system update (Safari) or rolled into the release cycle (Chrome) without the end user even being aware of it. The long tail3 of slightly oddball stuff like Java desktop apps, which tends to affect enterprise users, assorted weirdos like me, and niche use cases, took a bit longer to fix.

If every (client-server) app4 that could be in a browser, in the case of a security fail, fixing all those apps would be as simple as fixing the browser (and the server, but that’s a separate issue). If everything were a native app, you have to hope they are all using the system-level implementations of things like HTTP, TLS and JSON parsing, otherwise you have a hell of a job keeping them secure after vulnerabilities. We already see things going on in native-app-land (Napland?) that would cause a browser to throw a big stonking error: user data being sent in cleartext rather than over TLS being more common than I care to think about. But the native app won’t scream and shout and say “this is a bad, very wrong, no good idea, stop it unless you really really really want to”, because the UI was developed by the person who created the security issue to start with.

The downside to Postel’s Law5 is sometimes the graceful degradation is pretty damn graceless. Sadly, the least graceful degradation is often security-related. The web might still be better at mitigating those than all but the most attentive native app developers, or not. Browser manufacturers may be better at enforcing security policies retroactively than app stores, or they might not. We shall have to wait and see.

The Web is a hot mess.

But we still love it.

  1. Sausage typing: if it looks like a sausage, and tastes like a sausage, try not to think about what it really is.

  2. At risk of giving technical backing to rising reactionary movements, perhaps the modern day variation of Postel’s Law might be: “Be conservative in what you send, and be liberal in what you accept, to the degree that you can avoid catastrophic security failures.”

  3. Wow, it’s been a long time since we talked about them…

  4. Let’s leave aside the semantics of what is or isn’t an app. Incidentally, I was in a pub the other day and saw a few minutes of a football game. There was a big advert during the game from a major UK retail brand that said “DOWNLOAD THE [brand name] APP”. Why? I don’t know. We are back at the ghastly “VISIT OUR WEBSITE, WE WON’T TELL YOU WHY” stage with native mobile apps. I’m waiting for a soft drink brand to exhort me to download their app for no reason on the side of a bottle.

  5. I claim no originality in this observation. See here and here. Interestingly, if you take a look at RFC 760, where Postel’s Law is originally described, it has a rather less soundbitey remark just before it:

    The implementation of a protocol must be robust. Each implementation must expect to interoperate with others created by different individuals. While the goal of this specification is to be explicit about the protocol there is the possibility of differing interpretations. In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior. That is, it should be careful to send well-formed datagrams, but should accept any datagram that it can interpret (e.g., not object to technical errors where the meaning is still clear).

    The first two sentences are key…


Listening to @Ben_Hall show off katacoda.com - think Codecademy for already experienced developers wanting to learn new stuff (DevOps tools like Docker and Kubernetes etc.)

Really excellent way to try out new technologies.


Standups: just use text (and maybe Slack)

Having written one long post today on software methodology, I’m going to write a short one. I have one simple argument to make: replace your morning standup with a text-based update (e.g. via Slack).

I’ve rarely ever seen standup meetings work well. I know people who have worked at companies who say that they got standups working right. But I’ve seen so many problems with them that I’ve never really seen the value in them.

First of all, people make standups way too large and with people whose jobs have no relevance to the job of building software. I’ve seen standups with over 20 people in them. Then there’s the standups where everyone gets dragged in. Senior management, middle management, marketing, sales, pretty much anyone. Quite how what they were doing yesterday or today is of any relevance to the developers—who knows? What value is hearing about what the CEO had for lunch yesterday having on helping the developer build the product? None. But standups are agile and if you are cargo-culting, the best way to be more agile is to have more people in standups and more standups and so on.

I’ve also seen people running standups on the basis of office politics rather than based around product delivery. Rather than having separate standups for different products, just chuck everyone in together. Because what someone building a completely different product is doing is somehow of relevance to people on a different project.

The only time I’ve seen a standup work successfully is when it wasn’t a standup at all but a status update posted via Slack. Now, there’s the same format and the same time constraint. You tell the team “at 9:45am (or whenever) you need to post your status update”. But it gives them flexibility. If they do some kind of personal productivity mechanism like GTD or Pomodoro, they can copy in notes from that system into the standup comment. They can keep notes from the previous day. Being at their computer means they can actually look at what is in front of them (Git commits, issue tracker updates etc.) to determine what they are working on, and they can link to those if they feel like it.

It gives more flexibility for remote workers or even just people running a few minutes late for the office. Bus got stuck in traffic? You can still post your update on Slack.

It gives more permanence. You can see what you posted yesterday and see whether the things you said you’d do yesterday were the things you actually did. This gives you more accountability. Also, having a clear log of what you did every day you worked helps when compiling timesheets, or invoices if you are a contractor.

It prevents people from going off on pointless discussions about stuff that is of no relevance to the rest of the team. They can still discuss it, just not in the standup channel.

In every standup, there’s often someone who is extremely quiet or who doesn’t like speaking. Given that people with hearing loss now make up 20% of the population, the likelihood is that the quiet or shy person can’t be heard by at least one member of the team. If you are using text-based chat, no problem. Also, I bet that the place where you do your daily standup probably ain’t wheelchair accessible.

For the remote workers, they don’t have to futz around getting Skype or Hangouts or whatever setup so they can see their coworkers looking into the middle distance.

For teams working in a shared open-plan office (so much ugh), it means less noise bouncing around and annoying other people in the office.

That person whose job has almost no overlap with yours? You can skim read their contribution and only really pay attention if it is relevant. That pointless waffler who goes into long and dull detail about his day-to-day work? He’s not wasting anybody’s time but his own.

Slack or text-based standups are literally better in every single way. There’s now a whole litany of tools you can use to have text-based standups in Slack: Squash Reports, Tatsu, GeekBot, Jell and… just having a Slack channel that you ask people to post in every day. Or Hipchat. Or Campfire, or IRC, or Skype, or whatever. However you do it, it is probably going to be less dysfunctional than most actual standups.


The software assembly line fallacy

Many software developers now smile nervously when their employers say that they are “doing agile” (doing agile what?, the pedant in me asks—an adjective modifies a noun or pronoun). You just know that the old-school project manager has gone and ‘upgraded’ their PRINCE2 certificate over to a ScrumMaster certificate from some shady place that does a weekend-long course on how to do stand-ups and retros. Getting said certificate and buying a whole boxful of post-it notes from Ryman’s is apparently all you need to be an agile project manager. Unlike a number of the world’s great religions, there’s no baptism or genital mutilation required for the agile cult, just accepting the various doctrines and the transfer of money into the hands of a local friendly guru.

Obviously, I must now make the obligatory statement that there’s nothing at all wrong with agile, ever. It is perfectly fine and truthful and excellent and if it doesn’t work for you, you are just doing it wrong and need to go to confession (retrospective) and do penance (hire a consultant to train the team more). Praise be to agile. Amen.

A recurring theme I see in a lot of companies, and online at sites like StackOverflow/Programmers StackExchange, and in personal conversation with other developers is what I’ve come to call the assembly line view of software development. It seems to be popular with people who do agile really badly (i.e. as a cargo cult slavishly following the diktats of trainers and consultants). Many people seem to think that they can deliver software in an assembly line manner while also “being agile” (I try not to ask too many questions about what they mean by that so as to save the eventual embarrassment).

Waterfall is essentially an assembly line approach. You start with requirements, and then you proceed to design, then to construction, testing, release, documentation, integration, then support. Modern assembly line software development mostly isn’t this type of waterfall anymore. Waterfalls assembly line was subdivided up by the familiar steps of the waterfall.

These days, assembly line projects are divided up based on team ability, but the waterfall nature of those projects is broadly the same. You have a project that starts with business needs collection: this is done by salesman who ask the client what they want. He doesn’t really know anything about, say, the web or software development or about apps or the current state of what is possible with technology. But he does have nice shoes and an excellent line in buzzwords. So he’ll bring back a project brief that is extremely vague and keep it hidden away in his email account, inaccessible to developers or designers.

Then the project gets handed over to designers (or “creatives”, if you are in the kind of place that calls them that). Having got the business requirements from the salesman, they then design the product in Adobe Photoshop, a photo manipulation tool that is commonly used to model complex interactions, animations and modern responsive design approaches despite not being very good at doing any of those things. At the end of this, you will have PSD files.

The next step is to hand the project to front-end developers. The front-end developers will then convert the PSD files into HTML and CSS, without any specific knowledge of where the site will be deployed or which back-end technology stack is to be used (and thus what templating languages need to be used), nor any real knowledge of the complexity of any third-party services or APIs that the system needs to hook up to. Even worse, they may not really have any idea who is using the site, or on what devices. This is not their fault: nobody ever explained to them what needs doing. You don’t have to explain to assembly line workers anything about what the other people on the assembly line are doing.

Then the application is handed to back-end developers who try and cobble what the front-end developers have done to match the back-end technology of choice. They then have to set up the integrations with the third-party services, even though said services were not understood either by the front-end developers or the designers. When none of this works, well, there’s not really a plan for that.

Finally, the application is handed off to the operations team who deploy the application. They have had no input at any stage prior to this, so if some front-end build tool or back-end or database technology isn’t compatible with whatever the cloud deployment stack they are using demands, well, that’s going to be fun. And, sure, they may have to break some stuff to make it all deploy. But it’s not like anyone is going to have to change that code, so what’s the problem?

At this point, the familiar end-of-waterfall cycle happens. Design, front-end, back-end and operations all now get increasingly frantic to try and launch to deadline. The only functioning test server is the ops person letting you SSH into what will eventually become the production server. The front-end developer has moved onto another project, so the inevitable failures to map Photoshop designs pixel perfectly to HTML and CSS is remedied by having the original designer hover next to a developer pixel-tweaking it all in Chrome Dev Tools before checking it in with a commit message of “designers can’t make their bloody minds up”. It all gets out the door, but more by luck than judgment or careful planning.

You might assume wrongly that the assembly line metaphor would include time for testing and that there would be a person whose role would include spending a decent amount of time testing the finished product before it was handed off to customers. But that never really works out in reality, because hiring testing/QA people costs money and money is better spent on sales people to get you more projects that you’ll struggle to deliver due to, well, reasons too many to count.

By my tone, I’m sure you can tell that I am not seriously proposing this as a useful pattern for software development, but attempting to sign its death warrant.

The primary problem with this is a lack of communication. Each step in the process is treated as a predictable part of an assembly line when the expertise of each part of the process needs to be used to assist the other people working on the project. The developer with the expertise in how the third-party integration works needs to be talking to the project manager and the designer up front so they are fully aware of the limitations and issues that integrating with the platform will bring. The project manager needs to give some advice to the back-end developers how the thing is meant to operate, so they can have a first cut of a back-end working for the design to hook up to once the front-end developers are done.

Like the assembly line of Henry Ford, the intention of assembly line development is to try and introduce predictability and modularity to the process. Is one developer not available? Oh well. We’ll switch in another one. Sure, he hasn’t worked with the design team, hasn’t really understood the technical complexity of what we’re asking him to do, but he’s a developer. He’ll figure it out. (About a week after it was due to go out.)

Ultimately, assembly line programming, like most methodologies, ignores experience or detailed knowledge and tries to magic creativity out of an assembly line. I’ve seen these projects and they succeed only in the sense that they get the product out the door. They succeed in spite of chaos and lack of planning and people having to pull weekends and late nights. (The contractor getting double their day rate to work the weekend is fine with this; the permanent staff member who missed their kids birthday party is slightly less happy.)

Don’t let this happen to you. If you spot your company trying to make an assembly line type of process, there are ways to fix it. If you are a developer or a manager of a development team, insist on getting the developers likely to build it in on early meetings with management and designers so said developers can guide them into designing things that will actually be possible (both technically and with the team that you have available). If you manage a developer team, make clear what they can and can’t deliver with the skills they have.

Ultimately, if you can’t stop the assembly line nonsense, people will leave. They’ll get fed up with the waterfall-style last minute crunch times, the stress and anxiety, the lack of value placed on their creative and technical skills, the disinterest in fixing these issues on the part of management, and the recurring lost weekends and overnights. They’ll disappear and they’ll take their expertise (which is needed for the next project) away with them. You probably want to avoid that. Or not. I mean, the best thing about assembly lines is that if a worker on an assembly line stops working, you can easily replace them with someone else. Because hiring good technical people is super easy, right?

The assembly line is a terrible model. What we really need are strong teams where all the people with skills are able to demonstrate those skills to each other, to use those skills to guide each other to build great software. Software projects aren’t repeatable like cars or tin cans or hammers are. Assembly lines are for building the same project over and over. Software and websites and so on are not like that.


Pi is a constant unless you are a software developer

You may think that pi is a constant like 3.14159 (etc.), but that is because you learned mathematics in school. You learn a lot more once you enter the world of work.

As a software developer, you have to get used to the fact that pi is whatever management have decided it to be this week. For one project, pi might be 3.14159, but in other projects management may decide that pi is actually the integer 4, an API call to an old system that you need to use Java Remote Method Invocation to access, an XML document that is inside a Word file in an email account you don’t have access too, or it might be green, except on Thursday when it is furry and invisible. That’s what they sold the client, so that’s what it is.

You might think that if pi is being redefined, someone might file an issue about it or raise it with you in a meeting. No, they’ll mention it to you off-hand and indirectly, or they’ll put it on page 53 of a PowerPoint slide deck with a filename like CLIENT PRESENTATION DRAFT FINAL MAYBE? CHECK WITH CAROL (1).pptx that they CC to you, except the subject line says “Drinks on Friday” and the body is about the birthday party for Steve in accounting. Or, worse, they’ll upload it to Confluence, because it has to be useful for something. When you persist with your school-derived belief that pi is 3.14159 (or the value of java.lang.Math.PI and equivalent), you will be upbraided for not keeping up with the rapidly changing needs of a growing enterprise.

When you have learned to live with the fact that pi could be anything from one day to another, and once you have learned to listen patiently to the idiot telling you that pi is actually a Labrador puppy, suppressing your strong desire to grab him by the lapels and throw him down a ricin-laced lift shaft, you have progressed to the point where you can become a senior developer, a project manager or a consultant.

None of this means that pi is anything but 3.14159. It is a mathematical constant—your maths teacher is right. If you need to deal with the cognitive dissonance, step away from your computer, have a nice holiday—or a good gin and tonic, or dance your tits off in a nightclub or whatever hedonism gets you through life. Once you’ve coped with the dissonance, pretending pi and many other similar things aren’t constants makes dealing with companies much easier. Deadlines, requirements and functional specs are mostly make believe, so why can’t mathematical constants be fictional too?


50 Lies Programmers Believe

  1. The naming convention for the majority of the people in my country is the paradigm case and nobody really does anything differently.
  2. Names are all representable in US ASCII.
  3. Unicode has properly solved the problem of language encoding.
  4. Gender is immutable and fits cleanly into an enumerated list of two options.
  5. A person’s legal name is how they identify to the world.
  6. In general, openness is preferable to privacy.
  7. Postcodes or ZIP codes are a good way to identify the location someone is in rather than an arbitrary string used for routing mail.
  8. Everyone has a phone number and that phone numbers map 1-to-1 with people.
  9. Objects of any size can be delivered to one’s home at any time.
  10. Users give a fuck about security.
  11. The tech industry is a meritocracy.
  12. The tech industry is magically free of the prejudices of wider society.
  13. Date and times are precise rather than vague.
  14. We now have the one true data representation format: JSON.
  15. Names can be easily categorised by gender.
  16. Single sign-on services reduce complexity and ease user registration.
  17. Users have a single sign-on for the single sign-on provider.
  18. There is a meaningful distinction between an HTTP resource that has been called an API and one that serves HTML.
  19. A web app is a distinct and meaningfully different animal than a web site.
  20. CSS can be “object-oriented” or “functional” rather than a declarative rules language with a moderately complex inheritance model.
  21. Unit tests catch all the problems that type checkers or static analysers would.
  22. Writing unit tests is fun rather than a tiresome necessity.
  23. Getting 100% test coverage ensures bug free software.
  24. A methodology propagated primarily through expensive training courses will lead to the production of significantly better software.
  25. Reformulating an understandable bug report (“the Froobnicator class throws an uncaught exception when the input contains UTF-8”) into a long-winded user story (“as a developer, I want to be able to run this software without seeing a 500 line stack trace when…”) will magically make it easier to plan work.
  26. Having people wholly unfamiliar with a code base performing a quick review of code style and variable naming practices will ensure that bugs are caught.
  27. Having team members unfamiliar with a particular facet of a code base come up with arbitrary estimates based on their hunches will solve all estimation woes.
  28. “Rock stars” will fix all problems.
  29. This cool new thing you saw on Hacker News will solve all your problems and can be put directly into production with no issues.
  30. Security is simply a “layer” one need add to a piece of software.
  31. GPS signals are usually reasonably accurate in most circumstances.
  32. Only mobile devices need to provide geolocation support.
  33. Anything that runs Windows, Mac OS X or non-Android flavours of Linux should not be thought of as a mobile device even if it is a teeny ultraportable laptop you carry around with you everywhere.
  34. Mobile devices are used on the move with low bandwidth, even if they are being used by someone sitting on a sofa watching TV.
  35. Syncing over the Internet rather than directly between two computers is the simplest and most efficient way to share data.
  36. Distributed version control is made even more awesome by having GitHub as a single point of failure.
  37. There are no technical fixes to societal problems.
  38. Bitcoin is a technical fix for a societal problem.
  39. apt-get install bitcoin-qt solves the usability problems of Bitcoin. (I’m not making this one up.)
  40. People basically act rationally. (Don’t worry, the majority of economists believe this one too in spite of the existence of astrologers, homeopaths, theologians, the National Lottery, and psychics claiming to be able to talk to your dead pets.)
  41. People update their software frequently.
  42. If you have too many options in your software, you just hide them away in a “hamburger” menu and the problem is solved.
  43. The social networks used by programmers in the Western world broadly reflect the social networks used by people around the world.
  44. My behaviour-driven development tool’s fancy colourful feature list HTML output is ever looked at by non-technical management.
  45. Stated MIME types accurately reflect payload content.
  46. Being able to check code in at 30,000 feet using Git (or Mercurial etc.) is a feature I shall use, rather than taking advantage of all the free alcohol on the plane to make air travel slightly more tolerable.
  47. Seconds since epoch is a sensible date format. (And there is a commonly agreed epoch.)
  48. One’s database or application framework recognising timezone-aware dates solves timezone-related issues.
  49. Arguments about methodology will produce better software.
  50. Installing homebrew to install npm to install bower to install Angular (etc.) to avoid writing a raw AJAX call is reasonable.

Software is terrible.


I rediscovered James Clark’s excellent anti-XML Schema email yesterday. It’s a thing of beauty. There really is no justification for XML Schema.

If you want to smash your head against the desk repeatedly and contemplate genocide, XML Schema is the format for you. If you want to validate an XML file against a schema, try RELAX NG instead.

And if you are wondering why I need to validate XML files and don’t just use JSON instead… there are reasons.


All programming language communities eventually rewrite apt-get. Usually badly.


Everytime I use Calibre, it seems to get uglier. I only use it for the ebook-convert command line script, but you have to install the GUI in order to get the command line tools. It’s a hulk of an application too. The OS X DMG is 75.9Mb.


Ways to improve commuting

I saw on Twitter today that there is an event in Oxford called Design Jam going on, and they are trying to work on designing things to make commuting better.

I travel back and forth to London quite frequently on a commuter train, although I don’t do a daily commute. I did commute nearly every day for about three years though. (Telecommuting is a wonderful thing.)

Here are a few ways the commuting experience could be improved a lot.

No more high-capacity seating

On train services on Southern and Southeastern, there are two types of train carriage that are used: standard and high-capacity seating. Standard seating is arranged with two seats on either side of the aisle, while high-capacity is arranged with three seats on one side and two-seats on the other, and a very narrow aisle. The narrow aisle causes access problems for people with baggage, pushchairs/prams and so on. When they used to have refreshment carts on the train, they struggled to get through the very narrow gaps.

And the seating is unbelievably uncomfortable. Everyone I know who I have discussed this with say the same thing regardless of their size or body shape: it causes aches in the back and shoulder.

Adequate, obvious places to put luggage

The number of times I’ve got on a train and watched someone put their giant rolling suitcase on the seat next to them, when there is plenty of space in the overhead and underseat storage areas is ridiculous. On the old slam-door trains, those were clearly marked and obvious for people to put their luggage in.

Noise dampening: physical and social

I don’t know how it could be done, but ways to make it so that noise doesn’t transmit as far on the train and is dampened in various ways would be ideal. The chief problem with public transport is the other people who use it. They make a bloody racket: on their phones, on their digital gadgets, on their laptops. Finding ways both physical and social to reduce digital noise and nuisance is one of the main things needed to prevent long-suffering commuters from going postal.

For instance, imagine if electronic devices had some kind of thing where when they entered a quiet carriage (of which there aren’t enough), it could signal to the user that it might be an idea to pipe down. Perhaps a little Bluetooth transmission saying “hey, you are in an area with lots of other people who value a bit of peace and quiet”. You don’t have to abide by it: technology isn’t there to stop you being a dick. But it could help to set the social expectations of the environment you are in.

Low-cost super-effective noise-cancelling headphones that are very comfortable, have inline iPhone controls and don’t take up a lot of space and don’t break when manhandled by a security guy at the airport would be a good thing too…

Information

One major annoyance when commuting is lack of information. Many trains in the south-east stop at both London Bridge and then either go onto Cannon Street or Waterloo East and Charing Cross. For some people, if you intend to carry your journey on by bus or tube, knowing which stop to get off at is quite an important thing. Yes, we could carry smartphones and obsessively check tube and bus data, but it would be very useful if, when approaching London, train announcements or information boards were to say “By the way, the Circle line isn’t running” or something like that, so that people who intend to use the Underground can make informed decisions about which station to get off at. This needs to be tempered with not being annoying.

On the being annoying front, there is a guard on the train I take who is very annoying because he goes into information overload. At every stop, he’ll list every potential interchange you can make. So, for instance, when you finally get to London, he’ll explain that if you want to go to the North and Scotland, you can get off the train, get on the Thameslink and go to King’s Cross. Well, duh. If you are in Dover and you want to go to Edinburgh, you probably don’t need the guard on Southeastern Trains to tell you how to do that. But it would be nice if he could tell you when the Thameslink (sorry, Worst Crapital Connect) isn’t running.

And finally software

A popular diversion for many who commute is gadgets, including gadgets which go on the Internet. That means smartphones, iPads, Kindles, laptops and so on. Great way to pass the time. But there is one giant problem: tunnels.

The line I use to go to London has one of the longest railway tunnels in the country: 3,156 metres long. Now, I take great sadistic pleasure in watching people get frustrated when their noisy phone calls end abruptly as they enter the tunnel. But what about tweets, e-mails and other online interactions. Here is where software could be designed better. For a lot of software, it would be very useful if it could queue up network connections in the background. Take something like Twitter: I write a tweet, and say send. The iPhone/iPad client now blocks me from doing anything until it has sent. Well, what if I’ve just lost connectivity and am in the tunnel. I don’t care if my tweet is queued to be sent out once I’ve come out of the tunnel. This kind of thing should be built into mobile OSes like iOS and Android, and even into desktop OSes like Windows, OS X and Linux. Because for a lot of people, a lot of the time, connectivity will not be constant. It’ll be very patchy. So design around that constraint.

I feel this is the kind of thing which has been lost when we all got broadband. Pre-broadband, people didn’t have Internet access when you weren’t dialed-up. I remember Netscape’s mail and Usenet client had a mode that let you queue up all your sending and receiving, and then it would connect, do all it could as quickly as it could, then disconnect again. To save money and to keep one’s phone line unoccupied for incoming calls, batch processing of network transmissions was an essential thing. Will you find that in the Thunderbird of today? Or, indeed, in the web apps of today? Now that Google Gears seems to have disappeared, it is very difficult to get offline Gmail and Google Reader: the HTML 5 promised land hasn’t arrived quickly enough to give us back the functionality Gears used to have.

Similarly, think about synchronisation: there are lots of apps on mobile devices that need to sync with ‘the cloud’ (aka. ‘servers’ on ‘the Internet’). But the only way to do so is to open them up and wait for them to sync. Dude. Background processes. I still want my e-mail to be up-to-date even if I don’t open my e-mail client before going into the tunnel. I still want my calendars to be up-to-date even if I forgot to sync them. Synchronisation should be a background process. And to keep the users from going insane, this is an ideal thing to be an OS-level service. Mac OS tried to do this with iSync, but they never quite finished it. Have a way to register syncing services and give them a priority setting.

To see how this fails, look at the Dropbox client on the Mac. I have Dropbox set up on most of the computers I use, and I think it is a fairly mediocre implementation of what I envision from such a service. But look at the Mac implementation. It’s terrible. It only runs when you are logged in. So, you log into a machine you haven’t used for a while, and it suddenly spends half an hour downloading all the crap it hasn’t synced. Even though the machine may have been sitting there idle for hours. That’s why you need background processes and good process scheduling. I don’t want a stupid little icon in the menu bar, I want it so it works in the background when I’m not there.

Having to explain these bloody obvious things repeatedly to otherwise intelligent people constantly leaves me out-of-breath. Anyway, if this were a tweet, it’d be hashtagged #firstworldproblem

Finally then, the main problem with commuting.

Cost

To go from my local station to London at peak time without any discounts, it costs thirty pounds return. That’s fucking ridiculous and needs to change if we have any hope of getting people out of their cars and living in a post-oil, low-carbon economy. This can’t be designed away, it needs fundamental social and political reorganisation.


What I Use

The Hacker News crowd have asked what people use in terms of hardware and software. Instead of posting a comment on HN, I may as well blog it instead. I hope this isn’t considered too self-indulgent,

Hardware

I use a MacBook 13” as my primary machine. It is like the 13” MacBook Pro - the unibody metal ones - in fact, almost exactly the same as the Pro one except without the word ‘Pro’ and without FireWire. This is quite annoying: I have a FireWire hard drive. Attached to it is one of two 21” 1600x1200 Dell CRT monitors that is hernia-inducingly heavy. I like that it isn’t widescreen, but I would quite like to get one of the many pretty and ridiculously cheap Samsung LCD monitors you can get now.

I have a Linux box (an AMD Sempron 2300+ clocked at 1580 Mhz) under my desk that serves as a file server, general dogsbody machine, general hacking and place for me to SSH into when I’m out and about. It’s running an oldish version of Ubuntu which I really need to wipe clean and reinstall someday. I buggered up the Ubuntu install by adding Debian repositories to my sources.list. I’m a very silly boy. I’m slowly clearing stuff off the hard drive and plan to install the most recent Xubuntu on it.

I’ve also got two netbooks: an Acer AspireOne which I bought off a friend (Caz) and an Asus Eee 1005HA. I really like the Eee: I got it at the PayPal Innovate X conference last year in San Francisco - they gave them out for free, which is a price I can’t argue with.

I’ve got an iPod touch and an iPad (32Gb, not 3G). I use a 3 MiFi to go online with both and with whatever laptop I happen to be using.

I’ve got some little computer speakers from harman/kardon. I bought them from a dodgy computer shop in Narborough Road in Leicester for a tenner. They sound pretty good but have a Euro plug on the end which I always manage to knock out of the socket converter with my foot. This annoys me quite a lot. I have resorted in the past to using blue-tack, sellotape and string to keep the top-heavy Euro plug converter plugged in properly.1

I use the iPhone headset headphones on my computers and on my iPod touch and iPad. They aren’t great headphones, but the mic and controls are important for Skype and just for controlling music.

The phone I use is a Sony Ericsson W890i, which has a pretty average camera on it, and can play MP3s and run little JavaME apps. I won this at the Over the Air conference a year or so back. I’ve also got an Elonex e-book reader, which is kind of like a Kindle without the DRM: it can read ePub, HTML, TXT or Word.

I’ve got a Pentax K100D but I don’t get much chance to use it: SLRs are too heavy for me. I’ve got some film cameras too, including a Bronica ETRS which is an amazing camera.

Network connection: 8Mb “ADSL Max”. Some piece of shit Netgear modem hooked up to a stock Linksys WRT54G router. I bought it because I could install one of the many Linux WRT setups. I haven’t bothered though.

Software

On the MacBook, Mac OS X Snow Leopard. On the other machines, Ubuntu/Xubuntu.

I now use Alfred with the Powerpack on OS X. It’s still not quite as good as Quicksilver, except it uses dramatically less system resources. On Linux, I use GNOME Do.

Text editor: Vim/GVim/MacVim and TextMate. I tend to use Vim for most things, although when I’m dealing with Rails projects I tend to flip back to TextMate. I’m trying to wean myself off TextMate and onto Vim so I can platform switch easier – TextMate isn’t being maintained very well and I want to keep a viable escape plan to Linux in case Apple goes in a direction I don’t like. I have lots of Vim plugins including Command-T and snipMate which give me most of what I want from TextMate.

IDEs: I have Eclipse, NetBeans and IntelliJ IDEA Community Edition installed, although since I mostly use Scala now instead of Java, I don’t use them. Vim works better than all of them at mostly everything, although if I was on a desktop machine more I would probably find them more tolerable. I’ve got Xcode installed but I don’t use it. I’ve tried MonoDevelop and despite not knowing C#, MonoDevelop is lovely. It’s small, loads quickly and doesn’t make me want to swear. Make more IDEs like that people.

I do as much as I can from the command line. My most used commands are vim, git, scp, cd, rm, hg, ls, mv, ssh, rake, sudo, open, heroku and cat. I use screen a lot too: you should too. It rocks.

Web browser: Firefox. It’s slow and whatever, but I have a major plugin fetish. I use Vimperator and Tree Style Tabs, and Hide Tabbar. And the usual stuff: Adblock, Flashblock, FlashGot, Greasemonkey and Delicious. I have a handy little one called CopyAllUrls, and “Firefox PDF Plugin for Mac OS X” which opens PDFs in tabs. I use 1Password for managing passwords. I wish that was on Linux. I also use It’s All Text, which lets me hit a keyboard shortcut to take a textarea I’m in and open it in my text editor of choice (used to be TextMate, now MacVim or GVim).

Music and video: iTunes and VLC. Unlike lots of people, I think iTunes is pretty damn nice. Everyone gives it a lot of shit, but it does some important things very well. The sync experience between iPods and iTunes is pretty fucking sensible, and it remembers where I am in podcasts and audiobooks which nobody else seems to want to commit to doing.

Image editing: Pixelmator. It’s like Photoshop but it loads quickly. On Linux? GIMP is okay, I guess. I’m waiting for the interface to get better. I might try Pinta or one of the other Paint.NET clone type apps that people are producing for Linux.

IM and Twitter: On OS X, Skype and Adium. On Linux, I use Empathy but have been trying some alternatives. I want to find a good command line/curses-style IM client so I can have persistent IM sessions running in screen. For Twitter, I use Echofon on the Mac and iOS. I’ve paid for a license too. It’s not very pretty but has pretty good keyboard shortcuts and syncs pretty well between desktop and mobile.2

For IRC, I use irssi. I used to use Colloquy and before that X-Chat Aqua. I find irssi to be much more useful: you can keep it running on your machine at home, and it is secure. I like it a lot. I have quite a few IRC bots I’ve written, mostly using PircBot, a Java IRC bot framework. I tend to write them in JRuby though.

Mail: All my mail is on Gmail. On the desktop, I either use the Gmail web app or I use Mutt with OfflineIMAP. OfflineIMAP isn’t as reliable as I’d like it to be, but it works reasonably well. I use the Mail app on the iPhone and iPad but I hate it. The iOS 4 version is mostly tolerable, but it top-posts which I fucking hate with a passion.

I use Google Reader. I don’t like it, but I hate it marginally less than most other RSS readers which make some fundamentally stupid design and implementation mistakes.

I’ve tried various to-do list managers including Things on Mac and iOS. I tend to find that a todo.txt file and Vim works better. I use Redmine for bug tracking on work projects.

For version control, I use Git for open source stuff and Mercurial for some smaller projects. I prefer Git: it just fits better with how I think. I have commit access to a Subversion project and use git-svn to commit to that.

For writing, I use either a text editor or I use LyX. I can’t write LaTeX without LyX though but I should probably learn someday. I don’t really use Google Docs much, but I do use EtherPad and have accounts on a few different EtherPad hosts. I have no space in my life for office apps, so the fight between ODF and OOXML, Office, OpenOffice, iWork and GDocs is completely fucking pointless and resembles a roomful of idiots arguing over who gets to ride the unicorn next.

Large chunks of my life are run by shell scripts (by which I mean Ruby scripts, Python scripts, crappy little apps fused together into some Java-Ruby-C clusterfuck that just about works, even – sigh – AppleScripts). My ~/bin/ folder could kick your Outlook’s arse.

My dream setup

I’d switch to Linux completely. That would necessitate Linux being good enough that I wouldn’t miss some of the niceness of OS X. All software would be written by sane, competent non-psychopaths who would know how to manage memory properly and not waste my CPU cycles on pointless shit and work out that i don’t want to stare at their ugly icons all day (again, OS X menu bar icons, especially the ones which claim not to be intrusive: because putting stuff in the menu bar is the most obtrusive thing you could possibly do). They would also know how to do syncing properly: between computers, between mobiles, between desktop and the ‘net. And it’d all just work by magic.

Hardware: I’d love it if we could have good desktop PCs that could scale up and down very easily. I’d love to have a machine that had, say, an 8-core CPU inside and it was able to scale all the way down to very low power usage when it’s not doing much, so it was using little more than 10W, say. And then when I’m actually using it, it could push the power usage up to normal desktop amounts.

I want a really comfortable keyboard and mouse. Or, better yet, no mouse at all. I want to use a computer with the same sort of ‘playability’ as you see with classical pianists: there’s an expressiveness in their hands, almost like a magician just tapping away at the keys and sorcery happens. I feel this when I use Vim. I want the rest of my software to get more usable the more I use it. Instead, it just gets more annoying: like walking down a busy street in London with tourists stopping and getting confused while you are just trying to get somewhere and everyone and everything is getting in your way. Computers are like that too much, and I want a computer that isn’t. We now build computers to be friendly for people in the first ten seconds of use rather than the remaining lifetime after that.

Most of all, I want decent batteries. The iPad is close to this, but we can do better. I want to be able to use a computer all day without having to charge it up. And I want batteries to last much longer than they do and charge much quicker.

I want really cheap SSDs and cheaper 10K drives. I want to never have to deal with 5400rpm hard drives again. Even better, I want an operating system that could reliably work out what to store in an SSD and what to store on disk, so in my everyday life I can use my computer without ever having to wait for things to load. Basically, the SSD is becoming another layer of ‘persistent cache’ between the disk and the memory.

I want my computer to be contextually aware: I want it to know that when I’m using Skype to make a call, it shouldn’t try and download an operating system update in the background. I want it to know that it is perfectly okay to sync my Dropbox when I’m not logged in – in fact, it is a much better time to do it when I’m not logged in than when I am because when I am logged in, I might actually want to use the bandwidth and I/O cycles for what I want to do, not what a background process wants to do.

  1. The many different plugs used in different countries annoys me greatly. Standards, people, let’s do standards. Non-UK plugs seem scarily unsafe to me too.

  2. If you are building mobile software, remember that people use desktop software too. And remember that ‘desktops’ are mobile too: laptops! That you have lots of resources to play with on the desktop doesn’t mean you need to make a big bloated application - you need to exercise the same sort of frugality as you do on the mobile. And don’t put shit on my OS X menu bar – on a small screen, this is not an unlimited resource!