Discussing software, the web, politics, sexuality and the unending supply of human stupidity.

The software assembly line fallacy

Many software developers now smile nervously when their employers say that they are “doing agile” (doing agile what?, the pedant in me asks—an adjective modifies a noun or pronoun). You just know that the old-school project manager has gone and ‘upgraded’ their PRINCE2 certificate over to a ScrumMaster certificate from some shady place that does a weekend-long course on how to do stand-ups and retros. Getting said certificate and buying a whole boxful of post-it notes from Ryman’s is apparently all you need to be an agile project manager. Unlike a number of the world’s great religions, there’s no baptism or genital mutilation required for the agile cult, just accepting the various doctrines and the transfer of money into the hands of a local friendly guru.

Obviously, I must now make the obligatory statement that there’s nothing at all wrong with agile, ever. It is perfectly fine and truthful and excellent and if it doesn’t work for you, you are just doing it wrong and need to go to confession (retrospective) and do penance (hire a consultant to train the team more). Praise be to agile. Amen.

A recurring theme I see in a lot of companies, and online at sites like StackOverflow/Programmers StackExchange, and in personal conversation with other developers is what I’ve come to call the assembly line view of software development. It seems to be popular with people who do agile really badly (i.e. as a cargo cult slavishly following the diktats of trainers and consultants). Many people seem to think that they can deliver software in an assembly line manner while also “being agile” (I try not to ask too many questions about what they mean by that so as to save the eventual embarrassment).

Waterfall is essentially an assembly line approach. You start with requirements, and then you proceed to design, then to construction, testing, release, documentation, integration, then support. Modern assembly line software development mostly isn’t this type of waterfall anymore. Waterfalls assembly line was subdivided up by the familiar steps of the waterfall.

These days, assembly line projects are divided up based on team ability, but the waterfall nature of those projects is broadly the same. You have a project that starts with business needs collection: this is done by salesman who ask the client what they want. He doesn’t really know anything about, say, the web or software development or about apps or the current state of what is possible with technology. But he does have nice shoes and an excellent line in buzzwords. So he’ll bring back a project brief that is extremely vague and keep it hidden away in his email account, inaccessible to developers or designers.

Then the project gets handed over to designers (or “creatives”, if you are in the kind of place that calls them that). Having got the business requirements from the salesman, they then design the product in Adobe Photoshop, a photo manipulation tool that is commonly used to model complex interactions, animations and modern responsive design approaches despite not being very good at doing any of those things. At the end of this, you will have PSD files.

The next step is to hand the project to front-end developers. The front-end developers will then convert the PSD files into HTML and CSS, without any specific knowledge of where the site will be deployed or which back-end technology stack is to be used (and thus what templating languages need to be used), nor any real knowledge of the complexity of any third-party services or APIs that the system needs to hook up to. Even worse, they may not really have any idea who is using the site, or on what devices. This is not their fault: nobody ever explained to them what needs doing. You don’t have to explain to assembly line workers anything about what the other people on the assembly line are doing.

Then the application is handed to back-end developers who try and cobble what the front-end developers have done to match the back-end technology of choice. They then have to set up the integrations with the third-party services, even though said services were not understood either by the front-end developers or the designers. When none of this works, well, there’s not really a plan for that.

Finally, the application is handed off to the operations team who deploy the application. They have had no input at any stage prior to this, so if some front-end build tool or back-end or database technology isn’t compatible with whatever the cloud deployment stack they are using demands, well, that’s going to be fun. And, sure, they may have to break some stuff to make it all deploy. But it’s not like anyone is going to have to change that code, so what’s the problem?

At this point, the familiar end-of-waterfall cycle happens. Design, front-end, back-end and operations all now get increasingly frantic to try and launch to deadline. The only functioning test server is the ops person letting you SSH into what will eventually become the production server. The front-end developer has moved onto another project, so the inevitable failures to map Photoshop designs pixel perfectly to HTML and CSS is remedied by having the original designer hover next to a developer pixel-tweaking it all in Chrome Dev Tools before checking it in with a commit message of “designers can’t make their bloody minds up”. It all gets out the door, but more by luck than judgment or careful planning.

You might assume wrongly that the assembly line metaphor would include time for testing and that there would be a person whose role would include spending a decent amount of time testing the finished product before it was handed off to customers. But that never really works out in reality, because hiring testing/QA people costs money and money is better spent on sales people to get you more projects that you’ll struggle to deliver due to, well, reasons too many to count.

By my tone, I’m sure you can tell that I am not seriously proposing this as a useful pattern for software development, but attempting to sign its death warrant.

The primary problem with this is a lack of communication. Each step in the process is treated as a predictable part of an assembly line when the expertise of each part of the process needs to be used to assist the other people working on the project. The developer with the expertise in how the third-party integration works needs to be talking to the project manager and the designer up front so they are fully aware of the limitations and issues that integrating with the platform will bring. The project manager needs to give some advice to the back-end developers how the thing is meant to operate, so they can have a first cut of a back-end working for the design to hook up to once the front-end developers are done.

Like the assembly line of Henry Ford, the intention of assembly line development is to try and introduce predictability and modularity to the process. Is one developer not available? Oh well. We’ll switch in another one. Sure, he hasn’t worked with the design team, hasn’t really understood the technical complexity of what we’re asking him to do, but he’s a developer. He’ll figure it out. (About a week after it was due to go out.)

Ultimately, assembly line programming, like most methodologies, ignores experience or detailed knowledge and tries to magic creativity out of an assembly line. I’ve seen these projects and they succeed only in the sense that they get the product out the door. They succeed in spite of chaos and lack of planning and people having to pull weekends and late nights. (The contractor getting double their day rate to work the weekend is fine with this; the permanent staff member who missed their kids birthday party is slightly less happy.)

Don’t let this happen to you. If you spot your company trying to make an assembly line type of process, there are ways to fix it. If you are a developer or a manager of a development team, insist on getting the developers likely to build it in on early meetings with management and designers so said developers can guide them into designing things that will actually be possible (both technically and with the team that you have available). If you manage a developer team, make clear what they can and can’t deliver with the skills they have.

Ultimately, if you can’t stop the assembly line nonsense, people will leave. They’ll get fed up with the waterfall-style last minute crunch times, the stress and anxiety, the lack of value placed on their creative and technical skills, the disinterest in fixing these issues on the part of management, and the recurring lost weekends and overnights. They’ll disappear and they’ll take their expertise (which is needed for the next project) away with them. You probably want to avoid that. Or not. I mean, the best thing about assembly lines is that if a worker on an assembly line stops working, you can easily replace them with someone else. Because hiring good technical people is super easy, right?

The assembly line is a terrible model. What we really need are strong teams where all the people with skills are able to demonstrate those skills to each other, to use those skills to guide each other to build great software. Software projects aren’t repeatable like cars or tin cans or hammers are. Assembly lines are for building the same project over and over. Software and websites and so on are not like that.

I just learned about went, a Python library for webmentions that is built on mf2py.

Meta-point: it would be interesting to see if we could find a way to express “library X depends on library Y” using microformats and webmentions, so that when someone launches a new library that is based on a library you have created, you can be notified about it.

Question on signup form for a new web service: “Why do you want to use [product name]?”

My honest answer: “Pissing around, mostly.”

This is the usual answer for most things. Until you try them, you don’t know what you’ll be able to use them for.

On the dangers of a blockchain monoculture is an interesting takedown of Bitcoin’s flaws. It’s a standard refrain in the VC investments around Bitcoin and blockchains that while Bitcoin may not hold much value, blockchains are very interesting.

I see even less value in blockchains than I do in Bitcoin: an enormous, slow, lumbering, public database that you can’t remove private information from. Great. Just what privacy advocates have been fighting for.

I’m tempted to write a simple guide to current software development practices that explains them for developers. Specifically things like Scrum, Agile, TDD, BDD, DDD (and all the other *DD) and so on.

It’d take the form of answering the following questions.

  1. What is it? (e.g. TDD: “writing tests before you write code”)
  2. How do I do it? (e.g. TDD: “use a unit test library like JUnit or PyTest and enforce iron discipline on yourself”)
  3. Does it work? (Usual answer: we don’t know because we never fucking test any of our methodologies, we just adopt them on the basis of religious preference and/or hearsay)
  4. Who is making money off this trend? (Usual answer: consultants)

And, yes, the complete lack of useful answers to question 3 does tend to scupper this idea from the start.

What Does My Site Cost? shows you how much it costs to download your webpage.

And, yes, the extravagant cost of mobile bandwidth in various developing countries is a negative externality of shitty, but fashionable, web programming practices. Your pointlessly bloated JavaScript framework costs money for poor people to download.

Possible new emojis in Unicode 9.0.

With the addition of numerous sport-related emojis (wrestlers, water polo, handball, fencing, modern pentathlon), plus the first, second and third medal symbols, and the existing flag symbols, people will finally be able to report the results of the Olympics purely using emoji.

Probably won’t be ready for the Olympics in Rio de Janeiro in the summer, but maybe good to go for the 2020 Olympics in Tokyo.

This press release from the Royal College of Arts is an amazing and spectacular collection of bad writing.

This is an unprecedented opportunity to extend and deepen our subject landscape allowing a more leading-edge approach grounded in experience and expertise, with new initiatives underpinned by our reputation for innovation and skill. Our challenge is to combine scale with opportunity and agility, looking to lead change from the front. Our motivation is to help build a better world; our new structure is designed to support that ambition.

A perfect illustration of Orwell’s indictment: “phrases​ tacked together like the sections of a prefabricated hen-house”.

Do read the whole thing. There’s so much audaciously obscurantist art-speak that one could almost interpret it as a piece of performance art in its own right.

Lessons from trying to help with Android/iOS transfer

This weekend, I have been helping an Android user I know switch over to iOS.

What a fucking mess. The tech industry really ought to feel collective shame for the horror movie that is trying to switch from one platform to another.

Let’s start with the official route for moving from Android to iOS: Apple’s Move to iOS app.

  1. You can only run the app at the point of initial setup. If you managed to miss the option, enjoy wiping your iPhone and restarting.
  2. The fucking thing crashes repeatedly and there’s no way to resume the transfer. It starts from scratch. This is an infuriating process.
  3. If you try and Google anything related to this app, you get anything-but-helpful answers from the fandroid community. Instead of a detailed description of the technical issues with the app and how to get around them, you get helpful nuggets like:

Why would i move to ios, most android phones today out perform iphones. Quad core vs dual core, 4k vs 720p, freedom vs being told how your phone should look. Choice is yours.

FREEDOM! CHOICE! OPENNESS! (Except the freedom, choice and openness to move data between platforms easily etc.)

And things like this:

Moving to iOS was backwards logic. It’s like putting a straight jacket on your phone. Why jailbreak when you can use an open source OS like Android?

How about because the user might fucking prefer it? How about because you don’t want your phone pwned by the hilarious cascade of systemic security failure that is the Android ecosystem?

And one more:

Like I’d ever consider switching to using an iPhone… and as for that slow and buggy Watch of theirs, words can’t describe what a useless pile of overpriced crap it is.

Quite what the Apple Watch has to do with an app to help you move data between Android and iOS, I’m not sure. All discussions of going between mobile OSes ends up in pitiful religious argumentation and if you are just trying to get shit done, it does nothing but incite a lot of eye-rolling.

Fanboy awfulness aside, we re-ran the Move to iOS app once more but deselected the transfer of photos. I kind of figured that if we could get it to transfer things like old text messages and contacts and so on, then we could maybe move the 14Gb of pictures onto the iPhone by hand. They’ll all be stored as JPEGs on either the SD card or the internal storage on the phone.

After waiting some more, Move to iOS chunters everything it can except the pictures over. It kinda worked in a slow, clunky, crashy kind of way. Hardly as “seamless and simple as possible”, to quote Lifehacker.

Next job, let’s get the photos out. I had an Android phone many years ago. This should be easy. Plug it in, it mounts up as a USB mass storage device, drag and drop. That’s, y’know, what everyone tells me that Android did that iOS didn’t—no futzing around with iTunes, it’s just a USB mass storage device.

You’d think that. In the meantime, Samsung have deprecated USB mass storage in their Android devices and replaced it with the Media Transfer Protocol (MTP). Despite coming from the same place as Windows Media DRM nonsense, MTP sounds like a nice idea: unlike USB mass storage, it is a simpler protocol that implements much simpler operations than a traditional I/O interface, perfect for shunting the odd MP3 and JPEG around.

In practice though, actually using MTP from my Mac is soul-gnawingly awful. Obviously, there’s no filesystem that is mounted. For a while, I thought there must be some issue with the cable or that the Android phone had some formatting issue. Silent failures aren’t fun. Eventually, I find out how to switch the device into MTP mode and it then tells me that I ought to download the Android File Transfer app for Mac (you probably shouldn’t be surprised to learn that it isn’t open source). The UI is horrible. Cmd+A doesn’t work. It just randomly disconnects from the phone. And, worse, because MTP has no parallelism, transferring substantial quantities of data is godawfully slow. Especially if it is having to walk a directory tree and retrieve file lists. I’ve now dragged mostly all of the pictures off the phone and stored them on a computer, but systematically getting things other than the pictures off the filesystem has been unsuccessful.

There’s some alternatives, but all require effort. There’s libmtp, which I’m sure is excellent. There’s a rather nice looking Go program called go-mtpfs which will mount MTP devices as FUSE filesystems. That looks like it might be an improvement on the horrible OS X frontend at least.

What’s next then? How about WhatsApp messages? WhatsApp stores messages differently depending on whether you are using Android or iOS and all the methods for transferring between them look pretty damn rickety. They mostly involve shareware Windows apps from websites that give me that disconcerting feeling of talking to a sleazy second hand car dealer. I don’t quite trust any of the purported solutions to this.

I started looking into it myself. The files in WhatsApp for Android are stored in an encrypted SQLite store using a crypto system called crypt7 and/or crypt8. I have managed to extract this file from Android and I just need to decrypt it. There are some tools for this online. Pushing that data to iOS looks pretty simple. On the Mac, iCloud data is stored in ~/Library/Mobile Documents, and within there is a folder for WhatsApp containing a backup of WhatsApp conversations (in SQLite format) as well as media sent via WhatsApp. If one can retrieve the data from Android, it shouldn’t be too hard to push it into iCloud for the iPhone app to use.

All of this is way too fucking hard for non-programmers. The stuff on people’s phones is documentation of their life—their holidays, their families, their relationships, their co-workers. It’s not a part of some stupid platform war bullshit or an ideological debate about free software or DRM or whatever. It’s their stuff and they should be able to transfer it onto any device they choose to use. At the same time technologists have debated the ideal solutions for data portability, and churned out a thousand bullshit specs and documents that don’t do shit, ordinary people switching between iOS and Android (in either direction) have a hellish time doing ordinary stuff like WhatsApping with their friends and family. That’s a ludicrously poor show from our industry.

This is a shitshow. Be ashamed, fellow technologists. We have made a world that disempowers users and locks them in. And when they go online to find out why, all we have to show is a bunch of religious apologists telling them that this is okay because they are locked into a platform for their own freedom. Absolutely fucking terrible.

Today I learned: meta description tags are actually used by Slack for rendering link previews.

It may be invisible (partially visible, really) but when that data seeps through, it can often be wrong.