Discussing software, the web, politics, sexuality and the unending supply of human stupidity.

Oil & Gas International post-mortem: the hard security lessons are management-related, not technical

On Monday, a man named Dev George posted the following on Mozilla’s Bugzilla bug reporting database:

Your notice of insecure password and/or log-in automatically appearing on the log-in for my website, Oil and Gas International is not wanted and was put there without our permission. Please remove it immediately. We have our own security system and it has never been breached in more than 15 years. Your notice is causing concern by our subscribers and is detrimental to our business.

(Bugzilla have hidden the bug report, but you can view a version with broken CSS on

What happens next is both completely predictable and an excellent illustration of how to not do security.

Browsers are in the process of making it so insecure logins are flagged up as a potential security issue. This recently went live in Firefox and is slated to go live in Chrome soon (it is available behind a feature flag).

The website referred to in the comment—Oil and Gas International (or O&GI)—not only accepts logins via HTTP, but also accepts credit card payments sent unencrypted. At least, it did. It is now offline.

Not only was the site not using Transport Layer Security for logins or card transactions, the site itself was rife with an array of fundamental security failures.

It was running on ASP.NET, version 2.0. .NET 2.0 was released in 2006. The version the OGI site was running was likely teeming with old, unpatched vulnerabilities. Microsoft stopped supporting .NET 2.0 a long time ago, and they no longer support versions of Windows capable of running .NET 2.0. I’m not an expert in .NET, but from what I understand, it is pretty trivial to upgrade the underlying .NET Framework (just as one can run old Java code on a newer JVM).

Not that framework vulnerabilities are the main problem. The site was vulnerable to a SQL injection attack in the login form. Before the site went down, people found that submitting poorly formatted username/passwords led to the .NET server responding with a stack trace with highlighted source code.2

Here’s another screenshot showing the application vomiting back stack traces when the login code fails to handle a SqlException.

So far we’ve got no HTTPS, an extremely outdated back-end framework, SQL injection, and a server configured to reveal stack traces to the user.

According to this Reddit thread, someone took a peek at their user table and found all the passwords were stored in cleartext. And some charitable soul took it upon themselves to drop either the user table or the whole database (I can’t tell which). The Reddit thread also says that a user on there called the company and tried to explain their security failures, only to be rebuffed.

Nmap results posted in that thread and on Twitter showed that the server had open ports for Microsoft SQL Server, POP3, IMAP and… HTTPS. Yes, they had port 443 open for HTTPS traffic but didn’t use it.

At one level, the technical lessons of this are obvious. In no particular order:

  1. Keep your server patched. Keep an eye on the frameworks and libraries you use and patch/upgrade as appropriate. Automate this as much as possible.
  2. Don’t expose error messages/stack traces in production. Make the attacker do some work.
  3. Sanitise user input to avoid SQL injection.
  4. Don’t store user passwords in cleartext. At this point, Bcrypt or PBKDF2.
  5. Lock down your server ports.
  6. Use HTTPS. With Let’s Encrypt, certifiates are free. If you run a business, can you think of any examples where the traffic between your user’s browser and your server should be interceptible? No? Then use HTTPS. (Every US federal website is going HTTPS-only. Seriously. You should too.)
  7. PCI-DSS is a thing. Even if you think login-over-insecure-HTTP is fine (it isn’t), sending card details over insecure-HTTP is dumbness.

I do have some sympathy for the site owner. According to this biographical page from (an archived version of) the site, he’s a writer with genuine expertise and understanding of his field, and has worked with some of the biggest companies in the business. He knows about oil rigs, not infosec.

He has every right to sell access to his expertise and knowledge through running a premium website. But doing so comes with risk. Information security risks are genuine risks that businesses have to cope with.

And they are doing a poor job of it. Every business that builds anything online needs to have a strategy for how to handle information security risks. None of the issues I raised above are complex. Every professional developer worth their salt should know the importance of getting them right. And most do.

As I’ve noted in the past, business organisations often talk up the complex, ever-changing, difficult nature of these threats. Listen to the non-technical discourse on information security and you’ll end up so confused as to basically do nothing. The threat is insidious, ever-mutating, confusing and impossible to fight that it is quite rational to not do anything. This ignores the truth: the threat model that for small and medium sized companies are less like Wikileaks and/or 24 and more like… well, this kind of small potatoes nonsense.

The threat to most small-to-medium sized companies is that they don’t seem to get this. Poorly built websites, often outsourced to the cheapest supplier, filled with utterly boring vulnerabilities. SQL injections and insecure form submissions aren’t the constant, dynamic, ever-changing threat that pundits in the press pontificate about. It’s not cyberwar, just run-of-the-mill bad coding, lack of testing, and poor systems administration.

In so many companies, people haven’t fully understood how bad software is. They haven’t grasped how widespread poor business information security truly is. They hide behind certifications and assurances.

I’ve seen organisations proudly show off their ISO 27001 certification, then taken a look at their site and found an old, unpatched POP3 server running without TLS and accepting passwords in cleartext. I’ve seen organisations offering to do ISO 27001 certification, as well as government-approved cybersecurity work, without HTTPS on their website.

I have seen so many people and organisations in business who think software is something that gets “done”. That you just finish writing the software and nothing else needs to be done. Everyone who works in software knows this isn’t true, but this incorrect attitude is worryingly frequent in business.

I have also seen organisations where non-technical staff are not instructed on how to deal with reports of security vulnerabilities. In such organisations, it’s only through luck that a vulnerability is passed on to developers/sysadmins who can actually do something about it. When a security vulnerability is revealed, the staff run around like headless chickens worrying about what to do. People at all levels, in all areas (sales, customer service, support, management etc.) will need to learn what to do when someone tells them their website or app has a vulnerability—what the appropriate reporting procedures are etc. They also need to have confidence that the business will take it seriously, and know how to get it fixed quickly and properly.

Those at the management level need to take responsibility for understanding some basic information security. Without basic knowledge of information security, they won’t be able to evaluate risks rationally. If people were wandering around so worried about laser-shooting stalker drones that they didn’t bother locking their front doors, that’d be a sign of a colossal misunderstanding of risk. People getting worried about three-letter agencies and cyberwar while not fixing (or even having basic awareness) SQL injections, XSS vulnerabilities and other well-understood threats is the situation we see now.

The same misplaced fears affect the wider public. People read stories in the press about, oh, the CIA, NSA, GCHQ, Snowden, Wikileaks, mass surveillance, and cyberwar. That’s all worth thinking about. But most people, most businesses, are unlikely to be the target of surveillance by a spooky government agency, nor are they likely to be pawns in a global cyberwar. Much more likely is some incompetent website designed by inexperienced, self-taught developers with some major Dunning–Kruger issues, unwilling to learn anything new, is going to accidentally leak private data due to, oh, the sort of important but boring sins committed by sites like O&GI.

The real problem with information security is management-related: those in charge, unable to grasp elementary security issues or evaluate their relevance, properly handle security reports (sometimes from scary pseudonymous people on the Internet), or listen to experts. The existing certification processes, and the attitude that security is about “reassuring” customers (rather than actually being secure) is part of the problem. O&GI is a case study in how bad technology choices and poor management combine to produce a predictable (if undeserved) outcome. Here’s the scary bit: the same people who design websites with this level of insecurity are also designing your internet-connected coffee machine or teddy bear or, oh, car. We can’t carry on without some pretty fundamental fixes to how information security is handled in society, in business and in government.

  1. I did consider the ethics of whether to post this. I think the cat is out of the bag now and continuing to pretend otherwise is pointless.

  2. bool blnLogin. Yay for Hungarian notation. Systems Hungarian, really.

LWS talk: Break down the barriers - Self taught developers today

The following is contemporaneous notes from a talk at London Web Standards by Matt Bee (@mattbee) from futureLearn. There may be errors or material I missed.

Auto-didacticism: the act of self learning or self-education without the guidance of masters (such as teachers and professors) or institutions. (Audience member: “Just Stack Overflow!”)

Wholly self-taught front end developer. Two things happened when I first started developer: “The Facebook” and Firefox.

If you are self-taught, you are in good company: Leonard Da Vinci, Charles Darwin, Quentin Tarantino, David Bowie, Jimi Hendrix.

Our industry is amazing - the support/encouragement available is unbelievable. We are excluding new people though.

“I followed an unconventional route to being a web developer”. But that’s a lot of people.

Source for first website - table based layout, a lot of view source, a lot of Notepad, a lot of IE 6. Used to work mostly in HTML and CSS. With the help from books like “HTML for the World Wide Web - Visual Quickstart Guide”, learned a lot as a tinkerer.

Two years in: good with HTML (table layouts) and moderate CSS (fairly new), basic PHP, could use FTP and do basic web config. Could get a site up and running from scratch. This was enough to get my first developer job. This was without any computer science background.

Now: front end developer with 10 years experience, not an engineer, or a code ninja. I don’t know Angular, React, WebPack. I don’t even know JavaScript inside out. I am valuable to my team. Need more: empathy, honesty, being able to see stuff from a user’s perspective.

I worry for the future. Not in a political sense either. The openness is being removed. A lot harder for someone who is intrigued to learn. 69.1% of developers are self-taught according to Stack Overflow.

3 key areas that have made it harder for new people:

  1. Harnessing the power of devices.
  2. Exposure to the inner workings
  3. Mass adoption of complex tools and technologies

My first computer: ZX Spectrum 48K. Had to load programs off a tape. Instruction manual contained programs you could write and run, could produce beautiful graphics (if you had a colour TV). You could buy magazines to copy commands.

Compare the ZX Spectrum keyboard (with “str$”, “rnd”, “peek”, “poke”) keys with the PS2 controller. UX won out - but blackbox devices don’t enable developers.

Terminal and command line are hidden away. Accessible hardware is around, but you need to know about it and have surplus cash for it. Simulated environments aren’t always great experiences.

It used to be easy. View source was enough to get inspiration and troubleshoot. A lot of code was not dependent on anything else. Generally you saw exactly what people write. Didn’t depend on a node package that the author of the code you are looking at didn’t understand. The was true even for big production sites.

JavaScript is an example of how it used to be. In my early development days, a website like Dynamic Drive.1 You could copy, paste and tinker. The modern day equivalent is include some file then type $('.datepickerclass').datepicker();

Today: a new developer can create with no exposure to real code. View source is now potentially useless. Everything output is transformed from what is written. HTML and CSS sometimes feel like second class citizens.

Tweet from @paulcuth: “Just starting to use Web Components & Shadow DOM. Great fun, but going to raise the barrier to entry massively. View source is practically useless.”

Complex tools and mass adoption: “I first thought of learning AngularJS through their official documentation, but this turned out to be a terrible idea.” (source)

Ashley Nolan’s front end tooling survey: “the adoption rate of front-end tools shows no signs of letting up, with tools such as Webpack and JavaScript transpilers becoming ever more essential in our workflows” (source)

Try to install a Node.js library. Thousands of C++ compilation errors. Answer is to download a few gigabytes of Xcode that you probably won’t ever use. Why?2

We are exposing thousands of developers to this every day, and potentially putting them off a career in tech.

I am not saying do not use frameworks. They are valuable and save time and effort. I am saying provide a pathway to learn them.

In summary: I don’t think I’d cut it today. I’m not a great developer, I have huge gaps in my knowledge. I do have other attributes that make me valuable.

“The rapidness of getting the information doesn’t really correlate with the pace of the actual learning. This frustrates us” - Gavin Strange

What can we do? Consider the future developers. Run a code club. Run a Homebrew Website Club. Help teach people. Promote diversity on teams.

People learn at different rates and in different ways. Empathy is important. Provide pathways for great people to become great developers.

(All pictures from Wikimedia Commons!)

  1. Oh my. It still exists.

  2. “Everyone knows that debugging is twice as hard as writing a program in the first place. So if you are as clever as you can be when you write it, how will you ever debug it?” (Brian Kernighan)

LWS talk: Designing for the Web - Healthy practices for developers and designers

The following is contemporaneous notes from talk at London Web Standards by Matteo Pescarin (@ilPeach), development lead at Sainsbury’s (for now). There may be errors or material I missed.

“I’ve seen things you people wouldn’t believe”. I’ve seen developers and designers who didn’t want to touch a single line of HTML and CSS. I’ve seen fantastic designs that would work on paper. I’ve seen code that shouldn’t work in browsers.

There’s a lot to take in: learning the ever increasing complexity of front-end technologies.

At the beginning, we were very happy. Using tables to display data. Paginating millions of records. Then UX.

Then we started looking into content-first approaches. Personas, user journeys, interactions and behaviours, usability, cross-device compatibility, and many other aspects. A lot to take in.

Let’s say you don’t know web design. What is web design? Everything? Multi-faceted set of practices and doctrines that surround websites. Maybe it is more similar to product design. Quite difficult to grasp everything.

Semantic markup. Do you know?

  • What’s the difference between article and section?
  • How do you vertically align with CSS?

If developers have too much to take in, what should designers do? Learn to code?

The developers are masters in their field. Developers know the limits of the technologies. Designers know colour theory, typography, grid systems, use of whitespace etc. “Don’t expect to find the unicorn” - there are developers who learn design, and designers who learn to become developers, but they are rare.

Don’t expect tools to solve your problems: tools exist to help developers do design better. Things like SketchUp (and earlier Dreamweaver) enable designers to turn ideas into web pages. Developers have the same: CSS grid systems taught us indirectly how design work. Methodologies like Atomic Design too. But don’t let anybody decide which tools you should be using. Allow yourself to be flexible enough to experiment, try new things.

A better world is possible. We cannot shift responsibility.

1-10-100 rule: if you have to fix a problem during design time, cost is 1. Fix a problem during development time, cost is 10. Fix after delivery, cost is 100. Data-based analysis showed the 100 is more like 1m.

Iterate and validate: as developers and designers, we are biased. Involve your users. Design in the browser. Rapid prototyping. Use Design Sprints. Use style guides and pattern libraries - save thousands of hours in the long run.

Designers must sit with developers, to promote knowledge sharing. Work bottom-up, understand each other, elevate developers to work on front-end design.

LWS talk: Building a dementia friendly website

The following is contemporaneous notes from a talk given at London Web Standards by Imogen Levy (@teppie). There may be errors or material I missed.

Dementia: memory loss, mood changes, changes in reasoning. Caused by a number of diseases - most common is Alzheimer’s. Over 850,000 people in the United Kingdom with dementia. By 2021, over 1m people will live with dementia. Getting worse with ageing population.

Designing websites for people with dementia is an unexplored space. Redesigning the Alzheimer’s Society website, it was key that it worked for people with dementia, so they were included them at all stages in the design process.

Challenge 1: how to design for dementia

First step was to develop a series of personas. Personas help guide decisions in design. What are their needs? Used focus groups, interviews, help line call logs, surveys and quantitative data from existing website. Focus groups included both people with dementia and carers. Focus groups run through local branches of Alzheimer’s Society as well as local charitable groups included singing groups, “Alzheimer’s Cafés” etc.

Seven personas:

  1. Person with dementia
  2. Carer
  3. Volunteer
  4. Researcher
  5. Mature supporter
  6. ??
  7. Authority?

Research findings:

  1. Goal-oriented content emerged as key need. Content needed to be structured around stages of dementia.
  2. Increasing importance of mobile and tablet usage. Smartphones are equally accessible in the home to computers. Lots of users relying on smartphones and tablets alone.
  3. Content must be visual and interactive: more video and audio content.

Challenge 2

Challenge: “website is informative scary”. Contained 10,000 pages of content, but was added to without review. Overgrown, unclear structure. “Time to do some weeding”.

Google rankings were excellent, but navigating via the website wasn’t good. Old site started with four items on the left-hand navigation panel, but this grew. Had to start again, but didn’t want to lose out on Google rankings for key content.

New IA/site structure to make it easy for people to find stuff easily and quickly. Worked with consultants.

Digital team owned website, but dementia content was produced mostly by team responsible for print content including fact sheets and material handed out in doctors surgery. Printed material must meet NHS Information Standard.

Had to restructure content but not break compliance with NHS information standard.

The goal of the website is to help users find information and complete tasks.

Structure tested using online tool called TreeJack.

Set of tasks given to user, then user is asked to navigate through website. No design, just structure. Gives you a “spider graph”. The cleaner the graph, the easier the navigation.

Example task: find out more information about vascular dementia. 90% found information easily. Each click stage broken down.

Unsuccessful task: “how can you best look after yourself while caring for a relative?” Content was contained within the “About dementia” section, but most users went to the “Get Support” section, though that did not contain the relevant information. Result was moving the information.


  1. Don’t let the organisation dictate the website structure - this required tricky conversations internally
  2. Clear signposted entry points on website - “
  3. Needs and abilities are unique

Challenge 3: brand strategy

Wanted a new brand strategy as part of a wider goal to change societal perceptions of dementia - “I have dementia, and I’m OK.”

Use of logo across multiple devices. Square logos on a mobile phone was hard.

Colour palette: web use was limited. Lots of testing required. Two prototypes tested - one wireframe - to test out IA hierarchy. Testers liked the design and layout, gave positive impression

Challenge 4: in trouble with the law

ICO enforcement notice against AS. Data breaches including website hack. Great deal of scrutiny followed. Complete restructure necessitated a new environment. Involvement of lawyers, penetration testers and security companies. A great deal of security scrutiny was performed.

CMS limitations: 10,000 to 2,000 pages meant a lot of redirects. Significant number of spreadsheets sent for 301 redirects.

New site enabled “friendly URLs”. Not perfect, but better.


  1. “Love your testing team!”
  2. More time for managing redirects
  3. Use a CMS that supports SEO friendly URLs.

Challenge 5: The Future

AS wants to reach every person who has a dementia diagnosis, and “build stuff people want”. Team wanted to show how to be user focussed.

New stuff:

  • Implementing a new CMS
  • Extensive research into people at point of diagnosis
  • Partner project with NHS Digital - “we are the authority” on dementia content, so want to align with what NHS are doing
  • Continued website improvements

Some notes on Vim and Visual Studio Code

I’ve recently been playing with text editors a bit. I’ve been trying out Microsoft’s excellent Visual Studio Code and I have found it to be perhaps the best editor for both a language I love (Python) and a language I have to occasionally write (JavaScript, or one of its many spinoffs). I like it so much that I ported over a TextMate bundle for handling Mako templates.1

Visual Studio Code really is great, but my one true love is and probably always will be Vim. I recently upgraded to Vim 8.0 which came out in September. The async I/O and channels support should make it so things like linters can be adapted to not block the editing thread on save. This is good.

Today, I finally cleared up my .vimrc, refactored it out into some separate files, and took a position on the Vundle vs Pathogen debate. I’ve used Pathogen in the past, but Vundle seems preferable. Git submodules are an unending source of pain and confusion. Vundle lets me just specify a list of plugins and have that list stored in version control, much like a Gemfile or a requirements.txt or bundle.json or whatever Node is doing this week to download left-pad.

These are the packages I actually use lots in Vim:

  • editorconfig-vim: this is grade A amazing. The idea of EditorConfig is you create a file called .editorconfig and you put it in the root of your project. It specifies all the editor settings: character set, indent style, indent spacing, end-of-line coding, final newline etc.
  • vim-gitgutter: once set up right, it shows you what you have changed in the editor’s gutter (that is, margin) of your files
  • ctrlp: a pretty good fuzzy file finder. The only thing I wish it did was integrate with EditorConfig so that “don’t peek inside that stupidly large node_modules folder that the front-end guys use” could be specified in a shared config file.
  • vim-endwise: because if you find yourself writing Ruby, you may as well set up ways to spend less time actually writing Ruby.
  • vim-commentary: comment and uncomment things.

I have a bunch of other stuff installed in Vim including fugitive (Git integration), ack.vim (which gives you integration with the ack/ag search tools, which are basically faster versions of grep) and vim-rails (no prizes for guessing what that does). I don’t tend to use them. Two reasons. One is that I will often just use the command line. I don’t really feel like I need to use Vim as a command line replacement. 2 For something like Git, I either use the command line, or I use SourceTree. I don’t really feel like I need it baked into my editor.

The second reason I don’t use plugins I have downloaded: I don’t use them just because I don’t use them. They haven’t stuck in my head. They are quite likely to be useful but I haven’t burned the commands into my mind like I have with Vim itself and with the other plugins I use. One of the things that Visual Studio Code, Sublime Text 23 and TextMate before it did was make commands discoverable quite a bit easier by having a commands “palette” you could use. I should probably have a personal Vim cheatsheet. I already know the basic functionality of Vim well enough but each plugin brings some potentially powerful new stuff that you don’t really bother using until you actually learn how to use it.

There’s a few things I want to try soon. One is setting up ale, the asynchronous linting engine for Vim, which will asynchronously run your code through a whole variety of linters for different languages. The most useful for me being flake8 and whatever the least awful JavaScript one is when I next write JavaScript.

Oh, and while we are talking about linters, let’s talk about Proselint. Proselint is an actually useful linter for English prose. It isn’t a grammar checker. There are too many of those. I recently saw a grammar-checker-as-a-service called Grammarly being advertised at me during a YouTube video. The ad featured a rather clueless social media strategist needing a Chrome plugin to correctly spell basic words. The premium edition of Grammarly costs $139.95 a year. Jesus. Proselint is designed for people who are generally able to write in English but want the sort of thing one would consult a style guide over checked. I have a Proselint plugin setup in Visual Studio Code. Once I set up the ale plugin, I hope I will get the same in Vim.

I also tried vim-devicons and Powerline/Airline, but having to install custom fonts just wasn’t something I really felt like bothering with long term, so I stopped. I’m going to give Gundo a try soon.

  1. Mako templates aren’t such a good idea, but Pyramid seems to use them. I tend to normally use either Jinja or Django templates, or ERb in Rails land (because HAML annoys me, and “logic-less templates” are a lie).

  2. Try Emacs for that. It has a surprisingly pretty website for something hosted on!

  3. I’m told. I have never really used any version of Sublime Text. After TextMate was left without updates for such a long time, I lost interest in ever using an editor that wasn’t both free-as-in-beer and free-as-in-Stallman.

Jean Louis-Gassée has nailed the exact thing which the iPhone got right: wrestling control of the device away from the carriers.

This remains a problem in Android land. It isn’t a choice between empowering the device manufacturer or the user. The user doesn’t get the power–the network provider gets the power. And if the network provider doesn’t bother giving updates, the user gets all the vulnerabilities…

Bad news: Parliament voted down measures to make sex and relationship education required in British schools. Instead of comprehensive, rounded and LGBT-inclusive, British schools can continue to teach a minimal battery of reproductive biology combined with a tissue of religiously-inspired propaganda and call it education in order to satisfy the concerns of “faith communities”. In Britain not offending religious people is apparently a more important public policy outcome than teaching students that gay people are human, how to negotiate sexual consent and how to not get sexually transmitted diseases.

Fault tolerance on the Web and the dark side of Postel's Law

I’ve been reading @adactio’s new book. Pretty much all I have read is great, and I highly recommend reading it.

But in the grand spirit of pedantic blogging, I take issue with this:

“Imperative languages tend to be harder to learn than declarative languages”

That tend to is doing a lot of work.

I think Jeremy is conflating declarative with fault tolerant here. HTML and CSS are declarative and fault tolerant. SQL is declarative and decidedly fault intolerant (and quite hard to master to boot). PHP’s type system is a lot more permissive and tolerant (“weak”, one might say) than a language like Python. The former is great for newbies and non-programmers, because adding 1 + "1" (that is, adding a string and an integer together) will give you 2, or at least something that when printed to screen looks vaguely like a two, though under the covers it may be Cthulhu.1 And the behaviour of something like Python is great for jaded old gits like me who don’t want the system to magically convert strings into integers but to blow up as quickly as possible so that it can be fixed before all this nastiness gets stitched together with the rest of the code and causes some real major bugs. The same principle applies on a grander scale with stricter type systems like Java or the broad ML family (including things like Scala, F#, Haskell etc.).

“A misspelt tag in HTML or a missing curly brace in CSS can also cause headaches, but imperative programs must be well‐formed or they won’t run at all.”

Again, depends on the language. PHP is pretty permissive. If you make a very minor mistake, you can often just carry on and keep going. If you are feeling particularly careless about your errors, you prefix your expression with an @ sign and then security people get to laugh at you. I hesitate to say this was “designed” but it was at the very least chosen as a behaviour in PHP.

This may all seem rather pedantic, but it is actually quite important. HTML and CSS are declarative and permissively fault-tolerant. That’s a key strength. In an environment like the web, it creates a certain robustness. If your JavaScript fails to load entirely, you can get some very strange behaviour if, say, a function that is expected to be there isn’t. (I use a site frequently that makes multiple Ajax calls but if one fails, say due to a bad connection, the site is unusable and must be reloaded from scratch. It is also contained in an iOS app, which must be manually restarted.) But if some of your CSS doesn’t load, that’s not the end of the world: you still get something you can read. If some of your HTML has weird new stuff in it, as Jeremy points out elsewhere in the book, that’s still backwards compatible–the browser simply ignores that element and renders its content normally.

This error handling model, this permissiveness of web technologies, isn’t a side-effect of being declarative. It’s actually a property of them being designed for the web, of human choice by the creators. There is a cost to this. It has been incredibly hard for us to secure the web. Permissive error handling can and has enabled a whole class of security vulnerabilities.

If Postel’s Law gives you the ability to use border-radius in your CSS or aside in your HTML and some terrible old version of Internet Explorer happily ignoring it without vomiting XML warnings all across your screen, then Postel’s Law also comes with the cost of having to worry about downgrade attacks. We collectively left SSLv2 running long after it should have been dead and we got DROWN. We did the same with SSLv3 and we got POODLE. These are examples of ungraceful degradation and the sad cost is your server being vulnerable to being pwned.2

With the last few attacks on SSL/TLS, it wasn’t just nasty old versions of Internet Explorer on Windows XP getting shut out of the web, it was non-browser apps that talked to HTTPS-based APIs. The Logjam attack meant that a lot of people upgraded their servers to not serve DH keypairs that are below 1024-bit. For most current day browsers, this was not an issue. Apple, Mozilla, Google, Microsoft and others released patches for their current browsers. Java 6 didn’t get a patch for a very long time. If you had a Java desktop app that consumed an HTTPS-based RESTful API which had patched Logjam, that broke with no graceful degradation, and the general solution was to upgrade to a new version of Java. On OS X, Java used to be distributed by Apple, albeit as something of a reluctant afterthought. Upgrading every desktop Java user on OS X was a bit of a faff. (My particular Java Logjam issue was with… JOSM, the Java OpenStreetMap editor.)

Postel’s Law giveth and it taketh away. One could give this kind of rather pedantic counterexample to Jeremy’s account of Postel’s Law and then conclude hastily “given how badly the web is at solving these security problems, the grass on the native apps side of the fence might just be a little bit greener and yummier”. Oh dear. Just you wait. When every app on a platform is basically reimplementing a site-specific client, you might actually get more fragility. Consider our recent vulnerabilities with SSL/TLS. After something like Logjam, the bat signal went out: fix your servers, fix your clients. On the server side, generally it meant changing a config file for Apache or Nginx in a fairly trivial way and then restarting the web server process. On the client side, it meant downloading a patch for Chrome or Firefox or Safari or whatever. That may have just been rolled into the general system update (Safari) or rolled into the release cycle (Chrome) without the end user even being aware of it. The long tail3 of slightly oddball stuff like Java desktop apps, which tends to affect enterprise users, assorted weirdos like me, and niche use cases, took a bit longer to fix.

If every (client-server) app4 that could be in a browser, in the case of a security fail, fixing all those apps would be as simple as fixing the browser (and the server, but that’s a separate issue). If everything were a native app, you have to hope they are all using the system-level implementations of things like HTTP, TLS and JSON parsing, otherwise you have a hell of a job keeping them secure after vulnerabilities. We already see things going on in native-app-land (Napland?) that would cause a browser to throw a big stonking error: user data being sent in cleartext rather than over TLS being more common than I care to think about. But the native app won’t scream and shout and say “this is a bad, very wrong, no good idea, stop it unless you really really really want to”, because the UI was developed by the person who created the security issue to start with.

The downside to Postel’s Law5 is sometimes the graceful degradation is pretty damn graceless. Sadly, the least graceful degradation is often security-related. The web might still be better at mitigating those than all but the most attentive native app developers, or not. Browser manufacturers may be better at enforcing security policies retroactively than app stores, or they might not. We shall have to wait and see.

The Web is a hot mess.

But we still love it.

  1. Sausage typing: if it looks like a sausage, and tastes like a sausage, try not to think about what it really is.

  2. At risk of giving technical backing to rising reactionary movements, perhaps the modern day variation of Postel’s Law might be: “Be conservative in what you send, and be liberal in what you accept, to the degree that you can avoid catastrophic security failures.”

  3. Wow, it’s been a long time since we talked about them…

  4. Let’s leave aside the semantics of what is or isn’t an app. Incidentally, I was in a pub the other day and saw a few minutes of a football game. There was a big advert during the game from a major UK retail brand that said “DOWNLOAD THE [brand name] APP”. Why? I don’t know. We are back at the ghastly “VISIT OUR WEBSITE, WE WON’T TELL YOU WHY” stage with native mobile apps. I’m waiting for a soft drink brand to exhort me to download their app for no reason on the side of a bottle.

  5. I claim no originality in this observation. See here and here. Interestingly, if you take a look at RFC 760, where Postel’s Law is originally described, it has a rather less soundbitey remark just before it:

    The implementation of a protocol must be robust. Each implementation must expect to interoperate with others created by different individuals. While the goal of this specification is to be explicit about the protocol there is the possibility of differing interpretations. In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior. That is, it should be careful to send well-formed datagrams, but should accept any datagram that it can interpret (e.g., not object to technical errors where the meaning is still clear).

    The first two sentences are key…

There’s finally a reasonable looking podcasting client for Linux: Vocal. It has the things that iTunes (etc.) has had since the early days which everyone told me were unimportant like remembering where you left off.

Of course, now we have the Web Audio API, Electron and things like remoteStorage, we may even get cross-platform free software podcasting apps that sync what you have and haven’t listened to across devices. And then we’ll be back to where we were back in about 2008 with iTunes and the iPod…

Firefox 52 adding insecure form warnings

The latest version of Firefox’s Developer Edition (formerly known as Aurora) now ships with more prominent warnings for when you send passwords from an insecure context. For a long time, some sites have attempted to justify having login forms on insecure HTTP pages with the argument that the credentials would be encrypted in transmission as they were being sent to an HTTPS endpoint. The problem with this is that it doesn’t actually prevent password sniffing, it just makes it slightly harder. Rather than sniffing the credentials as they go over the wire, you instead use a man-in-the-middle attack to intercept the HTTP page and insert JavaScript into it that sniffs the password upon entry and then sends it via Ajax to the interceptor.

Firefox’s new implementation uses a fairly simple algorithm described in the new W3C Secure Contexts spec that is attempting to standardise some of the key concepts of browser-side security. Hopefully, users being warned that they are submitting passwords insecurely will start prompting websites to stop doing the login-form-on-HTTP-that-submits-via-HTTPS anti-pattern.

My usual go-to example when illustrating the problem is airline websites, specifically the online checkin or frequent flyer account login page. You give airlines quite substantial amounts of money and personal information. For a long time, most were vulnerable to this kind of attack. Malicious hackers also have been known to steal and sell frequent flyer miles, although not necessarily though man-in-the-middle attacks on the login forms.

British Airways used to have a login form for their Executive Club frequent flyer programme on their homepage—they’ve now fixed this and the whole site seems to be HTTPS.

But the following (mostly randomly selected) airlines still are vulnerable to man-in-the-middle password stealing through JavaScript injection.

And that’s just one sector: airlines. There’s plenty more sites that ordinary people use everyday that have potential vulnerabilities caused by these purportedly-secure-but-really-not login forms. Browsers giving prominent and irritating warnings about it is the first step to getting the companies to pay attention.

When the next big attack happens, there will–as always–be non-technical people from government and business lamenting how difficult all this information security stuff is, and how the vectors of attack are always changing. Let it be on the record that this kind of vulnerability is extremely simple, well-known and relatively easy to exploit. There are interesting and ingenious ways to attack Transport Layer Security, but if you don’t turn it on to start with, one doesn’t need to DROWN POODLEs or really do anything that will make technical people go “ooh, that’s clever”. Firefox warning users about this really old and boring way of breaking user security might mean that people spend less time speculating about the scary difficult emerging threats and fix the basic glaring security errors right in front of them.

Towards an Evernote replacement

Since the recent announcements by Evernote that they really, really will be able to poke around inside your notebooks without issue, and they’ll also apply the same sort of machine learning technology to your data that people have convinced themselves that paying for a product will help them avoid, lots of people have been looking at alternatives to Evernote. I’ve been evaluating a few of them.

Here’s some of the open source alternatives I’ve seen people talk about:

  • Laverna, a client-side only JavaScript app that uses localStorage for your notes. No syncing, sadly.
  • Paperwork, PHP serverside app
  • Standard Notes, which aims to be an encrypted protocol for storing Evernote-style notes

These all handle the plain text (or Markdown or whatever) case reasonably well, but there’s a few things Evernote provides which we should be aware of if we’re trying to find replacements.

  1. text/plain or RTF storage. A lot of people store a lot of simple text notes in Evernote.
  2. OCRed PDF storage. Evernote has an app called Scannable that makes it ludicrously easy to scan a lot of documents and store them in Evernote.
  3. Web Clipper: I don’t use this, but a lot of people use Evernote as a kind of bookmarking service using the Web Clipper plugin that they provide. If they see a news article or recipe or whatever on the web, they clip it out and store it in Evernote and use that almost like a reading list, like what people use Instapaper/Pocket for etc.

The solutions people have been building generally solve problem (1) but do little to address problems (2) and (3).

My own preferred solutions are basically this: for (1), I’m leaning towards just storing and syncing together plain text Markdown files of some description.

Solving (2) is a harder problem. My current plan is to try and create a way to store all these in email. Email is a pretty reliable, well-tested and widely implemented Everything Bucket. The process would be relatively simple: scan document, run it through an OCR process, then provide the relevant title and tags which could be stored in the subject line and body of the email. The OCR result would also be stored in the body of the email to make it more easily searchable. Then you just stick it all in an email folder (or Gmail label). You’ve got a security layer (whatever your email provides, and if you are storing lots of important data in there, you should probably ensure it is 2FA). You’ve got sync (IMAP). You’ve got universal capture (email it to yourself). And you have already made either the financial bargain (with, say, Fastmail) or the give-away-all-your-personal-information bargain (Gmail). Backing up IMAP is relatively trivial compared to backing up whatever weird binary blob format people come up with.

Solving (3) is somebody else’s problem because I don’t understand why anyone wants to stick all the websites they’ve ever visited into Evernote.

That said, let’s not promise users replacement for software if we are only replicating the features from that software that we actually use. If anyone has great suggestions for how they are going to sort out problem (2), I’m all ears.

Mako for VS Code

I just launched a Visual Studio Code extension: Mako. It provides syntax highlighting for the Mako template language. There’s already extensions for Jinja, Django and so on.

I can’t take much credit for this: literally all the extension does is takes an existing TextMate language theme. I needed it, so I created it very quickly and released it. It’s MIT licensed per the original TextMate plugin.

I’m really impressed with how great VS Code is. To see how to build your own plugin, check out this example tutorial and this page explaining how the vsce CLI tool works.

TIL: Vagrant doesn’t do any kind of error recovery when a VM (.box) download fails/stalls. This gets boring the third or fourth time it happens.

If you happen to be using a sporadic or error-prone Internet connection (and if you are using a London DSL connection, that definitely counts) you can get around this by manually downloading the box file (using wget, say) and then using vagrant box add boxname ~/Downloads/ (see this StackOverflow answer for details of the manual box adding process). This may also be of use if you are behind particularly fiddly or developer-unfriendly corporate firewalls.

Of course, in an ideal world, we’d have languages that are sensible enough that you don’t need to download a Linux distribution for each project you work on in said language.

You (probably) don't need a chatbot

There has been a great hullabaloo in the last few months about the rise of chatbots, and discussions of “conversational UIs” or, even more radically, the concept of “no UI”—the idea that services might not need a UI at all.

This latter concept is quite interesting: I’ve written in the past about one-shot interactions. For these one-shot interactions, UI is clutter. But chatbots aren’t the answer to that problem: because chatbots are UI, just a different sort of UI. Compare…

Scenario 1: App

  1. Alice hears an amazing song playing in a club.
  2. Alice pulls out her iPhone and unlocks it by placing her finger on the TouchID sensor.
  3. Alice searches on the homescreen for the Shazam app.
  4. Alice opens Shazam, then presses the button to start the process of Shazam identifying the song that is currently playing.
  5. Alice waits.
  6. Alice is told what the song is and offered links to stream it or download it from a variety of streaming and download services that vary depending on the day of the week, the cycle of the moon, and how Shazam’s business development team are feeling this week.

Scenario 2: Chat

Someone at Shazam decides that apps are a bit old-fashioned and decides to build a chatbot. They have read an article on that tells them that chatbots are better, and decide to build one based solely on this advice rather than any actual empirical evidence.

  1. Alice hears an amazing song playing in a club.
  2. Alice pulls out her iPhone and unlocks it by placing her finger on the TouchID sensor.
  3. Alice searches on the homescreen for the Facebook Messenger app.
  4. Alice opens Facebook Messenger, then locates the existing chat session with the Shazam bot.
  5. Alice scrolls back up the chat to work out what the magic phrase she needs to type in to trigger the chatbot into listening to music.
  6. Alice waits.
  7. Alice is told what the song is and offered whatever extra rich data the chat UI is allowed to show.

As you can see, this is a vast improvement, not because it makes the process less involved or elaborate, but because someone on told them that it is new and exciting.

Scenario 3: Idealised One-Shot Interaction

  1. Alice hears an amazing song playing in a club.
  2. Alice taps a button on her smartwatch. Everything else happens in the background. Alice continues partying and enjoying herself rather than being the saddo staring at her phone all night.

For those without a smartwatch, a lockscreen button on the phone could be substituted.

Anyway, this is a slight distraction from the broader point: chatbots are a bit of a silly fashion and a fad and that they seem to be adopted based on fashion rather than based on any actual utility.

But, but, there’s this awesome chatbot I use, and I really like it!

Great. I’m not saying that they have no purpose, but that chatbots are being adopted even though they often are worse at what they do than the alternative. They also come with considerable downsides.

First of all, chatbot UIs are poor at letting a user compare things. When someone browses, say, Amazon or eBay or another e-commerce service, they will often wish to compare products. They’ll open up competing products in different tabs, read reviews, check up on things on third-party sites, ask questions of their friends via messaging apps and social media sites like Facebook. Chatbot UIs remove this complexity and replace it with a linear stream.

Removing complexity sounds good, but when someone is ordering something, researching something or in any way committing to something, navigating through the complexity is a key part of what they are doing.

Imagine this scenario. Apple have 500 different iPhones to choose from. And instead of calling them iPhones, they give them memorable names like UN40FH5303FXZP (Samsung!) or BDP-BX110 (Sony!). Some marketing manager realises the product line is too complex and so suggests that there ought to be a way to help consumers find the product they want. I mean, how is the Average Joe going to know the difference between a BDP-BX110, a BDP-BX210, and a BDP-BX110 Plus Extra? You could build a chatbot. Or, you know, you could reduce the complexity of your product line. The chatbot is just a sticking plaster for a broader business failure (namely, that you have a process whereby you end up creating 17 Blu-Ray players and calling them things like BDP-BX110 rather than calling them something like “iPhone 7” or whatever).

Chatbots aren’t removing complexity as much as recreating it in another form. I called my bank recently because I wanted to enquire about a direct debit that I’d cancelled but that I needed to “uncancel” (rather than setup again). I was presented with an interactive voice response system which asked me to press 1 for payments, 2 for account queries, 3 for something else, and then each of those things had a layer more options underneath them. Of course, I now need to spend five minutes listening to the options waiting for my magic lucky number to come up.

Here’s another problem: the chatbot platforms aren’t necessarily the chat services people use. I’m currently in Brazil, where WhatsApp is everywhere. You see signs at the side of the road for small businesses and they usually have a WhatsApp logo. WhatsApp is the de facto communication system for Brazilians. The pre-pay SIM card I have has unlimited WhatsApp (and Facebook and Twitter) as part of the 9.99 BRL (about USD 3) weekly package. (Net neutrality? Not here.) The country runs on WhatsApp: the courts have blocked WhatsApp three times this year, each time bringing a grinding halt to both business and personal interactions. Hell, during Operação Lava Jato, the ongoing investigations into political corruptions, many of the leaks from judges and politicians have been of WhatsApp messages. Who needs Hillary Clinton’s private email servers when you have WhatsApp?

WhatsApp is not far off being part of the critical national telecoms infrastructure of Brazil at this point. Network effects will continue to place WhatsApp at the top, at least here in Brazil (as well as most of the Spanish-speaking world).

And, yet, WhatsApp does not have a bot platform like Facebook Messenger or Telegram. To get those users to use your chatbot, you need to convince them to set up an account on a chat network that supports your bot. For a lot of users, they’ll be stuck with WhatsApp, the app they use to talk to their friends, and Telegram, the app they use to talk to weird, slightly badly programmed robots. Why bother? Just build a website.

Now, in fairness, WhatsApp are planning to change this situation at some point, but you still have an issue to deal with: what if your users don’t have an account on the messaging service used by the bot platform?

One of the places chatbots are being touted for use is in customer service. “They’ll reduce customer service costs”, say proponents, because instead of customers talking to an expensive human you have to employ (and pay, and give breaks and holidays and parental leave and sick days and all that stuff) to, you just talk to a chatbot which will answer questions.

It won’t though. Voice recognition is still in its infancy, and natural language parsing is still fairly primitive keyword matching. If your query is simple enough that it can be answered by an automated chatbot, it’s simple enough for you to just put the information on your website, which means you can find it with your favourite search engine. If it is more complicated than that, your customer will very quickly get frustrated and need to talk to a human. The chatbot serves only as a sticking plaster for lack of customer service, or business processes that are so complicated that the user needs to talk to customer service rather than simply being able to complete the task themselves.

You know what else will suffer if there were a widespread move to chatbots? Internationalisation. Currently, the process of internationalising and localising an app or website is reasonably understandable. In terms of language, the process isn’t complex: you just replace your strings with calls to gettext or a locale file, and then you have someone translate all the strings. There’s sometimes a bit of back and forth because there’s something that doesn’t really make sense in a language so you have to refactor a bit. There’s a few other fiddly things like address formats (no, I don’t have a fucking ZIP code) and currency, as well as following local laws and social taboos.

In chatbot land, you have the overhead of parsing the natural language that the user presents. It’s hard enough to parse English. Where are the engineering resources (not to mention linguistic expertise) going to come from to make it so that the 390 million Spanish speakers can use your app? Or the Hindi speakers or the Russian speakers. If your chatbot is voice rather than text-triggered, are you going to properly handle the differences between, say, American English and British English? European Portuguese, Brazilian Portuguese and Angolan Portuguese? European Spanish and Latin American Spanish? Français en France versus Québécois? When your chatbot fucks up (and it will), you get to enjoy a social media storm in a language you don’t speak. Have fun with that.

And you can’t use the user’s location to determine how to parse their language. What language should you expect from a Belgian user: French, Dutch or German?

If you tell a user “here’s our website, it’s in English, but we’ve got a rough German translation”, that’s… okay. I use a website that is primarily in German everyday, and the English translation is incomplete. But I can still get the information I need. If, instead, your service promised to understand everything I say, then completely failed to speak my language, that’d be a bit of a fuck you to the user.

In the chatbot future, the engineering resources go into making it work in English, and then we just ignore anyone who speaks anything that isn’t English. World Wide Web? Well, if we’re getting rid of the ‘web’ bit, we may as well get rid of the ‘world’ and ‘wide’ while we’re at it.

Siri and Cortana are still a bit crap at language parsing, even with the Herculean engineering efforts of Apple and Microsoft behind them. An individual developer isn’t going to do much better. Why bother? There’s a web there and it works.

There’s far more to “no UI” or one-shot interactions than chat. But I’m cynical as to whether we’re ever going to reach the point of having “no UI”. We measure our success based on “engagement” (i.e. how much time people spend staring at the stuff we built). But the success criteria for the user isn’t how much time they spend “engaging” with our app, but how much value they get out of it divided by the amount of time they spend doing it. The less time I spend using your goddamn app, the more time I get to spend, oh, I dunno, looking at cat pictures or snuggling with my partner while rewatching Buffy or writing snarky blog posts about chatbots.

But so long as we measure engagement by how many “sticky eyeballs” there are staring at digital stuff, we won’t end up building these light touch “no UIs”, the interaction models of set-it-and-forget-it, “push a button and the device does the rest”. Because a manager won’t be able to stand up and show a PowerPoint of how many of their KPIs they met. Because “not using your app” isn’t a KPI.

Don’t not build a chatbot because of my snarkiness. They may solve a problem that your users have. They probably don’t but they might. But please don’t just build a chatbot because someone on a tech blog or a Medium post told you to. That’s just a damn cargo cult. Build something that delivers value to your users. That may be a chatbot, but most likely, it’s something as simple as making your website/app better.

Let's Encrypt: it just works, so use it

I’ve just switched over to Let’s Encrypt. My paid-for SSL certificate expires today. I don’t object too much to having to pay the €11 or whatever to renew it, but having to remember to do it every year is a huge faff. The process of making a CSR, logging into the SSL supplier website and all that is just boring.

Let’s Encrypt is nice not only because you don’t have to pay the (not very expensive) SSL certificate tax, but because ideally, it automates the renewal process. Instead of being an annual arse-ache, it’s hopefully set-it-and-forget-it.

For your personal or hobby site, if you currently do the annual certificate dance, switch to Let’s Encrypt when your SSL certificate expires. If you don’t currently do SSL, Let’s Encrypt takes one of the pain points out of it. There’ll still be a market for SSL certificates for businesses (especially for wildcard and EV certificates), but Let’s Encrypt lowers the bar to using SSL significantly.