Discussing software, the web, politics, sexuality and the unending supply of human stupidity.

Lessons from trying to help with Android/iOS transfer

This weekend, I have been helping an Android user I know switch over to iOS.

What a fucking mess. The tech industry really ought to feel collective shame for the horror movie that is trying to switch from one platform to another.

Let’s start with the official route for moving from Android to iOS: Apple’s Move to iOS app.

  1. You can only run the app at the point of initial setup. If you managed to miss the option, enjoy wiping your iPhone and restarting.
  2. The fucking thing crashes repeatedly and there’s no way to resume the transfer. It starts from scratch. This is an infuriating process.
  3. If you try and Google anything related to this app, you get anything-but-helpful answers from the fandroid community. Instead of a detailed description of the technical issues with the app and how to get around them, you get helpful nuggets like:

Why would i move to ios, most android phones today out perform iphones. Quad core vs dual core, 4k vs 720p, freedom vs being told how your phone should look. Choice is yours.

FREEDOM! CHOICE! OPENNESS! (Except the freedom, choice and openness to move data between platforms easily etc.)

And things like this:

Moving to iOS was backwards logic. It’s like putting a straight jacket on your phone. Why jailbreak when you can use an open source OS like Android?

How about because the user might fucking prefer it? How about because you don’t want your phone pwned by the hilarious cascade of systemic security failure that is the Android ecosystem?

And one more:

Like I’d ever consider switching to using an iPhone… and as for that slow and buggy Watch of theirs, words can’t describe what a useless pile of overpriced crap it is.

Quite what the Apple Watch has to do with an app to help you move data between Android and iOS, I’m not sure. All discussions of going between mobile OSes ends up in pitiful religious argumentation and if you are just trying to get shit done, it does nothing but incite a lot of eye-rolling.

Fanboy awfulness aside, we re-ran the Move to iOS app once more but deselected the transfer of photos. I kind of figured that if we could get it to transfer things like old text messages and contacts and so on, then we could maybe move the 14Gb of pictures onto the iPhone by hand. They’ll all be stored as JPEGs on either the SD card or the internal storage on the phone.

After waiting some more, Move to iOS chunters everything it can except the pictures over. It kinda worked in a slow, clunky, crashy kind of way. Hardly as “seamless and simple as possible”, to quote Lifehacker.

Next job, let’s get the photos out. I had an Android phone many years ago. This should be easy. Plug it in, it mounts up as a USB mass storage device, drag and drop. That’s, y’know, what everyone tells me that Android did that iOS didn’t—no futzing around with iTunes, it’s just a USB mass storage device.

You’d think that. In the meantime, Samsung have deprecated USB mass storage in their Android devices and replaced it with the Media Transfer Protocol (MTP). Despite coming from the same place as Windows Media DRM nonsense, MTP sounds like a nice idea: unlike USB mass storage, it is a simpler protocol that implements much simpler operations than a traditional I/O interface, perfect for shunting the odd MP3 and JPEG around.

In practice though, actually using MTP from my Mac is soul-gnawingly awful. Obviously, there’s no filesystem that is mounted. For a while, I thought there must be some issue with the cable or that the Android phone had some formatting issue. Silent failures aren’t fun. Eventually, I find out how to switch the device into MTP mode and it then tells me that I ought to download the Android File Transfer app for Mac (you probably shouldn’t be surprised to learn that it isn’t open source). The UI is horrible. Cmd+A doesn’t work. It just randomly disconnects from the phone. And, worse, because MTP has no parallelism, transferring substantial quantities of data is godawfully slow. Especially if it is having to walk a directory tree and retrieve file lists. I’ve now dragged mostly all of the pictures off the phone and stored them on a computer, but systematically getting things other than the pictures off the filesystem has been unsuccessful.

There’s some alternatives, but all require effort. There’s libmtp, which I’m sure is excellent. There’s a rather nice looking Go program called go-mtpfs which will mount MTP devices as FUSE filesystems. That looks like it might be an improvement on the horrible OS X frontend at least.

What’s next then? How about WhatsApp messages? WhatsApp stores messages differently depending on whether you are using Android or iOS and all the methods for transferring between them look pretty damn rickety. They mostly involve shareware Windows apps from websites that give me that disconcerting feeling of talking to a sleazy second hand car dealer. I don’t quite trust any of the purported solutions to this.

I started looking into it myself. The files in WhatsApp for Android are stored in an encrypted SQLite store using a crypto system called crypt7 and/or crypt8. I have managed to extract this file from Android and I just need to decrypt it. There are some tools for this online. Pushing that data to iOS looks pretty simple. On the Mac, iCloud data is stored in ~/Library/Mobile Documents, and within there is a folder for WhatsApp containing a backup of WhatsApp conversations (in SQLite format) as well as media sent via WhatsApp. If one can retrieve the data from Android, it shouldn’t be too hard to push it into iCloud for the iPhone app to use.

All of this is way too fucking hard for non-programmers. The stuff on people’s phones is documentation of their life—their holidays, their families, their relationships, their co-workers. It’s not a part of some stupid platform war bullshit or an ideological debate about free software or DRM or whatever. It’s their stuff and they should be able to transfer it onto any device they choose to use. At the same time technologists have debated the ideal solutions for data portability, and churned out a thousand bullshit specs and documents that don’t do shit, ordinary people switching between iOS and Android (in either direction) have a hellish time doing ordinary stuff like WhatsApping with their friends and family. That’s a ludicrously poor show from our industry.

This is a shitshow. Be ashamed, fellow technologists. We have made a world that disempowers users and locks them in. And when they go online to find out why, all we have to show is a bunch of religious apologists telling them that this is okay because they are locked into a platform for their own freedom. Absolutely fucking terrible.

Today I learned: meta description tags are actually used by Slack for rendering link previews.

It may be invisible (partially visible, really) but when that data seeps through, it can often be wrong.

The Guardian have an interesting piece about Marissa Mayer and Yahoo! I concur with the general point: the primary problem with Yahoo! isn’t Mayer, it is the company itself.

Open plan offices are basically terrible in every way

There’s a growing consensus in the scientific literature that open plan offices damage the mental and physical health of employees, destroy their morale and generally make their work lives less pleasant. Let us first look at some studies.

Evans and Johnson, 2000:

Forty female clerical workers were randomly assigned to a control condition or to 3-hr exposure to low-intensity noise designed to simulate typical open-office noise levels. The simulated open-office noise elevated workers’ urinary epinephrine levels, but not their norepinephrine or cortisol levels, and it produced behavioral aftereffects (fewer attempts at unsolvable puzzles) indicative of motivational deficits. Participants were also less likely to make ergonomic, postural adjustments in their computer work station while working under noisy, relative to quiet, conditions.

Epinephrine is another word for adrenaline, which plays a role in the body’s “fight or flight” response. So, open plan makes you feel like you are being preyed upon by someone trying to kill you, and it makes you less motivated to work on hard problems, and it makes you less likely to adjust your work station posture, meaning you are more likely to get a nasty physical problem like RSI. That sounds great. Just the sort of place you can relax and focus on solving hard problems, as knowledge workers are asked to do every day.

What about the noise in our offices? Professor Adrian Davis from University College London told The Guardian last year:

The noise in open-plan corporate offices and call centres, for instance, may not damage hearing but can cause raised blood pressure, sleep and mood disturbance and other long-term health problems.

Of course, to avoid the distraction caused by noise in the modern open plan office, people turn to playing music to drown out the noise. Which itself can cause hearing issues.

I currently have some temporary hearing loss, not caused by loud music but by a minor physical condition that I am awaiting surgery for, so I’m very interested in work environments and hearing loss at the moment.

The British deafness/hearing loss charity Action on Hearing Loss estimate that 1 in 5 people currently suffer from some form of hearing loss. In a recent report, ACH estimate that £24.8 billion is lost from the British economy due to people not able to work as a result of deafness/hearing loss. Under British law, specifically under the Equality Act 2010, employers must make reasonable adjustments for staff with disabilties including deafness/hearing loss.

Open plan offices present special issues for those with hearing difficulties, whether temporary (like mine) or permanent. A person who has damaged their hearing in one ear are often less able to parse spoken language in noisy rooms—impaired speech discrimination (the ability to discriminate between the speech one ought to be focussing on from the background noise, sometimes called the “cocktail party effect”).

Guess what—there’s a paper on this too: Suter, 1979:

Theoretically, open‐plan offices are designed so that their occupants may study without being distracted, as well as converse with each other and on the telephone. Design criteria using masking noise are based on assumptions that talkers, listeners, and listening conditions are “average.” These assumptions are reviewed, especially as they affect hearing impaired people. There is evidence that noise levels of 47–50 dBA can degrade communication of the hearing impaired without affecting those who hear normally. A study of speech discrimination in noise showed that even people with mild hearing losses had considerably more difficulty understanding speech than their normal‐hearing counterparts.

So even if the added noise that an open plan office is designed to have doesn’t affect those with normal hearing, it can make it much harder for those who have hearing difficulties to hear. The increased use of breakout areas, informal meetings in cafeterias and so on (which are often caused by the fact that there either aren’t enough meeting rooms, or because booking time in meeting rooms requires an overly bureaucratic process or requires managerial approval) don’t help here either. The technology industry’s increased use of “agile” standup meetings and scrums means more noise and more problems for the hearing impaired.

And the damage done on hearing impaired people isn’t simply audiological, it can also mean more stress and fatigue. So sayeth Jachncke and Hallin, 2012, in a study comparing two small groups of hearing impaired and normal hearing people working in open plan offices:

In each experimental session they worked for two hours with basic memory and attention tasks. We also measured physiological stress indicators (cortisol and catecholamines) and self-reports of mood and fatigue. The hearing impaired participants were more affected by high noise than the normal hearing participants, as shown by impaired performance for tasks that involve recall of semantic information. The hearing impaired participants were also more fatigued by high noise exposure than participants with normal hearing, and they tended to have higher stress hormone levels during the high noise compared to the low noise condition.

People with hearing impairments struggle more to understand what’s being said in an open plan office, and the general noise makes them worse at their job. Oh well. They are only a fifth of the British population.

If an accessibility issue were affecting a fifth of a company’s customer or user base, that would be a pretty reasonable thing to invest in, as web accessibility advocates frequently argue. It seems curious that such little attention is paid to improving the environment that people find themselves working in every day given the disproportionate effect that it has both on the well-being of people with a disability, and on the negative outcomes for their productivity.

Even if you don’t care about people with hearing loss (again, 20% of the population and growing), let me just give the most pragmatic, hard-nosed business reason why you should care.

Pejtersen et al., 2011:

Occupants sharing an office and occupants in open-plan offices (>6 occupants) had significantly more days of sickness absence than occupants in cellular offices.

They just don’t bloody work. They make people without hearing loss or disability spend more time off work sick.

Let’s recap then. Open plan offices make things worse for employees. Those employees take more time off work sick, are less able to focus at work, escape from the agony of the chatterbox by playing loud music on headphones so as to filter out the noise. They are more stressed, more likely to develop a whole variety of physical health problems, and less motivated at work. And it has a disproportionate effect on people with hearing disabilities, which could potentially lead to legal liability under the Equality Act. None of this seems good for employers or workers.

If there were a chemical substance that caused the same health outcomes as an open plan office, it would be restricted as a potential public health hazard. Copycat management and cost-cutting has led to the creation of these toxic workplaces. The same people who will understand that knowledge workers need to be “in the zone” to get productive, creative work done put them in environments least conducive to that happening. We have articles giving tips on how to survive an open plan office, as if it were a bad case of the flu or a screaming child on an aeroplane. The truth is far worse: open plan is very likely to be the rest of your working life.

Is there an alternative? Well, we could have old-style offices. Some people would object, but peace and quiet and privacy sounds like it would be absolutely amazing. We could have a mass transition to remote working, which means people wouldn’t have to commute either and could live wherever they felt like. Almost anything would be better than open plan at this point. Whatever the future is, we can’t carry on with open plan for much longer. It is a profoundly broken model.

Jean-Paul Sartre is often quoted as saying “hell is other people”.1 Open plan offices have been an extremely successful attempt to prove such sentiment right.

  1. He kinda didn’t say that though, like with so many quotations.

The Reverse XY Problem

  1. Mediocre tech blog tells me Service X solves Problem Y.
  2. Service X has a website filled with jargon and bullshit (“enterprise-grade”, “Gartner’s magic quadrant”).
  3. I sign up for Service X and download their software for my computer.
  4. Service X does not actually solve Problem Y.
  5. “Delete account”.
  6. Repeat.


vim-gitgutter is perhaps my new favourite Vim plugin.

It isn’t quite as nice visually as the same effect that you get in IntelliJ’s range of IDEs, but it is good to have something similar.

Ignore the talking heads: TalkTalk's security issues run much deeper

I have to say I’m rather intrigued by the TalkTalk hack. First of all, they’ve found the 15-year-old who allegedly did it, arrested him and bailed him pending investigations. Hopefully, if said person did it, he’ll be quite interested in helping the police with their inquiries and with a bit of luck, the customers aren’t going to have their personal or financial details released. TalkTalk have waived cancellation fees for customers who want to leave, but only if a customer has sustained a financial loss.

Meanwhile, Brian Krebs reports that the TalkTalk hackers demanded £80k worth of Bitcoin from the ISP. We’ve now had the media tell us it is “cyberjihadis”, a fifteen year old boy, and people holding it ransom for Bitcoin.

What’s curious though is how the mainstream media have not really talked very much to security experts. Yesterday, I listened to the BBC Today programme—this clip in particular. It featured an interview with Labour MP Hazel Blears (who was formerly a minister in the Home Office) and Oliver Parry, a senior corporate governance adviser at the Institute of Directors.

Here’s Mr Parry’s response to the issue:

The threat is changing hour by hour, second by second—and that’s one of the real problems, but as I said, I don’t think there’s one way to deal with this. We just need to reassure consumers, shareholders and other wider stakeholder groups that they have something in place.

Just a few things. This attack was a simple SQL injection attack. That threat isn’t “changing hour by hour, second by second”. It’s basic, common sense security that every software developer should know to mitigate, that every supervisor should be sure to ask about during code reviews, and that every penetration tester worth their salt will check for (and sadly, usually find).

As Paul Moore has pointed out in this excellent blog post, there are countless security issues with TalkTalk’s website. Craptastic SSL, no PCI compliance. The talking heads are going on about whether or not the data was “encrypted” or not. The SSL transport was encrypted but you could request that they encrypt traffic with an obsolete 40 or 56-bit key rather than the 128-bit that is considered secure.

There are new and changing threats, but SQL injection isn’t one of them. That’s a golden oldie. And it also doesn’t matter if the data is encrypted if the web application and/or the database is not secured against someone injecting a rogue query.

Mr Parry is right that there is not “one way to deal with this”. There are plenty. First of all, you need to hire people with some security expertise to build your systems. You need to hire independent experts to come and test your systems. That’s the in house stuff. Unfortunately, TalkTalk seem to have lost a whole lot of their senior technical staff in the last year, including their CIO. Perhaps they weren’t confident in the company’s direction on security and technology matters.

Then there’s the external facing stuff. Having a responsible disclosure process that works and which gives incentives for people to disclose. Have a reward system. If you can’t afford that, have a guarantee that you won’t seek prosecution or bring legal action against anyone who engages in responsible disclosure. Have an open and clear log where you disclose your security issues after fixing them. Actually fix the issues people report to you. Again, Paul Moore’s post linked above notes that he tried to contact TalkTalk and was ignored, disrespected and threatened. That’s not how you should treat security consultants helping you fix your issues.

All of this stuff should be simple enough for an ISP that has over four million paying customers. It isn’t rocket science. The fact that they aren’t doing it means they are either incompetent or don’t give a shit.

Brian Krebs nailed this corporate failure to care about security recently in a discussion on Reddit:

I often hear from security people at organizations that had breaches where I actually broke the story. And quite often I’ll hear from them after they lost their job or quit out of frustration, anger, disillusion, whatever. And invariably those folks will say, hey, we told these guys over and over…here are the gaps in our protection, here’s where we’re vulnerable….we need to address these or the bad guys will. And, lo and behold, those gaps turned out to be the weakest link in the armor for the breached organization. Too many companies pay good money for smart people to advise them on how to protect the organization, and then go on to ignore most of that advice.

Mr Parry said that the important thing was reassuring customers that their information was safe, not actually ensuring that customers data is safe. This is exactly the problem. I don’t want to be “reassured”, I want it to be safe. I don’t want to be reassured that my flight isn’t going to crash into the Alps—I actually want my flight to not crash into the Alps. Reassuring me requires salespeople and professional bullshitting, not crashing requires well trained pilots and staff, engineers doing proper checks, the designers of the plane making sure that they follow good engineering practices, constant testing. Engineering matters, not “reassurance”.

So long as business thinks “reassuring” customers matters more than actually fixing security problems, these kinds of things will keep happening. It would be really nice if the media actually spoke to security experts who could point out how trivially stupid and well-known the attack on TalkTalk was, so this kind of industry avoidance tactic could be properly squelched.

The story that’s been doing the rounds about the woman who presented her father a “certificate of purity” on her wedding day is itself a hilariously awful example of how American religious culture screws up sexuality but this blog post from Homeschoolers Anonymous made me see exactly why it is so horrible: in saying that virginity (especially female virginity) is ‘purity’, it automatically implies that anyone who has been raped or sexually assaulted or abused is ‘impure’.

That’s the message that is sent in such cultures: that a person—especially a woman—who has had sex, whether voluntarily or not, is ‘impure’, is ‘damaged goods’. And if they’ve been abused, no matter how much they’ve worked to free themselves of the shame or negative feelings about that abuse, they are haunted by the cultural stain of ‘impurity’.

We need to burn purity culture to the ground: it is truly nasty psychotic shit created by truly awful people.

Pi is a constant unless you are a software developer

You may think that pi is a constant like 3.14159 (etc.), but that is because you learned mathematics in school. You learn a lot more once you enter the world of work.

As a software developer, you have to get used to the fact that pi is whatever management have decided it to be this week. For one project, pi might be 3.14159, but in other projects management may decide that pi is actually the integer 4, an API call to an old system that you need to use Java Remote Method Invocation to access, an XML document that is inside a Word file in an email account you don’t have access too, or it might be green, except on Thursday when it is furry and invisible. That’s what they sold the client, so that’s what it is.

You might think that if pi is being redefined, someone might file an issue about it or raise it with you in a meeting. No, they’ll mention it to you off-hand and indirectly, or they’ll put it on page 53 of a PowerPoint slide deck with a filename like CLIENT PRESENTATION DRAFT FINAL MAYBE? CHECK WITH CAROL (1).pptx that they CC to you, except the subject line says “Drinks on Friday” and the body is about the birthday party for Steve in accounting. Or, worse, they’ll upload it to Confluence, because it has to be useful for something. When you persist with your school-derived belief that pi is 3.14159 (or the value of java.lang.Math.PI and equivalent), you will be upbraided for not keeping up with the rapidly changing needs of a growing enterprise.

When you have learned to live with the fact that pi could be anything from one day to another, and once you have learned to listen patiently to the idiot telling you that pi is actually a Labrador puppy, suppressing your strong desire to grab him by the lapels and throw him down a ricin-laced lift shaft, you have progressed to the point where you can become a senior developer, a project manager or a consultant.

None of this means that pi is anything but 3.14159. It is a mathematical constant—your maths teacher is right. If you need to deal with the cognitive dissonance, step away from your computer, have a nice holiday—or a good gin and tonic, or dance your tits off in a nightclub or whatever hedonism gets you through life. Once you’ve coped with the dissonance, pretending pi and many other similar things aren’t constants makes dealing with companies much easier. Deadlines, requirements and functional specs are mostly make believe, so why can’t mathematical constants be fictional too?

Some Netgear routers vulnerable to hack that allows modification of DNS server. Consumer-grade routing and DSL gear seems like a pretty vulnerable area: consumers have no idea about security, there’s no automatic software updates, and it is somewhat unlikely to be detected by the user.

Django: automatically testing admin pages are working

Today, I was working on a simple Django app. I was cranking away on something, then went to the admin panel and… something wasn’t working. I had made a typo and written foriegn rather than foreign in formfield_for_foreignkey.

And I hadn’t noticed. Computers are supposed to notice these things. Test suites and CI servers are supposed to catch my errors.

I realised then that having something that just checks to make sure that the admin panel is working is useful.

Something like this.

You could automate this some more: have it so it probes through your admin panel and clicks links for you. This will do for now though.

One could also go further and have the tests put data in the forms and so on. But this is good enough. It’s likely to blow up if you’ve made a mistake when you are writing the Python code that defines the admin panels.