The Iowa caucus app didn't fail because "we" are bad at software engineering. It failed because someone did a shitty job of understanding the problem space, deciding on an appropriate solution, and finding proper resources to make sure that solution was well-implemented.
I think properly implementing a solution appropriate to the problem space is the literal definition of engineering. So "we" aren't bad at software engineering but someone definitely is.
The someone in question were inexperienced devs fresh from a bootcamp. This is a recipe for failure in every industry from making coffee to making spacecraft.
Perhaps software is somewhat unique that we don't appreciate experience.
He has a legitimate point, though. Even as a software developer for nearly 30 years now, I’m surprised that there aren’t standardized solutions to these kinds of problems; I can sympathize with non-developers who figure, “this has been done in a similar form thousands of times, it should be almost a rubber stamp”.
Upfront I'll say I know the people at Shadow and worked right next to them at HFA. They're super competent (they worked on some of the more critical things) and some of the best people I know. Just so you know where I'm coming from. I haven't talked to them about any of this.
Political software is very wild west. There are dynamics that are just impossible to anticipate, like every Saturday this vendor's response time drops to nothing because they do weekly backups, or we're coordinating all the votes through this person but they've never used a smartphone before. There are some fundamental truths here:
- these systems and processes are underfunded, ad-hoc, and run by people who are compromised in significant ways (haven't slept more than 3 hours in 2 weeks, no experience with technology, responsible for literally 30 things, downright incompetent but bodies were needed, drug/drinking problems).
- we're now in an age where these systems have to be secure against state-level actors, and we are woefully unprepared for that in every way you can think of.
- very little in our political system can hold up to scrutiny. Caucuses in particular are and always have been junk, but even elections are... maybe impossible to verify. And even if we could perfectly record every vote, externalities (weather, voter suppression, malfunctions) and people's bad memories ensure there will always be a gap between the will of the people and the results of an election. They (and everything else about this process) are fundamentally imperfect.
At HFA we dealt with this by planning for it from the beginning, and when you're the presidential campaign you can do a lot to combat these fundamentals (run drills, hire super competent people for pennies who will work for you out of patriotism, run red teams, strongarm and replace vendors, etc). We still made mistakes that haunt me. It's just how it goes.
Nowhere else in politics is like that. So you do your best: don't put your app on the app store, subject to anyone downloading it and review by Apple/Google, use MFA even though it will undoubtedly confound 80% of your volunteers. Hope the internet holds up in rural areas or gyms with satellite trucks and crowded APs/towers. Hope no one learns the numbers of your backup hotline. Hope people know the rules of the caucus and count perfectly so the results aren't tainted and called into question ad nauseum on national news media. Hope no one thinks you're part of a conspiracy to cheat their favorite candidate and doxxes you. Make your app as simple as possible. Accept your client's insistence on secrecy in the name of security even though you know training and testing are essential. Hope there's no quirk in the vendor's API that will reject results on caucus night.
This whole thing is like a Rorschach test. Are you a person who thinks tech has no place in elections? That's what you see here. Are you a Sanders supporter who thinks there's a conspiracy (not for no reason) against him? Do you think Democrats are fundamentally incompetent? Do you lament the state of software robustness? Etc etc. I would implore people who are falling into into this stuff to recognize these mental reflexes. Very few people know anything about how any of this works (I certainly didn't) or the details of what happened in Iowa, and if you don't, well probably shouldn't extrapolate from this situation then.
You bring up a lot of valid considerations for designing software at scale. In good engineering, these known unknowns would be considered during the design phase.
If the argument is that those items weren't considered, that is definitionally bad engineering and I think GP is dead-on in their assessment of Shadow's development.
If the argument is that there were unknown unknowns that caused the app to malfunction and the items you listed were just examples, and we don't actually know what went wrong - well, I'll be happy to read the post-mortem.
But that's not what happened here. This wasn't an "unreliable vendor on Sunday" situation, or a "we couldn't budget for offline local storage for rural internet". This was a "our app literally doesn't even let users log in" scenario, and I find that really difficult to excuse under the blanket of "political software is hard."
There are postmortems. They were using Auth0 w/ MFA. Login problems were lots of user error (putting wrong creds in wrong boxes) and bad internet. Not a lot you can do about that without opening up an enumeration attack or erecting cell towers.
Basically everything else you wrote is outrage at the state of software robustness, so I'll refer you back to my last paragraph.
