Yes. I think this Etsy story showcases an interesting split in engineering approach, between being "right" vs being "effective". Perhaps a variant on "worse is better".
Etsy was originally built by two NYU students while they were still in college. It was their first production software project. It grew super fast in a period where the whole industry was still figuring out how to scale web applications; perhaps only Amazon & eBay had actually figured it out by then.
Then VCs got ahold of Etsy and tried to put out the fires by replacing the "kids" who built the thing. This was around 2006, which predates even Twitter's fail whale and Series A funding to round.
The founding team wasn't a bad engineering team. It was a young founding team who built a thing that grew like crazy, and which, had it not been built that way, would have meant no engineer could have ever come along to preside over a rewrite. But because it was crashing and dollars were at stake, and because there was some drama that prevented the founding team from having the time to build a proper scale-up engineering organization, a bunch of people came in and immediately reached for a rewrite.
They had a working thing, on a not-that-uncommon stack, but the new engineers insisted on a rewrite. Why?
Because new engineers always insist on a rewrite. Because, as my friend James Powell once put it, "Good code is code I wrote; bad code is code you wrote."
Python and Twisted. I am a big fan of Python and watched the evolution of Twisted -- the rise in popularity, and then the decline. I can imagine how they came across a group of engineers who were absolutely convinced Twisted was the silver bullet to solve their engineering challenges, at that specific moment in time -- when Twisted was at a peak of popularity.
Just like Django/Rails was the silver bullet in 2011. (Or Scala?) Just like Angular was in 2015 (or Go?). Just like React is now (Or k8s?).
Of course, I am not using the "silver bullet" term accidentally. Tech trends change, but "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity." (Brooks, 1976)
New tools are just an excuse for a rewrite, and rewrites are the ultimate shiny object to a new engineering team, brought in to "fix" an existing and working system.
As an aside: the founding software team at Etsy didn't get nearly enough credit for building a thing that worked and that grew from nothing. If they had focused on the "right" architecture, none of us would have ever have heard of Etsy, and we certainly wouldn't be debating the rewrite. That founding team didn't get enough credit, neither in equity nor in history. But that's another story.
It is hard as a new employee to not say "stop everything and let's rewrite this" when something is broken. New engineers are not only the ones to say it either as the craze and popularity of what's hot captivates longer tenure employees. If something vibes with the problem you are solving then it is only natural to want to use it. Proving the usefulness and viability is the part no one really wants to do as it is a long slog that is against the need of getting a promotion. Why not just use in a new app to prove it? and that will be the basis for let's use this everywhere (aka generalize it!)
It is very nice to try new things though because without that we can't move forward, but how much forward is enough seems to be the troubling part as investing time learning the "new unproven thing" every year or so is taxing.
> As an aside: the founding software team at Etsy didn't get nearly enough credit for building a thing that worked and that grew from nothing. If they had focused on the "right" architecture, none of us would have ever have heard of Etsy, and we certainly wouldn't be debating the rewrite. That founding team didn't get enough credit, neither in equity nor in history. But that's another story.
Indeed. And, to be clear, I don't think rewrites are always wrong. I just think they are very, very dangerous, and need to be treated as such. I wrote about a successful rewrite I presided over in this essay, "Shipping the Second System". But anyone who was on my team when we did that rewrite will tell you: we were far from optimistic about it. We only did it after exhausting ALL other options, and it was a risk-managed project all the way through.
> “the second system is the most dangerous one a man ever designs”
Never thought about it that way. Feature creep and aversions to what you thought went wrong from the last time get tested here.
> So, all in all, we did several “upgrades to stable” that were actually bleeding edge upgrades in disguise.
I can relate. Being burned by certain upgrades can push one to veer on the side of caution even if the previous version had bugs. Software is never finished as they say.
> Because new engineers always insist on a rewrite. Because, as my friend James Powell once put it, "Good code is code I wrote; bad code is code you wrote.
Good code is code that is testable and tested, bad code is code that is code that is not tested (and may not even be testable).
As someone who's written too much non-tested (and sometimes non-testable) code, I'm pushing myself to test, and to hold that line, on projects both large and small.
Came in to a project last year that had been around 4 years. It was not only untested, but the original code in place turned out to be somewhat untestable altogether. 16 months later, we're still finding "rock bottom" moments of "holy shit this is garbage". By "garbage" I mean finding out that data that people that was being recorded was being discarded or overwritten, for months. Financial data being simply wrong, and 'correct' data not being recoverable.
First reaction looking at it was "this needs to be rebuilt". No no no, we were told - it's already working it just needs a few more features. To the extent that things were "working", they were working on accident. There were no docs or tests to demonstrate that things were working as expected.
The last year has been spent keeping it 'going' while trying to unravel and repair years of bad data is continually uncovered every few weeks. "We don't have time to rewrite!". The fact is, probably 60% of the code that was in use before has been rewritten, but it's taken over a year. It would have been faster to do it from scratch, and worry about importing/transforming data as we verified it after the fact.
So... good code is testable and tested. Absent these qualities, if I'm the 'responsible party', I will advocate strongly for a rewrite. If that 'rewrite' is 'in place', people will be told up front everything will take 2-3x longer than they expect, because we're now doing discovery, planning, testing and whatever 'new feature dev' you were asking for.
Part of the problem with this particular system was that it was started as a rewrite, but for the wrong reasons, or... executed wrong. "We can't understand the old code", so they tacked on a second framework in place and started blending the code - but still documented nothing, nor wrote any tests, or had a repeatable build process. Nothing. So instead of one undocumented system, they had 2, but one was in their head, so it was 'OK'. Based on what we inherited, one could only assume they really didn't have any understanding of the domain problems they were tackling, which just compounded every bad decision.
Etsy was originally built by two NYU students while they were still in college. It was their first production software project. It grew super fast in a period where the whole industry was still figuring out how to scale web applications; perhaps only Amazon & eBay had actually figured it out by then.
Then VCs got ahold of Etsy and tried to put out the fires by replacing the "kids" who built the thing. This was around 2006, which predates even Twitter's fail whale and Series A funding to round.
The founding team wasn't a bad engineering team. It was a young founding team who built a thing that grew like crazy, and which, had it not been built that way, would have meant no engineer could have ever come along to preside over a rewrite. But because it was crashing and dollars were at stake, and because there was some drama that prevented the founding team from having the time to build a proper scale-up engineering organization, a bunch of people came in and immediately reached for a rewrite.
They had a working thing, on a not-that-uncommon stack, but the new engineers insisted on a rewrite. Why?
Because new engineers always insist on a rewrite. Because, as my friend James Powell once put it, "Good code is code I wrote; bad code is code you wrote."
Python and Twisted. I am a big fan of Python and watched the evolution of Twisted -- the rise in popularity, and then the decline. I can imagine how they came across a group of engineers who were absolutely convinced Twisted was the silver bullet to solve their engineering challenges, at that specific moment in time -- when Twisted was at a peak of popularity.
Just like Django/Rails was the silver bullet in 2011. (Or Scala?) Just like Angular was in 2015 (or Go?). Just like React is now (Or k8s?).
Of course, I am not using the "silver bullet" term accidentally. Tech trends change, but "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity." (Brooks, 1976)
New tools are just an excuse for a rewrite, and rewrites are the ultimate shiny object to a new engineering team, brought in to "fix" an existing and working system.
As an aside: the founding software team at Etsy didn't get nearly enough credit for building a thing that worked and that grew from nothing. If they had focused on the "right" architecture, none of us would have ever have heard of Etsy, and we certainly wouldn't be debating the rewrite. That founding team didn't get enough credit, neither in equity nor in history. But that's another story.