Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I test your way, but if you're not aware of it you might want to watch this talk called Integration Tests are a Scam: http://www.infoq.com/presentations/integration-tests-scam It may explain why the "gearbox shit the bed" and how to fix that problem.

If you want a talk that preaches to the/our choir, check out this one: http://vimeo.com/68375232

It's interesting how different they are. Mutually exclusive in fact. But is one wrong, both or neither?



> Integration Tests are a Scam: http://www.infoq.com/presentations/integration-tests-scam

Thank you, that was interesting! A lot of good points. I agree with some of them and don't agree with other parts of it.

First, let's start with this basic principle -- "customers don't care if I have 1M unit tests or 0 unit tests". They just don't. They will complain and not like my product if it keeps crashing or is showing other pathological behavior. First and foremost it is important to ship, and to ship working code as far as customers are concerned. Taken from that side, I want to ensure that when the system is setup in customer's configuration (and we do know it, in our line of work), then we want to stimulate the system as close as possible to what a customer would do. If that doesn't break and is behavior properly. We gain immediately a large degree of confidence because it is closer to "doesn't break when customer uses" use case.

Unit tests also promise that, and they would also promise that "if I have unit-tested my code well, there is a higher chance that the code will not break when it gets into customers' hands". However notice that this is further from being true than the first one.

The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results. When something is broken I might not be able to quickly zoom in on it. I can't experiment as often as I would like because it take too long to discover precise mistakes. And I agree with those, but, I still maintain it is better to have integration tests because once you get into a stable steady state and all of the sudden one commit breaks the tests it usually is pretty clear what broke, in the worst case just revert the commit or examine in detail until it works. That helps with the "I don't know what broke, something did" part. Usually that something is the one commit where tests started failing.

Yeah, it would be better to also have complete 100% unit test coverage as well and zoom in immediately on the problem. No doubt that 100% unit test coverage also costs a lot of time and money and I posit it is often un-necessary.

Another criticism, perhaps I misunderstood the presenter, it seems to me that the talk focuses more on machine-local, object oriented code. Today the systems are more and more distributed. In a pure, local, single threaded, object oriented code (say a 3D modeler or Photoshop).The behavior or objects is more predictable and it is easier to write what he calls "focused tests". Now introduce threads and well that starts to become hairy. Add in multiple machines (a distributed system) and it gets even more complicated. There is a whole slew of strange problems. Often involving consistency, timeouts, networks failure, partitions, hardware peculiarities. Un-expected input that is generated maybe because a front end service is broken and back-end write might never expect that case and crashes. Unit tests will catch basic errors but it will mask those harder to find those kind of errors.

One more criticism is psychological and it is the false sense of security. And I have seen this happen from those that advocate unit testing everything 100%, when their code breaks, they simply don't believe their code could break because it has 100% coverage and they "just ran it and it passed". Well in production it crashes. And customers don't pay for passing tests they pay for a working product. Anyway I am repeating myself at this point and this is longer rant, so, sorry about that.


> The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results.

I think you left out the main complaint: It's impossible to cover a majority of code paths.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: