Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If your product is used by humans, then it needs to be tested by humans - this cannot be automated. Those humans can be your QA people, or your customers. Perhaps your customers are happy to be testers, perhaps not.

Unit tests are very expensive and return little value. Conversely, a (manual?) 'smoke test' is very cheap and returns great value - the first thing you do when updating a server for example is to check it still responds (and nothing has gone wrong in the deployment process), takes 2 seconds to do, prevents highly embarrassing downtime due to a misconfigured docker pull or whatever.



> Unit tests are very expensive and return little value

Why are unit tests very expensive? This goes against everything I know.


Unit tests very roughly double the amount of effort required to make any meaningful change to your codebase. They are also require maintenance same as ordinary code - but the customer does not care in the slightest whether or not they pass. On the other side, they can only really tell you about low level bugs that you already expected, they cannot surface system level bugs - the actual hard bugs that cause problems for you and your customers.

Then there is the danger of thinking that green=all good, an example of 'automation bias' where we learn to trust the automation even as things go wrong.

As makers, it is also tempting to believe that [all] problems can be solved by making something (i.e. code), but actually many problems are not of that nature, and cannot be solved in that way.


Thank you, that makes sense. What I meant was that today all unit tests are basically written by an AI so the "cost" is almost zero. Am I wrong?


Sorry yes. If an LLM written unit test fails, then it has to be determined whether the test was wrong or the code was wrong. This is an expense in human oversight, unless of course we believe that LLMs will get it right at a high enough rate that they can be left to code everything themselves completely automatically.


Automated tests are no longer expensive to write and nowadays less expensive to execute.


Sad to see you being downvoted, but you're exactly right. Well, almost - if you can afford to invest in a good integration test suite, that can catch many errors without requiring a human to regression-test every time.

At the same time, many quality attributes can't really be automatically tested, so automation shouldn't try to replace manual testing, it should be used to augment it.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: