Toughts on testing an untested software

In the software engineering world testing is important, unfortunately in your career you may find yourself working on software that does not have automated tests.

In my own experience this can be quite frequent in research based projects. Since a fair amount of those projects starts as a POC or multiple scripts where the emphasis is more on exploration and research than producing high quality code (which is perfectly normal). The industrialisation phase, which aims to build a production ready software based on those scripts, may not always involve adding automated tests.

Recently I found myself working on a similar project, it was a rapidly evolving project that involved a lot of developments and updates.

It was clear at some point that the more we updated the higher the risk of regression. Fortunately the product was still in the release phase and were intended to be used internally. This made the users very forgiving on errors regressions and didn’t expect a 5 nines reliability.

But the more we integrated our tool to our existing system the less our users stayed forgiving. So we decided to add automated testing to our tool.

The following is my own take on the subject, the lessons I learnt and how will I recommend to do it.

Why do it ?

Adding automated tests make your code safer and the likeliness of regressions smaller. This is especially important if you update the software frequently. It’s always frustrating for you (and your users) to have to fix the same issue twice (or even more).

It also makes contributing to the project for newcomers easier, since they can safely update a part of the code without breaking other parts that they may be not aware of.

Adding automated tests on a non tested software does need (a lot) of time and doing it will make delivering new features for your users slower, at least in the short term.

But there is also a less obvious benefit for you: Doing it will force you to write better and testable code, since functions that do multiple non related things are arguably harder to test and their downsides are often visible later on when you need to change them.

How not to do it

First of all, although it may be tempting, you shouldn’t try to do it in one pass.

Depending on the project you can’t stop all development for 2 weeks (or more) to add tests. This can have a huge impact, especialy if your product is in the early stages. Understanding your clients needs may give you more value at the beginning that ensuring you hit 90% coverage.

How much should you test ?

So how much testing (per time unit) do you need to do?

There is no right answer. This will depend on multiple factors :

• How familiar are you with the code/structure of the project: Adding tests to a new project is often trying to answer: where can i add the most value by adding tests? This is a question easier answered if you’re already well aware of the critical parts of your software. If you’re new to the project you may instead prefer to spend more time to understand the current code.

• How often do you update the code: The more you do it the more the risk of adding regressions which means the more value you’ll get by automaticaly testing it

• How many people are actively developping the software: I believe the more you are on a given project the more the risk of breaking others people code by introducing a change.

How to do it ?

Again this can be reduced to the following problem: What are the parts of my code that will benefit me the most if tested.

In this regard, I think that starting with integration tests will probably give you the most value (cost wise) and will make catching obvious issues (for errors such Too many values to unpack in Python) easier.

After that you can start adding unit tests in the following order:

• When you solve an issue: This means, when you solve an issue try implementing the unit test that will prevent this issue from occurring again. Of course, this is just a rule of thumb and some issues will clearly be a one time thing, but still a good practice.
• Code that you just added: This will also encourage you to write cleaner and more testable code.
• If you’re familiar with the code / structure you can focus on critical parts of the system: Where will the regression really hurts ? For example if you work on a trading project function that monitor usage statics will typically be way less critical than the ones who compute the metrics which my directly impact traders decisions.

Ideally you need to prioritize the code that has both criteria : criticality and changing rate.

To get an idea of what parts of the code are the most updated you can do this:

git log --pretty=format: --name-only | sort | uniq -c | sort -rg | head -10

Source : https://stackoverflow.com/a/7686616

You should also, in my opinion, prioritise the code that is more easily testable. If the code doesn’t have unit tests, there are big chances that some parts may not be easily testable and will involve writing a lot of mocks and stubs. Refactoring is an option but can be very costly and hard if you’re new to the project.

What about code coverage ?

In my opinion code coverage can be important, but only in the early stages, the more unit tests you have the less coverage becomes important and the reasoning behind this is: If you have let’s say 70% code coverage you can already catch a fair amount of obvious issues, but chasing the remaining 30% may bring less value that adding a truly important unit test on the core parts of your software (as the metrics).

And focusing too much on code coverage can easily work against you by encouraging you or your team to increase it regardless of the quality of the tests, and it may give you a false confidence about the reliability of your automated tests.

Overall exerience

All together it was a great experience, I was lucky to work in a team fully aware of the importance of testing and who was encouranging investing time writing tests and enforcing them. Writing tests will make your work/dev on the software slower in the short term and some teams may see it badly. If this is your case, communicate with your team explain your point of view and give a clear amount of time you plan to dedicate writing tests.

It’s extremely rewarding when you (or another person on your team) makes a change, and one of the tests you wrote fail due the change breaking some parts of the existing code.