Stefan Zweifel

Writing Integration Tests for git-auto-commit

• 4 min read

A couple of weeks ago I had a break through moment for git-auto-commit: I finally managed to write a good integration test suite. But first a bit of backstory:

Backstory #

As the Action gets more and more popular (it's in use in more than 7000 public repositories 🤯), more and more feature requests came in. Good ideas which made the Action more flexible and useful for a lot of users. But it made the Action more susceptible to errors.

During the summer months, I released new versions every other day with the confidence, that "this little change won't break the Action for others". Oh how naive I was.
A couple of hours later the issue tracker was filled with reports, that the latest version broke everyones Workflow. Not a good feeling.

For a short moment of time I've contemplated to rewrite the Action in JavaScript to make testing easier. But would that be really worth it?

I deliberately wrote the Action in Bash for a couple of reasons:

  • The core logic of the Action is simple: If git status detects changes, commit everything and push it back to the remote repository on GitHub.
  • JavaScript projects come with a certain set of overhead: dependencies, upgrades to said dependencies, compatibility with different NodeJS versions, …
  • Speed: Dependencies would have to be installed whenever the Action is run in a Workflow. This can slow down Workflow runs.

It just doesn't make sense to me, to add all this stuff to this simple Action.

So I stayed with Bash and explored the different options available to write tests. The most popular framework is Bats which I've ended up using.

An honorable mention goes to bach. It takes a new approach of writing test, which I really like. I stuck with Bats though, as it was just simpler to use for this project.

First Version: Mocking git #

In Pull Request #100 I've added the first version of the test suite. As mentioned, It was written in Bats and made heavy use of shellmock.
I immediately knew that this wasn't perfect and would be a temporary solution. (I still spent some time and added tests to cover every feature in PR #109 though).

Mocking calls to git felt "wrong" – especially as the entire Action revolved around git. But why mock git, you might ask yourself now.

Well, as this Action creates commits and pushes them to GitHub, I first thought that my tests would have to make those pushes too. But that came with another problem: How should contributors run the test suite, if they don't have push access to the test repository? Writing a huge set of instructions? Doesn't sound like a good idea.
That's why I thought mocking git is the better solution. The tests would never actually run a git-command and certainly won't push to a remote GitHub repository. Win-Win (?)

But a couple of weeks later it showed, that this was a bad idea. The tests were green, but made the test suite brittle: A new feature was added and I immediately had to rewrite my existing tests and add additional mock calls to git to bring the tests back to green.

This made the test suite useless. Updating all tests after adding a new feature is way too time consuming.

I kept the Action and test suite in this broken state and kept thinking about this problem in the back of my head for a couple of weeks.

The breakthrough #

The breakthrough to a good test suite came on a November morning while lifting weights at the gym. (In hindsight it looks so obvious that I wonder why I didn't came up with the idea sooner.).
As mentioned, the problem was, that I didn't want to make actual requests to a remote repository on github.com (This would make tests slower and would require contributors to have access to said repository).

A git remote repository doesn't have to live on a remote server. It can also be on the same file system.
Instead of pointing the remote to github.com I just point to a local path:

- git remote add origin https://github.com/user/repo.git
+ git remote add origin ~/Sites/remote-repository

🤯

In Pull Request #128 I've rewritten the entire test suite to use those "local" remote repositories.

What a relief this discovery was. I had so much joy writing the tests! I could use all the different built-in tools of git to assert that the Action actually does what it should do. It's also fast and it works on my machine, on CI and can be run by contributors.

And the biggest improvement of all to the previous version: New features don't require me to rewrite all existing tests as nothing gets mocked. git just does its thing.

Conclusion #

Working on git-auto-commit showed me, that not everything has to be written in a modern programming language to have a good test suite. (And that good ideas need time to marinate)

Shoutout to Zach Holman for sharing how he wrote tests for During in Bats. The article doesn't mention it, but he was using bats-assert and bats-support to write cleaner assertions.
That short article also nudged me in the right direction.

As I spent quite some time researching testing in Bash without finding a good "Getting started" guide I've created my own: Getting Started with Bash Testing with Bats.

I hope it will help other developers new to Bash testing and make their lives easier.