We no longer use installed tests in CI, so lets reap the benefit and simplify our build setup for tests.