Restrict selective test results uploads to specific branch or environment

RFCs
1

Need/problem

Some organizations have a need for only uploading selective test results from a specific branch, usually main, to increase confidence in the selective test results or to make it easier to diff snapshot testing results. Additionally, some organizations might want to restrict the environment that the results are uploaded from – such as from the CI.

Detailed design

There are two ways I thought about fixing this, the solutions are not exclusive.

CLI flag

One way how we can accommodate the need is by adding a flag for explicitly telling Tuist that selective test results shouldn’t be uploaded.

My suggestion here is to use --no-upload:

tuist test --no-upload

This can be also extended to other commands like tuist cache:

tuist cache --no-upload

The test results would still be stored locally but they wouldn’t be uploaded to the remote storage.

Remote Tuist Project configuration

Alternatively, we can make it configurable in the remote Tuist Project configuration to enforce test results are only uploaded from a specific branch and from the CI. However, since developers could already encode on the CI which branch results should be uploaded from, I’m leaning to allow only configuring that CI is the only environment that the tests can be uploaded from:

tuist project update --ci-upload-only

Drawbacks

We add extra complexity around when test results are uploaded/fetched.

Alternatives

Instead of adding a new CLI option, we can push the configuration to be only at the remote Tuist project level.

Adoption strategy

Nothing would change for developers who don’t need this extra flexibility. They would opt-in to use something like this when such a need would arise in their organization.

Unresolved questions

Do you agree with the proposed solution of --no-upload at the CLI level and restricting to CI at the remote Tuist project level? What do you think about the naming?

1 Like
2

I wonder if --no-upload is explicit enough about what’s being uploaded by the flag. This is something that we can solve with the documentation of the flag, but I was wondering if we can make the naming explicit enough to make that unnecessary. Some ideas:

  • --no-upload-{results/binaries}-to-cache (upload implies the remote cache)
  • --no-persist-{results/binaries}-to-remote-cache (we separate the cache, remote in this case, from the action but the flag naming is longer)

Other than that, I’m on board with the approach.

3

The motivation behind using --no-upload was to have something a bit more generic as it’s likely that if you don’t want selective test results to be uploaded, nor would you want that for binary cache.

Alternatively, we can have specific categories as a list, such as tuist test --no-upload selectiveTests. If you don’t specify any, we would default to all categories. That would be in line with how tuist clean works where you can run tuist clean to clean everything, but you can also run tuist clean selectiveTests to clean only selective test results.

What do you think?

4

I’m on board with that idea.

1 Like