Long tests build times

Hi :wave:

After moving to microfeatures architecture, we are seeing our tests build time skyrocket (from 2 minutes to 9, on some off-cases to 20), has somebody seen something similar?
Every feature has its own tests target which are then added to Workspace 's testAction as a new TestableTarget

We have also a lot of dependencies with carthage (around 20) and sometimes the phase Embed Precompiled Frameworks seems to be slow

I also wonder if adding the tests to Workspace is the right thing to do, but I could not think of a different way how to be able trigger all the tests from one place (and also from command line with xcodebuild )

It’d be nice if the tests did not have to include all the pre-compiled frameworks, but I suppose they are necessary since our Core module is linked against them and every module imports the Core module.
Interestingly, the build times have not seem affected in a substantial way (only running tests)

(side note: non-test modules do not have embed pre-compiled frameworks phase, which is one thing that lead me to suspecting it)

W

ould be glad to hear any of your ideas!

2 to 9 minutes seems to be a lot. I’d start by looking where the time is mostly spent. You can add build phases manually that store a timestamp locally that you can later use for comparison.

We have also a lot of dependencies with carthage (around 20) and sometimes the phase Embed Precompiled Frameworks seems to be slow

Unless I’m missing something, I think the embed phase should be fast. 20 frameworks seems a reasonable amount of frameworks.

I also wonder if adding the tests to Workspace is the right thing to do, but I could not think of a different way how to be able trigger all the tests from one place (and also from command line with xcodebuild )

Can’t you define them in a project scheme?

t’d be nice if the tests did not have to include all the pre-compiled frameworks, but I suppose they are necessary since our Core module is linked against them and every module imports the Core module.
Interestingly, the build times have not seem affected in a substantial way (only running tests)

Have you tried running the tests removing that build phase. I think it’s necessary because the frameworks are transitive dynamic dependencies and the XCTest runner links them dynamically before running the tests. Could you confirm if removing the build phase results in tests not running?

(side note: non-test modules do not have embed pre-compiled frameworks phase, which is one thing that lead me to suspecting it)

By non-test modules do you mean framework targets? Because in that case copying frameworks is not necessary. The build phase should be defined in those targets that represent apps though.

1 Like

This is correct. As I mentioned in Slack, a nice hack that can save a LOT of disk space in this scenario (especially if you have a lot of test hosts) is to set the runpath search paths to be equal to the framework search paths. This works fine for unit tests still since they’re all running directly off builds, but you can’t do that for app builds obviously.

2 Likes

Thank you both for your replies.

Can’t you define them in a project scheme?

I do, but I also want to have one place from which I can run all the tests - which is why they are defined in the workspace, too.

Could you confirm if removing the build phase results in tests not running?

The build phase is necessary (if you do not define custom runpath search paths)

By non-test modules do you mean framework targets?

Yes and as you said, that phase is not necessary for framework targets and, indeed, it is not present there.

I have tried the solution proposed by @davidharris and it has actually reduced running our tests significantly :confetti_ball: Unfortunately, I am not sure how to force Tuist not to generate the Embed Precompiled Framework phase for frameworks that are available in the current run search paths. @pepibumur would you have any insight into this?

I have tried the solution proposed by @davidharris and it has actually reduced running our tests significantly :confetti_ball: Unfortunately, I am not sure how to force Tuist not to generate the Embed Precompiled Framework phase for frameworks that are available in the current run search paths. @pepibumur would you have any insight into this?

Let’s put it differently. We don’t need to force Tuist to not do something. We should rather set up Tuist to do what we think is the right thing for the user. If we think the most sensible approach here for test targets that are associated with frameworks is resorting to the run search path, then let’s adjust the project generation to do that.

We should aim for the mindset of is Tuist doing the right thing?, instead of: how can I workaround Tuist’s behavior?

I have no experience with embedding frameworks whatsoever, but if you feel like it makes sense to change the current behavior and modify runpath search path rather than copy the frameworks, I can create a proper RFC on Github where we can discuss the specifics.

Another thing that might be going on here: how do you usually run your tests? We’ve got ~100 targets in our existing project, and we’ve found that creating an AllTests test scheme that just adds ALL of the underlying projects is way faster that iterating over all the tests and then running them. This seems to hold across build systems, as Airbnb did it with Buck: https://github.com/airbnb/BuckSample/pull/31

There are a few caveats here worth considering:

  • This doesn’t work for all frameworks for some reason. We have a map framework in our app that has to be embedded to matter what
  • To some extent, this can work already by adding the search paths flags to an xcconfig on your test, and then not add any of the dynamic frameworks linked by the actual target to the tests. However, if you depend on a prebuilt Carthage framework for example and that needs to be linked to your tests, Tuist will embed that…

Most project generation frameworks follow this behavior, and tutorials for things like Carthage do instruct you to do this. For single target apps its not a big deal, and for modules that are built and stored in the built products directory things work fine, its really the precompiled things it matters for. I think it’s worth maybe instructing Tuist users how to do this and make it available, but as a default it might be a bit complex.

Thanks @davidharris for more of your insights!

how do you usually run your tests?

This is how our tests are defined - I believe this is the same approach that you have proposed for AllTests scheme

We have a map framework in our app that has to be embedded to matter what

That’s weird as I would expect adding a search path to be as reliable as embedding a framework

To some extent, this can work already by adding the search paths flags to an xcconfig on your test, and then not add any of the dynamic frameworks linked by the actual target to the tests. However, if you depend on a prebuilt Carthage framework for example and that needs to be linked to your tests, Tuist will embed that…

Since I have to link the dynamic framework that is being tested (eg module Detail for DetailTests) and Tuist automatically that adds its dependencies, I am not sure how to achieve it?
Besides, I really think that this should be solved by Tuist out of the box since our codebase is by no means large and we are already having to resolve long build times. Moving to @rpath instead of embedding might be a nice way to achieve that. It’d be good to know what framework had to embedded (and why).

Yeah that’s what we’ve got right now (we’re not migrated to project generation). Is this how things were done prior to Tuist or with the generation?

Yeah, so did we. It’s a third party vendor and I never really identified why, but it’s the only one the runpath hack didn’t work with.

Are these precompiled dependencies or does it just does this for every dynamic framework? If they aren’t precompiled like Carthage, you shouldn’t need them embedded. This might be something that definitely is slowing things down.

Mostly just curious if you’ve identified that the embed step is what is really the cause of difference here. It shouldn’t be the compilation if it’s the same files, so I suppose it makes sense. What might be helpful is to generate the project, time the build, then clear cached build data, delete the embed frameworks steps and try again.

Is this how things were done prior to Tuist or with the generation?

With the generation.

Are these precompiled dependencies or does it just does this for every dynamic framework?

It only embeds precompiled Carthage frameworks that are needed (either direct dependencies of the framework being linked or transitive)

Mostly just curious if you’ve identified that the embed step is what is really the cause of difference here.

Yes, the time has been cut cca by half from 6 minutes to a little bit under 3 minutes just by changing the runpaths and removing the embed precompiled frameworks phase.

Is that 3 minutes basically the same as the build times you expected prior to generation? I didn’t expect this to be a 50% increase in time; we solely did it to save disk space.

One other thing worth trying might be to figure out if its the framework, dsym or bcsymbolmap copy that’s taking up most of the time. Seems as if Tuist is identifying these are Carthage frameworks and then trying to replicate some of carthage copy-frameworks which has similar issues.

I have created an issue for this: https://github.com/tuist/tuist/issues/1443