XCode Parallel Testing — From Hours to Minutes
XCode Parallel Testing is a great option to save time on repetious tasks. iOS XCUITest is Apple’s native test automation framework for iOS/macOS/Apple Watch/AppleTV that helps to instrument UI interactions. This framework was released in 2015 during WWDC and has been evolving since then. It quickly became choice #1 due to its reliability and the fact that the XCUIText is intuitive and fast.
There are numerous articles written about the fundamentals of XCUITest and how to get started from scratch. Thus we will shift our focus towards the scalability of XCUITests execution.
In this article, Engenious will talk more about how to leverage XCUITest for CI — reliably running tests in parallel to qualify with other CI checks such as linter and unit tests.
What is XCUITest?
As we’ve mentioned above, XCUITest is a native test automation framework for iOS thus XCUITest doesn’t need you to download it, it ships as part of Xcode and this makes it extra comfortable to use this framework out of the box. Since it is an extension of XCTest — the unit test framework used by developers for a long time, you would get devs on your side pretty quickly. Your test codebase will be part of the app under test codebase and thus your Pull Requests could be reviewed by experienced iOS devs. If necessary, help will be always around the corner. Adding unique accessibility identifiers to UI elements to make tests resilient could be also very challenging. That is where pairing with iOS devs could be super helpful as well. Ultimately, living in the same codebase will get you a step closer to app development and will make you more proficient as an iOS dev in the long run.
As for cons, there aren’t lots of them. Mostly, the biggest downside of the XCUITest framework is that you can only use it with SWIFT and no other language. This also means that cross-platform testing won’t be available. It’s a native iOS framework at the end of the day.
How to run XCUITest?
As we’ve stated, XCUITest is a part of XCode, so to use it you have to set up Xcode first. Once you’ve dealt with that, you need to build a project for testing. Here’s a command you can use to generate a built that contains .xctestrun and other necessary files you need for the test execution.
Xcode build command:
-scheme Scheme \
-sdk iphoneos14.4 \
-destination ‘id=Device or Simulator UDID’ \
The command to run tests in parallel will be revealed in the next section.
Running Tests in parallel using XCode
UITests are much slower than unit tests when you run 100-200 tests sequentially, you will see that it takes 2-3 hours to run. Running tests in parallel might solve this problem.
With the 10th version of XCode, parallel testing has become possible with this framework. Now, with the XCode 12, this opportunity to speed up things is still there and you can use it for large projects to test various elements of your app at the same time.
There is also a possibility to use parallel distribution testing with this version, which allows adding more devices to the process. This may speed up your working processes by 30% and bring the release even closer without any losses in the quality of the tests themselves.
To run native parallel testing you may use the following command, with multiple destination devices:
-xctestrun /path/to/project_iphoneos13.4-arm64.xctestrun \
-destination ‘id=Device or Simulator UDID 1 \
-destination ‘id=Device or Simulator UDID 2 \
Test Time Imbalance and other issues with XCUI Testing parallelization
Native Xcode parallelization doesn’t support UITests for Mac OS and iPhone devices, it can be performed only in iOS simulators within the one Mac Machine. It’s very limited and is not scalable. The limitations produce flakiness and make the XCUI parallel testing unapplicable for enterprises.
Here are some of the issues you may encounter with XCUI parallel testing:
- Environment overloading – since simulators consume a lot of memory, a single MacPC can’t handle the parallel testing on 3 and more devices. This issue is hard to fix if you’re limited in finances;
- Flaky tests – the failure of a test will also ruin your green CI. The native solution won’t let you rerun the test and so this problem becomes a real struggle. The smooth test run might be interrupted by a simulator freeze, network failures, or even a sudden notification;
- Speed – the following issues might be bottlenecks since they slow down the parallel testing.
- Rerun tests – there is no way to handle test flakiness such as rerun quotes. This can save time for test failure investigation while processing testrun reports.
The remedy for the above-stated issues could be found outside of native XCode in an open-source library called “SIFT”. SIFT is indeed a scalable solution that enables running tests in parallel across multiple Mac machines while supporting MacOS, iPhone Devices, and Simulators.
SIFT – the open-source solution for XCUITest Sharding
SIFT is an open-source project for parallel Unit and UI Testing. It is aimed at speeding up the process and can support multiple devices as we’ve stated above.
Under the hood SIFT iOS uses the same XCodebuild commands so it has the same possibilities that XCodebuild has. But it doesn’t share the XCode bottlenecks since it has many pros that will help you test with the speed of light.
Here are some of the qualities that determine the value of SIFT project:
- Connect unlimited nodes – SIFT can execute tests on a few nodes, more so it was designed to do so. Thanks to this, the hardware issue of XCUI parallel testing can be solved with ease;
- Parallel execution – this makes the time fly. Every node runs tests in parallel, this way you can reduce 2-hour regression to 15 minutes with 4 nodes (2 devices on each). Output – SIFT supports different files of output like JUnit, Xcresult, XML. The additional processing isn’t needed.
- CI execution – it is okay for small companies to have test runs once before a release. But if you have a huge organization with hundreds of people working on applications – every minute is important. So SIFT can come in handy when you have to test your app on pre-merge or run the regression for every developing cycle.
- Rerun tests – now your test has a second chance to pass. It is possible to set rerun quotes for tests that may be handy when in complicated cases that require a lot of network connections or data loading.
And, if you wish to use the SIFT command to run from the console, use this:
sift run –config config.json – to run tests
sift run –config config.json –only-testing “DemoTests/DemoAppTests/testOne()” – to run specific tests
sift run –config config.json –tests-path tests.json – to run tests from file
sift list –config config.json – to print all tests from the bundle
The JSON configuration file you will find on our Github page. You’ll need it if you’re going to use SIFT without an Orchestrator.
Now let’s take a look at SIFT at practice; this example will let you better understand the principles of this tool’s work.
Example of XCode Parallel Testing
Let’s assume that we have a complicated project that combines features like login, auth, chat, locations, profile management, Facebook, Instagram integrations. There are around 200 UI tests and 2000 Unit tests for this project.
The average Unit test run takes 0.5 sec. So for the whole Unit test regression, we have to wait 2000 * 0.5 / 60 = 16.6 min. That’s not a huge number, plus unit tests are independent and have high stability, so no re-runs are required.
In the case of UI Tests situation is more difficult. Average UI Test run takes 1 min. To run all of them we need 400 minutes or 400 / 60 = 6.6 hours. This seems like a lot of time, and we need to add 10% to this number because UI Tests usually are not stable and more independent than Unit tests. To determine if the test is flaky we can rerun it. If it fails 2-3 times in a row then we can assume that test is failing.
So total time for UI Regression will be 6.6 * 1.1 = 7.26 hours.
To lower this number we can use default XCUITest parallelization, but its possibilities are quite limited. As we know 1 machine can handle up to 2 devices. With more tests become flakier.
With SIFT we can connect unlimited number of nodes. Let’s take for example 5 nodes and run 2 simulators or devices on each node.
Then our UI Test regression will take 7.26 / (5 * 2) = 0.726 hours or ~45 minutes total.
So, let’s wrap it up with a list of benefits you’ll get by using SIFT iOS.
- Connect unlimited nodes – Sift is designed to execute tests on multiple nodes. This feature resolves hardware issues.
- Parallel execution – every node is running tests in parallel, so 2-hour regression can be reduced to 15 minutes only with 4 nodes with 2 devices on each.
- Output – Different types of output files are supported by SIFT – JUnit, Xcresult, Xml.
- Orchestrator integration – SIFT is easily integrated with Orchestrator where you can manage your test suites, test statuses and observe test statistics.
Hopefully, this article helped you find the new and exciting solution for making iOS parallel testing quicker and less flaky.
These are not all the abilities of SIFT iOS. You can learn more about this technology by watching demo or getting on a consultation with our specialists. To contact please visit contact-us or drop us a message to our social media – Facebook, LinkedIn, Instagram or