Mobile Automation Stories — XCUITest — 2

iOS Mobile Testing — Running Parallel Tests with scan in Fastlane — Running Parallel Tests with multi_scan in test_center plugin

If you are going to develop an automation project, one of the important things you need to consider is whether the infrastructure you will use in your project supports parallelism. Especially in mobile automation, parallelism is an indispensable part.

In this article, I will explain how to parallelize an XCUITest project easily and flexibly, and discuss the steps in which this method will be useful to you.

WHAT IS FASTLANE?

Fastlane is the easiest way to automate beta distribution and versioning of your iOS and Android applications. 🚀 It handles all the complex tasks such as generating screenshots, code signing, publishing your application, and running your tests.

Fastlane for iOS

Fastlane can be installed in multiple ways. We can install fastlane directly through Homebrew. Let’s perform the installations from the official website.

Setup - fastlane docs

PARALLELISM VIA COMMAND LINE

Running tests through a CI server is another important aspect to consider in an automation project, specifically in the context of this article. Running tests on a CI server will be facilitated by executing tests through the command line. With the following commands, we can run our tests in parallel without any tools.

xcodebuild 
  -workspace Example.xcworkspace 
  -scheme ExampleSchema 
  -configuration Debug 
  -destination 'platform=iOS Simulator,name=iPhone 12,OS=15.5' 
  -derivedDataPath './DerivedData' 
  -parallel-testing-enabled YES 
  -parallel-testing-worker-count 2 
  -only-testing:ExampleUITests/ExampleRegisterTests 
  -only-testing:ExampleUITests/ExampleLoginTests 
clean build 
test ONLY_ACTIVE_ARCH=NO CODE_SIGN_IDENTITY="" CODE_SIGNING_ALLOWED="NO"
test-without-building

PARALLELISM VIA COMMAND LINE WITH FASTLANE

However, we can make this command line easier and more flexible with fastlane.

1.Go to the fastlane directory created in the project file through installation.

cd fastlane/ 

2. Open the FastFile with any editor.

3. Set your device, project name, and other settings as follows:

  desc "Execute login and register tests on simulators"
  lane :parallel_simulator_login_register_test do
      run_tests(
        workspace: 'Example.xcworkspace',
        scheme: 'ExampleSchema', # Project scheme name
        #clean: true, # clean project folder before test execution
        devices: "iPhone 12",
        xcargs: 'ONLY_ACTIVE_ARCH=NO CODE_SIGN_IDENTITY="" CODE_SIGNING_ALLOWED="NO"',
        only_testing: 'ExampleUITests/ExampleRegisterTests',
        only_testing: 'ExampleUITests/ExampleLoginTests'
        parallel_testing: "YES",
        concurrent_workers: 3,
        max_concurrent_simulators: 3,
        slack_use_webhook_configured_username_and_icon: true,
        slack_username: 'slack_username',
        slack_url: 'slack_url',
        slack_icon_url: 'slack_icon',
        slack_channel: "#ios-automation-result",
        open_report: true,
        code_coverage:true,
        output_directory: "test_output/",
        #test_without_building: true,
        result_bundle: "TestResults"
      )
  end

You can learn the functionality of the above parameters from the following website.

scan

4. Run your test with the following command.

fastlane parallel_simulator_login_register_test

5. After waiting for your tests to run, you will see that it sends both an html file and the results to the channel you specified with slack. fastlane/test_output

CONVENIENT PARALLELLING WITH multi_scan

multi_scan It is a plug-in that we can use instead of scan to increase the usefulness of iOS test results, reduce test run time, periodically review partial results during a test run, and provide better result reporting.

Here some important plugins may be a reason to use scan command instead.

You can resolve the retry event by running tests that fail each time with try_count multiple times.

With batch_count when too many tests are assigned to a simulator for tests, we can run the tests in groups to avoid the situation that will cause many tests to fail because it will consume all the resources.

You can also reduce our test times by running 2 to 6 simulators with parallel_testrun_count .

With testrun_completed_block we can make adjustments to how our tests are executed. False can be set to cancel subsequent tests on the progress of tests. And with only_testing we can change which tests to run.

And apart from all these cases, it supports the parameters on scan.

With result_bundle output_files, you can provide better reporting of results, combining the results of parallel tests on both separate and the same xcresult and giving their html outputs.

You can have more detailed usage by examining the following document.

fastlane-plugin-test_center/multi_scan.md at master · lyndsey-ferguson/fastlane-plugin-test_center

  
UI.important(
      'example: ' \
      'split the tests into 4 batches and run each batch of tests in ' \
      'parallel up to 3 times if tests fail. Abort the testing early ' \
      'if there are too many failing tests by passing in a ' \
      ':testrun_completed_block that is called by :multi_scan ' \
      'after each run of tests.'
    )

test_run_block = lambda do |testrun_info|
    failed_test_count = testrun_info[:failed].size
    passed_test_count = testrun_info[:passing].size
    try_attempt = testrun_info[:try_count]
    batch = testrun_info[:batch]

    # UI.abort_with_message!('You could conditionally abort')
    UI.message("\n everything is fine, let's continue try #{try_attempt + 1} for batch #{batch}")
    {
      continue: true,
      only_testing: ['ExampleUITests/UITests/LoginTests/testLoginPage'],
      only_testing: ['ExampleUITests/UITests/RegisterTests/testRegisterPage']
    }
end
  
sim_callback = lambda do |simulator_device_udid|
  puts "Start streaming system log for device #{simulator_device_udid}"
end

override_scan_options_callback = lambda do |options|
  options.delete(:xctestrun) unless FastlaneCore::Helper.xcode_at_least?(10)
end

desc "Execute tests on simulators"
  lane :uitest do
    multi_scan(
        workspace: 'Example.xcworkspace',
        scheme: 'ExampleSchema', # Project scheme name
        try_count: 2,
        batch_count: 3,
        parallel_testrun_count: 3,
        result_bundle: true,
        fail_build: false,
        collate_reports: true,
        testrun_completed_block: test_run_block,
        simulator_started_callback: sim_callback,
        override_scan_options_block: override_scan_options_callback,
        xcargs: 'ONLY_ACTIVE_ARCH=NO CODE_SIGN_IDENTITY="" CODE_SIGNING_ALLOWED="NO"',
        #clean: true,
        devices: "iPhone 12",
        output_directory: "test_output/",
        open_report: true,
        code_coverage: true
    )
  end

Now we can run our test with the following command.

bundle exec fastlane

You will see in the command lines that your enabled tests have been configured as worker 1 — worker 2 and worker 3 and started running. 3 separate clone simulators have been formed and your tests have started to run in groups of 3 on them.

As output, you will see batch outputs and a combined test output in the test_output file.

multi_scan ile rapor dosyalama yöntemi

XCTestHTMLReport ile detaylı raporlama — 1

XCTestHTMLReport ile detaylı raporlama — 2

XCTestHTMLReport ile detaylı raporlama — 3

XCPretty ile detaylı raporlama

Conclusion

In this article, we have seen how to run our tests with a fastlane integration, how to separate specific desired tests on fastlane and how to run our tests in groups in parallel. In addition, we were able to examine our tests with 2 different reports. In our next articles, we will touch on other issues that are important as infrastructure.

Feedback 📬

If you have patiently followed me up to this point and you think we could do it better this way, here is the reason for you to contact me! I am ready to listen to your advice, criticism and feedback that can improve me or you. Contact Us. Thanks…