Just run those tests in minutes

The Challange

I want to run UI tests on Bitrise CI asap. I have a sample test suite or I’m migrating from another CI. Perhaps I want to customize the workflow a bit to run custom annotation that scopes the test suite or sets device config, etc.

Who would benefit?

DevOps, Automation Engineer, Android Dev, Tech Lead.

Assumptions

I’m assuming you’re a beginner to intermediate Bitrise user, or you’re a skilled professional looking for a shortcut to just run your test suite asap. What I’m providing here is supposed to be a blueprint for you to jumpstart running your tests on Bitrise CI.

This article will cover the Mobile DevOps side of how to kick off UI testing on Bitrise for Android. Code included. I’ll also share a couple of good practices for UI automation configuration.

I’m assuming you want a quasi-infrastructure-as-a-code approach, meaning you are going to use bitrise.yaml instead of Bitrise web console. In fact, please use bitrise.yaml + version control system to maintain control over changes and learn more.

I’m assuming you have configured the auth mechanism between your repository and Bitrise, meaning you can clone your repo while running Bitrise builds.

I’m assuming you know where to find the most recent bitrise.yaml version. BTW See Workflow Editor -> bitrise.yaml tab and download it.

The Firebase Device Lab will be used to run the automation suite. The default account is linked with your Bitrise instance — it’s recommended to provide your own account.

If you would like to learn in-depth Bitrise concepts and how to navigate and use it in an optimized way, I would recommend you read my article on being effective with Bitrise CI, available on this site and under the link on medium.com. These two articles will share a few tips for sure.

The Blueprint

  runTests:
    steps:
      - git-clone: { }
      - cache-pull@2: { }
      - install-missing-android-tools@3: { }
      - android-sdk-update@1: { }
      - android-build-for-ui-testing@0:
          inputs:
            - variant: debug
            - module: app
      - virtual-device-testing-for-android@1:
          is_skippable: true
          inputs:
            - test_type: instrumentation
            - test_devices: "Pixel2,28,en_US,portrait"
            - inst_test_targets: "annotation com.example.annotation.SMOKE"
            - inst_use_orchestrator: 'true'
            - test_timeout: 3600
      - custom-test-results-export@0.1:
          inputs:
            - test_name: "*"
            - search_pattern: "*"
      - cache-push@2: { }

Just basic configs, nothing to see here, really. runTest steps live alongside other steps.

  runTests: #name of the workflow
    steps:
      - git-clone: { } # clones repo, more config may be required, out of scope
      - cache-pull@2: { } # gets prev cached artifacts, faster build
      - install-missing-android-tools@3: { } # update
      - android-sdk-update@1: { } # update
      # this is where testing happens
      - custom-test-results-export@0.1: # Test reports. You want them.
          inputs:
            - test_name: "*"
            - search_pattern: "*"
      - cache-push@2: { } # push artifacts
# Optionally you may like Slack notification

As said before, no magic happens here. Just housekeeping like cloning the repository and being sure things are up to date. Keep an eye on reports - those are useful to track what’s exactly wrong with your test including screenshots, logs, state, execution time, etc.

Ok, let’s explain the UI tests running part.

runTests:
    steps:
      # Test builds
      - android-build-for-ui-testing@0:
          inputs:
            - variant: debug
            - module: app
      # Testing
      - virtual-device-testing-for-android@1:
          is_skippable: true
          inputs:
            - test_type: instrumentation
            - test_devices: "Pixel2,28,en_US,portrait"
            - inst_test_targets: "annotation com.example.annotation.SMOKE"
            - inst_use_orchestrator: 'true'
            - test_timeout: 3600

How test builds are being prepared?

# Test builds creation
      - android-build-for-ui-testing@0:
          inputs:
            - variant: debug
            - module: app

No magic here again. Both your APK and testing APK will be built as a part of these steps. Bitrise will figure out the paths to APKs itself. If you want to see the guide — here is the official one.

The UI testing steps

- virtual-device-testing-for-android@1:
          is_skippable: true
          inputs:
            - test_type: instrumentation
            - test_devices: "Pixel2,28,en_US,portrait" #example
            - inst_test_targets: "annotation com.example.annotation.SMOKE" #OPTIONAL
            - inst_use_orchestrator: 'true'
            - test_timeout: 900

Step by step

is_skippable: true # not required

You may want to have a green build even though the UI test failed because some of the tests are flaky and brittle. Maybe you just care about configuration (for now) or maybe you’re running an experiment. It’s useful. Use it whenever necessary.

- test_type: instrumentation # required, other: game loop, robo

In most cases, Android UI tests are instrumentation tests. Ask Automation Engineer if that’s the truth for you.

# required, schema deviceID,version,language,orientation
- test_devices: "Pixel2,28,en_US,portrait"

You have to provide at least one device, you can provide many. Separated by comma.

You can find config details and a list of devices on the official Bitrise Github step codebase.

- inst_test_targets: "annotation com.example.annotation.SMOKE" # optional

You can define test targets — if not provided, every test in androidTest package will be triggered. You’re required to provide a fully qualified class or a package.

You can also configure your test targets via annotations. Annotations give you the opportunity to scope out what test to run by grouping them. It’s up to you what, how, and when to group which gives you the opportunity to adapt to the current context (release, feature branch or smoke testing?) — it’s all up to you and can be done both manually and dynamically. A short and sweet article on annotation is found here.

- inst_use_orchestrator: 'true' # optional, default false

This listing defines if tests are using Android Test Orchestrator or not. Default: false. Ask your Automation Engineer.

- test_timeout: 900 # optional, default 900=15m

A question: How much time must pass before a test is canceled? Default 900 = 15m. I would assume it should be much quicker for an efficient automation suite.

But there is obviously much more, like

- num_flaky_test_attempts: 0

And more, and more, and more.

You can compare what I have shown you with the official documentation on Github. In this case for the UI testing step. It’s way better than website documentation.

We’re finally at the end. Now, you can use provided a blueprint and run your test suite in minutes via Bitrise. Happy coding!

I hope you like this piece I love a fast feedback loop. If you have any objections, comments, or questions — please drop a comment or DM message.