AutomationTestSupervisor

AutomationTestSupervisor is a Python tool that is capable of creating, running Android Virtual Devices, building and installing .apk files on devices, running tests and generating logs. It’s strongest point is customizability as it allows you to easily manage which tests should be ran in session, devices to be used, paths on machine to be read and more. It allows to create run profiles which are set once and can be reused after.

Story behind AutomationTestSupervisor — our custom made tool for Android automation tests

Example Project

We are aware of that ATS has many functions and features. Some people prefer to see code rather then read blocks of text. That's why we have prepared example Android application - AzimoNote and wrote few Espresso tests so you can pull it and launch ATS by yourself.

In this example ATS is joined with AzimoNote in form of submodule. We use it the same way in Azimo project. It's quite convenient way as you can easily delegate ATS work to Jenkins or Gitlab. It is being downloaded along with Azimo repository and it's already pre-configured. You only need one CI command to launch it.

We have provided Quick Launch for you to follow.

Link to AzimoNote project

Quick Launch

Meeting minimum launch requirements step by step:

Make sure you have Python 3.6 or newer installed and available from terminal. (If you are MAC OS user we highly recommend Homebrew -> brew install python3.)
Clone AutomationTestSupervisor project to you machine from https://github.com/AzimoLabs/AutomationTestSupervisor.
Navigate to AutomationTestSupervisor folder on your machine.
Launch ATS with following command:

python3 Launcher.py

Launch command parameters

It is possible to add four parameters to ATS launch command:

pset < name of Path Module set >
lplan < name of Launch Module launch plan >
aset < name of AVD Module set >
tset < name of Test Module set >

Parameters can be added in any order you like. If you omit some parameter ATS will try to look for set/plan with name "default" in manifest JSON files.

In example:

python3 Launcher.py -tset deeplinkTests -aset 3AVD-API23 -pset JenkinsPaths

Module configurations

AutomationTestSupervisor can be divided into four main modules where each of them has it’s own config manifest file. Modules are integrated which each other and forming linear pipeline with set of steps. By modifying manifest of each module you can adjust behaviour of each step.

We wanted to make ATS project as self-contained as possible. That means we didn't wanted to force you to edit anything in the project. General idea is to just pull it and provide your own config files in right directory. In root of ATS project there is config_files_dir.json file which looks like this:

{
  "path_manifest_path": "../automationTestSupervisorConfig/pathManifest.json",
  "launch_manifest_path": "../automationTestSupervisorConfig/launchManifest.json",
  "test_manifest_path": "../automationTestSupervisorConfig/testManifest.json",
  "avd_manifest_path": "../automationTestSupervisorConfig/avdManifest.json"
}

It points for directory where you should put config .json files -> This is how it looks in test project.

Templates for .json manifest files are located in AutomationTestSupervisor project itself.

Path Module

File pathManifest.json:


{
  "path_set_list": [
    {
      "set_name": "default",
      "paths": [
        {
          "path_name": "sdk_dir",
          "path_value": "",
          "path_description": "Path to Android Software Development kit. By default stored in ANDROID_HOME env."
        },
        {
          "path_name": "avd_dir",
          "path_value": "",
          "path_description": "Location to .android folder where AVD images are stored. By default stored in ANDROID_SDK_HOME env."
        },
        {
          "path_name": "output_dir",
          "path_value": "",
          "path_description": "Path to where test process logs will be saved. By default ./output/."
        },
        {
          "path_name": "project_root_dir",
          "path_value": "",
          "path_description": "Path to root of application under test. (optional)"
        },
        {
          "path_name": "apk_dir",
          "path_value": "",
          "path_description": "Directory where .apk files of application under test are stored."
        }
      ]
    }
  ]
}

Overview:

You can provide both relative and absolute paths. ATS will convert relative paths to absolute paths anyway.
You can provide many set of paths (for different machines/developers) by adding another object path_set_list and giving it unique set_name.

Parameters (explanations in path_description):

sdk_dir
avd_dir
output_dir
project_root_dir - If you want to allow ATS to build .apk files of your project then you need to provide path to it so ATS can access ./gradlew file.
apk_dir - Directory where ATS will look for .apk files. If it won't find them and project_root_dir is properly set then there is a possibility to build them.

Launch Module

File: launchManifest.json

{
  "launch_plan_list": [
    {
      "plan_name": "default",
      "general": {
        "adb_call_buffer_size": 3,
        "adb_call_buffer_delay_between_cmd": 2000
      },
      "device_preparation_phase": {
        "avd_should_recreate_existing": true
      },
      "apk_preparation_phase": {
        "build_new_apk": true
      },
      "device_launching_phase": {
        "device_android_id_to_ignore": [],
        "avd_launch_sequentially": true,
        "avd_status_scan_interval_millis": 10000,
        "avd_wait_for_adb_boot_timeout_millis": 240000,
        "avd_wait_for_system_boot_timeout_millis": 120000,
        "device_before_launching_restart_adb": true
      },
      "testing_phase": {
        "record_tests": true
      }
    }
  ]
}

Overview:

You can create many launch plans by adding another object launch_plan_list and giving it unique plan_name.
Check how it's done in AzimoNote project -> link

Parameters:

ADB Buffer - as ADB has problem with too frequent calls to it and tends to process information too slow and return errors. Unfortunately it's limitation of ADB and all you can do is to wait for some major improvement/update. So to avoid spamming it, every command to ADB is entering buffer of size adb_call_buffer_size and then it's released after adb_call_buffer_delay_between_cmd milliseconds. This will allow you to reduce burden on your ADB in case you used weaker machine or decided to run tons of AVD.
avd_should_recreate_existing - Boolean value, if AVD that you have picked for your session already exists on your machine you can either choose to leave them as they are or ask ATS to re-create them (to completely reset memory state of device for example)
build_new_apk - If ATS won't find your .apk in directory specified in PathModule it will attempt to build it if you wish for that. In some cases you are debugging your tests and try to run it many times to trigger some non-deterministic action. That's why you don't want to wait for .apk being built from scratch every time if you are not making any changes to your code. If you are 100% sure .apk is already built you can use this flag to create "speed-run" launch plan.
device_android_id_to_ignore - Simply enter Android Device Id that should be ignored.
avd_launch_sequentially - After you've requested list of AVD to launch you can either wait for their boot in parallel or "one by one". Though non-sequential launch is dangerous for your OS. What's limiting how many AVD can be launched at the same time is HAXM. If you run 10 AVD in exactly same moment (where each takes ~1GB of RAM) - even though limit of your HAXM might be 6GB of RAM - SDK won't stop you from doing that. It will perform 10 checks and each of them will say - there is 6GB of free RAM, go ahead. That way you can launch more AVD - over the limit of your HAXM and in worst case you will crash your OS with out of memory error.
avd_status_scan_interval_millis - When AVD is booting and ATS is waiting for it this parameter allows you to set interval (millis) in which ADB will check device state.
avd_wait_for_adb_boot_timeout_millis - Timeout for so called ADB BOOT. It's time from launching terminal command that should start AVD to moment where ADB will display AVD in adb list with status device. At that point system of AVD just started booting.
avd_wait_for_system_boot_timeout_millis - Wait for system parameters of AVD: dev.bootcomplete, sys.boot_completed, init.svc.bootanim.
device_before_launching_restart_adb - You can choose if ADB should be restarted before it starts working with AVD.
record_tests - You can choose if every test should be recorded. It will put more burden on your machine.

AVD Module

File: avdManifest.json

{
  "avd_schema_list": [
    {
      "avd_name": "",
      "create_avd_package": "",
      "create_device": "",
      "create_avd_tag": "",
      "create_avd_abi": "",
      "create_avd_additional_options": "",
      "create_avd_hardware_config_filepath": "",
      "launch_avd_snapshot_filepath": "",
      "launch_avd_launch_binary_name": "",
      "launch_avd_additional_options": ""
    }
  ],
  "avd_set_list": [
    {
      "set_name": "default",
      "avd_list

AutomationTestSupervisor

Install / Use

README

AutomationTestSupervisor

Related Articles

Example Project

Quick Launch

Launch command parameters

Module configurations

Path Module

Launch Module

AVD Module