Table Of Contents

• Introduction
• Notable Files
• Getting Started On Local Machine
• Getting Started On Azure DevOps
• Explanation Of The Journey: Deadends, Pitfalls, Solutions
  • Must Use MacOS Agent
    • MacOS Agent Pitfalls
    • MacOS Agent Directories
  • Fresh Autogenerated Pipeline
  • General Pipeline YAML Advice
    • Pipeline Variables
    • Pipeline Triggers
    • Pipeline Scripts And Strings
  • Give Each Build An Increasing Android App Version
    • How To Choose The Version
    • Build The APK File
  • Sign The APK File
    • Keystore Background
    • AndroidSigning Task
  • Publish The APK Files As Build Artifacts
  • Build And Run Unit Tests
  • UI Tests In Azure DevOps
    • Set Up And Start Android Emulator
    • Build UI Tests
    • Run Emulator UI Tests
    • Publish Emulator UI Tests
  • UI Tests In App Center
    • Set Up App Center
    • Experiment With appcenter CLI
    • Run App Center UI Tests
    • Publish App Center UI Tests In Azure DevOps
    • Problems Accessing Stuff
• Thanks To Those Who Helped Me

Introduction

I'm making this demo repo and writeup because it was surprisingly and frustratingly difficult to get Xamarin.UITest tests for Android to run on a Microsoft-hosted agent in an Azure DevOps pipeline. NO App Center. NO self-hosted agents. I just wanted to do everything in Azure DevOps.

(Once I got that to work, I did add in App Center UI testing...which was also surprisingly difficult, so hopefully this demo is helpful for that as well.)

This demo has grown into showing how to accomplish quite a few common goals for an Azure Devops continuous integration pipeline for the Android portion of a Xamarin app...

• Each build gets its own versionCode and versionName.
• Build the APK.
• Sign the APK.
• Publish the APK as a pipeline artifact.
• Do unit tests (NUnit).
• Do UI tests (Xamarin.UITest) in Azure DevOps, which involves several Android emulator steps.
• Do UI tests in App Center.
• Publish all test results (including device-labeled App Center test results in Azure DevOps test explorer).

This demo is not about getting started on unit testing or UI testing; the demo is about getting these things to work in an Azure DevOps pipeline.

You can see a successful run, a successful job overview, published artifacts, and unit+UI test results.

This repo is available as a visualstudio.com repo and a github repo. As of 2020-Dec-24, Azure DevOps offers a free tier with 30 build hours per month and 2 GiB of artifact storage. The free tier was more than enough for all the pipeline needs of this demo.

This writeup is available as a github readme, visualstudio.com readme, and blog post. The repo readmes will be kept up to date, but the blog post may not receive many updates after 2020-12-24. Readme section links are oriented for GitHub.

Notable Files

The XamarinPipelineDemo.Android/AzureDevOps/ folder has most of the notable files...

• pipeline-android.yml: the pipeline definition and heart of this demo.
• AndroidSetVersion.ps1: the script that manipulates the Android manifest file to update the versionName and versionCode attributes.
• example.keystore: for signing the APK. Normally keystore files are sensitive and you wouldn't put them (and their passwords) in your repo, but this is a demo.

XamarinPipelineDemo.UITest/AppInitializer.cs: the autogenerated AppInitializer.cs has been modified so that you can specify which APK file to install for testing, or which keystore to match an already installed (signed) APK. I suggest the APK file methodology.

local_uitest_run.ps1: script to run UITest tests (on a local Android device or emulator) in way most similar to how the pipeline will do it.

appcenter_uitest_run.ps1: script to run UITest tests remotely via App Center. You'll need to set up your own App Center account (including app and device set) and modify the script to use that account.

Screenshots folder has some screenshots of the results of a working pipeline run, and some of the web interface you need to tangle with to get the pipeline working.

Getting Started On Local Machine

First, check that it works on your machine. Open the solution in Visual Studio 2019, and deploy the Release build to an Android emulator or connected Android device (just select Release build configuration and launch the debugger). The app should show you a page with a label that says "Some text.".

In Visual Studio's test explorer, run both the Nunit and UITest tests. Everything should pass.

Also, to run the UITest tests in the way most similar to how the pipeline will do it, install a recent stable nunit3-console release, go into the LocalScripts folder and run local_uitest_run.ps1. You'll get a test results file TestResult.xml and a detailed log uitest.log that is useful for troubleshooting. The script tries to use adb and msbuild from your PATH environment variable and a few other locations. You might have to add your adb or msbuild directories to your PATH. Also, you might have to set the ANDROID_HOME environment variable to something like C:\Program Files (x86)\Android\android-sdk and the JAVA_HOME environment variable to something like C:\Program Files\Android\jdk\microsoft_dist_openjdk_1.8.0.25

Getting Started On Azure DevOps

In the Pipelines => Library section of your Azure DevOps project, you need to do a few things.

You need to set up the keystore file and variables...

• Upload example.keystore as a secure file.
• Create a variable group named android_demo_var_group. In it, create the following variables...
  • androidKeystoreSecureFileName: example.keystore
  • androidKeyAlias: androiddebugkey
  • androidKeystorePassword: android
  • androidKeyPassword: android
• Make the androidKeystorePassword and androidKeyPassword secret by clicking the padlock icon.

You need to create a pipeline from the yaml pipeline definition file...

• Upload the repo to Azure DevOps.
• Create a new pipeline.
• When asked "where is your code?", choose "Azure Repos Git".
• Select the XamarinPipelineDemo repo.
• Select "Existing Azure Pipelines YAML file".
• Select the XamarinPipelineDemo/XamarinPipelineDemo.Android/AzureDevOps/pipeline-android.yml as the existing yaml file.

Run the pipeline and you'll have to click a few things to give the pipeline access to these secure files and secret variables. To grant permission to the pipeline, you might have to go to the run summary page (see Screenshots folder).

Explanation Of The Journey: Deadends, Pitfalls, Solutions

Note that things may change after this demo was made (2020-12-19). Some limitations may go away, and some workaround no longer needed. I'd love to hear about them if you ever encounter a way this demo should be updated.

Must Use MacOS Agent

First of all, doing Xamarin.UITest tests on Microsoft-hosted agent in an Azure DevOps pipeline has some important constraints. Microsoft-hosted agents for Window and Linux run in a virtual machine that can not run the Android emulator, so only the MacOS agent can run the Android emulator (MS docs page).

With a self-hosted agent that is not a virtual machine, you can use any of the three OSes. With App Center, the tests are run on a real device, and there is no need to run the Android emulator, so again, you can