Azure DevOps Backup

<p align="center"> <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool"><img src="https://img.shields.io/github/languages/top/michaelmsonne/AzureDevOpsBackupTool.svg"></a> <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool"><img src="https://img.shields.io/github/languages/code-size/michaelmsonne/AzureDevOpsBackupTool.svg"></a> <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool"><img src="https://img.shields.io/github/downloads/michaelmsonne/AzureDevOpsBackupTool/total.svg"></a><br> <a href="https://www.buymeacoffee.com/sonnes" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 30px !important;width: 117px !important;"></a> </p> <div align="center"> <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool/issues/new?assignees=&labels=bug&template=01_BUG_REPORT.md&title=bug%3A+">Report a Bug</a> · <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool/issues/new?assignees=&labels=enhancement&template=02_FEATURE_REQUEST.md&title=feat%3A+">Request a Feature</a> . <a href="https://github.com/michaelmsonne/AzureDevOpsBackupTool/discussions">Ask a Question</a> </div> <div align="center"> <br /> </div>

Table of Contents

• Introduction
• Contents
• Features
• Download
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Examples
• Contributing
• Support
• License

Introduction

Azure DevOps is a cloud service to manage source code and collaborate between development teams. It integrates perfectly with both Visual Studio and Visual Studio Code and other IDE´s and tools there is using the "Git". While your code is perfectly safe on the Azure infrastructure hosted by Microsoft, there are cases where a centralized local backup of all projects and repositories is needed. These might include Corporate Policies, Disaster Recovery and Business Continuity Plans. We can download a repository from Azure DevOps as a Zip file, but this may not be practical if we have a considerable amount of projects and repositories and need them backed up on a regular basis. To do this, we can use the Azure DevOps API, as explained here: https://learn.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-7.1 💪

Contents

Outline the file contents of the repository. It helps users navigate the codebase, build configuration and any related assets.

| File/folder | Description | |------------------------------|---------------------------------------------| | AdvancedInstaller | Code for installer project | | AzureDevOpsBackup | Source code for the main backup tool itself.| | AzureDevOpsBackupUnzipTool | Source code for the unzip tool itself. | | docs | Documents/pictures. | | .gitignore | Define what to ignore at commit time. | | CHANGELOG.md | List of changes to the sample. | | CONTRIBUTING.md | Guidelines for contributing to the AzureDevOpsBackupTool.| | README.md | This README file. | | SECURITY.md | Security file. | | LICENSE | The license for the AzureDevOpsBackupTool. |

Features

AzureDevOpsBackup:

Overall:

• Asynchronous Resolution: Utilizes asynchronous processing for improved performance and responsiveness, allowing users to continue working while backups are being created.
• Simplicity and Ease of Use: Provides a straightforward and user-friendly method for creating backups from Azure DevOps repositories.

List:

• Backup Functionality:

  • Repository backup: Enables users to create local backups of Azure DevOps repositories.
  • Customizable backup Options: Offers various command-line options to tailor the backup process, including specifying backup directories, token authentication, cleanup, backup retention period, and more.
  • Zip file handling: Ability to download repository items as ZIP files and organize them locally, optionally extracting them into a directory structure using the --unzip argument.
  • Automated cleanup: Supports cleanup functionality to manage the disk space occupied by the backup files.
  • Backup Retention: Allows users to specify the number of days to retain backups, automatically removing older backups based on the set retention period using the --daystokeepbackup argument.

• Email Reporting:

  • Email notifications: Sends detailed email reports upon backup completion, providing status updates, backup job information, and relevant details.
  • Email customization: Offers different email layouts (--simpelreportlayout) to cater to varying preferences or requirements.
  • Priority configuration: Provides the option to set email priority (--priority) for the backup completion notification.

• Logging:

  • Job logging: Stores logs for backup jobs in a designated folder (.\Log) for a defined period (default: 30 days) beside the AzureDevOpsBackup.exe executable.

AzureDevOpsBackup unzip tool:

Overall:

• Unzip functionality: allows users to extract ZIP files and JSON metadata into a directory structure, renaming GUIDs to file or folder names for a specific repository backup.
• Simplicity and ease of use: Provides a straightforward and user-friendly method for unzipping repository backups from Azure DevOps .zip files.

Download

Download the latest version

Version History

Getting Started

1. Downloaded the installer from releases to any location on a Windows machine where you want to use the tool
2. Run the downloaded installer and select where you will install the tool
3. When installed you need an access token to connect to Azure DevOps (read the guide under here)
4. Then run the main AzureDevOpsBackup.exe with the appropriate command line options to connect to your Azure DevOps and download the backup use with command line options in CMD, PowerShell or Windows Terminal!
5. You now have an local backup!

How to get a local backup of your repos on your backup/home server? 🤔

1. Get an Azure DevOps API personal access token

On the top right corner of the Azure DevOps portal we have our account picture. When clicking on it reveals the account menu where we find a Security option. Inside Security, we have Personal access tokens. We click on New token to create one. For this example we only need to check the Read, write & manage permission for Code. When we name the token and click Create, we get the token value, but since it won’t be shown again, we must copy and save it elsewhere. We will need this token later, to access the API!

2. Clone the repository, configure arguments and dependencies

To get the sample code, we can clone this GitHub repository or another and open it with etc. Visual Studio. To test the solution in debug mode, we still have to configure a few arguments in the project properties. These arguments will define the access token — obtained on the previous step — , organization — i.e. the Azure DevOps domain — and the local directory where we want to write the backed up data. There is a fourth optional argument to state if we want to unzip the downloaded files — more on that later.

N.B: Keep your generated access token in a safe place!

Here’s how the argument list will look like:

.\AzureDevOpsBackup.exe --token "our-auth-token" --org "our-org" --backup "C:\backup\out-directory" --server "smtp.server.com" --port "25" --from "azure-devops-backup@domain.com" --to "backupmail@domain.com" --unzip --cleanup --daystokeepbackup 50 --priority high

This solution uses two external libraries we need to reference: RestSharp to facilitate the API calls and Newtonsoft JSON to deserialize the API responses into objects.

3. Analyze and run the code

We start by declaring the data structures for the API responses. Then, the main function cycles through four nested levels of hierarchy — project / repository / branch / item — calling these API endpoints and deserializing the results into the corresponding structures. Hence, for each project, we get a list of the repositories it contains and for each repository, we get a list of the items it contains.

When we get to the repository level we don’t need to make individual API calls to download every single item on the repository. Instead, we can download all the items, packed into a Zip file with a single API call. In order to do this, we still need the item list, because we have to build a JSON structure containing every item on the list where the property gitObjectType equals “blob”. This JSON structure will then be posted to the /blobs endpoint to obtain the Zip file as a response.

Note we are also saving the original JSON item list we got from the repository call. This will be needed to map the files inside the Zip package, because these are presented in a single flat directory and their names are the object ids and not their actual names and extension. This is where the --unzip argument enters. If it’s omitted, the process does not go further and we get a simple backup: for every repository of each project we get a Zip file with the content and a JSON file to map it.

Usage

AzureDevOpsBackup:

Paramenters:

Backup:

• --token
  • token.bin: Use an encrypted .bin file (based on hardware ID´s) with your personal access token in. (Remember to run --tokenfile <token.data> to create the file first beside the application .exe!)
  • <token.data>: Replace this with your Azure DevOps personal access token.
• --org: Replace this with your Azure DevOps organization nam