AzureHPC

Overview

This project is aimed at simplifiying deployment and setup for HPC environments in Azure. The deployment scripts include setting up various building blocks available for Networking, Compute and Storage that are needed for an e2e setup. You can do this all with a few commands and fast as these are run in parallel.

They key motivation is:

• Simplified Automation
• Flexibility
• Speed of deployment

The basis for the project is a single JSON config file and some shell scripts for installing. The key point about the config file is that you can describe network, resources and installation steps. Tags are applied to resources that determines which resources run each install step.

The azhpc_* scripts only require the azure cli and a few utilities (bash, jq and ssh).

All of this is available in the Cloud Shell. Alternatively you can run on a Linux VM on Azure or from the Windows Ubuntu Shell.

Multiple examples for building blocks commonly used, scripts for building, installing and running some applications are included here so they can be used as you build your environment and run benchmarks.

We have also made some tutorials available that you can follow to not only learn more about the framework but also to understand how you can easily set an environment up e2e for your own application.

JSON configuration file

The JSON file is composed of the following:

• Variables dictionary
• Setup information
• Network dictionary
• Storage dictionary
• Resources dictionary
• Install list
• CycleCloud dictionary

Note : for the full config structure file see config.json

Variables dictionary

This allows variables to be created and used throughout the config file (see how this works here. This can be used when creating a template for others to use or just when the same value is repeated for many resources, e.g. choosing the OS image.

When creating templates for others to use the value should be <NOT-SET> so the azhpc-* commands will notify the user.

To allow a better reuse of variables across several configuration files the variable section can reference a file like this :

{
  "variables": "@variables.json"
}

The referenced file should contains variables this way :

{
    "image": "OpenLogic:CentOS:7.7:latest",
    "hpc_image": "OpenLogic:CentOS-HPC:7.7:latest",
    "location": "westeurope",
    "resource_group": "my resource group",
    "vm_type": "Standard_HB60rs",
    "vnet_resource_group": "variables.resource_group",
}

Setup information

The following properties are global :

| Name | Description | Required | Default | |------------------------------------|---------------------------------------------------|----------|---------| | location | The region where the resources are created | yes | | | resource_group | The resource group to put the resources | yes | | | install_from | The resource where the install script will be run | no | | | ssh_port | The port to use for SSH | no | 22 | | admin_user | The admin user for all resources | yes | | | proximity_placement_group_name | The proximity group name to create | no | | | global_tags | Global tags to apply to all ARM resources | no | |

The azhpc-build command will generate an install script from the configuration file. This will be run from the install_from VM. The install_from VM must either have a public IP address or be accessible by hostname from where azhpc-build is run (i.e. run azhpc-build from a VM on the same vnet). The ssh_port refers to an additional port number where the ssh daemon will listen on VMs with a public IP address. The NSG rules will only be applied to the ssh_port number although port 22 will be accessible from within the vnet.

Network dictionary

The config file will create or reuse vnet and subnets from the config file.

| Name | Description | Required | Default | |--------------------|------------------------------------------------------------------------------------|----------|---------| | resource_group | This can be used if different to the resources | no | | | name | Vnet name | yes | | | address_prefix | The address prefix (CIDR notation) | yes | | | subnets | Dictionary containing key-values for <subnet-name> and <subnet-address-prefix> | yes | | | dns_domain | Private domain name to create | no | | | peer | Dictionary of peer names to create | no | | | routes | Dictionary of route names to create | no | | | gateway | Specification of AAD based Azure VPN Gateway to create | no | |

Peer dictionary

This dictionary describes the virtual network peering to be created

| Name | Description | Required | Default | |----------------------------------|----------------------------------------------------------------------|----------|---------| | resource_group | Name of the resource group containing the vnet to peer to | yes | | | vnet_name | Name of the vnet to peer to | yes | | | peer_allow_vnet_access | Allow traffic from peer network to vnet | no | True | | peer_allow_forwarded_traffic | Allow traffic forwarded from vnet to peer network | no | True | | vnet_allow_vnet_access | Allow traffic from vnet to peer virtual network | no | True | | vnet_allow_forwarded_traffic | Allow traffic to be forwarded from peer virtual network to vnet | no | True | | gateway | Dictionary of peer-gateway to create | no | |

This dictionary describes the configuration of virtual nework gateway used in the peering

Peer-Gateway dictionary

| Name | Description | Required | Default | |--------------------------------|------------------------------------------------------------------------|----------|---------| | peer_allow_gateway_transit | Use the peer's gateway server | no | False | | peer_allow_remote_gateways | Use the vnet's gateway server | no | False | | vnet_allow_gateway_transit | Use the vnet's gateway server | no | False | | vnet_allow_remote_gateways | Use the peer's gateway server | no | False |

Route dictionary

This dictionary describes routes to be created

| Name | Description | Required | Default | |--------------------|------------------------------------------------------------------------------------|----------|---------| | address_prefix | Address space (CIDR) | yes | | | next_hop | TO DOCUMENT | yes | | | subnet | TO DOCUMENT | yes | |

Gateway dictionary

This dictionary describes the virtual network peering to be created

| Name | Description | Required | Default | |--------------------|------------------------------------------------------------------------------------|----------|---------| | name | Name of the VPN Gateway | yes | | | subnet | Subnet to be used for incoming clients of the gateway, | yes | | | | this MUST be "GatewaySubnet" and this subnet must be defined in the subnet section | | | | aad_tenant | URL of the AAD tenant as https://login.microsoftonline.com/<your Directory ID> | yes | | | aad_audience | Application ID of the Azure VPN Enterprise registration in AAD | yes | | | aad_issuer | URL of the AAD issuer as https://sts.windows.net/<your Directory ID>/ | yes | |

Here is an example setup with four subnets:

...
"vnet": {
    "resource_group": "vnet-resource-group",
    "name": "hpcvnet",
    "address_prefix": "10.2.0.0/20",
    "subnets": {
        "ad