Sketch2Code

Description

This lab will guide you through the process of using Azure Custom Vision to create a solution that is capable of transforming a handwritten user interface into valid HTML markup.

Process Flow

The process of transformation is detailed as follows:

1. The user uploads an image through the website.
2. A custom vision model predicts what HTML elements are present in the image and their location.
3. A handwritten text recognition service reads the text inside the predicted elements.
4. A layout algorithm uses the spatial information from all the bounding boxes of the predicted elements to generate a grid structure that accommodates all elements.
5. An HTML generation engine uses all this information to generate HTML markup code reflecting the result.

Architecture

The Sketch2Code solution is made up of the following components:

• A Microsoft Custom Vision Model: This model has been trained with images of different handwritten designs tagged with the information of common HTML elements such as buttons, text boxes and images.
• A Microsoft Computer Vision Service: Provides OCR character recognition capabilities that can turn handwritten images into digital text.
• An Azure Blob Storage: All steps involved in the HTML generation process are stored, including the original image, prediction results, and layout grouping information.
• An Azure Function: Serves as the backend entry point that coordinates the generation process by interacting with all the services.
• An Azure web site: A frontend application that enables the upload of a new design and outputs the generated HTML results. These components form the architecture as follows:

Setup your environment

A) Setup your Azure subscription

This lab requires an Azure subscription. If you delete the resources at the end of the session, total charges will be less than $1 so we strongly recommend using an existing subscription if available.

If you need a new Azure subscription, then there are a couple of options to get a free subscription:

1. The easiest way to sign up for an Azure subscription is with VS Dev Essentials and a personal Microsoft account (like @outlook.com). This does require a credit card; however, there is a spending limit in the subscription so it won't be charged unless you explicitly remove the limit.
   • Open Microsoft Edge and go to the Microsoft VS Dev Essentials site.
   • Click Join or access now.
   • Sign in using your personal Microsoft account.
   • If prompted, click Confirm to agree to the terms and conditions.
   • Find the Azure tile and click the Activate link.
2. Alternatively, if the above isn't suitable, you can sign up for a free Azure trial.
   • Open Microsoft Edge and go to the free Azure trial page.
   • Click Start free.
   • Sign in using your personal Microsoft account.
3. Complete the Azure sign up steps and wait for the subscription to be provisioned. This usually only takes a couple of minutes.

C) Download the lab materials

Download the sample code provided for this lab, it includes the following:

• Training images
• Training label data
• An Azure orchestrator function
• The Sketch2Code frontend app

To Download:

1. Click Clone or download from this repo.
2. You can clone the repo using git or click Download ZIP to directly download the code from your browser.

D) Create a Cognitive Services resource.

We'll use an Azure Cognitive Service resource to access the Custom Vision and Computer Vision APIs.

1. Log into the Azure Portal(portal.azure.com).

2. Click Create Resource [+] from the left menu and search for Cognitive Services.

3. Select the first result and click the Create button.

4. Provide the required information:

   • Name: sketch2code-vision-<your initials>.
   • Location (US) West US 2
   • Pricing tier: S0.
   • Create a new resource group: sketch2code-<your initials>.
   • Check the I confirm I have read and understood the notice below.

5. Click Create to create the resource and deploy it. This step might take a few minutes.

6. Once the deployment is completed you will see a Deployment succeeded notification.

7. Go to All Resources in the left pane and search for the new resource: sketch2code-vision-<your initials>.

8. Click on the first result to open it.

9. Click on the Quick start option from the menu.

10. Copy the Endpoint value into Notepad.

11. Copy the Key 1 value into Notepad.

12. Click Create to create the resource and deploy it. This step might take a few minutes.

Introduction to the Azure Custom Vision Service

A) Introduction to Azure Custom Vision

The Azure Custom Vision Service lets you easily build your own custom image classifiers, without needing any knowledge of the underlying machine learning models. It provides a web interface that allows you to upload and tag training data, as well as to train, evaluate, and make predictions. You can also interact with your custom models via a REST API.

The Custom Vision service has recently been enhanced to include Object Detection, which allows you to train the service to detect multiple objects inside an image with their locations.

This module will guide you through the process of creating an Object Detection project, and tagging your first image.

What is Object Detection?

Image Classification is the process of taking an image as input and outputting a class label out of a set of classes. Image Classification with Localization is the process that takes an image as input, outputs a class label and also draws a bounding box around the object to locate it in the image.

One step beyond that is Object Detection; this process applies image localization to all the objects in the image, which results in multiple bounding boxes and can have multiple applications like face detection, or autonomous driving.

B) Create an Object Detection project

We'll start by creating a new Custom Vision object detection project:

1. In your web browser, navigate to the Custom Vision web page. Select Sign in to begin using the service.
2. To create your first project, select New Project. For your first project, you are asked to agree to the Terms of Service. Select the check box, and then select the I agree button. The New project dialog box appears.
3. Enter a name and a description for the project. Then select Object Detection for the project type.
4. Select the Cognitive Service Resource previously created in Azure (sketch2code-vision-<your initials>).
5. To create the project, select Create project.

C) Tag an image

Next we'll use the Custom Vision web interface to tag images. In this section we are going to see how to manually tag an image, but later you'll also see how this can be done via the REST API.

1. To add images to the classifier, use the Add images button.

   Note: Custom Vision Service accepts training images in .jpg, .png, and .bmp format, up to 6 MB per image. (Prediction images can be up to 4 MB per image.) We recommend that images be 256 pixels on the shortest edge. Any images shorter than 256 pixels on the shortest edge are scaled up by Custom Vision Service.

2. Browse for the file sample.png inside the model folder of the downloaded repository files and select Open to move to tagging.

3. Click the Upload 1 file button.

4. Select Done once the image has been uploaded.

5. Then click the uploaded image to start tagging.

6. Draw a box surrounding the title of the image: LANGUAGE OPTIONS and add the tag Heading to the text field and click the + button.

7. Draw a box surrounding one of the checkbox in the image including the label, for example the checkbox and the word english; add the tag CheckBox in the same way than before.

8. Do the same for all the checkboxes in the image but now select the tag created before.

9. The final result of the tagging should look similar to this image:

Train an object detection model

In this section we are going to see how to train and test an Object Detection model. The more images you include in your training set, the better your model will perform. To save time, we've provided 150 images with a set of tags. The training set used to create the sample model in the project is located in the model folder. Each training image has a unique identifier that matches the tags contained in the dataset.json file. This file contains all the tag information used to train the sample model. A script is provided to upload the training images, along with the tag data.

A) Upload the full training set

1. Open Visual Studio 2019 from the Start Menu.
2. Click Open Project/Solution from the File > Open menu.
3. Select the solution file Sketch2Code\Sketch2Code.sln and wait for it to load.

Note: If you get compilation errors right click the solution and select Rebuild Solution.

1. Take a look at the Import project.
2. Return to Microsoft Edge and the Custom Vision (customvision.ai) portal.
3. Click on the settings icon in the top right corner of portal.
4. Copy the Key located under the Resources section.
5. Return to Visual Studio.
6. Open the Program.cs file in the Import project.
7. Find the string trainingKey = ""; code around line 19 and set the value with the copied key value.

   string trainingKey = "";
   

8. Open a terminal and go to the Import project folder.
9. Enter the command dotnet run and press the enter key.
10. When the import process is finished, you will see the message: Training data upload complete!.

NOTE: The import process assumes