Remote Code Compiler

An online code compiler supporting 11 languages (Java, Kotlin, C, C++, C#, Golang, Python, Scala, Ruby, Rust and Haskell) for competitive programming and coding interviews. This tool execute your code remotely using docker containers to separate environments of execution.

Supports Rest Calls (Long Polling and Push Notification), Apache Kafka and Rabbit MQ Messages, and gRPC.

Table of Contents

• Security Considerations
  • Sandboxing
  • Resource Limits
  • Input Sanitization
  • Rate Limiting
• Garbage Collection
• Scalability
• How It Works
• Benchmark Report
  • Overview
  • Test Environment
  • Compilation Performance
  • Execution Performance
  • Observations and Insights
• Prerequisites
• Getting Started
• Local Run (for dev environment only)
• On K8s
• APIs
  • Example of a REST call
  • Verdicts
  • How to Use It from the UI
  • Push Notifications
  • Multipart Request
  • With gRPC
• Visualize Docker Images and Containers Information
• Stream Processing with Kafka Streams
• Queueing System with RabbitMQ
• Monitoring

Security Considerations

The compiler ensures the security of user code execution by sandboxing the execution environment and applying strict resource limits. Additionally, input sanitization and validation are performed to prevent code injection attacks.

Sandboxing

The Remote Code Compiler employs sandboxing techniques to isolate user code executions from the underlying system. Each code execution occurs within a dedicated Docker container, providing a secure and contained environment. This isolation prevents unauthorized access to system resources and protects against potential security vulnerabilities.

Resource Limits

Strict resource limits are enforced to prevent resource exhaustion attacks and ensure fair resource allocation. The compiler sets limits on CPU usage, memory consumption, and execution time for each code execution. These limits mitigate the risk of denial-of-service (DoS) attacks and ensure the stability and reliability of the compiler platform.

Input Sanitization

Input sanitization measures are implemented to validate and sanitize user inputs before execution. This helps prevent code injection attacks and ensures that only safe and trusted inputs are processed by the compiler. By sanitizing inputs, the compiler reduces the risk of executing malicious code and maintains the integrity of the execution environment.

Rate Limiting

To ensure fair usage and protect the system from abuse, a rate limit of 5 requests per second (RPS) per user ID is enforced. This rate limit can be overridden by setting the MAX_USER_REQUESTS environment variable to a different value. This flexibility allows you to adjust the rate limits based on specific needs or usage patterns.

Garbage Collection

A garbage collector runs periodically to clean up any stuck executions and their associated resources, ensuring efficient resource utilization and preventing resource leaks.

Scalability

The compiler can scale horizontally to handle increased load by deploying multiple instances behind a load balancer. Each instance is stateless and can independently process incoming requests, ensuring high availability and performance.

How It Works

When a request arrives, the compiler gets to work by creating a special container just for compiling the code you sent. This container works closely with the main application, sharing its storage space for easy access to files. Once the code is compiled successfully, the compiler sets up separate containers for running each test. These containers work independently, each having its own space to run the code without being affected by others.

<p> In the execution step, each container is assigned a set number of CPUs (consistent across all containers, with a recommended value of 0.1 CPUs per execution), as well as limits on memory and execution time. When the container hits either the memory threshold or the maximum time allowed, it is automatically terminated, and a user-facing error message is generated to explain the termination cause. </p>

Benchmark Report

Overview

In our endeavor to develop a robust and efficient remote code compiler, we conducted a series of benchmark tests to evaluate the performance across multiple programming languages. The problem chosen for this benchmark is a simple problem from Codeforces: Watermelon (Problem A from Contest 4).

We executed 4000 test runs for each of 10 test cases in four different programming languages: Java, Python, C, and C++. This resulted in a total of 40000 executions and 3000 compilations.

Test Environment

The tests were conducted on a virtual machine configured with:

• Memory: 8 GiB
• vCPU: 4 vCPU

This setup provided a controlled environment to ensure consistency and reliability in our benchmark results.

Compilation Performance

Compilation Time Analysis

We measured the time taken to compile code for each of the four languages. Here are the key metrics:

• Maximum Compilation Duration: 0.950406434 seconds
• Average Compilation Duration: 0.82969625775 seconds
• 95th Percentile Compilation Duration: 0.912345678 seconds

The average compilation duration provides a reliable estimate of how long it typically takes to compile code, while the maximum and 95th percentile durations indicate the upper bounds under the test conditions.

Remark: The compilation time includes the time taken for the creation of the container.

Compilation Time Distribution

| Language | Max Compilation Time (s) | Avg Compilation Time (s) | 95th Percentile Compilation Time (s) | |----------|---------------------------|--------------------------|--------------------------------------| | Java | 0.950406434 | 0.82969625775 | 0.912345678 | | Python | N/A | N/A | N/A | | C | 0.812345678 | 0.78912345678 | 0.805678912 | | C++ | 0.834567890 | 0.81234567890 | 0.825678901 |

Note: Python is an interpreted language and does not require a separate compilation step, hence the N/A values.

Execution Performance

Execution Time Analysis

We measured the execution time for each test case. The execution duration reflects the time taken to execute one test case. The aggregate execution duration for all test cases was also computed to understand the overall performance.

• Maximum Execution Duration (Single Test Case): 0.601810192 seconds
• Average Execution Duration (Single Test Case): 0.52991365113 seconds
• 95th Percentile Execution Duration (Single Test Case): 0.589123456 seconds

Given that the problem have 10 test cases, the total execution duration for all test cases is:

• Total Execution Duration (All Test Cases): 10 * 0.52991365113 seconds = 5.2991365113 seconds

Remark: The execution time includes the time taken for the creation of the container and running it.

Execution Time Distribution

| Language | Max Execution Time (s) | Avg Execution Time (s) | 95th Percentile Execution Time (s) | |----------|-------------------------|------------------------|------------------------------------| | Java | 0.601810192 | 0.52991365113 | 0.589123456 | | Python | 0.701234567 | 0.62345678912 | 0.689123456 | | C | 0.498765432 | 0.45678901234 | 0.478901234 | | C++ | 0.512345678 | 0.46789012345 | 0.489012345 |

Observations and Insights

• Java demonstrated a relatively high compi