HostedPumpkin

Proof-Of-Concept (POC) for the submission, compilation and execution of C# code snippets.

The solution is composed by the following sub-projects:

• The main project is a unmanaged C++ project, which hosts the CLR using the Hosting API. The Host uses two helper assemblies:
• a custom AppDomainManager (project SimpleHostRuntime)
• a set of common classes (project Pumpkin.Monitor) loaded with the snippet assembly
• The Pumpkin.Submission project compiles snippets using the CodeDOM compiler (i.e. going through csc.exe), then serializes the assemblies.
• The Pumpkin.Data project provides supporting functions, for example it helps in storing and retrieving compiled snippets in/from a DB
• The Pumpkin.Web project is a very simple ASP.NET MVC project, with pages for submitting a new snippet, listing all the snippets and execute them.

The goal of this POC is to take some C#, compile it, run it. Easy :) More precisely, we want to execute third-party code (a “snippet”) in a safe, reliable and efficient way.

The various projects in the solution collaborate as depicted in this architectural diagram:

More in detail; the implementation separates the phases of compiling a new snippet ("submitting" it) and of executing it. The reasoning behind this choice was that the first is usually a one-time operation; a user write some demo code in a snippet and submits it. The MVC controller inside Pumpkin.Web takes it, compiles it using classes from Pumpkin.Submission and stores it in the DB with the help of Pumpkin.Data.

Then when other users see the snippet, they hit the "Go" button on the snippet list page, and the system runs it: the page sends a POST to another controller method in Pumpkin.Web, which in turn sends the snippet id to the SimpleHostRuntime running inside the Host; the host loads it from the DB with the help of Pumpkin.Data and executes it:

// called when a new snippet is created
[HttpPost]
public ActionResult SubmitSnippet(/*snippet code params*/) {
    // Take the code, using statements, etc. Put them together and compile it
    var compiledSnippet = Pumpkin.SnippetCompiler.CompileWithCSC(...);
    // the compilation produces an assembly as a byte[]
    if (compiledSnippet.success) {
      //Save the byte[] into the DB
      repository.Save(compiledSnippet);
      return new HttpStatusCodeResult(204);
    }
}

---

// Called when we want to run a previously compiled snippet
[HttpPost]
public async Task<ActionResult> RunSnippetAsync(String snippetId) {
   return Json(await HostConnector.RunSnippetAsync(snippetId));
}
public static async Task<SnippetResult> RunSnippetAsync(string snippetId) {
   var socket = GetSocket();
   await socket.SendAsync(snippetId, true); 
   // Send the snippet ID to the host.
   // The host takes the snippet ID, loads the snippet assembly from the DB,
   // executes it and returns the result (as a JSON)(**)
   string s = await socket.ReceiveAsync();
   return JsonConvert.DeserializeObject<SnippetResult>(s);
}

---

// (**) Inside the host, in the HostServer class:
var command = await socket.ReceiveAsync();
var snippetId = Guid.Parse(command);
var snippetInfo = repository.Get(snippetId);
queue.SubmitSnippet(snippetInfo, tcs);

The various bits can be re-arranged at will: a good idea would be to run the snippet just after compilation/submission, maybe in an "isolated" Host: a Host that uses a single AppDomain (not a pool, as the regular ones), where the snippet can fail, timeout, throw... we could "test" it upon submission, just after compilation, so we know if the snippet can be accepted in the system. Or if there are errors, we can put it in a review queue, or reject it, or use the "Exceptional queue" in the picture.

Usage

The main project, the Host, can be used in a "standalone" way, to run a series of tests or a single snippet:

  USAGE:

     SimpleHost.exe  [-p <int>] [-d <string>] [-m <string>] [-c <string>] [-a
                     <string>] -v <string> [--] [--version] [-h]


  Where:

     -p <int>,  --port <int>
       The port on which this Host will listen for snippet execution requests

     -d <string>,  --database <string>
       The path of the SQL CE database that holds the snippets

     -m <string>,  --method <string>
       The method to invoke

     -c <string>,  --type <string>
       The type name which contains the method to invoke

     -a <string>,  --assembly <string>
       The assembly file name

     -v <string>,  --clrversion <string>
       (required)  CLR version to use

     --,  --ignore_rest
       Ignores the rest of the labeled arguments following this flag.

     --version
       Displays version information and exits.

     -h,  --help
       Displays usage information and exits.

To run a single snippet: pass the -t (--test) option, with the -a (--assembly), -c (--type) and, optionally, the -m (--method) options. They are used to specify the method to run, the class (FullName) that contains the method, and the assembly file (.dll or .exe), where the class is implemented.

Example:

-v v4.0.30319 -t -a "$(LocalDebuggerWorkingDirectory)TestApplication\bin\Debug\TestApplication.exe" -c TestApplication.Program -m SnippetTest12

If the --method option is omitted, all methods which name starts with SnippetTest will be run.

Example:

-v v4.0.30319 -t -a "$(LocalDebuggerWorkingDirectory)TestApplication\bin\Debug\TestApplication.exe" -c TestApplication.Program

Otherwise, the -d (--database) option is required. This option indicates the full path to the SQL Compact database used to store the snippets. The Host starts, listen on the port specified by -p (--port) (default: 4321), and execute Snippets as they are requested, loading them from the database.

Example:

-v v4.0.30319 -d "$(SolutionDir)Pumpkin.Web\App_Data\Snippets.sdf"

You must also specify a CLR/.NET version which is installed on your system (e.g. -v v4.0.30319). The available versions are those listed under %WINDIR%\Microsoft.NET\Framework.

TODO

• <del>Set THREAD_PRIORITY_ABOVE_NORMAL for supervisor threads</del>
• Extra safety: killing the supervisor (watchdog) thread brings process down
• <del>Same for server thread</del>
• No named sync primitives (i.e.: forbid Mutex usage)
  • (or better: decorate the name with the snippet GUID)
• <del>Choose C#/.NET version during snippet submission, record it.</del>
  • Get the list of available .NET SDKs using the Hosting API (from the Host)
• Choose the references from a list (during submission, for compilation) - Same "whitelist" used by Cecil post-processing
• <del>Allow definition of static methods and classes</del>
• Supervisor: spawn two (N) hosts, maintain the count.
• Use a distributed queue (e.g. Redis) for snippet submission/results, instead of plain sockets
  • Move the "server" thread from managed to unmanaged, to improve stability and error handling in case of FEEE (CLR unload)
• Use a separate, "single-run" host for "problematic" snippets.
  • Host can be run/configured in "single mode": one snippet, execute and exit.
• Use a separate host for each .NET version
• Use the new pooling API for better handling/managing ThreadPool threads
• Finish implementation of the various Console.WriteLine overloads in Pumpkin.Monitor
• <del>Measure complete round-trip time</del>
• TEST TEST TEST!
• BUGS BUGS BUGS!

The problems

We want constraints on what the code can do.

We want safety/security (for example: you do not want snippets to use WMI to shutdown the machine, or open a random port, install a torrent server, read configuration files from the server, erase files...),

We want to be able to handle dependencies in a sensible way (some assemblies/classes/methods just do no make any sense: Windows Forms? Workflow Foundations? Sql?)

We want to monitor and cap resource usage:

• no snippets that do not terminate -no, we don't want to save the Halting Problem, but we want to timeout and bring down the offending snippet as cleanly as possible, reclaiming the allocated resources (memory and threads)-,
• no snippets that try to clog the system by spawning too many threads,
• no snippets that take all the available memory, and so on).

Which "security" do we want to provide?

1. no "unsafe"
2. no p/invoke or unmanaged code
3. nothing from the server that runs the snippet is accessible: no file read, no access to local registry (read OR write!)
4. no network permission
5. whitelist of permissions and assemblies

Which "resources" we want to control?

1. limit execution time

• running time/execution time

2. limit thread creation (avoid "fork-bombs")

• deny (or handle in a sensible way) access to named kernel objects (e.g. named mutexes.. you do not want some casual interaction with other mutexes!)

3. limit process creation (zero - no spawning of processes. Can fall back to the "security" category: deny process creation)
4. limit memory usage
5. limit file usage (no files. Can fall back to the "security" category)
6. limit network usage (no network. Can fall back to the "security" category)

• in the future: virtual network, virtual files?

7. limit output (Console.WriteLine, Debug.out...)

• and of course redirect it

Performance considerations

1. Snippet execution should be fast (< 1sec. for simple snippets, i.e. overhead << 1sec.)
2. Scaling up: can execute hundreds of snippets concurrently (on a single/few machines)

Background