LegoOS

<p align="center"> <img src="https://github.com/WukLab/LegoOS/blob/master/Documentation/logo.png" height="150" width="300"> </p>

LegoOS is a disseminated, distributed operating system built for hardware resource disaggregation. LegoOS is a research operating system being built from scratch and released by researchers from Purdue University. LegoOS splits traditional operating system functionalities into loosely-coupled monitors, and run those monitors directly on hardware device. You can find more details from our OSDI'18 paper.

[Paper] [Slide] [Keynote] [Tech Notes] [Google Trace Plot]

NEWS

• Jan 2020: LegoOS is able to run on RoCE Mellanox card, and we have tested on CloudLab R320 instance. Use CentOS if possible.

Table of Contents:

• 1. <a name='Developers'></a>Developers
• 2. <a name='CodebaseOrganization'></a>Codebase Organization
• 3. <a name='PlatformRequirement'></a>Platform Requirement
• 4. <a name='ConfigureandCompile'></a>Configure and Compile
  • 4.1. <a name='ConfigureProcessororMemoryManager'></a>Configure Processor or Memory Manager
    • 4.1.1. <a name='ConfigureProcessorExCacheSize'></a>Configure Processor ExCache Size
  • 4.2. <a name='ConfigureLinuxModules'></a>Configure Linux Modules
  • 4.3. <a name='ConfigureNetwork'></a>Configure Network
    • 4.3.1. <a name='FIT'></a>FIT
    • 4.3.2. <a name='QPN'></a>QPN
    • 4.3.3. <a name='LID'></a>LID
    • 4.3.4. <a name='NodeIDandNumberofMachines'></a>Node ID and Number of Machines
  • 4.4. <a name='ConfigureOutput'></a>Configure Output
    • 4.4.1. <a name='Setupprintk'></a>Setup printk()
    • 4.4.2. <a name='SetupSerialConnection'></a>Setup Serial Connection
• 5. <a name='InstallandRun'></a>Install and Run
  • 5.1. <a name='P-1M'></a>1P-1M
    • 5.1.1. <a name='Sample.config'></a>Sample .config
  • 5.2. <a name='P-1M-1S'></a>1P-1M-1S
    • 5.2.1. <a name='Configurations'></a>Configurations
    • 5.2.2. <a name='Boot'></a>Boot
    • 5.2.3. <a name='Sample.config-1'></a>Sample .config
  • 5.3. <a name='P-2M-1Sandmore'></a>1P-2M-1S (and more)
    • 5.3.1. <a name='Configurations-1'></a>Configurations
    • 5.3.2. <a name='Boot-1'></a>Boot
    • 5.3.3. <a name='Sample.confg'></a>Sample .confg
  • 5.4. <a name='VirtualMachine'></a>Virtual Machine
    • 5.4.1. <a name='VMSetup'></a>VM Setup
    • 5.4.2. <a name='InfiniBand'></a>InfiniBand

1. <a name='Developers'></a>Developers

• Profiling
• strace
• Counters and Watchdog
• Memory Manager Configurations

2. <a name='CodebaseOrganization'></a>Codebase Organization

Several terms in this repository are used differently from the paper description. Some of them might be used interchangeably here.

| Paper Term | Code Term| |:------------|:----------| | Resource Monitor | Resource Manager | | Global Resource Manager | Global Resource Monitor | | ExCache | pcache | |p-local|zerofill|

Now let's first get familiar with the codebase. If you have played with Linux kernel, welcome home. We reused most of Linux code to ease our own porting of InfiniBand drivers. The consequence is now LegoOS supports almost all essential Linux kernel functionalities. Overall, LegoOS has a similar directory organization:

• arch/ is for low-level ISA-specific hooks
• drivers/ has acpi, infiniband, pci, and tty drivers
• init/, kernel/, lib/, and mm/ are shared essential core kernel utilities
• linux-modules/ are Linux kernel modules for storage manager and global resource monitors

This code repository has many major subsystems (e.g., managers, monitors, networking). The following table describes where you can find the corresponding code:

| Major Subsystems | Directory | |:---------|:-----------| |Processor Manager| managers/processor/| |Memory Manager |managers/memory/| |Storage Manager |linux-modules/storage/| |Managers' Network Stack|net/| |Global Process Monitor |linux-modules/monitor/gpm/| |Global Memory Monitor |linux-modules/monitor/gmm/| |Monitors' Network Stack|linux-modules/fit/|

As for the Processor Manager (PM), it has the following subsystems:

| PM Internal | Purpose |Directory| |:---------------------------|:--------|:---------| |pcache|Virtual Cache Management|managers/processor/pcache/| |strace|Syscall Tracer|managers/processor/strace/| |fs|Filesystem State Layer|managers/processor/fs/| |mmap|Virtual Memory State Layer |managers/processor/mmap/| |replication|Memory Replication|managers/processor/replication.c| |fork|Process Creation Notification|managers/processor/fork.c| |exec|Execute Notification|managers/processor/exec.c| |misc|misc|all others|

As for the Memory Manager (MM), it has the following subsystems:

| MM Internal| Purpose | Directory| |:-----------------------|:--------|:---------| |pcache|Handle pcache Events|managers/memory/handle_pcache/| |loader|Program Loader|managers/memory/loader/| |pgcache|Page Cache|managers/memory/pgcache/| |replication|Handle Memory Replication|managers/memory/replica/| |vm|Virtual Memory|managers/memory/vm/| |fs|Filesystem Operations|managers/memory/m2s_read_write.c| |misc|misc| all others|

Storage manager and global resource monitors are not LegoOS's main focus at this stage, each of them has one simple task just as their name suggested.

3. <a name='PlatformRequirement'></a>Platform Requirement

LegoOS has been tested only in the following hardware setting:

|Hardware| Vendor and Model| |:--|:--| |Server| Dell PowerEdge R730| |CPU| Intel Xeon E5-2620 v3| |InfiniBand NIC|Mellanox MCX354A-TCBT ConnectX-3 VPI| |InfiniBand Switch|Melanox IS5035|

And the following toolchains:

|Software|Version| |:--|:--| |CentOS|7.2| |GCC|4.8.5 20150623 (Red Hat 4.8.5-16)| |GNU assembler|2.23.52.0.1 (x86_64-redhat-linux)| |GNU ld|2.23.52.0.1-55.el7 20130226| |GNU libc|2.17| |GRUB2|2.02|

Of all the above hardware and software requirments, the CPU and the Infiniband NIC are the hard requirements. Currently, LegoOS can only run on Intel x86 CPUs. As for the Infiniband NIC card, LegoOS has ported an mlx4_ib driver, which probably can run on other Mellanox cards, but we have not tested other than the one we used. As long as you have the CPU and the Infiniband NIC, we think you can run LegoOS on top your platform. You need at least two machines, connected by Infiniband switch (back-to-back connection is not supported now).

We understand that one key for an OS to be successful is let people be able to try it out. We are deeply sorry that we can not provide further technical support if you are using a different platform.

4. <a name='ConfigureandCompile'></a>Configure and Compile

The README is still raw and scratchy, it might not be complete and it might also seems confusing. The whole tutorial can be improved only if there are people trying out LegoOS and give us feedback. If you have any issues, please don't hesitate to contact us (Github Issue is preferred). We really appreciate your input here.

CAVEAT: Configure, compile, and run a LegoOS kernel is similar to test a new Linux kernel. You need to have root access to the machine. The whole process may involve multiple machine power cycles. Before you proceed, make sure you have some methods (e.g., IPMI) to monitor and reboot remote physical machine. It is possible to jus