HSCC---Hardware/Software Cooperative Caching for Hybrid DRAM/NVM Memory Architectures

       HSCC is implemented with zsim and NVMain simulators. Zsim is a fast x86-64 multi-core simulator. It exploits Intel Pin toolkit to collect traces of memory accesses for processes, and replays the traces in the zsim simulator. NVMain is a cycle-accurate memory simulator, it models components of DRAM and NVMs, and memory hierarchy in detail. The integrated "zsim + NVMain" simulators can be forked from "https://github.com/AXLEproject/axle-zsim-nvmain".

Based on the "zsim + NVMain" hybrid simulator, HSCC has added the following functions:

• Memory management simulations (such as MemoryNode, Zone, Buddy allocator etc.): As the pin-based zsim only replays virtual address in the x86 system architecture, and does not support OS simulation, HSCC has added memory management modules into zsim, including memory node, zone and buddy allocator.

• TLB simulation: The original "zsim + NVMain" simulator does not simulate the TLB. TLB simulation is implemented in zsim to accelerate address translations from virtual address to physical address.

• Implementation of HSCC, a hierarchical hybrid DRAM/NVM memory system that pushes DRAM caching management issues into the software level: DRAM cache is managed by hardware in tranditional DRAM/NVM hierarchical hybrid systems. HSCC is a novel software-managed cache mechanism that organizes NVM and DRAM in a flat physical address space while logically supporting a hierarchical memory architecture. This design simplifies the hardware design by pushing the burden of DRAM cache management to the software layers. Besides, HSCC only caches hot NVM pages into the DRAM cache to mitigate potential cache thrashing and bandwidth waste between DRAM cache and NVM main memory.

• Multiple DRMA/NVM hybrid architecture supports: HSCC supports both DRAM/NVM flat-addressable hybrid memory architecuture and DRAM/NVM hierarchical hybrid architecture. As shown in following figure, both DRAM and NVM are used as main memory and managed by OS in a single flat address space. In DRAM/NVM hierarchical hybrid memory architecture, DRAM is exploited as a cache to the NVM, and hardware-assisted hit-judgement mechanism is implemented to determine whether data is hit in DRAM cache. Besides, to reduce hardware overhead, DRAM cache is organized set-associative and usually uses demand-based caching policies.

•  Multiple DRAM/NVM hybrid system optimization policies: We have implemented Row Buffer Locality Aware (RBLA) page caching policy and MultiQueue-based (MultiQueue) page migration policy in DRAM/NVM flat addressable hybrid memory system. RBLA caching policy is a simple implementation of hybrid memory system proposed in the paper "Row Buffer Locality Aware Caching Policies for Hybrid Memories", MultiQueue migration policy is a simple implementation of system proposed in the paper "Page Placement in Hybrid Memory Systems". RBLA caching policy is aimed at migrating NVM pages with poor row buffer locality to DRAM since row buffer miss of NVM pages incur higher overhead than that of DRAM pages. The MultiQueue migration policy places hot NVM pages into DRAM, and MQ algorithm is used to update the hotness of pages based on time locality and access frequency.

The architecture and modules of HSCC are shown in the following figure:

Origianl License & Copyright of zsim

zsim is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 2.

zsim was originally written by Daniel Sanchez at Stanford University, and per Stanford University policy, the copyright of this original code remains with Stanford (specifically, the Board of Trustees of Leland Stanford Junior University). Since then, zsim has been substantially modified and enhanced at MIT by Daniel Sanchez, Nathan Beckmann, and Harshad Kasture. zsim also incorporates contributions on main memory performance models from Krishna Malladi, Makoto Takami, and Kenta Yasufuku.

zsim was also modified and enhanced while Daniel Sanchez was an intern at Google. Google graciously agreed to share these modifications under a GPLv2 license. This code is (C) 2011 Google Inc. Files containing code developed at Google have a different license header with the correct copyright attribution.

Additionally, if you use this software in your research, we request that you reference the zsim paper ("ZSim: Fast and Accurate Microarchitectural Simulation of Thousand-Core Systems", Sanchez and Kozyrakis, ISCA-40, June 2013) as the source of the simulator in any publications that use this software, and that you send us a citation of your work.

License & Copyright of HSCC (HUST SCTS & CGCL Lab)

HSCC is implemented by Yujie Chen, Dong Liu and Haikun Liu at Cluster and Grid Computing Lab & Services Computing Technology and System Lab in Huazhong University of Science and Technology(HUST SCTS & CGCL Lab), the copyright of this HSCC remains with CGCL & SCTS Lab of Huazhong University of Science and Technology.

Citing HSCC

If you use HSCC, please cite our reearch paper published at ICS 2017, included as doc/HSCC.pdf.

Haikun Liu, Yujie Chen, Xiaofei Liao, Hai Jin, Bingsheng He, Long Zhen and Rentong Guo, Hardware/Software Cooperative Caching for Hybrid DRAM/NVM Memory Architectures, in: Proceedings of the 31st International Conference on Supercomputing (ICS'17), Chicago, IL, USA, June 14-16, 2017

@inproceedings{Liu:2017:HCC:3079079.3079089,
 author = {Liu, Haikun and Chen, Yujie and Liao, Xiaofei and Jin, Hai and He, Bingsheng and Zheng, Long and Guo, Rentong},
 title = {Hardware/Software Cooperative Caching for Hybrid DRAM/NVM Memory Architectures},
 booktitle = {Proceedings of the International Conference on Supercomputing},
 series = {ICS 2017},
 year = {2017},
 isbn = {978-1-4503-5020-4},
 location = {Chicago, Illinois},
 pages = {26:1--26:10},
 articleno = {26},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/3079079.3079089},
 doi = {10.1145/3079079.3079089},
 acmid = {3079089},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {caching, hybird memory, non-volatile memory (NVM)},
}

Setup,Compiling and Configuration

1.External Dependencies
       Before install hybrid simulator zsim-nvmain, it's essential that you have already install dependencies listing below.

• gcc(>=4.6)
• Intel Pin Toolkit(Already integrates in project, or you can download from pin-2.13-61206-gcc.4.4.7-linux)
• boost (version 1.59 is ok)
• libconfig
• hdf5

2.Compiling

• Update environment script env.sh according to your machine configuration

#!/bin/sh
PINPATH= path of pin_kit
NVMAINPATH= path of nvmain
ZSIMPATH= path of zsim-nvmain
BOOST= path of boost
LIBCONFIG= path of libconfig
HDF5=path of hdf5
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PINPATH/intel64/lib:$PINPATH/intel64/runtime:$PINPATH/intel64/lib:$PINPATH/intel64/lib-ext:$BOOST/lib:$HDF5/lib:$LIBCONFIG:/lib
INCLUDE=$INCLUDE:$HDF5/include:$LIBCONFIG:/include
LIBRARY_PATH=$LIBRARY_PATH:$HDF5/lib
CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:$HDF5/include
export ZSIMPATH PINPATH NVMAINPATH LD_LIBRARY_PATH BOOST CPLUS_INCLUDE_PATH LIBRARY_PATH

• Compiling and Installation

[root @node1 HSCC]# cd zsim-nvmain
[root @node1 zsim-nvmain]# source env.sh  //init environmental values
[root @node1 zsim-nvmain]# scons -j16    //compiling, -j16 represents that compiling with 16 cores

If error "could not exec $PINPATH/intel64(ia32)/bin/pinbin" happens, it means that you are not authorized to execute pinbin, this can be solved with the following command:

[root @node1 zsim-nvmain]# chmod a+x $PINPATH/intel64(ia32)/bin/pinbin 

• Using a virtual machine
  If you use another OS, can't make system-wide configuration changes, or just want to test zsim without modifying your system, you can run zsim on a Linux VM. We have included a vagrant configuration file (http://vagrantup.com) that will provision an Ubuntu 12.04 VM to run zsim. You can also follow this Vagrantfile to figure out how to setup zsim on an Ubuntu system. Note that zsim will be much slower on a VM because it relies on fast context-switching, so we don't recommend this for purposes other than testing and development. Assuming you have vagrant installed (sudo apt-get install vagrant on Ubuntu or Debian), follow these steps: Copy the Vagrant file to the zsim root folder, boot up and provision the base VM with all dependencies, then ssh into the VM.

[root @node1 zsim-nvmain]# cp misc/Vagrantfile .
[root @node1 zsim-nvmain]# vagrant up
[root @node1 zsim-nvmain]# vagrant ssh

Vagrant automatically syncs the zsim root folder of your host machine to /vagrant/ on the guest machine. Now that you're in the VM, navigate to that synced folder, and simply build and use zsim (steps 5 and 6 above)

[root @node1 zsim-nvmain]# cd cd /vagrant/
[root @node1 zsim-nvmain]# scons -j4

3.zsim Configuration Keys (example zsim configuration files is in zsim-nvmain/config directory)

• Enable TLB、Page Table and Memory Management Simulation
  (1) sys.tlbs.tlb_type: type of TLB, default is "CommonTlb","HotMonitorTlb" enables HSCC policy;
  **(2) sys.t