nvitop

<!-- markdownlint-disable html -->

An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management. The full API references host at https://nvitop.readthedocs.io.

<p align="center"> <img width="100%" src="https://user-images.githubusercontent.com/16078332/171005261-1aad126e-dc27-4ed3-a89b-7f9c1c998bf7.png" alt="Monitor"> <br/> Monitor mode of <code>nvitop</code>. <br/> (TERM: GNOME Terminal / OS: Ubuntu 16.04 LTS (over SSH) / Locale: <code>en_US.UTF-8</code>) </p> <p align="center"> <a href="./nvitop-exporter"> <img width="100%" src="https://github.com/user-attachments/assets/e4867e64-2ca9-45bc-b524-929053f9673d" alt="Grafana Dashboard"> </a> <br/> A Grafana dashboard built on top of <code>nvitop-exporter</code>. </p>

Table of Contents <!-- omit in toc --> <!-- markdownlint-disable heading-increment -->

• Features
• Requirements
• Installation
• Usage
  • Device and Process Status
  • Resource Monitor
    • For Docker Users
    • For SSH Users
    • Command Line Options and Environment Variables
    • Keybindings for Monitor Mode
  • CUDA Visible Devices Selection Tool
  • Callback Functions for Machine Learning Frameworks (DEPRECATED)
    • Callback for TensorFlow (Keras)
    • Callback for PyTorch Lightning
    • TensorBoard Integration
  • More than a Monitor
    • Quick Start
    • Status Snapshot
    • Resource Metric Collector
    • Low-level APIs
      • Device
      • Process
      • Host (inherited from psutil)
• Screenshots
• Changelog
• License
  • Copyright Notice

---

nvitop is an interactive NVIDIA device and process monitoring tool. It has a colorful and informative interface that continuously updates the status of the devices and processes. As a resource monitor, it includes many features and options, such as tree-view, environment variable viewing, process filtering, process metrics monitoring, etc. Beyond that, the package also ships a CUDA device selection tool nvisel for deep learning researchers. It also provides handy APIs that allow developers to write their own monitoring tools. Please refer to section More than a Monitor and the full API references at https://nvitop.readthedocs.io for more information.

<p align="center"> <img width="100%" src="https://user-images.githubusercontent.com/16078332/202362811-34f2c01d-97c8-49d2-b19b-0d7da648f2d5.png" alt="Filter"> <br/> Process filtering and a more colorful interface. </p> <p align="center"> <img width="100%" src="https://user-images.githubusercontent.com/16078332/202362686-859bf4ad-6237-46ca-b2f7-f547d2f63213.png" alt="Comparison"> <br/> Compare to <code>nvidia-smi</code>. </p>

---

Features

• Informative and fancy output: show more information than nvidia-smi with colorized fancy box drawing.
• Monitor mode: can run as a resource monitor, rather than print the results only once.
  • bar charts and history graphs
  • process sorting
  • process filtering
  • send signals to processes with a keystroke
  • tree-view screen for GPU processes and their parent processes
  • environment variable screen
  • help screen
  • mouse support
• Interactive: responsive for user input (from keyboard and/or mouse) in monitor mode. (vs. gpustat & py3nvml)
• Efficient:
  • query device status using NVML Python bindings directly, instead of parsing the output of nvidia-smi. (vs. nvidia-htop)
  • support sparse query and cache results with TTLCache from cachetools. (vs. gpustat)
  • display information using the curses library rather than print with ANSI escape codes. (vs. py3nvml)
  • asynchronously gather information using multi-threading and correspond to user input much faster. (vs. nvtop)
• Portable: work on both Linux and Windows.
  • get host process information using the cross-platform library psutil instead of calling ps -p <pid> in a subprocess. (vs. nvidia-htop & py3nvml)
  • written in pure Python, easy to install with pip. (vs. nvtop)
• Integrable: easy to integrate into other applications, more than monitoring. (vs. nvidia-htop & nvtop)

<p align="center"> <img width="100%" src="https://user-images.githubusercontent.com/16078332/129374533-fe06c01a-630d-4994-b54b-821cccd0d33c.png" alt="Windows"> <br/> <code>nvitop</code> supports Windows! <br/> (SHELL: PowerShell / TERM: Windows Terminal / OS: Windows 10 / Locale: <code>en-US</code>) </p>

---

Requirements

• Python 3.8+
• NVIDIA Management Library (NVML)
• nvidia-ml-py
• psutil
• curses^* (with libncursesw)

NOTE: The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states. The runtime version of the NVML library ships with the NVIDIA display driver (available at Download Drivers | NVIDIA), or can be downloaded as part of the NVIDIA CUDA Toolkit (available at CUDA Toolkit | NVIDIA Developer). The lists of OS platforms and NVIDIA-GPUs supported by the NVML library can be found in the NVML API Reference.

This repository contains a Bash script to install/upgrade the NVIDIA drivers for Ubuntu Linux. For example:

git clone --depth=1 https://github.com/XuehaiPan/nvitop.git && cd nvitop

# Change to tty3 console (required for desktop users with GUI (tty2))
# Optional for SSH users
sudo chvt 3  # or use keyboard shortcut: Ctrl-LeftAlt-F3

bash install-nvidia-driver.sh --package=nvidia-driver-470  # install the R470 driver from ppa:graphics-drivers
bash install-nvidia-driver.sh --latest                     # install the latest driver from ppa:graphics-drivers

<p align="center"> <img width="100%" src="https://user-images.githubusercontent.com/16078332/174480112-e9a35edc-8f42-438e-a103-1d0ce998b381.png" alt="install-nvidia-driver"> <br/> NVIDIA driver installer for Ubuntu Linux. </p>

Run bash install-nvidia-driver.sh --help for more information.

<a name="curses">*</a> The curses library is a built-in module of Python on U