Modin
Modin: Scale your Pandas workflows by changing a single line of code
Install / Use
/learn @modin-project/ModinREADME
| <h3>Dev Community & Support</h3> | <h3>Forums</h3> | <h3>Socials</h3> | <h3>Docs</h3> |
|:---: | :---: | :---: | :---: |
| 
What is Modin?
Modin is a drop-in replacement for pandas. While pandas is single-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your cores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs out of memory. Also, Modin comes with the additional APIs to improve user experience.
By simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:
<img src="https://github.com/modin-project/modin/raw/main/docs/img/Import.gif" style="display: block;margin-left: auto;margin-right: auto;" width="100%"></img>
In the GIFs below, Modin (left) and pandas (right) perform the same pandas operations on a 2GB dataset. The only difference between the two notebook examples is the import statement.
<table class="tg"> <thead> <tr> <th class="tg-0lax" style="text-align: center;"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=True" height="35px"></th> <th class="tg-0lax" style="text-align: center;"><img src="https://pandas.pydata.org/static/img/pandas.svg" height="50px"></img></th> </tr> </thead> <tbody> <tr> <td class="tg-0lax"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin.gif"></img></td> <td class="tg-0lax"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Pandas.gif"></img></td> </tr> </tbody> </table>The charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found here. To learn more about the speedups you could get with Modin and try out some examples on your own, check out our 10-minute quickstart guide to try out some examples on your own!
<img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin_Speedup.svg" style="display: block;margin-left: auto;margin-right: auto;" width="100%"></img>
Installation
From PyPI
Modin can be installed with pip on Linux, Windows and MacOS:
pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.
If you want to install Modin with a specific engine, we recommend:
pip install "modin[ray]" # Install Modin dependencies and Ray.
pip install "modin[dask]" # Install Modin dependencies and Dask.
pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.
To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
it is required to have a working MPI implementation installed beforehand.
Otherwise, installation of modin[mpi] may fail. Refer to
Installing with pip
section of the unidist documentation for more details about installation.
Note: Since Modin 0.30.0 we use a reduced set of Ray dependencies: ray instead of ray[default].
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing ray[default] along with modin[ray].
Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.
From conda-forge
Installing from conda forge using modin-all
will install Modin and three engines: Ray, Dask and
MPI through unidist.
conda install -c conda-forge modin-all
Each engine can also be installed individually (and also as a combination of several engines):
conda install -c conda-forge modin-ray # Install Modin dependencies and Ray.
conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.
Note: Since Modin 0.30.0 we use a reduced set of Ray dependencies: ray-core instead of ray-default.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing ray-default along with modin-ray.
Refer to Installing with conda section of the unidist documentation for more details on how to install a specific MPI implementation to run on.
To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:
conda install -n base conda-libmamba-solver
and then use it during istallation either like:
conda install -c conda-forge modin-ray --experimental-solver=libmamba
or starting from conda 22.11 and libmamba solver 22.12 versions:
conda install -c conda-forge modin-ray --solver=libmamba
Choosing a Compute Engine
If you want to choose a specific compute engine to run on, you can set the environment
variable MODIN_ENGINE and Modin will do computation with that engine:
export MODIN_ENGINE=ray # Modin will use Ray
export MODIN_ENGINE=dask # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist
If you want to choose the Unidist engine, you should set the additional environment
variable UNIDIST_BACKEND. Currently, Modin only supports MPI through unidist:
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
This can also be done within a notebook/interpreter before you import Modin:
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("ray") # Modin will use Ray
modin_cfg.Engine.put("dask") # Modin will use Dask
modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend
Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior.
Which engine should I use?
On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required to use either of these engines as Modin abstracts away all of the complexity, so feel free to pick either!
Pandas API Coverage
<p align="center">| pandas Object | Modin's Ray Engine Coverage | Modin's Dask Engine Coverage | Modin's Unidist Engine Coverage |
|-------------------|:------------------------------------------------------------------------------------:|:---------------:|:---------------:|
| pd.DataFrame | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> |
| pd.Series | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg>
| pd.read_csv | ✅ | ✅ | ✅ |
| pd.read_table | ✅ | ✅ | ✅ |
| pd.read_parquet | ✅
Related Skills
oracle
334.1kBest practices for using the oracle CLI (prompt + file bundling, engines, sessions, and file attachment patterns).
prose
334.1kOpenProse VM skill pack. Activate on any `prose` command, .prose files, or OpenProse mentions; orchestrates multi-agent workflows.
claude-opus-4-5-migration
82.1kMigrate prompts and code from Claude Sonnet 4.0, Sonnet 4.5, or Opus 4.1 to Opus 4.5
Command Development
82.1kThis skill should be used when the user asks to "create a slash command", "add a command", "write a custom command", "define command arguments", "use command frontmatter", "organize commands", "create command with file references", "interactive command", "use AskUserQuestion in command", or needs guidance on slash command structure, YAML frontmatter fields, dynamic arguments, bash execution in commands, user interaction patterns, or command development best practices for Claude Code.
