Pystow

<h1 align="center"> PyStow </h1> <p align="center"> <a href="https://github.com/cthoyt/pystow/actions/workflows/tests.yml"> <img alt="Tests" src="https://github.com/cthoyt/pystow/actions/workflows/tests.yml/badge.svg" /></a> <a href="https://pypi.org/project/pystow"> <img alt="PyPI" src="https://img.shields.io/pypi/v/pystow" /></a> <a href="https://pypi.org/project/pystow"> <img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/pystow" /></a> <a href="https://github.com/cthoyt/pystow/blob/main/LICENSE"> <img alt="PyPI - License" src="https://img.shields.io/pypi/l/pystow" /></a> <a href='https://pystow.readthedocs.io/en/latest/?badge=latest'> <img src='https://readthedocs.org/projects/pystow/badge/?version=latest' alt='Documentation Status' /></a> <a href="https://codecov.io/gh/cthoyt/pystow/branch/main"> <img src="https://codecov.io/gh/cthoyt/pystow/branch/main/graph/badge.svg" alt="Codecov status" /></a> <a href="https://github.com/cthoyt/cookiecutter-python-package"> <img alt="Cookiecutter template from @cthoyt" src="https://img.shields.io/badge/Cookiecutter-snekpack-blue" /></a> <a href="https://github.com/astral-sh/ruff"> <img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff" style="max-width:100%;"></a> <a href="https://github.com/cthoyt/pystow/blob/main/.github/CODE_OF_CONDUCT.md"> <img src="https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg" alt="Contributor Covenant"/></a> <a href="https://zenodo.org/badge/latestdoi/318194121"> <img src="https://zenodo.org/badge/318194121.svg" alt="DOI"></a> </p>

👜 Easily pick a place to store data for your Python code

💪 Getting Started

Get a directory for your application.

import pystow

# Get a directory (as a pathlib.Path) for ~/.data/pykeen
pykeen_directory = pystow.join('pykeen')

# Get a subdirectory (as a pathlib.Path) for ~/.data/pykeen/experiments
pykeen_experiments_directory = pystow.join('pykeen', 'experiments')

# You can go as deep as you want
pykeen_deep_directory = pystow.join('pykeen', 'experiments', 'a', 'b', 'c')

If you reuse the same directory structure a lot, you can save them in a module:

import pystow

pykeen_module = pystow.module("pykeen")

# Access the module's directory with .base
assert pystow.join("pykeen") == pystow.module("pykeen").base

# Get a subdirectory (as a pathlib.Path) for ~/.data/pykeen/experiments
pykeen_experiments_directory = pykeen_module.join('experiments')

# You can go as deep as you want past the original "pykeen" module
pykeen_deep_directory = pykeen_module.join('experiments', 'a', 'b', 'c')

Get a file path for your application by adding the name keyword argument. This is made explicit so PyStow knows which parent directories to automatically create. This works with pystow or any module you create with pystow.module.

import pystow

# Get a directory (as a pathlib.Path) for ~/.data/indra/database.tsv
indra_database_path = pystow.join('indra', 'database', name='database.tsv')

Ensure a file from the internet is available in your application's directory:

import pystow

url = 'https://raw.githubusercontent.com/pykeen/pykeen/master/src/pykeen/datasets/nations/test.txt'
path = pystow.ensure('pykeen', 'datasets', 'nations', url=url)

Ensure a tabular data file from the internet and load it for usage (requires pip install pandas):

import pystow
import pandas as pd

url = 'https://raw.githubusercontent.com/pykeen/pykeen/master/src/pykeen/datasets/nations/test.txt'
df: pd.DataFrame = pystow.ensure_csv('pykeen', 'datasets', 'nations', url=url)

Ensure a comma-separated tabular data file from the internet and load it for usage (requires pip install pandas):

import pystow
import pandas as pd

url = 'https://raw.githubusercontent.com/cthoyt/pystow/main/tests/resources/test_1.csv'
df: pd.DataFrame = pystow.ensure_csv('pykeen', 'datasets', 'nations', url=url, read_csv_kwargs=dict(sep=","))

Ensure a RDF file from the internet and load it for usage (requires pip install rdflib)

import pystow
import rdflib

url = 'https://ftp.expasy.org/databases/rhea/rdf/rhea.rdf.gz'
rdf_graph: rdflib.Graph = pystow.ensure_rdf('rhea', url=url)

Also see pystow.ensure_excel(), pystow.ensure_rdf(), pystow.ensure_zip_df(), and pystow.ensure_tar_df().

If your data comes with a lot of different files in an archive, you can ensure the archive is downloaded and get specific files from it:

import numpy as np
import pystow

url = "https://cloud.enterprise.informatik.uni-leipzig.de/index.php/s/LHPbMCre7SLqajB/download/MultiKE_D_Y_15K_V1.zip"
# the path inside the archive to the file you want
inner_path = "MultiKE/D_Y_15K_V1/721_5fold/1/20210219183115/ent_embeds.npy"
with pystow.ensure_open_zip("kiez", url=url, inner_path=inner_path) as file:
    emb = np.load(file)

Also see pystow.module.ensure_open_lzma(), pystow.module.ensure_open_tarfile() and pystow.module.ensure_open_gz().

⚙️️ Configuration

By default, data is stored in the $HOME/.data directory. By default, the <app> app will create the $HOME/.data/<app> folder.

If you want to use an alternate folder name to .data inside the home directory, you can set the PYSTOW_NAME environment variable. For example, if you set PYSTOW_NAME=mydata, then the following code for the pykeen app will create the $HOME/mydata/pykeen/ directory:

import os
import pystow

# Only for demonstration purposes. You should set environment
# variables either with your .bashrc or in the command line REPL.
os.environ['PYSTOW_NAME'] = 'mydata'

# Get a directory (as a pathlib.Path) for ~/mydata/pykeen
pykeen_directory = pystow.join('pykeen')

If you want to specify a completely custom directory that isn't relative to your home directory, you can set the PYSTOW_HOME environment variable. For example, if you set PYSTOW_HOME=/usr/local/, then the following code for the pykeen app will create the /usr/local/pykeen/ directory:

import os
import pystow

# Only for demonstration purposes. You should set environment
# variables either with your .bashrc or in the command line REPL.
os.environ['PYSTOW_HOME'] = '/usr/local/'

# Get a directory (as a pathlib.Path) for /usr/local/pykeen
pykeen_directory = pystow.join('pykeen')

Note: if you set PYSTOW_HOME, then PYSTOW_NAME is disregarded.

X Desktop Group (XDG) Compatibility

While PyStow's main goal is to make application data less opaque and less hidden, some users might want to use the XDG specifications for storing their app data.

If you set the environment variable PYSTOW_USE_APPDIRS to true or True, then the appdirs or platformdirs package will be used to choose the base directory based on the user data dir option. This can still be overridden by PYSTOW_HOME.

🚀 Installation

The most recent release can be installed from PyPI with uv:

$ uv pip install pystow

or with pip:

$ python3 -m pip install pystow

The most recent code and data can be installed directly from GitHub with uv:

$ uv pip install git+https://github.com/cthoyt/pystow.git

or with pip:

$  python3 -m pip install git+https://github.com/cthoyt/pystow.git

👐 Contributing

Contributions, whether filing an issue, making a pull request, or forking, are appreciated. See CONTRIBUTING.md for more information on getting involved.

👋 Attribution

⚖️ License

The code in this package is licensed under the MIT License.

🍪 Cookiecutter

This package was created with @audreyfeldroy's cookiecutter package using @cthoyt's cookiecutter-snekpack template.

🛠️ For Developers

<details> <summary>See developer instructions</summary>

The final section of the README is for if you want to get involved by making a code contribution.

Development Installation

To install in development mode, use the following:

$ git clone git+https://github.com/cthoyt/pystow.git
$ cd pystow
$ uv pip install -e .

Alternatively, install using pip:

$ python3 -m pip install -e .

Updating Package Boilerplate

This project uses cruft to keep boilerplate (i.e., configuration, contribution guidelines, documentation configuration) up-to-date with the upstream cookiecutter package. Install cruft with either uv tool install cruft or python3 -m pip install cruft then run:

$ cruft update

More info on Cruft's update command is available here.

🥼 Testing

After cloning the repository and installing tox with uv tool install tox --with tox-uv or python3 -m pip install tox tox-uv, the unit tests in the tests/ folder can be run reproducibly with:

$ tox -e py

Additionally, these tests are automatically re-run with each commit in a GitHub Action.

📖 Building the Documentation

The documentation can be built locally using the following:

$ git clone git+https://github.com/cthoyt/pystow.git
$ cd pystow
$ tox -e docs
$ open docs/build/html/index.html

The documentation automatically installs the package as well as the docs extra specified in the [pyproject.toml](pyproje