<p align="center"><img src="https://raw.githubusercontent.com/eladrich/pyrallis/master/docs/pyrallis_logo.png" alt="logo" width="70%" /></p> <p align="center"> <a href="https://badge.fury.io/py/pyrallis"><img src="https://badge.fury.io/py/pyrallis.svg" alt="PyPI version" height="18"></a> <a href="https://github.com/eladrich/pyrallis/actions/workflows/pytest.yml"><img src="https://github.com/eladrich/pyrallis/actions/workflows/pytest.yml/badge.svg" alt="PyTest" height="18"></a> <a href="https://pepy.tech/project/pyrallis"><img src="https://pepy.tech/badge/pyrallis" alt="Downloads" height="18"></a> <a href="#contributors-"><img src="https://img.shields.io/badge/all_contributors-2-orange.svg" alt="All Contributors" height="18"></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT" height="18"></a> </p>

Pyrallis - Simple Configuration with Dataclasses

Pyrausta (also called pyrallis (πυραλλίς), pyragones) is a mythological insect-sized dragon from Cyprus.

Pyrallis is a simple library, derived from simple-parsing and inspired by Hydra, for automagically creating project configuration from a dataclass.

<p align="center"><img src="https://github.com/eladrich/pyrallis/raw/master/docs/argparse2pyrallis.gif" alt="GIF" width="100%" /></p>

Why pyrallis?

With pyrallis your configuration is linked directly to your pre-defined dataclass, allowing you to easily create different configuration structures, including nested ones, using an object-oriented design. The parsed arguments are used to initialize your dataclass, giving you the typing hints and automatic code completion of a full dataclass object.

My First Pyrallis Example 👶

There are several key features to pyrallis but at its core pyrallis simply allows defining an argument parser using a dataclass.

from dataclasses import dataclass
import pyrallis


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    workers: int = 8 # The number of workers for training
    exp_name: str = 'default_exp' # The experiment name

def main():
    cfg = pyrallis.parse(config_class=TrainConfig)
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')

The arguments can then be specified using command-line arguments, a yaml configuration file, or both.

$ python train_model.py --config_path=some_config.yaml --exp_name=my_first_exp
Training my_first_exp with 42 workers...

Assuming the following configuration file

exp_name: my_yaml_exp
workers: 42

Key Features

Building on that design pyrallis offers some really enjoyable features including

• Builtin IDE support for autocompletion and linting thanks to the structured config. 🤓
• Joint reading from command-line and a config file, with support for specifying a default config file. 😍
• Support for builtin dataclass features, such as __post_init__ and @property 😁
• Support for nesting and inheritance of dataclasses, nested arguments are automatically created! 😲
• A magical @pyrallis.wrap() decorator for wrapping your main class 🪄
• Easy extension to new types using pyrallis.encode.register and pyrallis.decode.register 👽
• Easy loading and saving of existing configurations using pyrallis.dump and pyrallis.load 💾
• Magical --help creation from dataclasses, taking into account the comments as well! 😎
• Support for multiple configuration formats (yaml, json,toml) using pyrallis.set_config_type ⚙️

Getting to Know The pyrallis API in 5 Simple Steps 🐲

The best way to understand the full pyrallis API is through examples, let's get started!

🐲 1/5 pyrallis.parse for dataclass Parsing 🐲

Creation of an argparse configuration is really simple, just use pyrallis.parse on your predefined dataclass.

from dataclasses import dataclass, field
import pyrallis


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    # The number of workers for training
    workers: int = field(default=8)
    # The experiment name
    exp_name: str = field(default='default_exp')


def main():
    cfg = pyrallis.parse(config_class=TrainConfig)
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')


if __name__ == '__main__':
    main()

Not familiar with dataclasses? you should probably check the Python Tutorial and come back here.

The config can then be parsed directly from command-line

$ python train_model.py --exp_name=my_first_model
Training my_first_model with 8 workers...

Oh, and pyrallis also generates an --help string automatically using the comments in your dataclass 🪄

$ python train_model.py --help
usage: train_model.py [-h] [--config_path str] [--workers int] [--exp_name str]

optional arguments:
  -h, --help      show this help message and exit
  --config_path str    Path for a config file to parse with pyrallis (default:
                  None)

TrainConfig:
   Training config for Machine Learning

  --workers int   The number of workers for training (default: 8)
  --exp_name str  The experiment name (default: default_exp)

🐲 2/5 The pyrallis.wrap Decorator 🐲

Don't like the pyrallis.parse syntax?

def main():
    cfg = pyrallis.parse(config_class=TrainConfig)
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')

One can equivalently use the pyrallis.wrap syntax 😎

@pyrallis.wrap()
def main(cfg: TrainConfig):
    # The decorator automagically uses the type hint to parsers arguments into TrainConfig
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')

We will use this syntax for the rest of our tutorial.

🐲 3/5 Better Configs Using Inherent dataclass Features 🐲

When using a dataclass we can add additional functionality using existing dataclass features, such as the post_init mechanism or @properties :grin:

from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
import pyrallis


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name


@pyrallis.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.exp_name}...')
    print(f'\tUsing {cfg.workers} workers and {cfg.eval_workers} evaluation workers')
    print(f'\tSaving to {cfg.exp_dir}')

$ python -m train_model.py --exp_name=my_second_exp --workers=42
Training my_second_exp...
    Using 42 workers and 42 evaluation workers
    Saving to /share/experiments/my_second_exp

Notice that in all examples we use the explicit dataclass.field syntax. This isn't a requirement of pyrallis but rather a style choice. As some of your arguments will probably require dataclass.field (mutable types for example) we find it cleaner to always use the same notation.

🐲 4/5 Building Hierarchical Configurations 🐲

Sometimes configs get too complex for a flat hierarchy 😕, luckily pyrallis supports nested dataclasses 💥

@dataclass
class ComputeConfig:
    """ Config for training resources """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers


@dataclass
class LogConfig:
    """ Config for logging arguments """
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name

# TrainConfig will be our main configuration class.
# Notice that default_factory is the standard way to initialize a class argument in dataclasses

@dataclass
class TrainConfig:
    log: LogConfig = field(default_factory=LogConfig)
    compute: ComputeConfig = field(default_factory=ComputeConfig)

@pyrallis.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.log.exp_name}...')
    print(f'\tUsing {cfg.compute.workers} workers and {cfg.compute.eval_workers} evaluation workers')
    print(f'\tSaving to {cfg.log.exp_dir}')

The argument parse will be updated accordingly

$ python train_model.py --log.exp_name=my_third_exp --compute.eval_workers=2
Training my_third_exp...
    Using 8 workers and 2 evaluation workers
    Saving to /share/experiments/my_third_exp

🐲 5/5 Easy Serialization with pyrallis.dump 🐲

As your config get longer you will probably want to start working with configuration files. Pyrallis supports encoding a dataclass configuration into a yaml file 💾

The command pyrallis.dump(cfg, open('run_config.yaml','w')) will result in the following yaml file

compute:
  eval_workers: 2
  workers: 8
log:
  exp_name: my_third_exp
  exp_root: /share/experiments

pyrallis.dump extends yaml.dump and uses the same synta