minydra 🦎

Minimal Python command-line parser inspired by Facebook's Hydra + dot-accessible nested dictionaries.

Easily parse arbitrary arguments from the command line without dependencies:

pip install minydra

minydra is tested on Python 3.7, 3.8 and 3.9.

<br/> <p align="center"> <a href="#getting-started"><strong>Getting Started</strong></a>&nbsp;&nbsp;•&nbsp; <a href="#forcing-types"><strong>Forcing types</strong></a>&nbsp;&nbsp;•&nbsp; <a href="#minydict"><strong>MinyDict</strong></a>&nbsp;&nbsp;•&nbsp; <a href="#dumpingloading"><strong>Save config</strong></a>&nbsp;&nbsp;•&nbsp; <a href="#strict-mode"><strong>Prevent typos</strong></a>&nbsp;&nbsp;•&nbsp; <a href="#using-default-configurations"><strong>Use default configs</strong></a>&nbsp;&nbsp;•&nbsp; <a href="/examples"><strong>Examples</strong></a> </p> <br/>

Getting Started

examples/parser.py

from minydra.parser import Parser

if __name__ == "__main__":
    parser = Parser(
        verbose=0, # print received args
        allow_overwrites=False, # allow repeating args in the command-line
        warn_overwrites=True, # warn repeating args if they are allowed
        parse_env=True, # get environment variable
        warn_env=True, # warn if an environment variable is specified but not found
        defaults=None, # path to a MinyDict-loadable dictionary of default values for the args
        strict=True, # if `defaults` is provided, whether to allow new keys in the command-line
                     # or restrict to `defaults`' keys
        keep_special_kwargs=True, # `defaults` and `strict` can be set from the command-line
                                  # with `@defaults=` and `@strict=`. This argument decides if
                                  # you want to keep those keys in the final arguments.
    )
    args = parser.args.pretty_print().resolve().pretty_print() # notice .resolve() transforms dotted.keys into nested dicts

examples/resolved_args.py

from minydra import resolved_args

if __name__ == "__main__":
    args = resolved_args()
    args.pretty_print()

examples/demo.py examples/demo.json

from minydra import MinyDict, resolved_args
from pathlib import Path

if __name__ == "__main__":
    # parse arbitrary args in 1 line
    args = resolved_args()

    # override default conf
    if args.default:
        args = MinyDict.from_json(args.default).update(args)

    # protect args in the rest of the code execution
    args.freeze()

    # print the args in a nice orderly fashion
    args.pretty_print()

    # access args with dot/attribute access
    print(f'Using project "{args.log.project}" in {args.log.outdir}')

    # save configuration
    args.to_json(Path(args.log.outdir) / f"{args.log.project}.json")

examples/decorator.py

import minydra
from minydra.dict import MinyDict

@minydra.parse_args(verbose=0, allow_overwrites=False) # Parser's init args work here
def main(args: MinyDict) -> None:
    args.resolve().pretty_print()


if __name__ == "__main__":
    main()

<br/><br/>

Parsing

• Simple strings are parsed to float and int automatically.
• A single keyword will be interpreted as a positive flag.
• A single keyword starting with - will be interpreted as a negative flag.
• If parse_env is True, environment variables are evaluated.

$ python examples/decorator.py outdir=$HOME/project save -log learning_rate=1e-4 batch_size=64
╭───────────────────────────────────────────╮
│ batch_size    : 64                        │
│ learning_rate : 0.0001                    │
│ log           : False                     │
│ outdir        : /Users/victor/project     │
│ save          : True                      │
╰───────────────────────────────────────────╯

• dotted keys will be resolved to nested dictionary keys:

$ python examples/decorator.py server.conf.port=8000
╭────────────────────╮
│ server             │
│ │conf              │
│ │ │port : 8000     │
╰────────────────────╯

• Using ast.literal_eval(value), minydra will try and parse more complex values for arguments as lists or dicts. Those should be specified as strings:

$ python examples/decorator.py layers="[1, 2, 3]" norms="{'conv': 'batch', 'epsilon': 1e-3}"
╭──────────────────────────────────────────────────╮
│ layers : [1, 2, 3]                               │
│ norms  : {'conv': 'batch', 'epsilon': 0.001}     │
╰──────────────────────────────────────────────────╯

<br/>

Forcing types

Adding ___<type> to a key will force this type to the value. Notice how 01 is parsed to an integer 1 but 04 is parsed to a string (as specified) "04", and hello is parsed to a list, not kept as a string

$ python examples/decorator.py n_jobs___str=04 job=01 chips___list=hello
╭────────────────────────────────────────╮
│ chips  : ['h', 'e', 'l', 'l', 'o']     │
│ job    : 1                             │
│ n_jobs : 04                            │
╰────────────────────────────────────────╯

Known types are defined in Parser.known_types and the separator (___) in Parser.type_separator

In [1]: from minydra import Parser

In [2]: Parser.known_types
Out[2]: {'bool', 'float', 'int', 'str'}

In [3]: Parser.type_separator
Out[3]: '___'

<br/>

Command-line configuration

You can configure the Parser from the command-line using special @ arguments. In other words, all __init__(self, ...) arguments can be set from the command-line with @argname=new_value.

In particular if you run python examples/decorator.py @defaults=./examples/demo.json you will see:

╭──────────────────────────────────────╮
│ @defaults : ./examples/demo.json     │
│ log                                  │
│ │logger                              │
│ │ │log_level   : DEBUG               │
│ │ │logger_name : minydra             │
│ │outdir  : /some/path                │
│ │project : demo                      │
│ verbose   : False                    │
╰──────────────────────────────────────╯

But if you add @strict=false @keep_special_kwargs=false you will now have:

$ python examples/decorator.py @defaults=./examples/demo.json @strict=false @keep_special_kwargs=false
╭──────────────────────────────╮
│ log                          │
│ │logger                      │
│ │ │log_level   : DEBUG       │
│ │ │logger_name : minydra     │
│ │outdir  : /some/path        │
│ │project : demo              │
│ verbose : False              │
╰──────────────────────────────╯

(you need to have @strict=false since @keep_special_kwargs is unknown in demo.json. It would not be the case if strict=false had been used in the script itself (but it can be overridden from the command-line!))

<br/><br/>

MinyDict

Minydra's args are a custom lightweight wrapper around native dict which allows for dot access (args.key), resolving dotted keys into nested dicts and pretty printing sorted keys in a box with nested dicts indented. If a key does not exist, it will not fail, rather return None (as dict.get(key, None)).

a MinyDict inherits from dict so usual methods work .keys(), .items() etc.

In [1]: from minydra.dict import MinyDict

In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).pretty_print(); args
╭──────────────────────────────╮
│ foo          : bar           │
│ yes.no.maybe : idontknow     │
╰──────────────────────────────╯
Out[2]: {'foo': 'bar', 'yes.no.maybe': 'idontknow'}

In [3]: args.resolve().pretty_print(); args
╭──────────────────────────╮
│ foo : bar                │
│ yes                      │
│ │no                      │
│ │ │maybe : idontknow     │
╰──────────────────────────╯
Out[3]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}

In [4]: args.yes.no.maybe
Out[4]: "idontknow"

In [5]: "foo" in args
Out[5]: True

In [6]: "rick" in args
Out[6]: False

In [7]: args.morty is None
Out[7]: True

In [8]: args.items()
Out[8]: dict_items([('foo', 'bar'), ('yes', {'no': {'maybe': 'idontknow'}})])

<br/>

Dumping/Loading

You can save and read MinyDict to/from disk in 3 formats: json and pickle without dependencies, yaml with the PyYAML dependency (pip install minydra[yaml]).

Methods to_pickle, to_json and to_yaml have 3 arguments:

1. file_path as a str or pathlib.Path which is resolved:
   1. expand env variable ($MYDIR for instance)
   2. expand user (~)
   3. make absolute
2. return_path which defaults to True. If those methods return the path of the created file
3. allow_overwrites which defaults to True. If False and path exists, a FileExistsError will be raised. Otherwise creates/overwrites the file at file_path
4. verbose which defaults to 0. If >0 prints the path of the created object

Note:

• to/from_yaml will fail with a ModuleNotFoundError if PyYAML is not installed.
• the json standard does not accept ints as keys in dictionaries so {3: 2} would be dumped -- and therefore loaded -- as {"3": 2}.

In [1]: from minydra.dict import MinyDict

In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).resolve(); args
Out[2]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}

In [3]: json_file_path = args.to_json("./args.json")

In [4]: yaml_file_path = args.to_yaml("./args.yaml")

In [5]: pkl_file_path = args.to_pickle("./args.pkl")

In [6]: _ = args.to_json("./args.json", verbose=1) # verbose argument prints the path
Json dumped to: /Users/victor/Documents/Github/vict0rsch/minydra/args.json