Easily parse arbitrary arguments from the command line without dependencies
Project description
minydra ๐ฆ
Minimal Python command-line parser inspired by Facebook's Hydra + dot-accessible nested dictionaries.
Easily parse arbitrary arguments from the command line without dependencies:
pip install minydra
minydra
is tested on Python 3.7
, 3.8
and 3.9
.
Getting Startedย โขย Forcing typesย โขย MinyDictย โขย Save configย โขย Prevent typos
Getting Started
from minydra.parser import Parser
if __name__ == "__main__":
parser = Parser(
verbose=0, # print received args
allow_overwrites=False, # allow repeating args in the command-line
warn_overwrites=True, # warn repeating args if they are allowed
parse_env=True, # get environment variable
warn_env=True, # warn if an environment variable is specified but not found
)
args = parser.args.pretty_print().resolve().pretty_print() # notice .resolve() transforms dotted.keys into nested dicts
from minydra import resolved_args
if __name__ == "__main__":
args = resolved_args()
args.pretty_print()
examples/demo.py
examples/demo.json
from minydra import MinyDict, resolved_args
from pathlib import Path
if __name__ == "__main__":
# parse arbitrary args in 1 line
args = resolved_args()
# override default conf
if args.default:
args = MinyDict.from_json(args.default).update(args)
# protect args in the rest of the code execution
args.freeze()
# print the args in a nice orderly fashion
args.pretty_print()
# access args with dot/attribute access
print(f'Using project "{args.log.project}" in {args.log.outdir}')
# save configuration
args.to_json(Path(args.log.outdir) / f"{args.log.project}.json")
import minydra
from minydra.dict import MinyDict
@minydra.parse_args(verbose=0, allow_overwrites=False) # Parser's init args work here
def main(args: MinyDict) -> None:
args.resolve().pretty_print()
if __name__ == "__main__":
main()
Parsing
- Simple strings are parsed to
float
andint
automatically. - A single keyword will be interpreted as a positive flag.
- A single keyword starting with
-
will be interpreted as a negative flag. - If
parse_env
isTrue
, environment variables are evaluated.
$ python decorator.py outdir=$HOME/project save -log learning_rate=1e-4 batch_size=64
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ batch_size : 64 โ
โ learning_rate : 0.0001 โ
โ log : False โ
โ outdir : /Users/victor/project โ
โ save : True โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
- dotted keys will be resolved to nested dictionary keys:
$ python examples/decorator.py server.conf.port=8000
โญโโโโโโโโโโโโโโโโโโโโโฎ
โ server โ
โ โconf โ
โ โ โport : 8000 โ
โฐโโโโโโโโโโโโโโโโโโโโโฏ
- Using
ast.literal_eval(value)
,minydra
will try and parse more complex values for arguments as lists or dicts. Those should be specified as strings:
$ python examples/decorator.py layers="[1, 2, 3]" norms="{'conv': 'batch', 'epsilon': 1e-3}"
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ layers : [1, 2, 3] โ
โ norms : {'conv': 'batch', 'epsilon': 0.001} โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Forcing types
Adding ___<type>
to a key will force this type to the value. Notice how 01
is parsed to an integer 1
but 04
is parsed to a string (as specified) "04"
, and hello
is parsed to a list
, not kept as a string
$ python examples/decorator.py n_jobs___str=04 job=01 chips___list=hello
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ chips : ['h', 'e', 'l', 'l', 'o'] โ
โ job : 1 โ
โ n_jobs : 04 โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Known types are defined in Parser.known_types
and the separator (___
) in Parser.type_separator
In [1]: from minydra import Parser
In [2]: Parser.known_types
Out[2]: {'bool', 'float', 'int', 'str'}
In [3]: Parser.type_separator
Out[3]: '___'
MinyDict
Minydra's args are a custom lightweight wrapper around native dict
which allows for dot access (args.key
), resolving dotted keys into nested dicts and pretty printing sorted keys in a box with nested dicts indented. If a key does not exist, it will not fail, rather return None (as dict.get(key, None)
).
a MinyDict
inherits from dict
so usual methods work .keys()
, .items()
etc.
In [1]: from minydra.dict import MinyDict
In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).pretty_print(); args
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ foo : bar โ
โ yes.no.maybe : idontknow โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Out[2]: {'foo': 'bar', 'yes.no.maybe': 'idontknow'}
In [3]: args.resolve().pretty_print(); args
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ foo : bar โ
โ yes โ
โ โno โ
โ โ โmaybe : idontknow โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Out[3]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [4]: args.yes.no.maybe
Out[4]: "idontknow"
In [5]: "foo" in args
Out[5]: True
In [6]: "rick" in args
Out[6]: False
In [7]: args.morty is None
Out[7]: True
In [8]: args.items()
Out[8]: dict_items([('foo', 'bar'), ('yes', {'no': {'maybe': 'idontknow'}})])
Dumping/Loading
You can save and read MinyDict
to/from disk in 3 formats: json
and pickle
without dependencies, yaml
with the PyYAML
dependency (pip install minydra[yaml]
).
Methods to_pickle
, to_json
and to_yaml
have 3 arguments:
file_path
as astr
orpathlib.Path
which is resolved:- expand env variable (
$MYDIR
for instance) - expand user (
~
) - make absolute
- expand env variable (
return_path
which defaults toTrue
. IfTrue
to_json
andto_pickle
return the path of the created objectallow_overwrites
which defaults toTrue
. IfFalse
andpath
exists, aFileExistsError
will be raised. Otherwise creates/overwrites the file atfile_path
verbose
which defaults to0
. If>0
prints the path of the created object
Note:
to/from_yaml
will fail with aModuleNotFoundError
ifPyYAML
is not installed.- the
json
standard does not accept ints as keys in dictionaries so{3: 2}
would be dumped -- and therefore loaded -- as{"3": 2}
.
In [1]: from minydra.dict import MinyDict
In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).resolve(); args
Out[2]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [3]: json_file_path = args.to_json("./args.json")
In [4]: yaml_file_path = args.to_yaml("./args.yaml")
In [5]: pkl_file_path = args.to_pickle("./args.pkl")
In [6]: _ = args.to_json("./args.json", verbose=1) # verbose argument prints the path
Json dumped to: /Users/victor/Documents/Github/vict0rsch/minydra/args.json
In [7]: MinyDict.from_json("args.json")
Out[7]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [8]: assert (
MinyDict.from_yaml(yaml_file_path)
== MinyDict.from_json(json_file_path)
== MinyDict.from_pickle(pkl_file_path)
== args
)
python examples/dumps.py path="./myargs.pkl" format=pickle cleanup
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ cleanup : True โ
โ format : pickle โ
โ path : ./myargs.pkl โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Dumped args to /Users/victor/Documents/Github/vict0rsch/minydra/myargs.pkl
Cleaning up
Strict Mode
To prevent typos from the command-line, the MinyDict.update
method has a strict mode: updating a MinyDict
with another one using strict=True
will raise a KeyError
if the key does not already exist:
from minydra import MinyDict, resolved_args
if __name__ == "__main__":
# parse arbitrary args in 1 line
args = resolved_args()
# override default conf
if args.default:
path = args.default
# delete otherwise it will be used to update the conf which does not have
# "default" as a key, therefore raising a KeyError in strict mode
del args.default
args = MinyDict.from_json(path).update(args, strict=True)
args.pretty_print()
No typo:
$ python examples/strict.py default=./examples/demo.json log.logger.log_level=INFO
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ log โ
โ โlogger โ
โ โ โlog_level : INFO โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/path โ
โ โproject : demo โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Typo:
$ python examples/strict.py default=./examples/demo.json log.logger.log_leveel=INFO
Traceback (most recent call last):
File "/Users/victor/Documents/Github/vict0rsch/minydra/examples/strict.py", line 13, in <module>
args = MinyDict.from_json(path).update(args, strict=True)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
self[k].update(v, strict=strict)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
self[k].update(v, strict=strict)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 100, in update
raise KeyError(
KeyError: 'Cannot create a non-existing key in strict mode ({"log_leveel":INFO}).'
pretty_print
Prints the MinyDict
in a box, with dicts properly indented. A few arguments:
indents
, which defaults to2
: the amount of indentation for nested dictionariessort_keys
, which defaults toTrue
: whether or not to alphabetically sort the keys before printing
to_dict
To produce a native Python dict
, use args.to_dict()
Protected attributes
MinyDict
's methods (including the dict
class's) are protected, they are read-only and you cannot therefore set attributes with there names, like args.get = 2
. If you do need to have a get
argument, you can access it through items: args["get"] = 2
.
Try with examples/protected.py
:
python examples/protected.py server.conf.port=8000 get=3
โญโโโโโโโโโโโโโโโโโโโโโฎ
โ get : 3 โ
โ server โ
โ โconf โ
โ โ โport : 8000 โ
โฐโโโโโโโโโโโโโโโโโโโโโฏ
<built-in method get of MinyDict object at 0x100ccd4a0>
3
dict_items([('get', 3), ('server', {'conf': {'port': 8000}})])
{'conf': {'port': 8000}}
Tests
Run tests and pre-commit checks (isort
, black
, flake8
) with
$ pip install -r requirements-test.txt
$ pre-commit run --all-files
$ pytest -vv --cov=minydra tests/
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.