naapc 1.7.0

Nested Automated Argument Parsing Configuration (NAAPC).

Nested Automated Argument Parsing Configuration (NAAPC)

NAAPC contains two classes: NConfig and NDict. NDict provides method to easily manipulate nested dictionaries. NConfig is a subclass of NDict and can automatically modify configurations according to CLI arguments.

Installation

pip install naapc

Or from source code:

pip install .

Typical Usage.

Assume a configuration file test.yaml:

task:
  task: classification
train:
  pretrain: false
  loss_args:
    lr: 0.1

The typical usage is as follows:

from naapc import nconfig
from argparse import parser

parser.add_argument("-c", type=str, dest="config")
args, extra_args = parser.parse_known_args(["-c", "test.yaml", "--task;task", "regression", "--train;loss_args;lr", "0.2", "--train;pretrain", "1", "--others", "other"])

with open(args.config, "r") as f:
  raw = yaml.safe_load(f)
config = nconfig(raw)
extra_args = config.parse_update(parser, extra_args)

The resulting configurations:

task:
  task: regression
train:
  pretrain: true
  loss_args:
    lr: 0.2

The data type is determined by the type in the configuration file. The boolean data is treated as integer number 1 and 0 during parsing.

You may custom the arguments:

task:
  task: regression
train:
  pretrain: true
  loss_args:
    lr: 0.2
_ARGUMENT_SPECIFICATION:
  task;task:
    flag: --task
    choices: ["regression", "classification"]
  train;lr:
    flag: lr

ndict Usages

for a sample configuration test.yaml file:

task:
  task: classification
train:
  loss_args:
    lr: 0.1

and a sample list configuration test_list.yaml file:

l:
- d:
    task:
      task: classification
- d:
    train:
      loss_args:
        lr: 0.1

from naapc import ndict

with open("test.yaml", "r") as f:
  raw = yaml.safe_load(f)
nd = ndict(raw["d"], delimiter=";")
nd1 = ndict.from_flatten_dict(nd.flatten_dict) # nd1 == nd
nd2 = ndict.from_list_of_dict(raw["l"]) # nd2 == nd1 == nd

"task;path" in nd                      # "task" in raw and "path" in raw["task"]
del nd["task;path"]                    # del raw["task]["path]
nd["task;path"] = "cwd"                # raw["task"]["path"] = Path(".").absolute()
nd.flatten_dict                        # {"task;task": "classification", "train;loss_args;lr": 0.1}
nd.flatten_dict_split                  # raw["l"]
nd.paths                               # ["task", "task;task", "train", "train;loss_args", "train;loss_args;lr"]
nd.get("task;seed", 1)                 # raw["task"].get("seed", 1)
nd.raw_dict                            # raw
nd.size                                # len(nd.flatten_dict)
nd.update({"task;here": "there"})      # raw["task]["here] = "there
nd.items()                             # raw.items()
nd.keys()                              # raw.keys()
nd.values()                            # raw.values()
len(nd)                                # len(raw)
bool(nd)                               # len(nd) > 0
nd1 == nd                              # nd1.flatten_dict == nd.flatten_dict
nd1["task;path"] = "xcwd"
nd1["task;extra"] = "ecwd"
nd["train;epochs"] = 100
nd.compare_dict(nd1)                   # {"task;path": ("cwd", "xcwd"), "task;extra": (None, ecwd), "train;epochs": (100, None)}
nd.is_matched(
        {
            "task;path": "ecwd", 
            "train;epochs": "!QUERY d[path] == d['train;minimum_epochs']"
        }
    )                                  # Test if this dictionary is what you want.

nconfig Usage

nconfig only supports int, str, float, bool, and list of these types. The nconfig automatically checks data type when modifications are applied. Note that argument specification ("_ARGUMENT_SPECIFICATION") does not count as part of the configurations but will be saved when use save() method. The path specified as "_IGNORE_IN_CLI" will not be added to the parser.

config.save("path.yaml")               # Save configurations as a yaml file
config.add_to_argparse(parser)         # Generate cli arguments for every configuration.
config.parse_update(parser, args)      # Parse cli arguments and update corresponding configuration. Extra arguments will be returned.

Typical specifications are as follows:

_ARGUMENT_SPECIFICATION:
  task;task:
    flag: --task
  task;seed:
    flag: --seed
  task;device:
    flag: -d
  data;dataset:
    choices: ["cifar", "imagenet", "asap"]
  log;comet_ml_key:
    _IGNORE_IN_CLI

Other functionalities are the same to NDict.

Typing

Add a type

NestedOrDict = Union[ndict, dict]