parser-binding 0.1.0

The parser to bind the dataclasses conviniently.

Documentation

Translated from 中文文档 by GPT 3.5

A tool for quickly binding dataclass classes to ArgumentParser to achieve rapid customization of command-line parameters. This binding supports most parameters of ArgumentParser, such as help documentation, whether it is a required parameter, multiple values, etc., as detailed below. At the same time, this tool supports Python's type hints as much as possible.

Quick Start

Python version: >=3.7

pip install parser-binding

demo.py：

from dataclasses import dataclass, field
from enum import Enum

from parser_binding import BindingParser, Field


class LogLevel(Enum):
    DEBUG = 'debug'
    INFO = 'info'
    WARNING = 'warning'
    ERROR = 'error'


@dataclass
class TestOptions:

    input_file: str = Field(
        default=None, aliases=['i'], help='The input file to read.'
    )
    workers: int = 1
    logging_level: LogLevel = LogLevel.WARNING

    verbose: bool = False


if __name__ == '__main__':
    parser = BindingParser(TestOptions)

    options = parser.parse_into_dataclasses((TestOptions, ))

    print(options)

Execute python demo.py -h, you will get the following output:

usage: demo.py [-h] [-i INPUT_FILE] [--workers WORKERS] [--logging-level {debug,info,warning,error}] [--with-verbose]

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE, --input_file INPUT_FILE
                        The input file to read. Optional. Default `None`.
  --workers WORKERS     Optional. Default `1`.
  --logging-level {debug,info,warning,error}, --logging_level {debug,info,warning,error}
                        Optional. Default `warning`.
  --with-verbose, --with_verbose
                        Optional. Default `False` as verbose disabled.

Execute python demo.py -i ./test.txt --workers 2 --logging-level debug --with-verbose, and you will automatically get an instance of the dataclass with corresponding parameters read from the command-line:

_MergedDataClass@1704877855657(input_file='./test.txt', workers=2, logging_level=<LogLevel.DEBUG: 'debug'>, verbose=True)

Formatting Guide

Command-line parameter names will be formatted into three types:

• --xxx-yyy, when the class property name contains _, it will be converted to a command-line parameter name starting with -- and separated by -.
• --xxx_yyy, when the class property name contains _, it will be converted to a command-line parameter name starting with -- and separated by _ as an alternative.
• -x, shortcut option support, when the property name is a single character or contains a single character in the specified alias, it will use the shortcut option - as one of the parameters.

Currently, it supports most types in Python:

• Basic Types

  • int: Integer type, automatically converts command-line parameters to integers.
  • float: Floating-point type, automatically converts command-line parameters to floating-point numbers.
  • str: String type, automatically converts command-line parameters to strings.
  • bytes: Byte type, supports direct conversion of strings in UTF-8 encoding, does not support other complex data, if other conversions are needed, it can be implemented by specifying the type.
  • bool: Boolean type, automatically adds a switch option to the command-line, the switch name is influenced by the default value; when the default value is True, the switch name is formatted as --without-xxx-yyy and --without_xxx_yyy, when the default value is False, the switch name is formatted as starting with --with. Note: Boolean value types will only be formatted as switches and do not need to pass parameters on the command-line. Pay attention to the change in parameter names caused by changes in default values.

• Types with Generic Annotations: When using generic annotations, please explicitly specify the type inside the generic. Complex types such as Union[str, int] are not directly supported：

  • Optional，Used to define optional properties, provide a default value when using this annotation. For all properties with default values, the corresponding parameters on the command-line will be treated as optional parameters and provide default values.
  • List/list: When using the List annotation, please specify the type of the list elements explicitly, such as List[int]. When using this type, the command-line parameter corresponding to it will be set as a multi-value type, and spaces will be used as the default separator for each value, such as --multiple 1 2 3. The explicit element type will format each value passed in as the target type, that is, each element in the List[int] property will be of type int, and not the result of List[str]. If using the list annotation, there is no additional type mapping for the passed elements by default. If additional type mapping is needed, please specify the type mapping using the type parameter (as shown in the example later).
  • Tuple/tuple: Same as List or list, the difference lies in the final collection class output.
  • Set/set: Same as above.
  • Deueue/dequeue: Same as above.
  • Queue: Same as above. These collection classes default to using a space as the separator between multiple values. If a custom separator is needed, it should be specified using sep (as shown in the subsequent example).
  • Dict/dict: When using Dict as the property type, please explicitly specify the key and value types. If using dict as an annotation, no mapping will be done for the key/value types. By default, the command-line parameters passed in will be treated as a JSON-string for parsing. If it cannot be parsed, it will try to parse it as a JSON file (in this case, pass in a JSON file directory). If both methods fail, it will result in failure.
  • Enum: When using an enumeration class, the command-line will construct an optional value parameter, and the input will be limited to a fixed list of values. The command-line will pass in a string, but during parsing, it will get a corresponding enumeration and provide it to the dataclass instance initialization.
  • Literal: Requires python >= 3.8 support, equivalent to Enum, but does not produce an enumeration.

• Complex Types

  • File type: For ease of reading from the command-line or setting output devices, you can quickly open files using this method to reduce the overhead of calling open in the code.
  • Nested or unknown types: To further expand the scenarios of complex types, support custom type conversion for command-line passing values through the Field and type parameters.

Usage Examples

Basic Type Scenario

from dataclasses import dataclass

from parser_binding import BindingParser


@dataclass
class TestOption():
    required_float: float
    a: int = 0
    string_type: str = None
    switch: bool = False
    bytes_data: bytes = None


parser = BindingParser(TestOption)
options = parser.parse_into_dataclasses((TestOption, ))

print(options)

python3 basic-demo.py --required-float 0.1 -a 10 --string-type 'Hello Parser!' --with-switch --bytes-data 'Hello Parser!'

Result:

_MergedDataClass@1704880154551(required_float=0.1, a=10, string_type='Hello Parser!', switch=True, bytes_data=b'Hello Parser!')

You can observe that --required-float is a required parameter; otherwise, parsing will fail. For parameters without default values, parser-binding will consider them as required parameters; otherwise, provide default values.

typing Annotation Types

from dataclasses import dataclass
from typing import List

from parser_binding import BindingParser


@dataclass
class TestOption:
    data: List[int] = None
    items: list = None


parser = BindingParser(TestOption)

opt = parser.parse_into_dataclasses((TestOption, ))

print(opt)

Execute python list-demo.py --data 1 2 3 --items 1 2 3 to parse and get the result _MergedDataClass@1704880514613(data=[1, 2, 3], items=['1', '2', '3'])。

From the above difference, if using the List[int] annotation, each element in the list will be converted to an integer, otherwise, no conversion will be performed. If conversion is required, the type should be specified, as shown below, changing the items annotation to:

items: list = Field(default=None, type=int)

Execute python list-demo.py --data 1 2 3 --items 1 2 3, and the parsed result will be _MergedDataClass@1704881428343(data=[1, 2, 3], items=[1, 2, 3]), where the type is now correctly converted.

For set, tuple, deque, queue, and other collection types, the effect is the same.

Note: If the property is annotated with list, tuple, set, queue, etc., the command-line will automatically convert the corresponding parameter to a multi-value parameter separated by spaces. At this time, if you specify type using Field, type should correspond to the class of the elements, not the final property collection type. For example, in the above Field(default=None, type=int), type specifies the element type as int, i.e., the property type is List[int].

Given the default behavior of command-line multi-value parameters, which default to being separated by spaces, further support for more separators can be achieved through Filed by specifying sep. For example: items: tuple = Field(sep=',', default=None, type=int), will result in a Tuple[int] parsing result, and the input parameters will be separated by ,. In this case, you should run the following command: python list-demo.py --data 1 2 3 --items 1,2,3, and the parsing result will be _MergedDataClass@1704893904876(data=(1, 2, 3), items=(1, 2, 3)).。

parser-binding also supports dictionary types, used to support JSON-like formats. When using Dict as the type annotation, you need to specify the key and value types so that they can be correctly converted. If the JSON string contains key-value pairs of multiple types, you can directly use dict for annotation, as shown in the following example:

from dataclasses import dataclass
from typing import Dict

from parser_binding import BindingParser, Field


@dataclass
class TestOption:
    data: Dict[str, int] = None
    items: dict = None


parser = BindingParser(TestOption)

opt = parser.parse_into_dataclasses((TestOption, ), )

print(opt)

Execute python dict-demo.py --data '{"1": "1", "2": "2"}' --items '{"1": "1", "2": "2"}', and the parsing result will be _MergedDataClass@1704944849798(data={'1': 1, '2': 2}, items={'1': '1', '2': '2'})。

From the above parsing result, when using dict for annotation, the key-type and value-type will remain consistent with the original JSON, but when using Dict and specifying types, the key/value will be converted to the corresponding types.

---

Enum Types

Enum types allow the command-line to define an optional value parameter, restricting the input to a fixed list of values, making the selection from the command-line effective.

As shown in the example below:

from dataclasses import dataclass
from enum import Enum

from parser_binding import parse_args


class Mode(Enum):
    train: str = 'train'
    eval: str = 'eval'


@dataclass
class TestOption:
    mode: Mode = Mode.train


args = parse_args((TestOption, ))

print(args.mode)

At this point, check the usage with usage: enum-demo.py [-h] [--mode {train,eval}], indicating that only the values train and eval are supported.

Enum classes are equivalent in effect to annotations like Literal。

Complex Type

To further support more scenarios, such as nested collections, multi-type inference, file reading and writing, etc., parser-binding supports implementation through Field combined with type.

We allow the use of Optional or Union[x, None] to declare optional properties, but we do not yet support multi-type inference, such as Union[str, int], or nested types like List[List[int]]. These types are considered complex types and require additional type configuration!

By default, for complex types when no type configuration is specified, we won't perform any processing. Consequently, the parameter values read will be of type str. This may lead to unexpected results, so please use caution.

Recommended: Provide the help argument for complex types to help the command-line understand how to pass values.

---

Nested Types

In complex scenarios, we use List[List[int]] as an example for parsing, as shown in the following code:

from dataclasses import dataclass
from typing import List

from parser_binding import parse_args


@dataclass
class TestOption:
    data: List[List[int]] = None


args = parse_args((TestOption, ))

print(args)

At this time, the data property is not given a type argument and is treated as a complex type, resulting in the following warning:

UserWarning: The filed "data" is complex but there is no type specified, this could make an error.

At this time, directly passing values from the command-line will result in the result: _MergedDataClass@1704955154188(data='1,2,3'), indicating that data does not match the expected type List[List[int]].

Therefore, a reasonable practice should be:

from dataclasses import dataclass
from typing import List

from parser_binding import Field, parse_args


@dataclass
class TestOption:
    data: List[List[int]] = Field(
        default=None,
        type=lambda x:
        list(map(lambda x: list(map(int, x.split(':'))), x.split(',')))
    )


args = parse_args((TestOption, ))

print(args)

Executing python complex-demo.py --data 1:4:5,2,3 will result in _MergedDataClass@1704955285471(data=[[1, 4, 5], [2], [3]]), which aligns with the expected type annotations.

---

File Type

To facilitate the input of files from the command line, parser-binding supports simple text reading and writing methods, making it easy to obtain file instances directly from a data class without the need for open method calls. Here is an example:

from dataclasses import dataclass
from typing import IO, Iterable, TextIO

from parser_binding import Field, parse_args


@dataclass
class TestOption:
    in_file: Iterable[str] = Field(
        default=None, file=True, file_mode='r', file_encoding='utf-8'
    )
    # in_file: IO[str] = Field(default=None, file='r', file_encoding='utf-8')
    out_file: IO[str] = Field(
        default='-', file_mode='w', file_encoding='utf-8'
    )


args = parse_args((TestOption, ))

for l in args.in_file:
    print('o', l.strip(), sep='\t', file=args.out_file)

For file handling, there are two annotation methods, as seen in the example above with the in_file property. The difference lies in whether the file argument needs to be specified as True when using the IO annotation. Otherwise, it must be specified as True, or else it will be treated as a complex type.

Suppose the content of the existing test-file.txt is:

a
b
c

By executing python file-demo.py --in-file test-file.txt, the above code will produce the output:

o       a
o       b
o       c

For file types, when the default value is '-', it is automatically set to stdin or stdout, depending on the file's read/write opening mode.

File types support plain text files. If the file name ends with .gz and the property type annotation is a generic str, such as IO[str]/TextIO, it will open the file using gzip.open() + rt mode.

使用文档

EN-DOC

用于快速将dataclass类绑定到ArgumentParser，以实现快速定制化command-line参数。该绑定支持argument-parser大部分参数，如help文档，是否必须参数、多选值等，具体见下。同时，该工具尽可能地支持了Python内置的类型提示。

快速使用

Python版本：>=3.7

pip install parser-binding

demo.py：

from dataclasses import dataclass, field
from enum import Enum

from parser_binding import BindingParser, Field


class LogLevel(Enum):
    DEBUG = 'debug'
    INFO = 'info'
    WARNING = 'warning'
    ERROR = 'error'


@dataclass
class TestOptions:

    input_file: str = Field(
        default=None, aliases=['i'], help='The input file to read.'
    )
    workers: int = 1
    logging_level: LogLevel = LogLevel.WARNING

    verbose: bool = False


if __name__ == '__main__':
    parser = BindingParser(TestOptions)

    options = parser.parse_into_dataclasses((TestOptions, ))

    print(options)

执行python demo.py -h，得到如下输出：

usage: demo.py [-h] [-i INPUT_FILE] [--workers WORKERS] [--logging-level {debug,info,warning,error}] [--with-verbose]

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE, --input_file INPUT_FILE
                        The input file to read. Optional. Default `None`.
  --workers WORKERS     Optional. Default `1`.
  --logging-level {debug,info,warning,error}, --logging_level {debug,info,warning,error}
                        Optional. Default `warning`.
  --with-verbose, --with_verbose
                        Optional. Default `False` as verbose disabled.

执行python demo.py -i ./test.txt --workers 2 --logging-level debug --with-verbose，将自动得到一个从command-line读取对应参数的dataclass实例：

_MergedDataClass@1704877855657(input_file='./test.txt', workers=2, logging_level=<LogLevel.DEBUG: 'debug'>, verbose=True)

格式化说明

command-line参数名将统一格式化为三种类型：

• --xxx-yyy，当类属性名带有_时，将转换为使用--开头，-分割的command-line参数名；
• --xxx_yyy, 当类属性名带有_时，将转换为使用--开头，_分割的command-line参数名作为备选；
• -x，快捷选项支持，当属性名为单字符或指定的别名中含有单字符名称时，将使用快捷选项-开头，作为参数之一；

当前支持python中大部分类型：

• 基本类型

  • int，整数类型，将自动将command-line的参数转化为整数；
  • float，浮点数类型，自动将command-line参数转化为浮点数；
  • str，字符串类型，自动将command-line参数转化为字符串；
  • bytes，字节类型，默认支持UTF8编码方式的字符串直接转换，其他复杂数据暂不支持，若需其他转换，可通过指定type实现；
  • bool，布尔类型，自动为command-line添加开关选项，开关名称受默认值影响；当默认值为True时，开关名称为--without-xxx-yyy或--without_xxx_yyy，当默认值为False时，则开关名称被格式化为以--with开头。注意：布尔值类型仅会被格式化为开关，无需在command-line中传入参数，且注意默认值变化所带来的参数名变化。

• 带有范型注释的类型，在使用范型注释时，请明确指出范型内的类型，暂不直接支持Union[str, int]此类复杂类型：

  • Optional，用于定义可选属性，使用该注释时，请为属性提供默认值。对于一切含有默认值的属性，command-line对应的参数将被视为可选参数，并提供默认值。
  • List/list，在使用List注释时，请明确指明列表元素的类型，如List[int]，使用该类型时，自动将command-line对应的参数设为多值类型，并默认使用空格表示每个值，如--multiple 1 2 3。明确的元素类型，将使得传入的每个值进行目标类型的格式化，即List[int]属性中的每个元素为int，而不会得到List[str]的结果； 若使用list注释，默认不对传入的元素进行额外的类型映射，若需要额外的类型映射，请指明type映射方法（使用可见后续介绍）；
  • Tuple/tuple，同List或list，区别在于输出的最终集合类不同；
  • Set/set，同上；
  • Deueue/dequeue，同上；
  • Queue，同上；上述集合类，均默认以空格作为多个值之间的分割，若需自定义分割符，需要使用sep指定，可参考后续实例使用方式。
  • Dict/dict，当使用Dict作为属性类型时，请明确指出key与value类型；若使用dict作为注释，则不对key/value类型做任何映射；默认情况下，command-line传入的参数，将被视为一个JSON-string解析，若无法解析，则尝试以JSON文件方式解析（此时传入一个JSON文件目录）；若两种方式均无法解析，则会导致失败；
  • Enum枚举，使用枚举类时，将使得command-line构造一个可选值的参数，并且将输入限定在固定的值列表内；command-line将传入字符串，但解析时将得到一个对应的枚举并提供给dataclass实例初始化；
  • Literal，需要python >= 3.8版本支持，与Enum等效，但不产出枚举；

• 复杂类型

  • 文件类型，为便于从command-line中读取或设置输出设备，可以通过该方式快速打开文件，减少编码open的开销；
  • 嵌套或未知类型，为进一步扩展复杂类型的场景，支持自定义type对command-line传值进行转换；

使用实例

基本类型场景

from dataclasses import dataclass

from parser_binding import BindingParser


@dataclass
class TestOption():
    required_float: float
    a: int = 0
    string_type: str = None
    switch: bool = False
    bytes_data: bytes = None


parser = BindingParser(TestOption)
options = parser.parse_into_dataclasses((TestOption, ))

print(options)

python3 basic-demo.py --required-float 0.1 -a 10 --string-type 'Hello Parser!' --with-switch --bytes-data 'Hello Parser!'

结果：

_MergedDataClass@1704880154551(required_float=0.1, a=10, string_type='Hello Parser!', switch=True, bytes_data=b'Hello Parser!')

通过-h可观察到，此时--required-float为必传参数，否则将解析失败； 对不不提供默认值的参数，parser-binding将视为必传参数处理，否则，请提供默认值。

typing注释类型

from dataclasses import dataclass
from typing import List

from parser_binding import BindingParser


@dataclass
class TestOption:
    data: List[int] = None
    items: list = None


parser = BindingParser(TestOption)

opt = parser.parse_into_dataclasses((TestOption, ))

print(opt)

执行python list-demo.py --data 1 2 3 --items 1 2 3将解析得到_MergedDataClass@1704880514613(data=[1, 2, 3], items=['1', '2', '3'])。

从上述区别可以看出，若使用List[int]注释，则列表中的每个元素将被转换为整数，否则将不进行任何转换。 若需要进行转换，需要指定type，如下，将items注释更换为：

items: list = Field(default=None, type=int)

再次执行python list-demo.py --data 1 2 3 --items 1 2 3，解析得到结果MergedDataClass@1704881428343(data=[1, 2, 3], items=[1, 2, 3])，此时类型将进行对应的转换。

对于set、tuple、deque、queue等集合类型，具有相同效果。

注意事项：若属性被list、tuple、set、queue等集合注释时，command-line将自动将对应的参数转化为以空格分割的多值参数。此时，若通过Field重新指定type时，type应该对应为元素的类，而非最终属性的集合类型。如上述Field(default=None, type=int)中，type指定元素类型为int，即属性类型为List[int]。

鉴于默认command-line多值参数都默认以空格区分，为进一步扩展支持更多分割符，可通过Filed指定sep来定义。 如：items: tuple = Field(sep=',', default=None, type=int)，将会得到Tuple[int]的解析结果，并且传入参数使用,分割。此时，应当执行以下命令：python list-demo.py --data 1 2 3 --items 1,2,3，解析结果为：_MergedDataClass@1704893904876(data=(1, 2, 3), items=(1, 2, 3))。

parser-binding额外支持字典类型，用于支持类似JSON的格式。当使用Dict作为类型注释时，需要为其分配Key-Type与Value-Type，以便于能够正确的转换。若JSON字符串中，存在多样类型的key-value，可直接使用dict进行注释，如下实例：

from dataclasses import dataclass
from typing import Dict

from parser_binding import BindingParser, Field


@dataclass
class TestOption:
    data: Dict[str, int] = None
    items: dict = None


parser = BindingParser(TestOption)

opt = parser.parse_into_dataclasses((TestOption, ), )

print(opt)

执行python dict-demo.py --data '{"1": "1", "2": "2"}' --items '{"1": "1", "2": "2"}'，得到解析结果为：_MergedDataClass@1704944849798(data={'1': 1, '2': 2}, items={'1': '1', '2': '2'})。

从上述解析结果可以看出，当使用dict进行注释时，key-type与value-type将与原JSON保持一致，但当使用Dict并指明类型时，key/value将进行对应的类型转换。

---

枚举类型

枚举类型将使得command-line定义一个可选值的参数，使得输入固定在某个集合内选择方为有效参数。

如以下示例：

from dataclasses import dataclass
from enum import Enum

from parser_binding import parse_args


class Mode(Enum):
    train: str = 'train'
    eval: str = 'eval'


@dataclass
class TestOption:
    mode: Mode = Mode.train


args = parse_args((TestOption, ))

print(args.mode)

此时，查看usage为usage: enum-demo.py [-h] [--mode {train,eval}]，即只支持train、eval二者之一的值；

枚举类在效果上与Literal注释等效。

复杂类型场景

为进一步支持更多的场景，如嵌套集合，多类型推断，文件读写等场景，parser-binding支持通过Field结合type的方式实现。

我们允许使用Optional或Union[x, None]的方式，用于声明可选属性，但尚不支持多类型推断，如Union[str, int]，或嵌套类型List[List[int]]，这些类型都将被视为复杂类型，且需要额外提供type属性，用于类型转换。

默认情况下，对于复杂类型且未指定type配置时，我们不会进行任何处理，因此读取的参数值是str类型，这可能会导致非预期的结果，请慎重！

推荐：推荐为复杂类型提供help属性，以帮助command-line理解应该如何传值。

---

嵌套类型

复杂场景下，我们以List[List[int]]为示例进行解析，如下代码：

from dataclasses import dataclass
from typing import List

from parser_binding import parse_args


@dataclass
class TestOption:
    data: List[List[int]] = None


args = parse_args((TestOption, ))

print(args)

此时，属性data并未给定type属性，并且被视为复杂类型，将得到如下警告： UserWarning: The filed "data" is complex but there is no type specified, this could make an error.

此时，直接通过command-line传值，将会得到结果：_MergedDataClass@1704955154188(data='1,2,3')，可见data与预期的List[List[int]] 类型不符。

因此，合理的实践应为：

from dataclasses import dataclass
from typing import List

from parser_binding import Field, parse_args


@dataclass
class TestOption:
    data: List[List[int]] = Field(
        default=None,
        type=lambda x:
        list(map(lambda x: list(map(int, x.split(':'))), x.split(',')))
    )


args = parse_args((TestOption, ))

print(args)

执行python complex-demo.py --data 1:4:5,2,3，将会得到_MergedDataClass@1704955285471(data=[[1, 4, 5], [2], [3]])的结果，符合类型注释预期。

---

文件类型

为便于command-line传入文件，parser-binding支持了简单的文本读、写方式，便于从dataclass中直接拿到文件实例，而避免open 方法调用，示例如下：

from dataclasses import dataclass
from typing import IO, Iterable, TextIO

from parser_binding import Field, parse_args


@dataclass
class TestOption:
    in_file: Iterable[str] = Field(
        default=None, file=True, file_mode='r', file_encoding='utf-8'
    )
    # in_file: IO[str] = Field(default=None, file='r', file_encoding='utf-8')
    out_file: IO[str] = Field(
        default='-', file_mode='w', file_encoding='utf-8'
    )


args = parse_args((TestOption, ))

for l in args.in_file:
    print('o', l.strip(), sep='\t', file=args.out_file)

对于文件，存在两种注释方式，如上述示例中的in_file属性定义，区别在于，若通过IO注释属性，则可以不指明file为True，否则必须指定file为True， 不然则视为复杂类型处理。

现有test-file.txt内容为：

a
b
c

通过执行python file-demo.py --in-file test-file.txt，上述代码得到输出为：

o       a
o       b
o       c

对于文件类型，当默认值为'-'时，则自动设默认值为stdin或stdout，这取决于文件的读/写的打开方式。

文件类型支持纯文本文件，若文件名以.gz结尾，且属性类型注释为str的范型，如IO[str]/TextIO，则会以gzip.open() + rt的模式打开文件。