billingflatfile 1.0.6

Generate the required fixed width format files from delimited files extracts for EMR billing purposes

billingflatfile

Generate the required fixed width format files from delimited files extracts for EMR billing purposes

How to run the program

How to install the program

For Linux and Windows, download the latest version from here (look under the "Assets" section) and run it on your system, no need to install anything else.

The program can also be installed from the Python Package Index:

pip install billingflatfile

Or it can be downloaded and ran directly from the Docker Hub:

docker run --rm e2jk/billingflatfile [add parameters here]

See below how to install from source.

Configuration file

In order for the program to know how to transform your delimited file into a fixed-width file, you will need to provide a configuration file describing the length and type of values expected for your output file.

An example configuration file can be found at tests/sample_files/configuration1.xlsx

A configuration file is a simple Excel .xlsx file in which each row represents a single field expected in the output file (the fixed-width file), and at least these 3 column headers, i.e. the first line in your Excel file:

• Length
• Output format
• Skip field

The Length value is self-explanatory: it represents how long the field will be in the generated fixed-width file. If the value in the input file is shorter than this defined length, it will be padded with 0s or spaces, depending on the type of Output format (see next section).

The Output format defines how the input value must be treated and transformed. The following values are supported:

• Integer

  • A numeric value that gets padded with 0s added to the left
  • Example: "123" becomes "000123" if a length of 6 is defined

• Decimal

  • Decimal numbers get sent as "cents" instead of "dollars", rounded to the nearest cent. (yeah, weird explanation -- better have a look at the example...). Also padded with 0s added to the left.
  • Example: "123.458" becomes "00012346" if a length of 8 is defined

• Keep numeric

  • strips all non-numeric characters from an input value and treats the remaining value as Integer
  • Example: "1-2.3a" becomes "000123" if a length of 6 is defined

• Date

  • A date to be converted from one format to another. The input value can be sent with either Day or Month as first element or as ISO format YYYYMMDD, and with a slash, dash, dot or no separator. When there is a separator defined, the day and month can omit the leading 0, if need be. See at the top of the test_main.py file for the full list of supported codes.
  • Examples:
    • "21/06/2020" becomes "20200621" with a format of Date (DD/MM/YYYY to YYYYMMDD) and a length of 8
    • "6-21-2020" becomes "20200621" with a format of Date (MM-DD-YYYY to YYYYMMDD) and a length of 8
    • "21062020" becomes "20200621" with a format of Date (DDMMYYYY to YYYYMMDD) and a length of 8
    • "6.21.2020" becomes "21/06/2020" with a format of Date (MM.DD.YYYY to DD/MM/YYYY) and a length of 10

• Time

  • A time sent as hour:minutes (with or without colon in the input data) will be sent out without the colon
  • Example: "20:06" becomes "2006" if a length of 4 is defined

• Text

  • The value gets sent without format changes (such as those outlined above for date and time), with spaces added at the end, on the right of the string
  • Example: "Hello" becomes "Hello     " if a length of 10 is defined

Finally, setting the value of the Skip field column to "True" allows to send a field as blank in the output file, respecting the field size and padding type: 0s or spaces depending on the defined output format.

Running the program

Open a Command Line window cmd and indicate your input file name and configuration file to use. You will also need to indicate a number of arguments needed to generate the metadata file, such as the Application ID and the Run ID. You can additionally indicate if your input file uses a specific field separator (default is ,), textual field wrapper (default is "), if you want to skip a specific number of header or footer files from your input file, a description for this billing run and the type of billing.

See the Program help information section below for details on how to populate these arguments.

An example run of the program could look like this:

billingflatfile.exe --input data\input_file.txt --config data\configuration_file.xlsx --application-id AB --run-id 456 --skip-header 1 --skip-footer 1 --delimiter "^" --run-description "Nice description" --billing-type "H"

Or it can be ran from the Docker Hub:

docker run --rm e2jk/billingflatfile [add parameters here]

If you've installed the program following how to install from source, you can run the program with pipenv run python billingflatfile.py.

Program help information

usage: billingflatfile.py [-h] [--version] (-i INPUT | -id INPUT_DIRECTORY) [-ie INPUT_ENCODING] [-od OUTPUT_DIRECTORY] [-m] -c CONFIG
                          [-dl DELIMITER] [-q QUOTECHAR] [-sh SKIP_HEADER] [-sf SKIP_FOOTER] [-l LOCALE] [-t TRUNCATE] [-dv DIVERT] [-x] -a
                          APPLICATION_ID [-ds RUN_DESCRIPTION] [-b BILLING_TYPE] [-r RUN_ID] [-rf RUN_ID_FILE] [-fv FILE_VERSION]
                          [-dr DATE_REPORT] [-txt] [-d] [-v]

Generate the required fixed width format files from delimited files extracts for EMR billing purposes

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -i INPUT, --input INPUT
                        Specify the input file
  -id INPUT_DIRECTORY, --input-directory INPUT_DIRECTORY
                        Specify the input directory from which to process input files
  -ie INPUT_ENCODING, --input-encoding INPUT_ENCODING
                        Specify the encoding of the input files (default: 'utf-8')
  -od OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                        The directory in which to create the output files
  -m, --move-input-files
                        Move the input files to the output directory after processing. Must be used in conjunction with the `--output-
                        directory` argument.
  -c CONFIG, --config CONFIG
                        Specify the configuration file
  -dl DELIMITER, --delimiter DELIMITER
                        The field delimiter used in the input file (default ,)
  -q QUOTECHAR, --quotechar QUOTECHAR
                        The character used to wrap textual fields in the input file (default ")
  -sh SKIP_HEADER, --skip-header SKIP_HEADER
                        The number of header lines to skip (default 0)
  -sf SKIP_FOOTER, --skip-footer SKIP_FOOTER
                        The number of footer lines to skip (default 0)
  -l LOCALE, --locale LOCALE
                        Change the locale, useful to handle decimal separators
  -t TRUNCATE, --truncate TRUNCATE
                        Comma-delimited list of field numbers for which the output will be truncated at the maximum line length, should the
                        input value be longer than the maximum defined field length. If not set, a field that is too long will cause the
                        script to stop with an error.
  -dv DIVERT, --divert DIVERT
                        Diverts to a separate file the content from rows containing a specific value at a specific place. The format of this
                        parameter is "<field number>,<value to divert on>" (without quotes). This parameter can be repeated several times to
                        support different values or different fields. The diverted content will be saved to a file whose name will be the
                        output filename with "_diverted" added before the file extension.
  -x, --overwrite-files
                        Allow to overwrite the output files
  -a APPLICATION_ID, --application-id APPLICATION_ID
                        The application ID. From the vendor specs: the first character will be filled with the first letter of the site that
                        is to be invoiced, and the second character will be filled with a significant letter to describe the application.
                        Must be unique for the receiving application to accept the files. Max 2 characters.
  -ds RUN_DESCRIPTION, --run-description RUN_DESCRIPTION
                        The description for this run. Free text, max 30 characters.
  -b BILLING_TYPE, --billing-type BILLING_TYPE
                        The billing type. Must be 'H' (internal billing), 'E' (external billing) or ' ' (both external and internal billing,
                        or undetermined). Max 1 character.
  -r RUN_ID, --run-id RUN_ID
                        The ID for this run. Must be unique for each run for the receiving application to accept it. Numeric value between 0
                        and 9999, max 4 characters.
  -rf RUN_ID_FILE, --run-id-file RUN_ID_FILE
                        Point to a file from which to retrieve the ID for this run. After processing, the Run ID + 1 is saved to this file,
                        allowing for automated recurring runs (for instance associated with the `--input-directory` and `--move-input-files`
                        arguments). Can be used in conjunction with the `--run-id` argument to seed the initial value of the Run ID.
  -fv FILE_VERSION, --file-version FILE_VERSION
                        The version of the output file to be generated. Only 'V1.11' is currently supported. Max 8 characters.
  -dr DATE_REPORT, --date-report DATE_REPORT
                        The column number of a Date column to report on in the metadata file. Numeric value between 0 and 99999.
  -txt, --txt-extension
                        Add a .txt extension to the output files' names.
  -d, --debug           Print lots of debugging statements
  -v, --verbose         Be verbose

Development information

How to install from source

Create the environment:

cd devel/billingflatfile/
python3 -m pip install --user pipenv
pipenv install

If you want to develop the script, replace that last command by pipenv install --dev

Activate the virtual environment:

cd devel/billingflatfile/
pipenv shell

You can also run the script using pipenv run billingflatfile.py instead of python3 billingflatfile.py without having to set up a subshell (which has some problems in Windows, with the history not being accessible with the up arrow)

When done:

exit

Update the dependencies:

pipenv update

Install a new dependency

pipenv install <package_name> [--dev]

Building the executable

Run the following command in your virtual environment:

pipenv run pyinstaller --onefile billingflatfile.py

The executable that gets created in the dist folder can then be uploaded to Github as a new release.

Packaging the source and publishing to the Python Package Index

Follow the instructions mentioned here, namely:

pipenv lock -r > requirements-no-dev.txt
pipenv run python setup.py sdist bdist_wheel
pipx run twine upload dist/*

Create the Docker image and publish it to Docker Hub

Run:

• docker build -t e2jk/billingflatfile:latest -t e2jk/billingflatfile:<version> --rm . to build the Docker image.
• docker run --rm e2jk/billingflatfile:<version> to test the Docker image locally.
• docker push e2jk/billingflatfile:latest and docker push e2jk/billingflatfile:<version> to push the Docker image to Docker Hub.

Changelog

These are the changes brought in each versions:

v1.0.6 (2021-07-09)

Non-breaking changes:

• Fix package published on PyPI

v1.0.5 (2020-11-21)

Breaking changes:

• --output_directory is now mandatory, don't create a default output directory in data/<today's date>
• Upper limit for the --run-id argument down from 99999 to 9999

Non-breaking changes:

• Process all the files in a directory as multiple input files through the new arguments --input-directory and --output-directory
• New --move-input-files argument to move the input files to the output directory after processing
• New --run-id-file arguments to read and save the next Run ID. Allows for automated recurring runs, for instance associated with the --input-directory and --move-input-files arguments
• New --input-encoding argument to specify the encoding of the input files, defaults to 'utf-8'

v1.0.4 (2020-11-02)

Breaking changes:

• Date and Time fields now default/pad to 0's instead of spaces

Non-breaking changes:

• New --txt-extension argument to add a .txt extension to the output files' names.
• The program can now be ran in a Docker container: https://hub.docker.com/r/e2jk/billingflatfile

v1.0.3 (2020-09-30)

Non-breaking changes:

• New --truncate argument to specify which fields can be cut when the input value is longer than the defined maximum field length
• New --locale argument, in case a different Decimal separator is used
• Fix: Spaces or empty string accepted as valid Integer and Decimal values (interpreted as 0)
• Support a large number of new date formats. See at the top of the test_main.py file for the full list of supported codes. Some examples:
  • Date (DD/MM/YYYY to DD/MM/YYYY)
  • Date (YYYYMMDD to DD.MM.YYYY)
  • Date (MM.DD.YYYY to YYYYMMDD)
  • Date (YYYYMMDD to MM-DD-YYYY)
• New format Keep numeric that strips all non-numeric characters from an input value
• New --divert argument to divert to a separate file the content from rows containing specific values

v1.0.2 (2020-09-16)

Non-breaking changes:

• Support for new date formats:
  • Date (DD-MM-YYYY to YYYYMMDD)
  • Date (MM-DD-YYYY to YYYYMMDD)
  • Date (DD.MM.YYYY to YYYYMMDD)
  • Date (MM.DD.YYYY to YYYYMMDD)
  • Date (DDMMYYYY to YYYYMMDD)
  • Date (MMDDYYYY to YYYYMMDD)

v1.0.1 (2020-09-16)

Other changes:

• Reduce the number of dependencies

v1.0.0 (2020-09-14)

Non-breaking changes:

• New --date-report argument

Other changes:

• Changes to the development toolchain and test suite

v0.0.1-alpha (2020-09-10)

• Initial release