Utilities to help build Workbench modules
Project description
Parsers for CJWorkbench modules.
Workbench modules may optionally depend on the latest version of this Python
package for its cjwparse.api.parse_file()
function.
Installation
This is meant to be used within a Docker container. It depends on executables
/usr/bin/(csv|json|xls|xlsx)-to-arrow
.
Your Dockerfile might look something like this:
FROM arrow-tools:v0.0.11 AS arrow-tools
FROM python:3.8.1-buster AS main
COPY --from=arrow-tools /usr/bin/csv-to-arrow /usr/bin/csv-to-arrow
COPY --from=arrow-tools /usr/bin/json-to-arrow /usr/bin/json-to-arrow
COPY --from=arrow-tools /usr/bin/xls-to-arrow /usr/bin/xls-to-arrow
COPY --from=arrow-tools /usr/bin/xlsx-to-arrow /usr/bin/xlsx-to-arrow
# And now that these binaries are accessible, you can install cjwparse...
Usage
import pyarrow
from cjwparse.api import parse_file
# Convert a CSV file 'input.csv' to Arrow file 'output.arrow'
input_path = Path("input.csv")
input_path.write_bytes(b"A,B\n1,2")
output_path = Path("output.arrow")
parse_file(input_path, output_path=output_path, has_headers=False)
# Read the output
with pyarrow.ipc.open_file(output_path) as reader:
table = reader.read_all()
Developing
- Run tests:
docker build .
- Write a failing unit test in
tests/
- Make it pass by editing code in
cjwparse/
black cjwparse tests && isort --recursive cjwparse tests
- Submit a pull request
Be very, very, very careful to preserve a consistent API. Workbench will upgrade this dependency without module authors' explicit consent. Add new features; fix bugs. Never change functionality.
I18n
Marking strings for translation
Strings in cjwparse
can be marked for translation using
cjwparse.i18n._trans_cjwparse
. Each translation message must have a
(unique) ID. ICU is supported for the content. For example:
from .i18n import _trans_cjwparse
err = "Error 404"
with_arguments = _trans_cjwparse(
"greatapi.exception.message",
"Something is wrong: {error}",
{"error": err}
)
without_arguments = _trans_cjwparse(
"greatapi.exception.general",
"Something is wrong",
)
Workbench is wired to accept the return value of _trans_cjwparse
wherever an error/warning or quick fix is expected.
Creating po
catalogs
Calls to _trans_cjwparse
can (and must) be parsed to create cjwparse
's .po
files.
Update the .po
files with:
./setup.py extract_messages
The first time you run this, you'll need to install dependencies: pip3 install .[maintenance]
Unit testing
In case a _trans_cjwparse
invocation needs to be unit tested, you can use cjwparse.testing.i18n.cjwparse_i18n_message
in a manner similar to the following:
from cjwparse.testing.i18n import cjwparse_i18n_message
import with_arguments, without_arguments
assert with_arguments == cjwparse_i18n_message("greatapi.exception.message", {"error": "Error 404"})
assert without_arguments == cjwparse_i18n_message("greatapi.exception.general")
Message deprecation
Never delete a trans()
call: each message ID, once assigned, must be preserved
forever.
When there is no more code path to a trans()
call, move it to
cjwparse/i18n/_deprecated_i18n_messages.py
. The file is only read by
extraction code. It is never executed.
Publishing
- Write a new
__version__
tocjwparse/__init__.py
. Adhere to semver. (As changes must be backwards-compatible, the version will always start with1
and look like1.x.y
.) - Prepend notes to
CHANGELOG.md
about the new version git commit
git tag v1.x.y
git push --tags && git push
- Wait for Travis to push our changes to PyPI
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.