strpipe
Reversible string processing pipe. Featuring reproducibility, serializability and performance.
Installation
pip install strpipe
Usage
import strpipe as sp
p = sp.Pipe()
p.add_step_by_op_name('ZhCharTokenizer')
p.add_step_by_op_name('AddSosEos')
p.add_checkpoint()
p.add_step_by_op_name('Pad')
p.add_step_by_op_name('TokenToIndex')
data = [
'你好啊',
'早安',
'你早上好',
]
p.fit(data)
result, tx_info, intermediates = p.transform(data) # convention: tx => tranform
back_data = p.inverse_transform(result, tx_info)Serialization
# Save it
p.save_json('/path/of/pipe')
# Load it
p = sp.Pipe.restore_from_json('/path/of/pipe')
result, meta = p.transform(['你好'])Test
$ make test
Docs
$ make docs Docs will be built in the `docs/build/html` folder. (Note: this also reinstalls the package because we need Cython code to be rebuilt.)
Extend Ops
Extend the new ops with BaseOp
Define input_type, output_type
Implement op creation
Implement fit, transform, inverse_transform. If the op is stateless, the fit method should return None.
Note: It is expected that an ops’s functionality will often be able to be decomposed into several functions. These functions should be written into (or imported from) the toolkit package for easy reuse. Ops in the ops package will, for the most part, be wrappers for functions in toolkit.
Write tests
Register to op_factory
2 maintainers
Metadata
2 maintainers
Release files for strpipe 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| strpipe-0.5.0.tar.gz | 615.2 kB | Details |
Release files / strpipe-0.5.0.tar.gz
| Download URL | strpipe-0.5.0.tar.gz |
|---|---|
| Size | 615.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
06153d1c56479a007715d3641bbd676a3ff7f51b9b62ef4bec71427dab7572e8
|
|
BLAKE2b-256 checksum How to use checksums |
5447ad998a103a9f3b1a0b02dc56be0f65f1770b93d6bc25726033ea0f0a6086
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.6.3
|
