This release is a pre-release and may not be stable for production use.
natlang: Natural Language Data Loading Tools
master: 
Data loader/common data structures and other tools
Most of the code are Python2/3 compatible. For the version of python for specific modules, please check the second line of each source file.
0. Usage
Install using pip will get you the latest tested version of natlang.
> pip install natlang
Alternatively, you can also install from source using the following command:
> python setup.py install
If you want to load up a dataset, then just do this:
> import natlang as nl
> data = nl.load(filePattern, format=ChoosenFormat)
> # ChoosenFormat here can be an actual imported format or string.
> # Alternatively, you can also pass a loader func in using nl.load(filePatttern, loader=func)
For parallel datasets:
> import natlang as nl
> data = nl.biload(srcPattern, tgtPattern, srcFormat, tgtFormat)
> # Loader option similar to nl.load also applies here. src stands for source, tgt stands for target.
1. Format
All supported formats are placed under src/format.
Currently the following formats are tested:
-
txt: simple text format. Sentences are separated by\n, tokens/words are separated by whitespace. -
tree: constituency tree format. Runpython -i format/tree.pyto play around. -
semanticFrame: Propbank/Nombank frame loader. Returns bundles of frames for analysis. -
AMR: Abstract Meaning Representation. Runpython -i format/AMR.pyto play around. -
conll: General CoNLL format loader. Default is CoNLL_U. Runpython -i format/conll.pyto play around.
1.1 Recommended Functions
For formats supporting being loaded from a file, one should implement a load
function in the format file (see 2.1).
For formats supporting being exported, each instance of that format should have
an export method that outputs a string.
2. Loader
2.1 Individual Loader
Each format has its own loader.
It is defined as format.FORMAT.load.
The load function has the following interface:
def load(file, linesToLoad=sys.maxsize)
At test time, the load function would be expected to parse the file
description and read from it.
It will return the first linesToLoad entries as a list.
For example, if one wishes to use load a file in constituency tree format (see
example in tests/sampleTree.txt), one could do the following:
from datatool.format import tree
x = tree.load("datatool/tests/sampleTree.txt")
2.2 Class ParallelDataLoader
This class allows one to load parallel corpora (L1, L2) in any format. You can specify the format for L1 and L2 side separately.
from datatool.loader import ParallelDataLoader
loader = ParallelDataLoader(srcFormat='txtOrTree', tgtFormat='txtOrTree')
Here, 'txtOrTree' is the default value for srcFormat and tgtFormat.
Note that under the format folder, except for data structures for specific
formats, there are also mere loaders and 'txtOrTree' is one that can handle
both tree and txt.
After initialising the loader, one can just go ahead and run:
loader.load(fFile, eFile, linesToLoad)
The loader will automatically align the parallel text and output a list of
tuples, each containing a single entry in L1 and L2.
Entries with either L1 or L2 being None or of length 0 will be omitted.
3. Exporter
Usage:
from datatool.exporter import exportToFile, RealtimeExporter
3.1 Function exportToFile
Export a txt format dataset or tree format dataset (not single entry, but
rather a dataset) to file.
3.2 Class RealtimeExporter
The code is pretty self-explanatory. If the export function of a specific format takes quite a bit of time, this method is recommended.
1 maintainer
Metadata
1 maintainer
Release files for natlang 0.3a29
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| natlang-0.3a29.tar.gz | 44.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| natlang-0.3a29-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 106.6 kB
Release files / natlang-0.3a29.tar.gz
| Download URL | natlang-0.3a29.tar.gz |
|---|---|
| Size | 44.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1baac178c035f1041dd4550328e812f6d81ea2abce49f7bbe0fb547e85b755a8
|
|
BLAKE2b-256 checksum How to use checksums |
7d216b4f865e8daeebdfdb482c1cc7e229cc16368c5df753b361faaa313d9a69
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.6.8
|
Release files / natlang-0.3a29-py3-none-any.whl
| Download URL | natlang-0.3a29-py3-none-any.whl |
|---|---|
| Size | 62.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
467cd0266cea3d0a71d035f9890562b452b3ca03af848b838277687c274654d8
|
|
BLAKE2b-256 checksum How to use checksums |
af3bf9981bae298c3d690b08868cd1931528ce3cad83c062e7ea72eba1f450f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.6.8
|
Security for natlang 0.3a29
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
