pytokr
Very simple, somewhat stoned tokenizer for teaching purposes.
Current version 1.1 for this repo, will become also the pip-installable version "soon".
Behaviorally inspired by the early versions of the
easyinput module;
shares with it some similar aims, but not the aim of
conceptual consistency with C/C++. A separate, different
evolution of easyinput is yogi.
Install
The usual incantation should work: pip install pytokr or,
in case you already have an earlier pytokr,
pip install --upgrade pytokr
(maybe with either sudo or --user or within a
virtual environment).
If that does not work, download or clone the repo, then
put the pytokr folder where Python can see it from
wherever you want to use it.
Simplest usage
Finds items (simple tokens, white-space separated) in a string-based iterable such as stdin (default). Ends of line are counted as white space but are otherwise ignored.
Simplest usage is
from pytokr import item
Then call item() to keep retrieving white-space-sparated
items from stdin. In case no items remain, a custom
EndOfDataError exception will be raised. Note that,
as white-space is ignored, including ends of line,
in case only white-space remains then the program is at
end of data. The outcomes are str: casting them into
int or float or whatever, if convenient, falls upon
the caller.
Of course you can assign to the function a different name
at import time by using a standard as clause.
Alternatively, you may import an iterator on the whole
contents of stdin:
from pytokr import items
It is most naturally employed in a for loop:
for itm in items():
Then, the iterator gracefully stops at end of data and
does not raise the EndOfDataError exception. Again the
renaming option applies, of course, and again ends of line
are ignored as white space.
In case you import both, they will interact naturally: the
individual item() function can be called inside a for
loop on the iterator, provided there is still at least
one item not yet read. That call will advance the items;
so, the next item at the loop will be the current one after
the local advances. Briefly: both advance the same iterator.
Slightly less simple usage
Alternatively, import the function that creates the reading functions:
from pytokr import pytokr
Call then pytokr to obtain the tokenizer function; give it
whatever name you see fit, say, item:
item = pytokr()
If a different source of items is desired, say source
(e.g. a file just open'ed or a list of strings),
simply pass it on:
item = pytokr(source)
In either case, a second output can be requested, namely, an
iterator over the items, say you want to name it items:
item, items = pytokr(iter = True)
(such a call would accept as well a source as first parameter).
Then you can run for itm in items(): or make up a ls = list(items())
and, with some care, avoid the dependence on the EndOfDataError
exception. Both combine naturally as explained above.
Also from pytokr import __version__ works as expected.
Example
Based on Jutge problem P29448 Correct Dates (and removing spoilers):
from pytokr import pytokr
item, items = pytokr(iter = True)
# alternative: from pytokr import item, items
for d in items():
m, y = item(), item()
if correct_date(int(d), int(m), int(y)):
print("Correct Date")
else:
print("Incorrect Date")
(Un)Deprecations
The import of item and items has gone through several
deprecation and undeprecation stages. They are currently
undeprecated and usable with normality. Please try to upgrade
to the most advanced version of pytokr and check the descriptions above.
The function make_tokr from earlier versions stays
deprecated. If employed on version 1.0 it will still work
but will print a deprecation message on stderr.
Metadata
Release files for pytokr 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pytokr-1.2.0.tar.gz | 6.5 kB | Without provenance | Details for pytokr-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pytokr-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for pytokr-1.2.0-py3-none-any.whl |
Total release size: 13.3 kB
Release files / pytokr-1.2.0.tar.gz
| Download URL | pytokr-1.2.0.tar.gz |
|---|---|
| Size | 6.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7fbcf57c6b2efc4066d4f3993b50d0797ba0e41602f093cf6d58f6692e2ca08f
|
|
BLAKE2b-256 checksum How to use checksums |
b28a15a32e42923125b1c8abe6fed168401968e39b923546e48a9daf185f0871
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Release files / pytokr-1.2.0-py3-none-any.whl
| Download URL | pytokr-1.2.0-py3-none-any.whl |
|---|---|
| Size | 6.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a795e8c8e1424e062e0f89db2f3de6890f5a3156221925ce78f5c8e234970dbd
|
|
BLAKE2b-256 checksum How to use checksums |
efd81f58f832e175e90ac6684822a6db1b48ad62a8246363a46e98f3953afc5a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Security for pytokr 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.