Skip to main content

pytokr

Very simple, somewhat stoned tokenizer for teaching purposes.

Current version 1.1 for this repo, will become also the pip-installable version "soon".

Behaviorally inspired by the early versions of the easyinput module; shares with it some similar aims, but not the aim of conceptual consistency with C/C++. A separate, different evolution of easyinput is yogi.

Install

The usual incantation should work: pip install pytokr or, in case you already have an earlier pytokr, pip install --upgrade pytokr (maybe with either sudo or --user or within a virtual environment).

If that does not work, download or clone the repo, then put the pytokr folder where Python can see it from wherever you want to use it.

Simplest usage

Finds items (simple tokens, white-space separated) in a string-based iterable such as stdin (default). Ends of line are counted as white space but are otherwise ignored.

Simplest usage is

from pytokr import item

Then call item() to keep retrieving white-space-sparated items from stdin. In case no items remain, a custom EndOfDataError exception will be raised. Note that, as white-space is ignored, including ends of line, in case only white-space remains then the program is at end of data. The outcomes are str: casting them into int or float or whatever, if convenient, falls upon the caller. Of course you can assign to the function a different name at import time by using a standard as clause.

Alternatively, you may import an iterator on the whole contents of stdin:

from pytokr import items

It is most naturally employed in a for loop:

for itm in items():

Then, the iterator gracefully stops at end of data and does not raise the EndOfDataError exception. Again the renaming option applies, of course, and again ends of line are ignored as white space.

In case you import both, they will interact naturally: the individual item() function can be called inside a for loop on the iterator, provided there is still at least one item not yet read. That call will advance the items; so, the next item at the loop will be the current one after the local advances. Briefly: both advance the same iterator.

Slightly less simple usage

Alternatively, import the function that creates the reading functions:

from pytokr import pytokr

Call then pytokr to obtain the tokenizer function; give it whatever name you see fit, say, item:

item = pytokr()

If a different source of items is desired, say source (e.g. a file just open'ed or a list of strings), simply pass it on:

item = pytokr(source)

In either case, a second output can be requested, namely, an iterator over the items, say you want to name it items:

item, items = pytokr(iter = True)

(such a call would accept as well a source as first parameter). Then you can run for itm in items(): or make up a ls = list(items()) and, with some care, avoid the dependence on the EndOfDataError exception. Both combine naturally as explained above.

Also from pytokr import __version__ works as expected.

Example

Based on Jutge problem P29448 Correct Dates (and removing spoilers):

from pytokr import pytokr
item, items = pytokr(iter = True)
# alternative: from pytokr import item, items
for d in items():
    m, y = item(), item()
    if correct_date(int(d), int(m), int(y)):
        print("Correct Date")
    else:
        print("Incorrect Date")

(Un)Deprecations

The import of item and items has gone through several deprecation and undeprecation stages. They are currently undeprecated and usable with normality. Please try to upgrade to the most advanced version of pytokr and check the descriptions above.

The function make_tokr from earlier versions stays deprecated. If employed on version 1.0 it will still work but will print a deprecation message on stderr.

Metadata

Release files for pytokr 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytokr 1.2.0
File Size Uploaded Provenance Actions
pytokr-1.2.0.tar.gz 6.5 kB Without provenance Details for pytokr-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytokr 1.2.0
File Interpreter ABI Platform Actions
pytokr-1.2.0-py3-none-any.whl Python 3 none any Details for pytokr-1.2.0-py3-none-any.whl

Total release size: 13.3 kB

Release files / pytokr-1.2.0.tar.gz

Download URL pytokr-1.2.0.tar.gz
Size 6.5 kB
Tags Source
SHA-256 checksum
How to use checksums
7fbcf57c6b2efc4066d4f3993b50d0797ba0e41602f093cf6d58f6692e2ca08f
BLAKE2b-256 checksum
How to use checksums
b28a15a32e42923125b1c8abe6fed168401968e39b923546e48a9daf185f0871
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / pytokr-1.2.0-py3-none-any.whl

Download URL pytokr-1.2.0-py3-none-any.whl
Size 6.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a795e8c8e1424e062e0f89db2f3de6890f5a3156221925ce78f5c8e234970dbd
BLAKE2b-256 checksum
How to use checksums
efd81f58f832e175e90ac6684822a6db1b48ad62a8246363a46e98f3953afc5a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for pytokr 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page