Skip to main content

pytokr

Very simple, somewhat stoned tokenizer for teaching purposes.

Behaviorally inspired by the early versions of the easyinput module; shares with it some similar aims, but not the aim of conceptual consistency with C/C++. A separate, different evolution of easyinput is yogi.

Install

The usual incantation should work: pip install pytokr or, in case you already have an earlier pytokr, pip install --upgrade pytokr (maybe with either sudo or --user or within a virtual environment). You may need to call pip preceded by python -m.

If that does not work, download or clone the repo, then put the pytokr folder where Python can see it from wherever you want to use it.

Simplest usage

Finds items (simple tokens, white-space separated) in a string-based iterable such as stdin (default). Ends of line are counted as white space but are otherwise ignored.

Simplest usage is

from pytokr import item

Then call item() to keep retrieving white-space-sparated items from stdin. In case no items remain, a custom EndOfDataError exception will be raised. Note that, as white-space is ignored, including ends of line, in case only white-space remains then the program is at end of data. The outcomes are str: casting them into int or float or whatever, if convenient, falls upon the caller. Of course you can assign to the function a different name at import time by using a standard as clause.

Alternatively, you may import an iterator on the whole contents of stdin:

from pytokr import items

It is most naturally employed in a for loop:

for itm in items():

Then, the iterator gracefully stops at end of data and does not raise the EndOfDataError exception. Again the renaming option applies, of course, and again ends of line are ignored as white space.

In case you import both, they will interact naturally: the individual item() function can be called inside a for loop on the iterator, provided there is still at least one item not yet read. That call will advance the items; so, the next item at the loop will be the current one after the local advances. Briefly: both advance the same iterator.

Slightly less simple usage

Alternatively, import the function that creates the reading functions:

from pytokr import pytokr

Call then pytokr to obtain the tokenizer function; give it whatever name you see fit, say, item:

item = pytokr()

If a different source of items is desired, say source (e.g. a file just open'ed or a list of strings), simply pass it on:

item = pytokr(source)

In either case, a second output can be requested, namely, an iterator over the items, say you want to name it items:

item, items = pytokr(iter = True)

(such a call would accept as well a source as first parameter). Then you can run for itm in items(): or make up a ls = list(items()) and, with some care, avoid the dependence on the EndOfDataError exception. Both combine naturally as explained above.

Also from pytokr import __version__ works as expected.

Example

Based on Jutge problem P29448 Correct Dates (and removing spoilers):

from pytokr import pytokr
item, items = pytokr(iter = True)
# alternative: from pytokr import item, items
for d in items():
    m, y = item(), item()
    if correct_date(int(d), int(m), int(y)):
        print("Correct Date")
    else:
        print("Incorrect Date")

(Un)Deprecations

The import of item and items has gone through several deprecation and undeprecation stages. They are currently undeprecated and usable with normality. Please try to upgrade to the most advanced version of pytokr and check the descriptions above.

The function make_tokr from earlier versions stays deprecated. If employed on the current version it will still work but will print a deprecation message on stderr.

Metadata

Release files for pytokr 1.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytokr 1.2.1
File Size Uploaded Provenance Actions
pytokr-1.2.1.tar.gz 5.9 kB Without provenance Details for pytokr-1.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytokr 1.2.1
File Interpreter ABI Platform Actions
pytokr-1.2.1-py3-none-any.whl Python 3 none any Details for pytokr-1.2.1-py3-none-any.whl

Total release size: 12.2 kB

Release files / pytokr-1.2.1.tar.gz

Download URL pytokr-1.2.1.tar.gz
Size 5.9 kB
Tags Source
SHA-256 checksum
How to use checksums
d2357974d13e9cae1e58d79f7556b65ebbcbe3d4d6669234433c03952863ac9c
BLAKE2b-256 checksum
How to use checksums
40e933636adbd20cfc4ea5182bbbd40e26349410f9f82d14fb122340a4e01a76
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / pytokr-1.2.1-py3-none-any.whl

Download URL pytokr-1.2.1-py3-none-any.whl
Size 6.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
beb3923f16a9a6a47df954f8da9d671d5beef0edae195af4db420d731e90e83b
BLAKE2b-256 checksum
How to use checksums
1acb224b799f397df2f8591a0f16085be73afc02d97413d391457c3948324746
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for pytokr 1.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page