InScript Lexer - Rust Implementation (v3.7.2)

Real, compilable Rust tokenizer for InScript language.

Features

✅ Zero-copy token references (fast, memory efficient) ✅ State machine lexer (no backtracking) ✅ 30+ token types (keywords, operators, literals) ✅ Accurate line/column tracking ✅ Comprehensive error reporting ✅ Python FFI via PyO3 ✅ Full test coverage

Build

cargo build --release

Token Types Supported

Literals

Numbers (integers and floats)
Strings (single and double quoted)
Identifiers

Keywords

if, else, while, for, fn, return
true, false, nil
and, or, not

Operators

Arithmetic: +, -, *, /, %
Comparison: ==, !=, <, <=, >, >=
Assignment: =
Logical: and, or, not

Delimiters

Parentheses: ( )
Braces: { }
Brackets: [ ]
Punctuation: , . ; : ->

Rust Usage

use inscript_lexer::Lexer;

fn main() {
    let code = "let x = 42;";
    let lexer = Lexer::new(code);
    
    for token in lexer {
        match token {
            Ok(t) => println!("{:?}: {}", t.token_type, t.value),
            Err(e) => eprintln!("Error: {}", e),
        }
    }
}

Python Usage (after compilation)

from inscript_lexer import PyLexer, tokenize_string

# Method 1: Using PyLexer class
lexer = PyLexer("let x = 42;")
tokens = lexer.tokens()
print(f"Found {lexer.count()} tokens")

# Method 2: Direct tokenization
tokens = tokenize_string("if true then x else y")
for token in tokens:
    print(f"{token.token_type}: {token.value}")

Performance

Expected Benchmarks

Simple tokens: < 100ns per token
Complex code: < 10µs per 100 tokens
Memory: O(1) per token

Optimizations

Inline hot paths
No allocations in common cases
SIMD-friendly string scanning
Zero-copy string references

Testing

# Run tests
cargo test

# Run with output
cargo test -- --nocapture

# Benchmark
cargo bench

Accuracy

✅ Correct line/column tracking ✅ Handles escape sequences ✅ Detects unterminated strings ✅ Error recovery

License

MIT

Metadata

Release files for inscript-lang 3.9.6.45

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for inscript-lang 3.9.6.45
File	Size	Uploaded
inscript_lang-3.9.6.45.tar.gz	338.8 kB	Jul 8, 2026	Details

Built distributions (wheels)

Table of built distributions (wheels) for inscript-lang 3.9.6.45
File	Interpreter	ABI	Platform	Reset
inscript_lang-3.9.6.45-cp312-cp312-win_amd64.whl 692.5 kB Jul 8, 2026	CPython 3.12	CPython 3.12	Windows x86-64	Details
inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_x86_64.whl 776.4 kB Jul 8, 2026	CPython 3.12	CPython 3.12	Linux glibc 2.28+ x86-64	Details
inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_aarch64.whl 744.6 kB Jul 8, 2026	CPython 3.12	CPython 3.12	Linux glibc 2.28+ ARM64	Details
inscript_lang-3.9.6.45-cp312-cp312-macosx_10_13_universal2.whl 1.1 MB Jul 8, 2026	CPython 3.12	CPython 3.12	macOS 10.13+ universal2 (ARM64, x86-64)	Details
inscript_lang-3.9.6.45-cp311-cp311-win_amd64.whl 688.9 kB Jul 8, 2026	CPython 3.11	CPython 3.11	Windows x86-64	Details
inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_x86_64.whl 773.2 kB Jul 8, 2026	CPython 3.11	CPython 3.11	Linux glibc 2.28+ x86-64	Details
inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_aarch64.whl 743.2 kB Jul 8, 2026	CPython 3.11	CPython 3.11	Linux glibc 2.28+ ARM64	Details
inscript_lang-3.9.6.45-cp311-cp311-macosx_10_12_universal2.whl 1.1 MB Jul 8, 2026	CPython 3.11	CPython 3.11	macOS 10.12+ universal2 (ARM64, x86-64)	Details
inscript_lang-3.9.6.45-cp310-cp310-win_amd64.whl 689.1 kB Jul 8, 2026	CPython 3.10	CPython 3.10	Windows x86-64	Details
inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_x86_64.whl 773.5 kB Jul 8, 2026	CPython 3.10	CPython 3.10	Linux glibc 2.28+ x86-64	Details
inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_aarch64.whl 743.5 kB Jul 8, 2026	CPython 3.10	CPython 3.10	Linux glibc 2.28+ ARM64	Details
inscript_lang-3.9.6.45-cp310-cp310-macosx_10_12_universal2.whl 1.1 MB Jul 8, 2026	CPython 3.10	CPython 3.10	macOS 10.12+ universal2 (ARM64, x86-64)	Details

Total release size: 10.3 MB

Release files / inscript_lang-3.9.6.45.tar.gz

Download URL	inscript_lang-3.9.6.45.tar.gz
Size	338.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`5d6d80bcc3f3a64e8cf352dd3b5fcf91bec5824d50c2aed5064a71482aa14036`
BLAKE2b-256 checksum How to use checksums	`c3b84d8e294c796f5ecd8738d33e44387a00624fa5b64928a1d6c819144349bb`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp312-cp312-win_amd64.whl

Download URL	inscript_lang-3.9.6.45-cp312-cp312-win_amd64.whl
Size	692.5 kB
Tags	CPython 3.12 Windows x86-64
SHA-256 checksum How to use checksums	`1d8aca4bd8cb7e0c7d4c7cf3bb31796adebbf02473f753993a35c219af2ef12b`
BLAKE2b-256 checksum How to use checksums	`2c5a25a56b244a771d7c5d0684e4c4faaf7be8d4dc64dda82d46450bd12d56bb`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_x86_64.whl

Download URL	inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_x86_64.whl
Size	776.4 kB
Tags	CPython 3.12 Linux glibc 2.28+ x86-64
SHA-256 checksum How to use checksums	`0628e0e16ae275f4fb974a237955c4d58dfafed6043cc065ea430443f5d3a3d6`
BLAKE2b-256 checksum How to use checksums	`65b3755f98336530d80b28f77ba2f2c3f0fb44cab30cf594705df39b3ace714d`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_aarch64.whl

Download URL	inscript_lang-3.9.6.45-cp312-cp312-manylinux_2_28_aarch64.whl
Size	744.6 kB
Tags	CPython 3.12 Linux glibc 2.28+ ARM64
SHA-256 checksum How to use checksums	`44fdc7c06f4f7c7072230f9be3f07769f7fdc9a2b88fb52136f7b63c39fb4903`
BLAKE2b-256 checksum How to use checksums	`6db6d46fc26f7a4f5e23aa308d4ec696b54c99b3c22f110a94f64df53ce3daec`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp312-cp312-macosx_10_13_universal2.whl

Download URL	inscript_lang-3.9.6.45-cp312-cp312-macosx_10_13_universal2.whl
Size	1.1 MB
Tags	CPython 3.12 macOS 10.13+ universal2 (ARM64, x86-64)
SHA-256 checksum How to use checksums	`d26f3e8c90a318c76b5fee79ed6cd512a725c908638c04f0eac83d105a74cfe9`
BLAKE2b-256 checksum How to use checksums	`9dc6477200960d41043402149febb77d4d13de7166831b1266924d04e04e9a3a`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp311-cp311-win_amd64.whl

Download URL	inscript_lang-3.9.6.45-cp311-cp311-win_amd64.whl
Size	688.9 kB
Tags	CPython 3.11 Windows x86-64
SHA-256 checksum How to use checksums	`254e8a7d05c0766fda13fe3b698c53e1f6fe56cd782dd0f82ac2caad41084fe2`
BLAKE2b-256 checksum How to use checksums	`680288586f74f642b6582aca4d9c350072ec3032daabfb3637c39057f8de1f72`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_x86_64.whl

Download URL	inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_x86_64.whl
Size	773.2 kB
Tags	CPython 3.11 Linux glibc 2.28+ x86-64
SHA-256 checksum How to use checksums	`af253b6f0a3776f3789d372ced1b5b61b5fb0b6fbd6cb608e962b40c3f590e46`
BLAKE2b-256 checksum How to use checksums	`1138d9defeb3b41861b96be26f58b44073844738cc0231546bc76822a6c2a23e`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_aarch64.whl

Download URL	inscript_lang-3.9.6.45-cp311-cp311-manylinux_2_28_aarch64.whl
Size	743.2 kB
Tags	CPython 3.11 Linux glibc 2.28+ ARM64
SHA-256 checksum How to use checksums	`16aeb82adef29e0d36879d904c5331fa6000ac9365bd08065b4d7d465e3c46f0`
BLAKE2b-256 checksum How to use checksums	`980cbcd71973e33d71ba60cd813cc17f7eca91b24025109bd954747e0d148db5`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp311-cp311-macosx_10_12_universal2.whl

Download URL	inscript_lang-3.9.6.45-cp311-cp311-macosx_10_12_universal2.whl
Size	1.1 MB
Tags	CPython 3.11 macOS 10.12+ universal2 (ARM64, x86-64)
SHA-256 checksum How to use checksums	`9b609634927487ac53e2e4bb4ab0af5f7bfe757183828b63f3f2caa9192f113c`
BLAKE2b-256 checksum How to use checksums	`968b2051a74d8b4de03fba7380958bff425b517001489834e0f0e4db1b79218b`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp310-cp310-win_amd64.whl

Download URL	inscript_lang-3.9.6.45-cp310-cp310-win_amd64.whl
Size	689.1 kB
Tags	CPython 3.10 Windows x86-64
SHA-256 checksum How to use checksums	`c4971783b133c9827d8f70092567177357f7bebe227dd6ff5944c8b85a1b6711`
BLAKE2b-256 checksum How to use checksums	`d0164e4d27daf2de9dfe15c71254593f3076dacc7443e97b8f03fc51c05ddf76`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_x86_64.whl

Download URL	inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_x86_64.whl
Size	773.5 kB
Tags	CPython 3.10 Linux glibc 2.28+ x86-64
SHA-256 checksum How to use checksums	`a9f2d4097e2cd32d1eb284097450c2901f29113b9560e3994cb2f45da1427410`
BLAKE2b-256 checksum How to use checksums	`a5865715d5f4f9e714f0535843699b4470cc529f4f72141866624f6db4a31805`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_aarch64.whl

Download URL	inscript_lang-3.9.6.45-cp310-cp310-manylinux_2_28_aarch64.whl
Size	743.5 kB
Tags	CPython 3.10 Linux glibc 2.28+ ARM64
SHA-256 checksum How to use checksums	`212bc0997cda068786b3cc5c84c954b814e4d82edd5215662dfd52445145b340`
BLAKE2b-256 checksum How to use checksums	`4da4e6501e17de5dd133facb20d3775d6b3afec41bdf47ea6801e0360f385698`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`

Release files / inscript_lang-3.9.6.45-cp310-cp310-macosx_10_12_universal2.whl

Download URL	inscript_lang-3.9.6.45-cp310-cp310-macosx_10_12_universal2.whl
Size	1.1 MB
Tags	CPython 3.10 macOS 10.12+ universal2 (ARM64, x86-64)
SHA-256 checksum How to use checksums	`d38703b389a9e90184afaffaa9d8fe11c87b4dc266890f85e144bf0449d3ecf3`
BLAKE2b-256 checksum How to use checksums	`7f6f00369c0ee967f362d356ec97243252daeedd4441172429ebcf520d43eddf`
Upload date	Jul 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.12`