Skip to main content

Demeuk

Documentation Status Tests

Demeuk is a simple tool to clean up corpora (like dictionaries) or any dataset containing plain text strings. Example use cases are: cleaning up language dictionaries, password sets (like for example RockYou) or any file / stdin containing plain text strings.

In those corpora you'll find encoding mistakes that have been made, or you want to remove some parts of a line. Instead of creating a huge bash oneliner you can use demeuk to do all your cleaning.

Example usages:

  • Cutting
  • Length checking
  • Encoding fixing

Demeuk is written in Python3, this means of course that it is slower than for example cut. However, Demeuk is written multithreaded and thus can use all your cores. Besides this Demeuk can easily be extended to match your needs.

This application is part of the CERBERUS project that has received funding from the European Union's Internal Security Fund - Police under grant agreement No. 82201

Please read the docs for more information.

Quick start

The recommended way to install demeuk is to install it in a virtual environment.

# Create virtual environment
virtualenv <virtual environment name>
# Activate the virtual environment
source <virtual environment name>/bin/activate
pip3 install -r requirements.txt

Now you can run Demeuk from bin/demeuk.py. Examples:

python3 bin/demeuk.py --version
python3 bin/demeuk.py --help
python3 bin/demeuk.py -i inputfile.tmp -o outputfile.dict -l droppedfile.txt
python3 bin/demeuk.py -i inputfile -o outputfile -j 24 -l logfile.log
python3 bin/demeuk.py -i inputfile.tmp -o outputfile.dict -l droppedfile.txt --leak
python3 bin/demeuk.py -i inputfile -o outputfile -j 24 -l logfile.log --leak-full
python3 bin/demeuk.py -i inputdir/*.txt -o outputfile.dict -l logfile.log
python3 bin/demeuk.py -o outputfile.dict -l logfile.log

Install

For a systemwide Demeuk do:

cd <path>/demeuk/
python3 -m pip install .

Check the version:

demeuk.py -version

Usage:

demeuk.py --help

Docs

The docs are available at: http://demeuk.rtfd.io/

Metadata

Release files for demeuk 4.6.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for demeuk 4.6.4
File Size Uploaded Provenance Actions
demeuk-4.6.4.tar.gz 26.4 kB Without provenance Details for demeuk-4.6.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for demeuk 4.6.4
File Interpreter ABI Platform Actions
demeuk-4.6.4-py3-none-any.whl Python 3 none any Details for demeuk-4.6.4-py3-none-any.whl

Total release size: 46.6 kB

Release files / demeuk-4.6.4.tar.gz

Download URL demeuk-4.6.4.tar.gz
Size 26.4 kB
Tags Source
SHA-256 checksum
How to use checksums
3b914c4ebf5c4d97103b9addcd6da5f8f80c45306a764168eac3cc6f79cd9e2c
BLAKE2b-256 checksum
How to use checksums
54921d64a577b780c5cf0ab969bec0c583d5afec6155dcbeadb25dfc55551c3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / demeuk-4.6.4-py3-none-any.whl

Download URL demeuk-4.6.4-py3-none-any.whl
Size 20.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5adeb7035e30e9d9ed07455a31f44e406e060b18eb98ecb07e133d2c00457a51
BLAKE2b-256 checksum
How to use checksums
23faf3f83d88f6c0aa881850b594270f006da09d556d83a4c86a6a02e68d98cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for demeuk 4.6.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page