disclaimer
mkv-this makes some of the features of the excellent markovify module available as a command line tool. i started on it because i wanted to process my own offline files the same way fedibooks processes mastodon toots. then i published it to share with friends. i'm a novice coder, so you are a programmer and felt like picking it up and improving on it, then by all means!
the rest of these notes are for end users.
mkv-this
mkv-this is a script that outputs a bunch of bot-like sentences based on a bank of text that you feed it, either from a local text file, a directory of text files, or a URL, and saves the results to another text file. if you run it again with the same output file, new results are appended after old ones.
both commands allow you to add a second file or URL to the input, so you can combine your secret diary with some vile shit from the webz.
installing
install it with pip, the python package manager:
python3 -m pip install mkv-this
or
pip install mkv-this
to do this you need python3 and pip. install them through your system's package manager. on debian (+ derivatives), for example, you'd run:
sudo apt install python3 python3-pip
markovify is a dependency, but should install along with mkv-this.
if you get sth like ModuleNotFound error: No module named '$modulename', just run pip install $modulename to get the missing module.
options
the script implements a number of the basic markovify options, so you can specify:
- how many sentences to output (default = 5; NB: if your input does not use initial capitals, each 'sentence' will be more like a paragraph).
- the state size, i.e. the number of preceding words to be used in calculating the choice of the next word (default = 2).
- a maximum sentence length, in characters.
- the amount of (verbatim) overlap allowed between input and output.
- if your text's sentences end with newlines rather than full-stops. handy for inputting poetry.
- an additional file or URL to use for text input.
- the relative weight to give to the second file/URL if used.
run mkv-this -h to see how to use all the options.
directories
mkv-this accepts a directory as input if you pass it the -d flag. it will read all text files found in it (including in all subdirectories) and use them as input.
if for some reason you want to concatenate some files yourself, you can easily do so from the command line, then process the resulting file:
- copy all your text files into a directory
- cd into the directory
- run
cat * > outputfile.txt - run mkv-this on the file:
mkv-this outputfile.txt
pdfs
since 0.2.3, mkv-this can take pdfs as input. but to do this you first need to download pdfminer.six with pip. because of its size, pdfminer is not installed by default with mkv-this. converting pdfs like this is not fast, and mkv-this must convert the pdf each time. so if you envisage using a large pdf many times, you would be better off converting it to plain text yourself.
file types
for directories of text files, the currently accepted file extensions are .txt, .org and .md. it is trivial to add others, so if you want one included just ask.
if you don't have text files, but odt files, use a tool like odt2txt or unoconv to convert them to text en masse. both are available in the repos.
for best results
feed mkv-this large-ish amounts of well punctuated text. it works best if you bulk replace/remove as much mess as possible (code, timestamps, HTML, links, metadata, stars, bullets, lines, tables, etc.), unless you want mashed versions of those things in your output. (no need to clean up the webpages you input though!)
you’ll probably want to edit or select things from the output. it doesn't rly output print-ready boilerplate bosh, although many bots are happily publishing its output directly.
for a few further tips, see https://github.com/jsvine/markovify#basic-usage.
happy zaning.
macos
it seems to run on macos.
you may already have python installed. if not, install homebrew, edit your PATH so that it works, then install python3 with brew install python3. if you are already running an old version of homebrew you might need to run brew install python3 && brew postinstall python3 to get python3 and pip running right.
i know nothing about macs so if you ask me for help i'll just send you random copypasta from the interwebs.
todo
- hook it up to a web-scraper.
- option to also append input model to a saved JSON file. (i.e.
text_model.to_json(),markovify.Text.from_json()). that way you could build up a bank over time. - learn how to programme.
1 maintainer
Release files for mkv-this 0.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mkv-this-0.2.5.tar.gz | 8.5 kB | Details |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mkv_this-0.2.5-py3.7.egg | Legacy Egg format | - | - | Details |
| mkv_this-0.2.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 47.5 kB
Release files / mkv-this-0.2.5.tar.gz
| Download URL | mkv-this-0.2.5.tar.gz |
|---|---|
| Size | 8.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e48e4eed5527009231b8a98c946f18864a5e0df71c6ca5a339aa8b198af86dc0
|
|
BLAKE2b-256 checksum How to use checksums |
a8cd8b46bf3cbdfa72b84bbe5afe99c1e6770172c6f7632bfce6127b50cab8f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.3
|
Release files / mkv_this-0.2.5-py3.7.egg
| Download URL | mkv_this-0.2.5-py3.7.egg |
|---|---|
| Size | 16.7 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
01c1757a8c8fd5be81b9a60a9fd0edb4af0ece133f22b5be0260171457c4f6b4
|
|
BLAKE2b-256 checksum How to use checksums |
194dc8335558b513cbf852e53b71989d7294b4b69707570c25fe09e3489de792
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.3
|
Release files / mkv_this-0.2.5-py3-none-any.whl
| Download URL | mkv_this-0.2.5-py3-none-any.whl |
|---|---|
| Size | 22.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
86b4c6abcb3514e2eae08038ebdbf1590529d5c5bf4fb7e7feba88d08f47bf3b
|
|
BLAKE2b-256 checksum How to use checksums |
546164207d8f96b425c5a13cba0e87943af2062c2d5cfb31d7ad86459a74db58
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.3
|
