tumblrbot 1.7.0

An updated bot that posts to Tumblr, based on your very own blog!

tumblrbot

Description of original project:

4tv-tumblrbot was a collaborative project I embarked on with my close friend Dima, who goes by @smoqueen on Tumblr. The aim of this endeavor was straightforward yet silly: to develop a Tumblr bot powered by a machine-learning model. This bot would be specifically trained on the content from a particular Tumblr blog or a selected set of blogs, allowing it to mimic the style, tone, and thematic essence of the original posts.

This fork is largely a rewrite of the source code with similarities in its structure and process.

Features:

• An interactive console for all steps of generating posts for the blog:
  1. Asks for OpenAI and Tumblr tokens.
     • Stores API tokens using keyring.
  2. Retrieves Tumblr OAuth tokens.
  3. Downloads posts from the configured blogs.
     • Skips redownloading already downloaded posts.
     • Shows progress and previews the current post.
  4. Creates examples to fine-tune the model from your posts.
     • Filters out posts that contain more than just text data.
     • Adds custom user messages and assistant responses to the dataset from the configured file.
  5. Filters out any posts flagged by the OpenAI Moderation API.
  6. Uploads examples to OpenAI and begins the fine-tuning process.
     • Provides cost estimates if the currently saved examples are used to fine-tune the configured model.
     • Resumes monitoring the same fine-tuning process when restarted.
     • Deletes the uploaded examples file if fine-tuning does not succeed (optional).
     • Stores the output model automatically when fine-tuning is completed.
  7. Generates and uploads posts to the configured blog using the configured fine-tuned model.
     • Creates tags by extracting keywords at the configured frequency using the configured model.
     • Uploads posts as drafts to the configured blog.
     • Reblogs posts from the configured blogs at the configured frequency.
     • Shows progress and previews the current post.
• Colorful output, progress bars, and post previews using rich.
• Automatically keeps the config file up-to-date and recreates it if missing.

To-Do:

• Create training data from a sample of posts (possible).
• User-specified list of words that will filter out posts.

Known Issues:

• Sometimes, you will get an error about the training file not being found when starting fine-tuning. We do not currently have a fix or workaround for this. You should instead use the online portal for fine-tuning if this continues to happen. Read more in fine-tuning.
• Post counts are incorrect when downloading posts. We are not certain what the cause of this is, but our tests suggest this is a Tumblr API problem that is giving inaccurate numbers.
• During post downloading or post generation, you may receive a "Limit Exceeded" error message from the Tumblr API. This is caused by server-side rate-limiting by Tumblr. The only workaround is trying again or waiting for a period of time before retrying. In most cases, you either have to wait for a minute or an hour for the limits to reset. You can read more about the limits in the Tumblr API documentation on rate limits.

Please submit an issue or contact us for features you want added/reimplemented.

Installation

1. Install the latest version of Python:
   • Windows: winget install python3
   • Linux (apt): apt install python-pip
   • Linux (pacman): pacman install python-pip
2. Install the pip package: pip install tumblrbot
   • Alternatively, you can install from this repository: pip install git+https://github.com/MaidThatPrograms/tumblrbot.git
   • On Linux, you will have to make a virtual environment or use the flag to install packages system-wide.
   • See keyring for additional requirements if you are not on Windows.

Usage

Run tumblrbot from anywhere. Run tumblrbot --help for command-line options. Every command-line option corresponds to a value from the config.

Obtaining Tokens

OpenAI

API token can be created here: OpenAI Tokens.

1. Leave everything at the defaults and set Project to Default Project.
2. Press Create secret key.
3. Press Copy to copy the API token to your clipboard.

Tumblr

API tokens can be created here: Tumblr Tokens.

1. Press + Register Application.
2. Enter anything for Application Name and Application Description.
3. Enter any URL for Application Website and Default callback URL, like https://example.com.
4. Enter any email address for Administrative contact email. It probably doesn't need to be one you have access to.
5. Press the checkbox next to I'm not a robot and complete the CAPTCHA.
6. Press Register.
7. You now have access to your consumer key next to Oauth Consumer Key.
8. Press Show secret key to see your Consumer Secret.

When running this program, you will be prompted to enter all of these tokens. The fields are password-protected, so there will be no output to the console. If something goes wrong while entering the tokens, you can always reset them by running the program again and answering y to the relevant prompt.

After inputting the Tumblr tokens, you will be given a URL that you need to open in your browser. Press Allow, then copy and paste the URL of the page you are redirected to into the console.

Configuration

All config options can be found in config.toml after running the program once. This will be kept up-to-date if there are changes to the config's format in a future update. This also means it may be worthwhile to double-check the config file after an update. Any changes to the config should be in the changelog for a given version.

All file options can include directories that will be created when the program is run.

All config options that involve blog identifiers expect any version of a blog URL, which is explained in more detail in the Tumblr API documentation on blog identifiers.

Specific Options:

• custom_prompts_file This file should follow the following file format:

  {"user message 1": "assistant response 1"}
  {"user message 1": "assistant response 1"}
  {"user message 2": "assistant response 2", "user message 3": "assistant response 3"}
  

  To be specific, it should follow the JSON Lines file format with one collection of name/value pairs (a dictionary) per line. You can validate your file using the JSON Lines Validator.

• developer_message - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.

• user_message - This setting is used and works in the same way as developer_message.

• expected_epochs - The default value here is the default number of epochs for base_model. You may have to change this value if you change base_model. After running fine-tuning once, you will see the number of epochs used in the fine-tuning portal under Hyperparameters. This value will also be updated automatically if you run fine-tuning through this program.

• token_price - The default value here is the default token price for base_model. You can find the up-to-date value in OpenAI Pricing, in the Training column.

• job_id - If there is any value here, this program will resume monitoring the corresponding job, instead of starting a new one. This gets set when starting the fine-tuning and is cleared when it is completed. You can read more in fine-tuning.

• base_model - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the fine-tuning portal by pressing + Create and opening the drop-down list for Base Model. Be sure to update token_price if you change this value.

• fine_tuned_model - Set automatically after monitoring fine-tuning if the job has succeeded. You can read more in fine-tuning.

• tags_chance - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.

• reblog_chance - This setting works the same way as tags_chance.

• reblog_user_message - This setting is a prefix that is directly prepended to the contents of the post being reblogged.

Manual Fine-Tuning

You can manually upload the examples file to OpenAI and start the fine-tuning here: fine-tuning portal.

1. Press + Create.
2. Select the desired Base Model from the dropdown. This should ideally match the model set in the config.
3. Upload the generated examples file to the section under Training data. You can find the path for this in the config.
4. Press Create.
5. (Optional) Copy the value next to Job ID and paste it into the config under job_id. You can then run the program and monitor its progress as usual.
6. If you do not do the above, you will have to copy the value next to Output model once the job is complete and paste it into the config under fine_tuned_model.