masquer 1.2.2

Weighted-random user-agent and referer data for GET requests

Masquer

A tool to generate random user-agent and referer data for HTTP requests.

Overview

Use masquer to obtain any combination of a random user-agent, referer or header data template, then use this with a library like requests to control the session data you send to other services.

The user-agent data is drawn from this list of the most common desktop user-agents, and referer data is taken from this list of search engines with the largest global market share.

Weighted random selections are made from those lists to approximate authentic header data patterns.

A basic header template with common attributes — like "Upgrade-Insecure-Requests": "1" — is also provided and defaults to the most common referer and user-agent data from the above lists.

Note on privacy

Controlling header data in this way can help to preserve privacy and hinder third-party tracking behaviour, by blending part of your web profile with the most common configurations.

It does not provide anonymity — that is a much more complex topic, and the open-source Privacy Guides are a good place to start.

Installation

To get hold of masquer either install the PyPI package into your project's virtual environment, clone the GitHub repo or pull the Docker image.

PyPI package

Install the masquer package from PyPI to retrieve just the tool with no extras.

Activate your existing project's virtual environment, then download masquer using a package manager. The below example uses Astral's uv; substitute pip by dropping "uv" or use another package manager as needed:

$ uv pip install masquer

GitHub repo

Clone the masquer repo from GitHub for the full source code. The repo includes the JSON source files used to generate the header data, a script to sync the programme if updates are made to the JSON files, and a test suite.

$ git clone [e-mail]:essteer/masquer

The functional code within the package src directory has no dependencies beyond Python built-in modules. If you intend to make changes to your cloned version of the repo, you may optionally install the project.optional-dependencies declared in the pyproject.toml file.

First create and activate a virtual environment — the below example uses Astral's uv; substitute pip or use another package manager as needed — then install the dev dependencies:

$ uv venv
$ source .venv/bin/activate
$ uv pip install fastapi==0.111.0 hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4

$ uv venv
$ .venv\Scripts\activate
$ uv pip install fastapi==0.111.0 hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4

FastAPI

The GitHub repo includes masquer API, a FastAPI version of masquer.

To self-host the API, install the FastAPI optional dependency as declared in the pyproject.toml file.

Activate the API from the root directory via:

$ fastapi run src/api/main.py

Then follow the instructions provided by FastAPI in the terminal.

By default, the FastAPI app will run on localhost. To view the API documentation, run the API and navigate to http://127.0.0.1:8000/docs.

Docker image

For no good reason, the masquer API is also available as a Docker image:

$ docker pull essteer/masquer

Operation

Interact with masquer via the masq method:

from masquer import masq

The masq function accepts up to three boolean parameters:

useragent = masq(
  ua = True,  # user-agent, defaults to True
  rf = False,  # referer, defaults to False
  hd = False,  # header-data, defaults to False
)

And returns the response in dictionary form:

{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3'}

FastAPI

masquer API is deployed on Fly.io - view the docs at https://masquer.fly.dev/docs.

An example of how to use masquer API:

$ curl -X GET \
  'https://masquer.fly.dev/masq?ua=true&rf=false&hd=false' \
  -H 'accept: application/json'

Docker

Run the masquer API from a container on the masquer Docker image as follows:

$ docker run -d --name masquer -p 8000:8000 esteer/masquer

Then interact as per the FastAPI instructions above.

Examples

User-agent only

By default only ua is set to True, so each of the following methods may be used to return just one randomly generated user-agent:

>>> useragent_1 = masq()
>>> useragent_2 = masq(True)
>>> useragent_3 = masq(ua=True)
>>>
>>> useragent_1
{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3"}
>>> useragent_2
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0."}
>>> useragent_3
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}

Referer only

By the same logic, these methods will each return just one randomly generated referer:

>>> referer_1 = masq(False, True)
>>> referer_2 = masq(ua=False, rf=True)
>>> referer_3 = masq(ua=False, rf=True, hd=False)
>>>
>>> referer_1
{"Referer": "https://www.google.com/"}
>>> referer_2
{"Referer": "https://www.google.com/"}
>>> referer_3
{"Referer": "https://bing.com/"}

Header-data

By default, the header data template supplies the most common user-agent and referer values as fixed, and can be accessed via the following methods:

>>> default_header_1 = masq(False, False, True)
>>> default_header_2 = masq(ua=False, hd=True)
>>> default_header_3 = masq(ua=False, rf=False, hd=True)
>>>
>>> default_header_1 == default_header_2 == default_header_3
True
>>> default_header_1
{"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-US,en;q=0.5;", "Referer": "https://www.google.com/", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}

As per the individual use-cases, get weighted random user-agent and referer values in the header by omitting the ua value (which defaults to True) and setting rf to True. A non-exhaustive selection of examples is provided below:

>>> # header with random user-agent and fixed referer
>>> random_header_1 = masq(hd=True)  
>>> random_header_2 = masq(True, False, True)
>>>
>>> # header with fixed user-agent and random referer
>>> random_header_3 = masq(False, True, True)
>>> random_header_4 = masq(ua=False, rf=True, hd=True)
>>>
>>> # header with random user-agent and random referer
>>> random_header_5 = masq(rf=True, hd=True)
>>> random_header_6 = masq(True, True, True)

Local development

The following details will assist with making and testing changes to a cloned version of the repository.

Updates

The root directory includes a shell script update.sh for convenience to update JSON assets for the masquer package.

$ chmod +x update.sh
$ ./update.sh
Update successful

If using your own JSON data you can still make use of the update_assets() function inside update.py to sync changes with the assets.py file inside the masquer package.

Tests

masquer uses Python's in-built unittest module.

To run the entire test suite using discover, or specify an individual test file from the tests directory — for example test_assets.py — run one of the following shell commands from the project root directory:

$ python3 -m unittest discover  # run all tests
$ python3 -m unittest tests.test_assets  # test a single file

$ python -m unittest discover  # run all tests
$ python -m unittest tests.test_assets  # test a single file

Deployment

Asset updates are performed on a regular basis to keep the user-agent and referer data current. The instructions that follow are written with these micro releases in mind, but the process will be similar for feature releases.

Activate the virtual environment then create and checkout a new branch such as asset-update.

$ source .venv/bin/activate
$ git branch asset-update
$ git checkout asset-update
Switched to branch 'asset-update'

Update

For an asset update, follow the update instructions.

Test

Ensure all tests are passing before proceeding further. See the Tests section above.

Version

Increment the version for a micro, minor or major release:

$ hatch version micro
Old: 1.2.1
New: 1.2.2

This will automatically update the version number in src/masquer/__about__.py, which is where the version information in pyproject.toml is read from for the main masquer program, and in src/api/main.py for the API.

Manually update the version number in the PyPI icon at the top of this README to match the new version number.

Format

Add and commit the version changes to git. Ruff is configured as a pre-commit hook to lint and format the package. Ruff will format files that aren't already in the correct format — those files will fail the check, so run all unit tests once again to be sure nothing was broken then add and commit the reformatted files.

Push the branch to GitHub.

Build

Run $ hatch build from the terminal to build the sdist and wheel targets inside the dist/ directory.

$ hatch build
──────────────────────────────────────────────────────── sdist ─────────────────────────────────────────────────────────
dist/masquer-1.2.2.tar.gz
──────────────────────────────────────────────────────── wheel ─────────────────────────────────────────────────────────
dist/masquer-1.2.2-py3-none-any.whl

Publish

After a successful build the package is ready to publish on PyPI:

$ hatch publish

Merge to main

Open a PR on GitHub to merge the updated package into main branch. GitHub workflows are in place to do a final format and test whenever a commit or PR is made into main.

Release and build Docker image

From the main repo page on GitHub click on the Releases heading then Draft a new release.

From the Choose a tag dropdown enter the new version number in the format v0.1.0, then click Generate release notes to automatically include information on changes made since the previous release.

Add any other necessary comments then click Publish release.

The docker.yaml GitHub workflow will then build a Docker image of the new version and push the build to Docker Hub.