masquer 1.0.1

Weighted-random user-agent and referer data for GET requests

Masquer

A tool to generate random user-agent and referer data for GET requests.

Overview

Use masquer to obtain any combination of a random user-agent, referer or header data template, then use this with a library like requests to control the session data you send to other services.

The user-agent data is drawn from this list of the most common desktop user-agents, and referer data is taken from this list of search engines with the largest global market share.

Weighted random selections are made from those lists to approximate authentic header data patterns.

A basic header template with common attributes — including the recommended "Upgrade-Insecure-Requests": "1" — is also provided and defaults to the most common referer and user-agent data from the above lists.

Note on privacy

Controlling header data in this way can help to preserve privacy and hinder third-party tracking behaviour, by blending part of your web profile with the most common configurations.

It does not provide anonymity — that is a much more complex topic, and the open-source Privacy Guides are a good place to start.

Installation

To get hold of masquer either install the package from PyPI into your project's virtual environment, or clone the GitHub repo for the full code base.

PyPI package

Install the masquer package from PyPI to retrieve just the tool with no extras.

Activate your existing project's virtual environment, then download masquer using a package manager. The below example uses Astral's uv; substitute pip by dropping "uv" or use another package manager as needed:

$ uv pip install masquer

GitHub repo

Clone the masquer repo from GitHub for the full source code. The repo includes the JSON source files used to generate the header data, a script to sync the programme if updates are made to the JSON files, and a test suite.

$ git clone [e-mail]:essteer/masquer

The functional code within the package src directory has no dependencies beyond Python built-in modules. If you intend to make changes to your cloned version of the repo, you may optionally install the project.optional-dependencies declared in the pyproject.toml file.

First create and activate a virtual environment — the below example uses Astral's uv; substitute pip or use another package manager as needed — then install the dev dependencies:

$ uv venv
$ source .venv/bin/activate
$ uv pip install hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4

$ uv venv
$ .venv\Scripts\activate
$ uv pip install hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4

Operation

Interact with masquer via the masq method:

from masquer import masq

The masq function accepts up to three boolean parameters:

useragent = masq(
  ua = True,  # user-agent, defaults to True
  rf = False,  # referer, defaults to False
  hd = False,  # header-data, defaults to False
)

Examples

User-agent only

By default only ua is set to True, so each of the following methods may be used to return just one randomly generated user-agent:

>>> useragent_1 = masq()
>>> useragent_2 = masq(True)
>>> useragent_3 = masq(ua=True)
>>>
>>> useragent_1
'{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3"}'
>>> useragent_2
'{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0."}'
>>> useragent_3
'{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}'

Referer only

By the same logic, these methods will each return just one randomly generated referer:

>>> referer_1 = masq(False, True)
>>> referer_2 = masq(ua=False, rf=True)
>>> referer_3 = masq(ua=False, rf=True, hd=False)
>>>
>>> referer_1
'{"Referer": "https://www.google.com/"}'
>>> referer_2
'{"Referer": "https://www.google.com/"}'
>>> referer_3
'{"Referer": "https://bing.com/"}'

Header-data

By default, the header data template supplies the most common user-agent and referer values as fixed, and can be accessed via the following methods:

>>> default_header_1 = masq(False, False, True)
>>> default_header_2 = masq(ua=False, hd=True)
>>> default_header_3 = masq(ua=False, rf=False, hd=True)
>>>
>>> default_header_1 == default_header_2 == default_header_3
True
>>> default_header_1
'{"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-US,en;q=0.5;", "Referer": "https://www.google.com/", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}'

As per the individual use-cases, get weighted random user-agent and referer values in the header by omitting the ua value (which defaults to True) and setting rf to True. A non-exhaustive selection of examples is provided below:

>>> # header with random user-agent and fixed referer
>>> random_header_1 = masq(hd=True)  
>>> random_header_2 = masq(True, False, True)
>>>
>>> # header with fixed user-agent and random referer
>>> random_header_3 = masq(False, True, True)
>>> random_header_4 = masq(ua=False, rf=True, hd=True)
>>>
>>> # header with random user-agent and random referer
>>> random_header_5 = masq(rf=True, hd=True)
>>> random_header_6 = masq(True, True, True)

Local development

The following details will assist with making and testing changes to a cloned version of the repository.

Updates

The root directory includes update.py: if you make changes to the JSON assets stored in the assets directory, sync those changes with the assets.py file inside the masquer package by running update.py from the terminal:

$ python3 update.py
Asset update successful

$ python update.py
Asset update successful

Tests

masquer uses Python's in-built unittest module.

To run the entire test suite using discover, or specify an individual test file from the tests directory — for example test_assets.py — run one of the following shell commands from the project root directory:

$ python3 -m unittest discover  # run all tests
$ python3 -m unittest tests.test_assets  # test a single file

$ python -m unittest discover  # run all tests
$ python -m unittest tests.test_assets  # test a single file