Weighted-random user-agent and referer data for GET requests
Project description
Masquer
A tool to generate random user-agent and referer data for GET requests.
Overview
Use masquer
to obtain any combination of a random user-agent, referer or header data template, then use this with a library like requests
to control the session data you send to other services.
The user-agent data is drawn from this list of the most common desktop user-agents, and referer data is taken from this list of search engines with the largest global market share.
Weighted random selections are made from those lists to approximate authentic header data patterns.
A basic header template with common attributes — like "Upgrade-Insecure-Requests": "1"
— is also provided and defaults to the most common referer and user-agent data from the above lists.
Note on privacy
Controlling header data in this way can help to preserve privacy and hinder third-party tracking behaviour, by blending part of your web profile with the most common configurations.
It does not provide anonymity — that is a much more complex topic, and the open-source Privacy Guides are a good place to start.
Installation
To get hold of masquer
either install the PyPI package into your project's virtual environment, clone the GitHub repo or pull the Docker image.
PyPI package
Install the masquer
package from PyPI to retrieve just the tool with no extras.
Activate your existing project's virtual environment, then download masquer
using a package manager. The below example uses Astral's uv
; substitute pip
by dropping "uv
" or use another package manager as needed:
$ uv pip install masquer
GitHub repo
Clone the masquer
repo from GitHub for the full source code. The repo includes the JSON source files used to generate the header data, a script to sync the programme if updates are made to the JSON files, and a test suite.
$ git clone git@github.com:essteer/masquer
The functional code within the package src
directory has no dependencies beyond Python built-in modules. If you intend to make changes to your cloned version of the repo, you may optionally install the project.optional-dependencies
declared in the pyproject.toml
file.
First create and activate a virtual environment — the below example uses Astral's uv
; substitute pip
or use another package manager as needed — then install the dev
dependencies:
$ uv venv
$ source .venv/bin/activate
$ uv pip install fastapi==0.111.0 hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4
$ uv venv
$ .venv\Scripts\activate
$ uv pip install fastapi==0.111.0 hatchling==1.24.2 pre-commit==3.7.1 ruff==0.4.4
FastAPI
The GitHub repo includes masquer API
, a FastAPI version of masquer
.
To self-host the API, install the FastAPI
optional dependency as declared in the pyproject.toml
file.
Activate the API from the root directory via:
$ fastapi run src/api/main.py
Then follow the instructions provided by FastAPI in the terminal.
By default, the FastAPI app will run on localhost. To view the API documentation, run the API and navigate to http://127.0.0.1:8000/docs
.
Docker image
For no good reason, the masquer API
is also available as a Docker
image:
$ docker pull essteer/masquer
Operation
Interact with masquer
via the masq
method:
from masquer import masq
The masq
function accepts up to three boolean parameters:
useragent = masq(
ua = True, # user-agent, defaults to True
rf = False, # referer, defaults to False
hd = False, # header-data, defaults to False
)
And returns the response in dictionary form:
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3'}
FastAPI
masquer API
is deployed on Fly.io - view the docs at https://masquer.fly.dev/docs
.
An example of how to use masquer API
:
$ curl -X GET \
'https://masquer.fly.dev/masq?ua=true&rf=false&hd=false' \
-H 'accept: application/json'
Docker
Run the masquer API
from a container on the masquer
Docker image as follows:
$ docker run -d --name masquer -p 8000:8000 esteer/masquer
Then interact as per the FastAPI instructions above.
Examples
User-agent only
By default only ua
is set to True
, so each of the following methods may be used to return just one randomly generated user-agent:
>>> useragent_1 = masq()
>>> useragent_2 = masq(True)
>>> useragent_3 = masq(ua=True)
>>>
>>> useragent_1
{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3"}
>>> useragent_2
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0."}
>>> useragent_3
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}
Referer only
By the same logic, these methods will each return just one randomly generated referer:
>>> referer_1 = masq(False, True)
>>> referer_2 = masq(ua=False, rf=True)
>>> referer_3 = masq(ua=False, rf=True, hd=False)
>>>
>>> referer_1
{"Referer": "https://www.google.com/"}
>>> referer_2
{"Referer": "https://www.google.com/"}
>>> referer_3
{"Referer": "https://bing.com/"}
Header-data
By default, the header data template supplies the most common user-agent and referer values as fixed, and can be accessed via the following methods:
>>> default_header_1 = masq(False, False, True)
>>> default_header_2 = masq(ua=False, hd=True)
>>> default_header_3 = masq(ua=False, rf=False, hd=True)
>>>
>>> default_header_1 == default_header_2 == default_header_3
True
>>> default_header_1
{"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-US,en;q=0.5;", "Referer": "https://www.google.com/", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}
As per the individual use-cases, get weighted random user-agent and referer values in the header by omitting the ua
value (which defaults to True
) and setting rf
to True
. A non-exhaustive selection of examples is provided below:
>>> # header with random user-agent and fixed referer
>>> random_header_1 = masq(hd=True)
>>> random_header_2 = masq(True, False, True)
>>>
>>> # header with fixed user-agent and random referer
>>> random_header_3 = masq(False, True, True)
>>> random_header_4 = masq(ua=False, rf=True, hd=True)
>>>
>>> # header with random user-agent and random referer
>>> random_header_5 = masq(rf=True, hd=True)
>>> random_header_6 = masq(True, True, True)
Local development
The following details will assist with making and testing changes to a cloned version of the repository.
Updates
The root directory includes a shell script update.sh
for convenience to update JSON assets for the masquer
package.
$ chmod +x update.sh
$ ./update.sh
Update successful
If using your own JSON data you can still make use of the update_assets()
function inside update.py
to sync changes with the assets.py
file inside the masquer
package.
Tests
masquer
uses Python's in-built unittest
module.
To run the entire test suite using discover
, or specify an individual test file from the tests
directory — for example test_assets.py
— run one of the following shell commands from the project root
directory:
$ python3 -m unittest discover # run all tests
$ python3 -m unittest tests.test_assets # test a single file
$ python -m unittest discover # run all tests
$ python -m unittest tests.test_assets # test a single file
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.