masquer 1.3.5

A tool to generate random user-agent and referer data for HTTP requests

Masquer

A tool to generate random user-agent and referer data for HTTP requests.

Overview

Use masquer to obtain any combination of a random user-agent, referer or header data template, then use this with a library like requests to control the session data you send to other services.

The user-agent data is drawn from this list of the most common desktop user-agents, and referer data is taken from this list of search engines with the largest global market share.

Weighted random selections are made from those lists to approximate authentic header data patterns.

A basic header template with common attributes — like "Upgrade-Insecure-Requests": "1" — is also provided and defaults to the most common referer and user-agent data from the above lists.

Note on privacy

Controlling header data in this way can help to preserve privacy and hinder third-party tracking behaviour by blending part of your web profile with the most common configurations.

It does not provide anonymity — that is a much more complex topic, and the open-source Privacy Guides are a good place to start.

Documentation

The sections that follow describe different ways to use masquer.

For development purposes see the docs directory for notes on development, deployment and Docker.

• API
• Python package
  • Installation
  • Operation
  • Examples
• Git repository
• Docker image

API

An API for masquer is in deployment at https://masquer.fly.dev/masq — try it out with the interactive Swagger UI or ReDoc documentation.

The API returns a JSON array, making it compatible with any language that can make HTTP requests and parse JSON.

Here is an example using curl from the command line to get a random user-agent and referer:

$ curl -X GET 'https://masquer.fly.dev/api/v1/masq?ua=true&rf=true' -H 'accept: application/json'
[
  {
    "Referer":"https://www.google.com",
    "User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.3"
  }
]

The optional count parameter specifies the number of objects to return in the response. The default value is 1.

Refer to the API docs for other examples, or see more details below in the package documentation. The tests/requests/ directory also contains HTTP request examples for use in development and production.

Python package

Installation

To use the masquer Python package, create and activate a virtual environment then install masquer using a package manager.

The below example uses Astral's uv — substitute pip by dropping "uv" or use another package manager as preferred:

$ uv venv
$ source .venv/bin/activate
$ uv pip install masquer

The core tool obtained this way has no dependencies.

Operation

Interact with masquer via the masq method:

>>> from masquer import masq

The masq function accepts up to three boolean parameters...

>>> useragent = masq(
  ua = True,  # user-agent, defaults to True
  rf = False,  # referer, defaults to False
  hd = False  # header-data, defaults to False
)

...and returns the response as a dict object:

{
  'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.'
}

Examples

User-agent

By default only the ua parameter is set to True, so each of the following methods may be used to obtain just one randomly selected user-agent:

>>> masq()
{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3"}
>>> masq(True)
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0."}
>>> masq(ua=True)
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"}

Referer

By the same logic, the following methods will each return one randomly selected referer:

>>> masq(False, True)
{"Referer": "https://www.google.com/"}
>>> masq(ua=False, rf=True)
{"Referer": "https://www.google.com/"}
>>> masq(ua=False, rf=True, hd=False)
{"Referer": "https://bing.com/"}

Header-data with default user-agent and referer

The default header-data template supplies the most common user-agent and refer values as fixed values, and can be accessed via the following methods:

>>> masq(False, False, True)
>>> masq(ua=False, hd=True)
>>> masq(ua=False, rf=False, hd=True)

Each of the above function calls would return the following:

>>> default_header = masq(ua=False, rf=False, hd=True)
>>> default_header
{
  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", 
  "Accept-Encoding": "gzip, deflate, br", 
  "Accept-Language": "en-US,en;q=0.5;", 
  "Referer": "https://www.google.com/", 
  "Sec-Fetch-Dest": "document", 
  "Sec-Fetch-Mode": "navigate", 
  "Sec-Fetch-Site": "none", 
  "Sec-Fetch-User": "?1", 
  "Upgrade-Insecure-Requests": "1", 
  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3"
}

Header-data with random user-agent and referer

To get the header-data with randomly selected user-agent and/or referer data, pass those arguments as True in addition to the hd parameter as per the below examples:

>>> random_header = masq(rf=True, hd=True)  # ua=True by default
>>> random_header
{
  # ...
  "Referer": "https://duckduckgo.com",
  # ...
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113."
}

Git repository

Clone the masquer repo for the full source code, including the FastAPI app used to host the API introduced above.

The repo includes the JSON source files used to generate the header data, a script to update the JSON content to the latest data, and a test suite.

$ git clone [e-mail]:essteer/masquer

The package code within the src/masquer directory has no dependencies beyond built-in Python modules, so can be run right away in a virtual environment.

The update.sh shell script in the root directory updates the header.json and referer.json files in the assets directory to the latest versions, then uses this data to update the assets.py file used by the masquer package.

To use the update.sh script first install Beautiful Soup into the virtual environment:

$ source .venv/bin/activate
$ uv pip install beautifulsoup4==4.12.3

Then grant execution permissions to update.sh and run it — the output should appear similar to that displayed below:

$ chmod +x update.sh
$ ./update.sh
2024-09-17 14:34:03 - INFO - update.py:29 - Fetched user-agent data
2024-09-17 14:34:04 - INFO - update.py:74 - Fetched referer data
2024-09-17 14:34:04 - INFO - update.py:133 - Saved user-agent and referer JSON data to assets.py
2024-09-17 14:34:04 - INFO - update.sh - Asset update OK

If interested in making changes to the repo, see the development notes for additional details.

FastAPI

The Git repo includes the FastAPI version of masquer that is introduced above and hosted at https://masquer.fly.dev — the relevant code is located at src/api.

To self-host the API, install the FastAPI optional dependency as declared in the pyproject.toml file.

$ source .venv/bin/activate
$ uv pip install fastapi==0.111.0

FastAPI runs on localhost port 8000 by default — to amend this change the relevant uvicorn argument in src/api/main.py:

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="127.0.0.1", port=8000)

Run the app from the root directory via:

$ fastapi run src/api/main.py

Then follow the instructions FastAPI provides in the terminal.

To view the API's interactive documentation, run the app and navigate to http://127.0.0.1:8000/docs or http://127.0.0.1:8000/redoc.

Docker image

A Docker image for masquer has been developed for production use, and is publicly available.

To run masquer from a container, first pull the image from DockerHub:

$ docker pull essteer/masquer

Launch a container using the masquer image as follows:

$ docker run -d --name masquer -p 8000:8000 essteer/masquer

Then interact as per the API instructions above.