Skip to main content

TLS and HTTP signature and fingerprint library

Project description

TS1 - A TLS and HTTP signature and fingerprint library

TS1 is a Python library for building fingerprints of web clients. A fingerprint in this context is a string that identifies a web client. Unlike a cookie, it does not distinguish between web users, but rather between different client software, e.g. different browsers, command line utilities and web libraries.

TS1 can currently calculate two kinds of signatures:

TS1 was created as part of the work on curl-impersonate, a curl fork that fakes its own TLS and HTTP/2 fingerprint to look like a browser.

Important note: TS1 is currently in an alpha stage. The signatures are likely to change slightly in the coming future before it stabilizies.

Installation

pip install ts1-signatures

TLS fingerprint

Each web client has a unique TLS fingerprint due to the multitude of ways of configuring the TLS protocol. Specifically, TS1 relies on the TLS client hello message which is the first message sent by the client in a TLS handshake.

In TS1, the parameters of the TLS client hello are encoded into a json (truncated here for clearness):

{
    "client_hello": {
        "record_version": "TLS_VERSION_1_0",
        "handshake_version": "TLS_VERSION_1_2",
        "session_id_length": 32,
        "ciphersuites": [
            "GREASE",
            4865,
            4866,
            52392,
            49171,
            49172,
            156,
            157,
            47,
            53
        ],
        "comp_methods": [
            0
        ],
        "extensions": [
            {
                "type": "server_name"
            },
            {
                "type": "extended_master_secret",
                "length": 0
            },
            {
                "type": "renegotiation_info",
                "length": 1
            },
            {
                "type": "supported_groups",
                "length": 10,
                "supported_groups": [
                    "GREASE",
                    29,
                    23,
                    24
                ]
            }
        ]
    }
}

The JSON is then converted to a canonical form, with one space after separators and keys sorted alphabetically:

{"client_hello": {"ciphersuites": ["GREASE", 4865, 4866, 52392, 49171, 49172, 156, 157, 47, 53], "comp_methods": [0], "extensions": [{"type": "server_name"}, {"length": 0, "type": "extended_master_secret"}, {"length": 1, "type": "renegotiation_info"}, {"length": 10, "supported_groups": ["GREASE", 29, 23, 24], "type": "supported_groups"}], "handshake_version": "TLS_VERSION_1_2", "record_version": "TLS_VERSION_1_0", "session_id_length": 32}}

which is then hashed with SHA1 to produce the TS1 signature hash:

cfee1dd35c55244cba1a7dba771d9df61d0dca47

Compared to JA3

JA3 is a library for creating TLS fingerprints which is already widely adopted. TS1 is similar to JA3 (and was inspired by it) but has the following advantages:

  • TS1 signatures encode more parameters than JA3. While it is possible for two different TLS client hello messages to have the same JA3 signature, it is far less likely with TS1.
  • TS1 signatures are JSON documents which are easier to read and understand by humans, and are forward-compatible with future extensions to the TLS protocol.

The disadvantage of TS1 is that its signatures, in their raw form, are much more verbose and not as succinct as JA3.

Usage

ts1.tls.TLSSignature is a class that encodes a TLS client's signature. It has two important functions:

  • TLSSignature.canonicalize() will produce the canonical JSON form.
  • TLSSignature.hash() will return the SHA1 hash as returned by hashlib.sha1

TS1 comes with a utility process_pcap() function to extract signatures from PCAP files:

import ts1

with open("/path/to/pcap", "rb") as pcap:
    for tls_client in ts1.tls.process_pcap(pcap):
        print("Client IP: {}".format(tls_client["src_ip"]))
        # TLSSignature object
        signature = tls_client["signature"]
        print("Client' TLS signature SHA1: {}".format(
            signature.hash().hexdigest()
        ))

HTTP/2 fingerprint

Each web client that supports HTTP/2 sends multiple HTTP/2 frames upon initiating a connection to a server. These frames contain various settings configured by the client. Each web client configures these settings differently, which makes it possible to build an HTTP/2 signature to identify the client.

The HTTP/2 frames are encoded into a JSON format (truncated here for clearness):

{
    "frames": [
        {
            "frame_type": "SETTINGS",
            "stream_id": 0,
            "settings": [
                {
                    "id": 1,
                    "value": 65536
                },
                {
                    "id": 4,
                    "value": 131072
                },
                {
                    "id": 5,
                    "value": 16384
                }
            ]
        },
        {
            "frame_type": "WINDOW_UPDATE",
            "stream_id": 0,
            "window_size_increment": 12517377
        },
        {
            "frame_type": "PRIORITY",
            "stream_id": 3,
            "priority": {
                "dep_stream_id": 0,
                "weight": 201,
                "exclusive": false
            }
        },
        {
            "frame_type": "HEADERS",
            "stream_id": 15,
            "pseudo_headers": [
                ":method",
                ":path",
                ":authority",
                ":scheme"
            ]
        }
    ]
}

which, as in the TLS case, is converted to a canonical form and hashed to produce a SHA1 hash:

c9bb208868a10863867841a2e5bcb3b903719784

Usage

ts1.http2.HTTP2Signature is a class that encodes an HTTP/2 signature. It has two important functions:

  • HTTP2Signature.canonicalize() will produce the canonical JSON form.
  • HTTP2Signature.hash() will return the SHA1 hash as returned by hashlib.sha1

TS1 comes with a utility process_nghttpd_log() function to extract signatures from nghttpd log (nghttpd is a small HTTP/2 server):

import ts1

with open("/path/to/log", "r") as logfile:
    for http2_client in ts1.http2.process_nghttpd_log(logfile.read()):
        print("Client ID: {}".format(http2_client["client_id"]))
        # HTTP2Signature object
        signature = http2_client["signature"]
        print("Client's HTTP/2 signature SHA1: {}".format(
            signature.hash().hexdigest()
        ))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ts1-signatures-0.1.6.tar.gz (16.5 kB view details)

Uploaded Source

Built Distribution

ts1_signatures-0.1.6-py3-none-any.whl (15.7 kB view details)

Uploaded Python 3

File details

Details for the file ts1-signatures-0.1.6.tar.gz.

File metadata

  • Download URL: ts1-signatures-0.1.6.tar.gz
  • Upload date:
  • Size: 16.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.8.10

File hashes

Hashes for ts1-signatures-0.1.6.tar.gz
Algorithm Hash digest
SHA256 bec4d11383d963a59010ac3f0490f4126bab686a78690c1ddefdf7b37c8e240d
MD5 ab6e3a899526c325c343d71b7fc1164e
BLAKE2b-256 01461ef1011a06cc029d9961a4e86245af57f8da57be960fe7d4fa766990f6e7

See more details on using hashes here.

File details

Details for the file ts1_signatures-0.1.6-py3-none-any.whl.

File metadata

  • Download URL: ts1_signatures-0.1.6-py3-none-any.whl
  • Upload date:
  • Size: 15.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.8.10

File hashes

Hashes for ts1_signatures-0.1.6-py3-none-any.whl
Algorithm Hash digest
SHA256 ec9399fc559a99e6fbafc51bcf049c820762446e301cc5fd9706e8ab67d8ff82
MD5 230fe7a12a8dc1eed033409cd8dddf2a
BLAKE2b-256 91f5cf5c1bd986c510eb52e1ce4d5d2e1a52ceccedeeaf009fc856fc4821bcb9

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page