humanbot 0.1.0

A human-like HTTP client: randomized browser headers, TLS impersonation, delays, and built-in HTML parsing.

🤖 humanbot

A human-like HTTP client for Python.

Randomized browser headers · TLS/JA3 impersonation · Smart delays · Built-in HTML parsing

---

✨ Why humanbot?

Plain requests looks nothing like a real browser — no realistic headers, no TLS fingerprint, no natural pacing between requests. humanbot fixes that. It wraps curl_cffi and BeautifulSoup into one clean, requests-style client that just feels human.

• 🧑‍💻 Realistic headers — randomized per request, tuned for desktop, mobile, and tablet
• 🔒 TLS/JA3 impersonation — looks like real Chrome/Firefox/Safari at the network level
• ⏱️ Natural pacing — optional randomized delays between requests
• 🥣 Parsing built-in — client.soup, CSS selectors, tables, lists, links, images — no extra imports
• 💾 One-line saving — dump HTML, JSON, text, or binary responses straight to disk
• 🧩 Drop-in feel — if you know requests, you already know humanbot

📦 Installation

pip install humanbot

Requirements: Python 3.8+

🚀 Quickstart

from humanbot import Human

with Human(device="desktop") as client:
    client.get("https://example.com")
    title = client.get_text("h1")
    links = client.get_links()
    print(title, links)

That's it — realistic headers, TLS impersonation, and HTML parsing, all in three lines. 🎉

🛠️ Usage

Creating a client

from humanbot import Human

client = Human(
    device="desktop",         # 📱 "mobile" · 💻 "desktop" · 📱 "tablet"
    delay_min_s=1,             # ⏱️ minimum delay before each request (seconds)
    delay_max_s=3,             # ⏱️ maximum delay before each request (seconds)
    lang_code="en-US",         # 🌍 override Accept-Language
    impersonate="chrome120",   # 🎭 curl_cffi TLS impersonation target
    parser="lxml",             # 🥣 BeautifulSoup parser
)

Making requests

client.get("https://example.com")
client.post("https://example.com/api", json={"key": "value"})
client.put(...)
client.patch(...)
client.delete(...)
client.head(...)
client.options(...)

Every method returns a standard Response object, and HTML responses are automatically parsed into client.soup behind the scenes. 🪄

🔍 Parsing helpers

client.get("https://example.com")

client.find("div.article")               # all matching elements
client.find_one("h1")                    # first matching element
client.get_text("h1")                    # text of first match, whitespace-normalized
client.get_attribute("img", "src")       # attribute of first match
client.get_all_attributes("img", "src")  # attribute of all matches
client.get_links()                       # all href values
client.get_images()                      # all img src values
client.extract_table("table")            # 2D list of cell text
client.extract_list("ul.items")          # list of item text

💾 Saving responses

client.save_html("page.html")
client.save_json("data.json")
client.save_text("page.txt")
client.save_binary("file.bin")
client.save_soup("pretty.html")     # prettified parsed HTML
client.download_file(url, "output.zip")

🧢 Managing headers

client.get_headers()
client.set_header("X-Custom", "value")
client.update_headers({"X-A": "1", "X-B": "2"})

Or generate a header set on its own, without spinning up a client:

from humanbot import HeadersGenerator

gen = HeadersGenerator()
gen.desktop_headers()
gen.mobile_headers()
gen.tablet_headers()
gen.random_headers()

📋 Full example

from humanbot import Human

with Human(device="mobile", delay_min_s=1, delay_max_s=2) as client:
    client.get("https://news.ycombinator.com")

    for title, link in zip(
        client.extract_list("table.itemlist", "span.titleline"),
        client.get_links("span.titleline a"),
    ):
        print(f"📰 {title} → {link}")

    client.save_html("hn.html")

⚠️ A responsible-use note

humanbot is built for legitimate scraping, testing, and automation. Always respect a site's robots.txt, terms of service, and rate limits — please don't use this to abuse or overload services you don't own or have permission to test.

🤝 Contributing

Issues and pull requests are welcome! Open one on GitHub.

📄 License

MIT © Arad Ahmadi

---

Made with 🖤 by Arad Ahmadi