Skip to main content
=================
Browler
=================

Browler is a Selenium based web crawler. It was built out of a need to crawl and scrape content from Javascript heavy
web sites.

----------------------
Requirements
-----------------------

* Redis on localhost
* ``pip install browler``


------------
Example
------------
Perform a crawl with two concurrent processes with a limit of two visits per process. Additionally, do not follow any
links that contain ``bad-stuff`` in the url::

from browler import Browler
import re

config = {
"browser": "firefox",
"url": "https://en.wikipedia.org/wiki/Main_Page",
"limit": 2,
"processes": 2,
"plugins": [

],
"filters": [
re.compile('bad-stuff'),
]
}
crawler = Browler(config)
crawler.crawl()

-------------
Features
-------------

* Concurrent processing for faster web crawls (Use at your own risk)
* Selenium Grid support with remote browser configuration
* Plugin support with hooks to customize the crawler

------------------
Configuration
------------------

:browser: ``(string)(required)`` must be supported Selenium browser ``firefox``, ``chrome``, ``phantomjs``, or (``remote`` requires see ``remote`` below)
:remote: ``(dict)(optional)`` required when ``browser`` is set to ``remote``

:browser: ``(string)`` see ``browser``
:url: ``(string)`` url of the remote host ie. ``http://localhost:4444/wd/hub``

:limit: ``(int)`` maximum number of visits per process
:plugins: ``(list)`` instances which inherit from ``browler.plugins.base.Plugin`` see :doc:`plugins`
:filters: ``(list)`` filters out any links that match the following regular expressions compiled with ``re.complile``

Metadata

Release files for browler 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for browler 0.1.2
File Size Uploaded Provenance Actions
browler-0.1.2.tar.gz 6.2 kB Without provenance Details for browler-0.1.2.tar.gz

Release files / browler-0.1.2.tar.gz

Download URL browler-0.1.2.tar.gz
Size 6.2 kB
Tags Source
SHA-256 checksum
How to use checksums
2eecca215afa55d7cd91c70abd445746123fe646d7cef6a4ff032b1ff33a65ba
BLAKE2b-256 checksum
How to use checksums
2270a8242a11311bb3e691b07f3f18e15ba90f5a0007c7a08368322f559468ba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for browler 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.2 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page