Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Abstractions of web interactions

Project description

Web Traversal Library

The Web Traversal Library (WTL) is a Python library for abstracting web interactions on top of a base execution layer such as Selenium.

Build Status License Developed at Klarna Documentation Status


Run pip install webtraversallibrary. That's it.

Usage example


You will find more information in the API docs. As a high-level overview, common terms in the documentation are:

  • Workflow: The main orhcestrating class handling the main "event loop". Sometimes "schema" is also used for the specification of a certain workflow.

  • View: A static snapshot of a current website in a tab, with metadata associated to the page and its elements, possibly augmented with certain ML classifiers.

  • Policy: WTL is based on principles of reinforcement learning, where a policy is a function of the current state (here, the snapshots of current open tabs) to a set of actions.

  • Classifier: These, along with preload_callbacks and postload_callbacks are arbitray code that is executed on each workflow iteration. A classifier takes a set of elements and returns either a subset or a mapping from elements to numeric scores.

  • Config: A helper class containing string, numeric, or boolean values for a number of configurations related to WTL. Some are pregrouped under certain umbrella names, such as desktop (running as a Desktop browser, the default is mobile emulation), but all values can be arbitrarily set. See the documentation for the Config class for more information.

Getting started

See the documentation at!

Also watch "Machine Learning to Auto-Navigate Websites" given at PyCon SE 2020 for an introduction and examples.

General architecture

The flow in a workflow is as follows:

  1. Initialize the workflow with given config
  2. Navigate to given URLs
  3. Snapshot the pages
  4. Run all classifiers
  5. Check if the goal is fulfilled, if so exit
  6. Call policy with the current view(s)
  7. Execute the returned action(s)
  8. Goto 3

For more examples and usage, please run make docs and look at docs/build/html/index.html.

Development setup

All development requirements are in requirements.txt. Install the packages from pip. Helper commands are available to create a virtual environment - make env-create and make env-update.

To lint the JavaScript files (not required unless you're editing them) you need jshint which is available from npm.

How to contribute

See our guide on contributing.

Release History

See our changelog.


Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for webtraversallibrary, version 0.13.2
Filename, size File type Python version Upload date Hashes
Filename, size webtraversallibrary-0.13.2-py3-none-any.whl (241.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size webtraversallibrary-0.13.2.tar.gz (210.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page