Skip to main content

Abstractions of web interactions

Project description

Web Traversal Library

The Web Traversal Library (WTL) is a Python library for abstracting web interactions on top of a base execution layer such as Selenium.

Build Status License Developed at Klarna

Installation

Run pip install webtraversallibrary. That's it.

Usage example

Glossary

You will find more information in the API docs. As a high-level overview, common terms in the documentation are:

  • Workflow: The main orhcestrating class handling the main "event loop". Sometimes "schema" is also used for the specification of a certain workflow.

  • View: A static snapshot of a current website in a tab, with metadata associated to the page and its elements, possibly augmented with certain ML classifiers.

  • Policy: WTL is based on principles of reinforcement learning, where a policy is a function of the current state (here, the snapshots of current open tabs) to a set of actions.

  • Classifier: These, along with preload_callbacks and postload_callbacks are arbitray code that is executed on each workflow iteration. A classifier takes a set of elements and returns either a subset or a mapping from elements to numeric scores.

  • Config: A helper class containing string, numeric, or boolean values for a number of configurations related to WTL. Some are pregrouped under certain umbrella names, such as desktop (running as a Desktop browser, the default is mobile emulation), but all values can be arbitrarily set. See the documentation for the Config class for more information.

Getting started

See the documentation at webtraversallibrary.readthedocs.io!

Also watch "Machine Learning to Auto-Navigate Websites" given at PyCon SE 2020 for an introduction and examples.

General architecture

The flow in a workflow is as follows:

  1. Initialize the workflow with given config
  2. Navigate to given URLs
  3. Snapshot the pages
  4. Run all classifiers
  5. Check if the goal is fulfilled, if so exit
  6. Call policy with the current view(s)
  7. Execute the returned action(s)
  8. Goto 3

For more examples and usage, please run make docs and look at docs/build/html/index.html.

Development setup

All development requirements are in requirements.txt. Install the packages from pip. Helper commands are available to create a virtual environment - make env-create and make env-update.

To lint the JavaScript files (not required unless you're editing them) you need jshint which is available from npm.

How to contribute

See our guide on contributing.

Release History

See our changelog.

License

Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

webtraversallibrary-0.11.1.tar.gz (203.0 kB view hashes)

Uploaded Source

Built Distribution

webtraversallibrary-0.11.1-py3-none-any.whl (234.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page