Abstractions of web interactions
Web Traversal Library
The Web Traversal Library (WTL) is a Python library for abstracting web interactions on top of a base execution layer such as Selenium.
pip install webtraversallibrary. That's it.
You will find more information in the API docs. As a high-level overview, common terms in the documentation are:
Workflow: The main orhcestrating class handling the main "event loop". Sometimes "schema" is also used for the specification of a certain workflow.
View: A static snapshot of a current website in a tab, with metadata associated to the page and its elements, possibly augmented with certain ML classifiers.
Policy: WTL is based on principles of reinforcement learning, where a policy is a function of the current state (here, the snapshots of current open tabs) to a set of actions.
Classifier: These, along with
postload_callbacksare arbitray code that is executed on each workflow iteration. A classifier takes a set of elements and returns either a subset or a mapping from elements to numeric scores.
Config: A helper class containing string, numeric, or boolean values for a number of configurations related to WTL. Some are pregrouped under certain umbrella names, such as
desktop(running as a Desktop browser, the default is mobile emulation), but all values can be arbitrarily set. See the documentation for the
Configclass for more information.
See the documentation at webtraversallibrary.readthedocs.io!
Also watch "Machine Learning to Auto-Navigate Websites" given at PyCon SE 2020 for an introduction and examples.
The flow in a workflow is as follows:
- Initialize the workflow with given config
- Navigate to given URLs
- Snapshot the pages
- Run all classifiers
- Check if the goal is fulfilled, if so exit
- Call policy with the current view(s)
- Execute the returned action(s)
- Goto 3
For more examples and usage, please run
make docs and look at
All development requirements are in requirements.txt. Install the packages from pip. Helper commands are available to create a virtual environment -
make env-create and
jshint which is available from npm.
How to contribute
See our guide on contributing.
See our changelog.
Copyright © 2020 Klarna Bank AB
For license details, see the LICENSE file in the root of this project.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size webtraversallibrary-0.13.2-py3-none-any.whl (241.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size webtraversallibrary-0.13.2.tar.gz (210.2 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for webtraversallibrary-0.13.2-py3-none-any.whl
Hashes for webtraversallibrary-0.13.2.tar.gz