Web Data Extraction Library Written in Python
Project description
Wextracto is a toolkit for command-line web data extraction.
Installation
$ pip install wextracto
Kicking the Tyres
$ echo -e "[wex]\nsitemaps=wex.sitemaps:urls_from_sitemaps" > entry_points.txt $ wex "http://www.ebay.com/robots.txt"
Documentation
The documentation can be found here:
http://wextracto.readthedocs.org/en/latest/index.html
Release History
0.15.1 (2017-10-24)
- Add warc_protocol, warc_version, warc_headers to wex response
- Some (partial) support for Python 2.6
0.14.1 (2017-08-17)
- Fix errors in PhantomJS responses
- Handle non utf-8 urls
0.14.0 (2017-07-10)
- Ensure utf-8 is tried first even if not declared
0.12.0 (2017-04-03)
- Support onInitialized in PhantomJS required modules
0.10.1 (2016-09-26)
- Add –label argument for easy process-wide labelling
0.9.6 (2016-09-09)
- Fix shutdown error caused by daemon thread for timeout with phantomjs
- Fix handling of directories in tarfiles read from stdin (-)
0.9.5 (2016-09-06)
- Small fix to avoid non-integer status code when error occur with PhantomJS
0.9.4 (2016-06-21)
- Support ‘params’ keyword argument on URL.get
0.9.2 (2016-05-13)
- Fix bug in handling HTML comments when fixing numeric character references
0.9.1 (2016-04-26)
- Fix bug when using nested Cache objects
0.9.0 (2016-04-16)
- Add support for reading WARC response format
0.8.8 (2016-04-11)
- Fix bug in handling of invalid numeric character references
0.8.5 (2015-12-07)
- Allow utf-8 in HTTP headers (only applies to PY2)
0.8.3 (2015-09-23)
- Fix bug in HTTP decode caused by magic bytes handling.
0.8.2 (2015-09-21)
- Add magic_bytes to Response for more reliable wex.http:decode behaviour.
0.7.9 (2015-08-18)
- Re-worked encoding for HTML to pre-parse
0.7 (2015-06-04)
- Better proxy support
0.4 (2015-02-12)
- Now we flatten labels and values.
- href and src become href_url and src_url.
0.3 (2014-12-29)
- Some API changes + switch to “tab-separated JSON”.
0.2.2 (2014-10-24)
- Uploaded sdist to PyPI for “pip install wextracto” simplicity.
0.1 (2014-10-16)
- Initial release as open source
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size Wextracto-0.15.1-py2.py3-none-any.whl (50.9 kB) | File type Wheel | Python version 2.7 | Upload date | Hashes View |
Filename, size Wextracto-0.15.1.tar.gz (45.3 kB) | File type Source | Python version None | Upload date | Hashes View |
Close
Hashes for Wextracto-0.15.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93ebefc43330e63806ede165f081ffb0ce7b07e87254996f00283707e8a97325 |
|
MD5 | 69c84082f78e4f35ebf0eddad4fb0a7e |
|
BLAKE2-256 | 2394bcc5cbfb977d03b6fec9ef06265c4cf0d511ac75e55b1029d50b315a0cad |