Highlights
Re-use the same socket connection for multiple requests (HTTPConnectionPool and HTTPSConnectionPool) (with optional client-side certificate verification).
File posting (encode_multipart_formdata).
Built-in redirection and retries (optional).
Supports gzip and deflate decoding.
Thread-safe and sanity-safe.
Small and easy to understand codebase perfect for extending and building upon. For a more comprehensive solution, have a look at Requests.
What’s wrong with urllib and urllib2?
There are two critical features missing from the Python standard library: Connection re-using/pooling and file posting. It’s not terribly hard to implement these yourself, but it’s much easier to use a module that already did the work for you.
The Python standard libraries urllib and urllib2 have little to do with each other. They were designed to be independent and standalone, each solving a different scope of problems, and urllib3 follows in a similar vein.
Why do I want to reuse connections?
Performance. When you normally do a urllib call, a separate socket connection is created with each request. By reusing existing sockets (supported since HTTP 1.1), the requests will take up less resources on the server’s end, and also provide a faster response time at the client’s end. With some simple benchmarks (see test/benchmark.py ), downloading 15 URLs from google.com is about twice as fast when using HTTPConnectionPool (which uses 1 connection) than using plain urllib (which uses 15 connections).
This library is perfect for:
Talking to an API
Crawling a website
Any situation where being able to post files, handle redirection, and retrying is useful. It’s relatively lightweight, so it can be used for anything!
Examples
Go to urllib3.readthedocs.org for more nice syntax-highlighted examples.
But, long story short:
import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://google.com/')
print r.status, r.data
The PoolManager will take care of reusing connections for you whenever you request the same host. For more fine-grained control of your connection pools, you should look at ConnectionPool.
Run the tests
Running the test command will install the necessary dependencies and run the tests.
$ python setup.py test ...........E
Tests can also be run using nosetests.
Some tests require the dummy_server.py running to pass. In another terminal:
$ python -O test/dummy_server.py
Then run the tests again:
$ nosetests ............
Success!
Contributing
Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a Contributor Friendly tag for issues that should be ideal for people who are not very familiar with the codebase yet.
Fork the urllib3 repository on Github to start making your changes.
Write a test which shows that the bug was fixed or that the feature works as expected.
Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to CONTRIBUTORS.txt.
Changes
1.0 (2011-10-08)
Added PoolManager with LRU expiration of connections (tested and documented).
Added ProxyManager (needs tests, docs, and confirmation that it works with HTTPS proxies).
Added optional partial-read support for responses when preload_content=False. You can now make requests and just read the headers without loading the content.
Made response decoding optional (default on, same as before).
Added optional explicit boundary string for encode_multipart_formdata.
Convenience request methods are now inherited from RequestMethods. Old helpers like get_url and post_url should be abandoned in favour of the new request(method, url, ...).
Refactored code to be even more decoupled, reusable, and extendable.
License header added to .py files.
Embiggened the documentation: Lots of Sphinx-friendly docstrings in the code and docs in docs/ and on urllib3.readthedocs.org.
Embettered all the things!
Started writing this file.
0.4.1 (2011-07-17)
Minor bug fixes, code cleanup.
0.4 (2011-03-01)
Better unicode support.
Added VerifiedHTTPSConnection.
Added NTLMConnectionPool in contrib.
Minor improvements.
0.3.1 (2010-07-13)
Added assert_host_name optional parameter. Now compatible with proxies.
0.3 (2009-12-10)
Added HTTPS support.
Minor bug fixes.
Refactored, broken backwards compatibility with 0.2.
API to be treated as stable from this version forward.
0.2 (2008-11-17)
Added unit tests.
Bug fixes.
0.1 (2008-11-16)
First release.
3 maintainers
Metadata
3 maintainers
Release files for urllib3 1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| urllib3-1.0.tar.gz | 22.8 kB | Without provenance | Details |
Release files / urllib3-1.0.tar.gz
| Download URL | urllib3-1.0.tar.gz |
|---|---|
| Size | 22.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ca6b37da0353a907bd3b93a15400c0826bf9ec4088b60911b3e8fe0fdec6865b
|
|
BLAKE2b-256 checksum How to use checksums |
d3224cbcf82178d038efe03fc33093df8d0792c7e143748fc5bc3f43be4b83fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for urllib3 1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
