Unicodec Package Documentation
This package provides functions for:
- decoding bytes content of HTML document into Unicode text
- detecting encoding of bytes content of HTML document
- normalization of encoding's name to canonical form, according to WHATWG HTML standard
Feel free to give feedback in Telegram groups: @grablab and @grablab_ru.
Installation
pip install -U unicodec
Usage Example #1
Download web document with urllib and convert its content to Unicode.
from urllib.request import urlopen
from unicodec import decode_content, detect_content_encoding
res = urlopen("http://lib.ru")
rawdata = res.read()
data = decode_content(rawdata, content_type_header=res.headers["content-type"])
print(data[:70])
print(detect_content_encoding(rawdata, res.headers["content-type"]))
Output:
<html><head><title>Lib.Ru: Библиотека Максима Мошкова</title></head><b
koi8-r
Usage Example #2
Download web document with urllib3 and convert its content to Unicode.
from urllib3 import PoolManager
from unicodec import decode_content, detect_content_encoding
res = PoolManager().urlopen("GET", "http://lib.ru")
rawdata = res.data
data = decode_content(rawdata, content_type_header=res.headers["content-type"])
print(data[:70])
print(detect_content_encoding(rawdata, res.headers["content-type"]))
Output:
<html><head><title>Lib.Ru: Библиотека Максима Мошкова</title></head><b
koi8-r
Usage Example #3
Convert names of encodings to canonical form (according to WHATWG HTML standard).
from unicodec.normalization import normalize_encoding_name
for name in ["iso8859-1", "utf8", "cp1251"]:
print("{} -> {}".format(name, normalize_encoding_name(name)))
Output:
iso8859-1 -> windows-1252
utf8 -> utf-8
cp1251 -> windows-1251
References
1 maintainer
Metadata
1 maintainer
Release files for unicodec 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| unicodec-0.2.0.tar.gz | 13.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| unicodec-0.2.0-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 24.3 kB
Release files / unicodec-0.2.0.tar.gz
| Download URL | unicodec-0.2.0.tar.gz |
|---|---|
| Size | 13.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a59748691835826f5f9915434543f3d96882a8eb70a882ba7f76445de0c96863
|
|
BLAKE2b-256 checksum How to use checksums |
3223508cd345d57c621da7a19925fdfb7dccabd379c727a4c493f353cd87c77a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.2
|
Release files / unicodec-0.2.0-py2.py3-none-any.whl
| Download URL | unicodec-0.2.0-py2.py3-none-any.whl |
|---|---|
| Size | 10.8 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
609dae02f66490a53a0ad0f84a340e2563424362c2d247c50e1e65dc5cef4655
|
|
BLAKE2b-256 checksum How to use checksums |
6e55a9d9d997205cde95f4f38f490011efd354f35a3a06b88e15c29ed597e1f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.2
|
