Python requests.HTTPAdapter for acw_sc__v2
acw_sc__v2 is a cookie used by some websites to prevent crawlers.
When the website detects that the request is sent by a crawler, it returns a javascript challenge. The crawler needs to solve the challenge and resend the request with the cookie set to the challenge value.
This project provides a Python requests.HTTPAdapter to resolve the challenge automatically.
Usage
pip install acw-sc-v2-py
# start acw_sc_v2_solver
git clone https://github.com/WangYihang/acw-sc-v2.js.git
cd acw-sc-v2.js
npm install
node app.js
import requests
session = requests.Session()
# add the following code to your original requests code
from acw_sc_v2_py import acw_sc__v2
adapter = acw_sc__v2.AcwScV2Adapter(acw_sc_v2_solver_url="http://localhost:3000/")
session.mount("http://", adapter)
session.mount("https://", adapter)
response = session.get("https://www.example.com/")
print(response.text)
Use Case
Before using acw-sc-v2-py
import requests
session = requests.Session()
for i in range(8):
response = session.get("https://www.example.com/")
print(response.text)
Usually, you will get blocked after sending 2 consecutive requests to the same website. The response will be like the following HTML code which requires you to solve a javascript challenge. If the web page is opened in a browser, the browser will automatically solve the challenge.
<html><script>
var arg1='70D9569CD5E5895C84F284A09503B1598C5762A1';
var _0x4818=['\x63\x73\x4b\x48\x77\x71\x4d\x49,...
function setCookie(name,value){var expiredate=new Date();...
function reload(x) {setCookie("acw_sc__v2", x);...
</script></html>
After using acw-sc-v2-py
import requests
# step 1: import
from acw_sc_v2_py import acw_sc__v2
session = requests.Session()
# step 2: create adapter
adapter = acw_sc__v2.AcwScV2Adapter()
# step 3: mount adapter
session.mount("http://", adapter)
session.mount("https://", adapter)
for i in range(8):
response = session.get("https://www.example.com/")
print(response.text)
By using acw-sc-v2-py, you will get the normal response. The
acw_sc__v2 will handle the javascript challenge and automatically
update the cookie.
Enable logging
Prepend the following code to enable detailed log for acw-sc-v2-py.
import logging
logger = logging.getLogger("acw_sc_v2_py.acw_sc__v2")
logger.setLevel(logging.DEBUG)
logger.propagate = True
import requests
from acw_sc_v2_py import acw_sc__v2
session = requests.Session()
adapter = acw_sc__v2.AcwScV2Adapter()
session.mount("http://", adapter)
session.mount("https://", adapter)
for i in range(8):
response = session.get("https://www.example.com/")
print(response.text)
The log will be like the following.
[2024-01-26 22:01:26] INFO:root:detected anti spam is triggered
[2024-01-26 22:01:28] INFO:root:cookie generated acw_sc__v2=65b3bb3601fe9ab002c5c1ff58fc71a1115e8322
[2024-01-26 22:01:28] INFO:root:resending the origin request
<!DOCTYPE html></html>
[2024-01-26 22:01:29] INFO:root:cookie set acw_sc__v2=65b3bb3601fe9ab002c5c1ff58fc71a1115e8322
[2024-01-26 22:01:30] INFO:root:anti spam is not triggered
<!DOCTYPE html></html>
...
References
- NodeJS version for Server API (acw-sc-v2.js)
- GoLang version for client code (acw-sc-v2-go)
1 maintainer
Release files for acw-sc-v2-py 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| acw_sc_v2_py-0.0.6.tar.gz | 3.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acw_sc_v2_py-0.0.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.5 kB
Release files / acw_sc_v2_py-0.0.6.tar.gz
| Download URL | acw_sc_v2_py-0.0.6.tar.gz |
|---|---|
| Size | 3.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2389e6970b1a4ebd5edc74e3ea4467256a7541c765a8cb6f4a98fe941d5444d4
|
|
BLAKE2b-256 checksum How to use checksums |
2e2677b495b6704db8b80e28cc0cf3ecf172fd437cfcb6f4b70f52509bd0ca9e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.14
|
Release files / acw_sc_v2_py-0.0.6-py3-none-any.whl
| Download URL | acw_sc_v2_py-0.0.6-py3-none-any.whl |
|---|---|
| Size | 4.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6f59916faa9850da765849a7dca3d45a9cef7a5a5c5411cb2d8875577c0f7dab
|
|
BLAKE2b-256 checksum How to use checksums |
25e538bc5c6a53662fc9afb68b883a7859bdab951b553f85c4f0181f189b4b03
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.14
|
