Skip to main content

项目说明

  • ...

更新历史

  • 加入爬虫案例

爬虫案列

爬取皮肤图片

代码

import json
import os
import re
from concurrent.futures import ThreadPoolExecutor

from loguru import logger

from craws import AirSpider


class WZRYSkinSpider(AirSpider):
    def parse_url(self, url, b=False):
        try:
            res = self.get(url)
            res.encoding = "GBK"
            assert res.status_code == 200, "Code not is 200"
            return res.content if b else res.text
        except:
            pass

    def download_img(self, img_url, hero_name, hero_img, num):
        b_data = self.parse_url(img_url, b=True)
        if b_data is None:
            return
        with open(hero_img, "wb") as f:
            f.write(b_data)
        logger.success(f"{hero_name} 第{num}张皮肤图片 下载完毕")

    def process_hero(self, hero_id, name):
        logger.info(f"{hero_id}\t{name}\t处理中...")

        hero_dir = f"./英雄皮肤/{name}"
        if not os.path.exists(hero_dir):
            os.makedirs(hero_dir, exist_ok=True)

        with ThreadPoolExecutor(max_workers=20) as pool:
            for num in range(1, 20):
                hero_img = f"{hero_dir}/皮肤_{num}.png"
                if os.path.exists(hero_img):
                    logger.warning(f"{hero_img}已下载过,跳过")
                    continue
                img_url = f"https://game.gtimg.cn/images/yxzj/img201606/skin/hero-info/{hero_id}/{hero_id}-bigskin-{num}.jpg"
                pool.submit(self.download_img, img_url, name, hero_img, num)

    def crawl(self):
        api_url = "https://game.gtimg.cn/images/yxzj/web201706/js/heroid.js"
        text = self.parse_url(api_url)
        search_result = re.search('var module_exports = ({.*?})', text, re.S)
        hero_info_str = search_result.group(1)
        hero_info_str = re.sub("'", '"', hero_info_str)
        hero_info_dict = json.loads(hero_info_str)

        with ThreadPoolExecutor(max_workers=10) as pool:
            for hero in hero_info_dict:
                name, id = hero_info_dict[hero], hero
                pool.submit(self.process_hero, id, name)


if __name__ == '__main__':
    WZRYSkinSpider().crawl()

Metadata

Release files for craws 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for craws 0.1.1
File Size Uploaded Provenance Actions
craws-0.1.1.tar.gz 8.6 kB Without provenance Details for craws-0.1.1.tar.gz

Release files / craws-0.1.1.tar.gz

Download URL craws-0.1.1.tar.gz
Size 8.6 kB
Tags Source
SHA-256 checksum
How to use checksums
1f59f65dbd8b30d13cbc048672fb819cc06f58a691be7aca1a4797b8fa1fd1cb
BLAKE2b-256 checksum
How to use checksums
ba36c45773710f668de77b35ebd8e1e896b4ee23755b2053ba4126e75cb09432
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.11.10

Security for craws 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page