shopee-crawler 0.1.6

Crawler for shopee

Shopee-crawler

This is a crawling tool.

High performance.

Easy to use.

---

Requirement

python>=3.7

Install

• Using setup.py

git clone https://github.com/lthoangg/shopee-crawler
python/python3 setup.py install

• Using pip

pip install shopee-crawler

How to use

• Crawl by shop url

from shopee_crawler import crawl_by_shop_url

data = crawl_by_shop_url(shop_url='shop_url')
# print(data)

• Crawl by category url

from shopee_crawler import crawl_by_cat_url

data = crawl_by_cat_url(cat_url='cat_url')
# print(data)

• Crawl by keyword (search)

from shopee_crawler import crawl_by_search

data = crawl_by_search(keyword='keyword')
# print(data)

• Crawl all categories

from shopee_crawler import crawl_cat_list

data = crawl_cat_list()
# print(data)

Usage

• About 12.000 rows (products) in 2-3 seconds (32 workers)

• About 12.000 rows (products) in 5-6 seconds (16 workers)

Test yourself

import time
from shopee_crawler.crawler import crawl_by_search

keyword='điện thoại samsung'

# Start time
start = time.time()

# Crawl
crawl_by_search(keyword=keyword)

# End time
end = time.time()

# Time crawling
print("Time : ",end - start, "seconds")