Skip to main content

A simple web-crawling framework, based on aiohttp.

Project description

PyPI Documentation Status

🔍 A powerful web-crawling framework, based on aiohttp.


  • Write your crawler in one Python script with asyncio

  • Schedule task with priority, fingerprint, exetime, recrawl…

  • Middleware: add handlers before or after task’s execution

  • Simple shortcuts to speed up scripting

  • Parse html conveniently with Parsel

  • Parse with rules and chained processors

  • Support JavaScript/browser-automation with pyppeteer

  • Stop and Resume: crawl periodically and persistently

  • Distributed work support with Redis


To install, simply use pip:

$ pip install acrawler

$ pip install uvloop      #(only Linux/macOS, for faster asyncio event loop)
$ pip install aioredis    #(if you need Redis support)
$ pip install motor       #(if you need MongoDB support)
$ pip install aiofiles    #(if you need FileRequest)


Documentation and tutorial are available online at and in the docs directory.

Sample Code


from acrawler import Crawler, Request, ParselItem, Handler, register, get_logger

class MovieItem(ParselItem):
   log = True
   css = {
      # just some normal css rules
      # see Parsel for detailed information
      "date": ".subtext a[href*=releaseinfo]::text",
      "time": ".subtext time::text",
      "rating": "span[itemprop=ratingValue]::text",
      "rating_count": "span[itemprop=ratingCount]::text",
      "metascore": ".metacriticScore span::text",

      # if you provide a list with additional functions,
      # they are considered as field processor function
      "title": ["h1::text", str.strip],

      # the following four fules is for getting all matching values
      # the rule starts with [ and ends with ] comparing to normal rules
      "genres": "[.subtext a[href*=genres]::text]",
      "director": "[h4:contains(Director) ~ a[href*=name]::text]",
      "writers": "[h4:contains(Writer) ~ a[href*=name]::text]",
      "stars": "[h4:contains(Star) ~ a[href*=name]::text]",

class IMDBCrawler(Crawler):
   config = {"MAX_REQUESTS": 4, "DOWNLOAD_DELAY": 1}

   async def start_requests(self):
      yield Request("", callback=self.parse)

   def parse(self, response):
      yield from response.follow(
            ".lister-list tr .titleColumn a::attr(href)", callback=self.parse_movie

   def parse_movie(self, response):
      url = response.url_str
      yield MovieItem(response.sel, extra={"url": url.split("?")[0]})

class HorrorHandler(Handler):
   family = "MovieItem"
   logger = get_logger("horrorlog")

   async def handle_after(self, item):
      if item["genres"] and "Horror" in item["genres"]:
            self.logger.warning(f"({item['title']}) is a horror movie!!!!")

def process_time(value):
   # a self-defined field processing function
   # process time to minutes
   # '3h 1min' -> 181
   if value:
      res = 0
      segs = value.split(" ")
      for seg in segs:
            if seg.endswith("min"):
               res += int(seg.replace("min", ""))
            elif seg.endswith("h"):
               res += 60 * int(seg.replace("h", ""))
      return res
   return value

if __name__ == "__main__":


# Scrape quotes from
from acrawler import Parser, Crawler, ParselItem, Request

logger = get_logger("quotes")

class QuoteItem(ParselItem):
   log = True
   default = {"type": "quote"}
   css = {"author": ""}
   xpath = {"text": ['.//span[@class="text"]/text()', lambda s: s.strip("“")[:20]]}

class AuthorItem(ParselItem):
   log = True
   default = {"type": "author"}
   css = {"name": "", "born": ""}

class QuoteCrawler(Crawler):

   main_page = r"\d+"
   author_page = r"*"
   parsers = [
            follow_patterns=[main_page, author_page],
      Parser(in_pattern=author_page, item_type=AuthorItem),

   async def start_requests(self):
      yield Request(url="")

if __name__ == "__main__":

See examples.


  • Replace parsel with parselx

  • clean redundant handlers

  • Cralwer’s name for distinguishing

  • Use dynaconf as configuration manager

  • Add delta_key support for request

  • Monitor all crawlers in web

  • Write detailed Documentation

  • Testing

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acrawler-0.1.6.tar.gz (34.9 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page