Skip to main content

Mocy

Overview

Mocy is a simple web crawling framework that is flexible and easy to use.

  • Concurrent downloads

  • Decorators like @before_download, @after_download, @pipe

  • Rate limit and retry mechanism

  • Session keeping

  • More...

Installation

$ pip install mocy

A Quick Example

The following is a simple spider to extract upcoming Python events.

from mocy import Spider, Request, pipe


class SimpleSpider(Spider):
    entry = 'https://www.python.org/'

    def parse(self, res):
        for link in res.select('.event-widget li a'):
            yield Request(
                link['href'],
                state={'name': link.text},
                callback=self.parse_detail_page
            )

    def parse_detail_page(self, res):
        date = ' '.join(res.select('.single-event-date')[0].stripped_strings)
        yield res.state['name'], date

    @pipe
    def output(self, item):
        print(f'{item[0]} will be held on "{item[1]}"')


SimpleSpider().start()

The Result is:

[2021-06-08 00:59:22] INFO   : Spider is running...
[2021-06-08 00:59:23] INFO   : "GET https://www.python.org/" 200 0.27s
[2021-06-08 00:59:23] INFO   : "GET https://www.python.org/events/python-events/1094/" 200 0.61s
[2021-06-08 00:59:23] INFO   : "GET https://www.python.org/events/python-events/964/" 200 0.63s
[2021-06-08 00:59:23] INFO   : "GET https://www.python.org/events/python-events/1036/" 200 0.69s
[2021-06-08 00:59:23] INFO   : "GET https://www.python.org/events/python-events/1085/" 200 0.69s
[2021-06-08 00:59:24] INFO   : "GET https://www.python.org/events/python-events/833/" 200 0.79s
[2021-06-08 00:59:24] INFO   : Spider exited; total running time 1.12s.

PyFest will be held on "From 16 June through 18 June, 2021"
EuroPython 2021 will be held on "From 26 July through 01 Aug., 2021"
PyCon Namibia 2021 will be held on "From 18 June through 19 June, 2021"
PyOhio 2021 will be held on "31 July, 2021"
SciPy 2021 will be held on "From 12 July through 18 July, 2021"

There are some detailed examples in the directory /examples.

API

Request

class Request(url: str,
              method: str = 'GET',

              callback: Optional[Callable] = None,
              session: Union[bool, dict] = False,
              state: Optional[dict] = None,

              headers: Optional[dict] = None,
              cookies: Optional[dict] = None,
              params: Optional[dict] = None,
              data: Optional[dict] = None,
              json: Optional[dict] = None,
              files: Optional[dict] = None,
              proxies: Optional[dict] = None,
              verify: bool = True,
              timeout: Optional[Union[Tuple[Number, Number], Number]] = None,
              **kwargs)

The popular HTTP library requests is used under the hood. Please refer to its documentation: requests.request.

It accepts some extra parameters:

Parameters:

  • callback: It will be used to handle response to this request. The default value is self.parse.
  • session: It provides cookie persistence, connection-pooling, and configuration. It can be a Bool or dict. The default value is False, that means no new requests.Session will be created.
    • If set to True, a requests.Session object is created, and subsequent requests will be sent under the same session.
    • if set to a dict, in addition to the above, the value can be used to provide default data to the Request. For example: session={'auth': ('user', 'pass'), 'headers': {'x-test': 'true'}}.
  • state: It is shared between a request and the corresponding response.

Response

This object contains a server’s response to an HTTP request. Actually it is the same object as requests.Response.

Several attributes and methods are attached to this object:

Attributes:

  • req: The Request object.
  • state: The same object that was passed by a Request.

Methods:

  • select(self, selector: str, **kw) -> List[bs4.element.Tag]

    Perform a CSS selection operation on the HTML element. The powerful HTML parser Beautiful Soup is used.

Spider

Base class for spiders. All spiders must inherit from this class.

Class attributes:

  • WORKERS

    Default: os.cpu_count() * 2

    The number of concurrent requests that will be performed by the downloader.

  • TIMEOUT:

    Default: 30

    The amount of time (in secs) that the downloader will wait before timeout.

  • DOWNLOAD_DELAY

    Default: 0

    The amount of time (in secs) that the downloader should wait before download.

  • RANDOM_DOWNLOAD_DELAY

    Default: True

    If enabled, the downloader will wait a random time (0.5 * delay ~ 1.5 * delay by default) before downloading the next page.

  • RETRY_TIMES

    Default: 3

    Maximum number of times to retry when encountering connection issues or unexpected status codes.

  • RETRY_CODES:

    Default: (500, 502, 503, 504, 408, 429)

    HTTP response status codes to retry. Other errors (DNS or connection issues) are always retried.

    502: Bad Gateway, 503: Service Unavailable, 504: Gateway Timeout, 408: Request Timeout, 429: Too Many Requests.

  • RETRY_DELAY

    Default: 1

    The amount of time (in secs) that the downloader will wait before retrying a failed request.

  • DEFAULT_HEADERS

    Default: {'User-Agent': 'mocy/0.1'}

Attributes:

  • entry: Union[str, Request, Iterable[Union[str, Request]], Callable] = []

Methods:

  • entry() -> Union[str, Request, Iterable[Union[str, Request]], Callable] = []

  • on_start(self) -> None

    Called when the spider starts up.

  • on_finish(self) -> None

    Called when the spider exits.

  • on_error(self, reason: SpiderError) -> None

    Called when the spider encounters an error when downloading or parsing. It may be called multiple times.

  • parse(self, res: Response) -> Any

    Parse a response and generate some data or new requests.

  • collect(self, item: Any) -> Any

    Called when the spider outputs a result. Usually it will be called multiple times.

  • collect(self, item: Any, res: Response) -> Any

    Called when the spider outputs a result. Usually it will be called multiple times.

  • start(self) -> None

    Starts up the spider. It will keep running until all requests were processed.

Decorators

The decorators can be applied to multiple methods of the Spider class. They are called in the same order as they were defined.

  • before_download

    The decorated method is used to modify request objects. If it does not return the same or a new Request object, the passed request will be ignored.

  • after_download

    The decorated method is used to modify response objects. If it does not return the same or a new Response object, the passed response will be ignored. If it returns a Request object, then the object will be added to the request queue.

  • pipe

    The decorated method is used to process yielded items. If it returns None, the item won't be passed to the next pipeline. Spider.collect is a default pipe.

Exceptions

  • class SpiderError(msg: str, cause: Optional[Exception] = None)

    Base Class for spider errors. The following exceptions inherit from this class.

    Attributes:

    • msg: a brief text that explains what happened.
    • cause: the underlying exception that raised this error.
    • req: the Request object.
    • res: the Response object; it may be None.
  • class RequestIgnored(url: str, cause: Optional[Exception] = None)

    Indicates a decision was made not to process a request.

  • class ResponseIgnored(url: str, cause: Optional[Exception] = None)

    Indicates a decision was made not to process a response.

  • class DownLoadError(url: str, cause: Optional[Exception] = None)

    Indicates an error when downloading.

  • class ParseError(url: str, cause: Optional[Exception] = None)

    Indicates an error when parsing.

Tests

$ pytest

License

MIT

Metadata

Release files for mocy 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mocy 0.2.3
File Size Uploaded Provenance Actions
mocy-0.2.3.tar.gz 15.9 kB Without provenance Details for mocy-0.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mocy 0.2.3
File Interpreter ABI Platform Actions
mocy-0.2.3-py3-none-any.whl Python 3 none any Details for mocy-0.2.3-py3-none-any.whl

Total release size: 30.5 kB

Release files / mocy-0.2.3.tar.gz

Download URL mocy-0.2.3.tar.gz
Size 15.9 kB
Tags Source
SHA-256 checksum
How to use checksums
6df8d7e1c6a5df961227913eee178e3c781994969f9b7d44417fb18c70b638ab
BLAKE2b-256 checksum
How to use checksums
aea71f28b3cdee429ab0997cd0bff889454b9fb8794e147eda9829aa653b3e81
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.25.1 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.1

Release files / mocy-0.2.3-py3-none-any.whl

Download URL mocy-0.2.3-py3-none-any.whl
Size 14.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
be5d9b083414617beee514c6d590f59c400de61898e3b4254d0dd29fc01e7dba
BLAKE2b-256 checksum
How to use checksums
cdb66b00364533483b4ac5758e4bd3dabcde3d4b78a012c065def67ecbabd036
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.25.1 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.1

Security for mocy 0.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page