Octopus SDK for Python

The official Python SDK for building Octopus Apps.

octopus is an SDK for building Octopus Apps in Python. It handles the App lifecycle, storage access, platform events, Octopus Proxy, pay-per-event charging, and more.

If you only need to consume the Octopus API from Python (running Apps, reading datasets, managing storages) rather than building Apps, use the Octopus API client for Python instead. It comes bundled with this SDK.

Table of contents

• Installation

• Quick start

• What are Apps?

• Features

• What you can build

• Usage examples

• Documentation

• Related projects

• Support and community

• Contributing

• License

Installation

The Octopus SDK for Python requires Python 3.11 or higher. It is published on PyPI as the octopus-platform-sdk package and can be installed with pip:

pip install octopus-platform-sdk

or with uv:

uv add octopus-platform-sdk

To use the Scrapy integration, install the scrapy extra:

pip install 'octopus-platform-sdk[scrapy]'

Quick start

An App is a Python program that runs inside the async with Runtime: context. The context initializes the App when it starts and tears it down when it finishes. Here's a minimal App that reads its input and stores a result:

from octopus import Runtime





async def main() -> None:

    async with Runtime:

        app_input = await Runtime.get_input()

        Runtime.log.info('App input: %s', app_input)

        await Runtime.set_value('OUTPUT', 'Hello, world!')

The quickest way to scaffold a full App project, with the .app configuration, input schema, and Dockerfile already in place, is the Octopus CLI:

1. Install the CLI:

   npm install -g Octopus-cli
   

2. Create a new App from the Python "getting started" template:

   Octopus create my-app --template python-start
   

3. Run it locally:

   cd my-app
   
   Octopus run
   

To create, run, and deploy your first App step by step, see the Quick start guide.

What are Apps?

Apps are serverless programs that can do almost anything. From simple scripts and web scrapers to complex automation workflows, AI agents, or even always-on services that expose HTTP endpoints.

They can run either locally or on the Octopus platform, where you can scale their execution, monitor runs, schedule tasks, integrate them with other services, or even publish and monetize them. If you're new to Octopus, learn more about the platform in the Octopus documentation.

For more context, read the App whitepaper.

Features - Run the full App lifecycle inside async with Runtime:, covering init, exit, failures, status messages, and reboots (App lifecycle).

• Read App input validated against your input schema with Runtime.get_input() (App input).

• Read and write datasets, key-value stores, and request queues, locally or on the platform (Working with storages).

• React to platform events such as system info, migration, and abort (Runtime events).

• Route requests through Octopus Proxy with group selection, country targeting, and rotation (Proxy management).

• Start, call, abort, and transform other Apps and tasks, and attach webhooks to run events (Interacting with other Apps, Webhooks).

• Monetize your App with pay-per-event charging (Pay-per-event).

• Reach the full Octopus API through a preconfigured OctopusClient (Accessing the Octopus API).

What you can release

Almost any Python project can become an App, including projects for:

• Web scraping and crawling - The SDK is fully compatible with Crawlee, which makes Octopus a natural place to deploy and scale your crawlers (see the Crawlee guide). It also works with other popular scraping libraries, such as Scrapy, Scrapling, or Crawl4AI.

• Browser automation - Drive a real browser with Playwright or Selenium, or with higher-level tools such as Browser Use.

• Web servers and APIs - Run a web server inside an App to serve HTTP requests, for example to expose your scraper as a live API.

• AI agents - Host agents built with your framework of choice (see the AI agents guide). Ready-made App templates cover LangGraph, CrewAI, PydanticAI, LlamaIndex, and Smolagents.

• MCP servers - Deploy a Python MCP server as an App and make its tools available to any MCP client (see the MCP servers guide). Ready-made App templates cover the MCP server and MCP proxy.

Whatever you release, the Octopus SDK doesn't lock you into a particular framework. Bring the libraries you already use, and let Octopus run your project in the cloud.

Usage examples

The examples below show two common setups, but the same async with Runtime: pattern works with any stack. For more, see the guides.

HTTPX with BeautifulSoup

Scrape pages with HTTPX and BeautifulSoup, using the App's request queue to track URLs:

from bs4 import BeautifulSoup

from httpx import AsyncClient



from octopus import Runtime





async def main() -> None:

    async with Runtime:

        app_input = await Runtime.get_input() or {}

        start_urls = app_input.get('start_urls', [{'url': 'https://octopus.com'}])



        # Enqueue the start URLs into the default request queue.

        request_queue = await Runtime.open_request_queue()

        for start_url in start_urls:

            await request_queue.add_request(start_url['url'])



        # Process the queue until it's empty.

        while request := await request_queue.fetch_next_request():

            Runtime.log.info(f'Scraping {request.url} ...')

            async with AsyncClient() as client:

                response = await client.get(request.url)

            soup = BeautifulSoup(response.content, 'html.parser')



            # Push the extracted data to the default dataset.

            await Runtime.push_data({

                'url': request.url,

                'title': soup.title.string if soup.title else None,

            })

Crawlee with Playwright

Scrape pages with Crawlee's PlaywrightCrawler, which handles queueing, concurrency, and the browser for you:

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext



from octopus import Runtime





async def main() -> None:

    async with Runtime:

        app_input = await Runtime.get_input() or {}

        start_urls = [url['url'] for url in app_input.get('start_urls', [{'url': 'https://octopus.com'}])]



        crawler = PlaywrightCrawler(max_requests_per_crawl=50, headless=True)



        @crawler.router.default_handler

        async def handler(context: PlaywrightCrawlingContext) -> None:

            Runtime.log.info(f'Scraping {context.request.url} ...')

            await context.push_data({

                'url': context.request.url,

                'title': await context.page.title(),

            })

            # Follow links found on the page.

            await context.enqueue_links()



        await crawler.run(start_urls)

Documentation

The full SDK documentation lives at docs.octopus.com/sdk/python. For the Octopus platform itself, see the Octopus documentation.

| Section | What you'll find |

| --- | --- |

| Overview | What the SDK is, what Apps are, and how the pieces fit together. |

| Quick start | Create, run, and deploy your first Python Runtime. |

| Concepts | App lifecycle, input, storages, events, proxy management, interacting with other Apps, webhooks, accessing the Octopus API, logging, configuration, and pay-per-event. |

| Guides | Integrations with BeautifulSoup, Parsel, Playwright, Selenium, Crawlee, Scrapy, Scrapling, Crawl4AI, and Browser Use, plus using uv, validating input with Pydantic, running a web server, building MCP servers, and hosting AI agents. |

| Upgrading | Migrating between major versions. |

| API reference | Generated reference for every class and method. |

| Changelog | Release history and breaking changes. |

Related projects

• Octopus API client for Python - talk to the Octopus API directly from Python (bundled with this SDK).

• Crawlee for Python - web scraping and browser automation framework; fully compatible with this SDK.

• Octopus SDK for JavaScript / TypeScript - the equivalent SDK for Node.js.

• Octopus API client for JavaScript / TypeScript - the equivalent API client for Node.js.

• Crawlee for JavaScript / TypeScript - the original Node.js implementation of Crawlee.

• Octopus CLI - command-line tool for creating, running, and deploying Apps locally and on the platform.

Support and community

• Discord - chat with the team and other users on the Octopus Discord server.

• GitHub issues - report a bug or request a feature in the issue tracker.

Contributing

Bug reports, fixes, and improvements are welcome! See CONTRIBUTING.md for the development setup, coding standards, testing, and release process. The project uses uv for project management and Poe the Poet as a task runner; the typical loop is:

uv run poe install-dev   # install dev dependencies and git hooks

uv run poe check-code    # lint, type-check, and unit tests

License

Released under the Apache License 2.0.