arcgis-scraper

A simple scraper to download data from ArcGIS FeatureServer and MapServer layers.

Installation

The Python module can be installed in the following ways:

Install from PyPI

pip install --upgrade arcgis-scraper

Install from wheel

If you have downloaded a .whl file from the Releases page, install it with:

pip install path/to/arcgis_scraper‑<version>‑py3‑none‑any.whl

Replace <version> with the actual version number in the filename. For example:

pip install path/to/arcgis_scraper-1.0.1-py3-none-any.whl

Install from source

git clone https://github.com/pgroenbaek/arcgis-scraper.git
pip install --upgrade ./arcgis-scraper

Usage

from arcgisscraper import ArcGISScraper

# Base URL for the ArcGIS REST services
base_url = "https://services1.arcgis.com/QcgJt0vVxSaqMKl7/arcgis/rest/services/"

# List of layers to scrape
query_urls = [
    "Adgangsveje_OD/FeatureServer/0/query",
    "Afsnitsmidter_OD/FeatureServer/0/query",
    "Basisspor_OD/FeatureServer/0/query",
]

# Initialize the scraper
scraper = ArcGISScraper(
    base_url=base_url,
    export_directory="./data",
    export_format="csv", # Options: "csv", "json", "parquet"
    max_requests_per_second=1.0,
    token=None, # Optional: token for secured services
    max_retries=3,
)

# Scrape multiple layers
scraper.scrape_layers(query_urls)

# Scrape a single layer with filtering and custom filename
scraper.scrape_layer(
    "Basisspor_OD/FeatureServer/0/query",
    filename="basisspor_filtered",
    where="SporType='Main'",
    out_fields="ID,Name,SporType"
)

Running Tests

You can run tests manually or use tox to test across multiple Python versions.

Run Tests Manually

First, install the required dependencies:

pip install pytest

Then, run tests with:

pytest

Run Tests with tox

tox allows you to test across multiple Python environments.

1. Install tox

pip install tox

2. Run Tests

tox

This will execute tests in all specified Python versions.

3. tox.ini Configuration

The tox.ini file should be in your project root:

[tox]
envlist = py36, py37, py38, py39, py310

[testenv]
deps = pytest
commands = pytest

Modify envlist to match the Python versions you want to support.

License

This Python module was created by Peter Grønbæk Andersen and is licensed under the MIT License.