OSM Fastly Logs
This is ofastlylog, the software that processes OpenStreetMap logs from Fastly.
Install
Packaged versions of ofastlylog can be installed like any other python software. To do so in a virtual environment do
python3 -m venv venv
venv/bin/pip install ofastlylog
venv/bin/ofastlylog --help
Contributing
This code is difficult to contribute to unless you have access to the OSMF AWS accounts. Recommendations to improve this are welcome.
Development is done using uv.
git clone https://github.com/pnorman/ofastlylog.git
cd ofastlylog
uv sync
uv run pre-commit install
uv run ofastlylog --help
Development could also be done using a venv and pip but this is not officially supported
python3 -m venv venv
venv/bin/pip install -e .
venv/bin/ofastlylog --help
Usage
AWS credentials need to be set up for ofastlylog to access the service logs. In a typical setup this is done with
export AWS_PROFILE=osm-service-logs
ofastlylog has two commands right now: one to create tables, and another to perform hourly tasks.
All commands take a list of service tables to create or process. Specify these with --<servicename> <tabletype>, e.g. --tile success for the tile.openstreetmap.org success table. See Table Types for a full list of combinations.
ofastlylog tools
Contains tools for creating and working with setup tables
ofastlylog tools create-table
Creates tables in athena.
ofastlylog tools check-hour
TODO: Check if a particular hour has data in the table
ofastlylog process
Peform log generation and processing
ofastlylog process hourly
Perform log processing intended to be run hourly
Table types
| Table | --tile |
--vector |
--nominatim |
|---|---|---|---|
base |
✓ | ✓ | ✓ |
success |
✓ | ✓ | ✓ |
minimized |
Not implemented | not impelemented | not impelemented |
Architecture
Several OSMF-hosted services use Fastly as a CDN. Fastly is set up to deposit compressed log files in a s3 bucket. The log files contain line-delimited JSON objects deposited every 10 minutes.
Because compressed text is slow to read, these files are processed with ofastlylog process hourly. This creates a number of tables. All tables are stored as Parquet data with hourly partitioning.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file ofastlylog-0.3.0.tar.gz.
File metadata
- Download URL: ofastlylog-0.3.0.tar.gz
- Upload date:
- Size: 57.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4e897587a20031e2a6033f1cc581ae804d6c98860d628c7ea78d1e38093cedfe
|
|
| MD5 |
a976bf85e32eb5fb079442a02706a292
|
|
| BLAKE2b-256 |
920a687528df3231d0e9277efac37e42090698e104112a403468aa1319f08127
|
Provenance
The following attestation bundles were made for ofastlylog-0.3.0.tar.gz:
Publisher:
package.yml on pnorman/ofastlylog
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
ofastlylog-0.3.0.tar.gz -
Subject digest:
4e897587a20031e2a6033f1cc581ae804d6c98860d628c7ea78d1e38093cedfe - Sigstore transparency entry: 2239726891
- Sigstore integration time:
-
Permalink:
pnorman/ofastlylog@16ef11a3aff3455498c6580696d0cd1f7b68ecaf -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/pnorman
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
package.yml@16ef11a3aff3455498c6580696d0cd1f7b68ecaf -
Trigger Event:
push
-
Statement type:
File details
Details for the file ofastlylog-0.3.0-py3-none-any.whl.
File metadata
- Download URL: ofastlylog-0.3.0-py3-none-any.whl
- Upload date:
- Size: 21.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d5376ad49adb6e7493a9f631259930fae6e82c4fd7bb1be8c09fd770fe04b853
|
|
| MD5 |
d01ab410b514de06a76b4156d4cea751
|
|
| BLAKE2b-256 |
af0de5b0be9ac30df4be9139adc88808020d90bcafd472c4d066a00be15bd1fc
|
Provenance
The following attestation bundles were made for ofastlylog-0.3.0-py3-none-any.whl:
Publisher:
package.yml on pnorman/ofastlylog
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
ofastlylog-0.3.0-py3-none-any.whl -
Subject digest:
d5376ad49adb6e7493a9f631259930fae6e82c4fd7bb1be8c09fd770fe04b853 - Sigstore transparency entry: 2239727132
- Sigstore integration time:
-
Permalink:
pnorman/ofastlylog@16ef11a3aff3455498c6580696d0cd1f7b68ecaf -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/pnorman
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
package.yml@16ef11a3aff3455498c6580696d0cd1f7b68ecaf -
Trigger Event:
push
-
Statement type:
