PySpark Vector Files
Read vector files into a Spark DataFrame with geometry encoded as Well Known Binary (WKB).
Full documentation is available here.
Requirements
This library was developed using Databricks Runtime 10.4 LTS and uses the versions of python, pandas and pyspark that come pre-installed on that runtime. However, it also requires GDAL 3.4.3 as this is the most recent version of GDAL available from ubuntugis-unstable as of 2022-08-11.
You can install GDAL on your cluster using an init script. See here for an example.
Install pyspark-vector-files
Within a Databricks notebook
%pip install pyspark-vector-files
From the command line
python -m pip install pyspark-vector-files
Quick start
Read the first layer from a file or files with given extension into a single Spark DataFrame:
from pyspark_vector_files import read_vector_files
sdf = read_vector_files(
path="/path/to/files/",
suffix=".ext",
)
More examples are available here.
Local development
To ensure compatibility with Databricks Runtime 10.4 LTS, this package was developed on a Linux machine running the Ubuntu 20.04 LTS operating system using Python3.8.10, GDAL 3.4.3, and spark 3.2.1..
Install Python 3.8.10 using pyenv
See the pyenv-installer's Installation / Update / Uninstallation instructions.
Install Python 3.8.10 globally:
pyenv install 3.8.10
Then install it locally in the repository you're using:
pyenv local 3.8.10
Install GDAL 3.4.3
Add the UbuntuGIS unstable Private Package Archive (PPA) and update your package list:
sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable \
&& sudo apt-get update
Install gdal 3.4.3, I found I also had to install python3-gdal (even though
I'm going to use poetry to install it in a virtual environment later) to
avoid version conflicts:
sudo apt-get install -y gdal-bin=3.4.3+dfsg-1~focal0 \
libgdal-dev=3.4.3+dfsg-1~focal0 \
python3-gdal=3.4.3+dfsg-1~focal0
Verify the installation:
ogrinfo --version
# GDAL 3.4.3, released 2022/04/22
Install poetry 1.1.13
See poetry's osx / linux / bashonwindows install instructions
Clone this repository
git clone https://github.com/Defra-Data-Science-Centre-of-Excellence/pyspark_vector_files.git
Install dependencies using poetry
poetry install
Metadata
Release files for pyspark-vector-files 0.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyspark_vector_files-0.2.5.tar.gz | 16.8 kB | Without provenance | Details for pyspark_vector_files-0.2.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pyspark_vector_files-0.2.5-py3-none-any.whl | Python 3 | none | any | Details for pyspark_vector_files-0.2.5-py3-none-any.whl |
Total release size: 34.1 kB
Release files / pyspark_vector_files-0.2.5.tar.gz
| Download URL | pyspark_vector_files-0.2.5.tar.gz |
|---|---|
| Size | 16.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
933e8384fa5afa519d6f314352c43e7a6bcf48a7b894249310dd42eb6439f751
|
|
BLAKE2b-256 checksum How to use checksums |
d90a3270c3e1cf4e74fdd0733fd7845bab2e57a0985d20143f91322feac1cd3d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.8.10 Linux/5.15.0-1027-aws
|
Release files / pyspark_vector_files-0.2.5-py3-none-any.whl
| Download URL | pyspark_vector_files-0.2.5-py3-none-any.whl |
|---|---|
| Size | 17.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b4a10d4e71e7c5eb08bd30934efe3c313b03bf441175569e6b807a27f87bdd93
|
|
BLAKE2b-256 checksum How to use checksums |
142be41c582e882866c8a55662bbea1e9af25d3b752e864babae4bc59848359e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.8.10 Linux/5.15.0-1027-aws
|
Security for pyspark-vector-files 0.2.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.