fused 1.1.2

No project description provided

Fused Public Python package

🌎 Geospatial, with Python, at scale.

Fused.io is a Python library to process and store geospatial data - at scale. Express workflows as a set of shareable UDFS (user defined functions) without thinking about the underlying compute. The Fused Python library is maintained by Fused.io.

Prerequisites

Python >= 3.8

Install

The Fused Python package is currently distributed via a private beta. Email [e-mail] for access.

Quickstart

import fused


# Load data
census = 's3://fused-asset/infra/census_bg_us/'
buildings = 's3://fused-asset/infra/building_msft_us/'

# Declare UDF
@fused.udf()
def census_buildings_join(left, right):
    import fused
    df_joined = fused.utils.geo_join(left.data, right.data)
    df_cnt = df_joined.groupby(['fused_index','GEOID']).size().to_frame('count_building').reset_index()
    return df_cnt

# Instantiate job configuration that runs the data against the UDF
job = census_buildings_join(census, buildings)

# Run locally
job.run_local()

# Run on remote compute managed by Fused and view logs
job_id = job.run_remote(output_table='s3://my-s3-bucket/census_buildings_join')
job_id.tail_logs()

# Export job to local directory
job.export('census_buildings_join', overwrite=True)

# Re-import job
fused.load_job('census_buildings_join')

Available operations

The following are some of the key functions:

• ingest: Upload a dataset into S3 with the Fused format.
• open_table: Open a Table object given a path to the root of the table
• run_local: Execute data processing tasks locally while you test and debug.
• run_remote: Submit jobs to run on a remote clusters - by changing a single line of code.
• export: Save a job and its configuration as a local directory, zip file, or gist.
• load_job: Open a previously saved job.
• load_udf: Open a previously saved UDF.
• show: Debugger tool.
• render: Render job or UDF to new Notebook cell and edit.

See the Fused documentation for the full list of available functions.

Docs

The documentation is a work in progress. It follows the Diátaxis system:

• Getting Started Tutorial: A hands-on introduction to Fused.
• How-to guides: Simple step-by-step user guides to accomplish specific tasks.
• Reference guide: Commands, modules, classes and methods.
• Explanation: Discussion of key decisions and design philosophy of the library.

Changelog

See the changelog for the latest changes.

Releases

The project manages releases with Semantic Versioning Specification (SemVer).