Skip to main content

insitupy

https://img.shields.io/pypi/v/insitupy.svg

Manage reading and analyzing raw files of insitu data. The goal is to get raw insitu data files into manageable classes with helpful functions and provide access to the data as GeoPandas Arrays.

The first application of this will be for SnowEx pit data.

THIS IS A WORK IN PROGRESS (use at your own risk as outlined in the MIT license). We fully welcome any contribution and ideas. Snow science works better together!

  • Free software: MIT license

Features

  • Parsing of raw insitu data files with a variable number of header lines

  • Reading csv data files into a pandas DataFrame, and parsing metadata into usable format

  • Flexible user-defined variables

  • Reading both pit and point data

  • Parsing of date and location data

Definitions

Types of variables

Insitupy uses two types of variable definitions to parse CSV information:

1. primary variables - These are the data that expected to be found in the data columns. Think of them as the column headers that describe the data. In the example file <example_file_> below it would be the last header row indicated by the ‘#’. 2. metadata variables - These are the data that are expected to be found in the header lines. These are assumed to start with a ‘#’ sign by default. All data above the last row with the ‘#’ is assumed to be additional information that describes the data.

Variable definitions

Both variables types are defined the same way, in a separate yaml file. A standard single variable definition looks like this:

TOTAL_DEPTH:  # <- YAML root key
  code: total_depth
  description: Total depth of measurement
  map_from:
  - total_snow_depth
  - hs
  match_on_code: true
  auto_remap: true
  • code: The string that will be used to reference this variable within the insitupy code

  • description: A description of the variable

  • map_from: A list of strings that will be used to match the entry as primary or metadata variables

  • match_on_code: If true, the variable will be matched if the code values is found

    in the data, not just the map_from values

  • auto_remap: If true, the variable will be remapped to the code value

Overriding variables

One can override a variable by defining the same code as in the default supplied files within insitupy. The user definition always has precedence over the internal ones.

Example

I want to read in a file that looks like this:

# Location,East River
# Date/Local Standard Time,2020-04-27T08:45
# Latitude,38.92524
# Longitude,-106.97112
# Flags,random flag
# Top (cm),Bottom (cm),Density (kg/m3)
95.0,85.0,401.0
85.0,75.0,449.0
75.0,65.0,472.0

Parsing data

The metadata, variable and data parsers are configured with a lot of defaults for working with SnowEx data within insitupy, so the simple approach is

from insitupy.campaigns.snowex import SnowExProfileData
my_data = SnowExProfileDataCollection.from_csv(
    "./some_data.csv",
    # Don't fail when there are unknown variables in the header
    allow_map_failure=True
)
# Inspect the data
print(my_data.profiles[0].df)
# Look at the parsed metadata
print(my_data.profiles[0].metadata)

Defining your own variables

If you want to try your hand at defining variables yourself, you can do as follows.

A user custom metadata YAML file:

LATITUDE:
  auto_remap: true
  code: latitude
  description: Latitude
  map_from:
  - lat
  - latitude
  match_on_code: true
LONGITUDE:
  auto_remap: true
  code: longitude
  description: Longitude
  map_from:
  - long
  - lon
  - longitude
DATETIME:
  auto_remap: true
  code: datetime
  description: Combined date and time
  map_from:
  - Date/Local Standard Time
  - date/local_standard_time
  - datetime
  - "date&time"
  - date/time
  - date/local_time
  match_on_code: true
SITE_NAME:
  auto_remap: true
  code: site_name
  description: Name of campaign site
  map_from:
      - location
  match_on_code: true

and a separate primary variable YAML file:

BOTTOM_DEPTH:
  auto_remap: true
  code: bottom_depth
  description: Lower edge of measurement
  map_from:
  - bottom
  - bottom_depth
  match_on_code: true
DENSITY:
  auto_remap: true
  code: density
  description: measured snow density
  map_from:
  - density
  - density_mean
  match_on_code: true
DEPTH:
  auto_remap: true
  code: depth
  description: top or center depth of measurement
  map_from:
  - depth
  - top
  match_on_code: true
LAYER_THICKNESS:
  auto_remap: true
  code: layer_thickness
  description: thickness of layer
  map_from: null
  match_on_code: true

Save the two files to your local hard drive. They will be used as arguments in Python code with the next step.

Then use the new definitions and read in the file:

from insitupy.campaigns.snowex import SnowExProfileData
my_data = SnowExProfileDataCollection.from_csv(
    "./some_data.csv",
    # Don't fail when there are unknown variables in the header
    allow_map_failure=True,
    # Use the files YOU defined here
    primary_variable_files="/path/to/saved/primaryvariables.yaml",
    metadata_variable_files="/path/to/saved/metadatavariables.yaml",
)
print(my_data.profiles[0].df)
print(my_data.profiles[0].metadata)

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.0 (2024-03-27)

  • First release on PyPI.

Metadata

Release files for insitupy 0.4.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for insitupy 0.4.5
File Size Uploaded Provenance Actions
insitupy-0.4.5.tar.gz 42.3 kB Without provenance Details for insitupy-0.4.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for insitupy 0.4.5
File Interpreter ABI Platform Actions
insitupy-0.4.5-py2.py3-none-any.whl Python 3, Python 2 none any Details for insitupy-0.4.5-py2.py3-none-any.whl

Total release size: 74.2 kB

Release files / insitupy-0.4.5.tar.gz

Download URL insitupy-0.4.5.tar.gz
Size 42.3 kB
Tags Source
SHA-256 checksum
How to use checksums
709a791e245f3db96fc1a9016490e602506713d70165cb597faa7850faa87b8e
BLAKE2b-256 checksum
How to use checksums
79bd9fb226b593f2c5cb584ae6eaa23f3e4e980fb09ec223c7cfce3ec7525b2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Release files / insitupy-0.4.5-py2.py3-none-any.whl

Download URL insitupy-0.4.5-py2.py3-none-any.whl
Size 31.9 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
5029e97a0884049149f8e443d484e991a93494810ccbabd10c7b266c3b31f594
BLAKE2b-256 checksum
How to use checksums
0ba369a3da85f1232a8f70cb98ab7c4b5bee30c4b1d9739f9de690e3c4e307bd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Security for insitupy 0.4.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page