pyFile

pyFile is a Python command-line application for working with data files from dataloggers. It can:

• Split or merge data files
• Identify gaps in timestamped measurements
• Upload data from supported TOA5 files to the project's database

Requirements

• Python 3.12 or newer
• To upload file data to the database, you must run pyFile on the tower's Raspberry Pi

Installation

pyFile is available on PyPI. Install it with:

pip install pyfile-prod

Commands

Use --help with pyfile or any command to display usage information and available options:

pyfile --help
pyfile arrange --help
pyfile report --help
pyfile uploaddb --help

Arrange files

Arrange and split file data by day, month, or year, or merge the data into one file. All files must belong to the same dataset and have the same table columns. Supported file formats are TOA5, SWD, and CSV.

pyfile arrange --mode d <directory or file paths separated by spaces>
pyfile arrange --mode m <directory or file paths separated by spaces>
pyfile arrange --mode y <directory or file paths separated by spaces>
pyfile arrange --mode a <directory or file paths separated by spaces>

Report missing data

Find gaps between consecutive timestamps. For example:

pyfile report --expected-gap 30 --time-unit minutes <directory or file paths separated by spaces>

Supported time units are milliseconds, seconds, minutes, and hours. The report can be printed in the terminal, saved as CSV, or both.

Compare two previously saved reports to list the gaps found in only one file:

pyfile report compare <first report.csv> <second report.csv>

The comparison is printed in the terminal and can optionally be saved as CSV.

Upload TOA5 data to the database

Upload one or more supported, uncompressed TOA5 .dat files data to the database. Run this on the production Raspberry Pi:

pyfile uploaddb <file paths or directory>

Currently supported filenames are:

• FERLD01_dendroTable.dat through FERLD06_dendroTable.dat
• FERLD01_sapflowTable.dat through FERLD06_sapflowTable.dat
• FERLD07_dendroTable.dat
• FERLD07_soilTable.dat
• FERLD08_dendroTable.dat
• FERLD08_soilTable.dat
• fluxcsformat.dat
• fluxnotesabridged.dat
• fluxstorage.dat
• intavg.dat
• particules.dat

The idea is to take a valid TOA5 file with the data that you want to upload and give it the corresponding name from the ones above. When you upload a file to the database, duplicate records are ignored, and only new records are inserted.

Project layout

src/pyfile_prod/
├── commands/          # Typer command definitions
├── utils/
│   ├── database/      # PostgreSQL queries and connection pool
│   ├── files.py       # Input detection, parsing, and collections
│   ├── processors.py  # Database upload transformations
│   ├── reporter.py    # Missing-data detection
│   ├── divider.py     # Period-based output splitting
│   └── merger.py      # File merging
├── enums.py
└── main.py