licelformat 1.4.0

Parser for Licel binary format — lidar measurement profiles (analog & photon counting)

licelformat

Parser for Licel binary format files — a common data format in atmospheric lidar remote sensing. Reads analog and photon‑counting measurement profiles, extracts metadata, and provides scaling/unscaling consistent with the reference Go implementation.

Features

• Parse Licel .dat/.licel files with full header + binary profile extraction
• Scale raw ADC counts → millivolts and raw photon counts → MHz
• Round‑trip: save → reload preserves data losslessly (unscale on write)
• Multi‑file loading via glob masks and ZIP archives
• Filter profiles by any predicate (LicelFile.filter) and files by any criteria (LicelPack.filter)
• Average channels across multiple files with LicelPack.average()
• Save to ZIP with configurable compression method and level
• Profile selection by wavelength and channel type
• NumPy arrays for all profile data — ready for further analysis

Installation

pip install licelformat

Quick start

Load a single Licel file

from licelformat import LoadLicelFile

lf = LoadLicelFile("path/to/file.DAT")

print(lf.MeasurementSite)       # "Vladivos"
print(lf.MeasurementStartTime)  # datetime(2020, 2, 10, 19, 22, 35)
print(len(lf.Profiles))         # 12

# First profile
p = lf.Profiles[0]
print(p.Wavelength)   # 355.0
print(p.Photon)       # False  (analog channel)
print(p.NShots)       # 2001
print(p.Data[:5])     # numpy float64 array, units: mV or MHz

Select profiles by wavelength

# Pick the 355 nm photon‑counting channel
profile = lf.select_certain_wavelength(is_photon=True, wavelength=355.0)

Batch processing with LicelPack

from licelformat import NewLicelPack

pack = NewLicelPack("/data/2020/*.DAT")
profiles = pack.select_certain_wavelength(is_photon=True, wavelength=532.0)

for p in profiles:
    print(p.Data.mean())

Glue (merge) analog and photon channels

# Merge analog and photon at 355 nm, polarization "o",
# using altitude range 1500-2500 m for calibration
lf.glue(wavelength=355.0, polarization="o", h1=1500, h2=2500)

# Same for all files in a pack — returns new pack with only glued files
glued_pack = pack.glue(wavelength=532.0, polarization="s", h1=1000, h2=2000)

Subtract background

# By mean of tail beyond 8000 m
lf.subtract_background(method="mean", bgrRange=8000.0)

# By median of tail beyond 8000 m
lf.subtract_background(method="median", bgrRange=8000.0)

# Using a dark signal file
lf.subtract_background(method="dark", dark_file=dark_licel_file)

# Same for all files in a pack
pack.subtract_background(method="mean", bgrRange=8000.0)

Average channels across files

from licelformat import NewLicelPack

pack = NewLicelPack("/data/2020/*.DAT")

# Create a new LicelFile with channel-wise averages across all files
avg_file = pack.average()

# Each profile in avg_file is the element-wise mean of the
# corresponding profiles from all files in the pack
print(avg_file.Profiles[0].Data[:5])  # averaged data

Filter profiles in a LicelFile

# Only photon‑counting channels
photon = lf.filter(lambda p: p.Photon)

# Only 532 nm channels
at_532 = lf.filter(lambda p: p.Wavelength == 532.0)

Filter files in a LicelPack

# Keep only files from a specific site
site_pack = pack.filter_files(lambda lf: lf.MeasurementSite == "Vladivos")

print(site_pack.StartTime, site_pack.StopTime)  # recomputed from filtered set

# Collect all photon‑counting profiles across all files
photon_profiles = pack.filter(lambda p: p.Photon)

# Collect all 532 nm profiles
at_532_profiles = pack.filter(lambda p: p.Wavelength == 532.0)

Load from a ZIP archive

from licelformat import NewLicelPackFromZip

pack = NewLicelPackFromZip("archive.zip")

Save to a ZIP archive

from licelformat import NewLicelPack

pack = NewLicelPack("/data/2020/*.DAT")

# Save with default compression (DEFLATED, level 6)
pack.save_to_zip("output.zip")

# Store without compression
import zipfile
pack.save_to_zip("output.zip", compression=zipfile.ZIP_STORED)

# Maximum compression
pack.save_to_zip("output.zip", compresslevel=9)

Data scaling

Raw int32 values stored on disk are automatically scaled during loading:

Channel type	Scale factor	Result unit
Analog	DiscrLevel × 1000 / (2^AdcBits × NShots)	mV
Photon	1 / (NShots × 0.05)	MHz

When saving, the inverse scaling is applied (round(scaled / scale)), so round‑trip is lossless.

API reference

licelformat.LicelProfile

Field	Type	Description
Active	bool	Channel active flag
Photon	bool	True for photon counting
LaserType	int	Laser index (1, 2, 3)
NDataPoints	int	Number of bins
Reserved	list	3 reserved values
HighVoltage	int	PMT high voltage (V)
BinWidth	float	Range bin width (m)
Wavelength	float	Laser wavelength (nm)
Polarization	str	"o", "s", or "p"
BinShift	int	Bin shift
DecBinShift	int	Decimal bin shift
AdcBits	int	ADC resolution (bits)
NShots	int	Number of laser shots
DiscrLevel	float	Discriminator level (mV)
DeviceID	str	2‑char device ID
NCrate	int	Crate slot number
Data	np.ndarray	Scaled float64 profile data

Property	Type	Description
isPhoton	bool	True if DeviceID == 'BC'
isAnalog	bool	True if DeviceID == 'BT'
isGlued	bool	True if DeviceID == 'BG'

Methods: metadata(), profile(), scale_factor(), to_dict(), truncate(rmax), subtract_background()

licelformat.LicelFile

Fields: MeasurementSite, MeasurementStartTime, MeasurementStopTime, AltitudeAboveSeaLevel, Longitude, Latitude, Zenith, Laser1NShots, Laser1Freq, Laser2NShots, Laser2Freq, NDatasets, Laser3NShots, Laser3Freq, FileLoaded, Profiles

Methods: select_certain_wavelength(), glue(), filter(), save(), to_bytes(), to_dict(), truncate(rmax), subtract_background()

licelformat.LicelPack

Fields: StartTime, StopTime, Data (dict of LicelFile)

Methods: select_certain_wavelength(), filter(), filter_files(), glue(), average(), save(), save_to_zip(), to_dict(), truncate(rmax), subtract_background()

Module‑level functions

Function	Description
LoadLicelFile(path)	Load a single Licel file from disk
LoadLicelFileFromReader(r)	Load from a binary stream
NewLicelPack(mask)	Load all files matching a glob pattern
NewLicelPackFromZip(path)	Load all valid files from a ZIP archive

License

LGPL v3 or later. See LICENSE for details.

Related projects

This is a Python port of the Go package github.com/physicist2018/licelfile. Both packages produce bit‑identical results on the same input files.