Database for NECST.
Project description
NECSTDB
Database for NECST.
TL;DR
NECST, an abbreviation of NEw Control System for Telescope, is a flexible control system for radio telescopes. Its efficient data storage is provided here.
The database contains tables, which keep individual topic of time series data with some metadata attached to them, e.g. spectral data from one spectrometer board (array of data + timestamp), weather data (temperature + humidity + wind speed + ... + timestamp), etc.
Features
This package provides:
- database writer with quite flexible data format support
- database reader which supports various output format
Installation
pip install necstdb
Usage
Database Writer
-
Create new database instance
>>> db_dir = "path/to/new_database_directory" >>> db = necstdb.opendb(db_dir, mode="w") # "w" stands for "write"
-
Create table with data information
>>> data1_info = { ... "data": [ ... {"key": "recorded_time", "format": "d", "size": 8}, ... {"key": "n_scan_lines", "format": "i", "size": 4}, ... {"key": "obsmode", "format": "3s", "size": 3}, ... {"key": "timestamp", "format": "d", "size": 8}, ... ], ... "memo": "generated by db_logger_operation", ... "necstdb_version": necstdb.__version__, ... } >>> db.create_table("data1", data1_info)
The keys "key" and "format" in "data" list are required. The format characters and the sizes are listed below. For more information, see the struct module documentation and/or the ROS message wiki.
- Changed in v0.2.5: "size" in "data" list is no longer required, but is optional.
The "memo" and "necstdb_version" keys are not necessary. You can also add any other keys to the data information dict.
ROS format Format character Size [byte] bool ?
1 int8 b
1 int16 h
2 int32 i
4 int64 q
8 uint8 B
1 uint16 H
2 uint32 I
4 uint64 Q
8 float32 f
4 float64 d
8 string [length]s
1 * length NOTE: Array data are also supported. To write them, use repeat count syntax (e.g.
3d
) or format character sequence syntax (e.g.ddd
) for the "format", and the sum of the elements' sizes for the "size".{"key": "data_array", "format": "3d", "size": 3 * 8} # repeat count syntax {"key": "data_array", "format": "ddd", "size": 8 + 8 + 8} # format character sequence syntax
-
Write data into table
>>> table1 = db.open_table("data1", mode="ab") # "ab" stands for "append binary" >>> data1 = [1.6294488758e9, 3, b"SKY", 1.6294488757e9] # string data are not allowed, use bytes instead >>> table1.append(*data1)
Call the
append
method every time you get new data.NOTE: Data to pass to
append
method should be flattened. The nested structure will be reconstructed on reading.data = [1, 2, [3, 4, 5], 6, 7] data = necstdb.utils.flatten_data(data) table.append(*data)
NOTE: Multi-dimensional data are not supported.
Database Reader
-
Open the database instance
>>> db = necstdb.opendb("path/to/database_directory")
-
Read a desired topic
>>> data1 = db.open_table("data1").read(astype="array") >>> data1 array([[1.6294488758e9, 3, b'SKY', 1.6294488757e9], ...], dtype=[('recorded_time', '<f8'), ('n_scan_lines', '<i4'), ('obsmode', '|S3'), ('timestamp', '<f8')])
The supported
astype
keywords forread
method are:Output type Keywords Notes tuple "tuple" *1 dict "dict" pandas.DataFrame "pandas", "dataframe", "data_frame", "df" *2 numpy.ndarray "array", "structuredarray", "structured_array", "sa" bytes "buffer", "raw" *1: Array data are not supported, but will be flattened.
*2: Changed in v0.2.5: Keyworddf
is now supported.
Misc
List all the tables contained in the database
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.list_tables()
['data1', 'spectral_data', 'weather_data', ...]
Archive the database
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.checkout(saveto="path/to/archive.tar.gz", compression="gz")
Get informations of all tables in the database
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.get_info()
"""
file size [byte] #records record size [byte] format
table name
data1 2448 102 24 di3sd
spectral_data 41948160 320 131088 d32768fd
weather_data 6328 113 56 ddddddd
"""
Read particular columns and/or rows of the database
>>> db = necstdb.opendb("path/to/database_directory")
>>> data = db.open_table("data1").read(num=5, start=3, cols=["timestamp", "obsmode"], astype="tuple") # order of cols won't be preserved
((b'SKY', 1.6294488788e9) # 4th element (caution 0-based indexing)
(b'SKY', 1.6294488798e9)
(b'SKY', 1.6294488808e9)
(b'SKY', 1.6294488818e9)
(b'HOT', 1.6294488828e9))
Flatten nested array
>>> data = [1, 2, 3, [4, 5], 6]
>>> flattened = necstdb.utils.flatten_data(data)
[1, 2, 3, 4, 5, 6]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.