Skip to main content

A python implementation of the flat-file streaming database

Project description

Objective

A python implementation of the FSDB flat-file streaming database.

Installation

Using pip:

pip3 install pyfsdb

Or manually:

git clone git@github.com:gawseed/pyfsdb.git
cd pyfsdb
python3 setup.py build
python3 setup.py install

Example Usage

The FSDB file format contains headers and footers that supplement the data within a file. The most common separator is tab-separated, but can wrap CSVs and other datatypes (see the FSDB documentation for full details). The file also contains footers that trace all the piped commands that were used to create a file, thus documenting the history of its creation within the metadata in the file.

Example pyfsdb code for reading a file

Reading in row by row:

import pyfsdb
db = pyfsdb.Fsdb("myfile.fsdb")
print(db.column_names)
for row in db:
    print(row)

Example FSDB file

#fsdb -F t col1 two andthree
1	key1	42.0
2	key2	123.0

Example writing to an FSDB formatted file.

import pyfsdb
db = pyfsdb.Fsdb(out_file="myfile.fsdb")
db.out_column_names=('one', 'two')
db.append([4, 'hello world'])
db.close()

Read below for further usage details.

Installation

pip3 install pyfsdb

Additional Usage Details

The real power of the FSDB comes from the build up of tool-suites that all interchange FSDB formatted files. This allows chaining multiple commands together to achieve a goal. Though the original base set of tools are in perl, you don't need to know perl for most of them.

Let's create a ./mydemo command:

import sys, pyfsdb

db = pyfsdb.Fsdb(file_handle=sys.stdin, out_file_handle=sys.stdout)
value_column = db.get_column_number('value')

for row in db:     # reads a row from the input stream
    row[value_column] = float(row[value_column]) * 2
    db.append(row) # sends the row to the output stream

db.close()

And then feed it this file:

#fsdb -F t col1 value
1	42.0
2	123.0

We can run it thus'ly:

# cat test.fsdb | ./mydemo
#fsdb -F t col1 value
1	84.0
2	246.0
#   | ./test.py

Or chain it together with multiple FSDB commands:

# cat test.fsdb | ./mydemo | dbcolstats valueq
cat test.fsdb | PYTHONPATH=pyfsdb python3 ./test.py | dbcolstats value | dbcol mean stddev sum min max | dbfilealter -R C
#fsdb -R C mean stddev sum min max
mean: 165
stddev: 114.55
sum: 330
min: 84
max: 246

#   | ./test.py
#   | dbcolstats value
#   | dbcol mean stddev sum min max
#   | dbfilealter -R C

See also

The FSDB website and manual page for the original perl module:

https://www.isi.edu/~johnh/SOFTWARE/FSDB/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyfsdb, version 0.9.2
Filename, size File type Python version Upload date Hashes
Filename, size pyfsdb-0.9.2-py3-none-any.whl (18.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size pyfsdb-0.9.2.tar.gz (14.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page