Skip to main content

A python implementation of the flat-file streaming database

Project description


A python implementation of the FSDB flat-file streaming database.


Using pip:

pip3 install pyfsdb

Or manually:

git clone
cd pyfsdb
python3 build
python3 install

Example Usage

The FSDB file format contains headers and footers that supplement the data within a file. The most common separator is tab-separated, but can wrap CSVs and other datatypes (see the FSDB documentation for full details). The file also contains footers that trace all the piped commands that were used to create a file, thus documenting the history of its creation within the metadata in the file.

Example pyfsdb code for reading a file

Reading in row by row:

import pyfsdb
db = pyfsdb.Fsdb("myfile.fsdb")
for row in db:

Example FSDB file

#fsdb -F t col1 two andthree
1	key1	42.0
2	key2	123.0

Example writing to an FSDB formatted file.

import pyfsdb
db = pyfsdb.Fsdb(out_file="myfile.fsdb")
db.out_column_names=('one', 'two')
db.append([4, 'hello world'])

Read below for further usage details.


pip3 install pyfsdb

Additional Usage Details

The real power of the FSDB comes from the build up of tool-suites that all interchange FSDB formatted files. This allows chaining multiple commands together to achieve a goal. Though the original base set of tools are in perl, you don't need to know perl for most of them.

Let's create a ./mydemo command:

import sys, pyfsdb

db = pyfsdb.Fsdb(file_handle=sys.stdin, out_file_handle=sys.stdout)
value_column = db.get_column_number('value')

for row in db:     # reads a row from the input stream
    row[value_column] = float(row[value_column]) * 2
    db.append(row) # sends the row to the output stream


And then feed it this file:

#fsdb -F t col1 value
1	42.0
2	123.0

We can run it thus'ly:

# cat test.fsdb | ./mydemo
#fsdb -F t col1 value
1	84.0
2	246.0
#   | ./

Or chain it together with multiple FSDB commands:

# cat test.fsdb | ./mydemo | dbcolstats valueq
cat test.fsdb | PYTHONPATH=pyfsdb python3 ./ | dbcolstats value | dbcol mean stddev sum min max | dbfilealter -R C
#fsdb -R C mean stddev sum min max
mean: 165
stddev: 114.55
sum: 330
min: 84
max: 246

#   | ./
#   | dbcolstats value
#   | dbcol mean stddev sum min max
#   | dbfilealter -R C


Wes Hardaker @ USC/ISI

See also

The FSDB website and manual page for the original perl module:

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyfsdb, version 1.2.1
Filename, size File type Python version Upload date Hashes
Filename, size pyfsdb-1.2.1-py3-none-any.whl (75.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pyfsdb-1.2.1.tar.gz (27.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page