A FUSE wrapper around MongoDB gridfs using python and llfuse.
Project description
# python gridfs fuse
A FUSE wrapper around MongoDB gridfs using python and llfuse.
## Usage
```bash
gridfs_fuse --mongodb-uri="mongodb://127.0.0.1:27017" --database="gridfs_fuse" --mount-point="/mnt/gridfs_fuse" # --options=allow_other
```
### fstab example
```
mongodb://127.0.0.1:27017/gridfs_fuse.fs /mnt/gridfs_fuse gridfs defaults,allow_other 0 0
```
Note this assumes that you have the `mount.gridfs` program (or `mount_gridfs` on MacOS X) symlinked
into `/sbin/` e.g.
```bash
sudo ln -s $(which mount.gridfs) /sbin/`
```
## Requirements
* pymongo
* llfuse
## Install
Ubuntu 16.04:
```bash
sudo apt-get install libfuse python-llfuse
sudo -H pip install gridfs-fuse
```
MacOSX:
```bash
brew install osxfuse
sudo -H pip install gridfs-fuse
```
## Operations supported
* create/list/delete directories => folder support.
* read files.
* delete files.
* open and write once (like HDFS).
* rename
## Operations not supported
* modify an existing file.
* resize an existing file.
* hardlink
* symlink
* statfs
## Performance
### Setup
* AWS d2.xlarge machine.
* 4 @ 2.40Ghz (E5-2676)
* 30 gigabyte RAM
* filesystem: ext4
* block device: three instance storage disks combined with lvm.
```
lvcreate -L 3T -n mongo -i 3 -I 4096 ax /dev/xvdb /dev/xvdc /dev/xvdd
```
* mongodb 3.0.1
* mongodb storage engine WiredTiger
* mongodb compression: snappy
* mongodb cache size: 10 gigabyte
### Results
* sequential write performance: ~46 MB/s
* sequential read performance: ~90 MB/s
Write performance was tested by copying 124 files, each having a size of 9 gigabytes and different content.
Compression factor was about factor three.
Files were copied one by one => no parallel execution.
Read performance was tested by randomly picking 10 files out of the 124.
Files were read one by one => no parallel execution.
```bash
# Simple illustration of the commands used (not the full script).
# Write
pv -pr /tmp/big_file${file_number} /mnt/gridfs_fuse/
# Read
pv -pr /mnt/gridfs_fuse${file_number} > /dev/null
```
A FUSE wrapper around MongoDB gridfs using python and llfuse.
## Usage
```bash
gridfs_fuse --mongodb-uri="mongodb://127.0.0.1:27017" --database="gridfs_fuse" --mount-point="/mnt/gridfs_fuse" # --options=allow_other
```
### fstab example
```
mongodb://127.0.0.1:27017/gridfs_fuse.fs /mnt/gridfs_fuse gridfs defaults,allow_other 0 0
```
Note this assumes that you have the `mount.gridfs` program (or `mount_gridfs` on MacOS X) symlinked
into `/sbin/` e.g.
```bash
sudo ln -s $(which mount.gridfs) /sbin/`
```
## Requirements
* pymongo
* llfuse
## Install
Ubuntu 16.04:
```bash
sudo apt-get install libfuse python-llfuse
sudo -H pip install gridfs-fuse
```
MacOSX:
```bash
brew install osxfuse
sudo -H pip install gridfs-fuse
```
## Operations supported
* create/list/delete directories => folder support.
* read files.
* delete files.
* open and write once (like HDFS).
* rename
## Operations not supported
* modify an existing file.
* resize an existing file.
* hardlink
* symlink
* statfs
## Performance
### Setup
* AWS d2.xlarge machine.
* 4 @ 2.40Ghz (E5-2676)
* 30 gigabyte RAM
* filesystem: ext4
* block device: three instance storage disks combined with lvm.
```
lvcreate -L 3T -n mongo -i 3 -I 4096 ax /dev/xvdb /dev/xvdc /dev/xvdd
```
* mongodb 3.0.1
* mongodb storage engine WiredTiger
* mongodb compression: snappy
* mongodb cache size: 10 gigabyte
### Results
* sequential write performance: ~46 MB/s
* sequential read performance: ~90 MB/s
Write performance was tested by copying 124 files, each having a size of 9 gigabytes and different content.
Compression factor was about factor three.
Files were copied one by one => no parallel execution.
Read performance was tested by randomly picking 10 files out of the 124.
Files were read one by one => no parallel execution.
```bash
# Simple illustration of the commands used (not the full script).
# Write
pv -pr /tmp/big_file${file_number} /mnt/gridfs_fuse/
# Read
pv -pr /mnt/gridfs_fuse${file_number} > /dev/null
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
gridfs_fuse-0.1.2.dev1.tar.gz
(10.4 kB
view details)
Built Distributions
File details
Details for the file gridfs_fuse-0.1.2.dev1.tar.gz
.
File metadata
- Download URL: gridfs_fuse-0.1.2.dev1.tar.gz
- Upload date:
- Size: 10.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b7b75c54c100f10ef6afc9ab0d950c5df9a17cb1c8a352ea7e54c3e45be690e |
|
MD5 | 4a4f4fe788cc264b00bb443fdfa878d4 |
|
BLAKE2b-256 | 15404bca1eb18a2e21a16859ade87737c8191b8686dc9491bd267a5d4e2153df |
File details
Details for the file gridfs_fuse-0.1.2.dev1-py3-none-any.whl
.
File metadata
- Download URL: gridfs_fuse-0.1.2.dev1-py3-none-any.whl
- Upload date:
- Size: 12.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2759848858121cfc5120c2bc82f2232894c57dd0dd5d3e8061f7df9ea7b8dff3 |
|
MD5 | dbbe975400974aaa4d318c84f02c3ab2 |
|
BLAKE2b-256 | c4a9c57fae09b6f977d6a67d6e6a178db2bdfa4b2e40fbc611f8ba88afce235d |
File details
Details for the file gridfs_fuse-0.1.2.dev1-py2-none-any.whl
.
File metadata
- Download URL: gridfs_fuse-0.1.2.dev1-py2-none-any.whl
- Upload date:
- Size: 12.6 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5fbf92c3383c7458e03bdbf3b6091d7eb53a170d6d1818dd315778d75705c67 |
|
MD5 | f64c725266395f6a974956ae98b52b72 |
|
BLAKE2b-256 | 7f3e8beb97e02e7bad6276230a29a59016dae9ada4585ba12d579940cabbc320 |