ncdu-reader 1.0.0

Parser for binary exports generated by NCDU

ncdu-reader

---

Table of Contents

• What is it?
• Usage
• Performance
• License

What is it?

ncdu-reader is a parser for binary exports generated by NCDU available since 2.6.

Usage

In order to parse a binary export generated by NCDU (i.e., ncdu -O export.bin), you can use the ncdu_reader.ncdu module.

from ncdu_reader import ncdu

export_path = "/tmp/export.bin"
root = ncdu.parse_ncdu(export_path)

For a simple file structure like this:

/tmp/ncdu-playground
└── foo
    ├── bar
    │   ├── qux
    │   └── quxx.bin
    └── baz.bin

You will get this kind of output:

Parsed output

Directory(reference=AbsoluteReference(block_index=0, offset=126),
          name='/tmp/ncdu-playground',
          type=<ItemType.Directory: 0>,
          prev=None,
          asize=4096,
          dsize=4096,
          uid=None,
          gid=None,
          mode=None,
          mtime=None,
          dev=2096,
          rderr=None,
          cumasize=16384,
          cumdsize=16384,
          shrasize=0,
          shrdsize=0,
          items=5,
          sub=RelativeReference(offset=-29),
          children=[Directory(reference=AbsoluteReference(block_index=0,
                                                          offset=97),
                              name='foo',
                              type=<ItemType.Directory: 0>,
                              prev=None,
                              asize=4096,
                              dsize=4096,
                              uid=None,
                              gid=None,
                              mode=None,
                              mtime=None,
                              dev=0,
                              rderr=None,
                              cumasize=12288,
                              cumdsize=12288,
                              shrasize=0,
                              shrdsize=0,
                              items=4,
                              sub=RelativeReference(offset=-20),
                              children=[File(reference=AbsoluteReference(block_index=0,
                                                                         offset=77),
                                             name='baz.bin',
                                             type=<ItemType.File: 1>,
                                             prev=RelativeReference(offset=-29),
                                             asize=0,
                                             dsize=0,
                                             uid=None,
                                             gid=None,
                                             mode=None,
                                             mtime=None),
                                        Directory(reference=AbsoluteReference(block_index=0,
                                                                              offset=48),
                                                  name='bar',
                                                  type=<ItemType.Directory: 0>,
                                                  prev=None,
                                                  asize=4096,
                                                  dsize=4096,
                                                  uid=None,
                                                  gid=None,
                                                  mode=None,
                                                  mtime=None,
                                                  dev=0,
                                                  rderr=None,
                                                  cumasize=8192,
                                                  cumdsize=8192,
                                                  shrasize=0,
                                                  shrdsize=0,
                                                  items=2,
                                                  sub=RelativeReference(offset=-21),
                                                  children=[File(reference=AbsoluteReference(block_index=0,
                                                                                             offset=27),
                                                                 name='quxx.bin',
                                                                 type=<ItemType.File: 1>,
                                                                 prev=RelativeReference(offset=-27),
                                                                 asize=0,
                                                                 dsize=0,
                                                                 uid=None,
                                                                 gid=None,
                                                                 mode=None,
                                                                 mtime=None),
                                                            Directory(reference=AbsoluteReference(block_index=0,
                                                                                                  offset=0),
                                                                      name='qux',
                                                                      type=<ItemType.Directory: 0>,
                                                                      prev=None,
                                                                      asize=4096,
                                                                      dsize=4096,
                                                                      uid=None,
                                                                      gid=None,
                                                                      mode=None,
                                                                      mtime=None,
                                                                      dev=0,
                                                                      rderr=None,
                                                                      cumasize=4096,
                                                                      cumdsize=4096,
                                                                      shrasize=0,
                                                                      shrdsize=0,
                                                                      items=0,
                                                                      sub=None,
                                                                      children=[])])])])

For more information about what the different fields mean, refer to the official documentation.

Partial reads

ncdu-reader does not support lazily reading exports (e.g., parsing the root at first and then parsing children when requested). This means that the entire tree needs to be parsed at once. This can be problematic for very large exports.

It is however possible to only parse down to a certain depth, e.g. ncdu.parse_ncdu(export_path, depth=4). If all you care about is getting information about the top-level item (i.e., the root of your export), pass a depth of 0.

Performance

Here is a very unscientific benchmark of parsing exports for varying tree sizes to give an idea of scale, not expected performance. This includes the startup time for the Python interpreter, hence why small exports take around the same time.

Tree Size	Time (mean ± σ)
5	93.0 ms ± 1.6 ms
12,605	95.1 ms ± 3.0 ms
1,064,009	7.708 s ± 0.114 s
7,905,281	87.761 s ± 0.710 s

Note that RAM availability greatly impacts parsing of larger exports.

License

ncdu-reader is distributed under the terms of the MIT license.