Skip to main content

A python package that splits large files into smaller chunks.

Project description

Tutorial using the pysplit package

pysplit is a Python package used for splitting large files into smaller chunks.

Currently the default chunk size is 100MB. This size was chosen to work around GitHub's upload file size limit.

Install the latest version of pysplitter

Run the cell below to ensure you have the latest version of pysplitter installed on your machine.

# !pip install --upgrade pysplitter

Import required packages

import pysplitter as pysp
import numpy as np

Import helpful packages

import sys
import os

Create a numpy array that will exceed 100MB when saved to disk.

The numeric values of the data are not important. Random values were used for convenience only.

dim = 250
num = int(dim * dim * dim)
x = np.random.normal(size=num).reshape(dim, dim, dim)
(250, 250, 250)

Save numpy array to disk and list directory contents'x.npy', x)

Display size of file on disk

size = os.path.getsize('x.npy')
print(f'{size / 1e6} MB')
125.000128 MB

As many people may know, GitHub will not allow files exceeding 100 MB to be uploaded.

Use the commands below to split the original (and too large) file into multiple .split files.

Currently the default split size is <= 100 MB, but this may become a variable paramter in furture distributions.

src = 'x.npy'
2 file(s) written.

Check file size of the two chunks that were just written.

print(os.path.getsize('x0000.npy.split') / 1e6, 'MB')
100.0 MB
print(os.path.getsize('x0001.npy.split') / 1e6, 'MB')
25.000128 MB

As is clearly shown from the output of the above cells, both chunks are <= 100MB. This means that this data can now pushed to GitHub as any other file would.

Recombine the data chunks back into a single file

search_pattern = './x*.split'
dst = '.'
pysp.unsplit(search_pattern, dst, validate=True, orig_src=src)
File reconstructed without loss: True
x_unsplit = np.load('x(unsplit).npy')
(250, 250, 250)

Show that the manipulated data x_unsplit is the same as the original data x.

np.allclose(x, x_unsplit)


Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pysplitter-0.0.17.tar.gz (3.4 kB view hashes)

Uploaded Source

Built Distribution

pysplitter-0.0.17-py3-none-any.whl (4.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page