Skip to main content

A python package that splits large files into smaller chunks.

Project description

Tutorial using the pysplit package

pysplit is a Python package used for splitting large files into smaller chunks.

Currently the default chunk size is 100MB. This size was chosen to work around GitHub's upload file size limit.

Install the latest version of pysplitter

Run the cell below to ensure you have the latest version of pysplitter installed on your machine.

# !pip install --upgrade pysplitter

Import required packages

import pysplitter as pysp
import numpy as np

Import helpful packages

import sys
import os

Create a numpy array that will exceed 100MB when saved to disk.

The numeric values of the data are not important. Random values were used for convenience only.

dim = 250
num = int(dim * dim * dim)
x = np.random.normal(size=num).reshape(dim, dim, dim)
x.shape
(250, 250, 250)

Save numpy array to disk and list directory contents

np.save('x.npy', x)
os.listdir()
['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x.npy']

Display size of file on disk

size = os.path.getsize('x.npy')
print(f'{size / 1e6} MB')
125.000128 MB

As many people may know, GitHub will not allow files exceeding 100 MB to be uploaded.

Use the commands below to split the original (and too large) file into multiple .split files.

Currently the default split size is <= 100 MB, but this may become a variable paramter in furture distributions.

os.listdir()
['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy']
src = 'x.npy'
pysp.split(src)
2 file(s) written.

Check file size of the two chunks that were just written.

os.listdir()
['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy',
 'x0000.npy.split',
 'x0001.npy.split']
print(os.path.getsize('x0000.npy.split') / 1e6, 'MB')
100.0 MB
print(os.path.getsize('x0001.npy.split') / 1e6, 'MB')
25.000128 MB

As is clearly shown from the output of the above cells, both chunks are <= 100MB. This means that this data can now pushed to GitHub as any other file would.

Recombine the data chunks back into a single file

search_pattern = './x*.split'
dst = '.'
pysp.unsplit(search_pattern, dst, validate=True, orig_src=src)
File reconstructed without loss: True
os.listdir()
['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy',
 'x0000.npy.split',
 'x0001.npy.split']
x_unsplit = np.load('x(unsplit).npy')
x_unsplit.shape
(250, 250, 250)

Show that the manipulated data x_unsplit is the same as the original data x.

np.allclose(x, x_unsplit)
True


          

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pysplitter, version 0.0.17
Filename, size File type Python version Upload date Hashes
Filename, size pysplitter-0.0.17-py3-none-any.whl (4.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pysplitter-0.0.17.tar.gz (3.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page