No project description provided
Project description
Threaded File Loader
Multithreaded Python package for faster file loading in machine learning.
Installation
pip install ThreadedFileLoader
Usage:
Loading Image Files:
from ThreadedFileLoader.ThreadedFileLoader import *
instance = ThreadedImageLoader("path_to_/*.jpg")
instance.start_loading()
images = instance.loaded_objects
print(len(images))
print(images[0].shape)
Loading Text Files:
from ThreadedFileLoader.ThreadedFileLoader import *
instance = ThreadedTextLoader("path_to_/*.txt")
instance.start_loading()
images = instance.loaded_objects
Loading Custom File Formats
Threaded FileLoader can load different file types.
This examples shows how the ThreadedTextLoader
class
overloads the ThreadedFileLoader
class to load text files.
from ThreadedFileLoader.ThreadedFileLoader import *
class ThreadedTextLoader(ThreadedFileLoader):
def object_loader(self, path):
with open(path) as afile:
data = afile.readlines()
return data
instance = ThreadedTextLoader("path_to_/*.txt")
instance.start_loading()
texts = instance.loaded_objects
Machine Learning Example
Loading Dataset
import numpy as np
from ThreadedFileLoader.ThreadedFileLoader import *
from sklearn.cluster import KMeans
instance = ThreadedImageLoader("path_to_dataset/*.jpg")
instance.start_loading()
images = instance.loaded_objects
images = np.array(images)
images = images.reshape(len(images), -1)
kmeans = KMeans(n_clusters=10, random_state=0).fit(images)
print(kmeans.labels_)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for ThreadedFileLoader-1.0.0.9.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cf14e08872e7b095f109092380fce3881fa940d8d09151429d4a5332c25e509 |
|
MD5 | c392d930a07c67138f6ac42d9f8d2aaa |
|
BLAKE2b-256 | 9a6d7bd4bfdde2d25ea38098643d841a4445405af5ea3f97573761580a518d0c |
Close
Hashes for ThreadedFileLoader-1.0.0.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e10b8cd239d1135456042a72edf14a9f849ca266df48ea7f1b7b82e711d9e1d |
|
MD5 | cddffefb73b60930f4f7aec5c25a29bd |
|
BLAKE2b-256 | 73dd076a8df9e804a5d9b23d125494c7b18740d69259092274dfa54018969263 |