Uaine.Pydat
A python package to assist in data and database handling.
-Daniel Stamer-Squair
Purpose
This contains a few functions that make data handling a little simpler, including a dedicated psv function, hash functions, data table cleaning methods and snippet queries for duckdb.
STABILITY STATUS
DEVELOPMENT:
Description
A Python package that streamlines data handling, processing, and database operations through a collection of utility functions and tools.
Uaine.Pydat provides a comprehensive toolkit for data scientists, analysts, and developers working with structured data. The package simplifies common data manipulation tasks while offering specialized functionality for file operations, data transformation, table cleaning, and database interactions.
Key Features include File I/O Operations for reading/writing various file formats, listing files by extension, and managing system paths; Data Transformation tools for reshaping, converting, and manipulating data structures with minimal code; Data Cleaning methods to sanitize, standardize, and prepare data tables for analysis; DuckDB Integration with helper functions and snippet queries; Cryptographic Hashing for data integrity and anonymization; System Information utilities to gather system metrics and resource usage; Data Generation for testing and development scenarios; and Configuration Handling for XML, INI, and other formats.
Core Modules include dataio.py for data input/output operations and format conversion; fileio.py for file system operations and path management; datatransform.py for data structure transformation and manipulation; dataclean.py for data cleaning and standardization; duckfunc.py for DuckDB database interactions; datahash.py for cryptographic hashing functions; systeminfo.py for system information gathering; datagen.py for random data generation; and bitgen.py for low-level bit generation utilities.
Common Use Cases for this package include simplifying ETL workflows, streamlining data preparation for analysis and machine learning, managing database operations with less boilerplate code, generating test data for development, monitoring system resources during data processing tasks, and securing sensitive data through hashing and anonymization.
This package aims to reduce the complexity of common data handling tasks by providing ready-made solutions that follow best practices while remaining flexible enough to adapt to various data processing requirements.
Getting Started
Follow the documentation here on github: https://uaineteine.github.io/uaine-pydat-docs/
Requirements
- Python
- requests: This library is used to send HTTP requests.
- pyreadstat: This library is used for most read and writing of dataframes.
- pandas: This library is used to store and manipulate memory held data.
- duckdb: This library is used to create duck db requests and queries from its metadata tables
- wheel: This is for making the package yourself
- twine: This is for making the package yourself
- psutil: This is for checking process availability of the machine
- lxml: This for parsing xml files and content
- polars: For lazy stream reading and writing of parquets
- azure-storage-blob: For listing and downloading from blob containers
- tqdm: For progress bars
Changelog
See the full version history in the changelog.
1 maintainer
Metadata
1 maintainer
Release files for uainepydat 1.6.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| uainepydat-1.6.4.tar.gz | 44.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| uainepydat-1.6.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 91.2 kB
Release files / uainepydat-1.6.4.tar.gz
| Download URL | uainepydat-1.6.4.tar.gz |
|---|---|
| Size | 44.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
963b7ed8822105508786a0c5e301ca90fe973f1bd796f93f8661402fc68b5eeb
|
|
BLAKE2b-256 checksum How to use checksums |
848dc4e46b8b19dab69e19aac1b64fe08b5dfa5b371fc863081e40dae528cce0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.8
|
Release files / uainepydat-1.6.4-py3-none-any.whl
| Download URL | uainepydat-1.6.4-py3-none-any.whl |
|---|---|
| Size | 46.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ccc3babf7a68e8db44b9c5afd9d0ef6ab82f10e99e28358c45feaa2073451c5d
|
|
BLAKE2b-256 checksum How to use checksums |
5ec3a725b6881055b4a7756efd48a52eee35a936103836ebcd6ea8c997613628
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.8
|
Security for uainepydat 1.6.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
