Skip to main content

FileGenie

FileGenie is a Python library that simplifies file parsing from AWS S3 across various formats (eg.- TEXT, CSV, EXCEL, ZIP, XML, PDF etc.) and enables users to define custom functions for data massaging and transformation, ensuring seamless processing and tailored output generation based on provided configurations.

Features

  • Multi-format Support: Effortlessly parse files in formats such as TEXT, CSV, EXCEL, ZIP, XML, and PDF directly from AWS S3.
  • Flexible Response Types: Generate responses tailored to user needs, including DATAFRAME, JSON, or FILE outputs.
  • Password-Protected Files: Seamlessly parse files secured with passwords.
  • Custom Edge Case Handling: Apply user-defined custom functions to address specific data massaging and transformation requirements, such as sanitizing data, converting values, reformatting date fields etc.
  • AWS S3 Integration: Fetch files directly from AWS S3 buckets using IAM roles for secure access.
  • Streamlined Configuration: Set up easily with minimal configuration, eliminating the need of writing parser for specific file type.

Installation

Install the SDK using pip:

pip install file_genie

Prerequisites

  • Your application should be deployed on AWS EKS to enable the SDK to utilize AWS S3 credentials.
  • Python: >= '3.6'
  • Pandas: '2.0.0'

Getting Started

Define Custom Edge Cases: Let's say you need to sanitize columns (e.g., standardise column values to a common format before applying custom logic) during file parsing, you can define custom functions for the SDK to use.

To implement this:

  • Create an edgeCases folder in your project.
  • Add a file named user_edge_cases.py.
  • Define your custom functions in this file.
  • Reference these functions in the edge_case section of the file_config.
  • The SDK will automatically import and apply these functions during file parsing or transformation.
from edgeCases import user_edge_cases
self.edge_cases = user_edge_cases

Define the configuration required for file parsing logic and S3 bucket names

    s3_config: {
        upload_bucket: s3_bucket_name
        download_bucket: s3_bucket_name
    }
    file_config: {
        "file_source_1": {
            "read_from_s3_func":"read_complete_excel_file",
            "parameters_for_read_s3": None,
            "file_dtype":{
                "Order_Number": str,
                "Added On":str,
                "Added By":str
            },
            "columns_mapping": {
                <!-- "Column Name in file": "Column name required in output" -->
                "Transaction Type": "TransactionType",
                "Cust Name": "CustomerName",
                "Cust ID": "CustomerId",
                "Transaction Amount": "Amount",
                "OrderNumber": "TransactionReference",
                "Reference ID": "CustomerReferenceId",
                "Target Date": "TargetDate",
                "TransactionDate": "TransactionDate",
                "FeeAmount": "ServiceCharge",
                "TaxAmount": "ServiceTax",
                "NetAmount": "NetAmount"
            }
            "edge_case": {
                <!-- edge case function name which you have defined in user_edge_case.py : params required for that function
                there can be different type of params. For eg. - dict, list, str -->
                <!-- In this convert_amount_as_per_currency is the edge case function which you want to apply while transforming the entries and "Amount" is the param to this function where you will apply the currency conversion -->
                "convert_amount_as_per_currency": "Amount"
            }
        },
    }

read_from_s3_func: This filed in FileGenie configuration specifies the function to be used for parsing a specific file type from AWS S3. Depending on the file format, you can choose from the following available functions:

  • readFromS3 - parse the TXT, EXCEL, CSV, XML, PDF files
  • readZipFromS3 - parse the zip files
  • read_complete_excel_file - Use this function when working with EXCEL files containing multiple sheets.

parameters_for_read_s3: This field in FileGenie configuration specifies the additional parameters required for reading the file such as password_protected, password, sep etc. you can choose from the following available params:

  • password_protected: If file is password protected or not
  • passowrd_secret_key: Secret key name for password.
  • skiprows: Rows to skip at the start.
  • sep: Delimiter for CSV parsing.
  • header: Row number(s) to use as column names.
  • has_header: Specify if the file has a header.
  • skip_header: Skip the header row during processing.
  • sheet_name: Target sheet in an Excel file.
  • parser_func: Custom parser function.
  • chunksize: Number of rows to read per chunk.
  • skip_footer: Rows to skip at the end.

Import and initialise the file genie

from file_genie import FileGenie

file_genie = FileGenie(config={s3_config: s3_config, file_config: file_config})
parsed_data = file_genie.parse("s3://your-bucket-name/path/to/your/file.csv", file_source, ParsedDataResponseType.DATAFRAME.value)
//By default SDK will provide response as DATAFRAME

Metadata

Release files for file-genie 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for file-genie 0.0.4
File Size Uploaded Provenance Actions
file_genie-0.0.4.tar.gz 15.3 kB Without provenance Details for file_genie-0.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for file-genie 0.0.4
File Interpreter ABI Platform Actions
file_genie-0.0.4-py3-none-any.whl Python 3 none any Details for file_genie-0.0.4-py3-none-any.whl

Total release size: 31.2 kB

Release files / file_genie-0.0.4.tar.gz

Download URL file_genie-0.0.4.tar.gz
Size 15.3 kB
Tags Source
SHA-256 checksum
How to use checksums
4f52b5c23041043bc0d3ee1bc4a51067f7f74f312a74f312c49664af9baf5a9b
BLAKE2b-256 checksum
How to use checksums
2bc242816fd4b42457a7ecda363389f59d05d5e780641af8f760897457f99d5b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.9.20

Release files / file_genie-0.0.4-py3-none-any.whl

Download URL file_genie-0.0.4-py3-none-any.whl
Size 15.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d4e839dfa286b7fec56ee1a677fb0f5f2b93db9877bc885e2153d18b38570e98
BLAKE2b-256 checksum
How to use checksums
c9a60babeec729d455ec84ae59a65a0fdc319d345611c9c62381781c6fa2a800
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.9.20

Security for file-genie 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page