A small library to chunk large files into smaller arrays that can be used for generating RAG embeddings

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for harpreetset from gravatar.com harpreetset

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: Harpreet Sethi
  • Requires: Python >=3.12
  • Provides-Extra: test , formatting , type-checking , docs , publishing
Classifiers
Report project as malware

Project description

chunking4rag

For implementing a good RAG framework, we need 3 steps:

  1. Data Ingestion/ Parsing
  2. Chunking
  3. Embedding/ Vectorization

Data Ingestion is a important steps where we extract data from a document. Aa document can be available in various formats like text, pdf, excel, csv or even Images. The data_extraction folder contains classes to extract data from different types of documents.

Chunking is the process of breaking up a document into smaller chunks. The chunking methods are implemented in the chunkingmethods folder.

The strategies discussed in this repo are:

  1. Fixed length chunking

  2. Keyword chunking

  3. Adaptive chunking

  4. Sliding window

  5. Paragraph chunking

  6. Sentence chunking

Embedding is the process of converting text into vectors. The embedding methods are implemented in the embeddingmethods folder.

To use the library

The library is quite simple to use. Below example uses sentence chunking by extraction from text data

from data_extraction.document_parser import DocumentParser
from chunkingmethods.sentence_chunking import SentenceChunking
from chunkingdatamodel.chunking_model import ChunkingInput


text_content = "<html><body><p>Sample text content. This is a test. It has multiple sentences. It also has some punctuation.</p></body></html>"


text_document = DocumentParser(doc_type={"kind":"HTMLDocument"}).doc_type
content = text_document.get_content(text_content).pages[0].textual_content
data = ChunkingInput(text=content)

sentence_chunking = SentenceChunking(data)
chunks = sentence_chunking.chunk()
print(chunks)

# for pdf document data extraction use the following
pdf_document = DocumentParser(doc_type={"kind":"PDFDocument"}).doc_type
with open("test.pdf", "rb") as f:
    extracted_content = pdf_document.get_content(f.read()).pages[0].textual_content
print(extracted_content)

To install this library

Run the following command

pip install chunking4rag

To start with contribution to the project

  1. Clone the repository using git

  2. Create a virtual environment using uv

uv create chunking4rag
  1. Activate the virtual environment
source .venv/bin/activate
  1. Install the dependencies by running
uv install -r requirements.txt
  1. Run tests to make sure everything is working fine
python chuking_tests.py

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for harpreetset from gravatar.com harpreetset

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: Harpreet Sethi
  • Requires: Python >=3.12
  • Provides-Extra: test , formatting , type-checking , docs , publishing
Classifiers

Release history Release notifications | RSS feed

This version

0.0.5

0.0.4

0.0.3

0.0.1 yanked

Reason this release was yanked:

deprecated and old

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

chunking4rag-0.0.5.tar.gz (12.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

chunking4rag-0.0.5-py3-none-any.whl (16.3 kB view details)

Uploaded Python 3

File details

Details for the file chunking4rag-0.0.5.tar.gz.

File metadata

  • Download URL: chunking4rag-0.0.5.tar.gz
  • Upload date:
  • Size: 12.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for chunking4rag-0.0.5.tar.gz
Algorithm Hash digest
SHA256 bf527221cc1d473fc331e4d7e0d969a5851d4061c0db7cab03c02782391ba5f6
MD5 01c427d1df9cdc435509bf07cd967bae
BLAKE2b-256 73c66b7ee8c00a30134dfa33181f9627556a3d670aa1aa67877bf98de8f393af

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunking4rag-0.0.5.tar.gz:

Publisher: python-publish.yml on harpreetset1/chunking4rag

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file chunking4rag-0.0.5-py3-none-any.whl.

File metadata

  • Download URL: chunking4rag-0.0.5-py3-none-any.whl
  • Upload date:
  • Size: 16.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for chunking4rag-0.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 ca7c43257ac39ea54467fb8f19dde7afe0a505c8eebc44b71e4fc107bce1d38d
MD5 9138c77a55c07c0a012226836c39c3af
BLAKE2b-256 2bf8bbbcb5b5f9d03b8637ba49f8a1484fd618b1f265d1092e48704828d5f7e4

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunking4rag-0.0.5-py3-none-any.whl:

Publisher: python-publish.yml on harpreetset1/chunking4rag

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page