qullm 0.3.0

Of course, here is the detailed example of the README.md file, which describes the functionality, installation method, and usage examples of the qullm package.

# QULLM

QUllm is a Python package for preprocessing and augmenting data for large language models using quantum neural networks.

## Features

- **Data Loading**: Load data from various file formats.
- **Data Preprocessing**: Clean, normalize, and filter text data.
- **Data Augmentation**: Enhance data with techniques like synonym replacement, random insertion, deletion, and swapping.
- **Data Analysis**: Analyze data to get word frequencies, unique words, and other statistics.

## Installation

You can install the package using pip:

```bash
pip install qullm

Usage

Here is an example of how to use the qullm package:

from qullm.llm_data_processor import DataLoader, DataPreprocessor, DataAugmenter, DataAnalyzer, save_data

# Specify the path to your data file
data_file_path = 'path/to/your/data.txt'
processed_data_file_path = 'path/to/save/processed_data.txt'

# Step 1: Load data
data_loader = DataLoader(data_file_path)
data = data_loader.load_data()
print("Original Data:")
print(data[:5])  # Print first 5 lines for inspection

# Step 2: Preprocess data
preprocessor = DataPreprocessor(data)
cleaned_data = preprocessor.clean_data()
normalized_data = preprocessor.normalize_data()
filtered_data = preprocessor.filter_data()

print("Cleaned Data:")
print(cleaned_data[:5])  # Print first 5 lines for inspection

print("Normalized Data:")
print(normalized_data[:5])  # Print first 5 lines for inspection

print("Filtered Data:")
print(filtered_data[:5])  # Print first 5 lines for inspection

# Step 3: Augment data
augmenter = DataAugmenter(filtered_data)
augmented_data = augmenter.augment_data()
print("Augmented Data:")
print(augmented_data[:5])  # Print first 5 lines for inspection

# Step 4: Analyze data
analyzer = DataAnalyzer(augmented_data)
word_freq = analyzer.get_word_frequency()
stats = analyzer.get_data_statistics()

print("Word Frequency:")
print(word_freq.most_common(10))  # Print top 10 most common words

print("Data Statistics:")
print(stats)

# Step 5: Save processed data
save_data(augmented_data, processed_data_file_path)
print(f"Processed data saved to {processed_data_file_path}")

Modules

DataLoader

A module for loading data from files.

Methods

• load_data(): Load data from a file and return as a list of strings.
• load_data_as_string(): Load data from a file and return as a single string.
• load_data_as_lines(): Load data from a file and return as a list of lines.

DataPreprocessor

A module for preprocessing text data.

Methods

• clean_data(): Clean the data by removing extra whitespace and newlines.
• normalize_data(): Normalize the data by converting text to lowercase.
• filter_data(min_length): Filter out lines with fewer words than min_length.
• remove_special_characters(): Remove special characters from the data.
• remove_stopwords(stopwords): Remove stopwords from the data.

DataAugmenter

A module for augmenting text data.

Methods

• augment_data(augment_factor): Augment the data by duplicating and shuffling.
• synonym_replacement(word_map): Replace words with their synonyms.
• random_insertion(insertion_words, insert_prob): Randomly insert words into the data.
• random_deletion(delete_prob): Randomly delete words from the data.
• random_swap(swap_prob): Randomly swap words in the data.

DataAnalyzer

A module for analyzing text data.

Methods

• get_word_frequency(): Get the frequency of words in the data.
• get_data_statistics(): Get statistics such as the number of lines and words.
• get_unique_words(): Get unique words in the data.
• get_average_line_length(): Get the average length of lines in the data.
• get_most_common_words(n): Get the top n most common words in the data.

Utils

Utility functions for saving data and loading configurations.

Methods

• save_data(data, file_path): Save data to a specified file path.
• load_config(config_path): Load configuration from a JSON file.
• save_config(config, config_path): Save configuration to a JSON file.
• split_data(data, train_ratio): Split data into training and testing sets.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

Thanks to the contributors and the open-source community for their support and contributions.

### Explanation

1. **Features**:
   - Briefly introduces the main functionalities of the package, including data loading, preprocessing, augmentation, and analysis.

2. **Installation**:
   - Provides the command to install the package.

3. **Usage**:
   - Offers a complete usage example demonstrating how to load, preprocess, augment, analyze, and save data.

4. **Modules**:
   - Detailed introduction of each module and its methods, including `DataLoader`, `DataPreprocessor`, `DataAugmenter`, `DataAnalyzer`, and `Utils`.

5. **License**:
   - Provides information about the project's license.

6. **Acknowledgements**:
   - Thanks the contributors and the open-source community for their support and contributions.

In this way, users can easily understand the functionality and usage of the `qullm` package to better preprocess and augment large language data.