Utility tools for Desync AI users
Project description
desync-data
Utility functions for working with PageData objects from the desync_search library. These tools help you clean, filter, deduplicate, extract links, and export structured data from websites crawled via Desync.
🚀 Features
- 🔍 Remove boilerplate from page content (prefix/suffix delimiters)
- 📌 Filter pages by URL substring
- 🧹 Remove duplicate pages (based on
text_content) - 🔗 Extract link graphs (internal navigation structure)
- 📤 Export pages to CSV, JSON, or SQLite
📦 Installation
pip install desync-tools
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
desync_data-0.1.0.tar.gz
(4.3 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file desync_data-0.1.0.tar.gz.
File metadata
- Download URL: desync_data-0.1.0.tar.gz
- Upload date:
- Size: 4.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8ffb0d9209c954443a82f224703bcb809893002ba5cb4add413d93e14d63a194
|
|
| MD5 |
0f945230ed42ccb990c5f086c20530e8
|
|
| BLAKE2b-256 |
222e7e77d1549d3689c001b7c23f9fea64b2d48a41cc1c220e0363ebe8b67408
|
File details
Details for the file desync_data-0.1.0-py3-none-any.whl.
File metadata
- Download URL: desync_data-0.1.0-py3-none-any.whl
- Upload date:
- Size: 4.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
63e28b77789eeb3c62b93716d629b37fa7960bef3bd127c0cab29b4188d49a17
|
|
| MD5 |
f252b7daf745cd34bf36c10f75bad21f
|
|
| BLAKE2b-256 |
88a1b9e97449618a40cac1edbec9ce90cabf7f2026671ea6ee22b30fb9a958e8
|
