Project description

scrapy-folder-tree

PyPI PyPI - Format PyPI - Status Code style: black Checked with mypy Imports: isort

This is a scrapy pipeline that provides an easy way to store files and images using various folder structures.

Given this scraped file: 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg, you can choose the following folder structures:

Using file name

full
├── 0
.   ├── 5
.   .   ├── b
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

Using crawling time

full
├── 0
.   ├── 11
.   .   ├── 48
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

Using crawling date

full
├── 2022
.   ├── 1
.   .   ├── 24
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

pip install scrapy_folder_tree

Use the following settings in your project:

ITEM_PIPELINES = {
    'scrapy_folder_tree.FilesHashTreePipeline': 300
}

FOLDER_TREE_DEPTH = 3

These details have not been verified by PyPI

This version

0.1.3

Feb 1, 2022

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Uploaded Feb 1, 2022 Source

Uploaded Feb 1, 2022 Python 3

Details for the file scrapy-folder-tree-0.1.3.tar.gz.

Hashes for scrapy-folder-tree-0.1.3.tar.gz
Algorithm	Hash digest
SHA256	`5568ea3ec11aae42155e08cdb585516292779139216640cadfcb59759448252d`
MD5	`ac2df3c865c5477548b26092dda441cc`
BLAKE2b-256	`06790ab65d7b75e6848b80ebd9b2ca4b09a6a439d1a22458d6ff9bbb6f4c401a`

Details for the file scrapy_folder_tree-0.1.3-py3-none-any.whl.

Hashes for scrapy_folder_tree-0.1.3-py3-none-any.whl
Algorithm	Hash digest
SHA256	`a407ffc0ba26d66cbb870880a1fe90a59e8e8a2aec83e55f8a7cd24b40d83f63`
MD5	`4de8133bf21ff78c720243ac088ba76c`
BLAKE2b-256	`7263ee311de80558f06c38ab4132bf3ffdd26b781a5bbdd049955fe59f4a6aab`