Crawl your personal favorite images, photo albums, comics from website. Support pixiv, yande.re for now.
Project description
FavoritesCrawler
Crawl your personal favorite images, photo albums, comics from website.
Warning!
- Not ready for production.
- Appropriately reduce the crawling speed in the future and provide options to set performance, but your account is still at risk of being disabled by the website.
Plan to support
- instagram.com
Already support
- pixiv.net (crawl your liked illust, must login), Thanks for project PixivPy.
- yande.re (crawl your voted posts, require your username)
- lmmpic.com (crawl your favorite albums, must login)
- nhentai.net (crawl your favorite comics, must login)
- twitter.com (crawl your liked posts, must login)
Requirements
- Python3.7+
Install
pip install -U favorites_crawler
Config Proxy (Optional)
# on Windows
set https_proxy=http://localhost:8080 # replace with your proxy server
# on Liunx/macOS
export https_proxy=http://localhost:8080
Login
favors login [-h] {pixiv,yandere}
Login Pixiv
Thanks for @ZipFile Pixiv OAuth Flow
- run command
favors login pixiv
- input your user_id (Access your pixiv personal page, copy from address bar), after press Enter, Pixiv login page will open in browser.
- Open dev console (F12) and switch to network tab.
- Enable persistent logging ("Preserve log").
- Type into the filter field: callback?
- Proceed with Pixiv login.
- After logging in you should see a blank page and request that looks like this: https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback?state=...&code=.... Copy value of the code param into the prompt and hit the Enter key.
Login Yandere
- run command:
favors login yandere
- input your username and hit the Enter key.
Login Lmmpic
- Open lmmpic on browser and login.
- Use "Get cookies.txt" extension download cookie file.
- Copy cookie file to {user_home}/.favorites_crawler.
Login NHentai
- Open nhentai on browser and login.
- Use "Get cookies.txt" extension download cookie file.
- Copy cookie file to {user_home}/.favorites_crawler.
Login Twitter
- run command
favors login twitter
- input your username, after press Enter, likes page will open in browser.
- Open dev console (F12) and switch to network tab.
- Enable persistent logging ("Preserve log").
- Type into the filter field: Likes?
- Refresh Page.
- Copy Authorization, X-Csrf-Token and RequestURL from request(Likes?variables...) input on terminal.
- Use "Get cookies.txt" extension download cookie file.
- Copy cookie file to {user_home}/.favorites_crawler.
Crawl
favors crawl [-h] {lemon,nhentai,pixiv,yandere}
Crawl Pixiv
Before run this command, make sure you are already login.
favors crawl pixiv
Crawl Yandere
Before run this command, make sure you are already login.
favors crawl yandere
Crawl Lmmpic
Before run this command, make sure you are already login.
favors crawl lemon
Crawl NHentai
Before run this command, make sure you are already login.
favors crawl nhantai
Crawl Twitter
Before run this command, make sure you are already login.
favors crawl twitter
Config
Config file locate on {your_home}/.favorites_crawler/config.yml
.
You can set any scrapy built-in settings in this file.
By default, file content likes this:
pixiv:
ACCESS_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
REFRESH_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
USER_ID: xxxx
yandere:
USERNAME: xxxx
Download location
By default, pictures will download to working directory.
If you want to change download location, you can add FILES_STORE option to config.
For example, if you want save pixiv files to pictures/a
, and want save yandere files to pictures/b
, you can modify config file like this:
pixiv:
ACCESS_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
REFRESH_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
USER_ID: xxxx
FILES_STORE: pictures/a
yandere:
USERNAME: xxxx
FILES_STORE: pictures/b
Organize file by artist
if you want to organize pixiv illust by user, add this line to your config:
pixiv:
# FAVORS_PIXIV_ENABLE_ORGANIZE_BY_USER: true # (Deprecation)
ENABLE_ORGANIZE_BY_ARTIST: true # add this line to your yandere config
if you want to organize yandere post by artist, add this line to your config:
yandere:
ENABLE_ORGANIZE_BY_ARTIST: true # add this line to your yandere config
Store tags to IPTC/Keywords
only support pixiv and yandere.
yandere:
ENABLE_WRITE_IPTC_KEYWORDS: true # default: true
EXIF_TOOL_EXECUTABLE: '<Path to your exiftool executable>' # default None
pixiv:
ENABLE_WRITE_IPTC_KEYWORDS: true # default: true
EXIF_TOOL_EXECUTABLE: '<Path to your exiftool executable>' # default None
Restore your favorites
Vote yandere posts
$ favors restore yandere -h
usage: favors restore yandere [-h] -s {0,1,2,3} -t CSRF_TOKEN -c COOKIE path
positional arguments:
path The location of the post to vote. (Sub-folders are ignored)
optional arguments:
-h, --help show this help message and exit
-s {0,1,2,3}, --score {0,1,2,3}
Set 1, 2 or 3 to vote, 0 to cancel vote.
-t CSRF_TOKEN, --csrf-token CSRF_TOKEN
CSRF token. To get it:
1. Open your browser DevTools.
2. Switch to network tab.
3. Vote any post on yandere.
4. Copy x-csrf-token value from request headers.
-c COOKIE, --cookie COOKIE
Cookie. To get it:
1. Open your browser DevTools.
2. Switch to network tab.
3. Vote any post on yandere.
4. Copy cookie value from request headers.
Example:
favors restore yandere -s 3 -t "xxxx" -c "xx=x; xx=x; xx=x" .
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file favorites_crawler-0.2.1.tar.gz
.
File metadata
- Download URL: favorites_crawler-0.2.1.tar.gz
- Upload date:
- Size: 24.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 390a83452b3baa1ac0bde29f536b65ce554c0227de41643e7f106bdb18f66d62 |
|
MD5 | 355fe49154c8199d8ca47e123d6ad75a |
|
BLAKE2b-256 | d081acee5f380d7a35bc894f2dbc606b2e6ea58074c41dd74f03df2a8e823727 |
File details
Details for the file favorites_crawler-0.2.1-py3-none-any.whl
.
File metadata
- Download URL: favorites_crawler-0.2.1-py3-none-any.whl
- Upload date:
- Size: 26.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 759789e7abcd38557c9f577db6aed4c128e1e410454f0ac541882b8a868050ad |
|
MD5 | 9b5fbd1755a4d2b9e9c13420102c4131 |
|
BLAKE2b-256 | 37e633284ced171d56fdf3b3e84e131dce8f8c38bef1dec11abd9b649f0f665c |