Pure Python program to download any user's gallery/scraps/favorites and more from FurAffinity forum in an easily handled database.
Project description
FALocalRepo
Pure Python program to download any user's gallery/scraps/favorites and more from FurAffinity forum in an easily handled database.
Warning: New version under development in pre-alpha stage. Current version 2.11.2 does not work.
Warning: A cookie file named FA.cookies.json in json format is needed.
Warning: You need to set the theme to 'beta' on FurAffinity
Warning: On Windows safe interruption does NOT work prior to version 2.10
Warning: On Unix system the provided binaries require installing the pypi modules used in this software (See Build instructions
)
Introduction
This program was born with the desire to provide a relatively easy-to-use method for FA users to download submissions that they care about from the forum. At the moment its a little more than a text interface in a terminal window with only basic search functionality, a GUI will be hopefully added in the near future.
Contents
Usage
Use the provided binaries or build your own (build instructions at the end) and run from the command line on Linux or double click on Windows.
[FA|FA.exe] [options]
Options:
--raise
Allow exceptions to raise without special handling, useful for troubleshooting--log
Log major program operations to a file named 'FA.log'--logv
Log all operations (including sub-steps)
To always use specific options please consult a guide on how to modify default arguments for your operative system.
When the program starts a simple menu will appear, type the indicated number or key to select an option, there is no need to press ENTER.
Download & Update
This menu allows to download a user gallery, scraps, favorites, extras or to update specific users and/or sections for the database.
When a submission is downloaded all its informations (except for the comments and user-made folders) are downloaded into a database located in the same folder the program is launched in. The file (artwork, story, audio, etc...) is saved inside a folder named 'FA.files' which contains all submissions in a tiered structure based on their ID (e.g submission '3704554' will be saved in the folder 'FA.files/0/3/704/0003704554'). Backup .txt and .html files are also saved with the file, they contain the basic informations and the description and are there for safety (in case the database is accidentally deleted). For a guide on the database structure see Database
below.
-
Username:
First field is reserved for users. To download or sync a specific user/s insert the username/s (url or userpage name are both valid). Usernames can be separated with spaces or commas. -
Sections:
Second field is reserved for sections. These can be:g
- Gallerys
- Scrapsf
- Favoritese
- Extras partial
Searches submissions that contain ':iconusername:' OR ':usernameicon:' in the description and are not from the user's gallery/scraps.E
- Extras full
Like partial but also searches for 'username' in the description, keywords and title.
Sections can be omitted if 'update' option is used.
-
Options:
Last field is reserved for options. These can be:sync
Stops download when a submission already present in the user database entry is encountered.update
Reads usernames from the database and downloads new submissions in the respective sections. This option can be used without specifying users or sections, if either is specified then the update will be limited to those user/s and/or section/s.forceN
Prevents update and sync from stopping the download at the first already present submission. Download stops at the first downloaded submission from page N+1. Example: 'force4' will download the first 4 pages with no interruption and will allow the download to stop from page 5.all
Like 'force' but it will prevent interrupting the download for the whole section (this means ALL pages from each user will be checked, only use for a limited amount of users).slow
Use lowest possible download speed and make sure each submission doesn't take less than 1.5 seconds.noindex
Do not update indexes after completing the download.dbonly
Do not save any file during download/update, save submissions only in the database.quit
Quits the program when the current operation is completed.
Note: options can be inserted with or without spaces between them.
-
After inserting the necessary usernames/sections/options (and making sure their combination is valid) the program will:
- Check connection to FA website
- Build a Session object and add the provided cookies
- Check validity of cookies and bypass cloudflare
If all these steps are completed without errors then the program will proceed to download the targets. As a bonus feature the program will also handle filetypes to make sure the submission files have the correct extension.
If the the program cannot verify the cookies and connect to the forum then it will abort the download and check the cookies for some common errors.The program throttles download speed down to 100KB/sec to avoid taxing the forum's servers with a huge number of requests and downloads close to each other.
Search
This menu allows to search in the database using one or more among user (with or w/o sections), title, tags, category, species, gender and rating.
-
User
Search users. Multiple users can be matched.Section
If a user is selected the search can be restricted to a specific section/s using g, s, f, e.
-
Title
Search titles. -
Description
Search inside submissions' descriptions. -
Tags
Tags are sorted automatically before search. -
Category
*
Matches the category of submissions, like 'Artwork', 'Story', etc... -
Species
*
Search species, like 'Vulpine', 'Feline', etc... -
Gender
*
Gender can be 'Male', 'Female', 'Any'. -
Rating
*
The rating can be 'general', 'mature' or 'adult'. -
Options
There are two possible options:regex
Use regular expressions to search the database. Full regex syntax is supported in all fields.case
Turn on case sensitivity on ALL fields. This works in both normal and regex mode.web
Search on the website directly. Only user, title, tags, description and rating will be used.
If indexes aren't up to date, and web option wasn't used, they will be updated.
Results are ordered by username and date.
When turned on case sensitivity will be enabled for all fields, this means that, for example, 'tiger' won't match anything as the species is saved as 'Tiger' on FA.
If no results can be found in the local database the program will prompt to run the search on the website instead.
* As shown on the submission page on the main site.
Repair database
Selecting this entry will open the database repair menu. There are 6 possible choices.
-
Submissions
This will start the analysis and repair of the SUBMISSIONS table.ID
Missing IDs will be flagged.
This error type doesn't have a fix yet as there is no clear way to identify the submission on FA. However the program cannot create this type of error.Fields
If the id passes the check then the other fields in the submission entry will be searched for misplaced empty strings, incorrect value types and incorrect location.
The program will try and fix the errors in-place, replacing NULL values with empty strings. If the automatic fixes are successful then the submission will be checked for missing files, if any is missing then the submission will be passed to the next step. However if the automatic fixes do not work then the corrupted entry will be erased from the database, the files (if any present) deleted and the submission downloaded again, thus also fixing eventual missing files.Files
If the previous checks have passed then the program will check that all submission files are present.
The program will simply erase the submission folder to remove any stray file (if any is present) and then download them again.
Indexes will be redone and database optimized with sqlite
VACUUM
function. -
Submission files
This will start the analysis of the submission table in search of submissions whoseFILENAME
column was set to 0 (i.e. an error during the download or in the file itself) and whoseSERVER
column is not 0 (i.e. that are still present on the main website as far the database knows).
This missing submission files will be downloaded from scratch again. -
Users
Empty users
Users with no folders and no submissions saved will be deleted.Repeating users
Users with multiple entries.
All the entries will be merged, the copies deleted and a new entry created. This new entry will be checked for incorrectFOLDERS
and empty sections.Names
Usernames with capital letters or underscores will be updated to lowercase and the underscores removed.Full names
Full usernames that do not match with their url version will be collected from the submissions database or the website. If both fail they will be substituted with the url version.No folders
Users whose folders entry is empty or missing sections with saved submissions (SeeDatabase
→USERS
→FOLDERS
) will be updated with said sections (e.g. user 'tiger' has submissions saved in theGALLERY
andFAVORITES
sections but theFOLDERS
column only contains 'g' so 'g' will be added toFOLDERS
).Empty sections
Users with folders but no submissions saved (e.g.FOLDERS
containss
but theSCRAPS
column is empty) will be redownloaded from FA (submissions already present in the database won't be downloaded again but simply added to the user's database entry).
Indexes will be redone and database optimized with sqlite
VACUUM
function. -
Infos
Repeated entries
Repeated entries will be deleted.Version error
If the database version is missing or incorrect it will be fixed.Database name
If the database name is not set to '' it will be fixed.Numbers errors
Missing or incorrect totals will be fixed.Update & Download time
Missing or incorrect epoch and duration of last update and download will be reset to 0.Index flag
Missing or not 0/1INDEX
flag value.
Indexes will be redone and database optimized with sqlite
VACUUM
function. -
All
Submissions, submission files, users and infos will all be checked and the database re-indexed and optimized. -
Analyze
Search for errors in all three tables but do not repair them. -
Index
Indexes will be updated. -
Optimize
Database will be optimized with sqliteVACUUM
function.
Interrupt
While the program is running you can use CTRL-C to interrupt the current operation safely. If a download is in progress it will complete the current submission and exit at the first safe point. Safe interrupt works in all sections of the program.
If you're using a version of the program lower than 2.10 safe interruption won't work on Windows systems.
Database
The database (named 'FA.db') contains three tables:
-
INFOS
DBNAME
Database custom name, unused at the moment.VERSION
Database version, this can differ from the program version.USERN
Number of users inUSERS
table.SUBN
Number of submissions in theSUBMISSIONS
table.LASTUP
Time when the last update was started (epoch time, in seconds).LASTUPT
Duration of last update (in seconds).LASTDL
Time when the last download was started (epoch time).LASTDLT
Duration of last download (in seconds).INDEX
Boolean value indicating whether indexes were updated after the last download/update.
-
USERS
The USERS table contains a list of all the users that have been download with the program. Each entry contains the following:USER
The url username of the user (no caps and no underscores).USERFULL
The full username as chosen by the user (with caps and underscores if present).FOLDERS
The sections downloaded for that specific user. A '!' beside a section means that the user was disabled, it is used as a flag for the program.*GALLERY
,SCRAPS
,FAVORITES
,EXTRAS
These contain a list of the submissions IDs downloaded for each section.*
* For a guide on what each section means see
Usage
→Sections
-
SUBMISSIONS
The last table is a list of all the single submissions downloaded by the program. Each entry has 14 different values:ID
The id of the submissionAUTHOR
,AUTHORURL
The author username in normal format and url format (e.g. 'Flying_Tiger' and 'flyingtiger')TITLE
The titleUDATE
Upload dateDESCRIPTION
Description in html format.TAGS
The submission's keywords sorted alphanumericallyCATEGORY
,SPECIES
,GENDER
,RATING
The category, species, gender and rating as listed on the submission's page on the forumFILELINK
,FILENAME
The link to the submission file on the forum and the name of the downloaded file (all files are named 'submission' + their proper extension) (an empty or 0 value means the file has an error on the forum and has not been downloaded)LOCATION
The location in the current folder of the submission's file, description and backup informations file.SERVER
This last field is defaulted to 1 and only updated to 0 if the program checks the submission on the forum and finds it missing (because the uploaded has either disabled or deleted it)
The database is built using sqlite so it can be easily opened and searched with a vast number of third-party programs. A good one is 'DB Browser for SQLite' (http://sqlitebrowser.org/) which is open source, cross-platform and has a project page here on GitHub.
Upgrading from earlier versions
When the program is started it will check the database for its version. If the database version is lower than the program then it will update it depending on the difference between the two.
0.x
to1.x
→2.x
New informations handled by version 2 and onward will be downloaded and added to the database, these include submission category, rating, gender and species. Depending on the size of the database to be updated this process may take a long time.
The older version of the database will be saved as 'FA.v1.db'2.0
to2.2
→2.3
Full versions of users' nicknames will be collected from the submissions database or the website. If both fail the url version will be used instead.
The older version of the database will be saved as 'FA.v2.db'2.3
→2.6
Two columns in theUSERS
will be renamed (NAME
→USER
andNAMEFULL
→USERFULL
) and descriptions will be moved inside the database. It is recommended to runRepair
(SeeUsage
→Repair database
for details) with a version lower than 2.6 to make sure all description files are present. When the upgrade is completed indexes will be created for all fields.
Size of the database will increase by about 2,8KB per submission (averaged on a database of over 234k submissions which increased by about 653MB).
The older version of the database will be saved as 'FA.v2_3.db'2.6
→2.7
EntryINDEX
will be added toINFOS
table. This upgrade cannot be interrupted and resumed later, however it is very fast being mostly a copy-paste.
At each update step the program will save a backup copy of the database.
Update can be interrupted and resumed later without losing progress.
Warning: The update cannot be skipped, to keep using a specific version of the database you need to download the release relative to that version
Cookies
The program needs to use cookies from a login session to successfully connect to FA. These cookies need to be in json format and can be easily extracted from Firefox/Chrome/Opera/Vivaldi/etc... using extensions or manually. The value must be written in a file named FA.cookies.json
(The value is fake)
[
{
"domain": ".furaffinity.net",
"expirationDate": 1511387940,
"hostOnly": false,
"httpOnly": false,
"name": "__asc",
"path": "/",
"sameSite": "no_restriction",
"secure": false,
"session": false,
"storeId": "0",
"value": "kee3gpzjurkaq9fbyrubhys7epk",
"id": 1
},
]
The following cookie names are needed in order to successfully connect:
- __asc
- __auc
- __cfduid
- __gads
- __qca
- a
- b
- _adb
Only 'name' and 'value' are needed for each cookie so the file can also be composed by entries like this:
[
{
"name": "__asc",
"value": "kee3gpzjurkaq9fbyrubhys7epk",
},
]
Build Instructions
This program is written with Python 3.x, Python 2.x will NOT work.
To run and/or build the program you will need the following pypi modules:
The following non-pypi modules (already included in this repo as submodules):
The following modules are used but available by default:
Once these modules are installed (suggest using pip
) then the program can be run through the Python 3.x interpreter or built using pyinstaller
or any other software.
Troubleshooting & Logging
The program is set up so that any unforeseen error interrupts the program after displaying the error details. To get extra information the program can be run with the --raise
argument which will allow exceptions to raise normally.
To get details of all operations the program can be run with '--log' or '--logv' arguments, warning and errors are saved regardless of the settings. Details will be saved in a file named 'FA.log' with the format: "YYYY-MM-DD hh:mm:ss.ssssss | LOG TYPE [N|V|W] | OPERATION -> detail
". Using '--log' will only log major passages; '--logv' will log all operations to file.
Opening issues
Before opening an issue please run the program with '--raise' and '--logv' arguments and copy the resulting log and exception/s details printed on screen.
Appendix
Unverified commits
All commits before the 27th of January 2018 show as unverified because I accidentally revoked my old GPG key before adding a new one. They have all been added by me and can vouch for their authenticity.
Some commits before e76e993b show as unverified, these were done with GitHub's editor and thus their GPG signature cannot be verified by GitLab.
Commits 96143db5-7d40880b (09-10/07/2018) are unverified because of an error with the GPG key.
Early releases
Release binaries before and including v2.10.2 can be found on GitHub -> Releases
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for falocalrepo-3.0.0a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c08c4391e5b57b2b3150b3b5b608918f615c23778ffbcbe8b0d95deb3827c978 |
|
MD5 | d82041c3029a644c656cdb3c54dc3acd |
|
BLAKE2b-256 | c02e87c9b4aca13288b3282d6d8360699c3a8b2ccde0d3e5729941d17b2ed7d1 |