filecrawl 1.0.0

A simple crawler obtaining all available files from the campus management platform Studip and moodle for University Trier

Filecrawl

A simple crawler obtaining all available files from the campus management platforms Studip and moodle for University Trier

Disclaimer

This version works only for the University Trier and Studip 4. Use the Studip RESTAPI if possible as Crawlers are forbidden by many universities.

There are other open Source Clients supporting the API, e.g. studip-fuse or STUD.IP-FileSync.

Installation

Windows

1. Download Python 3.4+
2. Download filecrawl and unzip it.
3. open cmd
4. cd /path/to/filecrawl/
5. python -m pip install -r requirements.txt

MacOS and Linux

1. Download Python 3.4+
2. Download filecrawl and unzip it.
3. open terminal
4. cd /path/to/filecrawl/
5. pip3 install -r requirements.txt
   • When you use a Linux Distro which isn't supported by the keyrings backend (e.g. Raspbian) you also need to run pip3 install keyrings.alt

Usage

Windows

Simply run the script with python filecrawl.py when you are in the filecrawl folder. It will guide you through the setup when you run the script for the first time.

MacOS and Linux

Simply run the script with python3 filecrawl.py when you are in the filecrawl folder. It will guide you through the setup when you run the script for the first time.

• You may get an Error when using help in the Setup on macOS stating that FIFinderSyncExtensionHost is implemented in two paths - just ignore this. This also leads to a Finder window which is stuck as long as the script is running.

Crontabs

When you want to run filecrawl as a crontab, you have to have root privileges or run everything with sudo. Also, you may have to change the path where the config will be saved in filecrawl/L233, config_handling/L89 and config_handling/L100 as the root user has no /User/ folder where the config will be stored.

Credits & Licence

Under MIT LICENCE.