prim-sync 0.4.0

Primitive Sync - Bidirectional and unidirectional sync over SFTP. Multiplatform Python script optimized for the Primitive FTPd Android SFTP server.

[!WARNING] This repository currently works much better with my modified version of the Primitive FTPd Android SFTP server!

• install my fork from https://github.com/lmagyar/prim-ftpd - and use all the new features and bugfixes
• install the original version from https://github.com/wolpi/prim-ftpd - and disable hashing (-H option), enable file rename (-v option), and be patient with the extreme slow SD card access (SAF)
• or wait until the new features got merged into the original version, for the PR's statuses see https://github.com/wolpi/prim-ftpd/pulls/lmagyar

Primitive Sync

Bidirectional and unidirectional sync over SFTP. Multiplatform Python script optimized for the Primitive FTPd Android SFTP server.

Why another sync solution? Because none of the professional solutions can write SD cards and follow local symlinks, or are extremely slow or full of bugs (Syncthing, Resilio Sync, rsync, osync, rclone, Unison). I gave up and wrote it.

See my other project, https://github.com/lmagyar/prim-ctrl, for remote control of your phone's Primitive FTPd SFTP server and optionally Tailscale VPN.

See my other project, https://github.com/lmagyar/prim-batch, for batch execution of prim-ctrl and prim-sync commands.

Note: These are my first ever Python projects, any comments on how to make them better are appreciated.

Features

• Follow local symlinks
• Hash files for fast comparison ( !!! currently requires the forked Primitive FTPd !!! )
• Write SD card (with Primitive FTPd and Storage Access Framework) ( !!! fast operation currently requires the forked Primitive FTPd !!! )
• Dual access in case of SD card (reading plain-old file-system for fast scan and download, and writing with the slower Storage Access Framework)
• Failsafe, restartable operation (costs some time, renames on SD card are slow)

Notes on following local symlinks

• File symlinks just work
• File hardlinks are OK for unidirectional outward sync, but have to use --overwrite-destination option in case of bidirectional or unidirectional inward sync
  • Better not to use file hardlinks, use symlinks
• Folder symlinks or junctions are OK for unidirectional outward sync, but be very-very-very careful with bidirectional or unidirectional inward sync
  • If you have a folder symlink and you think you delete files on your phone under the symlinked folder, or even you delete the symlinked folder, because "they are only under a symlink", you shoot yourself in the foot
  • Syncing file deletions means file deletions synced first, then the containing folder deletion synced
  • So first all the files will be deleted in the symlink target folder, then the folder symlink itself will be deleted, though the target folder is not deleted
  • So symlinking the family picture albums' folder better done with an unidirectional outward sync
  • Or symlink only the files if you enable deletion on the phone
  • You have been warned!

Installation

You need to install:

• Primitive FTPd on your phone

  • My forked version - see: https://github.com/lmagyar/prim-ftpd download from Releases
  • Original version - see: https://github.com/wolpi/prim-ftpd install from F-Droid (not from Google Play)

• Python 3.12+, pip and venv on your laptop - see: https://www.python.org/downloads/ or

  Unix

  sudo apt update
  sudo apt upgrade
  sudo apt install python3 python3-pip python3-venv
  

  Windows

  • Install from Microsoft Store the latest Python 3 (search), Python 3.12 (App)
  • Install from Chocolatey: choco install python3 -y

• pipx - see: https://pipx.pypa.io/stable/installation/#installing-pipx or

  Unix

  sudo apt install pipx
  pipx ensurepath
  

  Windows

  py -m pip install --user pipx
  py -m pipx ensurepath
  

• This repo

  pipx install prim-sync
  

Optionally, if you want to edit or even contribute to the source, you also need to install:

• poetry - see: https://python-poetry.org/

  pipx install poetry
  

Configuration

Android

You have to enable Primitive FTPd to run as much in the background as possible, please see the relevant Readme section.

Networking

Either use the built-in zeroconf (DNS-SD) functionality in Primitive FTPd (see below), or set up a constant address (IP or host name, for the -a option) for your phone (fixed LAN IP, VPN, hosts file, your choice).

Primitive FTPd

• Home tab
  • Select "Virtual folders" and follow the relevant Readme section
• Configuration tab
  • Authentication
    • Anonymous Login: disable
    • Username/Password: eg. sftp/sftp (will be disabled)
    • Public Key Authentication: disable (will be enabled)
  • Connectivity
    • Server(s) to be started: SFTP only
    • Secure Port: eg. 2222
    • Server Idle Timeout: 0
    • Idle timeout to stop server: eg. 60 or 0
    • Allowed IPs pattern, IP to bind to: at first leave them empty, you can harden your security later
  • UI
    • This is based on your preferences
  • System
    • Server Start Directory: eg. /storage/emulated/0
    • Prevent Standby: enable
    • Announce server in LAN: enable if you use zeroconf (DNS-SD)
    • Servername: make it unique, even if you don't use zeroconf, especially when multiple phones are synced, because this will be used as unique identifier to store the per-device-sync-state between runs
    • SFTP Hostkey algorithms: enable at least ed25519
    • Other options can be left unchanged
• Stop the server (if you have started)
• Close and restart the whole app
• Start the server

SSH keys

You need to generate an SSH key pair.

Unix

sudo apt install openssh-client
mkdir ~/.ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_sftp -N ""

Windows

Go to Settings / System / Optional features / Add an optional feature and add "OpenSSH Client"

mkdir %USERPROFILE%\.ssh
ssh-keygen -t ed25519 -f %USERPROFILE%\.ssh\id_ed25519_sftp -N ""

Then install it in Primitive FTPd:

• Use your favorite SFTP client (eg. WinSCP, FileZilla) to access the Primitive FTPd, use username/password to authenticate.
• Open for editing the /fs/storage/emulated/0/Android/data/org.primftpd/files/.ssh/authorized_keys file.
• Append the content of the previously generated .ssh/id_ed25519_sftp.pub file to it. It is something like "ssh-ed25519 XXXxxxXXXxxx you@your-device"

Then add your phone to the known_hosts file if your favorite SFTP client hasn't done it:

• Use ssh to access the Primitive FTPd, use username/password to authenticate.

  Unix

  ssh -oUserKnownHostsFile=~/.ssh/known_hosts -oPort=2222 [e-mail]
  

  Windows

  ssh -oUserKnownHostsFile=%USERPROFILE%\.ssh\known_hosts -oPort=2222 [e-mail]
  

• Acceph host key

• The error "shell request failed on channel 0" is OK, there is no SSH server in Primitive FTPd, our goal was to connect and store the server key in the known_hosts file.

Primitive FTPd again

• Configuration tab:
  • Authentication
    • Password: delete it
    • Public Key Authentication: enable
• Stop the server
• Close and restart the whole app
• Start the server

Usage

Create a backup of your files!!! Really!!! If you use symlinks, this is only question of time when will you delete something unintendedly!!!

The first upload is better done over USB connection and manual copy, because copying files over Wi-Fi is much slower. The prim-sync script handles both this "external" upload and the changes in the future.

The first run will be longer than a regular run, because without prior knowledge, the prim-sync script handles all files on both sides as newly created and compares them or their hashes (hashing is much faster than downloading and comparing the content).

On regular runs the meaning of the log lines are:

• Scanning - Name of the remote folder that is scanned (only remote is logged, remote is the bottleneck)
• Comparing, Hashing - Comparing the content or the hash of the files on the two sides.
• <<< !!! >>> - Conflicting changes that are not resolved by any command line option, the details are in the next line.
• RECOVER - The previous run failed (probably network/connection problem), and there are intermediate/leftover files that are deleted on the next (ie. this) run.
• INVALID - Invalid characters in the filename are replaced because --valid-chars command line option is used.
• HARDLNK - There are hardlinks on the destination side and --overwrite-destination command line option is not used.
• SYMLINK - There are folder symlinks or junctions on the destination side and --folder-symlink-as-destination command line option is not used.
• CHANGED - The destination file changed after the decision is made to update it and before it replaced by the new content, this conflict will be handled on the next run.

Notes:

• In the log lines the left side is the Local and the right side is the Remote
• Local file creation times (birthtime) are:
  • preserved on Windows but not on Unix when the default restartable operation is used
  • unchanged when --overwrite-destination option is used
• You can brainwash (ie. delete the state under the .prim-sync folder) between two runs. After this, the script will behave, as if the next run is the first run (see "first run" above).
• Never ever delete any files where the name ends with .prim-sync.new or .tmp or .old, the pure existence of these files are the "transaction state", if you delete any of these files, the recovery algorythm won't be able to figure out in which phase got the restartable operation interrupted. If you delete any of these files, you are on your own to figure out how to recover from the interruption.

Some example

Unix

prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" "~/Mobile" "/fs/storage/XXXX-XXXX" "/saf" "Camera" "DCIM/Camera"
prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" -uo -m --overwrite-destination "~/Mobile" "/fs/storage/XXXX-XXXX" "/saf" "Music" "*"
prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" -a your.phone.host.name 2222 "~/Mobile" "/fs/storage/emulated/0" "*" "Screenshots" "DCIM/Screenshots"

Windows

prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" "D:\Mobile" "/fs/storage/XXXX-XXXX" "/saf" "Camera" "DCIM/Camera"
prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" -uo -m --overwrite-destination "D:\Mobile" "/fs/storage/XXXX-XXXX" "/saf" "Music" "*"
prim-sync your-phone-pftpd id_ed25519_sftp -t -sh -rs "/fs/storage/emulated/0" -a your.phone.host.name 2222 "D:\Mobile" "/fs/storage/emulated/0" "*" "Screenshots" "DCIM/Screenshots"

Options

usage: prim-sync [-h] [-a host port] [-V] [-ui | -uo] [-d] [-D] [-v [CHARS]] [-rs PATH] [--overwrite-destination] [--folder-symlink-as-destination] [--ignore-locks [MINUTES]] [-t] [-s] [-ss] [-sh] [--debug] [-M] [-C] [-H]
                 [-n | -o] [-cod | -doc] [-l [PATTERN ...]] [-r [PATTERN ...]] [-m [PATTERN ...]]
                 server-name keyfile local-prefix remote-read-prefix remote-write-prefix local-folder remote-folder

Bidirectional and unidirectional sync over SFTP. Multiplatform Python script optimized for the Primitive FTPd Android SFTP server (https://github.com/wolpi/prim-ftpd), for more details see https://github.com/lmagyar/prim-sync

positional arguments:
  server-name                        unique name for the server (if zeroconf is used, then the Servername configuration option from Primitive FTPd, otherwise see the --address option also)
  keyfile                            key filename located under your .ssh folder
  local-prefix                       local path to the parent of the folder to be synchronized
  remote-read-prefix                 read-only remote path to the parent of the folder to be synchronized, eg. /fs/storage/XXXX-XXXX or /rosaf
  remote-write-prefix                read-write remote path to the parent of the folder to be synchronized, eg. /saf (you can use * if this is the same as the read-only remote path above)
  local-folder                       the local folder name to be synchronized
  remote-folder                      the remote folder name to be synchronized (you can use * if this is the same as the local folder name above)

options:
  -h, --help                         show this help message and exit
  -a host port, --address host port  if zeroconf is not used, then the address of the server
  -V, --dont-validate-server-name    cached zeroconf address and server-name pairing validation is available only with prim-ftpd, disable on other servers
  -ui, --unidirectional-inward       unidirectional inward sync (default is bidirectional sync)
  -uo, --unidirectional-outward      unidirectional outward sync (default is bidirectional sync)
  -d, --dry                          no files changed in the synchronized folder(s), only internal state gets updated and temporary files get cleaned up
  -D, --dry-on-conflict              in case of unresolved conflict(s), run dry
  -v [CHARS], --valid-chars [CHARS]  replace [] chars in filenames with chars from CHARS (1 or 2 chars long, default is '()')
                                     Note: this is required only for the original Primitive FTPd SAF SD card access, will be removed
  -rs PATH, --remote-state-prefix PATH
                                     stores remote state in a common .prim-sync folder under PATH instead of under the remote-folder argument (decreases SD card wear), eg. /fs/storage/emulated/0
                                     Note: currently only the .lock file is stored here
                                     Note: if you access the same server from multiple clients, you have to specify the same --remote-state-prefix option everywhere to prevent concurrent access
  --overwrite-destination            don't use temporary files and renaming for failsafe updates - it is faster, but you will definitely shoot yourself in the foot when used with bidirectional sync
  --folder-symlink-as-destination    enables writing and deleting symlinked folders and files in them on the local side - it can make sense, but you will definitely shoot yourself in the foot
  --ignore-locks [MINUTES]           ignore locks left over from previous run, optionally only if they are older than MINUTES minutes

logging:
  -t, --timestamp                    prefix each message with a timestamp
  -s, --silent                       only errors printed
  -ss, --silent-scanning             don't print scanned remote folders as progress indicator
  -sh, --silent-headers              don't print headers
  --debug                            use debug level logging and add stack trace for exceptions, disables the --silent and enables the --timestamp options

comparison:
  -M, --dont-use-mtime-for-comparison
                                     beyond size, modification time or content must be equal, if both are disabled, only size is compared
  -C, --dont-use-content-for-comparison
                                     beyond size, modification time or content must be equal, if both are disabled, only size is compared
  -H, --dont-use-hash-for-content-comparison
                                     not all sftp servers support hashing, but downloading content for comparison is mush slower than hashing

bidirectional conflict resolution:
  -n, --newer-wins                   in case of conflict, newer file wins
  -o, --older-wins                   in case of conflict, older file wins
  -cod, --change-wins-over-deletion  in case of conflict, changed/new file wins over deleted file
  -doc, --deletion-wins-over-change  in case of conflict, deleted file wins over changed/new file
  -l [PATTERN ...], --local-wins-patterns [PATTERN ...]
                                     in case of conflict, local files matching this Unix shell PATTERN win, multiple values are allowed, separated by space
                                     if no PATTERN is specified, local always wins
  -r [PATTERN ...], --remote-wins-patterns [PATTERN ...]
                                     in case of conflict, remote files matching this Unix shell PATTERN win, multiple values are allowed, separated by space
                                     if no PATTERN is specified, remote always wins

unidirectional conflict resolution:
  -m [PATTERN ...], --mirror-patterns [PATTERN ...]
                                     in case of conflict, mirror source side files matching this Unix shell PATTERN to destination side, multiple values are allowed, separated by space
                                     if no PATTERN is specified, all files will be mirrored