Syncweb

An offline-first distributed web

This ‘World Wide Web’ was just a lame text format and a lot of connected directories.

Ted Nelson

Syncweb is not a physical network, but rather a logical network resting on top of several existing physical networks. These networks include, but are not limited to, Syncthing nodes and relays, NNCP, the Internet, an Ethernet. What is important is that two neighboring systems on Syncweb have some method to exchange files, from one system to the other, and once on the receiving system, processed by the Block Exchange Protocol software on that system.

Install

pip install syncweb

Install syncweb-automatic (optional)

Syncweb-automatic is an optional daemon that will auto-accept new local devices and their folders

curl -s https://raw.githubusercontent.com/chapmanjacobd/syncweb-py/refs/heads/main/examples/install.sh | bash

Usage

Start your syncweb cluster by creating your first folder (can be an existing folder)

$ syncweb create ./audio/
sync://audio#CKTVWGQ-XBRFFRH-YTRPQ5G-YDA5YXI-N66GA5J-XVBGEZ3-PD56G6Y-N7TEAQC

Share your new syncweb folder with someone else by having them run:

$ syncweb join sync://audio#CKTVWGQ-XBRFFRH-YTRPQ5G-YDA5YXI-N66GA5J-XVBGEZ3-PD56G6Y-N7TEAQC
Local Device ID: NXL7XBL-VPNDOSR-QXU7WI7-NEUI65A-TWN7YGT-WS2U457-NTZNGB4-J6IYDQH

You will then need to accept their device ID. Their device ID is printed for convenience when running the previous command syncweb join but they can also find their 🏠 device ID by running syncweb devices.

$ syncweb accept --folders=audio NXL7XBL-VPNDOSR-QXU7WI7-NEUI65A-TWN7YGT-WS2U457-NTZNGB4-J6IYDQH

Now you're ready to share files!

List files

$ syncweb ls --long audio/
Type       Size      Modified  Name
-----------------------------------
d        2.7GiB  06 Oct 20:56 Recordings/
d        3.2MiB  28 Jul  2022 Documentation/

Find files

$ syncweb find --type=f --ext MKA --size=-20M --min-depth=2 Test
audio/Recordings/TestRecording_1.mka
audio/Recordings/TestRecording_22.mka
audio/Recordings/TestRecording_23.mka
...

# shorthand
$ syncweb find -tf -eMKA -S=-20M -d=+2 Test

Sort

$ syncweb find -tf -eMKA -S=-20M -d=+2 Test | syncweb sort "balanced,frecency" | tee download_list.txt
audio/Recordings/TestRecording_23.mka
audio/Recordings/TestRecording_22.mka
audio/Recordings/TestRecording_1.mka

Download

$ cat download_list.txt | syncweb download --yes
Download Summary:
---------------------------------------------------------------------------------------
Folder ID    Files   Total Size       Usable      Pending       Buffer  Shared   Status
---------------------------------------------------------------------------------------
audio           87     216.9MiB     114.6GiB            - 18.4GiB (1%)      No       OK
---------------------------------------------------------------------------------------
TOTAL           87     216.9MiB
---------------------------------------------------------------------------------------

Mark 87 files (216.9MiB) for download? [y/N]:

List devices

$ syncweb devices
Device ID                                                        Name     Last Seen              Duration    Bandwidth Limit
---------------------------------------------------------------  -------  ---------------------  ----------  -----------------
FXVNMWB-RHVAHJX-GG6QRDK-BMBO7KH-526SY6Y-6YVKEIE-2PPM5P7-F62C2AT  syncweb  🏠                                 Unlimited
7ZEWMJV-3JBF4CI-F4Z4CWC-U6232BS-3CP6BUI-FSXZORS-JHGJ4V3-5QDWCAE  syncweb  😴 23 days ago, 19:34  1.3 days    Unlimited

List folders

syncweb folders
Folder ID    Label    Path                         Local             Needed             Global                Free    Sync Status      Peers  Errors
-----------  -------  ---------------------------  ----------------  -----------------  --------------------  ------  -------------  -------  -------------------
content      -        /tmp/tmp.WwxPwoouIa/content  0 files (0Bytes)  1 files (10.8GiB)  753 files (170.5GiB)  -       94% error            1  folder path missing

Debugging

You can start another instance of Syncweb like this:

syncweb --home=/tmp/1/ join sync://test#CKTVWGQ-XBRFFRH-YTRPQ5G-YDA5YXI-N66GA5J-XVBGEZ3-PD56G6Y-N7TEAQC
syncweb --home=/tmp/1/ repl

Future Aspirations

What Syncweb is

Syncweb builds on top of Syncthing as an opinionated selective-sync configuration helper.

The advantages to using it are that it is offline first. You can download a whole website and use the site fully offline. When you come back online the new changes and updates will be synced and your comments and interactions will be automatically uploaded. It is delay-tolerant.

The disadvantage is that browser support for Syncweb URLs is virtually non-existant at this time. I have no plans to work on this aspect. Feel free to lead the charge!

The other really big disadvantage is that Syncweb is fragmented. But this limitation encourages small, productive, file-sharing groups! See what other people are sharing and find a group that matches your interests.

What Syncweb is not

Syncweb will never replace your online banking app. While it may be possible to write something equivalent, I imagine doing so will be very clunky. The traditional web has very mature patterns for building. Requests are atomically mapped out across multiple services.

The traditional web has a robust line of authority via the Domain Name System so you can easily know whether you are on your bank's website or not. Syncweb has left this authority up to the community. Syncthing does not have a built-in certificate revocation mechanism like Certificate Authorities (CAs) do. You control the trust relationships of your devices directly.

Links

For example, a browser can be used in AFS by using “file://” rather than “http://” in addresses. All of the powerful caching and consistence-maintenance machinery that is built into AFS would then have been accessible through a user-friendly tool that has eventually proved to be enormously valuable. It is possible that the browser and AFS could have had a much more symbiotic evolution, as HTTP and browsers eventually did.

Mahadev Satyanarayanan

      _____________
     '-------------.`-.
        /..---..--.\\  `._________________________________________
       //||   ||   \\\   `-\\-----\\-----\\-----\\-----\\-----\\--\
   __.'/ ||   ||    \\\     \\     \\     \\     \\     \\     \\  \
  /   /__||___||___.' \\     \\     \\     \\     \\     \\     \\ |
  |       |  -|        \\     \\     \\     \\     \\     \\     \\/
  |       |___|________ \\     \\     \\     \\     \\     \\_.-'
  [ ____ /.-----------.\ \\     \\     \\     \\     \\   .'
 | |____|/ .-'''''''-. \\ \\     \\     \\ .-'''''''-.\\_/
 | |____|.'           '.\\ \\____....----.'           '.
 | |___ /    .-----.    \\\______....---/    .-----.    \
 | |___|    / o o o \    \|============|    / o o o \    \
 | |__ |   | o     o |   ||____________|   | o     o |   |
[_.|___\    \ o o o /    |          LGB\    \ o o o /    |
  .  .  \    '-----'    /  .   ..  . .  \    '-----'    /  . .
 .  .  . '.           .'   .  .   .   .  '.           .' .  .
 ..  .   . '-._ _ _.-'   .  .   .   .   .  '-._ _ _.-' . .

UNDER CONSTRUCTION