http file sharing hub
Project description
โ๐ copyparty
- http file sharing hub (py2/py3) (on PyPI)
- MIT-Licensed, 2019-05-26, ed @ irc.rizon.net
summary
turn your phone or raspi into a portable file server with resumable uploads/downloads using any web browser
- server only needs
py2.7
orpy3.3+
, all dependencies optional - browse/upload with IE4 / netscape4.0 on win3.11 (heh)
- resumable uploads need
firefox 34+
/chrome 41+
/safari 7+
for full speed - code standard:
black
๐ท screenshots: browser // upload // unpost // thumbnails // search // fsearch // zip-DL // md-viewer // ie4
get the app
'' ''
(basic upload client, nothing fancy yet)
readme toc
- top
- quickstart - download copyparty-sfx.py and you're all set!
- on servers - you may also want these, especially on servers
- on debian - recommended additional steps on debian
- notes - general notes
- status - feature summary
- testimonials - small collection of user feedback
- quickstart - download copyparty-sfx.py and you're all set!
- motivations - project goals / philosophy
- future plans - some improvement ideas
- bugs
- FAQ - "frequently" asked questions
- accounts and volumes - per-folder, per-user permissions
- the browser - accessing a copyparty server using a web-browser
- tabs - the main tabs in the ui
- hotkeys - the browser has the following hotkeys
- navpane - switching between breadcrumbs or navpane
- thumbnails - press
g
to toggle grid-view instead of the file listing - zip downloads - download folders (or file selections) as
zip
ortar
files - uploading - drag files/folders into the web-browser to upload
- file-search - dropping files into the browser also lets you see if they exist on the server
- unpost - undo/delete accidental uploads
- file manager - cut/paste, rename, and delete files/folders (if you have permission)
- batch rename - select some files and press
F2
to bring up the rename UI - markdown viewer - and there are two editors
- other tricks
- searching - search by size, date, path/name, mp3-tags, ...
- server config - using arguments or config files, or a mix of both
- file indexing
- upload rules - set upload rules using volume flags
- compress uploads - files can be autocompressed on upload
- database location - in-volume (
.hist/up2k.db
, default) or somewhere else - metadata from audio files - set
-e2t
to index tags on upload - file parser plugins - provide custom parsers to index additional tags, also see ./bin/mtag/README.md
- upload events - trigger a script/program on each upload
- complete examples
- browser support - TLDR: yes
- client examples - interact with copyparty using non-browser clients
- up2k - quick outline of the up2k protocol, see uploading for the web-client
- why chunk-hashes - a single sha512 would be better, right?
- performance - defaults are usually fine - expect
8 GiB/s
download,1 GiB/s
upload- client-side - when uploading files
- security - some notes on hardening
- gotchas - behavior that might be unexpected
- recovering from crashes
- client crashes
- frefox wsod - firefox 87 can crash during uploads
- client crashes
- HTTP API
- dependencies - mandatory deps
- optional dependencies - install these to enable bonus features
- install recommended deps
- optional gpl stuff
- sfx - there are two self-contained "binaries"
- sfx repack - reduce the size of an sfx by removing features
- install on android
- reporting bugs - ideas for context to include in bug reports
- building
- todo - roughly sorted by priority
quickstart
download copyparty-sfx.py and you're all set!
running the sfx without arguments (for example doubleclicking it on Windows) will give everyone read/write access to the current folder; see -h
for help if you want accounts and volumes etc
some recommended options:
-e2dsa
enables general file indexing-e2ts
enables audio metadata indexing (needs either FFprobe or Mutagen), see optional dependencies-v /mnt/music:/music:r:rw,foo -a foo:bar
shares/mnt/music
as/music
,r
eadable by anyone, and read-write for userfoo
, passwordbar
- replace
:r:rw,foo
with:r,foo
to only make the folder readable byfoo
and nobody else - see accounts and volumes for the syntax and other permissions (
r
ead,w
rite,m
ove,d
elete,g
et)
- replace
--ls '**,*,ln,p,r'
to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
on servers
you may also want these, especially on servers:
- contrib/systemd/copyparty.service to run copyparty as a systemd service
- contrib/systemd/prisonparty.service to run it in a chroot (for extra security)
- contrib/nginx/copyparty.conf to reverse-proxy behind nginx (for better https)
on debian
recommended additional steps on debian which enable audio metadata and thumbnails (from images and videos):
-
as root, run the following:
apt install python3 python3-pip python3-dev ffmpeg
-
then, as the user which will be running copyparty (so hopefully not root), run this:
python3 -m pip install --user -U Pillow pillow-avif-plugin
(skipped pyheif-pillow-opener
because apparently debian is too old to build it)
notes
general notes:
- paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
- because no browsers currently implement the media-query to do this properly orz
browser-specific:
- iPhone/iPad: use Firefox to download files
- Android-Chrome: increase "parallel uploads" for higher speed (android bug)
- Android-Firefox: takes a while to select files (their fix for โ๏ธ)
- Desktop-Firefox:
may use gigabytes of RAM if your files are massiveseems to be OK now - Desktop-Firefox: may stop you from deleting files you've uploaded until you visit
about:memory
and clickMinimize memory usage
status
feature summary
- backend stuff
- โ sanic multipart parser
- โ multiprocessing (actual multithreading)
- โ volumes (mountpoints)
- โ accounts
- upload
- download
- โ single files in browser
- โ folders as zip / tar files
- โ FUSE client (read-only)
- browser
- โ navpane (directory tree sidebar)
- โ file manager (cut/paste, delete, batch-rename)
- โ audio player (with OS media controls and opus transcoding)
- โ image gallery with webm player
- โ textfile browser with syntax hilighting
- โ thumbnails
- โ ...of images using Pillow
- โ ...of videos using FFmpeg
- โ ...of audio (spectrograms) using FFmpeg
- โ cache eviction (max-age; maybe max-size eventually)
- โ SPA (browse while uploading)
- server indexing
- โ locate files by contents
- โ search by name/path/date/size
- โ search by ID3-tags etc.
- markdown
- โ viewer
- โ editor (sure why not)
testimonials
small collection of user feedback
good enough
, surprisingly correct
, certified good software
, just works
, why
motivations
project goals / philosophy
- inverse linux philosophy -- do all the things, and do an okay job
- quick drop-in service to get a lot of features in a pinch
- there are probably better alternatives if you have specific/long-term needs
- run anywhere, support everything
- as many web-browsers and python versions as possible
- every browser should at least be able to browse, download, upload files
- be a good emergency solution for transferring stuff between ancient boxes
- minimal dependencies
- but optional dependencies adding bonus-features are ok
- everything being plaintext makes it possible to proofread for malicious code
- no preparations / setup necessary, just run the sfx (which is also plaintext)
- as many web-browsers and python versions as possible
- adaptable, malleable, hackable
- no build steps; modify the js/python without needing node.js or anything like that
future plans
some improvement ideas
- the JS is a mess -- a preact rewrite would be nice
- preferably without build dependencies like webpack/babel/node.js, maybe a python thing to assemble js files into main.js
- good excuse to look at using virtual lists (browsers start to struggle when folders contain over 5000 files)
- the UX is a mess -- a proper design would be nice
- very organic (much like the python/js), everything was an afterthought
- true for both the layout and the visual flair
- something like the tron board-room ui (or most other hollywood ones, like ironman) would be :100:
- some of the python files are way too big
up2k.py
ended up doing all the file indexing / db managementhttpcli.py
should be separated into modules in general
bugs
- Windows: python 3.7 and older cannot read tags with FFprobe, so use Mutagen or upgrade
- Windows: python 2.7 cannot index non-ascii filenames with
-e2d
- Windows: python 2.7 cannot handle filenames with mojibake
--th-ff-jpg
may fix video thumbnails on some FFmpeg versions (macos, some linux)--th-ff-swr
may fix audio thumbnails on some FFmpeg versions
general bugs
- all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
- probably more, pls let me know
not my bugs
-
iPhones: the volume control doesn't work because apple doesn't want it to
- future workaround: enable the equalizer, make it all-zero, and set a negative boost to reduce the volume
- "future" because
AudioContext
is broken in the current iOS version (15.1), maybe one day...
- "future" because
- future workaround: enable the equalizer, make it all-zero, and set a negative boost to reduce the volume
-
Windows: folders cannot be accessed if the name ends with
.
- python or windows bug
-
Windows: msys2-python 3.8.6 occasionally throws
RuntimeError: release unlocked lock
when leaving a scoped mutex in up2k- this is an msys2 bug, the regular windows edition of python is fine
-
VirtualBox: sqlite throws
Disk I/O Error
when running in a VM and the up2k database is in a vboxsf- use
--hist
or thehist
volflag (-v [...]:c,hist=/tmp/foo
) to place the db inside the vm instead
- use
FAQ
"frequently" asked questions
-
is it possible to block read-access to folders unless you know the exact URL for a particular file inside?
- yes, using the
g
permission, see the examples there - you can also do this with linux filesystem permissions;
chmod 111 music
will make it possible to access files and folders inside themusic
folder but not list the immediate contents -- also works with other software, not just copyparty
- yes, using the
-
can I make copyparty download a file to my server if I give it a URL?
- not officially, but there is a terrible hack which makes it possible
accounts and volumes
per-folder, per-user permissions - if your setup is getting complex, consider making a config file instead of using arguments
- much easier to manage, and you can modify the config at runtime with
systemctl reload copyparty
or more conveniently using the[reload cfg]
button in the control-panel (if logged in as admin)
configuring accounts/volumes with arguments:
-a usr:pwd
adds accountusr
with passwordpwd
-v .::r
adds current-folder.
as the webroot,r
eadable by anyone- the syntax is
-v src:dst:perm:perm:...
so local-path, url-path, and one or more permissions to set - granting the same permissions to multiple accounts:
-v .::r,usr1,usr2:rw,usr3,usr4
= usr1/2 read-only, 3/4 read-write
- the syntax is
permissions:
r
(read): browse folder contents, download files, download as zip/tarw
(write): upload files, move files into this folderm
(move): move files/folders from this folderd
(delete): delete files/foldersg
(get): only download files, cannot see folder contents or zip/tar
examples:
- add accounts named u1, u2, u3 with passwords p1, p2, p3:
-a u1:p1 -a u2:p2 -a u3:p3
- make folder
/srv
the root of the filesystem, read-only by anyone:-v /srv::r
- make folder
/mnt/music
available at/music
, read-only for u1 and u2, read-write for u3:-v /mnt/music:music:r,u1,u2:rw,u3
- unauthorized users accessing the webroot can see that the
music
folder exists, but cannot open it
- unauthorized users accessing the webroot can see that the
- make folder
/mnt/incoming
available at/inc
, write-only for u1, read-move for u2:-v /mnt/incoming:inc:w,u1:rm,u2
- unauthorized users accessing the webroot can see that the
inc
folder exists, but cannot open it u1
can open theinc
folder, but cannot see the contents, only upload new files to itu2
can browse it and move files from/inc
into any folder whereu2
has write-access
- unauthorized users accessing the webroot can see that the
- make folder
/mnt/ss
available at/i
, read-write for u1, get-only for everyone else, and enable accesskeys:-v /mnt/ss:i:rw,u1:g:c,fk=4
c,fk=4
sets thefk
volume-flag to 4, meaning each file gets a 4-character accesskeyu1
can upload files, browse the folder, and see the generated accesskeys- other users cannot browse the folder, but can access the files if they have the full file URL with the accesskey
the browser
accessing a copyparty server using a web-browser
tabs
the main tabs in the ui
[๐]
search by size, date, path/name, mp3-tags ...[๐งฏ]
unpost: undo/delete accidental uploads[๐]
and[๐]
are the uploaders[๐]
mkdir: create directories[๐]
new-md: create a new markdown document[๐]
send-msg: either to server-log or into textfiles if--urlform save
[๐บ]
audio-player config options[โ๏ธ]
general client config options
hotkeys
the browser has the following hotkeys (always qwerty)
B
toggle breadcrumbs / navpaneI/K
prev/next folderM
parent folder (or unexpand current)V
toggle folders / textfiles in the navpaneG
toggle list / grid viewT
toggle thumbnails / iconsESC
close various thingsctrl-X
cut selected files/foldersctrl-V
pasteF2
rename selected file/folder- when a file/folder is selected (in not-grid-view):
Up/Down
move cursor- shift+
Up/Down
select and move cursor - ctrl+
Up/Down
move cursor and scroll viewport Space
toggle file selectionCtrl-A
toggle select all
- when a textfile is open:
I/K
prev/next textfileS
toggle selection of open fileM
close textfile
- when playing audio:
J/L
prev/next songU/O
skip 10sec back/forward0..9
jump to 0%..90%P
play/pause (also starts playing the folder)
- when viewing images / playing videos:
J/L, Left/Right
prev/next fileHome/End
first/last fileS
toggle selectionR
rotate clockwise (shift=ccw)Esc
close viewer- videos:
U/O
skip 10sec back/forwardP/K/Space
play/pauseF
fullscreenC
continue playing next videoV
loopM
mute
- when the navpane is open:
A/D
adjust tree width
- in the grid view:
S
toggle multiselect- shift+
A/D
zoom
- in the markdown editor:
^s
save^h
header^k
autoformat table^u
jump to next unicode character^e
toggle editor / preview^up, ^down
jump paragraphs
navpane
switching between breadcrumbs or navpane
click the ๐ฒ
or pressing the B
hotkey to toggle between breadcrumbs path (default), or a navpane (tree-browser sidebar thing)
[+]
and[-]
(or hotkeysA
/D
) adjust the size[๐ฏ]
jumps to the currently open folder[๐]
toggles between showing folders and textfiles[๐]
shows the name of all parent folders in a docked panel[a]
toggles automatic widening as you go deeper[โต]
toggles wordwrap[๐]
show full name on hover (if wordwrap is off)
thumbnails
press g
to toggle grid-view instead of the file listing, and t
toggles icons / thumbnails
it does static images with Pillow and uses FFmpeg for video files, so you may want to --no-thumb
or maybe just --no-vthumb
depending on how dangerous your users are
audio files are covnerted into spectrograms using FFmpeg unless you --no-athumb
(and some FFmpeg builds may need --th-ff-swr
)
images with the following names (see --th-covers
) become the thumbnail of the folder they're in: folder.png
, folder.jpg
, cover.png
, cover.jpg
in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
- indicated by the audio files having the โถ icon instead of ๐พ
zip downloads
download folders (or file selections) as zip
or tar
files
select which type of archive you want in the [โ๏ธ] config
tab:
name | url-suffix | description |
---|---|---|
tar |
?tar |
plain gnutar, works great with curl | tar -xv |
zip |
?zip=utf8 |
works everywhere, glitchy filenames on win7 and older |
zip_dos |
?zip |
traditional cp437 (no unicode) to fix glitchy filenames |
zip_crc |
?zip=crc |
cp437 with crc32 computed early for truly ancient software |
- hidden files (dotfiles) are excluded unless
-ed
up2k.db
anddir.txt
is always excluded
zip_crc
will take longer to download since the server has to read each file twice- this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
- how are you accessing copyparty actually
- this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right
uploading
drag files/folders into the web-browser to upload
this initiates an upload using up2k
; there are two uploaders available:
[๐] bup
, the basic uploader, supports almost every browser since netscape 4.0[๐] up2k
, the fancy one
you can also undo/delete uploads by using [๐งฏ]
unpost
up2k has several advantages:
- you can drop folders into the browser (files are added recursively)
- files are processed in chunks, and each chunk is checksummed
- uploads autoresume if they are interrupted by network issues
- uploads resume if you reboot your browser or pc, just upload the same files again
- server detects any corruption; the client reuploads affected chunks
- the client doesn't upload anything that already exists on the server
- much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
- the last-modified timestamp of the file is preserved
see up2k for details on how it works
protip: you can avoid scaring away users with contrib/plugins/minimal-up2k.html which makes it look much simpler
protip: if you enable favicon
in the [โ๏ธ] settings
tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress
the up2k UI is the epitome of polished inutitive experiences:
- "parallel uploads" specifies how many chunks to upload at the same time
[๐]
analysis of other files should continue while one is uploading[๐ญ]
ask for confirmation before files are added to the queue[๐ค]
sync uploading between other copyparty browser-tabs so only one is active[๐]
switch between upload and file-search mode- ignore
[๐]
if you add files by dragging them into the browser
- ignore
and then theres the tabs below it,
[ok]
is the files which completed successfully[ng]
is the ones that failed / got rejected (already exists, ...)[done]
shows a combined list of[ok]
and[ng]
, chronological order[busy]
files which are currently hashing, pending-upload, or uploading- plus up to 3 entries each from
[done]
and[que]
for context
- plus up to 3 entries each from
[que]
is all the files that are still queued
note that since up2k has to read each file twice, [๐ bup]
can theoretically be up to 2x faster in some extreme cases (files bigger than your ram, combined with an internet connection faster than the read-speed of your HDD, or if you're uploading from a cuo2duo)
if you are resuming a massive upload and want to skip hashing the files which already finished, you can enable turbo
in the [โ๏ธ] config
tab, but please read the tooltip on that button
file-search
dropping files into the browser also lets you see if they exist on the server
when you drag/drop files into the browser, you will see two dropzones: Upload
and Search
on a phone? toggle the
[๐]
switch green before tapping the big yellow Search button to select your files
the files will be hashed on the client-side, and each hash is sent to the server, which checks if that file exists somewhere
files go into [ok]
if they exist (and you get a link to where it is), otherwise they land in [ng]
- the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much
unpost
undo/delete accidental uploads
you can unpost even if you don't have regular move/delete access, however only for files uploaded within the past --unpost
seconds (default 12 hours) and the server must be running with -e2d
file manager
cut/paste, rename, and delete files/folders (if you have permission)
file selection: click somewhere on the line (not the link itsef), then:
-
space
to toggle -
up/down
to move -
shift-up/down
to move-and-select -
ctrl-shift-up/down
to also scroll -
cut: select some files and
ctrl-x
-
paste:
ctrl-v
in another folder -
rename:
F2
you can move files across browser tabs (cut in one tab, paste in another)
batch rename
select some files and press F2
to bring up the rename UI
quick explanation of the buttons,
[โ apply rename]
confirms and begins renaming[โ cancel]
aborts and closes the rename window[โบ reset]
reverts any filename changes back to the original name[decode]
does a URL-decode on the filename, fixing stuff like&
and%20
[advanced]
toggles advanced mode
advanced mode: rename files based on rules to decide the new names, based on the original name (regex), or based on the tags collected from the file (artist/title/...), or a mix of both
in advanced mode,
[case]
toggles case-sensitive regexregex
is the regex pattern to apply to the original filename; any files which don't match will be skippedformat
is the new filename, taking values from regex capturing groups and/or from file tags- very loosely based on foobar2000 syntax
presets
lets you save rename rules for later
available functions:
$lpad(text, length, pad_char)
$rpad(text, length, pad_char)
so,
say you have a file named meganeko - Eclipse - 07 Sirius A.mp3
(absolutely fantastic album btw) and the tags are: Album:Eclipse
, Artist:meganeko
, Title:Sirius A
, tn:7
you could use just regex to rename it:
regex
=(.*) - (.*) - ([0-9]{2}) (.*)
format
=(3). (1) - (4)
output
=07. meganeko - Sirius A.mp3
or you could use just tags:
format
=$lpad((tn),2,0). (artist) - (title).(ext)
output
=7. meganeko - Sirius A.mp3
or a mix of both:
regex
=- ([0-9]{2})
format
=(1). (artist) - (title).(ext)
output
=07. meganeko - Sirius A.mp3
the metadata keys you can use in the format field are the ones in the file-browser table header (whatever is collected with -mte
and -mtp
)
markdown viewer
and there are two editors
- the document preview has a max-width which is the same as an A4 paper when printed
other tricks
-
you can link a particular timestamp in an audio file by adding it to the URL, such as
&20
/&20s
/&1m20
/&t=1:20
after the.../#af-c8960dab
-
enabling the audio equalizer can help make gapless albums fully gapless in some browsers (chrome), so consider leaving it on with all the values at zero
-
get a plaintext file listing by adding
?ls=t
to a URL, or a compact colored one with?ls=v
(for unix terminals) -
if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
-
click the bottom-left
ฯ
to open a javascript prompt for debugging -
files named
.prologue.html
/.epilogue.html
will be rendered before/after directory listings unless--no-logues
-
files named
README.md
/readme.md
will be rendered after directory listings unless--no-readme
(but.epilogue.html
takes precedence)
searching
search by size, date, path/name, mp3-tags, ...
when started with -e2dsa
copyparty will scan/index all your files. This avoids duplicates on upload, and also makes the volumes searchable through the web-ui:
- make search queries by
size
/date
/directory-path
/filename
, or... - drag/drop a local file to see if the same contents exist somewhere on the server, see file-search
path/name queries are space-separated, AND'ed together, and words are negated with a -
prefix, so for example:
- path:
shibayan -bossa
finds all files where one of the folders containshibayan
but filters out any results wherebossa
exists somewhere in the path - name:
demetori styx
gives you good stuff
add the argument -e2ts
to also scan/index tags from music files, which brings us over to:
server config
using arguments or config files, or a mix of both:
- config files (
-c some.conf
) can set additional commandline arguments; see ./docs/example.conf kill -s USR1
(same assystemctl reload copyparty
) to reload accounts and volumes from config files without restarting- or click the
[reload cfg]
button in the control-panel when logged in as admin
- or click the
file indexing
file indexing relies on two database tables, the up2k filetree (-e2d
) and the metadata tags (-e2t
), stored in .hist/up2k.db
. Configuration can be done through arguments, volume flags, or a mix of both.
through arguments:
-e2d
enables file indexing on upload-e2ds
also scans writable folders for new files on startup-e2dsa
also scans all mounted volumes (including readonly ones)-e2t
enables metadata indexing on upload-e2ts
also scans for tags in all files that don't have tags yet-e2tsr
also deletes all existing tags, doing a full reindex
the same arguments can be set as volume flags, in addition to d2d
, d2ds
, d2t
, d2ts
for disabling:
-v ~/music::r:c,e2dsa,e2tsr
does a full reindex of everything on startup-v ~/music::r:c,d2d
disables all indexing, even if any-e2*
are on-v ~/music::r:c,d2t
disables all-e2t*
(tags), does not affect-e2d*
-v ~/music::r:c,d2ds
disables on-boot scans; only index new uploads-v ~/music::r:c,d2ts
same except only affecting tags
note:
- the parser can finally handle
c,e2dsa,e2tsr
so you no longer have toc,e2dsa:c,e2tsr
e2tsr
is probably always overkill, sincee2ds
/e2dsa
would pick up any file modifications ande2ts
would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise- the rescan button in the admin panel has no effect unless the volume has
-e2ds
or higher
to save some time, you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting --no-hash \.iso$
or the volume-flag :c,nohash=\.iso$
, this has the following consequences:
- initial indexing is way faster, especially when the volume is on a network disk
- makes it impossible to file-search
- if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected
similarly, you can fully ignore files/folders using --no-idx [...]
and :c,noidx=\.iso$
if you set --no-hash [...]
globally, you can enable hashing for specific volumes using flag :c,nohash=
upload rules
set upload rules using volume flags, some examples:
:c,sz=1k-3m
sets allowed filesize between 1 KiB and 3 MiB inclusive (suffixes:b
,k
,m
,g
):c,nosub
disallow uploading into subdirectories; goes well withrotn
androtf
::c,rotn=1000,2
moves uploads into subfolders, up to 1000 files in each folder before making a new one, two levels deep (must be at least 1):c,rotf=%Y/%m/%d/%H
enforces files to be uploaded into a structure of subfolders according to that date format- if someone uploads to
/foo/bar
the path would be rewritten to/foo/bar/2021/08/06/23
for example - but the actual value is not verified, just the structure, so the uploader can choose any values which conform to the format string
- just to avoid additional complexity in up2k which is enough of a mess already
- if someone uploads to
:c,lifetime=300
delete uploaded files when they become 5 minutes old
you can also set transaction limits which apply per-IP and per-volume, but these assume -j 1
(default) otherwise the limits will be off, for example -j 4
would allow anywhere between 1x and 4x the limits you set depending on which processing node the client gets routed to
:c,maxn=250,3600
allows 250 files over 1 hour from each IP (tracked per-volume):c,maxb=1g,300
allows 1 GiB total over 5 minutes from each IP (tracked per-volume)
compress uploads
files can be autocompressed on upload, either on user-request (if config allows) or forced by server-config
- volume flag
gz
allows gz compression - volume flag
xz
allows lzma compression - volume flag
pk
forces compression on all files - url parameter
pk
requests compression with server-default algorithm - url parameter
gz
orxz
requests compression with a specific algorithm - url parameter
xz
requests xz compression
things to note,
- the
gz
andxz
arguments take a single optional argument, the compression level (range 0 to 9) - the
pk
volume flag takes the optional argumentALGORITHM,LEVEL
which will then be forced for all uploads, for examplegz,9
orxz,0
- default compression is gzip level 9
- all upload methods except up2k are supported
- the files will be indexed after compression, so dupe-detection and file-search will not work as expected
some examples,
-v inc:inc:w:c,pk=xz,0
folder named inc, shared at inc, write-only for everyone, forces xz compression at level 0-v inc:inc:w:c,pk
same write-only inc, but forces gz compression (default) instead of xz-v inc:inc:w:c,gz
allows (but does not force) gz compression if client uploads to/inc?pk
or/inc?gz
or/inc?gz=4
database location
in-volume (.hist/up2k.db
, default) or somewhere else
copyparty creates a subfolder named .hist
inside each volume where it stores the database, thumbnails, and some other stuff
this can instead be kept in a single place using the --hist
argument, or the hist=
volume flag, or a mix of both:
--hist ~/.cache/copyparty -v ~/music::r:c,hist=-
sets~/.cache/copyparty
as the default place to put volume info, but~/music
gets the regular.hist
subfolder (-
restores default behavior)
note:
- markdown edits are always stored in a local
.hist
subdirectory - on windows the volflag path is cyglike, so
/c/temp
meansC:\temp
but use regular paths for--hist
- you can use cygpaths for volumes too,
-v C:\Users::r
and-v /c/users::r
both work
- you can use cygpaths for volumes too,
metadata from audio files
set -e2t
to index tags on upload
-mte
decides which tags to index and display in the browser (and also the display order), this can be changed per-volume:
-v ~/music::r:c,mte=title,artist
indexes and displays title followed by artist
if you add/remove a tag from mte
you will need to run with -e2tsr
once to rebuild the database, otherwise only new files will be affected
but instead of using -mte
, -mth
is a better way to hide tags in the browser: these tags will not be displayed by default, but they still get indexed and become searchable, and users can choose to unhide them in the [โ๏ธ] config
pane
-mtm
can be used to add or redefine a metadata mapping, say you have media files with foo
and bar
tags and you want them to display as qux
in the browser (preferring foo
if both are present), then do -mtm qux=foo,bar
and now you can -mte artist,title,qux
tags that start with a .
such as .bpm
and .dur
(ation) indicate numeric value
see the beautiful mess of a dictionary in mtag.py for the default mappings (should cover mp3,opus,flac,m4a,wav,aif,)
--no-mutagen
disables Mutagen and uses FFprobe instead, which...
- is about 20x slower than Mutagen
- catches a few tags that Mutagen doesn't
- melodic key, video resolution, framerate, pixfmt
- avoids pulling any GPL code into copyparty
- more importantly runs FFprobe on incoming files which is bad if your FFmpeg has a cve
file parser plugins
provide custom parsers to index additional tags, also see ./bin/mtag/README.md
copyparty can invoke external programs to collect additional metadata for files using mtp
(either as argument or volume flag), there is a default timeout of 30sec
-mtp .bpm=~/bin/audio-bpm.py
will execute~/bin/audio-bpm.py
with the audio file as argument 1 to provide the.bpm
tag, if that does not exist in the audio metadata-mtp key=f,t5,~/bin/audio-key.py
uses~/bin/audio-key.py
to get thekey
tag, replacing any existing metadata tag (f,
), aborting if it takes longer than 5sec (t5,
)-v ~/music::r:c,mtp=.bpm=~/bin/audio-bpm.py:c,mtp=key=f,t5,~/bin/audio-key.py
both as a per-volume config wow this is getting ugly
but wait, there's more! -mtp
can be used for non-audio files as well using the a
flag: ay
only do audio files, an
only do non-audio files, or ad
do all files (d as in dontcare)
-mtp ext=an,~/bin/file-ext.py
runs~/bin/file-ext.py
to get theext
tag only if file is not audio (an
)-mtp arch,built,ver,orig=an,eexe,edll,~/bin/exe.py
runs~/bin/exe.py
to get properties about windows-binaries only if file is not audio (an
) and file extension is exe or dll
upload events
trigger a script/program on each upload like so:
-v /mnt/inc:inc:w:c,mte=+a1:c,mtp=a1=ad,/usr/bin/notify-send
so filesystem location /mnt/inc
shared at /inc
, write-only for everyone, appending a1
to the list of tags to index, and using /usr/bin/notify-send
to "provide" that tag
that'll run the command notify-send
with the path to the uploaded file as the first and only argument (so on linux it'll show a notification on-screen)
note that it will only trigger on new unique files, not dupes
and it will occupy the parsing threads, so fork anything expensive, or if you want to intentionally queue/singlethread you can combine it with --mtag-mt 1
if this becomes popular maybe there should be a less janky way to do it actually
complete examples
- read-only music server with bpm and key scanning
python copyparty-sfx.py -v /mnt/nas/music:/music:r -e2dsa -e2ts -mtp .bpm=f,audio-bpm.py -mtp key=f,audio-key.py
browser support
TLDR: yes
ie
= internet-explorer, ff
= firefox, c
= chrome, iOS
= iPhone/iPad, Andr
= Android
feature | ie6 | ie9 | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
---|---|---|---|---|---|---|---|---|
browse files | yep | yep | yep | yep | yep | yep | yep | yep |
thumbnail view | - | yep | yep | yep | yep | yep | yep | yep |
basic uploader | yep | yep | yep | yep | yep | yep | yep | yep |
up2k | - | - | *1 |
*1 |
yep | yep | yep | yep |
make directory | yep | yep | yep | yep | yep | yep | yep | yep |
send message | yep | yep | yep | yep | yep | yep | yep | yep |
set sort order | - | yep | yep | yep | yep | yep | yep | yep |
zip selection | - | yep | yep | yep | yep | yep | yep | yep |
file rename | - | yep | yep | yep | yep | yep | yep | yep |
file cut/paste | - | yep | yep | yep | yep | yep | yep | yep |
navpane | - | yep | yep | yep | yep | yep | yep | yep |
image viewer | - | yep | yep | yep | yep | yep | yep | yep |
video player | - | yep | yep | yep | yep | yep | yep | yep |
markdown editor | - | - | yep | yep | yep | yep | yep | yep |
markdown viewer | - | yep | yep | yep | yep | yep | yep | yep |
play mp3/m4a | - | yep | yep | yep | yep | yep | yep | yep |
play ogg/opus | - | - | - | - | yep | yep | *3 |
yep |
= feature = | ie6 | ie9 | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
- internet explorer 6 to 8 behave the same
- firefox 52 and chrome 49 are the final winxp versions
*1
yes, but extremely slow (ie10:1 MiB/s
, ie11:270 KiB/s
)*3
iOS 11 and newer, opus only, and requires FFmpeg on the server
quick summary of more eccentric web-browsers trying to view a directory index:
browser | will it blend |
---|---|
links (2.21/macports) | can browse, login, upload/mkdir/msg |
lynx (2.8.9/macports) | can browse, login, upload/mkdir/msg |
w3m (0.5.3/macports) | can browse, login, upload at 100kB/s, mkdir/msg |
netsurf (3.10/arch) | is basically ie6 with much better css (javascript has almost no effect) |
opera (11.60/winxp) | OK: thumbnails, image-viewer, zip-selection, rename/cut/paste. NG: up2k, navpane, markdown, audio |
ie4 and netscape 4.0 | can browse, upload with ?b=u |
SerenityOS (7e98457) | hits a page fault, works with ?b=u , file upload not-impl |
client examples
interact with copyparty using non-browser clients
-
javascript: dump some state into a file (two separate examples)
await fetch('//127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});
var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');
-
curl/wget: upload some files (post=file, chunk=stdin)
post(){ curl -b cppwd=wark -F act=bput -F f=@"$1" http://127.0.0.1:3923/;}
post movie.mkv
post(){ wget --header='Cookie: cppwd=wark' --post-file="$1" -O- http://127.0.0.1:3923/?raw;}
post movie.mkv
chunk(){ curl -b cppwd=wark -T- http://127.0.0.1:3923/;}
chunk <movie.mkv
-
bash: when curl and wget is not available or too boring
(printf 'PUT /junk?pw=wark HTTP/1.1\r\n\r\n'; cat movie.mkv) | nc 127.0.0.1 3923
(printf 'PUT / HTTP/1.1\r\n\r\n'; cat movie.mkv) >/dev/tcp/127.0.0.1/3923
-
python: up2k.py is a command-line up2k client (webm)
- file uploads, file-search, autoresume of aborted/broken uploads
- see ./bin/README.md#up2kpy
-
FUSE: mount a copyparty server as a local filesystem
- cross-platform python client available in ./bin/
- rclone as client can give ~5x performance, see ./docs/rclone.md
-
sharex (screenshot utility): see ./contrib/sharex.sxcu
copyparty returns a truncated sha512sum of your PUT/POST as base64; you can generate the same checksum locally to verify uplaods:
b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
b512 <movie.mkv
you can provide passwords using cookie cppwd=hunter2
, as a url query ?pw=hunter2
, or with basic-authentication (either as the username or password)
up2k
quick outline of the up2k protocol, see uploading for the web-client
- the up2k client splits a file into an "optimal" number of chunks
- 1 MiB each, unless that becomes more than 256 chunks
- tries 1.5M, 2M, 3, 4, 6, ... until <= 256 chunks or size >= 32M
- client posts the list of hashes, filename, size, last-modified
- server creates the
wark
, an identifier for this uploadsha512( salt + filesize + chunk_hashes )
- and a sparse file is created for the chunks to drop into
- client uploads each chunk
- header entries for the chunk-hash and wark
- server writes chunks into place based on the hash
- client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload
up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would probably have noticed as well (thanks to tls also functioning as an integrity check)
why chunk-hashes
a single sha512 would be better, right?
this is due to crypto.subtle
not providing a streaming api (or the option to seed the sha512 hasher with a starting hash)
as a result, the hashes are much less useful than they could have been (search the server by sha512, provide the sha512 in the response http headers, ...)
hashwasm would solve the streaming issue but reduces hashing speed for sha512 (xxh128 does 6 GiB/s), and it would make old browsers and iphones unsupported
performance
defaults are usually fine - expect 8 GiB/s
download, 1 GiB/s
upload
below are some tweaks roughly ordered by usefulness:
-
-q
disables logging and can help a bunch, even when combined with-lo
to redirect logs to file -
--http-only
or--https-only
(unless you want to support both protocols) will reduce the delay before a new connection is established -
--hist
pointing to a fast location (ssd) will make directory listings and searches faster when-e2d
or-e2t
is set -
--no-hash .
when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable -
-j
enables multiprocessing (actual multithreading) and can make copyparty perform better in cpu-intensive workloads, for example:- huge amount of short-lived connections
- really heavy traffic (downloads/uploads)
...however it adds an overhead to internal communication so it might be a net loss, see if it works 4 u
client-side
when uploading files,
-
chrome is recommended, at least compared to firefox:
- up to 90% faster when hashing, especially on SSDs
- up to 40% faster when uploading over extremely fast internets
- but up2k.py can be 40% faster than chrome again
-
if you're cpu-bottlenecked, or the browser is maxing a cpu core:
- up to 30% faster uploads if you hide the upload status list by switching away from the
[๐]
up2k ui-tab (or closing it)- switching to another browser-tab also works, the favicon will update every 10 seconds in that case
- unlikely to be a problem, but can happen when uploding many small files, or your internet is too fast, or PC too slow
- up to 30% faster uploads if you hide the upload status list by switching away from the
security
some notes on hardening
on public copyparty instances with anonymous upload enabled:
- users can upload html/css/js which will evaluate for other visitors in a few ways,
- unless
--no-readme
is set: by uploading/modifying a file namedreadme.md
- if
move
access is granted AND none of--no-logues
,--no-dot-mv
,--no-dot-ren
is set: by uploading some .html file and renaming it to.epilogue.html
(uploading it directly is blocked)
- unless
other misc:
- you can disable directory listings by giving permission
g
instead ofr
, only accepting direct URLs to files- combine this with volume-flag
c,fk
to generate per-file accesskeys; users which have full read-access will then see URLs with?k=...
appended to the end, andg
users must provide that URL including the correct key to avoid a 404
- combine this with volume-flag
gotchas
behavior that might be unexpected
- users without read-access to a folder can still see the
.prologue.html
/.epilogue.html
/README.md
contents, for the purpose of showing a description on how to use the uploader for example
recovering from crashes
client crashes
frefox wsod
firefox 87 can crash during uploads -- the entire browser goes, including all other browser tabs, everything turns white
however you can hit F12
in the up2k tab and use the devtools to see how far you got in the uploads:
-
get a complete list of all uploads, organized by statuts (ok / no-good / busy / queued):
var tabs = { ok:[], ng:[], bz:[], q:[] }; for (var a of up2k.ui.tab) tabs[a.in].push(a); tabs
-
list of filenames which failed:
โvar ng = []; for (var a of up2k.ui.tab) if (a.in != 'ok') ng.push(a.hn.split('<a href=\"').slice(-1)[0].split('\">')[0]); ng
-
send the list of filenames to copyparty for safekeeping:
await fetch('/inc', {method:'PUT', body:JSON.stringify(ng,null,1)})
HTTP API
- table-column
params
= URL parameters;?foo=bar&qux=...
- table-column
body
= POST payload - method
jPOST
= json post - method
mPOST
= multipart post - method
uPOST
= url-encoded post FILE
= conventional HTTP file upload entry (rfc1867 et al, filename inContent-Disposition
)
authenticate using header Cookie: cppwd=foo
or url param &pw=foo
read
method | params | result |
---|---|---|
GET | ?ls |
list files/folders at URL as JSON |
GET | ?ls&dots |
list files/folders at URL as JSON, including dotfiles |
GET | ?ls=t |
list files/folders at URL as plaintext |
GET | ?ls=v |
list files/folders at URL, terminal-formatted |
GET | ?b |
list files/folders at URL as simplified HTML |
GET | ?tree=. |
list one level of subdirectories inside URL |
GET | ?tree |
list one level of subdirectories for each level until URL |
GET | ?tar |
download everything below URL as a tar file |
GET | ?zip=utf-8 |
download everything below URL as a zip file |
GET | ?ups |
show recent uploads from your IP |
GET | ?ups&filter=f |
...where URL contains f |
GET | ?mime=foo |
specify return mimetype foo |
GET | ?raw |
get markdown file at URL as plaintext |
GET | ?txt |
get file at URL as plaintext |
GET | ?txt=iso-8859-1 |
...with specific charset |
GET | ?th |
get image/video at URL as thumbnail |
GET | ?th=opus |
convert audio file to 128kbps opus |
GET | ?th=caf |
...in the iOS-proprietary container |
method | body | result |
---|---|---|
jPOST | {"q":"foo"} |
do a server-wide search; see the [๐] search tab raw field for syntax |
method | params | body | result |
---|---|---|---|
jPOST | ?tar |
["foo","bar"] |
download folders foo and bar inside URL as a tar file |
write
method | params | result |
---|---|---|
GET | ?move=/foo/bar |
move/rename the file/folder at URL to /foo/bar |
method | params | body | result |
---|---|---|---|
PUT | (binary data) | upload into file at URL | |
PUT | ?gz |
(binary data) | compress with gzip and write into file at URL |
PUT | ?xz |
(binary data) | compress with xz and write into file at URL |
mPOST | act=bput , f=FILE |
upload FILE into the folder at URL |
|
mPOST | ?j |
act=bput , f=FILE |
...and reply with json |
mPOST | act=mkdir , name=foo |
create directory foo at URL |
|
GET | ?delete |
delete URL recursively | |
jPOST | ?delete |
["/foo","/bar"] |
delete /foo and /bar recursively |
uPOST | msg=foo |
send message foo into server log |
|
mPOST | act=tput , body=TEXT |
overwrite markdown document at URL |
server behavior of msg
can be reconfigured with --urlform
admin
method | params | result |
---|---|---|
GET | ?reload=cfg |
reload config files and rescan volumes |
GET | ?scan |
initiate a rescan of the volume which provides URL |
GET | ?stack |
show a stacktrace of all threads |
general
method | params | result |
---|---|---|
GET | ?pw=x |
logout |
dependencies
mandatory deps:
jinja2
(is built into the SFX)
optional dependencies
install these to enable bonus features
enable music tags:
- either
mutagen
(fast, pure-python, skips a few tags, makes copyparty GPL? idk) - or
ffprobe
(20x slower, more accurate, possibly dangerous depending on your distro and users)
enable thumbnails of...
- images:
Pillow
(requires py2.7 or py3.5+) - videos/audio:
ffmpeg
andffprobe
somewhere in$PATH
- HEIF pictures:
pyheif-pillow-opener
(requires Linux or a C compiler) - AVIF pictures:
pillow-avif-plugin
install recommended deps
python -m pip install --user -U jinja2 mutagen Pillow
optional gpl stuff
some bundled tools have copyleft dependencies, see ./bin/#mtag
these are standalone programs and will never be imported / evaluated by copyparty, and must be enabled through -mtp
configs
sfx
there are two self-contained "binaries":
- copyparty-sfx.py -- pure python, works everywhere, recommended
- copyparty-sfx.sh -- smaller, but only for linux and macos, kinda deprecated
launch either of them (use sfx.py on systemd) and it'll unpack and run copyparty, assuming you have python installed of course
pls note that copyparty-sfx.sh
will fail if you rename copyparty-sfx.py
to copyparty.py
and keep it in the same folder because sys.path
is funky
sfx repack
reduce the size of an sfx by removing features
if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL)
393k
size of original sfx.py as of v1.1.3310k
after./scripts/make-sfx.sh re no-cm
269k
after./scripts/make-sfx.sh re no-cm no-hl
the features you can opt to drop are
cm
/easymde, the "fancy" markdown editor, saves ~82khl
, prism, the syntax hilighter, saves ~41kfnt
, source-code-pro, the monospace font, saves ~9kdd
, the custom mouse cursor for the media player tray tab, saves ~2k
for the re
pack to work, first run one of the sfx'es once to unpack it
note: you can also just download and run scripts/copyparty-repack.sh -- this will grab the latest copyparty release from github and do a few repacks; works on linux/macos (and windows with msys2 or WSL)
install on android
install Termux (see ocv.me/termux) and then copy-paste this into Termux (long-tap) all at once:
apt update && apt -y full-upgrade && termux-setup-storage && apt -y install python && python -m ensurepip && python -m pip install -U copyparty
echo $?
after the initial setup, you can launch copyparty at any time by running copyparty
anywhere in Termux
reporting bugs
ideas for context to include in bug reports
if something broke during an upload (replacing FILENAME with a part of the filename that broke):
journalctl -aS '48 hour ago' -u copyparty | grep -C10 FILENAME | tee bug.log
building
dev env setup
mostly optional; if you need a working env for vscode or similar
python3 -m venv .venv
. .venv/bin/activate
pip install jinja2 # mandatory
pip install mutagen # audio metadata
pip install Pillow pyheif-pillow-opener pillow-avif-plugin # thumbnails
pip install black bandit pylint flake8 # vscode tooling
just the sfx
first grab the web-dependencies from a previous sfx (assuming you don't need to modify something in those):
rm -rf copyparty/web/deps
curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
python3 x.py -h
rm x.py
mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/
then build the sfx using any of the following examples:
./scripts/make-sfx.sh # both python and sh editions
./scripts/make-sfx.sh no-sh gz # just python with gzip
complete release
also builds the sfx so skip the sfx section above
in the scripts
folder:
- run
make -C deps-docker
to build all dependencies git tag v1.2.3 && git push origin --tags
- upload to pypi with
make-pypi-release.(sh|bat)
- create github release with
make-tgz-release.sh
- create sfx with
make-sfx.sh
todo
roughly sorted by priority
- nothing! currently
discarded ideas
- reduce up2k roundtrips
- start from a chunk index and just go
- terminate client on bad data
- not worth the effort, just throw enough conncetions at it
- single sha512 across all up2k chunks?
- crypto.subtle cannot into streaming, would have to use hashwasm, expensive
- separate sqlite table per tag
- performance fixed by skipping some indexes (
+mt.k
)
- performance fixed by skipping some indexes (
- audio fingerprinting
- only makes sense if there can be a wasm client and that doesn't exist yet (except for olaf which is agpl hence counts as not existing)
os.copy_file_range
for up2k cloning- almost never hit this path anyways
- up2k partials ui
- feels like there isn't much point
- cache sha512 chunks on client
- too dangerous
- comment field
- nah
- look into android thumbnail cache file format
- absolutely not
- indexedDB for hashes, cfg enable/clear/sz, 2gb avail, ~9k for 1g, ~4k for 100m, 500k items before autoeviction
- blank hashlist when up-ok to skip handshake
- too many confusing side-effects
- blank hashlist when up-ok to skip handshake
- hls framework for Someone Else to drop code into :^)
- probably not, too much stuff to consider -- seeking, start at offset, task stitching (probably np-hard), conditional passthru, rate-control (especially multi-consumer), session keepalive, cache mgmt...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for copyparty-1.1.11-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 342ab13182dd9c74982528d29ed9a9d60da600c038d4355dba7e7a2a0cc9453e |
|
MD5 | 22ab032fe58f07071871d475293f6d4a |
|
BLAKE2b-256 | 31e45c0acf25ede4af8949d851af88dc4ba1f177b96305c161d3b70c078d4bfb |