No project description provided
Project description
oarepo-upload-cli
Package that synchronizes documents between the student system and repository up to some date.
CLI Usage
To use the upload CLI tool, you first have to install the package somewhere.
Installing upload CLI in a separate virtualenv
Create a separate virtualenv and install upload CLI into it:
python3.10 -m venv .venv-upload-cli
(source .venv-upload-cli/bin/activate; pip install -U pip setuptools wheel; pip install oarepo-upload-cli)
Configuration
Ini file
In order for the configuration file to be parsed correctly, create the file following these rules:
- name -
~/.repository-uploader.ini
(or anywhere else using --config param) - content template
[authentication] token = enter-token-here [repository] collection_url = url_of_the_collection # use dot notation to select the metadata field. Note that this should not be the repository's # modified but a different field that will reflect the modified attribute of your source record record_modified_field = metadata.dateModified file_modified_field = metadata.dateModified # field inside the record that holds the id from the source id_query_field = metadata.originalId # to be able to upload only non-modified records, provide a max(..) aggregation # inside the repository and set the path within the aggs element here last_modification_date_agg = dateModifiedAgg.value [entrypoints] # name of the entrypoint inside oarepo_upload_cli.dependencies # that gives implementation of RecordSource source = # name of the entrypoint inside oarepo_upload_cli.dependencies that gives implementation # of RepositoryClient. In most cases, not need to specify it repository =
Environment variables
Values in the configuration can be overriden by these environment variables:
REPOSITORY_UPLOADER_BEARER_TOKEN
REPOSITORY_UPLOADER_COLLECTION_URL
REPOSITORY_UPLOADER_FILE_MODIFIED_FIELD_NAME
REPOSITORY_UPLOADER_RECORD_MODIFIED_FIELD_NAME
REPOSITORY_UPLOADER_ID_QUERY_FIELD
REPOSITORY_UPLOADER_LAST_MODIFICATION_DATE_AGG
REPOSITORY_UPLOADER_SOURCE
REPOSITORY_UPLOADER_REPOSITORY
Command-line options
Commandline options take the highest priority:
oarepo_upload
--config config-file-location
--token bearer_token
--collection-url collection-url
--file-modified-field file-modified-field
--record-modified-field record-modified-field
--source source-entrypoint
--repository repository-entrypoint
The following options handle which records should be uploaded:
--modified_after
- Timestamp that represents date after modification. If not specified, the last updated timestamp from repository will be used.--modified_before
- Timestamp that represents date before modification.
Implementing Source of records
Step 1: Before calling the client, you have to implement your own source of records. To do so, inherit from
RecordSource
:
from oarepo_upload_cli.source import RecordSource, SourceRecord, SourceRecordFile
from oarepo_upload_cli.config import Config
class MySource(RecordSource):
"""
Describes a source that is used to generate records.
"""
def __init__(self, config: Config) -> None:
self._config = config
def get_records(self, modified_after, modified_before):
for rec in backend.records(modified_after, modified_before):
yield SourceRecord(
record_id=rec.id,
datetime_modified=rec.modified,
deleted=rec.deleted,
# make sure that the metadata include datetimeModified
# or whatever the modified record name is in config
metadata=rec.metadata.as_json(),
files=[
SourceRecordFile(
key=f.filename,
content_type='application/octet-stream', # use always this
datetime_modified=f.modified,
metadata={
# always include the datetimeModified inside the metadata
# or whatever the modified field name is in config
'datetimeModified': f.modified.isoformat()
},
reader=lambda: f.open_for_read()
) for f in rec.files
]
)
def get_records_count(self, modified_after, modified_before):
return backend.records_changed_count(modified_after, modified_before)
Step 1: Register the implementation to entrypoints:
[options.entry_points]
oarepo_upload_cli.dependencies =
my_source = "my_source_module:MySource"
Step 3: Run the client with --source=my_source
or put then source to configuration.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for oarepo_upload_cli-1.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ca4c1f264c94389153256ac1bbcce253cd2a6a70e147ade55da49fb491b42af |
|
MD5 | 6bd7195023a1a7dccf72a7f6b0d86f8f |
|
BLAKE2b-256 | 73ada56a6407adf1766f51eb8d1c3da0b3c73f8112366a1f6aa863da2f0d2824 |