This release is a pre-release and may not be stable for production use.
What is the Big Data Smart Socket (BDSS)?
The increasing size of datasets used in scientific computing has made it difficult or impossible for a researcher to store all their data at the compute site they are using to process it. This has necessitated that a data transfer step become a key consideration in experimental design. Accordingly, scientific data repositories such as NCBI have begun to offer services such as dedicated data transfer machines and advanced transfer clients. Despite this, many researchers continue familiar but suboptimal practices: using slow transfer clients like a web browser or scp, transferring data over wireless networks, etc.
BDSS aims to alleviate this problem by shifting the burden of learning about alternative file mirrors, transfer clients, tuning parameters, etc. from the end user researcher to a group of “data curators”. It consists of three parts:
Components
Metadata repository
Central database managed by data curators
Matches patterns of data file URLs and maps them to alternate sources
Includes information about the transfer tool to use to retrieve the data
BDSS transfer client
Consumes information from metadata repository
Invokes transfer tools
Reports analytics to metadata repository
Integration as a Galaxy data transfer tool
Get Started
Moving data with the BDSS client:
Setting up a new metadata repository
Examples
All examples here require a metadata repository configured to support them. The default metadata repository at https://bdss.bioinfo.wsu.edu/ supports these examples and the necessary configuration is also listed here.
NCBI SRA archive
NCBI makes files available for transfer using Aspera Connect, a tool with “improved data transfer characteristics” vs FTP or HTTP. If ascp is installed on your machine, BDSS can handle building the appropriate command.
Without BDSS:
ascp -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh -T anonftp@ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra ./With BDSS:
bdss transfer -u 'ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra'Metadata repository configuration:
{
"data_sources": [
{
"description": "",
"label": "NCBI Sequence Read Archive with FTP",
"test_files": [],
"transfer_mechanism": {
"options": {},
"type": "curl"
},
"transforms": [
{
"for_destinations": [],
"options": {
"new_scheme": "aspera"
},
"target": "NCBI Sequence Read Archive with Aspera",
"type": "change_scheme"
}
],
"url_matchers": [
{
"options": {
"pattern": "^ftp://ftp\\.ncbi\\.nlm\\.nih\\.gov/sra"
},
"type": "regular_expression"
}
]
},
{
"description": "",
"label": "NCBI Sequence Read Archive with Aspera",
"test_files": [],
"transfer_mechanism": {
"options": {
"disable_encryption": true,
"username": "anonftp"
},
"type": "aspera"
},
"transforms": [],
"url_matchers": [
{
"options": {
"pattern": "^aspera://ftp\\.ncbi\\.nlm\\.nih\\.gov/sra"
},
"type": "regular_expression"
}
]
}
],
"destinations": []
}JGI Genome Portal
To download files from the JGI Genome Portal, you must first authenticate. BDSS can prompt for credentials and handle storing your session cookies.
Without BDSS:
curl 'https://signon.jgi.doe.gov/signon/create' --data-urlencode 'login=USER_NAME' --data-urlencode 'password=USER_PASSWORD' -c cookies > /dev/null
curl 'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10' -b cookies > get-directoryWith BDSS:
bdss transfer -u 'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10'
JGI Genome Portal username?USER_NAME
JGI Genome Portal password?USER_PASSWORDMetadata repository configuration:
{
"data_sources": [
{
"description": "",
"label": "JGI Genome Portal",
"test_files": [],
"transfer_mechanism": {
"options": {
"auth_url": "https://signon.jgi.doe.gov/signon/create",
"password_field": "password",
"password_prompt": "JGI Genome Portal password?",
"username_field": "login",
"username_prompt": "JGI Genome Portal username?"
},
"type": "session_authenticated_curl"
},
"transforms": [],
"url_matchers": [
{
"options": {
"pattern": "http:\\/\\/genome\\.jgi\\.doe\\.gov\\/ext-api"
},
"type": "regular_expression"
}
]
}
],
"destinations": []
}
Release files for bdss-client 1.0.1b4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bdss_client-1.0.1b4.tar.gz | 18.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bdss_client-1.0.1b4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 70.1 kB
Release files / bdss_client-1.0.1b4.tar.gz
| Download URL | bdss_client-1.0.1b4.tar.gz |
|---|---|
| Size | 18.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
97e39aa649bd0454fb73e7f3c52db5805f0d5935dea7298533d520ac780b6e00
|
|
BLAKE2b-256 checksum How to use checksums |
5516408c57feefda772b6b07f5b71557f7fe35d5bffb5f16cb7bbf2613127d8f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / bdss_client-1.0.1b4-py3-none-any.whl
| Download URL | bdss_client-1.0.1b4-py3-none-any.whl |
|---|---|
| Size | 52.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
724bdbd8c2adc0461f0f7bc7f9ba055e858aea725ffe94a1ffdc0abddbab6656
|
|
BLAKE2b-256 checksum How to use checksums |
1ec623af8c8115ae2ee62bc9831f3fc8a30f132ee7760bab390d8052367151c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
