A package to access sciencedata.dk
Project description
sddk
This is a simple Python package to upload data to- and dowload data from sciencedata.dk. It is especially designed for working with group folders. It relies mainly on Python requests library.
sciencedata.dk is a project managed by DEiC (Danish e-infrastrcture cooperation) aimed to offer a robust data storage, data management and data publication solution for researchers in Denmark and abroad (see docs and dev for more info). The storage is accessible either through (1) the web interface, (2) WebDAV clients or (3) an API relaying on HTTP Protocol (see docs and dev for more info). One of the strength of sciencedata.dk is that it currently supports institutional login from 2976 research and educational institutions around the global (using WAYF). That makes it a perfect tool for international research collaboration.
The main functionality of the package is in uploading any Python object (dict, list, dataframe) as a text or json file to a preselected shared folder and getting it back into a Python environemnt as the original Python object. It uses sciencedata.dk API in combination with Python requests library.
Install and import
So far the package is in a testing PyPi repository here.
To install and import the package within your Python environment (i.e. jupyter notebook) run:
!pip install sddk
import sddk
Configure session and access endpoint for a shared folder
To run the main configuration function below, you have to know the following:
- your sciencedata.dk username (e.g. "123456@au.dk" or "kase@zcu.cz"),
- your sciencedata.dk password (has to be previously configured in the sciencedata.dk web interface),
In the case you want to access a shared folder, you further need:
-
name of the shared folder you want to access (e.g. "our_shared_folder"),
-
username of the owner of the folder (if it is not yours)
(Do not worry, you will be asked to input these values interactively while running the function)
To configure a personal session, run:
s, sddk_url = sddk.configure_session_and_url()
To configure a session pointing to a shared folder, run:
s, sddk_url = sddk.configure_session_and_url("our_shared_folder", "owner_username@au.dk")
Running this function, you configurate two key variables:
s
: a request session authorized by your username and passwordsddk_url
: default url address (endpoint) for your request Below you can inspect how these two are used in typical request commands
Usage
String to TXT
Upload (export) simple text file:
s.put(sddk_url + "testfile.txt", data="textfile content")
Get it back (import) to Python:
string_testfile = ast.literal_eval(s.get(sddk_url + "testfile.txt").text)
print(string_testfile)
Pandas DataFrame to JSON
Upload a dataframe as a json file:
import pandas as pd
df = pd.DataFrame([("a1", "b1", "c1"), ("a2", "b2", "c2")], columns=["a", "b", "c"])
s.put(sddk_url + "df.json", data=df.to_json())
Get it back:
df = pd.DataFrame(s.get(sddk_url + "df.json").json())
Pandas DataFrame to CSV
import pandas as pd
df = pd.DataFrame([("a1", "b1", "c1"), ("a2", "b2", "c2")], columns=["a", "b", "c"])
df.to_csv("df.csv") ### temporal file
s.put(sddk_url + "df.csv", data = open("df.csv", 'rb'))
Dictionary to JSON
To sciencedata.dk:
dict_object = {"a" : 1, "b" : 2, "c":3 }
s.put(sddk_url + "dict_file.json", data=json.dumps(dict_object))
From sciencedata.dk:
dict_object = json.loads(s.get(sddk_url + "dirgot_data/dict_file.json").content)
Matplotlib figure to PNG
import matplotlib.pyplot as plt
fig = plt.figure()
plt.plot(range(10))
fig.savefig('temp.png', dpi=fig.dpi) ### works even in Google colab
s.put(sddk_url + "temp.png", data = open("temp.png", 'rb'))
Next steps
- to develop our own functions for uploading files and getting them back (asking in case of already existing files, etc.:
def file_from_object(file_name_and_loc, python_object):
if s.get(sciencedata_groupurl + file_name_and_loc).ok: ### if there already is a file with the same name
new_name = input("file with name \"" + file_name_and_loc.rpartition("/")[2] + "\" already exists in given location. Press Enter to overwrite it or enter a different name (without path)")
if len(new_name) == 0:
s.put(sciencedata_groupurl + file_name_and_loc, data=json.dumps(python_object))
else:
if "/" in new_name: ### if it is a path
s.put(sciencedata_groupurl + new_name, data=json.dumps(python_object))
else:
s.put(sciencedata_groupurl + file_name_and_loc.rpartition("/")[0] + new_name, data=json.dumps(python_object))
else:
s.put(sciencedata_groupurl + file_name_and_loc, data=json.dumps(python_object))
def object_from_file(file_name_and_loc):
if s.get(sciencedata_groupurl + file_name_and_loc).ok:
print("file exists")
try:
return json.loads(s.get(sciencedata_groupurl + file_name_and_loc).content) ### if there already is a file with the same name
except:
print("file import failed")
else:
print("file does not found; check file name and path.")
The package is built following this tutorial.
Versions history
- 0.0.6 - first functional configuration
- 0.0.7 - configuration of individual session by default
- 0.0.8 - shared folders reading&writing for ordinary users finally functional
- 0.1.1 - added shared folder owner argument to the main configuration function; migration from test.pypi to real pypi
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.