pycanvasdata 0.0.3

Python utilities for working with Canvas Data

Full documentation

This python module is designed to make it easy to access Canvas Data files.

Currently, this module makes it possible to:

• Convert downloaded Canvas Data files to CSV files with headers

• export SQL table creation statements

• list files for a table

• download all files or files for a specific table

• view the Canvas Data schema (fields, field types, etc)

• connect to a database (uses sqlalchemy) to create tables, import data, run SQL queries

---

Module Usage

This module can be used programatically in other scripts and software. An example of creating a canvas_data object is found below.:

from canvas_data_utils.canvas_data_auth import CanvasData

canvas_data_object = CanvasData(
    API_KEY=YOUR_API_KEY ,
    API_SECRET=YOUR_API_SECRET,
    base_folder = YOUR_BASE_DIR,
    data_folder = YOUR_DATA_DIR)

Once you have that object created, you can…

generate mysql table creation statements

mysql_table_creation_statement = canvas_data_object.table_creation_statement('mysql')

generate sqlite table creation statements

sqlite_table_creation_statement = canvas_data_object.table_creation_statement('sqlite')

generate postgres table creation statements

postgres_table_creation_statement = canvas_data_object.table_creation_statement('postgres')

create tables in a database given by a connection string

canvas_data_object.create_tables('sqlite:///{}'.format(db_filename))

fetch the current schema (as json)

schema = canvas_data_object.fetch_schema()

get a list of columns in a table

user_dim_columns = canvas_data_object.get_schema_columns( 'user_dim')

convert an text file download from TSV (Tab Separated Values) to CSV

canvas_data_object.convert_tsv_to_csv(tsv_filepath)

list all the tables in the schema

table_list = canvas_data_object.table_list()

download and convert all files to CSV

canvas_data_object.convert_all_to_csv()

list all downloadable files for a table

file_list = canvas_data_object.list_all_files('user_dim')

---

Config File

You need to create a config file somewhere. This config file is a typical .INI file. It should look something like the following example.

[config]
API_SECRET = replace_with_api_secret_from_canvas_data
API_KEY = replace_with_api_key_from_canvas_data

base_folder = /path/to/base/folder/for/downloads/
data_folder = %(base_folder)s/test2

connection_string = sqlite:///%(base_folder)s/sample.db

Note: The connection_string configuration follows the connection pattern needed by SQLAlchemy at http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html.

This library supports any database type than SQLAlchemy does.

---

Command-line Tool

This library includes a command line utility called canvasdata.

Usage

canvasdata [-h] [--config CONFIG] [-t TABLE] [--offline OFFLINE]
                {convert_to_csv,import,create_tables,reset,sql_create_statement,list_files,download,sample_queries,schema}

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       path to the configuration file
  -t TABLE              specify a specific table
  --offline OFFLINE     run in offline mode