Statcan Dialogue Dataset

This repository contains the code for our project. For detailed information, instructions on requesting access, API user guide and documentation, please visit the website. You can find useful links above. A quickstart is provided below.

Quickstart

First, you need to request access (please go to the main webpage for more information).

Then, you can install our Python library to load the dataset:

pip install statcan-dialogue-dataset

Now, inside Python, you can use various functions to work with the data:

import statcan_dialogue_dataset as sdd

# Download the dataset (from dataverse)
sdd.download_dataverse("your_api_token")
# Alternatively, Download from huggingface
sdd.download_huggingface("your_api_token")

# Extract ZIP into ~/.statcan_dialogue_dataset/
sdd.extract_task_data_zip(remove_zip=True)

# Load task specific split for a specific language
train_ret = sdd.load_task_data(task="retrieval", split="train", lang="en")

# Load all task conversations
task_conversations = sdd.load_task_conversations()

# Load table metadata
table = sdd.metadata.Table.from_code(10100002)
# Get subjects or surveys
subjects = table.get_subjects()
surveys = table.get_surveys()

For more information, head to the Core API Reference page.

Reproduce paper results

The full eacl_code directory contains the code used to produce the results in the paper. To reproduce the results, start by reading the README.md file in the eacl_code directory.

StatCan resources

You may find the following external resources useful:

• Web Data Service User Guide
• Web Data Service API
• Developer homepage