scribe-data 6.0.4

Wikidata and Wiktionary language data extraction

Scribe-Data is a command-line interface (CLI) for extracting and formatting language data from Wikidata, Wiktionary and Unicode for Scribe applications. Written in a way that other projects can adopt it for their language data needs, Scribe-Data helps users list, download, manage, convert, and filter language data directly from the terminal.

Scribe applications are available on iOS, Android (WIP) and Desktop (planned).

Check out Scribe-Data's architecture diagram for an overview of the data process. Also see and Scribe's architecture diagram to learn about our organization including our applications, services and processes. It depicts the projects that Scribe is developing as well as the relationships between them and the external systems with which they interact. Also check out the Wikidata and Scribe Guide for an overview of Wikidata and getting language data from it.

Contents

• Process
• Installation
• CLI Usage
• Data Contracts
• Contributing
• Environment Setup
• Featured By

Process

The CLI commands defined within scribe_data/cli are used to update all data for Scribe-iOS and Scribe-Android, with this functionality later being expanded to update Scribe-Desktop.

The main data update process triggers language based SPARQL queries to query language data from Wikidata using SPARQLWrapper as a URI. The base command for getting data from Wikidata is scribe-data get -lang LANGUAGE -dt DESIRED_DATA, with DESIRED_DATA being replaced by data types like nouns, verbs, etc. Emojis are further sourced from Unicode CLDR, with this process being ran via the scribe-data get -lang LANGUAGE -dt emoji-keywords command.

_{Back to top.}

Installation

Scribe-Data is available for installation via uv (recommended) or pip.

For Users

# Using uv (recommended - fast, Rust-based installer):
uv pip install scribe-data

# Or using pip:
pip install scribe-data

For Development Build

git clone https://github.com/scribe-org/Scribe-Data.git  # or ideally your fork
cd Scribe-Data

# With uv (recommended):
uv sync --all-groups  # install all dependencies
source .venv/bin/activate  # activate venv (macOS/Linux)
# .venv\Scripts\activate  # activate venv (Windows)

# Or with pip:
python -m venv .venv  # create virtual environment
source .venv/bin/activate  # activate venv (macOS/Linux)
# .venv\Scripts\activate  # activate venv (Windows)
pip install -e .

_{Back to top.}

CLI Usage

Scribe-Data provides a command-line interface (CLI) for efficient interaction with its language data functionalities. Please see the usage guide or the official documentation for detailed instructions.

Basic Usage

To utilize the Scribe-Data CLI, you can execute variations of the following commands in your terminal:

scribe-data -h  # view the cli options
scribe-data [command] [arguments]

Available Commands

• list (l): List languages, data types and combinations of each that Scribe-Data can be used for.
• get (g): Get data from Wikidata and other sources for the given languages and data types.
• total (t): Check Wikidata for the total available data for the given languages and data types.
• convert (c): Convert data returned by Scribe-Data to different file types.
• download (d): Download Wikidata lexeme or Wiktionary dumps.
• export_contracts (ec): Export Scribe-Data contracts to a local directory.
• audit_wd_lexeme_forms (awdlf): Run an audit of the available language data forms on Wikidata.
• generate_wd_lexeme_queries (gwdlq): Generate Wikidata language data queries from contracts.
• check_contracts (cc): Check the data in a Scribe-Data export directory to see that all needed language data is included to fulfill data contracts.
• filter_data (fd): Filter exported Scribe-Data data based on provided data contract values.
• interactive (i): Run in interactive mode.

Command Examples

scribe-data list
scribe-data list --language
scribe-data list --data-type
scribe-data get --language English --data-type verbs --output-dir ./scribe-data
scribe-data total --language English

scribe-data get --interactive
scribe-data total --interactive

_{Back to top.}

Data Contracts

Wikidata has lots of language data available, but not all of it is useful for all applications. By default the Scribe-Data get requests returns only the data for the given languages and data types that is needed for Scribe applications. Larger Scribe-Data requests should be parsing Wikidata lexeme dumps as the data source.

Scribe's solution for the modularity of Wikidata's data is data contracts. The data contracts for Scribe's client applications can be found in the scribe_data_contracts directory. Data contracts are YAML objects where the values that are used in end applications are the keys and the resulting data identifiers based on Wikidata lexeme forms are the values. If the forms for a lexeme change, then the values would also change, but all that's needed is to update the contract for the application to function again.

Efficient client application data updates using Scribe-Data follow as such:

• Wikidata is audited by the Scribe community to see if new data exists
• Contracts are written to map the data values to keys that are used in the application
• Scribe-Data queries are regenerated based on the new contracts and these queries are ran to get new data

scribe-data gwdlq

• The contracts are checked to make sure that all contract values still exist within the resulting data
  • The question is whether a form was added or removed from a data point such that its identifier has changed
  • This is done via the following command:

scribe-data cc -cd DATA_CONTRACTS_DIRECTORY  # default data path is used

• If the check above passes, then new data can be added to the client applications
• If the check fails, then the contract values should be updated given the directions from the CLI and then new data can be loaded
• If contracts are updated and a user wants to filter a current dataset for the new contract values, then this can be done via the following:

scribe-data fd -cd DATA_CONTRACTS_DIRECTORY  # default data paths are used

Updating contracts shouldn't be something that Scribe-Data users should have to do often if they're using stable data from Wikidata. We provide this functionality given the wiki nature of the underlying data so that the Scribe community and others can easily react to potential changes in the lexeme data. The base structure of your YAML objects should be consistent, and the final values can be updated such that new data can easily be loaded into applications simply by fulfilling the contracts.

_{Back to top.}

Contributing

Scribe uses Matrix for communications. You're more than welcome to join us in our public chat rooms to share ideas, ask questions or just say hi to the team :) We'd suggest that you use the Element client and Element X for a mobile app.

Please see the contribution guidelines and Wikidata and Scribe Guide if you are interested in contributing to Scribe-Data. Work that is in progress or could be implemented is tracked in the issues and projects.

Those interested can further check the -next release- and -priority- labels in the issues for those that are most important, as well as those marked good first issue that are tailored for first-time contributors.

After your first few pull requests organization members would be happy to discuss granting you further rights as a contributor, with a maintainer role then being possible after continued interest in the project. Scribe seeks to be an inclusive and supportive organization. We'd love to have you on the team!

Ways to Help

• Reporting bugs as they're found 🐞
• Working on new features ✨
• Documentation for onboarding and project cohesion 📝
• Adding language data to Scribe-Data via Wikidata! 🗃️

Road Map

The Scribe road map can be followed in the organization's project board where we list the most important issues along with their priority, status and an indication of which sub projects they're included in (if applicable).

Data Edits

Scribe does not accept direct edits to the queries or any resulting Scribe-Data data files as they are sourced from data contracts and Wikidata. Edits can be discussed and the contracts themselves will be changed used to generate queries before an update. If there is a problem with data in Scribe applications, then the fix should be made on Wikidata and not on Scribe. Feel free to let us know that edits have been made by opening a data issue and we'll be happy to integrate them!

_{Back to top.}

Environment Setup

The development environment for Scribe-Data can be installed via the following steps:

1. Fork the Scribe-Data repo, clone your fork, and configure the remotes:

# Clone your fork of the repo into the current directory.
git clone https://github.com/<your-username>/Scribe-Data.git
# Navigate to the newly cloned directory.
cd Scribe-Data
# Assign the original repo to a remote called "upstream".
git remote add upstream https://github.com/scribe-org/Scribe-Data.git

• Now, if you run git remote -v you should see two remote repositories named:
  • origin (forked repository)
  • upstream (Scribe-Data repository)

2. Create a virtual environment for Scribe-Data (Python >=3.12), activate it and install dependencies:

uv sync --all-groups  # create .venv and install all dependencies from uv.lock

# Unix or macOS:
source .venv/bin/activate

# Windows:
.venv\Scripts\activate.bat  # .venv\Scripts\activate.ps1 (PowerShell)

After activating the virtual environment, set up prek by running:

prek install
# uv run prek run --all-files  # lint and fix common problems in the codebase

See the contribution guidelines for a more detailed explanation and troubleshooting.

_{Back to top.}

Featured By

Please see the blog posts page on our website for a list of articles on Scribe, and feel free to open a pull request to add one that you've written at scribe-org/scri.be!

Organizations

The following organizations have supported the development of Scribe projects through various programs. Thank you all! 💙

                   

                   

_{Back to top.}

Powered By

Contributors

Many thanks to all the Scribe-Data contributors! 🚀

Wikimedia Communities