Wikidata and Wikipedia data extraction for Scribe applications
Project description
Wikidata and Wikipedia data extraction for Scribe applications
Scribe-Data contains the scripts for extracting and formatting data from Wikidata and Wikipedia for Scribe applications. Updates to the language keyboard and interface data can be done using scribe_data/load/update_data.py and the notebooks within the scribe_data/load directory.
Note: The contributing section has information for those interested, with the articles and presentations in featured by also being good resources for learning more about Scribe.
Scribe applications are available on iOS, Android (WIP) and Desktop (planned).
Contents
Process ⇧
scribe_data/load/update_data.py and the notebooks within the scribe_data/load directory are used to update all data for Scribe-iOS, with this functionality later being expanded to update Scribe-Android and Scribe-Desktop when they're active.
The main data update process in update_data.py triggers SPARQL queries to query language data from Wikidata using SPARQLWrapper as a URI. The autosuggestion process derives popular words from Wikipedia as well as those words that normally follow them for an effective baseline feature until natural language processing techniques are employed. Functions to generate autosuggestions are ran in gen_autosuggestions.ipynb. Emojis are further sourced from Unicode CLDR, with this process being ran in gen_emoji_lexicon.ipynb.
The ultimate goal is that this repository will house language packs that are periodically updated with new Wikidata lexicographical data and data from other sources. These packs would then be available to download by users of Scribe applications.
Contributing ⇧
Scribe uses Matrix for communications. You're more than welcome to join us in our public chat rooms to share ideas, ask questions or just say hi :)
Please see the contribution guidelines if you are interested in contributing to Scribe-Data. Work that is in progress or could be implemented is tracked in the issues and projects. Also check the -priority- labels in the issues for those that are most important, as well as those marked good first issue that are tailored for first time contributors.
After your first few pull requests organization members would be happy to discuss granting you further rights as a contributor, with a maintainer role then being possible after continued interest in the project. Scribe seeks to be an inclusive and supportive organization. We'd love to have you on the team!
Ways to Help ⇧
- Reporting bugs as they're found 🐞
- Working on new features ✨
- Documentation for onboarding and project cohesion 📝
- Adding language data to Scribe-Data via Wikidata! 🗃️
Road Map ⇧
The Scribe road map can be followed in the organization's project board where we list the most important issues along with their priority, status and an indication of which sub projects they're included in (if applicable).
Data Edits ⇧
Scribe does not accept direct edits to the grammar JSON files as they are sourced from Wikidata. Edits can be discussed and the queries themselves will be changed and ran before an update. If there is a problem with one of the files, then the fix should be made on Wikidata and not on Scribe. Feel free to let us know that edits have been made by opening a data issue and we'll be happy to integrate them!
Supported Languages ⇧
Scribe's goal is functional, feature-rich keyboards and interfaces for all languages. Check the extract_transform directory for queries for currently supported languages and those that have substantial data on Wikidata.
The following table shows the supported languages and the amount of data available for each on Wikidata and via Unicode CLDR for emojis:
| Languages | Nouns | Verbs | Translations* | Prepositions† | Emoji Keywords |
|---|---|---|---|---|---|
| French | 17,070 | 6,572 | 67,652 | - | 2,488 |
| German | 102,789 | 3,592 | 67,652 | 190 | 2,898 |
| Italian | 8,669 | 73 | 67,652 | - | 2,457 |
| Portuguese | 5,437 | 536 | 67,652 | - | 2,327 |
| Russian | 194,448 | 12 | 67,652 | 15 | 3,827 |
| Spanish | 38,755 | 4,828 | 67,652 | - | 3,134 |
| Swedish | 44,624 | 4,474 | 67,652 | - | 2,913 |
* Given the current beta status where words are machine translated.
† Only for languages for which preposition annotation is needed.
Featured By ⇧
Articles and Presentations on Scribe
2023
- Presentation slides for a talk at Berlin Hack and Tell (Hack of the month winner 🏆)
2022
- Presentation slides for a session at the 2022 Wikimania Hackathon
- Presentation slides for a talk at CocoaHeads Berlin
- Video on Scribe for Wikimedia Celtic Knot 2022
- Presentation slides for a talk with the LD4 Wikidata Affinity Group
- Scribe featured for new developers on MediaWiki
- Presentation slides for Wikimedia Hackathon 2022
- Blog post on Scribe-iOS for Wikimedia Tech News (DE / Tweet)
- Presentation slides for Wikidata Data Reuse Days 2022
Powered By
Contributors
Many thanks to all the Scribe-Data contributors! 🚀
Blog posts
List of referenced posts
Wikimedia Communities
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scribe_data-3.0.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d6849dbf78785c846e3aff8caaf7a2605b34ed6f3b2dace7715afbfa8bc60d24 |
|
| MD5 | 231fc11205f9631e48ea3a569d31f871 |
|
| BLAKE2b-256 | 5d7a821aff7137169e6ee93931e8afc14b790da4488c16e690970f0cf9a98788 |
