Ingest all data from a Typeform account and put the data into a SQL database; can be run regularly to sync updates to DB
A python class and main function to extract and continuously sync data from a Typeform.com account.
You should prefer Python 3 packages from your operating system.
So under Fedora Linux:
dnf install python3-requests python3-sqlalchemy python3-pandas python3-configobj python3-mysql
Under Red Hat Enterprise Linux 8:
dnf install python3-requests python3-sqlalchemy python3-configobj mariadb-connector-c-devel mariadb-connector-c gcc pip3 install mysqlclient --user
This MariaDB connector is the one that works with SQLAlchemy (used by the module). Other connectors as PyMySQL failed our tests.
pip requires compilers and MariaDB development framework pre-installed in the system, as shown in the Red Hat Enterprise Linux section.
Install the module
pip3 install TypeformETL --user
All unsatisfied dependencies (such as Pandas on RHEL) will be installed along.
With plain command line
python3 -m TypeformETL --typeform "6w___API_KEY___nPZ" --database 'mysql://user:password@host/dbname'
--restart to get and sync all data from Typeform, not just last updates.
--updatedb to do everything but update database.
--debug to be more verbose.
If you want to not pass arguments through command line, you can also use a config file.
--config to pass a config file name. By default, the main module will look for a file name called
syncFromTypeform.conf in the current directory.
Arguments from config file and command line will be combined giving priority to the ones passed in command line.
So if a database URL is passed both in command line and config file, the command line will be used.
I have these 2 entries at the same time on my crontab:
@hourly python3 -m TypeformETL --typeform "6w___API_KEY___nPZ" --database 'mysql://user:password@host/dbname' 30 3 * * 0 python3 -m TypeformETL --typeform "6w___API_KEY___nPZ" --database 'mysql://user:password@host/dbname' --restart
Which will run a sync every hour. And once a week will reset data tables and bring all data from scratch.
Into a Python program
from TypeformETL import TypeformETL tf = TypeformETL( token=context['typeform_token'], dburl=context['database'], restart=context['restart'], dbupdate=context['dbupdate'], tableprefix=context['tableprefix'] ) # Read Typeform updates and write to DB tf.sync()
A SQL database must exist. MySQL and MariaDB tested.
tf_” as a table prefix, these are the objects (tables and views) that will be created and updated:
||Contains metadata about forms|
||Contains metadata about form items (text fields, checkboxes) related to their parent forms|
||Contains all answers to all fields of all forms; each complete form response has an entry in the
||Contains all responses and metadata to all forms; each form response has an entry here, each form field answer has an entry in the
||Operational table used by the syncer|
||Operational table that logs every sync with some simple statistics|
||A convenient view that joins together table
||The calculated current NPS (Net Promoter Score) of all numerical fields (only a few fields might have a real NPS semantic)|
||The NPS of all numerical fields per day; can be used to see evolution of some NPS along time.|
||Auxiliary view used to calculate cumulative NPS|
The module makes
TRUNCATE operations. Make sure the database connection user has granted permission to all these operations.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size TypeformETL-0.5.12-py3-none-any.whl (27.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size TypeformETL-0.5.12.tar.gz (9.6 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for TypeformETL-0.5.12-py3-none-any.whl