pykobo 0.1.0

A Python module to fetch data from the Kobo API

pykobo

Pykobo is a Python module that fetches data from Kobo forms via the Kobo API.

The data is returned as a pandas DataFrame making it easy to integrate in your workflow for further cleaning, filtering, aggregatation, etc...

Functionalities

Pykobo can:

• Get the list of forms a user has access to
• Fetch the data of a form (with names or labels for the columns and choices) as a pandas DataFrame
• Download a form in XLS or XML format

Requirements

• requests
• pandas
• numpy

Examples

How to start

NB: The example below uses kf.kobotoolbox.org but pykobo also works with kobo.humanitarianresponse.info and any other Kobo server.

To get your API token, see here.

import pykobo


URL_KOBO_API = "https://kf.kobotoolbox.org/api/v2"
MYTOKEN = "2bc8e0201d23dac4ec1c334107698147b81513a2"

# Initialize the Manager object
km = pykobo.Manager(url_api=URL_KOBO_API, token=MYTOKEN)

Get the list of forms you have access to

my_forms = km.get_forms()

This returns a Python list of KoboForm objetcs

print(my_forms)

[   KoboForm('tpz2buHAdXxcN0JVrZaSdk'),
    KoboForm('vyARFbyE8Gv3RUvXNfdTRj'),
    KoboForm('wogyYJzUu2ZFVnzqGg8K7q'),
    KoboForm('bQLZapErE3UqqG9Avntkhd')]

# Each form contains metadata
for f in my_forms:
      print(f.metadata)

{   'date_created': '2022-07-14T20:44:11.929901Z',
    'date_modified': '2022-10-02T07:49:19.714891Z',
    'geo': True,
    'has_deployment': True,
    'name': 'Household survey',
    'owner': 'pvernier',
    'version_id': 'aqUMoSqANiEgH3j4Nn3Cr7'}
{   'date_created': '2022-07-14T12:41:14.665314Z',
    'date_modified': '2022-09-28T11:55:15.408542Z',
    'geo': True,
    'has_deployment': True,
    'name': 'Health facilities monitoring',
    'owner': 'pvernier',
    'version_id': 'abLugnJGURSyyZ8RZxC0wQ'}
{   'date_created': '2022-07-14T13:40:32.033446Z',
    'date_modified': '2022-09-28T09:19:20.691620Z',
    'geo': False,
    'has_deployment': True,
    'name': 'Post emergency evaluation',
    'owner': 'pvernier',
    'version_id': 'aQQUmPns7xLUL4Ro0amqwS'}
{   'date_created': '2022-09-14T16:54:06.990672Z',
    'date_modified': '2022-09-20T13:27:52.410261Z',
    'geo': True,
    'has_deployment': True,
    'name': 'Identification of burnt areas',
    'owner': 'pvernier',
    'version_id': 'xes8JkQRpbDcbct9sqmCYZ'}  

Fetch a single form with its uid.

uid = 'tpz2buHAdXxcN0JVrZaSdk'

my_form = km.get_form(uid)

print(my_form.metadata)

{   'date_created': '2022-07-14T20:44:11.929901Z',
    'date_modified': '2022-10-02T07:49:19.714891Z',
    'geo': True,
    'has_deployment': True,
    'name': 'Household survey',
    'owner': 'pvernier',
    'version_id': 'aqUMoSqANiEgH3j4Nn3Cr7'}

Fetch the data of a form

my_form.fetch_data()

# The data is accessible via the `data` attribute as a pandas DataFrame

print(my_form.data)
                             start                            end       today         username                  deviceid            phonenumber  ...     _submission_time _tags _notes _validation_status    _submitted_by _index
0    2022-09-01T15:47:55.797+02:00  2022-09-01T15:51:48.302+02:00  2022-09-01  surveyer_1  collect:4vUec4gLVJx3GP1D                    NaN  ...  2022-09-01T13:52:04    []     []                 {}  surveyer_1      1
1    2022-09-01T15:58:08.251+02:00  2022-09-01T16:08:14.548+02:00  2022-09-01  surveyer_1  collect:Xk9Z5f1VTW5nig68                    NaN  ...  2022-09-01T14:08:46    []     []                 {}  surveyer_1      2
2    2022-09-01T14:05:08.484+02:00  2022-09-01T16:17:59.305+02:00  2022-09-01  surveyer_1  collect:0Y8Cozz5fzI8jczs                    NaN  ...  2022-09-01T14:18:36    []     []                 {}  surveyer_1      3
3    2022-09-01T16:20:39.699+02:00  2022-09-01T16:32:03.393+02:00  2022-09-01  surveyer_1  collect:MPi52tvGiPY6AuK3                    NaN  ...  2022-09-01T14:32:27    []     []                 {}  surveyer_1      4
...
...
[595 rows x 38 columns]

print(type(my_form.data))

<class 'pandas.core.frame.DataFrame'>

# The method `fetch_data` returns the data using the Kobo columns and choices names

print(my_form.data.columns)

Index(['start', 'end', 'today', 'username', 'deviceid', 'phonenumber', 'date',
       'health_area', 'village_name', 'team_number', 'cluster_number',
       'household_number', 'gps', '_gps_latitude', '_gps_longitude',
       '_gps_altitude', '_gps_precision', 'hhh_present',
       'age_hhh', 'consent', 'number_children',
       '__version__', '_id', '_uuid', '_status', '_submission_time',
       '_tags', '_notes', '_validation_status', '_submitted_by', '_index'],
      dtype='object')

Display the data using Kobo labels for columns and/or choices

my_form.display(columns_as='label', choices_as='label')

print(my_form.data.columns)

Index(['start', 'end', 'today', 'username', 'deviceid', 'phonenumber',
       'Date of the survey', 'Health zone', 'Name of the village', 'team number',
       'Cluster number', 'Household number', 'GPS Coordinates',
       '_GPS Coordinates_latitude', '_GPS Coordinates_longitude',
       '_GPS Coordinates_altitude', '_GPS Coordinates_precision',
       'Head of the household present?',
       'Age of the head of the household ',
       'Consent obtained',
       'Number of children in the household',
       '__version__', '_id', '_uuid', '_status', '_submission_time', '_tags',
       '_notes', '_validation_status', '_submitted_by', '_index'],
      dtype='object')


# You can go back and forth between names and labels as much as you want 
my_form.display(columns_as='label', choices_as='name')
my_form.display(columns_as='name', choices_as='label')
my_form.display(columns_as='name', choices_as='name')
my_form.display(columns_as='label', choices_as='label')

Note

• For questions of type select_multiple the different answers are separated by a '|'.

• If a form contains n columns with the same label, a suffix (1) to (n) will be added to each of the columns.

Repeats

Repeats are supported (only one level, not repeats inside repeats). In this case data of the repeat groups are separated from the 'main' data and accessible via the 'repeats' attribute which returns a Python dictionary

print(my_form.has_repeats)

True
# This means that the form has at least 1 repeat group

print(my_form.repeats.keys())

dict_keys(['children_questions'])
# The form has 1 repeat group called 'children_questions'


print(my_form.repeats['children_questions'])


     index_repeat Sex of the child Age of the child  ... Going to school?  _parent_index
0               1                           Male              No                    2
1               2                           Female            No                    2
2               1                           Female            No                    4
3               1                           Female            Yes                   5
4               2                           Female            No                    5
...
...
[1040 rows x 27 columns]

The column _index in the main DataFrame (my_form.data) and the column _parent_index in the DatFrame of the repeat group can be used to join the 2 DataFrames.

df_join = pd.merge(
    my_form.data,
    my_form.repeats['groupe_questions_enfants'],
    how="left",
    left_on='_index',
    right_on='_parent_index'
)

Save the data to file

Because the data is a pandas DataFrame, we can take advantage of the many pandas methods to export it to a file.

# CSV
df_join.to_csv('household_survey.csv', index=False)

# Excel
df_join.to_excel('household_survey.xlsx', index=False)

Download a form in XLS or XML format

my_form.download_form('xls')

This downloads the XLSForm tpz2buHAdXxcN0JVrZaSdk.xls in the current working directory

Also

Pykobo has a bunch of utility methods that make easy to clean you data (not documented yet).

Note

Pykobo only reads and fetches data from Kobo forms. It doesn't update or delete the forms and their data on the Kobo server.

TO DO

• Add possibility to display group name as a prefix
• Add method to download media files
• Clean and document utility functions
• Be more consistent and robust in case of errors
• Calculate stats on forms time duration
• Build Python package