Introduction
What is this repository for?
- This repository include Python scripts that transform raw sensor data collected from Android mobile devices into intermediate CSV file for analysis of various purposes.
- The preprocessing scripts here intend to be used to provide reliable, consistent, and structured intermediate data file for data analysis related to MicroT project.
Summary of Preprocessed Intermediate File
The discussion on data source to include can bee found in the #86 Issue of DataProcessing repository
1. Android Smartwatch
| Information | Raw Data File | Included Data | Omitted Data | Intermediate file |
|---|---|---|---|---|
| Prompt response | [participant_folder] / logs-watch / DATE / HOUR / PromptResponses.log.csv | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response | None | [save_path]/[participant_id]/DATE/watch_prompt_response_PARTICIPANT_DATE.csv |
| Battery level | [participant_folder] / data-watch / DATE / HOUR / Battery.##.event.csv | timestamp, battery_level, battery_charging | None | [save_path]/[participant_id]/DATE/watch_battery_PARTICIPANT_DATE.csv |
| Accelerometer data | [participant_folder] / data-watch / DATE / HOUR / AndroidWearWatch-AccelerationCalibrated-NA.*.sensor.baf | header_timestamp, accelation_meters_per_second_squared(X,Y,Z axis), MIMS-unit | None | [participant_folder] / data-watch / DATE / HOUR / 020000000000-AccelerationCalibrated.*.sensor.csv, mims_DATE_HOUR.csv; a copy of former two in [save_path]/[participant_id]/DATE/ |
| App usages | [participant_folder] / data-watch / DATE / HOUR / AppEventCounts.csv | log_time, last_hour_timestamp, current_hour_time_stamp, app_package_name, event_time_stamp, app_event | None | [save_path]/[participant_id]/DATE/phone_app_usage_PARTICIPANT_DATE.csv |
2. Android Smartphone
| Information | Raw Data File | Included Data | Omitted Data | Intermediate file |
|---|---|---|---|---|
| Prompt response | [participant_folder] / logs / DATE / HOUR / PromptResponses.log.csv | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response | None | [save_path]/[participant_id]/DATE/phone_prompt_response_PARTICIPANT_DATE.csv |
| GPS data | [participant_folder] / data / DATE / HOUR / GPS.csv | log_time, location_time, lat, long, horizontal_accuracy, provider, speed, altitude, bearing | None | [save_path]/[participant_id]/DATE/phone_GPS_PARTICIPANT_DATE.csv |
| step count | [participant_folder] / data / DATE / HOUR / StepCounterService.csv | log_time_stamp, steps_last_hour, accumulated_steps | None | [save_path]/[participant_id]/DATE/phone_stepCount_PARTICIPANT_DATE.csv |
| Phone state and detected activities | [participant_folder] / data / DATE / HOUR / ActivityDetected.csv | log_time, in_vehicle, on_bike, on_foot, running, still, tilting, walking, unknown | None | [save_path]/[participant_id]/DATE/phone_detected_activity_PARTICIPANT_DATE.csv |
| Phone usage events and broadcasts | [save_path]/[participant_id]/DATE/phone_usage_broadcasts_PARTICIPANT_DATE.csv | |||
| Environmental sensors | LightSensorStats.csv, ProximitySensorManagerService.csv, AmbientPressManagerService.csv, AmbientTempManagerService.csv, AmbientHumidManagerService.csv | log time, sensor value, sensor max | None |
Code Book
Detailed explanation of columns in intermediate files can be found in the code book.
Python Version
Python 3.6+ (Other versions haven't been tested but should be fine)
Dependencies
- For user who wants to include accelerometer and MIMS-unit, the below are extra set-up:
- The MIMS-unit depends on particular R package, so install R on your system.
- Add R to your environment variables. For windowsOS users, put path similar to "C:\Program Files\R\R-4.0.2\bin\x64" to your path in system variables, and reboot your computer.
Usage Option1: Install and use as a python package
Install package
The pypi link can be accessed here.
> pip install microt-preprocessing
Import package in python
from microt_preprocessing import time_study_preprocessing_main
microT_root_path = <microT_root_path> # path to the data source folder
intermediate_file_save_path = <intermediate_file_save_path> # path to the destination folder
decrypt_password = <decrypt_password> # decryption password for GPS file
delete_raw = "0" or "1" # "0" denotes not deleting data source, "1" denotes deleting data source
date = "2020-06-11" # specific dates to be preprocessed
time_study_preprocessing_main.preprocessing_all_ema.run_ema_main(microT_root_path, intermediate_file_save_path, decrypt_password, delete_raw, date)
Usage Option2: Clone this project and run scripts
-
Run script
- For smartphone
python preprocessing_all_ema.py <microT_root_path> <intermediate_file_save_path> [date_start]- For smartwatch
python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end]
Who do I talk to?
Maintained by Aditya and Jixin
1 maintainer
Release files for microt-preprocessing 0.1.19
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| microt_preprocessing-0.1.19.tar.gz | 39.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| microt_preprocessing-0.1.19-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 152.6 kB
Release files / microt_preprocessing-0.1.19.tar.gz
| Download URL | microt_preprocessing-0.1.19.tar.gz |
|---|---|
| Size | 39.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4bd4eb6338325ee381674e51ef233ed2cf5b38ca2f4b07ab1c8b82154d7ed16d
|
|
BLAKE2b-256 checksum How to use checksums |
2f6ea5124e1fbbf8ee8f16e242fddb886583ebbe9fdf40dd211563b9688dd52e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0
|
Release files / microt_preprocessing-0.1.19-py3-none-any.whl
| Download URL | microt_preprocessing-0.1.19-py3-none-any.whl |
|---|---|
| Size | 112.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b2cae5d2de1a41b423a40554828241b9b9e9f9dad0854b8ea5c7675151db2d65
|
|
BLAKE2b-256 checksum How to use checksums |
1aba0d247ef0200e5f88881cb3c06e33ae9b694e036215127617f4a9d183375b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0
|
