# CerebralCortex Kernel
Cerebral Cortex is the big data cloud companion of mCerebrum designed to support population-scale data analysis, visualization, model development, and intervention design for mobile sensor data.
You can find more information about MD2K software on our [software website](https://md2k.org/software) or the MD2K organization on our [MD2K website](https://md2k.org/).
CerebralCortex Kernel is part of our [CerebralCortex cloud platform](https://github.com/MD2Korg/CerebralCortex). CerebralCortex Kernel is mainly responsible to store/retrieve mobile sensor data along with it's metadata.
**Note**:
We have renamed following repositories.
* CerebralCortex-Platform -> CerebralCortex
* CerebralCortex - > CerebralCortex-Kernel
## Documentation
- [Source code documentation](https://cerebralcortex-kernel.readthedocs.io/en/3.0.1/)
## Installation
CerebralCortex-Kernel is a part of CerebralCortex cloud platform. To test the complete cloud platform, please visit [CerebralCortex](https://github.com/MD2Korg/CerebralCortex).
CerebralCortex-Kernel requires minimum [Python3.6](https://www.python.org/downloads/release/python-360/). To install CerebralCortex-Kernel as an API:
```pip3 install cerebralcortex-kernel```
- Note: please use appropriate pip (e.g., pip, pip3, pip3.6 etc.) installed on your machine
## Code Example
Please have a look at [example code](https://github.com/MD2Korg/CerebralCortex-Kernel-Examples), if you want to only see an example on how to call CerebralCortex-Kernel API.
## FAQ
**1 - Do I need whole CerebralCortex cloud platform to use CerebralCortex-Kernal?**
No! If you want to use CerebralCortex-Kernel independently then you would need:
* Backend storage (FileSystem/HDFS and MySQL) with some data. Here is [some sample data](TODO) to play with.
* Setup the [configurations](https://github.com/MD2Korg/CerebralCortex-Kernel/tree/master/conf)
* Use the [examples](TODO) to start exploring data
**2 - I do not want tCerebralCortex-kernel (Examples)
================================
This directory contains some of the following examples on how to
get/save data streams and perform basic operations:
- Window stream data into 1 minute chunks
Dependencies
------------
- `Python3.6`_
- Note: Python3.7 is not compatible with some of the requirements
- Make sure pip version matches Python version
- spark 2.4
- Download and extract `Spark 2.4`_
- ``cd ~/``
- ``wget http://apache.claz.org/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz``
- ``tar -xzf spark*.tar.gz``
- Spark requires Java-8 installed
- Edit ``SPARK_HOME`` in
``CerebralCortex-Kernel-Examples/examples/basic_windowing/run.sh``
to point to the location you extracted Spark to.
- ex: ``export SPARK_HOME=~/spark-2.4.0-bin/hadoop2.7/`` if Spark
was extracted to your home directory.
- ``export PYTHONPATH="${PYTHONPATH}":PATH-OF-MAIN-DIR/CerebralCortex-Kernel-Examples``
(This is defined in
``CerebralCortex-Kernel-Examples/examples/basic_windowing/run.sh``)
- MySQL > 5.7
- You might have to set up a MySQL user.
Installation
------------
- ``git clone https://github.com/MD2Korg/CerebralCortex-Kernel-Examples.git``
- ``cd CerebralCortex-Kernel-Examples``
- ``sudo pip3 install -r requirements.txt``
- Note: please use appropriate pip (e.g., pip, pip3, pip3.6 etc.)
installed on your machine
Configure CerebralCortex-Kernel
-------------------------------
- Update MySQL settings in
``CerebralCortex-Kernel-Examples/conf/cerebralcortex.yml`` file, for
example, mysql username, password etc.. Please look at the comments
on what params shall be updated.
How to run the example code?
----------------------------
- **Import MySQL Database:**
- ``cd CerebralCortex-Kernel-Examples/resources/db``
- ``mysql -u MySQL-USERNAME -pMySQL-PASSWORD < cerebralcortex.sql``
- **Run example**
- ``cd CerebralCortex-Kernel-Examples/examples/basic_windowing``
- ``sh run.sh``
If everything works well then example code will produce similar output
on console as below:
::
+--------------------+--------------------+--------------------+
| user| window| battery_level|
+--------------------+--------------------+--------------------+
|00000000-afb8-476...|[2019-01-09 11:35...|[100, 100, 100, 1...|
|00000000-afb8-476...|[2019-01-09 11:41...|[97, 97, 97, 97, ...|
|00000000-afb8-476...|[2019-01-09 11:43...|[96, 96, 96, 96, ...|
|00000000-afb8-476...|[2019-01-09 11:38...|[99, 99, 99, 99, ...|
|00000000-afb8-476...|[2019-01-09 11:39...|[98, 98, 98, 98, ...|
+--------------------+--------------------+--------------------+
only showing top 5 rows
BATTERY--org.md2k.phonesensor--PHONE-windowed-data has been stored.
.. _Python3.6: https://www.python.org/downloads/release/python-360/
.. _Spark 2.4: https://spark.apache.org/downloads.htmlo use FileSystem/HDFS as NoSQL storage. How can I change NoSQL storage backend?**
CerebralCortex-Kernel follows component based structure. This makes it easier to add/remove features.
* Add a new class in [Data manager-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/).
* New class must have read/write methods. Here is a sample [skeleton class](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/storage_blueprint.py) with mandatory methods required in the new class.
* Create an object of new class in [Data-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/data.py) with appropriate parameters.
* Add appropriate configurations in [cerebralcortex.yml](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml) in (NoSQL Storage)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml#L8] section.
**3 - How can I replace MySQL with another SQL storage system?**
* Add a new class in [Data manager-SQL](https://github.com/MD2Korg/CerebralCortex-Kernel/tree/master/cerebralcortex/core/data_manager/sql).
* New class must implement all of the methods available in (stream_handler.py)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/sql/stream_handler.py] class.
* Create an object of new class in [Data-SQL](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/sql/data.py) with appropriate parameters.
* Add appropriate configurations in [cerebralcortex.yml](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml) in (Relational Storage)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml#L31] section.
**4 - Where are all the backend storage related classes/methods?**
In [Data manager-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/). You can add/change any backend storage.
## Contributing
Please read our [Contributing Guidelines](https://md2k.org/contributing/contributing-guidelines.html) for details on the process for submitting pull requests to us.
We use the [Python PEP 8 Style Guide](https://www.python.org/dev/peps/pep-0008/).
Our [Code of Conduct](https://md2k.org/contributing/code-of-conduct.html) is the [Contributor Covenant](https://www.contributor-covenant.org/).
Bug reports can be submitted through [JIRA](https://md2korg.atlassian.net/secure/Dashboard.jspa).
Our discussion forum can be found [here](https://discuss.md2k.org/).
## Versioning
We use [Semantic Versioning](https://semver.org/) for versioning the software which is based on the following guidelines.
MAJOR.MINOR.PATCH (example: 3.0.12)
1. MAJOR version when incompatible API changes are made,
2. MINOR version when functionality is added in a backwards-compatible manner, and
3. PATCH version when backwards-compatible bug fixes are introduced.
For the versions available, see [this repository's tags](https://github.com/MD2Korg/CerebralCortex/tags).
## Contributors
Link to the [list of contributors](https://github.com/MD2Korg/CerebralCortex-Kernel/graphs/contributors) who participated in this project.
## License
This project is licensed under the BSD 2-Clause - see the [license](https://md2k.org/software-under-the-hood/software-uth-license) file for details.
## Acknowledgments
* [National Institutes of Health](https://www.nih.gov/) - [Big Data to Knowledge Initiative](https://datascience.nih.gov/bd2k)
* Grants: R01MD010362, 1UG1DA04030901, 1U54EB020404, 1R01CA190329, 1R01DE02524, R00MD010468, 3UH2DA041713, 10555SC
* [National Science Foundation](https://www.nsf.gov/)
* Grants: 1640813, 1722646
* [Intelligence Advanced Research Projects Activity](https://www.iarpa.gov/)
* Contract: 2017-17042800006
Cerebral Cortex is the big data cloud companion of mCerebrum designed to support population-scale data analysis, visualization, model development, and intervention design for mobile sensor data.
You can find more information about MD2K software on our [software website](https://md2k.org/software) or the MD2K organization on our [MD2K website](https://md2k.org/).
CerebralCortex Kernel is part of our [CerebralCortex cloud platform](https://github.com/MD2Korg/CerebralCortex). CerebralCortex Kernel is mainly responsible to store/retrieve mobile sensor data along with it's metadata.
**Note**:
We have renamed following repositories.
* CerebralCortex-Platform -> CerebralCortex
* CerebralCortex - > CerebralCortex-Kernel
## Documentation
- [Source code documentation](https://cerebralcortex-kernel.readthedocs.io/en/3.0.1/)
## Installation
CerebralCortex-Kernel is a part of CerebralCortex cloud platform. To test the complete cloud platform, please visit [CerebralCortex](https://github.com/MD2Korg/CerebralCortex).
CerebralCortex-Kernel requires minimum [Python3.6](https://www.python.org/downloads/release/python-360/). To install CerebralCortex-Kernel as an API:
```pip3 install cerebralcortex-kernel```
- Note: please use appropriate pip (e.g., pip, pip3, pip3.6 etc.) installed on your machine
## Code Example
Please have a look at [example code](https://github.com/MD2Korg/CerebralCortex-Kernel-Examples), if you want to only see an example on how to call CerebralCortex-Kernel API.
## FAQ
**1 - Do I need whole CerebralCortex cloud platform to use CerebralCortex-Kernal?**
No! If you want to use CerebralCortex-Kernel independently then you would need:
* Backend storage (FileSystem/HDFS and MySQL) with some data. Here is [some sample data](TODO) to play with.
* Setup the [configurations](https://github.com/MD2Korg/CerebralCortex-Kernel/tree/master/conf)
* Use the [examples](TODO) to start exploring data
**2 - I do not want tCerebralCortex-kernel (Examples)
================================
This directory contains some of the following examples on how to
get/save data streams and perform basic operations:
- Window stream data into 1 minute chunks
Dependencies
------------
- `Python3.6`_
- Note: Python3.7 is not compatible with some of the requirements
- Make sure pip version matches Python version
- spark 2.4
- Download and extract `Spark 2.4`_
- ``cd ~/``
- ``wget http://apache.claz.org/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz``
- ``tar -xzf spark*.tar.gz``
- Spark requires Java-8 installed
- Edit ``SPARK_HOME`` in
``CerebralCortex-Kernel-Examples/examples/basic_windowing/run.sh``
to point to the location you extracted Spark to.
- ex: ``export SPARK_HOME=~/spark-2.4.0-bin/hadoop2.7/`` if Spark
was extracted to your home directory.
- ``export PYTHONPATH="${PYTHONPATH}":PATH-OF-MAIN-DIR/CerebralCortex-Kernel-Examples``
(This is defined in
``CerebralCortex-Kernel-Examples/examples/basic_windowing/run.sh``)
- MySQL > 5.7
- You might have to set up a MySQL user.
Installation
------------
- ``git clone https://github.com/MD2Korg/CerebralCortex-Kernel-Examples.git``
- ``cd CerebralCortex-Kernel-Examples``
- ``sudo pip3 install -r requirements.txt``
- Note: please use appropriate pip (e.g., pip, pip3, pip3.6 etc.)
installed on your machine
Configure CerebralCortex-Kernel
-------------------------------
- Update MySQL settings in
``CerebralCortex-Kernel-Examples/conf/cerebralcortex.yml`` file, for
example, mysql username, password etc.. Please look at the comments
on what params shall be updated.
How to run the example code?
----------------------------
- **Import MySQL Database:**
- ``cd CerebralCortex-Kernel-Examples/resources/db``
- ``mysql -u MySQL-USERNAME -pMySQL-PASSWORD < cerebralcortex.sql``
- **Run example**
- ``cd CerebralCortex-Kernel-Examples/examples/basic_windowing``
- ``sh run.sh``
If everything works well then example code will produce similar output
on console as below:
::
+--------------------+--------------------+--------------------+
| user| window| battery_level|
+--------------------+--------------------+--------------------+
|00000000-afb8-476...|[2019-01-09 11:35...|[100, 100, 100, 1...|
|00000000-afb8-476...|[2019-01-09 11:41...|[97, 97, 97, 97, ...|
|00000000-afb8-476...|[2019-01-09 11:43...|[96, 96, 96, 96, ...|
|00000000-afb8-476...|[2019-01-09 11:38...|[99, 99, 99, 99, ...|
|00000000-afb8-476...|[2019-01-09 11:39...|[98, 98, 98, 98, ...|
+--------------------+--------------------+--------------------+
only showing top 5 rows
BATTERY--org.md2k.phonesensor--PHONE-windowed-data has been stored.
.. _Python3.6: https://www.python.org/downloads/release/python-360/
.. _Spark 2.4: https://spark.apache.org/downloads.htmlo use FileSystem/HDFS as NoSQL storage. How can I change NoSQL storage backend?**
CerebralCortex-Kernel follows component based structure. This makes it easier to add/remove features.
* Add a new class in [Data manager-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/).
* New class must have read/write methods. Here is a sample [skeleton class](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/storage_blueprint.py) with mandatory methods required in the new class.
* Create an object of new class in [Data-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/raw/data.py) with appropriate parameters.
* Add appropriate configurations in [cerebralcortex.yml](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml) in (NoSQL Storage)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml#L8] section.
**3 - How can I replace MySQL with another SQL storage system?**
* Add a new class in [Data manager-SQL](https://github.com/MD2Korg/CerebralCortex-Kernel/tree/master/cerebralcortex/core/data_manager/sql).
* New class must implement all of the methods available in (stream_handler.py)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/sql/stream_handler.py] class.
* Create an object of new class in [Data-SQL](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/sql/data.py) with appropriate parameters.
* Add appropriate configurations in [cerebralcortex.yml](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml) in (Relational Storage)[https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/conf/cerebralcortex.yml#L31] section.
**4 - Where are all the backend storage related classes/methods?**
In [Data manager-Raw](https://github.com/MD2Korg/CerebralCortex-Kernel/blob/master/cerebralcortex/core/data_manager/). You can add/change any backend storage.
## Contributing
Please read our [Contributing Guidelines](https://md2k.org/contributing/contributing-guidelines.html) for details on the process for submitting pull requests to us.
We use the [Python PEP 8 Style Guide](https://www.python.org/dev/peps/pep-0008/).
Our [Code of Conduct](https://md2k.org/contributing/code-of-conduct.html) is the [Contributor Covenant](https://www.contributor-covenant.org/).
Bug reports can be submitted through [JIRA](https://md2korg.atlassian.net/secure/Dashboard.jspa).
Our discussion forum can be found [here](https://discuss.md2k.org/).
## Versioning
We use [Semantic Versioning](https://semver.org/) for versioning the software which is based on the following guidelines.
MAJOR.MINOR.PATCH (example: 3.0.12)
1. MAJOR version when incompatible API changes are made,
2. MINOR version when functionality is added in a backwards-compatible manner, and
3. PATCH version when backwards-compatible bug fixes are introduced.
For the versions available, see [this repository's tags](https://github.com/MD2Korg/CerebralCortex/tags).
## Contributors
Link to the [list of contributors](https://github.com/MD2Korg/CerebralCortex-Kernel/graphs/contributors) who participated in this project.
## License
This project is licensed under the BSD 2-Clause - see the [license](https://md2k.org/software-under-the-hood/software-uth-license) file for details.
## Acknowledgments
* [National Institutes of Health](https://www.nih.gov/) - [Big Data to Knowledge Initiative](https://datascience.nih.gov/bd2k)
* Grants: R01MD010362, 1UG1DA04030901, 1U54EB020404, 1R01CA190329, 1R01DE02524, R00MD010468, 3UH2DA041713, 10555SC
* [National Science Foundation](https://www.nsf.gov/)
* Grants: 1640813, 1722646
* [Intelligence Advanced Research Projects Activity](https://www.iarpa.gov/)
* Contract: 2017-17042800006
1 maintainer
Metadata
1 maintainer
Release files for cerebralcortex-kernel 3.0.0.post13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cerebralcortex-kernel-3.0.0.post13.tar.gz | 51.7 kB | Without provenance | Details for cerebralcortex-kernel-3.0.0.post13.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cerebralcortex_kernel-3.0.0.post13-py3-none-any.whl | Python 3 | none | any | Details for cerebralcortex_kernel-3.0.0.post13-py3-none-any.whl |
Total release size: 160.8 kB
Release files / cerebralcortex-kernel-3.0.0.post13.tar.gz
| Download URL | cerebralcortex-kernel-3.0.0.post13.tar.gz |
|---|---|
| Size | 51.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c7fd441d75c7a1f6eb53a7a885dc9a6b9708e0e1ce058bf87a06babf4b048f7e
|
|
BLAKE2b-256 checksum How to use checksums |
a474cff7b0873a2a66e58754eaa26d987c9a7410cf5af6d8c4dbdc86911ff430
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.7
|
Release files / cerebralcortex_kernel-3.0.0.post13-py3-none-any.whl
| Download URL | cerebralcortex_kernel-3.0.0.post13-py3-none-any.whl |
|---|---|
| Size | 109.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bff3d98629596465f3b758f71483623317cbe79baf286e5b21e32b00371ade02
|
|
BLAKE2b-256 checksum How to use checksums |
2390286b150d30ce432bb4ce6c68bd1dd1fd29586daae468d05b8d02a2c90a81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.7
|
Security for cerebralcortex-kernel 3.0.0.post13
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
