Kqlmagic: KQL (Kusto Query Language), enable query Azure Monitor data from a Jupyter compatibe notebook using magic
Project description
.. image:: https://mybinder.org/badge.svg :target: https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStart.ipynb
.. image:: https://mybinder.org/badge.svg :target: https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStartAI.ipynb
jupyter-Kqlmagic
===========
Extension (Magic) to Jupyter notebook and Jupyter lab, that enable notebook experience working with Kusto, ApplicationInsights, and LogAnalytics data.
===========
:Author: Michael Binshtock, mbnshtck@gmail.com
Introduces a %kql (or %%kql) magic.
Connect to kusto, using a connect strings, then issue KQL
commands within IPython or IPython Notebook.
.. image:: https://github.com/mbnshtck/jupyter-kql-magic/master/examples/writers.png
:width: 600px
:alt: screenshot of jupyter-kql-magic in the Notebook
Examples
--------
.. code-block:: python
In [1]: %load_ext kql
In [2]: %%kql kusto://username('myname').password('mypassword').cluster('mycluster').database('mydatabase')
...: character
...: | where abbrev = 'ALICE'
...:
Out[2]: [(u'Alice', u'Alice', u'ALICE', u'a lady attending on Princess Katherine', 22)]
In [3]: result = _
In [4]: print(result)
charid charname abbrev description speechcount
=================================================================================
Alice Alice ALICE a lady attending on Princess Katherine 22
In [4]: result.columns_name
Out[5]: [u'charid', u'charname', u'abbrev', u'description', u'speechcount']
In [6]: result[0][0]
Out[6]: u'Alice'
In [7]: result[0].description
Out[7]: u'a lady attending on Princess Katherine'
After the first connection, connect info can be omitted::
In [8]: %kql work | count
Out[8]: [(43L)]
Connections to multiple databases can be maintained. You can refer to
an existing connection by database@cluster
.. code-block:: python
In [9]: %%kql mydatabase1@mycluster
...: character
...: | where speechcount = (character | summarize max(speechcount))
...: | project charname, speechcount
...:
Out[9]: [(u'Poet', 733)]
In [10]: print(_)
charname speechcount
======================
Poet 733
If no connect string is supplied, ``%kql`` will provide a list of existing connections;
however, if no connections have yet been made and the environment variable ``KQLMAGIC_CONNECTION_STR``
is available, that will be used.
For secure access, you may dynamically access your credentials (e.g. from your system environment or `getpass.getpass`) to avoid storing your password in the notebook itself. Use the `$` before any variable to access it in your `%kql` command.
.. code-block:: python
In [11]: user = os.getenv('SOME_USER')
....: password = os.getenv('SOME_PASSWORD')
....: connection_string = "kusto://username('{user}'.password('{password}').cluster('some_cluster').database('some_database')".format(user=user, password=password)
....: %kql $connection_string
Out[11]: u'Connected: some_database@some_cluster'
You may use multiple Kql statements inside a single cell, but you will
only see any query results from the last of them, so this really only
makes sense for statements with no output
.. code-block:: python
In [11]: %%kql
....: work | limit 1
....: work | count
....:
Out[11]: [(43L)]
Bind variables (bind parameters) can be used in the "named" (:x) style.
The variable names used should be defined in the local namespace
.. code-block:: python
In [12]: name = 'Countess'
In [13]: %kql select description from character where charname = :name
Out[13]: [(u'mother to Bertram',)]
As a convenience, dict-style access for result sets is supported, with the
leftmost column serving as key, for unique values.
.. code-block:: python
In [14]: result = %kql work
In [15]: result['richard2']
Out[15]: (u'richard2', u'Richard II', u'History of Richard II', 1595, u'h', None, u'Moby', 22411, 628)
Results can also be retrieved as an iterator of dictionaries (``result.dicts_iterator()``)
or a single dictionary with a tuple of scalar values per key (``result.to_dict()``)
Assignment
----------
Ordinary IPython assignment works for single-line `%kql` queries:
.. code-block:: python
In [16]: works = %kql work | project title, year
The `<<` operator captures query results in a local variable, and
can be used in multi-line ``%%kql``:
.. code-block:: python
In [17]: %%kql works << work
...: | project title, year
...:
Returning data to local variable works
Connecting
----------
Some example connection strings::
kusto://username('username').password('password').cluster('clustername').database('databasename')
kusto://username('username').password('password').cluster('clustername')
kusto://username('username').password('password')
kusto://cluster('clustername').database('databasename')
kusto://cluster('clustername')
kusto://database('databasename')
Configuration
-------------
Query results are loaded as lists, so very large result sets may use up
your system's memory and/or hang your browser. There is no auto_limit
by default. However, `auto_limit` (if set) limits the size of the result
set (usually with a `LIMIT` clause in the KQL). `display_limit` is similar,
but the entire result set is still pulled into memory (for later analysis);
only the screen display is truncated.
.. code-block:: python
In [2]: %config Kqlmagic
Kqlmagic options
--------------
Kqlmagic.auto_limit=<Int>
Current: 0
Automatically limit the size of the returned result sets
Kqlmagic.auto_dataframe=<Bool>
Current: False
Return Pandas DataFrames instead of regular result sets
Kqlmagic.display_limit=<Int>
Current: 0
Automatically limit the number of rows displayed (full result set is still
stored)
Kqlmagic.feedback=<Bool>
Current: True
Print number of records returned, and assigned variables
Kqlmagic.short_errors=<Bool>
Current: True
Don't display the full traceback on KQL Programming Error
Kqlmagic.prettytable_style=<Unicode>
Current: 'DEFAULT'
Set the table printing style to any of prettytable's defined styles
(currently DEFAULT, MSWORD_FRIENDLY, PLAIN_COLUMNS, RANDOM)
In[3]: %config Kqlmagic.feedback = False
Please note: if you have auto_dataframe set to true, the option will not apply. You can set the pandas display limit by using the pandas ``max_rows`` option as described in the `pandas documentation <http://pandas.pydata.org/pandas-docs/version/0.18.1/options.html#frequently-used-options>`_.
Pandas
------
If you have installed ``pandas``, you can use a result set's
``.DataFrame()`` method
.. code-block:: python
In [3]: result = %kql character | where speechcount > 25
In [4]: dataframe = result.DataFrame()
.. _Pandas: http://pandas.pydata.org/
Graphing
--------
If you have installed ``matplotlib``, you can use a result set's
``.plot()``, ``.pie()``, and ``.bar()`` methods for quick plotting
.. code-block:: python
In[5]: result = %kql work | where genretype = 'c' | project title, totalwords
In[6]: %matplotlib inline
In[7]: result.pie()
.. image:: https://raw.github.com/catherinedevlin/ipython-sql/master/examples/wordcount.png
:alt: pie chart of word count of Shakespeare's comedies
Dumping
-------
Result sets come with a ``.csv(filename=None)`` method. This generates
comma-separated text either as a return value (if ``filename`` is not
specified) or in a file of the given name.
.. code-block:: python
In[8]: result = %kql work | where genretype = 'c' | project title, totalwords
In[9]: result.csv(filename='work.csv')
Installing
----------
Install the lastest release with::
pip install jupyter-kql-magic
or download from https://github.com/mbnshtck/jupyter-kql-magic and::
cd jupyter-kql-magic
sudo python setup.py install
Development
-----------
https://github.com/mbnshtck/jupyter-kql-magic
Credits
-------
- Kql_
- Kusto_
- Distribute_
- Buildout_
- modern-package-template_
.. _Distribute: http://pypi.python.org/pypi/distribute
.. _Buildout: http://www.buildout.org/
.. _modern-package-template: http://pypi.python.org/pypi/modern-package-template
.. _Kql: https://kusto.azurewebsites.net/docs/queryLanguage/query_language.html
.. _Kusto: https://kusto.azurewebsites.net/docs/
News
====
0.1.1
---
*Release date: 06-Sep-2017*
* Initial release
0.1.3
---
*Release date: 01-July-2018*
KQL Interfaces
* Query kusto
- Multi Tenant autheticate with client/secret or user/pw or code
* Query Application insight
- Authenticate with appid/appkey
* Query Log Analytics
- Authenticate with workspace/appkey
ConnectionString
* Explicit connection, that starts with one of the support schems: kusto://, appinsights://, loganalytics://
i.e. kusto://tenant('authority').username('name').password('pw').clustername('cname').database('dbname')
kusto://tenant('authority').clientid('cid').clientsecret('secret').clustername('cname').database('dbname')
kusto://tenant('authority').code().clustername('cname').database('dbname')
appinsights://appid('id').appkey('key')
loganalytics://workspace('ws').appkey('key')
* Reference to an already existing connection: database@cluster, appid@appinsights, workspace@loganalytics
* Reference to a section in config file, that contains all the fields of the connection string (DSN style), file name must be set in config.dsn_filename
* Substring of the form $name or ${name}, in windows also %name% if found in env variabes.
i.e. $MY_CONNECTION
${MY_CONNECTION}
%MY_CONNECTION%
* Current connection. If connection string is not specified, current (last) connection is used.
* Partial explicit connection string. If some fields are missing they are set to a default value if exist, or are taken from current (last) connection, or user is prompted.
i.e. if kusto://... is missing tenant(...) component, 'microsoft.com' is used
if kusto://... is missing credentials component (username.password or clientid/clientsecret or code()), credentials are taken from current(last) connection
if kusto://... is missing cluster(...) component, cluster value is taken from current (last) connection
if credential secret is missing, user is prompted to enter the credential secret
Authentication
* loganalytics://, workspace/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* appinsights://, appid/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* kusto://, username/password. If password is missing in connection string, user is prompted to enter password
* kusto://, clientid/clientsecret. If clientsecret is missing in connection string, user is prompted to enter clientsecret
* kusto://, code. code is displayed, and user is reffered to a the authentication page to enter the code.
ConnectionString validation
* If a single line kql magic (%kql) contains the connectionString part, but without the query part, an implicit validation query is used, to validate the connectionString
UI
* info, warning and error messages - are displayed in colors, blue, green, and brown
TODO
*Support parametrize based on local/global ns
Authetication
- Authenticate loganalytics with additional schemes (current authentication is appKey only which used mainly for demo)
- Solve ambiguity if kusto authentication to same cluster is definned by two differnt schemes, for example code and user/pw
- Make kusto code() authentication more user friendly
- Make user/pw authentication when used from Windows browser, automatic based on windows user/pw
- when missing components in connection string are taken from current connection, they should be taken from the same type of connection, if exist
Ingestion
- support basic ingestion to kusto, appinsight and loganalytics
Performance
- Add retries to kusto ,appinsights and loganalytics for some errors
- Add option to use WebGl in rendering using plotly Scattergl instead of scatter for increased speed (see plotly interface)
UI
- make kql errors that return from server easier to read (open json)
- detect unknown refernces to connection string of type db@cluster and provide a better result, instead of current error message
Connection Management
- if connection fails to connect either should be marked as such or shouldn't be in list
08/25/2018 - Kusto Connection to a new database on a cluster that already have a database connected, will skip authentication, and all databases on same cluster will use same authenticated client to that cluster.
The new connection_str can be either kusto://... or database@cluster
08/26/2018 - replaced show_conn_list option with show_conn_info that has 3 states: 'current', 'list' and 'None'
08/26/2018 - added the following methods:
display_table - will display the result table in the cell
display_chart - will display the result chart in the cell
display - will display the result in the cell
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Kqlmagic-0.1.50.tar.gz
(68.3 kB
view hashes)
Built Distribution
Kqlmagic-0.1.50-py3-none-any.whl
(71.0 kB
view hashes)
Close
Hashes for Kqlmagic-0.1.50-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 712cb27ab34ff5f211c22bfe1fc83c4bacee92b93630da529cdaab0efd9e2603 |
|
MD5 | 869f627fae49155146e9df3dab9239fd |
|
BLAKE2b-256 | 6dde5e5f61a064e17aa3642f25edc5a42958c76458d54e0d67b71b565f24e29e |