Skip to main content

Graphene Elasticsearch (DSL) integration

Project description

Elasticsearch (DSL) integration for Graphene.

PyPI Version Supported Python versions Build Status GPL-2.0-only OR LGPL-2.1-or-later Coverage

Prerequisites

  • Graphene 2.x. Support for Graphene 1.x is not planned, but might be considered.

  • Python 3.6, 3.7. Support for Python 2 is not intended.

  • Elasticsearch 6.x, 7.x. Support for Elasticsearch 5.x is not intended.

Main features and highlights

  • Implemented ElasticsearchConnectionField and ElasticsearchObjectType are the core classes to work with graphene.

  • Pluggable backends for searching, filtering, ordering, etc. Don’t like existing ones? Override, extend or write your own.

  • Implemented search backend.

  • Implemented filter backend.

  • Implemented ordering backend.

  • Implemented pagination.

See the Road-map for what’s yet planned to implemented.

Documentation

Documentation is available on Read the Docs.

Installation

Install latest stable version from PyPI:

pip install graphene-elastic

Or latest development version from GitHub:

pip install https://github.com/barseghyanartur/graphene-elastic/archive/master.zip

Examples

Install requirements

pip install -r requirements.txt

Populate sample data

The following command will create indexes for User and Post documents and populate them with sample data:

./scripts/populate_elasticsearch_data.sh

Sample document definition

search_index/documents/post.py

See examples/search_index/documents/post.py for full example.

import datetime
from elasticsearch_dsl import (
    Boolean,
    Date,
    Document,
    InnerDoc,
    Keyword,
    Nested,
    Text,
    Integer,
)

class Comment(InnerDoc):

    author = Text(fields={'raw': Keyword()})
    content = Text(analyzer='snowball')
    created_at = Date()

    def age(self):
        return datetime.datetime.now() - self.created_at


class Post(Document):

    title = Text(
        fields={'raw': Keyword()}
    )
    content = Text()
    created_at = Date()
    published = Boolean()
    category = Text(
        fields={'raw': Keyword()}
    )
    comments = Nested(Comment)
    tags = Text(
        analyzer=html_strip,
        fields={'raw': Keyword(multi=True)},
        multi=True
    )
    num_views = Integer()

    class Index:
        name = 'blog_post'
        settings = {
            'number_of_shards': 1,
            'number_of_replicas': 1,
            'blocks': {'read_only_allow_delete': None},
        }

Sample apps

Sample Flask app

Run the sample Flask app:

./scripts/run_flask.sh

Open Flask graphiql client

http://127.0.0.1:8001/graphql

Sample Django app

Run the sample Django app:

./scripts/run_django.sh runserver

Open Django graphiql client

http://127.0.0.1:8000/graphql

ConnectionField example

ConnectionField is the most flexible and feature rich solution you have. It uses filter backends which you can tie to your needs the way you want in a declarative manner.

Sample schema definition

import graphene
from graphene_elastic import (
    ElasticsearchObjectType,
    ElasticsearchConnectionField,
)
from graphene_elastic.filter_backends import (
    FilteringFilterBackend,
    SearchFilterBackend,
    OrderingFilterBackend,
    DefaultOrderingFilterBackend,
)
from graphene_elastic.constants import (
    LOOKUP_FILTER_PREFIX,
    LOOKUP_FILTER_TERM,
    LOOKUP_FILTER_TERMS,
    LOOKUP_FILTER_WILDCARD,
    LOOKUP_QUERY_EXCLUDE,
    LOOKUP_QUERY_IN,
)

# Object type definition
class Post(ElasticsearchObjectType):

    class Meta(object):
        document = PostDocument
        interfaces = (Node,)
        filter_backends = [
            FilteringFilterBackend,
            SearchFilterBackend,
            OrderingFilterBackend,
            DefaultOrderingFilterBackend,
        ]

        # For `FilteringFilterBackend` backend
        filter_fields = {
            # The dictionary key (in this case `title`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value could be simple or complex structure (in this case
            # complex). The `field` key points to the `title.raw`, which
            # is the field name in the Elasticsearch document
            # (`PostDocument`). Since `lookups` key is provided, number
            # of lookups is limited to the given set, while term is the
            # default lookup (as specified in `default_lookup`).
            'title': {
                'field': 'title.raw',
                # Available lookups
                'lookups': [
                    LOOKUP_FILTER_TERM,
                    LOOKUP_FILTER_TERMS,
                    LOOKUP_FILTER_PREFIX,
                    LOOKUP_FILTER_WILDCARD,
                    LOOKUP_QUERY_IN,
                    LOOKUP_QUERY_EXCLUDE,
                ],
                # Default lookup
                'default_lookup': LOOKUP_FILTER_TERM,
            },

            # The dictionary key (in this case `category`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `category.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'category': 'category.raw',

            # The dictionary key (in this case `tags`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `tags.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'tags': 'tags.raw',

            # The dictionary key (in this case `num_views`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `num_views`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'num_views': 'num_views',
        }

        # For `SearchFilterBackend` backend
        search_fields = {
            'title': {'boost': 4},
            'content': {'boost': 2},
            'category': None,
        }

        # For `OrderingFilterBackend` backend
        ordering_fields = {
            # The dictionary key (in this case `tags`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `tags.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'title': 'title.raw',

            # The dictionary key (in this case `created_at`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `created_at`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'created_at': 'created_at',

            # The dictionary key (in this case `num_views`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `num_views`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'num_views': 'num_views',
        }

        # For `DefaultOrderingFilterBackend` backend
        ordering_defaults = (
            '-num_views',  # Field name in the Elasticsearch document
            'title.raw',  # Field name in the Elasticsearch document
        )

# Query definition
class Query(graphene.ObjectType):
    all_post_documents = ElasticsearchConnectionField(Post)

# Schema definition
schema = graphene.Schema(query=Query)
Filter
Sample queries

Since we didn’t specify any lookups on category, by default all lookups are available and the default lookup would be term. Note, that in the {value:"Elastic"} part, the value stands for default lookup, whatever it has been set to.

query PostsQuery {
  allPostDocuments(filter:{category:{value:"Elastic"}}) {
    edges {
      node {
        id
        title
        category
        content
        createdAt
        comments
      }
    }
  }
}

But, we could use another lookup (in example below - terms). Note, that in the {terms:["Elastic", "Python"]} part, the terms is the lookup name.

query PostsQuery {
  allPostDocuments(
        filter:{category:{terms:["Elastic", "Python"]}}
    ) {
    edges {
      node {
        id
        title
        category
        content
        createdAt
        comments
      }
    }
  }
}

Or apply a gt (range) query in addition to filtering:

{
  allPostDocuments(filter:{
        category:{term:"Python"},
        numViews:{gt:"700"}
    }) {
    edges {
      node {
        category
        title
        comments
        numViews
      }
    }
  }
}
Implemented filter lookups

The following lookups are available:

  • contains

  • ends_with (or endsWith for camelCase)

  • exclude

  • exists

  • gt

  • gte

  • in

  • is_null (or isNull for camelCase)

  • lt

  • lte

  • prefix

  • range

  • starts_with (or startsWith for camelCase)

  • term

  • terms

  • wildcard

See dedicated documentation on filter lookups for more information.

Ordering

Possible choices are ASC and DESC.

query {
  allPostDocuments(
        filter:{category:{term:"Photography"}},
        ordering:{title:ASC}
    ) {
    edges {
      node {
        category
        title
        content
        numViews
        tags
      }
    }
  }
}
Pagination

The first, last, before and after arguments are supported. By default number of results is limited to 100.

query {
  allPostDocuments(first:12) {
    pageInfo {
      startCursor
      endCursor
      hasNextPage
      hasPreviousPage
    }
    edges {
      cursor
      node {
        category
        title
        content
        numViews
      }
    }
  }
}

Road-map

Road-map and development plans.

Lots of features are planned to be released in the upcoming Beta releases:

  • Geo-spatial backend

  • Filter lookup geo_bounding_box (or geoBoundingBox for camelCase)

  • Filter lookup geo_distance (or geoDistance for camelCase)

  • Filter lookup geo_polygon (or geoPolygon for camelCase)

  • Aggregations (faceted search) backend

  • Post-filter backend

  • Nested backend

  • Highlight backend

  • Suggester backend

  • Global aggregations backend

  • More-like-this backend

  • Complex search backends, such as Simple query search

  • Source filter backend

Stay tuned or reach out if you want to help.

Testing

Project is covered with tests.

By defaults tests are executed against the Elasticsearch 7.x.

Running Elasticsearch

Run Elasticsearch 7.x with Docker

docker-compose up elasticsearch

Running tests

Make sure you have the test requirements installed:

pip install -r requirements/test.txt

To test with all supported Python versions type:

tox

To test against specific environment, type:

tox -e py37

To test just your working environment type:

./runtests.py

To run a single test module in your working environment type:

./runtests.py src/graphene_elastic/tests/test_filter_backend.py

To run a single test class in a given test module in your working environment type:

./runtests.py src/graphene_elastic/tests/test_filter_backend.py::FilterBackendElasticTestCase

Debugging

For development purposes, you could use the flask app (easy to debug). Standard pdb works (import pdb; pdb.set_trace()). If ipdb does not work well for you, use ptpdb.

Writing documentation

Keep the following hierarchy.

=====
title
=====

header
======

sub-header
----------

sub-sub-header
~~~~~~~~~~~~~~

sub-sub-sub-header
^^^^^^^^^^^^^^^^^^

sub-sub-sub-sub-header
++++++++++++++++++++++

sub-sub-sub-sub-sub-header
**************************

License

GPL-2.0-only OR LGPL-2.1-or-later

Support

For any issues contact me at the e-mail given in the Author section.

Author

Artur Barseghyan <artur.barseghyan@gmail.com>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

graphene-elastic-0.1.tar.gz (76.1 kB view hashes)

Uploaded Source

Built Distribution

graphene_elastic-0.1-py2.py3-none-any.whl (106.0 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page