lookout-sdk 0.2.0

SDK for writing lookout analyzers

lookout SDK

What is lookout SDK?

Lookout SDK is a toolkit for writing new analyzers for the lookout.

What does it provide

It provides to an analyzer an easy access to the DataService API though a gRPC service.

DataService abstracts all data access and details of dealing with actual Git repositories, UAST extraction, programming language detection, etc. This way the analyzer can focus focus only on source code analysis logic.

The architecture of lookout and its components are described in src-d/lookout/docs

SDK includes:

• proto definitions
• pre-generated libraries code for Golang and Python
• low-level helpers to workaround protobuf/gRPC caveats. Check go documentation
• quickstart documentation on how to write an analyzer

How to create a new Analyzer

Essentially, every analyzer is just a gRPC server that implements Analyzer service. Lookout itself acts as a gRPC client for this server and it will push the analyzer server whenever a new Pull Request is ready for analysis.

Golang

Steps:

• use pre-generated code for gRPC from gopkg.in/src-d/lookout-sdk.v0/pb
• implement Analyzer service interface. Example:

  NotifyReviewEvent(ctx context.Context, review *pb.ReviewEvent) (*pb.EventResponse, error)
  NotifyPushEvent(context.Context, *pb.PushEvent) (*pb.EventResponse, error)
  

  • analyzer should request a stream of files and UASTs from DataService that lookout exposes, by default, on localhost:10301
  • analyzer has options to ask either for all files, or just the changed ones, as well as UASTs, language, full file content and/or exclude some paths: by regexp, or just all vendored paths
  • analyzer has to return a list of Comment messages
• run gRPC server to listen for requests from the lookout

SDK contains a quickstart example of an Analyzer that detects language and number of functions for every file language-analyzer.go:

• go get -u .
• go run language-analyzer.go

Python

• pip install lookout-sdk
• use pre-generated code for gRPC from lookout_sdk library
• implement Analyzer class that extends AnalyzerServicer. Example:

  def NotifyReviewEvent(self, request, context):
  def NotifyPushEvent(self, request, context):
  

  • analyzer should request a stream of files and UASTs from DataService that lookout exposes, by default, on localhost:10301
  • analyzer has options to ask either for all files, or just the changed ones, as well as UASTs, language, full file content and/or exclude some paths: by regexp, or just all vendored paths
  • analyzer has to return a list of Comment messages
• start grpc server and add Analyzer instance to it

SDK contains a quickstart example of an Analyzer that detects language and number of functions for every file language-analyzer.py:

• python3 language-analyzer.py

How to test analyzer

One can test analyzer locally without the need for Github access and a full lookout server installation using lookout-sdk binary. You can think about it as curl-like tool to call an analyzer gRPC endponts. For convenience, it also exposes a DataServer backed by a git repository in local FS.

• get lookout-sdk binary from src-d/lookout releases
• run bblfshd
• build and start analyzer e.g. Golang
  • go get -u .
  • go run language-analyzer.go , or Python
  • pip install lookout-sdk
  • python3 language-analyzer.py
• test without Github access, on the latest commit in some Git repository in local FS

  $ lookout-sdk review \
    --log-level=debug \
    --git-dir="$GOPATH/src/gopkg.in/src-d/lookout-sdk.v0" \
    "ipv4://localhost:2020"
  

this will create a "mock" Review event and notify the analyzer, as if you were creating a Pull Request from HEAD~1.

Check src-d/lookout for further details on lookout-sdk binary CLI options.

Extracting UAST from arbitrary code

Lookout server and lookout-sdk binary proxy bblfsh gRPC protocol through DataServer.

Please check how to create client in bblfsh documentation and use DataServer address to connect.

Caveats

• client: disable secure connection on dialing with grpc.WithInsecure()
• client: set max gRPC message size:
  • go: use pb.DialContext instead.
  • python: use lookout.sdk.grpc.create_channel instead.
• client: turn off gRPC fail-fast mode If your analyzer greedy creates a connection to DataServer before one was actually started, you might want to disable fail-fast mode. This way the RPCs are queued until the chanel ready. Here is an example.
• client/server RFC 3986 URI scheme support:
  • go: use pb.ToGoGrpcAddress and pb.Listen
  • python: use lookout.sdk.grpc.to_grpc_address

Release Process

• Make sure the code is up to date using make protogen.
• Update VERSION in python/setup.py with the same version that you will use for the tag (manual step required until #2 is implemented).
• Create the release tag.

License

Apache License v2.0