opal-server 0.0.13

OPAL is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and data and pushing live updates to your agents. The opal-server creates a pub/sub channel clients can subscribe to (i.e: acts as coordinator). The server also tracks a git repository (via webhook) for updates to policy (or static data) and accepts continuous data update notifications via REST api, which are then pushed to clients.

⚡OPAL⚡

Open Policy Administration Layer

OPAL is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and data and pushing live updates to your agents.

OPAL brings open-policy up to the speed needed by live applications. As you push updates to your application's stores (e.g. Git, DBs, S3, SaaS services) OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need.)

Table of contents

• Getting started
• Intro to OPAL
• Architecture
• Key-Concepts
• HOW-TOs
• Design choices
• Join the Community

🛠️ Getting started

• As containers

  • Server

    docker pull authorizon/opal-server
    

    docker run authorizon/opal-server
    

  • Client (prebuilt with OPA inside)

    docker pull authorizon/opal-client
    

    docker run authorizon/opal-client
    

• Or install as directly as Python packages (python >3.7 )

  • pip install opal-client
  • pip install opal-server

📖 Introduction to OPAL - data and policy realtime delivery

• Modern applications are complex, distributed, multi-tenant and serve at scale - creating (often) overwhelming authorization challenges. OPA (Open-Policy-Agent) brings the power of decoupled policy to the infrastructure layer (especially K8s), and light applications. OPAL supercharges OPA to meet the pace of live applications, where the picture may change with every user click and api call.

• OPAL builds on top of OPA adding realtime updates (via Websocket Pub/Sub) for both policy and data.

• OPAL embraces decoupling of policy and code, and doubles down on decoupling policy (Git driven) and data (distributed data-source fetching engines).

Why use OPAL

• OPAL is the easiest why to keep your solution's authorization layer up-to-date in realtime.
  • OPAL aggregates policy and data from across the field and integrates them seamlessly into the authorization layer.
  • OPAL is microservices and cloud-native (see key concepts below)

Why OPA + OPAL == 💪💜

OPA (OpenPolicyAgent) is great! it decouples policy from code in a highly performant and elegant way. But keeping policy agents up to the date is hard- especially in applications - where each user interaction or API call can affect access-control. OPAL runs in the background - supercharging policy-agents, keeping them in sync with events in realtime.

What OPAL is not

• A Policy Engine:

  • OPAL uses a policy-engine, but isn't one itself
  • Check out Open-Policy-Agent, and OSO

• Large scale Global FGA:

  • Currently OPAL is not meant for managing ridiculous (>100GB) amounts of data within one layer. Though it can complement a CDN to achieve a similar result - see below.
  • Check out Google-Zanzibar

• Fullstack authorization:

  • OPAL and OPA essentially provide microservices for authorization
  • Developers still need to add control interfaces on top (e.g. user-management, api-key-management, audit, impersonation, invites) both as APIs and UIs
  • Check out authorizon

📡 Architecture

See a more detailed diagram

• OPAL consists of two key components that work together:

  1. OPAL Server

     • Creates a Pub/Sub channel client's subscribe to
     • Tracks a Git repository (via webhook) for updates to policy (or static data)
       • Additional versioned repositories can be supported (e.g. S3, SVN)
     • Accepts data update notifications via Rest API
     • pushes updates to clients (as diffs)
     • scales with other server instances via a configurable backbone pub/sub (Postgre, Redis, Kafka , (more options to be added) )

  2. OPAL Client -

     • Deployed alongside a policy-agent keeping it up to date
     • Subscribes to Pub/Sub updates by topics for data and policy
     • Downloads data source configurations from server
       • Fetches data from multiple sources (DBs, APIs, 3rd party services)
     • Downloads policy from server
     • Keeps policy agents up to date

• Further reading

  • Architecture deep dive
  • Code modules review

💡 Key Concepts

• OPAL is realtime (with Pub/Sub updates)

  • OPAL is all about easily managing your authorization layer in realtime.
  • This is achieved by a Websocket Pub/Sub channel between OPAL clients and servers.
  • Each OPAL-client (and through it each policy agent) subscribes to and receives updates instantly

• OPAL is stateless

  • OPAL is designed for scale, mainly via scaling out both client and server instances, as such neither are stateful.
  • State is retained in the end components (the policy agent) and source components (e.g. GIT, databases, API servers)

• OPAL is extendable

  • OPAL's Pythonic nature makes extending and embedding it extremely easy.
  • Built with typed Python3, Pydantic, and FastAPI - OPAL is balanced just right for stability and fast development.
  • A key example is OPAL's FetchingEngine and FetchProviders. Want to use authorization data from a new source (a SaaS service, a new DB, your own proprietary solution)? Simply implement a new fetch-provider

👩‍🏫 HOW-TOs

• How to extend OPAL to fetch data from your sources with FetchProviders

🗿 Foundations

OPAL is built on the shoulders of open-source giants, including:

• OpenPolicyAgent- the default policy agent managed by OPAL.
• FastAPI - the ASGI server framework used by OPAL-servers and OPAL-clients.
• FastAPI WS PubSub - powering the live realtime update channels
• Broadcaster allowing syncing server instances through a backend backbone (e.g. Redis, Kafka)

🎨 Design choices

• Networking

  • OPAL creates a highly efficient communications channel Using websocket Pub/Sub connections to subscribe to both data and policy updates allows OPAL clients (and the services they support) to be deployed anywhere - in your VPC, at the edge, on-premises, etc.
  • By using outgoing websocket connections to establish the Pub/Sub channel most routing/firewall concerns are circumnavigated
  • Using Websocket connections allows network connections to be dormant most fo the time (saving CPU cycles for both clients and servers) - especially when comparing to polling based methods.

• Implementation with Python

  • OPAL is written completely in Python3 using FastAPI and Pydantic. OPAL was initially created as a component of authorizon.com, and we've chosen Python for development speed, ease of use and extendability (e.g. Fetcher providers).
  • Python3 with coroutines (Asyncio) and FastAPI has presented significant improvements for Python server performance. While still not on par with GO or Rust - the results match and in some cases even surpass NodeJS.

• Performance

  • It's important to note that OPAL doesn't replace the direct channel to the policy-engine - so for example with OPA all authorization queries are processed directly by OPA's GO based engine.

  • Pub/Sub benchmarks - While we haven't run thorough benchmarks yet- we are using OPAL in production- seeing its Pub/Sub channel handle 100s of events per second per server instance with no issue.

• Decouple Data from Policy

  • Open-Policy-Agent sets the stage for authorization-data and policy decoupling by providing a separate API to manage each.
  • OPAL takes this approach a step forward by enabling independent update channels for each into the policy cache.
  • POLICY - Policy as code - is code, and as such is naturally maintained best within version control (e.g. GIT). OPAL allows open-policy agents to get the subset of policy they need directly from repositories (as part of CI/CD or independently)
  • DATA - OPAL takes a more distributed approach to authorization data - recognizing that there are many potential data sources we'd like to include in the authorization conversion (e.g. billing data, compliance data, usage data, etc. etc. ). OPAL-clients can be configured and extended to aggregate any data-source into whichever service needs it.

• Decouple data/policy management from policy agents

  • OPAL was built initially with OPA in mind, and OPA is mostly a first-class citizen in OPAL. That said OPAL can support various and multiple policy agents, even in parallel - allowing developers to choose the best policy and agent for their needs.

• FGA, large scale / global authorization (e.g. Google Zanzibar)

  • OPAL is built for fine grained authorizon (FGA), allowing developers to aggregate all and any the data they need and restructure it for the authorization layer
  • By making sure each policy-agent is loaded with only the data it needs (via topic subscriptions) - i.e. data focus and separation.
    • examples of data-separation: the back-office service doesn't need to know about customer users, a tenant specific service doesn't need the user list of other tenants, ...
  • That said OPAL with OPA as it is - is still limited in capacity of authorization data that can be available per policy decision; but after applying said data focus this will only impact applications with truly insane scales (where the data for every decision is larger than ~5GB).
  • For these larger scale cases, OPAL can potentially become a link between a solution like Google Zanzibar (or equivalent CDN) and local policy-agents. (Allowing both Google like scales, low latency, and high performance)
  • I you're developing such a service, or considering such high-scale scenarios; you're welcome to contact us, we'd be happy to share our plans for OPAL in that area.

• Using OPAL for other live update needs

  • While OPAL was created and primarily designed for open-policy and authorization needs; it can be generically applied for other live updates and data/code propagation needs
  • If you'd like to use OPAL or some of its underlying modules for other update cases - please contact us (See below), we'd love to help you do that.

• Administration capabilities and UI

  • We've already built policy editors, back-office, frontend-embeddable interfaces, and more as part of authorizon.com
  • We have plans to migrate more parts of authorizon.com to be open-source; please let us know what you'd like to see next

Joining the community

• We are eager to hear from you 😃
• Raise questions and ask for features to be added to the road-map in our Github discussions
• Report issues in Github issues
• Chat with us in our Slack community

Contacting us (the authors)

• We love talking about authorization, open-source, realtime communication, and tech in general
• feel free to reach out to us on our GitHub discussions or directly over email

Contributing to OPAL

• Pull requests are welcome! (please make sure to include passing tests and docs)
• Prior to submitting a PR - open an issue on GitHub, or make sure your PR addresses an existing issue well.