opal-common-fork 0.1.25

OPAL is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and data and pushing live updates to your agents. opal-common contains common code used by both opal-client and opal-server.

⚡OPAL⚡

Open Policy Administration Layer

OPAL is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents.

OPAL brings open-policy up to the speed needed by live applications. As your application state changes (whether it's via your APIs, DBs, git, S3 or 3rd-party SaaS services), OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need).

Check out our main site at OPAL.ac
and this Microsoft video briefly explaining OPAL and how it works with OPA.

Table of contents

• Getting Started
• Intro to OPAL
  • Why use OPAL
• Architecture
• Key Concepts
• HOW-TOs
• Design Choices
• Join the Community

🛠️ Getting started

OPAL is available both as python packages with a built-in CLI as well as pre-built docker images ready-to-go.

Getting started with the pre-built docker containers

• Play with a live playground environment in docker-compose
• Try the getting started guide for containers
• Check out the Helm Chart for Kubernetes

Getting started with the python packages and CLI

• Install
  • pip install opal-client
  • pip install opal-server
• Run server (example):

  # Run server 
  #  in secure mode -verifying client JWTs (Replace secrets with actual secrets ;-) )
  export OPAL_AUTH_PRIVATE_KEY=~/opal 
  export OPAL_AUTH_PUBLIC_KEY=~/opal.pub 
  export OPAL_AUTH_MASTER_TOKEN="RANDOM-SECRET-STRING"
  #  Watching a GIT repository from a webhook
  export OPAL_POLICY_REPO_URL=https://github.com/authorizon/opal-example-policy-repo.git
  export OPAL_POLICY_REPO_WEBHOOK_SECRET="RANDOM-SECRET-STRING-SHARED-WITH-GITHUB"
  opal-server run
  

• Run client (example):

  # Run client
  #  authenticating with a JWT (replace 'JWT-CRYPTOGRAPHIC-CONTENT' with actual token )
  export OPAL_CLIENT_TOKEN="JWT-CRYPTOGRAPHIC-CONTENT"
  # connect to server
  export OPAL_SERVER_URL=https://opal.mydomain.com:7002
  # Subscribe to specific data-topics
  export OPAL_DATA_TOPICS=tenants/my-org,stripe_billing,tickets
  opal-client run
  

• Try it yourself - Read the getting started guide

📖 Introduction to OPAL - data and policy realtime delivery

• Modern applications are complex, distributed, multi-tenant and run at scale - often creating overwhelming authorization challenges. OPA (Open-Policy-Agent) brings the power of decoupled policy to the infrastructure layer (especially K8s), and light applications. OPAL supercharges OPA to meet the pace of live applications, where the state relevant to authorization decisions may change with every user click and api call.

• OPAL builds on top of OPA adding realtime updates (via Websocket Pub/Sub) for both policy and data.

• OPAL embraces decoupling of policy and code, and doubles down on decoupling policy (git driven) and data (distributed data-source fetching engines).

Why use OPAL

• OPAL is the easiest way to keep your solution's authorization layer up-to-date in realtime.
  • OPAL aggregates policy and data from across the field and integrates them seamlessly into the authorization layer.
  • OPAL is microservices and cloud-native (see key concepts below)

Why OPA + OPAL == 💪 💜

OPA (Open Policy Agent) is great! It decouples policy from code in a highly-performant and elegant way. But the challege of keeping policy agents up-to-date is hard - especially in applications - where each user interaction or API call may affect access-control decisions. OPAL runs in the background, supercharging policy-agents, keeping them in sync with events in realtime.

What OPAL is not

• A Policy Engine:

  • OPAL uses a policy-engine, but isn't one itself
  • Check out Open-Policy-Agent, and OSO

• Large scale Global FGA:

  • Currently OPAL is not meant for managing ridiculous (>100GB) amounts of data within one layer. Though it can complement a CDN to achieve a similar result - see below.
  • Check out Google-Zanzibar

• Fullstack authorization:

  • OPAL and OPA essentially provide microservices for authorization
  • Developers still need to add control interfaces on top (e.g. user-management, api-key-management, audit, impersonation, invites) both as APIs and UIs
  • Check out authorizon

📡 Architecture

See a more detailed diagram

• OPAL consists of two key components that work together:

  1. OPAL Server

     • Creates a Pub/Sub channel clients subscribe to
     • Tracks a git repository (via webhook / polling) for updates to policy (or static data)
       • Additional versioned repositories can be supported (e.g. S3, SVN)
     • Accepts data update notifications via Rest API
     • pushes updates to clients (as diffs)
     • scales with other server instances via a configurable backbone pub/sub (Currently supported: Postgres, Redis, Kafka; more options will be added in the future)

  2. OPAL Client

     • Deployed alongside a policy-agent, and keeping it up to date
     • Subscribes to Pub/Sub updates, based on topics for data and policy
     • Downloads data-source configurations from server
       • Fetches data from multiple sources (e.g. DBs, APIs, 3rd party services)
     • Downloads policy from server
     • Keeps policy agents up to date

• Further reading

  • Architecture deep dive
  • Code modules review

💡 Key Concepts

• OPAL is realtime (with Pub/Sub updates)

  • OPAL is all about easily managing your authorization layer in realtime.
  • This is achieved by a Websocket Pub/Sub channel between OPAL clients and servers.
  • Each OPAL-client (and through it each policy agent) subscribes to and receives updates instantly.

• OPAL is stateless

  • OPAL is designed for scale, mainly via scaling out both client and server instances; as such, neither are stateful.
  • State is retained in the end components (i.e: the OPA agent, as an edge cache) and data-sources (e.g. git, databases, API servers)

• OPAL is extensible

  • OPAL's Pythonic nature makes extending and embedding new components extremely easy.
  • Built with typed Python3, pydantic, and FastAPI - OPAL is balanced just right for stability and fast development.
  • A key example is OPAL's FetchingEngine and FetchProviders. Want to use authorization data from a new source (a SaaS service, a new DB, your own proprietary solution)? Simply implement a new fetch-provider.

👩‍🏫 HOW-TOs

• How to get started with OPAL (Packages and CLI)
• How to get started with OPAL (Container Images)
• How to trigger Data Updates via OPAL
• How to extend OPAL to fetch data from your sources with FetchProviders

🗿 Foundations

OPAL is built on the shoulders of open-source giants, including:

• Open Policy Agent- the default policy agent managed by OPAL.
• FastAPI - the ASGI API framework used by OPAL-servers and OPAL-clients.
• FastAPI Websocket PubSub - powering the live realtime update channels
• Broadcaster allowing syncing server instances through a backend backbone (e.g. Redis, Kafka)

🎨 Design choices

• Networking

  • OPAL creates a highly efficient communications channel using websocket Pub/Sub connections to subscribe to both data and policy updates. This allows OPAL clients (and the services they support) to be deployed anywhere - in your VPC, at the edge, on-premises, etc.
  • By using outgoing websocket connections to establish the Pub/Sub channel most routing/firewall concerns are circumnavigated.
  • Using Websocket connections allows network connections to stay idle most of the time, saving CPU cycles for both clients and servers (especially when comparing to polling-based methods).

• Implementation with Python

  • OPAL is written completely in Python3 using asyncio, FastAPI and Pydantic. OPAL was initially created as a component of authorizon.com, and we've chosen Python for development speed, ease of use and extensibility (e.g. fetcher providers).
  • Python3 with coroutines (Asyncio) and FastAPI has presented significant improvements for Python server performance. While still not on par with Go or Rust - the results match and in some cases even surpass Node.js.

• Performance

  • It's important to note that OPAL doesn't replace the direct channel to the policy-engine - so for example with OPA all authorization queries are processed directly by OPA's Go based engine.

  • Pub/Sub benchmarks - While we haven't run thorough benchmarks yet, we are using OPAL in production - seeing its Pub/Sub channel handle 100s of events per second per server instance with no issue.

• Decouple Data from Policy

  • Open Policy Agent sets the stage for policy code and policy data decoupling by providing separate APIs to manage each.
  • OPAL takes this approach a step forward by enabling independent update channels for policy code and policy data, mutating the policy agent cache separately.
  • Policy (Policy as code): is code, and as such is naturally maintained best within version control (e.g. git). OPAL allows OPA agents to subscribe to the subset of policy that they need directly from source repositories (as part of CI/CD or independently).
  • Data: OPAL takes a more distributed approach to authorization data - recognizing that there are many potential data sources we'd like to include in the authorization conversation (e.g. billing data, compliance data, usage data, etc etc). OPAL-clients can be configured and extended to aggregate data from any data-source into whichever service needs it.

• Decouple data/policy management from policy agents

  • OPAL was built initially with OPA in mind, and OPA is mostly a first-class citizen in OPAL. That said OPAL can support various and multiple policy agents, even in parallel - allowing developers to choose the best policy agent for their needs.

• FGA, large scale / global authorization (e.g. Google Zanzibar)

  • OPAL is built for fine grained authorizon (FGA), allowing developers to aggregate all and any data they need and restructure it for the authorization layer.
  • OPAL achieves this by making sure each policy-agent is loaded with only the data it needs via topic subscriptions (i.e: data focus and separation).
    • Examples of data separation: the back-office service doesn't need to know about customer users, a tenant specific service doesn't need the user list of other tenants, ...
  • That said OPAL is still limited by OPA's resource utilization capacity.
    • If the size of the dataset you need to load into OPA cache is huge (i.e: > 5GB), you may opt to pass this specific dataset by overloading input to your policy.
    • OPAL can still help you if you decide to shard your dataset across multiple OPA agents. Each agent's OPAL-client can subscribe only to the relevant shard.
  • For these larger scale cases, OPAL can potentially become a link between a solution like Google Zanzibar (or equivalent CDN) and local policy-agents, allowing both Google-like scales, low latency, and high performance.
  • If you're developing such a service, or considering such high-scale scenarios; you're welcome to contact us, and we'd be happy to share our plans for OPAL in that area.

• Using OPAL for other live update needs

  • While OPAL was created and primarily designed for open-policy and authorization needs; it can be generically applied for other live updates and data/code propagation needs
  • If you'd like to use OPAL or some of its underlying modules for other update cases - please contact us (See below), we'd love to help you do that.

• Administration capabilities and UI

  • We've already built policy editors, back-office, frontend-embeddable interfaces, and more as part of authorizon.com.
  • We have plans to migrate more parts of authorizon.com to be open-source; please let us know what you'd like to see next.

Joining the community

• We are eager to hear from you 😃
• Raise questions and ask for features to be added to the road-map in our Github discussions
• Report issues in Github issues
• Chat with us in our Slack community

Contacting us (the authors)

• We love talking about authorization, open-source, realtime communication, and tech in general.
• Feel free to reach out to us on our GitHub discussions or directly over email.

Contributing to OPAL

• Pull requests are welcome! (please make sure to include passing tests and docs)
• Prior to submitting a PR - open an issue on GitHub, or make sure your PR addresses an existing issue well.