Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

BigQuery Foreign Data Wrapper for PostgreSQL

Project Description
bigquery\_fdw: BigQuery Foreign Data Wrapper for PostgreSQL
===========================================================

bigquery\_fdw is a BigQuery foreign data wrapper for PostgreSQL using
`Multicorn <https://github.com/Kozea/Multicorn>`__.

It allows to write queries in PostgreSQL SQL syntax using a foreign
table. It supports most of BigQuery's `data
types <docs/data_types.md>`__ and `operators <docs/operators.md>`__.

Features and limitations
------------------------

- Table partitioning is supported. `You can use partitions in your SQL
queries <docs/table_partitioning.md>`__.
- Queries are parameterized when sent to BigQuery
- BigQuery's standard SQL support (legacy SQL is not supported)
- Authentication works with a "`Service
Account <docs/service_account.md>`__" Json private key

`Read more <docs/README.md>`__.

Requirements
------------

- PostgreSQL >= 9.5
- Python 3

Dependencies
------------

Dependencies required to install bigquery\_fdw:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- ``postgresql-server-dev-X.Y``
- ``python3-pip``
- ``python3-dev``
- ``make``
- ``gcc``

Major dependencies installed automatically during the installation process:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- `Google Cloud
BigQuery <https://pypi.org/project/google-cloud-bigquery/>`__
- `Multicorn <https://github.com/Kozea/Multicorn>`__

Installation
------------

.. code:: bash

# Install `setuptools` if necessary
pip3 install --upgrade setuptools

# Install Multicorn
git clone git://github.com/Kozea/Multicorn.git && cd Multicorn
export PYTHON_OVERRIDE=python3
make && make install

# Install bigquery_fdw
pip3 install bigquery-fdw

Usage
-----

We recommend testing the `BigQuery client
connectivity <docs/test_client.md>`__ before trying to use the FDW.

With ``psql``:

.. code:: sql

CREATE EXTENSION multicorn;

CREATE SERVER bigquery_srv FOREIGN DATA WRAPPER multicorn
OPTIONS (
wrapper 'bigquery_fdw.fdw.ConstantForeignDataWrapper'
);

CREATE FOREIGN TABLE my_bigquery_table (
column1 text,
column2 bigint
) SERVER bigquery_srv
OPTIONS (
fdw_dataset 'my_dataset',
fdw_table 'my_table',
fdw_key '/opt/bigquery_fdw/user.json'
);

Options
-------

List of options implemented in ``CREATE FOREIGN TABLE`` syntax:

+--------+-------+-------+
| Option | Defau | Descr |
| | lt | iptio |
| | | n |
+========+=======+=======+
| ``fdw_ | - | BigQu |
| datase | | ery |
| t`` | | datas |
| | | et |
| | | name |
+--------+-------+-------+
| ``fdw_ | - | BigQu |
| table` | | ery |
| ` | | table |
| | | name |
+--------+-------+-------+
| ``fdw_ | - | Path |
| key`` | | to |
| | | priva |
| | | te |
| | | Json |
| | | key |
| | | (See |
| | | `Key |
| | | stora |
| | | ge |
| | | recom |
| | | menda |
| | | tions |
| | | <doc |
| | | s/key |
| | | _stor |
| | | age.m |
| | | d>`__ |
| | | ) |
+--------+-------+-------+
| ``fdw_ | - | Conve |
| conver | | rt |
| t_tz`` | | BigQu |
| | | ery |
| | | time |
| | | zone |
| | | for |
| | | dates |
| | | and |
| | | times |
| | | tamps |
| | | to |
| | | selec |
| | | ted |
| | | time |
| | | zone. |
| | | Examp |
| | | le: |
| | | ``'US |
| | | /East |
| | | ern'` |
| | | `. |
+--------+-------+-------+
| ``fdw_ | ``'fa | See |
| group` | lse'` | `Remo |
| ` | ` | te |
| | | group |
| | | ing |
| | | and |
| | | count |
| | | ing < |
| | | docs/ |
| | | remot |
| | | e_gro |
| | | uping |
| | | .md>` |
| | | __. |
+--------+-------+-------+
| ``fdw_ | - | See |
| castin | | `Cast |
| g`` | | ing < |
| | | docs/ |
| | | casti |
| | | ng.md |
| | | >`__. |
+--------+-------+-------+
| ``fdw_ | ``'fa | Set |
| verbos | lse'` | to |
| e`` | ` | ``'tr |
| | | ue'`` |
| | | to |
| | | outpu |
| | | t |
| | | debug |
| | | infor |
| | | matio |
| | | n |
| | | in |
| | | Postr |
| | | geSQL |
| | | 's |
| | | logs |
+--------+-------+-------+
| ``fdw_ | ``'st | BigQu |
| sql_di | andar | ery |
| alect` | d'`` | SQL |
| ` | | diale |
| | | ct. |
| | | Curre |
| | | ntly |
| | | only |
| | | ``sta |
| | | ndard |
| | | `` |
| | | is |
| | | suppo |
| | | rted. |
+--------+-------+-------+

More documentation
------------------

See `bigquery\_fdw documentation <docs/README.md>`__.


Release History

Release History

This version
History Node

1.2

History Node

1.1

History Node

1.0.2

History Node

1.0.1

History Node

1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
bigquery_fdw-1.2-py2.py3-none-any.whl (12.1 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Oct 17, 2017
bigquery-fdw-1.2.tar.gz (8.9 kB) Copy SHA256 Checksum SHA256 Source Oct 17, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting