Skip to main content

BigQuery Foreign Data Wrapper for PostgreSQL

Project description

# bigquery_fdw: BigQuery Foreign Data Wrapper for PostgreSQL

bigquery_fdw is a BigQuery foreign data wrapper for PostgreSQL using [Multicorn](

It allows to write queries in PostgreSQL SQL syntax using a foreign table. It supports most of BigQuery's [data types](docs/ and [operators](docs/

## Features and limitations

- Table partitioning is supported. [You can use partitions in your SQL queries](docs/
- Queries are parameterized when sent to BigQuery
- BigQuery's standard SQL support (legacy SQL is not supported)
- Authentication works with a "[Service Account](docs/" Json private key

[Read more](docs/

## Requirements

- PostgreSQL >= 9.5
- Python 3

## Dependencies

### Dependencies required to install bigquery_fdw:

- `postgresql-server-dev-X.Y`
- `python3-pip`
- `python3-dev`
- `make`
- `gcc`

### Major dependencies installed automatically during the installation process:

- [Google Cloud BigQuery](
- [Multicorn](

## Installation

# Install `setuptools` if necessary
pip3 install --upgrade setuptools

# Install Multicorn
git clone git:// && cd Multicorn
export PYTHON_OVERRIDE=python3
make && make install

# Install bigquery_fdw
pip3 install bigquery-fdw

## Usage

We recommend testing the [BigQuery client connectivity](docs/ before trying to use the FDW.

With `psql`:


wrapper 'bigquery_fdw.fdw.ConstantForeignDataWrapper'

CREATE FOREIGN TABLE my_bigquery_table (
column1 text,
column2 bigint
) SERVER bigquery_srv
fdw_dataset 'my_dataset',
fdw_table 'my_table',
fdw_key '/opt/bigquery_fdw/user.json'

## Options

List of options implemented in `CREATE FOREIGN TABLE` syntax:

| Option | Default | Description |
| `fdw_dataset` | - | BigQuery dataset name |
| `fdw_table` | - | BigQuery table name |
| `fdw_key` | - | Path to private Json key (See [Key storage recommendations](docs/ |
| `fdw_convert_tz` | - | Convert BigQuery time zone for dates and timestamps to selected time zone. Example: `'US/Eastern'`. |
| `fdw_group` | `'false'` | See [Remote grouping and counting](docs/ |
| `fdw_casting` | - | See [Casting](docs/ |
| `fdw_verbose` | `'false'` | Set to `'true'` to output debug information in PostrgeSQL's logs |
| `fdw_sql_dialect` | `'standard'` | BigQuery SQL dialect. Currently only `standard` is supported. |

## More documentation

See [bigquery_fdw documentation](docs/

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bigquery-fdw-1.3.2.tar.gz (8.0 kB view hashes)

Uploaded Source

Built Distribution

bigquery_fdw-1.3.2-py2.py3-none-any.whl (11.4 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page