tinybird-cli 1.0.0b205

Tinybird Command Line Tool

The Tinybird command-line tool allows you to use all the Tinybird functionality directly from the command line. Additionally, it includes several functions to create and manage data projects easily.

Changelog

---

1.0.0b205

• Improved handling of some SQL queries.

1.0.0b204

• Fix tb test run <file> –fail to show only failed tests.

1.0.0b203

• Append a new node to a Pipe fixed: tb pipe append.

1.0.0b202

• When running tb auth –host XXXX, we strip the host parameter to keep just the domain (https://XXXX.tinybird.co)

1.0.0b201

• Improved how we deal with workspace_map replacements, making it more flexible, and less prone to errors.

1.0.0b200

• Revisited the tb test suite, now reduced to tb test [init,run,parse], simplified the tests, and based in the YAML format.

1.0.0b199

• Better error message on missing arguments to tb datasource append.

1.0.0b198

• When running the pipe checker, we were checking that the processed bytes of the new version not to increase more than 25%. Now, this validation will be optional.

1.0.0b197

• When running the pipe checker, we were checking that the processed bytes of the new version not to increase more than 25%. Now, this validation will be optional.

1.0.0b196

• Improve delete Data Source prompt message when they are the target of materialized views.

1.0.0b195

• Minor internal fixes and improvements.

1.0.0b194

• Support for role management using the subcommand tb workspace members set-role.

1.0.0b193

• Improve tb fmt, now Python code is better formatted

1.0.0b192

• Allow to hide token values in the output using the –hide-tokens flag.

1.0.0b191

• Add Data Source dependency information to tb datasource rm, also allowing Data Source forced deletion if it is the target of a materialized view.

1.0.0b190

• Pipe checker to support non-json endpoints (.ndjson, .csv)

1.0.0b189

• Minor internal fixes and improvements.

1.0.0b188

• Improve error reporting when tb push pipe_file and there’s a timeout

1.0.0b187

• Minor fixes in tb fmt

1.0.0b186

• Added a workspace creation wizard, triggered by calling tb workspace create without additional arguments.

• Added the new option –starter-kit to the tb workspace create command to allow applying an starter kit during workspace creation.

1.0.0b185

• Minor fix in tb fmt to better report usage

1.0.0b184

• Add tb fmt datafile_path to format a local datafile

1.0.0b183

• Fix tb pipe data to actually respect the query pass as parameter

1.0.0b182

• Fix when running regression-test

1.0.0b181

• Add option to truncate all depending Data Sources in cascade using a new flag tb truncate –cascade DS_name

1.0.0b180

• Remove when running tb push or tb regression-test the validation of response time not increasing > 25% as the response time can variate according the the status of the cluster. Instead we have added a validation of the bytes read as it remains constant

1.0.0b179

• Improve error reporting

1.0.0b178

• Fix issue when dealing with server gateway errors

1.0.0b177

• Internal improvements when dealing with server gateway errors

1.0.0b176

• Added –override-datasource to tb push to override an existing Data Source by a new one with different settings when creating a Materialized View

1.0.0b175

• Avoid requesting used_by field when doing calls to /v0/datasources to avoid unncessary work

1.0.0b174

• Added –ignore-empty to tb datasource replace to not throw error if the data is empty

1.0.0b173

• Fix: avoid removing a Pipe when running tb push –force and the push fails because there’s an error in any Node. If the Pipe can’t be overridden, the current Pipe remains untouched.

• Fix: avoid removing the Data Source of a Materialized View when running tb push –force. The Materialized View gets unlinked, but the Data Source won’t be removed.

1.0.0b172

• Added support for CODECS when using json schemas in .datasource files

1.0.0b171

• When doing tb push or tb regression-test, the pipe checker will verify if new versions of the endpoint does not have a degradation of respose time.

1.0.0b170

• Update clickhouse-toolset to 0.21.dev0 to provide better dependency detection when pushing projects.

1.0.0b169

• Added api column to tb auth ls output.

1.0.0b168

• Fix encoding when pushing a pipe.

1.0.0b167

• Added new workspace members management commands.

1.0.0b166

• Fix tb datasource append command by making URL not mandatory.

1.0.0b165

• Add mandatory target Data Source argument for most tb datasource X commands.

1.0.0b164

• Fix bug tb workspace delete should ask for a user token if not provided.

1.0.0b163

• Add –token as a valid flag for tb auth.

1.0.0b162

• Internal fix

1.0.0b161

• Add support for modifying the Data Source TTL

1.0.0b160

• Fix default array values in Data Source Schemas: arr Array(String) DEFAULT [‘-‘]

1.0.0b159

• Added tb datasource share to be able to share a datasource with another workspace

• Added –fork option to tb workspace create to share all the datasources when the workspace gets created

1.0.0b158

• Change tb workspace clear to skip Shared Data Sources correctly.

1.0.0b157

• Remove –populate-sql-condition alias since it was misleading, it’s enough with tb push pipe –populate –sql-condition “the_condition”`

1.0.0b156

• Improved tb auth. Now it automatically selects the proper region based on the provided token.

1.0.0b155

• Partially fix a bug that makes comments and SQL formatting to change before pushing a datafile

1.0.0b154

• Fixed problem with update version warning being displayed when using –token. Now the update version warning is never displayed with this command.

1.0.0b153

• New tb pipe populate command

1.0.0b152

• Deprecate ENGINE_FULL option.

1.0.0b151

• Update command tb workspace clear. Now deletes all files in the workspace, not only the ones you have in your local folder.

1.0.0b150

• Added option –sample-by-params when running tb push -f or tb pipe regression-test to specify the number of sample calls to validate against. For each combination of query parameters, we will sample the data and validate the results.

1.0.0b149

• Update clickhouse-toolset to 0.20.dev0

1.0.0b148

• Add tb pipe publish and tb pipe unpublish commands

1.0.0b147

• Add –recursive flag to tb dependencies to get recursive dependencies

• Add –populate-sql-condition as an alias for tb push pipe –populate –sql-condition

1.0.0b146

• Improved documentation on tb test commands

• Improved readibility of tb test

• tb test run return non-zero exit code on failure

• –fixtures option NDJSON fixtures (only CSV was supported before)

1.0.0b145

• Add –sql-condition to send an arbitrary SQL condition together with the –populate flag when pushing a materialized pipe. See tb push –help for usage information.

1.0.0b144

• Report errors 500

1.0.0b143

• Add support to create/update tokens from datasource definition

1.0.0b142

• When running the pipe checker, checker endpoints will added to the TOKEN scope if token is specified

1.0.0b141

• Publishing Docker images for linux/arm64 arch next to the existing linux/amd64 arch

1.0.0b140

• Fix CLI for OS without clickhouse-toolset available

1.0.0b139

• Add force format option (json) to datasource / pipe ls

1.0.0b138

• Fix CLI for OS without clickhouse-toolset available

1.0.0b137

• Drop support for –skip-table-checks

1.0.0b136

• Deprecate –skip-table-checks

1.0.0b135

• Update clickhouse-toolset to 0.19.dev2

1.0.0b134

• Update clickhouse-toolset to 0.19.dev0: Improved performance and better checks around ClickHouse functions

1.0.0b133

• Now tb check verifies also query syntax

1.0.0b132

• Upgraded click package from version 7.0.0 to 8.0.0

1.0.0b131

• New tb test feature.

1.0.0b130

• Fix tinybird-cli releases for python 3.10

1.0.0b129

• Connectors performing multiple appends at once but honoring GCP limitations

1.0.0b128

• Changed the HTTP Client for the CLI

1.0.0b127

• Added new metrics about endpoint response times (max,min,mean,median an p90) on pipe command regression-test

1.0.0b126

• Connectors fixes and improvements

1.0.0b125

• Connectors performing multiple appends at once to improve performance

1.0.0b124

• Fixes in tb materialize

• Now tb materialize properly supports –prefix and datafiles with VERSION

• Removed unneeded prompts, the command does not ask you to override resources when it’s not necessary

• Changed command prompts so they explain better the action to be performed

• Added tb materialize documentation

1.0.0b123

• Revert to 1.0.0b121

1.0.0b122

• Update clickhouse-toolset to 0.17.dev2: Improved performance and better checks around ClickHouse functions

1.0.0b121

• Connectors fix and improvements

1.0.0b120

• Added –failfast/-ff and –ignore-order to tb pipe regression-test and tb push

1.0.0b119

• Add support for creating/deleting workspaces

1.0.0b118

• Improved syntax error reporting

1.0.0b117

• Fix dependencies check on tb push

1.0.0b116

• Return better error when accessing wrong resources

1.0.0b115

• tb materialize command now is interactive

1.0.0b114

• Add support for description in datasources

1.0.0b113

• Fixed problem with tb push datasource

1.0.0b112

• Fixed problem with web requests on Windows

1.0.0b111

• Requests send by the checker will be tag to avoid running always, so we can ignore them if run again the checker

• Now checker will run a sample of requests by each group of params. This way, we improve the coverage of the endpoints

• Improve checker diff to provide much more feedback when something is different between original result and checker one

1.0.0b110

• Nodes with TYPE materialized won’t be published as endpoints anymore. This was a historic design decision we are changing due to multiple side effects.

• This change adds a breaking change: if you had a pipe using the name of a materialized node, and the materialized node is not published as an endpoint the pipe won’t work. To fix the issue you have to make your pipes depend on the Materialized Node target Data Source.

1.0.0b109

• Improve feedback if server returns something that the HTTP Client can not parse

1.0.0b108

• Replace –populate –sample by –populate –subset, since sample was misleading.

1.0.0b107

• Add option –skip-incompatible-partition-key when replacing a data source with condition, i.e: tb datasource replace <name> –skip-incompatible-partition-key

1.0.0b106

• Fixed push Kafka DS with custom columns (!3079)

1.0.0b105

• Change tb pipe ls, now it returns when a pipe was modified and not just when it was created

1.0.0b104

• Fix CLI format auto detection with URLs (!3054)

1.0.0b103

• Fix tb push path/ allowing to push the content of a directory.

1.0.0b102

• Add option to clear the workspace by running tb workspace clear

1.0.0b101

• Add support for Parquet files

1.0.0b100

• Add tb materialize path/to/pipe.pipe node_name. It analyzes the node_name SQL query to generate the .datasource and .pipe files needed to push a new materialize view.

1.0.0b99

• Fix broken release 1.0.0b98. The published package missed one dependency required for some commands.

1.0.0b98

• tb auth –interactive now supports custom regions

1.0.0b97

• Add –sample option for –populate

1.0.0b96

• Use ‘replace’ instead of ‘append’ when pushing fixtures.

1.0.0b95

• Fix broken release 1.0.0b94.

1.0.0b94

• Changed the name for workspace plans.

1.0.0b93

• Improved error message format pushing a pipe.

1.0.0b92

• Added a new command to perform regression tests. See tb pipe regression-test –help for more information about the new command.

1.0.0b91

• Update clickhouse-toolset to 0.15dev1: Performance improvements in query parsing and replacement and fixes an issue with the M1 build depending on system libraries.

1.0.0b90

• Rollback to 1.0.0b88

1.0.0b89

• Update clickhouse-toolset to 0.15dev0: Performance improvements in query parsing and replacement and fixes an issue with the M1 build depending on system libraries.

1.0.0b88

• Increased timeout for tb dependencies for situations where a Data Source contains a large number of dependent Data Sources.

1.0.0b87

• Added a way to check the dependencies of a Data Source that would be affected by a partial replace. Example: tb dependencies –datasource my_datasource –check-for-partial-replace

1.0.0b86

• Return 1 as an exit code (not zero) in case of a problem appending to a datasource

1.0.0b85

• Checker should raise an error when fails retrieving most common requests while performing regression tests

1.0.0b84

• Improved error message suggesting –skip-table-checks

• Return file path for push errors

1.0.0b83

• Fixed a bug when pushing a pipe with the TOKEN command

1.0.0b82

• Forcing checker to be a standard pipe even when pushing with force a materialized view pipe

1.0.0b81

• Improve message showed in imports/replaces when the operation is not correctly configured.

1.0.0b80

• When pulling a resource with versioned dependencies, dependecy names are correctly replaced by the original names

1.0.0b78

• When pushing a new resource with versioning, sucessful create message displays the name with the version

1.0.0b77

• Changed dry-run prompts to include resources versions

1.0.0b76

• Fix authentication using localhost without http/https prefix

1.0.0b75

• Add delete subcommand to datasource to delete rows matching a SQL condition

1.0.0b74

• Fixed problem with new version warning not being displayed

1.0.0b73

• Added shortcut -i to tb auth –interactive

• Removed explorations folder as it is not used and cause confusion

1.0.0b72

• Update clickhouse-toolset to 0.14dev1.

1.0.0b71

• Fix TABLE macro to work with shared data sources

1.0.0b70

• Update clickhouse-toolset to 0.14dev0.

1.0.0b69

• Removed –skip_update_validation in favor of –version_warning

1.0.0b68

• Fix installation when the toolset binary dependency isn’t available. Improve OSX 11 support.

1.0.0b67

• Add support to validate and replace queries locally (on selected architectures and OSs).

1.0.0b66

• Added support for multi-region auth

1.0.0b65

• Fixes a problem when pushing a Kafka Data Source with the new KAFKA_STORE_RAW_VALUE key.

• Improved the message shown when a user tries to push –force a Kafka Data Source.

• When using Snowflake connector, the CSV exported to TB will be represented by and empty space between delimeters ( ex: ,,)

1.0.0b64

• Fixes tb auth when already ran a sucessful auth

1.0.0b63

• Fix Auth error when trying to remove datasource/pipe

1.0.0b62

• Added –timeout option when running tb push <PIPE>

1.0.0b61

• CLI client will retry when reach request limit (HTTP code 429)

1.0.0b60

• push, pull and append now work with NDJSON

1.0.0b59

• Improve CLI message when GCS compose does not have data because SQL query did not return any

1.0.0b58

• Added option –skip-update-validation to skip the update validation that the CLI does to check for a new version

1.0.0b57

• Support –token and –host param for auth and workspace commands. This is useful when you want to automate commands via shell scripts or similar and you want to pass those parameters via environment variables without having to do tb auth with the token prompt. For instance tb –token <token> workspace ls will list the workspaces for the token passed as a parameter.

1.0.0b56

• Fixed push when shared datasources are used in a SQL. Shared datasources names are pushed as-is.

1.0.0b55

• Removed the following ‘tb auth’ commands: ‘tb auth use’, ‘tb auth ls’, ‘tb auth add’, and ‘tb auth rm’.

• Fixed ‘tb workspace ls’ command to list correctly the workspaces a user has access to.

• Added ‘tb workspace use’ command to switch between the workspaces a user has access to.

1.0.0b54

• Fixed push pipes with escaped quotes

• Bump snowflake client to 2.6.1

1.0.0b53

• Changed tb datasource generate and tb datasource analyze to use the new v0/analyze API. It works with local and remote CSV and NDJSON files and Snowflake tables.

1.0.0b52

• Added the posibility of using env variables in a template. For example, you can have a statements like this

` INCLUDE "includes/${env}_secrests.incl" `

and call tb like this

` env=test tb push file.datasource ` - Added –no-versions to push so dependencies versions are not used. Useful for data migrations.

1.0.0b51

• Changed internal user authentication.

• Added the possibility to use a workspace as a dependecy. See tb push –workspace option.

1.0.0b50

• Fixed Fixed duplicated rows in snowflake connector import

1.0.0b49

• Fixed snowflake secrets file generation when running tb auth –connector snowflake wizard

1.0.0b48

• When new version available, the message will return the pip command to execute

1.0.0b47

• Fix error given when trying to delete a non-existent data source

• Add flag to save data sources and pipes into their own directories, it respects the default behaviour

• Instead of avoid push an already defined data source from a materialized node, only raise an error if the schema and the engine don’t match

• Capture error properly during a populate job that fails

1.0.0b46

• Enable connection commands

• Support push and pull Kafka Data Sources

1.0.0b45

• Add shared from to the datasource ls command to show Data Source’s original Workspace.

1.0.0b44

• Support SQL in nodes with no indentation

• Standardize ls command output

• Add –only-response-times flag when pushing a pipe. It’ll just run regression tests related to check response times

1.0.0b43

• Improve pull command to make it able to interact with shared Data Sources.

1.0.0b42

• Improve parser’s error handling of files: add line number

1.0.0b41

• Fix URL escaping with tb generate

1.0.0b40

• Automatically prettify .tinyb files

1.0.0b39

• Improved host option. You can use URLs ended with a backslash or not. Example: tb –host https://my-host/ auth

1.0.0b38

• Fixed misleading warnings when pushing a pipe, related to join tables.

• Fixed a bug parsing the data sources statistics when running tb datasource ls

• Changed error message in case the datasource pushed has incorrect syntax, now it gives hints on the part of the schema that is wrong.

1.0.0b37

• Gives better error messages in case the SQL in a node exceeds the current size limits (8KB).

1.0.0b36

• The option –ignore-sql-errors in tb push is no longer needed as we have fixed one of the underlying issues that was forcing its use on certain occasions.

1.0.0b35

• Added tb connection' commands. Supported connectors: Kafka

• Added tb datasource connect <id> command. Supported connectors: Kafka

1.0.0b34

• Improved the message showed in case a Pipe being pushed gets removed in the middle of the operation.

• Improved the way the CLI detect changes when working with resources with and without prefixes.

1.0.0b33

• Enables the possibility to add new columns to an existing datasource.

1.0.0b32

• Fix drop-prefix command, now it drops first pipes and its dependencies and then source data sources

1.0.0b31

• Do no report negative lines on append

• Auth improvements and changes. Credentials precendence set to –token => TB_TOKEN envvar => .tinyb file.

1.0.0b30

• Improve server error handling and sort datasource’s dependencies.

• Fix the Job tracking in cli operations when the Job enters cancelled status.

• Fix returning an error if there are wrong options when creating a data source

1.0.0b29

• Added support for Python 3.9

1.0.0b28

• Changed --skip-table-check error message to indicate how to use the flag

• Fixed checking Content-Type header

1.0.0b27

• Added flag --skip-table-check to skip materialized view and table checks

1.0.0b26

• Fix regression tests execution.

1.0.0b25

• Added support for job cancellation tb job cancel <job_id>

1.0.0b24

• Added support for ingesting data from Snowflake

1.0.0b23

• Added support for truncate operations tb datasource truncate <datasource_name>

• Added support for replace operations tb datasource replace <datasource_name> <URL or file>

• Added support for replace with condition operations tb datasource replace <datasource_name> <URL or file> –sql-condition=”country=’ES’”

• Added support for ingesting data from BigQuery

• Fix –wait option on populate

• Report progress bar when waiting for a populate job to finish

• Pull also resources with prefixes

• Disable detecting TOKEN envvar

• Fix auth from other commands

1.0.0b22

• Add support for ENGINE_SETTINGS and ENGINE_TTL

1.0.0b21

• Added population time to populate pipe option. It returns the actual population time from the population job when debug is used along with populate and wait options.

1.0.0b20

• Add missing dependency. It fixes version 1.0.0b19

1.0.0b19

• Remove red color on pull (#847) (!1243)

• Change color palette feedback (!1243)

• When doing a query against a datasource that doesn’t exist, the CLI returns a proper error (#846) (!1243)

• Fix ‘blog_log’ key error when pulling (#656) (!1243)

• Make tb commands async (!1243)

• Display a warning if there is a more recent version (#950) (!1243)

1.0.0b18

• Fix sql command (!1264)

1.0.0b17

• Fix auth CLI workspace commands (!1256)

1.0.0b16

• Add CLI workspace commands (!1246)

1.0.0b15

• Support for CSV and JSON output to “sql” CLI command

• Fixed datasource rm command not working when there is a MV pointing to the datasource being removed

1.0.0b14

• Added missed dependency. It fixes version 1.0.0b13.

1.0.0b13

• Added wait option to push command for waiting the populate job to finish.

• Fixed appending large files (more than 2GB). Now, files are uploaded using multipart and without loading the whole file in memory. (!833)(!1221)

• Added feedback about appended rows, total rows and errors to append command output (!1205)

1.0.0b12

• Added support for ENGINE options to match the API parameters

• Deprecating ENGINE_FULL, SORTING_KEY, PRIMARY_KEY, SAMPLING_KEY, and TTL options.

• Added job commands to list jobs and see job details

• Added pipe_stats command

• New set of tokens are created per prefix

• Fixed drop-prefix to remove resources in the right order

1.0.0b11

• Better error reporting when a table for a data source does not exist (!1020)

1.0.0b10

• Better error reporting when pushing a not supported file extension (!966)

1.0.0b9

• Fix parsing schemas in datasource files when using aggregate function (!956)

1.0.0b8

• Fix CLI error when pushing fixtures (!938)

2020-08-28 1.0.0b7

• Support for Python versions 3.6, 3.7 and 3.8 for both MacOSX and Linux

2020-08-26: 1.0.0b6

• Fixes saving pipe descriptions on pull and push

2020-08-24: 1.0.0b5

• Fixes appending a datasource from a URL

2020-08-10: 1.0.0b4

• Fixes the version parser.

2020-08-10: 1.0.0b3

• Limited the amount of rows fetched when running tb sql to 100. Added --row-limit.

• Fixed warning when pipes use internal tables like datasources_ops_log

• Raise an error when INSERT sql statement is used.