Modular SQL Linting for Humans
Project description
SqlFluff :scroll: :black_nib: :sparkles:
The SQL Linter for humans
Bored of not having a good SQL linter that works with whichever dialiect you're working with? Fluff is an extensible and modular linter designed to help you write good SQL and catch errors and bad SQL before it hits your database.
Sqlfluff is still in an open alpha phase - expect the tool to change significantly over the coming months, and expect potentially non-backward compatable api changes to happen at any point. In particular moving from 0.0.x to 0.1.x introduced some non backward compatible changes and potential loss in functionality. If you'd like to help please consider contributing.
Getting Started
To get started just install the package, make a sql file and then run sqlfluff and point it at the file.
$ pip install sqlfluff
$ echo " SELECT a + b FROM tbl; " > test.sql
$ sqlfluff lint test.sql
== [test.sql] FAIL
L: 1 | P: 1 | L003 | Single indentation uses a number of spaces not a multiple of 4
L: 1 | P: 14 | L006 | Operators should be surrounded by a single space unless at the start/end of a line
L: 1 | P: 27 | L001 | Unnecessary trailing whitespace
Usage
For more details on usage see the docs on github here.
Progress
There's lots to do in this project, and we're just getting started. NB: This list has started again from the top due to the re-write. If you want to understand more about the architecture of sqlfluff, you can find more here.
- Command line interface
- Basic linting, both of paths and files
- Version information
- Nicely formatted readout of linting success or fail
- Exit codes which reflect linting success or fail
- Filtering to particular codes in the linting step
- Allow basic correction of some linting codes
- Basic ANSI linting
- Simple whitespace testing
- Whitespace around operators
- Indentation (size and mix of tabs and spaces)
- Indentation between lines and when to indent
- Number of blank lines
- Indentation of comments
- Inconsistent capitalisation of keywords
- Inconsistent capitalisation of unquoted identifiers
- (idea) Implement a context manager in the parse and match functions to avoid passing around so many variables.
- Configurable linting
- Command line options for config
- Ability to read from config files
- Ability to read config from block comment
sections in
.sql
files. - Ignore particular rules (blacklisting)
- Specifying particlar dialects to use
- Preconfiguring verbosity
- Dialects
- ANSI
- Implement singleton matching for
::
,:
and||
. - Bring in a much wider selection of test queries to identify next gaps.
- Flesh out function coverage.
- Implement singleton matching for
- MySQL
- Redshift
- Snowflake
- Detecting dialect from a config file of some kind
- jinja2 compatible linting (for dbt)
- ANSI
- Documentation
- Basic architectural principles
- Update CLI docs to match current state
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[Unreleased]
Changed
[0.1.2] - 2019-10-30
Changed
- Fixed the errors raised by the lexer.
[0.1.1] - 2019-10-30
Changed
- Fixed which modules from sqlfluff are installed in the setup.py. This affects
the
version
command.
[0.1.0] - 2019-10-29
Changed
- Big Rewrite - some loss in functionality might be apparent compared to pre-0.1.0. Please submit any major problems as issues on github
- Changed unicode handling for better escape codes in python 2. Thanks @mrshu
- BIG rewrite of the parser, completely new architecture. This introduces
breaking changes and some loss of functionality while we catch up.
- In particular, matches now return partial matches to speed up parsing.
- The
Delimited
matcher has had a significant re-write with a major speedup and broken the dependency onSequence
. - Rewrite of
StartsWith
andSequence
to use partial matches properly. - Different treatment of numeric literals.
- Both
Bracketed
andDelimited
respect bracket counting. - MASSIVE rewrite of
Bracketed
.
- Grammars now have timers.
- Joins properly parsing,
- Rewrite of logging to selectively output commands at different levels
of verbosity. This uses the
verbosity_logger
method. - Added a command line
sqlfluff parse
option which runs just the parsing step of the process to better understand how a file is being parsed. This also has options to configure how deep we recurse. - Complete Re-write of the rules section, implementing new
crawlers
which implement the linting rules. Now with inbuilt fixers in them. - Old rules removed and re implemented so we now have parity with the old rule sets.
- Moved to using Ref mostly within the core grammar so that we can have recursion.
- Used recursion to do a first implementation of arithmetic parsing. Including a test for it.
- Moved the main grammar into a seperate dialect and renamed source and test files accordingly.
- Moved to file-based tests for the ansi dialect to make it easier to test using the tool directly.
- As part of file tests - expected outcomes are now encoded in yaml to make it easier to write new tests.
- Vastly improved readability and debugging potential of the _match logging.
- Added support for windows line endings in the lexer.
[0.0.7] - 2018-11-19
Added
- Added a
sqlfluff fix
as a command to implement auto-fixing of linting errors. For now onlyL001
is implemented as a rule that can fix things. - Added a
rules
command to introspec the available rules. - Updated the cli table function to use the
testwrap
library and also deal a lot better with longer values. - Added a
--rules
argument to most of the commands to allow rule users to focus their search on a specific subset of rules.
Changed
- Refactor the cli tests to use the click CliRunner. Much faster
[0.0.6] - 2018-11-15
Added
- Number matching
Changed
- Fixed operator parsing and linting (including allowing the exception of
(*)
)
[0.0.5] - 2018-11-15
Added
- Much better documentation including the DOCS.md
Changed
- Fixed comma parsing and linting
[0.0.4] - 2018-11-14
Added
- Added operator regexes
- Added a priority for matchers to resolve some abiguity
- Added tests for operator regexes
- Added ability to initialise the memory in rules
[0.0.3] - 2018-11-14
Added
- Refactor of rules to allow rules with memory
- Adding comma linting rules (correcting the single character matchers)
- Adding mixed indentation linting rules
- Integration with CircleCI, CodeCov and lots of badges
Changed
- Changed import of version information to fix bug with importing config.ini
- Added basic violations/file reporting for some verbosities
- Refactor of rules to simplify definition
- Refactor of color cli output to make it more reusable
[0.0.2] - 2018-11-09
Added
- Longer project description
- Proper exit codes
- colorama for colored output
Changed
- Significant CLI changes
- Much improved output from CLI
[0.0.1] - 2018-11-07
Added
- Initial Commit! - VERY ALPHA
- Restructure into package layout
- Adding Tox and Pytest so that they work
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.