A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
Project description
sqlitebiter
Summary
A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
Features
- Create a SQLite database file from:
- File(s):
CSV
Microsoft Excel TM
HTML
JSON
Line-delimited JSON(LDJSON)/NDJSON/JSON Lines
Markdown
Mediawiki
Space separated values (SSV)
SQLite
Tab separated values (TSV)
URL (scrape data from web pages)
Multi-byte character support
Automatic file encoding detection
Usage
Create SQLite database from files
Create SQLite database from URL
Following is an example that convert HTML table tags within a web page to SQLite tables.
- Example:
$ sqlitebiter url "https://en.wikipedia.org/wiki/Comparison_of_firewalls" [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html1' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html2' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html3' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html4' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html5' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html6' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html7' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html8' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html9' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html10' table [INFO] sqlitebiter url: convert 'en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html11' table [INFO] sqlitebiter url: number of created tables: 11 [INFO] sqlitebiter url: database path: out.sqlite
- Output:
sqlite> .schema CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html1' (Firewall TEXT, License TEXT, Costandusagelimits TEXT, OS TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html2' (Firewall TEXT, License TEXT, Cost TEXT, OS TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html3' (CanTarget TEXT, Changingdefaultpolicytoacceptrejectbyissuingasinglerule TEXT, IPdestinationaddresses TEXT, IPsourceaddresses TEXT, TCPUDPdestinationports TEXT, TCPUDPsourceports TEXT, EthernetMACdestinationaddress TEXT, EthernetMACsourceaddress TEXT, Inboundfirewallingress TEXT, Outboundfirewallegress TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html4' (Can TEXT, [workatOSILayer4statefulfirewall] TEXT, [workatOSILayer7applicationinspection] TEXT, ChangeTTLTransparenttotraceroute TEXT, ConfigureREJECTwithanswer TEXT, DMZdemilitarizedzoneallowsforsingleseveralhostsnottobefirewalled TEXT, Filteraccordingtotimeofday TEXT, RedirectTCPUDPportsportforwarding TEXT, RedirectIPaddressesforwarding TEXT, FilteraccordingtoUserAuthorization TEXT, TrafficratelimitQoS TEXT, Tarpit TEXT, Log TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html5' (Features TEXT, ConfigurationGUItextorbothmodes TEXT, [RemoteAccessWebHTTPTelnetSSHRDPSerialCOMRS232] TEXT, Changeruleswithoutrequiringrestart TEXT, Abilitytocentrallymanageallfirewallstogether TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html6' (Features TEXT, Modularitysupportsthirdpartymodulestoextendfunctionality TEXT, [IPS : Intrusion prevention system] TEXT, OpenSourceLicense TEXT, [supports IPv6 ?] TEXT, ClassHomeProfessional TEXT, OperatingSystemsonwhichitruns TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html7' (Can TEXT, [NAT44staticdynamicwoportsPAT] TEXT, [NAT64NPTv6] TEXT, IDSIntrusionDetectionSystem TEXT, VPNVirtualPrivateNetwork TEXT, AVAntiVirus TEXT, Sniffer TEXT, Profileselection TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html9' (A TEXT, B TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html10' (A TEXT, B TEXT); CREATE TABLE 'Comparison_of_firewalls_Wikipedia_html11' (A TEXT, B TEXT);
For more information
More examples are available at http://sqlitebiter.rtfd.io/en/latest/pages/usage/index.html
Installation
Install via pip (recommended)
sqlitebiter can be installed via pip (Python package manager).
- Example:
pip install sqlitebiter
Installation for Debian/Ubuntu from a deb package
wget https://github.com/thombashi/sqlitebiter/releases/download/<version>/sqlitebiter_<version>_amd64.deb
dpkg -iv sqlitebiter_<version>_amd64.deb
- Example:
$ wget https://github.com/thombashi/sqlitebiter/releases/download/v0.12.0/sqlitebiter_0.12.0_amd64.deb $ sudo dpkg -i sqlitebiter_0.12.0_amd64.deb
Installing executable files in Windows
sqlitebiter can be used in Windows environments without Python installation as follows:
Navigate to https://github.com/thombashi/sqlitebiter/releases
Download the latest version of the sqlitebiter_win_x64.zip
Unzip the file
Execute sqlitebiter.exe in either Command Prompt or PowerShell
>cd sqlitebiter_win_x64
>sqlitebiter.exe -h
Usage: sqlitebiter.exe [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-a, --append append table(s) to existing database.
-i, --index TEXT comma separated attribute names to create indices.
-v, --verbose
--debug for debug print.
--quiet suppress execution log messages.
-h, --help Show this message and exit.
Commands:
configure Configure the following application settings:...
file Convert tabular data within...
gs Convert a spreadsheet in Google Sheets to a...
url Scrape tabular data from a URL and convert...
Installation for macOS via Homebrew
$ brew tap thombashi/sqlitebiter
$ brew install sqlitebiter
Dependencies
Python 2.7+ or 3.4+
Python package dependencies are as follows.
Python package dependencies
Mandatory dependencies
Following mandatory Python packages are automatically installed during sqlitebiter installation process:
Google Sheets dependencies (Optional)
Following Python packages are required to manual installation when you use Google Sheets feature:
The above packages can be installed with the following pip command;
$ pip install sqlitebiter[gs]
Test dependencies
Misc dependencies (Optional)
Dependencies other than Python packages (Optional)
libxml2 (faster HTML/Markdown conversion)
pandoc (required when converting MediaWiki files)
Documentation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sqlitebiter-0.17.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 129d7d436f74abcb67adba6ba5200f8fe063bff1013de3e5622b721d149ed629 |
|
MD5 | 9d58ac538b5af61f0e605a641a930073 |
|
BLAKE2b-256 | 223e030da73c617111cc85f29f41b158379b0d27caebc8f929d26d4fb3492cda |