sqlitebiter 0.6.0

sqlitebiter is a CLI tool to convert CSV/Excel/HTML/JSON/LTSV/Markdown/TSV/Google-Sheets to a SQLite database file.

sqlitebiter

Summary

sqlitebiter is a CLI tool to convert CSV/Excel/HTML/JSON/LTSV/Markdown/TSV/Google-Sheets to a SQLite database file.

Features

• Create a SQLite database file from:

  • file(s):

    • CSV

    • Microsoft Excel ^TM

    • HTML: extract table tag data

    • JSON

    • Labeled Tab-separated Values (LTSV)

    • Markdown: extract Markdown table

    • Tab separated values (TSV)

  • Google Sheets

  • URL (fetch data from the Internet)

• Multi-byte character support

Usage

Create SQLite database from files

Create SQLite database from URL

Following is an example that convert HTML table tags within a web page to SQLite tables.

$ sqlitebiter -v url "https://en.wikipedia.org/wiki/Comparison_of_firewalls"
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html1 (Firewall TEXT, License TEXT, CostUsageLimits TEXT, OS TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html2 (Firewall TEXT, License TEXT, Cost TEXT, OS TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html3 (CanTarget TEXT, Changingdefaultpolicytoacceptrejectbyissuingasinglerule TEXT, IPdestinationaddresses TEXT, IPsourceaddresses TEXT, TCPUDPdestinationports TEXT, TCPUDPsourceports TEXT, EthernetMACdestinationaddress TEXT, EthernetMACsourceaddress TEXT, Inboundfirewallingress TEXT, Outboundfirewallegress TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html4 (Can TEXT, [workatOSILayer4statefulfirewall] TEXT, [workatOSILayer7applicationinspection] TEXT, ChangeTTLTransparenttotraceroute TEXT, ConfigureREJECTwithanswer TEXT, DMZdemilitarizedzoneallowsforsingleseveralhostsnottobefirewalled TEXT, Filteraccordingtotimeofday TEXT, RedirectTCPUDPportsportforwarding TEXT, RedirectIPaddressesforwarding TEXT, FilteraccordingtoUserAuthorization TEXT, TrafficratelimitQoS TEXT, Tarpit TEXT, Log TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html5 (Features TEXT, ConfigurationGUItextorbothmodes TEXT, [RemoteAccessWebHTTPTelnetSSHRDPSerialCOMRS232] TEXT, Changeruleswithoutrequiringrestart TEXT, Abilitytocentrallymanageallfirewallstogether TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html6 (Features TEXT, Modularitysupportsthirdpartymodulestoextendfunctionality TEXT, [IPS : Intrusion prevention system] TEXT, OpenSourceLicense TEXT, [supports IPv6 ?] TEXT, ClassHomeProfessional TEXT, OperatingSystemsonwhichitruns TEXT)' table
[INFO] convert 'https://en.wikipedia.org/wiki/Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html7 (Can TEXT, [NAT44staticdynamicwoportsPAT] TEXT, [NAT64NPTv6] TEXT, IDSIntrusionDetectionSystem TEXT, VPNVirtualPrivateNetwork TEXT, AVAntiVirus TEXT, Sniffer TEXT, Profileselection TEXT)' table

For more information

More examples are available at http://sqlitebiter.rtfd.io/en/latest/pages/usage/index.html

Installation

Install via pip (recommended)

sqlitebiter can be installed via pip (Python package manager).

sudo pip install sqlitebiter

Python package dependencies are as follows.

Mandatory Python package dependencies

Mandatory Python packages are automatically installed during sqlitebiter installation via pip.

• click

• DataPropery

• logbook

• path.py

• pytablereader

• SimpleSQLite

• sqliteschema

Google Sheets dependencies

Manual installation required to use Google Sheets feature.

• oauth2client

• pyOpenSSL

Test dependencies

• pytest

• pytest-runner

• tox

• XlsxWriter

Misc

• lxml (Faster HTML convert if installed)

Installing executable file in Windows

1. Navigate to https://github.com/thombashi/sqlitebiter/releases

2. Download the latest version of the sqlitebiter_win_x64.zip

3. Unzip the file

Documentation

http://sqlitebiter.rtfd.io/