gaiaoffline 1.0.0

No project description provided

GaiaOffline

GaiaOffline is a Python package for offline querying of Gaia DR3 data. It enables you to download Gaia catalog subsets, store them in a local SQLite database, and perform efficient queries without relying on online services.

The point of this repository is to enable you to create a local catalog with some flexibility, while keeping the on disk size of the catalog small. This optimizes for smallest size on disk at any time, not for

Features

• Download Gaia DR3 catalog data as CSV files and convert them to a local SQLite database.
• Perform offline queries, including rectangular searches in RA/Dec.
• Configure stored columns, magnitude limits, and other settings via a persistent config file.

Installation

Install using pip

You can install this package with pip using

pip install gaiaoffline

Install the package using Poetry

Clone the repository:

   git clone https://github.com/christinahedges/gaiaoffline.git
   cd gaiaoffline

Install dependencies using Poetry:

    poetry install

Configuration

GaiaOffline uses a persistent configuration file to manage settings. The configuration file is automatically created at:

• macOS: ~/Library/Application Support/gaiaoffline/config.ini
• Linux: ~/.config/gaiaoffline/config.ini
• Windows: %LOCALAPPDATA%\gaiaoffline\config.ini

You can use this file to customize the behavior of the package without modifying the code.

Key Sections

1. SETTINGS

   • archive_url: URL to the Gaia DR3 archive for downloading data. Should be a listing of CSV files to download.
   • db_dir: Directory where the SQLite database is stored.
   • db_name: Name of the database file (default: gaiadr3.db).
   • table_name: Name of the database table for Gaia data.
   • log_level: Logging level (INFO, DEBUG, etc.).

2. DATABASE

   • stored_columns: List of columns to save from the Gaia data. Customize this to store only the data you need, reducing database size. You must store, at minimum, phot_g_mean_flux
   • zeropoints: Zeropoints for converting fluxes to magnitudes for G, BP, and RP bands. These are set to current best estimates from the Gaia mission.
   • magnitude_limit: Faintest magnitude to store in the database. Filters out faint sources during database creation. If you set this limit to a bright magnitude (e.g. 10), when downloading the database any fainter sources will be removed. This will reduce the amount of data stored on disk. If you set no limit, and have all columns, expect the database to be ~3Tb.

Example Configuration

[SETTINGS]
db_dir = /path/to/database
db_name = gaiadr3.db
log_level = INFO
table_name = gaiadr3
archive_url = https://cdn.gea.esac.esa.int/Gaia/gdr3/gaia_source/

[DATABASE]
stored_columns = source_id,ra,dec,parallax,pmra,pmdec,phot_g_mean_flux,phot_bp_mean_flux,phot_rp_mean_flux,radial_velocity,teff_gspphot,logg_gspphot,mh_gspphot
zeropoints = 25.6873668671,25.3385422158,24.7478955012
magnitude_limit = 16

Modifying the Configuration

You can edit the configuration file manually or update it programmatically:

from gaiaoffline import config, save_config
config["SETTINGS"]["log_level"] = "DEBUG"
save_config(config)

Resetting the Configuration

To reset the configuration file to its default values use the following function. Keep in mind this will reset your config file on disk, you must restart your session to have these configurations take effect.

from gaiaoffline import reset_config
reset_config()

Managing the Database

Creating the Database

If you don't have a copy of the database, you can create one using

from gaiaoffline import populate_gaiadr3
populate_gaiadr3()

This will download ~3500 csv files and will take a long time. If you interupt the download for any reason, simply repeat the command and the database will pick up the download from wherever you've left off.

Once this is complete, you can optionally download the gaia-2MASS cross match using

from gaiaoffline import populate_tmass_xmatch
populate_tmass_xmatch()

Once this is finished you can then download the 2MASS database using

from gaiaoffline import populate_tmass
populate_tmass()

You must complete these steps in order, otherwise your database will be incomplete.

Once you have completed this, you can check on the completeness by looking at the repr of the Gaia object.

from gaiaoffline import Gaia
with Gaia() as gaia:
   print(gaia)

This repr should look something like:

Offline Gaia Database
   gaiadr3: 100.0% Populated
   tmass xmatch: 100.0% Populated
   tmass: 100.0% Populated

Adding a precomputed database

If you've recieved a database file from a colleague or downloaded from Zenodo make sure that

1. Your config files match. All but the db_dir location should match.
2. Your database file is in the db_dir location. You can also find this by running from gaiaoffline import DATABASEPATH. This string will tell you where the file should be.

If you are using the default settings of this repository you can download a precomputed catalog here.

Delete the database

The database can get large, and you may wish to delete it. Remember you can find the database file location in the config file.

from gaiaoffline.utils import delete_database

# Remove the database
delete_database()

Usage

Querying the Database

gaiaoffline provides you with an object that you can manage using context, this ensures that the database behind any queries is always closed after your have finished your queries.

To perform a cone search around a given RA/Dec use:

from gaiaoffline import Gaia
with Gaia() as gaia:
    results = gaia.conesearch(ra=45.0, dec=6.0, radius=0.5)

This should give a result that looks like the following:

	source_id	ra	dec	parallax	pmra	pmdec	phot_g_mean_flux	phot_bp_mean_flux	phot_rp_mean_flux	radial_velocity	teff_gspphot	logg_gspphot	mh_gspphot
0	7090720423502592	44.509845	5.929635	0.747558	6.432207	4.551372	16856.086102	8409.319290	12121.273134	NaN	5114.8706	4.2540	-1.3869
1	7097317492675328	44.519449	6.081799	4.258604	5.068064	5.079538	47298.968398	27576.148423	72616.670451	0.469183	NaN	NaN	NaN
2	7097317493262720	44.519675	6.081777	NaN	NaN	NaN	34481.298446	NaN	NaN	13.451580	NaN	NaN	NaN
3	7096905176403968	44.525031	6.043208	0.081763	-0.017625	-2.614324	9935.154498	4022.459319	8514.830261	NaN	NaN	NaN	NaN
4	7103188712818304	44.526470	6.097983	1.191807	7.503639	-10.218165	84525.893200	44516.084209	57364.797278	-0.059489	6297.8930	4.0977	-0.4296

You can add a magnitude limit to your conesearch using the following code. This will execute larger searches faster by applying the magnitude limit first.

from gaiaoffline import Gaia
with Gaia(magnitude_limit=(-3, 10)) as gaia:
    results = gaia.conesearch(ra=45.0, dec=6.0, radius=0.5)

You can include the 2MASS crossmatch data using

from gaiaoffline import Gaia
with Gaia(tmass_crossmatch=True) as gaia:
    results = gaia.conesearch(ra=45.0, dec=6.0, radius=0.5)

The default is to output fluxes in the catalog, but you can switch to magnitudes using

from gaiaoffline import Gaia
with Gaia(photometry_output='mag') as gaia:
    results = gaia.conesearch(ra=45.0, dec=6.0, radius=0.5)

If you are doing a large query and want only the top 10 results to test the query, you can use

from gaiaoffline import Gaia
with Gaia(limit=10) as gaia:
    results = gaia.conesearch(ra=45.0, dec=6.0, radius=0.5)

Any of the above can be used in combination.

License

This project is licensed under the MIT License.