fpbot 1.1.2

Forced photometry pipeline for the Zwicky Transient Facility

fpbot

This package a Forced Photometry Pipeline based on ztfquery and ztflc. It needs IPAC access to download the images, as well as access to the AMPEL Archive to obtain information on the transients.

If you are planning to run forced photometry on many ZTF transients, this is the right tool for you!

Note: Requires Python >= 3.10. Also requires a MongoDB instance for storing the metadata, reachable under port 27017. This can be modified in database.py.

Installation

1. Note that libpq-dev needs to be present. On Debian/Ubuntu, issue sudo apt install libpq-dev. On Mac OS, run brew install postgresql.

2. Then install via: pip install fpbot. Alternatively, clone this repo and install it with poetry. To do so, run

git clone https://github.com/simeonreusch/fpbot.git
cd fpbot
poetry install

3. If MongoDB is not present, it can easily be installed. On Debian/Ubuntu, just follow this instruction set. After this, make sure the demon runs. Issue

sudo systemctl start mongod
sudo systemctl enable mongod

On MacOS, make sure brew is present. To do so, follow this tutorial.

4. fpbot requires an environment variable to know where to store the data. Include a line in your .bashrc or .zshrc like export ZTFDATA='/absolute/path/to/ZTF-data-folder/'. If you don't need AMPEL access, you are done!

---

5. If you want to use the AMPEL API for alert data (you don't have to!), you need credentials for the API. You can get these here.

6. NOTE: If you are planning to run fpbot on a headless system which does not provide the luxury of a systemwide keychain, please add export ZTFHUB_MODE='HEADLESS' to your .bashrc or .zshrc. The pipeline will then uses ztfquery's base64-obfuscated password storage.

ALTERNATIVE: Use Docker container

fpbot comes shipped with a Dockerfile and a docker-compose.yml. Use them to build the docker container (this includes all dependencies as well as a MongoDB instance). Note: You have to provide a .ztfquery file in the fpbot directory containing access data for ztfquery (see ztfquery or ztflc for details).

First, do the following:

git clone https://github.com/simeonreusch/fpbot.git
cd fpbot
docker-compose build

in the directory containing 1) the Dockerfile, 2) the docker-compose.yml and 3) the .ztfquery credentials file and run with

docker-compose run -p 8000:8000 fpbot. This exposes the web API to port 8000 of your local machine.

Troubleshooting

In case way too few images are downloaded, check your IRSA credentials. These are stored in ~.ztfquery. If there is a problem with these, ztfquery will not complain but simply only download publicly accessible images.

Usage

By importing class

All functionality of the command-line tool is present in the class. Just call it according to the commands available in pipeline.py.

For example:

from fpbot.pipeline import ForcedPhotometryPipeline

pl = ForcedPhotometryPipeline(
    file_or_name="ZTF19aatubsj",
    daysago=90,
    nprocess=24
)

pl.download()
pl.psffit()
pl.plot()

By systemwide command (fp name -operations --options)

Always:

name A ZTF name has to be provided, or an ASCII file containing one ZTF name in each line or an arbitrary name if followed by the ra/dec-option as to be provided.

optionally:

-radec [RA DEC] If this is given, the name can be chosen arbitrarily (but a name MUST be provided). Radec must be given in a format that can be parsed by astropy; e.g. -radec 218.487548 +40.243758.

Additional commands

-dl Downloads the images used for forced photometry from IPAC. Needs a valid IPAC account.

-fit Performs the PSF-photometry fit and generates plots of the lightcurve(s).

-plot Plots the lightcurve(s).

-plotflux Plots the lightcurve(s), but with flux instead of magnitude.

-sciimg Experimental: Also downloads the science images from IPAC (note: to create thumbnails if specified)

-thumbnails Experimental: Generates thumbnails for all science-images. Science images have to be downloaded (see -sciimg)

Options

--nprocess [int] Specifies the number of processes spawned for parallel computing. Default is 4. Note: download is always performed with 32 processes in parallel, as IPAC upload-speed is the bottleneck there.

--daysago [int] Determines how old the photometric data should be. Default: all.

--daysuntil [int] Determines how new the photometric data should be. Default: all.

--snt [float] Specifies the signal-to-noise ratio for plotting and SALT-fitting.

--magrange [float float] Defines upper and lower magnitude bound for plotting the lightcurves; order is irrelevant.

--fluxrange [float float] Defines lower and upper flux bound for plotting the flux lightcurves; order is irrelevant.

Examples

fp ZTF19aatubsj downloads this ZTF object, does forced photometry, plots it and saves it to the default directory in "forcephotometry" (ZTFDATA, located at $ZTFDATA in your .bashrc/.zshrc/..., see ztfquery doc).

fp ZTF19abimkwn -dl -fit --nprocess 16 downloads all images for ZTF19abimkwn found on IPAC, performs PSF-fitting and plots the lightcurve with 16 processes in parallel.

fp supernovae.txt -dl -fit Downloads all difference images for ZTF transients found in supernovae.txt, each line a ZTFname. These are then fitted, but not plotted. To get a nice example of ZTF lightcurves, issue: fp example_download.txt -dl -fit -plot.

fp this_looks_interesting -radec 143.3123 66.42342 -dl -fit -plot --daysago 10 -magrange 18 20 Downloads all images of the last ten days of the location given in RA and dDecec, performs PSF-fits and plots the lightcurve in the 18--20 magnitude range.

By systemwide bulk command (fpbulk file.txt -operations --options)

file.txt must be an ASCII file containing one ZTF-ID per line. The usual options apply (e.g. -dl, -fit).

Requirements

• ztfquery is used to download the image files from IPAC.
• ztflc is used for PSF-fitting.
• AMPEL credentials are neccessary for the pipeline to work.

Notes

Slackbot

There is a bot for Slack included, based on the SlackRTM-API. You have to create a classic Slack app for this, because the newer version depends on the Events API, which itself seems to need a web server to run. Classic slack Apps can be created here. Make sure not to convert to the new permission/privilege system in the process (Slack tries to push you towards it, be careful). After successfully setting up the App/bot and giving it permissions, change the bot-username to the one of your bot in start_slackbot.py and it should basically work (first start requires you to enter the bot- and bot-user credentials, also provided by Slack).

Resulting dataframe

The dataframes resulting after plotting (located at ZTDATA/forcephotometry/plot/dataframes) consists of the following columns:

• sigma(.err): The intrinsic error
• ampl(.err): The flux amplitude (error)
• fval: Total minimized value
• chi2(dof): PSF-fit chi square (per degrees of freedom)
• Columns 9-39: The science image header
• target_x/y: pixel position of target
• data_hasnan: Data contains NaN-values (should always be False)
• F0: Zero point magnitude from header converted to flux
• Fratio(.err): Flux to flux zero point ratio (error)
• upper_limit: For forced photometry result < signal to noise threshold, this is the limiting magnitude from the Marshal (see maglim column)
• mag(_err): Flux amplitude (error) converted to magnitude. For detections below signal to noise threshold, this value is set to 99.