Welcome to Live Pipeline User Manual#

The Goodman Spectroscopic Pipeline Live is a web implementation of the Goodman Spectroscopic Pipeline package. In summary the reduction process has been broken in several parts that might include more than one step of the full process. Each of these parts has a correspondent web API end point.

All this is nicely presented to the user as a web page where the user can visualize fully reduced spectra just seconds after it was obtained. Also the user has the option of manually triggering certain process blocks.

Note

The user get access permission by proposal association. We ask our users not to share their credentials. We aim to make it very easy for PIs to add new collaborators.

Overview#

We use Django and Django Rest Framework for build our backend API.

The goodman-pipeline package processes are very granulated as it should be. But for a web application such as ours we found out that the best option was to group many of the processes in blocks as described below. But in an observing run there are many types of files. We have calibration and science files.

File Types#

Calibration files:: Calibration files are BIAS and FLAT files. They need to be combined into master BIAS and master FLATs. Each proposal has also associated a dedicated data reduction settings and there can be defined, among other things, what are the minimum number of calibration files required to trigger a combination of a master BIAS or FLAT.
Science files:: Science files are EXPOSE SPECTRUM and ARC though one can argue that ARC are calibration files, in a general sense they are treated as science files in this case.

Process Blocks#

Reduce

Identifies suitable master flats and master bias and applies the basic data reduction process. For spectroscopic data it also trims the non-illuminated areas which produce artifacts. In general the process looks like this:

An EXPOSE files, which is the type given to imaging data can have the following process.

Image Trim → Bias Subtract → Flatfielding → Cosmic Ray Removal

An SPECTRUM file will have the following:

Image Trim → Slit Trim → Bias Subtract → Flatfielding → Cosmic Ray Removal

Extract

Identifies targets in a SPECTRUM file and finds a suitable and compatible ARC lamps and performs fractional pixel extractions.

Identify → Trace → Extract

Wavelength Calibration

Using the extracted lamps it performs the wavelength calibration process using a template lamp from a library and series of cross correlations, then fits a mathematical model, resamples the spectrum to a linear dispersion axis and saves the file.

Flux Calibration

This is not implemented yet but it will.

Astrometric Solution

This functionality is under development.

Getting Access#

Warning

We allow everyone to have an account. Please don’t share your credentials with anyone.

Any astronomer with an observing proposal can have access. Principal Investigators (PIs) can also create users to be added as collaborators to their respective proposals.

The server is protected under a VPN connection that you can get from your Instrument Scientist or by any trusted contact you have at SOAR Telescope.

At the beginning of the semester all PIs are registered with their respective proposals. And they will receive an email that contains a link to set a safe password. Our backend does not allow simple or too common password. In fact you will get the following messages.

Your password can’t be too similar to your other personal information.

Your password must contain at least 8 characters.

Your password can’t be a commonly used password.

Your password can’t be entirely numeric.

Data Visualization#

One of the greatest benefits from having a web user interface (UI) is the great flexibility it provides without having to install any special software and the data is downloaded through channels we use in our everyday life.

Understanding the UI#

We will go into details later but for now this is what you see most of the time. It is meant to be simple to understand but we will go into explaining every part.

_images/data_table.png — Search view, implements some filtering functions.#

For a detailed exploration we will divide the view in several subsections, for their global position please consider the following image.

_images/data_detail_annotated.png — Detail: 1. Access to user controls and information. 2. Main navigation bar. 3. Quick access to files. 4. Related or child files. 5. Toggle header view. 6. Visualization area. 7. Visualization options. 8. Contextual actions. 9. Comments.#

Main Navigation Bar#

An always present navigation bar allows you to access:

Data Visualization: The view were you can visualize data as it arrives as well as trigger actions.
Search: Changes to Search view and implements filters.
Proposals: Where you can see the proposals details, such as Title, Abstract, Principal Investigator and Collaborators. Also you can add or delete collaborators and edit the Data Reduction Settings.
Logs: Important log messages, not very detailed but the most relevants.
Management Menu: This is for staff users only and will be explained in the staff chapter.
API Docs: Contains the documentation of the Public API in case you want to develop your own tools.

User Controls#

By clicking the username button you get access to some user details and settings. Also you can change some preferences.

File Index#

This area allows you to navigate the files, they are ordered from newest to oldest, if you want to filter the data you should go to Search.

In this area you can also see only combined calibration files such as, master BIAS and master FLAT, those are visible to everyone.

Visualization Choices#

Allows to select a different color map for images, show or not saturated pixels and an image title.

Contextual Actions#

This area is where you can trigger actions.

Download: For download the FITS file.
Advanced Visualization: Takes you to another view where you can adjust the sampling limits for images. Or change the color map.
Calibrate Flux: It’s an action that changes according the type of file that is on display. For instance for a raw file will be Reduce Raw File.
Delete: Only possible for processed files. Raw files can’t be deleted.

Visualization Area#

In this area you see the plots or the images. There is an interactive way that uses Bokeh but is slower. and for now it only allows you to zoom in or out.

Data Reduction Settings#

In your navigation bar click in Proposals then the cogs icon.

Now you can change the settings that will be used for this particular proposal in all reduction steps and in every new file.

Collaborators#

Click in Proposals and then the Person + icon.

Now you can add or remove collaborators for all the proposals you have permissions to do so. If you are a collaborator you can’t add or remove other collaborators.

API Documentation#

More on this can be found on Developer’s Guide chapter. You can get here from clicking API Docs in the navigation bar.

In this section you can even try the API if you are authenticated.

_images/api_swagger_documentation.png — Public API documentation.#

Screenshots#

Adding new Users#

Warning

This section is meant for staff users only

There are several ways you can access the user creation form. The faster would be:

Management → Add User

Another way that will take you to the same form is:

Management → Manage Users → click Add new

_images/staff_new_users.png — Add new user form#

Note

Staff users can only be added from a super user account for now.

And the last one would be through Proposals’s Collaborators

Adding new Observing Proposals#

Warning

This section is meant for staff users only

If you created a user as described in Adding new Users you will be redirected to the proposal creation form However there are other ways too.

Management → Add Proposal

and

Management → Manage Proposals → click Add new

_images/staff_new_proposals.png — Add new proposal form#

Make sure the SOAR ID is correct or PI or collaborators will not be able to see their data. The Soar id default value is set automatically depending on current semester and count.

Overview#

The Public API is thoroughly documented using swagger.

API Examples#

Warning

Always trust the API docs page instead because it is updated along the API itself.

Get the list of raw files#

import requests

api_key = 'some-fake-api-key-that-does-not-work'

# define request headers
headers = {'Authorization': f"Token {api_key}",
           'Content-Type': 'application/json'}

#define parameters
# 0: Raw files
# 1: Calibration Files (Master flats or bias)
# 2: Reduced files
# 3: Extracted 1D spectrum
# 4: Wavelength calibrated
# 5: Flux Calibrated
parameters = {"data_type": "0"}

# remotehost should be replaced with the actual host
# also the url can be composed as http://remotehost/api/files/?data_type=0
# in that case you don't need to pass the params argument.
response = requests.get(
         url='http://remotehost/api/files/',
         params=parameters,
         headers=headers)

if response.status_code == 200:
    json_response = response.json()

# alternatively you can use response.raise_for_status() and catch any exception

from request.exceptions import HTTPError

try:
    response.raise_for_status()
except HTTPError as http_error:
    log.exception(http_error)

Will produce a response like this. In this particular case the results have been truncated to just a couple of results.

{
    "count": 545,
    "next": "http://remothost/gsp/api/files/?data_type=0&page=2",
    "previous": null,
    "results": [
        {
            "id": 952,
            "created": "2021-02-18T10:22:10.340053-03:00",
            "last_modified": "2021-02-18T10:22:13.470928-03:00",
            "original_file": "0001_GHTS_B_400m1_2x2_18-02-2021.fits",
            "original_file_id": 952,
            "parent_file": "0001_GHTS_B_400m1_2x2_18-02-2021.fits",
            "parent_file_id": 952,
            "file_name": "0001_GHTS_B_400m1_2x2_18-02-2021.fits",
            "directory_name": "/pipeline/data/20210218/RAW",
            "full_path": "/pipeline/data/20210218/RAW/0001_GHTS_B_400m1_2x2_18-02-2021.fits",
            "obstype": "LAMPFLAT",
            "object": "DFLAT",
            "filter": "NO_FILTER",
            "filter_2": "NO_FILTER",
            "grating": "400_SYZY",
            "slit": "1.0_LONG_SLIT",
            "cam_targ": "11.6",
            "grt_targ": "5.8",
            "obsra": "17:38:33.320",
            "obsdec": "-48:23:59.899",
            "gain": "1.4",
            "rdnoise": "4.74",
            "roi": "Spectroscopic 2x2",
            "wavmode": "400_M1",
            "proposal_id": "calibrate",
            "observation_id": null,
            "configuration_id": null,
            "block_id": null,
            "image_id": 1,
            "date": "2021-02-18",
            "data_type": "0",
            "normalized": false,
            "technique": "Spectroscopy",
            "saturation_value": 49928
        },
        {
            "id": 900,
            "created": "2020-11-26T21:30:52.692308-03:00",
            "last_modified": "2020-11-26T21:30:55.204647-03:00",
            "original_file": "0354_Orion-Burst_26-11-2020_comp.fits",
            "original_file_id": 900,
            "parent_file": "0354_Orion-Burst_26-11-2020_comp.fits",
            "parent_file_id": 900,
            "file_name": "0354_Orion-Burst_26-11-2020_comp.fits",
            "directory_name": "/pipeline/data/20201126/RAW",
            "full_path": "/pipeline/data/20201126/RAW/0354_Orion-Burst_26-11-2020_comp.fits",
            "obstype": "ARC",
            "object": "ZTF20actqfpc",
            "filter": "NO_FILTER",
            "filter_2": "GG455",
            "grating": "400_SYZY",
            "slit": "1.0_LONG_SLIT",
            "cam_targ": "16.1",
            "grt_targ": "7.5",
            "obsra": "05:34:22.372",
            "obsdec": "-5:24:52.448",
            "gain": "1.48",
            "rdnoise": "3.89",
            "roi": "Spectroscopic 2x2",
            "wavmode": "400_M2",
            "proposal_id": "SOAR2020B-008",
            "observation_id": 1,
            "configuration_id": 1,
            "block_id": 0,
            "image_id": 34304,
            "date": "2020-11-26",
            "data_type": "0",
            "normalized": false,
            "technique": "Spectroscopy",
            "saturation_value": 69257
        },
        {
            "id": 895,
            "created": "2020-11-26T19:37:53.519510-03:00",
            "last_modified": "2020-11-26T19:37:55.892156-03:00",
            "original_file": "0353_Orion-Burst_26-11-2020_comp.fits",
            "original_file_id": 895,
            "parent_file": "0353_Orion-Burst_26-11-2020_comp.fits",
            "parent_file_id": 895,
            "file_name": "0353_Orion-Burst_26-11-2020_comp.fits",
            "directory_name": "/pipeline/data/20201126/RAW",
            "full_path": "/pipeline/data/20201126/RAW/0353_Orion-Burst_26-11-2020_comp.fits",
            "obstype": "ARC",
            "object": "ZTF20actqfpc",
            "filter": "NO_FILTER",
            "filter_2": "GG455",
            "grating": "400_SYZY",
            "slit": "1.0_LONG_SLIT",
            "cam_targ": "16.1",
            "grt_targ": "7.5",
            "obsra": "05:34:22.372",
            "obsdec": "-5:24:52.448",
            "gain": "1.48",
            "rdnoise": "3.89",
            "roi": "Spectroscopic 2x2",
            "wavmode": "400_M2",
            "proposal_id": "SOAR2020B-008",
            "observation_id": 1,
            "configuration_id": 1,
            "block_id": 0,
            "image_id": 34304,
            "date": "2020-11-26",
            "data_type": "0",
            "normalized": false,
            "technique": "Spectroscopy",
            "saturation_value": 69257
        }
    ]
}

Add Collaborator to Proposal#

import requests

api_key = 'some-fake-api-key-that-does-not-work'

# define request headers
headers = {'Authorization': f"Token {api_key}",
           'Content-Type': 'application/json'}

#define payload
payload = {
    "email": "user@server.net",
    "action": "add"
    }


# remotehost should be replaced with the actual host
response = requests.get(
         url='http://remotehost/api/proposals/collaborators/5/',
         data=payload,
         headers=headers)

if response.status_code == 200:
    json_response = response.json()

You get a serialized version of the proposal database instance.

{
    "id": 5,
    "soar_id": "calibrate",
    "semester": "2020A",
    "title": "Calibrations",
    "abstract": "How calibrations will be handled.",
    "user": {
        "id": 3,
        "username": "observer",
        "last_login": "2020-10-24T18:56:45.976000-03:00",
        "first_name": "Observer",
        "last_name": "Observer",
        "email": "observer@observatory.cl",
        "is_staff": true,
        "is_active": true,
        "date_joined": "2019-11-13T10:26:23-03:00"
    },
    "collaborators": [
        {
            "id": 41,
            "username": "user",
            "last_login": null,
            "first_name": "",
            "last_name": "",
            "email": "user@server.net",
            "is_staff": false,
            "is_active": true,
            "date_joined": "2021-03-24T15:32:23.329399-03:00"
        }
    ]
}

Since the user did not exists a new user was created.

Welcome to Live Pipeline User Manual#

Overview#

File Types#

Process Blocks#

Getting Access#

Data Visualization#

Understanding the UI#

Main Navigation Bar#

User Controls#

File Index#

Related Files#

Header#

Visualization Choices#

Contextual Actions#

Visualization Area#

Data Reduction Settings#

Collaborators#

API Documentation#

Screenshots#

Adding new Users#

Adding new Observing Proposals#

Overview#

API Examples#

Get the list of raw files#

Add Collaborator to Proposal#