Tags:
create new tag
view all tags

Observation Portal (Share and track observations)

This page gathers initial ideas & inputs related to improve observers deal with their programs (many targets / OB) and perform (real-time) tracking, scheduling ...

Context

General scope: VLTI and CHARA observations offered to any observer: end-user, large programs, instrument teams (GTO?) like GRAVITY, MATISSE, VEGA teams.

This sketch illustrates the existing tools:

JmmcUserPortal.png

- JMMC Aspro2 (http://www.jmmc.fr/aspro):

This Java GUI to help astronomers prepare their observations.

The session is stored in asprox files (xml based):

  • Portfolio: name, general log (text)
  • Target list + calibrators + ancillary stars (AO / FT) with their models + target notes + specific constraints (HA ranges...) + user groups (backup, priority ...)
  • Interferometer setup: observatory + period (eso) + array setup: configurations (baselines) + PoPs / beams (chara only) + AO setup (VLTI:
  • Instrument setup: instrument mode (resolution + wavelength range) + sampling (periodicity + obs. time)
  • Constraints: elevation, date ?

ASPRO2 computes observability (target can be observed with current setup during the ... range), UV coverage (coordinates in the fourier plane) that illustrates the target sampling that the user can compare with previous observations using the JMMC OIFits Explorer tool (OIFITS standard files contains targets & uv coordinates already).

ASPRO2 can export Observing blocks (O.B.) to ESO p2 thanks to the A2P2 tool: SAMP integration + specific OBxml exchange format (including observable ranges expressed as HA and LST ranges).

For history ASPRO2 / PIVOT (deprecated) integration was a powerful combination to prepare & manage VEGA programs (Denis Mourard - Instrument PI). PIVOT GUI can import ASPRO2 OBs in its database (proposal) and send global schedule to ASPRO2 (targets with the same interferometer / instrument setup) for the Run PI to finalize the night preparation i.e. the proper night schedule.

- JMMC A2P2 (https://github.com/JMMC-OpenDev/a2p2):

A2P2 is a python tool (CLI or GUI) that acts as a SAMP bridge between ASPRO2 and ESO p2 / CHARA backends. This is an open source project maintained by JMMC and ESO instrument scientists (GRAVITY, PIONIER, MATISSE OB constraints). It handles login to ESO portal (user account) and convert ASPRO2 obxml to fill ESO OB templates (generic time ranges) and submit OBs into ESO p2 portal.

For CHARA, it only logs OBs as text log (ASCII art) but in the future A2P2 could submit OBs into CHARA backend (cosmic debris).

- JMMC OiDB (http://oidb-beta.jmmc.fr/index.html):

This JMMC web portal contains both (VLTI / CHARA) observation logs (L0) and OIFITS data (L2 & L3). It is based on a TAP server (taplib + postgres + pgsphere extension) and will soon support private collections (restricted access to data files, meta-data are always public).

Information is stored according to the ObsCore IVOA schema (see http://www.ivoa.net/documents/ObsCore/) (data PI, program Id, target, facility, instrument, time & wavelength range) + minor extensions. Several integrations are in place to perform authentication (JMMC custom user db), query SIMBAD (target validation), ESO logs (cds B/ESO catalog) and OIFits Explorer to provide quick plots.

Metadata description: http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/OiDbColumns

The database could evolve to provide the array setup (configuration) and/or baselines (UV coordinates ?) for observation data...

Examples:

- JMMC OITools / OIFits Explorer:

The OITools open-source library () deals with OIFITS file handling (read / write + processing) and OIFits Explorer () provides plots ...

These tools provides a command line interface and could be helpful to scan reduced OIFITS files (local or in OiDB) & extract detailed observation logs (target + configuration + instrument + UV coordinates)...

- ESO p2 (http://www.eso.org/p2): Phase-2 preparation tool (once Observation proposals are accepted)

P2 is the Science user portal that manages user OBs (corresponding to an accepted proposal) before running observations (service or visitor modes).

Such OB contains detailed setup (instrument, AO specific fields) ...

- ESO Archives:

ESO Obs schedule (http://archive.eso.org/wdb/wdb/eso/sched_rep_arc/form) : this gives the schedule of the observatory (done or planned observations with prog_id / abstract). For example: this query lists recent program ids accepted on the MATISSE instrument: http://archive.eso.org/wdb/wdb/eso/sched_rep_arc/query?wdbo=html%2fdisplay&max_rows_returned=100&tab_period=on&period=&tab_obs_mode=on&tab_run_type=on&run_type=%25&start=&tab_nights=on&tel=%25&tab_instrument=on&instrument=MATISSE%25&instrument_user=&progid=&tab_pi_coi=on&pi_coi=&pi_coi_name=PI_only&title=&tab_dp_id=on&tab_publications=on&order=PERIOD_PROGID&

The ESO raw archive gathers observation logs associated to a run PI or program Id. For example:

http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?prog_id=60.A-9134(A)&max_rows_returned=1000

It gives access to detailed information (instrument modes, configuration + uv coordinates from raw file headers).

It also provides specific search forms for VLTI instruments: http://archive.eso.org/cms/eso-data/instrument-specific-query-forms.html (explaining some interesting keywords)

Note: astroquery provides a python module to query the ESO archive: see https://astroquery.readthedocs.io/en/latest/eso/eso.html

note: ESO archive provides a TAP access point: http://archive.eso.org/programmatic/#TAP

It contains also raw information in dbo_raw table (but seems very slow).

Preliminary ideas / Features

JMMC Observation portal / ObsDB

  • user login + team (group) management to share observation preparation (confidential); super user & instrument PI can bypass visibility restrictions at certain stages (submitted proposals)
  • Adopt TAP interface (VO) ? to allow cone-search queries ? See ObsLocTAP (http://www.ivoa.net/documents/ObsLocTAP/index.html) and OIDB schema (L0)
  • Custom REST API to upload data (observation proposals...)
  • Integrate and process OiDB meta-data to present a simple view on all observation log (VLTI / CHARA L0 to L3). The tricky point consists in gathering individual obs logs per nights, get array configuration (baselines) and UV tracks (coordinates ?) but also deal with duplicates ...
  • Database design: observation proposals + logs (relations / links ?)
  • Basic workflow ? proposals -> accepted -> planned -> done / cancelled
  • Simple Web interface ?
A2P2 enhancements to be the Obs portal client and share information with ASPRO2 (xml + samp)
  • user login
  • new SAMP messages + ObsPortal REST client / ObsTAP client ?
  • CHARA integration (OB submission)

Support observation tracking in ASPRO2

  • First improve asprox & obxml DM to incorporate observation logs (PI, program ... + obs ranges / configuration / UV tracks); see new ESO UV plane optimiser tool (python)
  • Enter manually logs
  • Scan local OIFITS files => ingest observation logs

  • Update Observability / UV Coverage plots + adapt portfolio setup (program...)
  • Enhance A2P2 integration to deal with Obs Portal

Share observation preparation (ASPRO2 into ObsPortal)

Share up-to-date information about observations (log, progress, status, quality information) from ASPRO2 into ObsPortal

  • sharing asprox files could help (as a first step) ? versioning ?
  • To mimic PIVOT integration, ASPRO2 can load a VOTable (target + setup) to dynamically import observations from the ObsPortal ?

For now Asprox files acts as a local storage that users can share with colleagues (send document by email): it is generic.

Note: One science target can correspond to mutiple OBs on several nights (different setups) !

Other ideas:

  • Night scheduling ?
Observation proposal => night scheduling : complicated task in ASPRO2
  • Boards for Instrument scientists / runs ? merge OBs into nights ?

Priority / Actions

Observation tracking

Objective:

  1. get observations logs from (ESO) archives to indicate what observations have been done in ASPRO2
  2. show the Calibrator OBs observed corresponding to the Science targets (CAL - SCI - CAL sequence): it supposes to interpret the OB list along the night to retrieve the SCI-CAL relationships.

Search criteria:

  • Per object (cross match by position) : TAP / cone search is needed as target name are not enough
  • others ? program id, instrument, date ranges ?
  • the user will be able to filter in ASPRO2 which OBs are displayed (i.e. enabled) by configuration (same baselines or not), instrument, instrument mode (spectral resolution)

ASPRO2 changes:

  • show observed LST ranges in Observability plot : every OB is represented by one range (exposure) and the OB details can be displayed in a tooltip
  • show the OB list in a new table widget with filtering options (to be refined)
  • store OBs in asprox files to keep this information in offline mode
  • handle OB list updates (regularly or manually)
  • show the concrete UV points on the UV Coverage panel

Open issues:

  • quality flags: how to get ESO QC flags or night logs ? seeing values could give a rough indication
  • granularity: should OB be grouped or not by instrument setup or night ? for now, every OB corresponds to 1 measurement (few minutes) but headers give LST / UV ranges per baseline => time / UV coverages must be expressed as value ranges
  • instrument mode:
    • the mapping between ASPRO2 modes and ESO OB keywords is instrument-specific. TODO: define rules per instruments
  • configurations:
    • ESO OB can have been observed at stations that are not present in ASPRO2 offered configuration: it depends on the period or it can be a telescope relocation issue
    • mapping rules are needed to interpret these cases and find 'equivalent' configurations like small, medium, large configurations (new in ESO Period 104)
  • note: Target groups could be used to group targets by program id, instrument setups or nights in the future.
WARNING a cone search on ESO's headers for many interferometric stars is bound to fail as the COORDINATES in the HEADER are NOT precise enough. Similarly the OBJECT NAME is not well standardized and sometimes ABSENT. It is thus not possible to search for a star unsing its SIMBAD coodinates or its SIMBAD name.

Proposed strategy: find the headers in the database (observed objects) that are not found when looking for the coordinates, and/or the various SIMBAD names, of the objects listed in the headers. Report this list to ESO as invalid headers. Eventually, propose to help them working on a solution.

Data model: see JmmcObsPortalDataAnalysis

Use cases:

  1. GRAVITY / AMBER (Gilles Duvert)
    • YSO large program ids:
      • 099.C-0667(A,B,C) ---> seul B a des données.
      • 0103.C-0347(A,B,C,D,E)
      • 100.C-0278(A...L)
      • 0102.C-0408(A..D)
  2. MATISSE (Alexis Matter)
  3. PIONIER (JBLB)

TODO

  • Study data models (asprox, obxml, p2, eso cfp, archive ...) to extract missing information in every component (oidb, obsPortal, aspro2) and define properly shared information
  • Data life cycle & Software Architecture

  • Build a tiger team ...
  • SPICA requirements ?

Other links of interest

Corner cases in Obs logs:

Data analysis

Software

Architecture and database design

Deployment

  • Application
    • Repository : https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/JMMC/jmmc-obsportal
    • Source code uses the Git-Flow pattern :
      • The branch "develop" is the main integration branch merging all developments
      • The branch "master" is the production branch ready to deploy on a server
    • Instructions : see the "Readme.md" file at the root of the project to have up-to-date instructions

Usage

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng JmmcUserPortal.png r1 manage 200.4 K 2019-07-02 - 13:39 LaurentBourges  
Edit | Attach | Watch | Print version | History: r20 < r19 < r18 < r17 < r16 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r20 - 2020-02-07 - GillesDuvert
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback