Here comes the release notes of the amber data reduction software

V3.0.8 (21-July-2014)

• VISAMP (amplitude of the differential visibility) is now correctly computed. A consequence is the addition of 3 new 'colums' (if the correction is applied) in the OI_VIS table of the OI-FITS files produced by amdlib. The 'columns' are named T3BIAS, T3INTERBIAS and VISCOVRI and should be normally ignored by all OI-FITS related programs.
• correction for phase closure bias as described in http://www.aanda.org/articles/aa/pdf/2012/05/aa17335-11.pdf Please note that, although this correction is enabled by default in the (seldom used) amdlib commandline mode, it is disabled (option -T) in the yorick amdlib program.
• Fix missing checkClosureBiasRemoval calls
• The yorick version of amdlib still has an issue with compressed P2VM files ending in '.Z'. In this case, gunzip the culprit P2VM and retry.

V3.0.7 (05-Sep-2013)

• fix minor bug for amdlibPreferences using binary distribution of amdlib in terminal. E.g. catalogsDir was not found.
• fix bug on recent linux distributions that didn't interpret properly big numerical values. E.g. 1e500 was ignored reading yorick code and generated Syntax errors.
• improve calibration routines to deal with fits file with empty keyword.
• amdlib launched from terminal provides autocompletion. Use tab to complete with filenames or main amdlib commands.

V3.0.6 (17-Apr-2013)

• corrected bug reported by Armin Gabasch (ESO) where a zeroing of arrays used in amdlibComputeSpectralCalibration was not properly done depending on processor architecture. Possibly explains in retrospect some spectral calibration problems.
• update specCalShifts in amdlibComputeP2vm with the default value for LR and MR but HR since the intervention of 2013-01.Ticket #451
• reduce the length of the INFO field into the browser, to better display info (fix bug on some linux boxes). Ticket #456
• amdlibCalibrateAllOiData now reorder the baselines before writting the OIDATA product file so that they should appear in the same order as in the RAW data (if not possible, an warning is issued). Ticket #367.
• Patch (to be tested thoroughly): The repositioning errors of the grating/prism turret are not taken into account by the existing observation software (on the instrument at Paranal). Consequently the wavelength for each spectral channel may be inaccurate: they are shifted by an unknown amount (between 0 and 15 pixels). This is somewhat corrected during the computation of the P2VM matrix (command amdlibComputeP2vm) by measuring the position of the phase shift between the system of fringes in H and the one in K. The autoShift mode (option -a) of amdlib has been introduced to this effect. However, there have been reports that the wavelength were sometimes shifted by an enormous and unphysical amount, mainly in recent (> 2011) observations. It seems that the algorithm is too sensitive to the exactness of the displacement of the photometric channels (another AMBER curse due to misalignment of the roof prisms used to separate photometric channels). With this patch, computation of the p2vm proceeds in 2 steps. The first step uses 'global photometry' to get an estimate of the wavelength shift, since with global photometry one does not depend anymore on the photometric channels displacement. The second step is done as usual and computes the rest of the p2vm values. Note that in LR mode the value of the photometric channels displacement is really important since we need to separate the J H and K bands! Please be aware that this step is crucial and should not be used without double-checks of the wavelength table (example: if you notice that your wavelength range is within the visible, well, something's wrong!)
• update yorick revision (yorick-2.2.00x) that fixes some bugs encountered by linux users on recent machines.

V3.0.5 (17-Dec-2012)

This release gets some minor bug fixes but provides one source package that runs on MacOS Mountain Lion.

The user manual has been updated with a new paragraph on spectal shifts.

V3.0.4 (15-June-2012)

This release mainly concerns the "core" of amdlib, i.e., the C code library and all the amdlibXXX shell commands and their numerous options. Note that while the namesake amdlib Yorick interface, which presents simpler options and add visualization tools, should be preferred over the shell commands for the general user, reading these notes can be enlightening.

1. added option "-w " to amdlibComputeP2vm. This option permits to pass a new value for the spectral resolution, in nanometers. This should be used in conjunction with the -S option to correct the wavelengths of AMBER. Based on repeated measurements, AMBER not only suffers from a bad repositioning of the grating turret but also of a false spectral dispersion formula. The former produces the global shift of the spectrum, which is not corrected at time of observation but can be measured afterwards on known atmospheric absorption lines for example. This global shift is cured by the -S option where one can shift the wavelengths accordingly. In addition, the "-w" option permits to give a better spectral dispersion value for the 2 gratings. I have found that in Medium Resolution from H to K band, a constant value per pixel of 0.6858 nm gives perfect results. For all Low Resolution observations, do not panic, the shift is done automatically and a correct wavelength table is applied.
2. All the history of commands is now stored in the HISTORY keywords of the P2VM and OI-FITS files.
3. Added a new command, amdlibP2vm2Oidata. This command uses the P2VM calibration files as if they were normal observation files and produces an OI-FITS. This give insights on the signature of the P2VM in all the other observations. There is no closure data since the P2VM does not calibrate the closure.
4. Slightly changed the noise formula estimate for closures.
5. Documented all the options of amdlibComputeOiData.
6. Added Keyword NULLVALS in OI-FITS headers (gives the blanking value used by amdlib at the time of data reduction).
7. Photometry is now also computed on the Interferometric Channel and gives another two columns "INTERF_SPECTRUM" and "INTERF_SPECTRUM_ERROR" in the AMBER_DATA table of our OI-FITS. (This Photometry is the one that should be used when checking the spectral calibration.)

Here comes a new amdlib version aimed at speedup!:

1. the reduced Chi2 of the fringe fit is kept and stored in the raw visibilities OI-FITS files (table AMBER_DATA, column FRINGE_CHI2). To be complete, we should add in a next release the possibility to select visibilities according to this chi2 in amdlibPerform[Frame]Selection.
2. contrary to a well enforced useage, tagged values are no more blanked when it is possible. I should have set up a switch to permit this, but it would have been too complicated. So only the flags will really tell if a value is deemed wrong or not, which is what the OI-FITS norm expects, in fact.
3. The amdlibComputeOiData program has now a fast (switch -Z0) and a fussy (switch -Z2) mode. The "fussy" mode is the same as before: inversion of the full covariance matrix previous to complex coherent flux extraction, iteration on piston fit until high precision is reached. The fast mode considers that the covariance matrix is diagonal and the piston is evaluated with less precision. For most cases (especially when the S/N is not bad), the fast mode is enough. And it is fast, especially since the matrix routines have been optimized also. The -Z1 switch is reserved for an intermediate, "clever" mode based on S/N, that I have not developed yet.
4. The full O3 optimization with inlining etc is set by default in the Makefile. Moreover, the really slow parts of code have been parallelized with OpenMp pragma. The speed of amdlibComputeOiData (which is the only program really time-consuming in amdlib) scales now roughly with the number of cores, and is now limited by the cfitsio reading and writing of files. With gcc and modern computers this should be transparent, but one has to test it on several platforms.
5. All these changes have been reported in the Yorick interface. To set the speed switch to fast, use amdlibSetPreference,"fastMode",0.

Package changes:

1. rlterm is not delivered anylonger with the amdlib packages. Please install rlwrap or rlterm on your system by yourself following the instructions present in the installation guide.
2. Mac OS Tiger is now very old so we do not build and distribute binary package anylonger.

Additionnally, this version closes two "tickets" we had:

1. Ticket 93: the VISDATA AND VISERR columns in OI_VIS (which are not OI-FITS compliant but ESO compliant) now store the Complex Coherent Fluxes & error (not the complex visibilities). One may use these values (on the RAW OI-FITS obviously) to add, combine and average the coherent fluxes as he/she likes.
2. Ticket 92: In case of selection averaging (amdlibPerformFrameSelection), the phase closure is retained as soon as two baselines are OK (in older versions (amdlib2 and amdlib3.0.1, etc) the phase closure would be deemed valid if the 3 baselines were retained by the selection criterium, giving really too few values for something which is essentially robust).

V3.0.3 (27-June-2011)

• corrected a regression introduced in previous version (3.0.2): flagged values were sometimes used when computing averaged closure phases, giving weird closures.
• 'catalogsDir' preference is now properly defined for the users who are using the binary packages
• appendOiData now properly tags appended 'OIDATA_RAW' files as OIDATA_RAW.
• amdlibSearchAllStarDiameters now use the UD in H-band. It uses the estimation from only one catalog, the one with the best confidence (in order: Merand, Borde, JSDC).
• slight changes in Piston estimation (should be more robust).
• amdlibCheckSpectralShift has been fixed and now works
• a memory leak in amdlibComputePiston2T (typo follwing the change above) has been corrected (10 May 2011).
• packaging script have been updated following cvs->svn migration
• yorick code version is checked against its C programs complement (a warning is displayed if versions differ)
• amdlib command get one new -e option to display one help message to configure emacs environment
• amdlib packages for beta test will be numbered as SVN_YYYYMMDD_REV. this should increase tracability and help developpers, beta testers or advanced users

V3.0.2 (18-march-2011)

1. Fix bugs on MacOs platforms related to keyword writing in amdlibAppendOiData.

V3.0.1 (14-march-2011)

1. Correct a bug in amdlibShowOiData when no selection file is found
2. Rewrote function amdlibAppendOiData: much faster; contrary to previous version, preserves the Spectrum table. To be tested. Also, updates important Header parameters: the list if files concatenated, the baselines start and end...
3. Corrected Bug in the frame selection on Piston OPD, method Percentage (OPD_PCG). The selection was in the wrong direction (i.e., not selecting the 10% smallest pistons but the 10% worst). Note that:
   1. Selection on THReshold was OK.
   2. Percentage selection in the case of OPD is kept for historical reasons but should be used with caution. Prefer the threshold selection (say, 15 microns for Low_Res)
4. Corrected Feature: The instrumental fringe contrast was at 1.0 when the P2VM was normalized to 1 (which is the default). Now reports the true instrumental fringe contrast.
5. Improvement: In case of a frame selection, Closure Phases are now computed for all frames for which one or the other baseline is selected, not for frames where all baselines are selected (which amounted to next to nothing in many cases).
6. Documentation: added hint that since v3.0 the closure phase sign is not the same as for previous versions of amdlib.
7. This version includes bug corrections on the calibration routines.
8. The yoco package development has been separated from the amdlib one and has its own distribution webpage: http://ipag.osug.fr/twiki/bin/view/Ipag/Distrib/YoCo ( Yoco is still distributed together with amdlib)
9. Yoco reverts to cfitsio V2.510 so it runs correctly on all supported platform.
10. Implement final data calibration routines with 2 main flavors, one from J.-B. Lebouquin and one from F. Millour. See documentation for details.

-- FlorentinMillour, GillesDuvert, JeanBaptisteLeBouquin

Amdlib version 3 Important note

It should be noted that the sign of the phase closure given by amdlib3 is not the same as with amdlib2. If you need to use together phase closure measurement coming from old data reduced with amdlib2 and new data reduced with amdlib3, it is recommended to re-reduce the old data with amdlib3, preferably to changing the sign of the amdlib2 values.

Do not hesitate to contact the user support of JMMC on this subject, we are interested to know about the validity of amdlib3's closure sign.

We apologize for not having listed this difference in the early documentation of amdlib3's new features.

-- GillesDuvert - 29 Jul 2010

V3.0b5

This is the first to be distributed to the public. No big changes on the core library side. I have changed the comments about the use of FLATS below, but "repetita placent": amdlib v3 is robust wrt. the use of flats. In any case, use of a Flat seem unnecessary to the author. We advise to use this really flat flat instead of ESO-provided ones. In doubt, test the data reduction using a real flat, but in this case, use only the flat obtained at MPIfR, attached at the end of this page. There are two flats, one for the old detector (chip id is "ESO-test" in the raw data files headers), http://www.jmmc.fr/twiki/pub/Jmmc/Software/AmberDrsReleaseNotes/AMBER_FFM_CHIP-247.fits.gz (chip id is "247" in the raw data files headers).

-- GillesDuvert - 01 Apr 2010

V3.0b4

Two options for amdlibComputeP2vm have been added, "-z" and "-a". Unfold usage for amdlibComputeP2vm in the V3.0 section below for explanation. Additionnally, if you use the "-a" option, which is handy at removing automatically wavelength shifts due to the repositioning error of the amber grating turret, be aware that the phase shifts (and thus the repositioning) seems sensitive to the quality of the Flat Field used. It may be prudent not to use any flat field at all with this option.

---

GillesDuvert - 01 Jul 2009

V3.0beta

The V3.0beta release consists in:

1. The C library, amdlib, which is autonomous since it provides command-line tools to reduce the data
2. The Yorick wrapper calling the amdlib.i yorick library, that provides added pipe-line processing and visualization tools.
   Starting from this version, users are requested, if they use this software for a scientific publication, to use the following acknowledgement:http://www.jmmc.fr/amberdrs/acknowledgement

The amdlib Library and its command-line tools

The usual command-line tools are :

• amdlibComputeP2vm: is the first thing to do. It computes a P2VM fits file wich contains all the calibration aspects. At the moment it is recommended to always use the options of amdlibComputeP2vm to modify manually the "spectral shifts" and the wavelength values.

• amdlibComputeOiData : reduce a science or calibrator observation and produce an OI-FITS file. An observation is one raw data file. There are typically 5 raw datat files on an object (science or calibrator) plus 1 dark and 1 sky in each Observing Block.

Show usage  Hide usage 

  Usage: amdlibComputeOiData [-h] [-f] [-s] [-v level] [-b <num>] [-p PHASOR] [-e STATISTIC|THEORIC] [-c FLUX|FRG|OPD] [-r <ratio>] [-B J|H|K] <badPixelMap> <flatFieldMap> <p2vm> <dark> <input> <output>

Options (new Options are in bold face):

• -h help
• -v verbosity level [0..4]
• -f force the use of another P2VM
• -s split OI-FITS; one file per band
• -b of frames to average (bin) over frames
• -p PHASOR (default) for piston algorithm
• -e THEORIC (default) or STATISTIC for error bars estimation
• -c FRG, FLUX or OPD criterion for frame selection
• -r for frame selection:
  if 0<ratio<=1, its a ratio, else it is a threshold.
  flux are in (e), OPD in microns.
• -B J, H or K to select the corresponding spectral band
• -J to tag Pistons with Jitter/Error gt. value (in microns)
• -X to tag visibilities with chi2 gt. value. This removes P2VM fits of complex coherent fluxes that did not behave well, that is, above some chi2. The default is -X0 , that is: tag data whose chi2 is above the mean chi2 by 3 times the chi2 rms. -X -1 or any negative value tages nothing. Normally AMBER data gives a mean chi2 between 0.9 and 1.8. Thus, -X 1.8 would select "normal" frames.
• -C to enhance piston precision using piston closure info. To be used with caution: The idea is to compute the piston on the baseliens with the smallest S/N as minus the sum of the two others. useful sometimes in Low Res.
• -P to retain photometries gt. value (in e). Default is 0, and default should not normally cause problems.
• -F to avoid using the continuum correction factor. This linearisation factor is part of the new way to reduce AMEBR data. Wizard only.
• -D to drop the first frames. Absolutely MANDATORY until the bias effects on the camera have been mastered. Drop at least the first 20 or 30 frames if possible. For FINITO-assisted data when the number of frames is too small, just pray .
• -n to avoid applying global bias estimate and correction. This should be used along with the -D option. If you drop frames, you do not need the global bias estimate.
• -A to avoid updating the badPixel info with bad pixels found in the dark file (Such On-The-Fly Bad Pixels are computed only if the number of frames is >500 anyway).
• -S to normalize star's spectrum by the spectrum of the P2VM (not exactly a transmission, the optical path being different: mostly useful in LowRes.). Wizard only.

The default command should be something like that:

amdlibComputeOiData -D30 -n -b 99999 -p PHASOR -e THEORIC -s <badPixelMap> <flatFieldMap> <p2vm> <dark> <input> <output>

Note that it is recommendend to avoid selection (option -b 9999) unless you are in Low Res, where selection on PISTON ( -c OPD -r 15 ) would be a good idea if you are interested in differential visibilities.

• amdlibAppendOiData : Adds several OI-FITS files together.

Show usage  Hide usage 

 Usage: amdlibAppendOiData [-h] [-v level] <inOiFitsFile1> <inOiFitsFile2> [... <inOiFitsFileN>] <outOiFitsFile>
  Options:
    -h help
    -v verbosity level [0..4]
 

• amdlibPerformFrameSelection : from various selection criteria, produces a file with an index of frames to be retained to compute a final OI-FITS.

Show usage  Hide usage 

Usage: amdlibPerformFrameSelection [-h] [-v level] [-c FLUX_THR|FLUX_PCG|FRG_THR|FRG_PCG|OPD_THR|OPD_PCG] [-r <ratio>] [-i <inSelFile>] [-o <outSelFile>] <inOiFitsFile> [<outOiFitsFile>]
  Options:    -h help
    -v verbosity level [0..4]
    -c <criterion> for frame selection. Possible values are:
        FLUX_THR - threshold on flux (electron count)
        FLUX_PCG - percentage of frames kept based on flux selection
        FRG_THR  - threshold on fringe SNR (number)
        FRG_PCG  - percentage of frames kept based on fringe SNR selection
        OPD_THR  - threshold on piston (microns)
        OPD_PCG  - percentage of frames kept based on piston selection
    -r <value> frame selection value
    -i <inSelFile> input selection file
    -o <outSelFile> output selection file
 

Changes in the library wrt. version 2

General issues

• This is a completely new release involving a rewriting of many parts of the amdlib C library.
• The code implements most of the algorithms published by Chelli et al. 2009 as well as workarounds of some of the problems audited by the ATF team (see ATF report).
• We have tried to maintain, at least this time, the interfaces of the amdlib C library 'almost' compatible with the previous ones, to avoid rewriting of the Yorick interface.However, a number of internal changes were necessary, as for example the fact that all computations are now in double precision. Another important change is that amdlib OI-FITS files have FLAGS and BLANKING VALUES . User programs reading these files should be aware that, depending on filters implicit or explicit in amdlibV3, some values may be flagged or blanked.

In Particular

• detector noise is computed per pixel from the dark file (darks thus preferably contain more than 500 frames for good statistics)
• on-the-fly bad pixel detection can be done
• since the dark in Amber is usually a hot dark, dark noise is taken into account AND the additional use of SKY files is FORBIDDEN
• However, one can use a SKY file in place of a DARK, which should be better in case of, e.g., sky glow.
• Starting June 2009, day cold darks are taken by ESO. These should be used preferentially to any other type of dark or sky.
• amdlib v3 is robust wrt. the use of flats. In any case, use of a Flat seem unnecessary to the author. We advise to use this really flat flat instead of ESO-provided ones. In doubt, test the data reduction using a real flat, but in this case, use only the flat obtained at MPIfR, attached at the end of this page. There are two flats, one for the old detector (chip id is "ESO-test" in the raw data files headers), http://www.jmmc.fr/twiki/pub/Jmmc/Software/AmberDrsReleaseNotes/AMBER_FFM_CHIP-247.fits.gz (chip id is "247" in the raw data files headers).
• additionnally, the software

1. implements the linearization factor of Chelli et al 2008. Information is displayed as:
   Average Linearization Factor: 0.953354, meaning that the data is good. if this factor is markedly different from 1.0, (say, below 0.8 or above 1.2 ), you could wonder if the P2VM is good enough or if the bad pixel map is up to date.
2. uses a refined algorithm to compute pistons, and an heuristic scheme to evaluate this piston 'goodness of fit'. This is mostly useful in Low Resolution witout FINITO where the atmosphere piston can cause a complete loss of coherence. The present version does not use the improved estimate of piston to correct from the theoretical extenuation. However, it has options ( -J and -C ) to tag bad piston values and use piston closure for piston evaluation improvement. Use the -c and -r options to filter out pistons larger than some fraction of the coherence length (say, 20 microns in LRK).
3. uses a goodness of fit test to tag individual visibilities which are not well fitted by the carrying waves of the interferogram. A message is associated, like: Correlated Flux fit statistics: mean = 1.015236, rms = 0.276510, median = 0.988546. The fact that this chi2 is very close to 1 means that all the variances have been well estimated. amdlibComputeOiData rejects by default all fits with a chi2 > 5 times the rms. Can be overriden with the -X option
4. drops all visibilities where one of the photometries is < 0. The option -P permits to raise this value above zero.
5. tags all bad values in the OI-FITS file using the FLAG columns (Tables OI_VIS, OI_VIS2 and OI_T3) and put BLANK values for the bad values in the AMBER_DATA table.
6. All of the above is known to permit to avoid Frame Selection in MR or HR mode. In LR mode w/o FINITO, a selection on piston seems necessary. All FINITO-aided modes are abolutely not guaranteed yet, due to the absence of FINITO residuals in the AMBER raw data files.
7. Bug fix for the stations indexes management

The Yorick Interface

• In construction.
• The GUI Browser gets one "remove" button
• Bug correction: put back wlen table to meters in OI fits file
• A suffix is added to amdlibGetProductDir
• Bug correction: spaces are properly handled into keywords !
• Bug correction: amdlibReadLog supports some empty log file
• Bug correction for hardcopy file of graphs
• Bug corrections: amdlibSetSpec, is now able to write spectra !
• amdlibP2VM.i, amdlibRawData.i: Add bandwidth to amdlibRawDataXXX functions and correct a long-lasting bug about P2VM and science data wavelength matching. Hopefully it will fix other side-effect bugs also

---

---

V2.1 -> V2.2

• Correction of bugs happening when appending OI-FITS files amd memory leaks in the C code part.
• Correction of bugs in the yorick scripts:
  • corrected amdlibDivideOiData crashes (due to zeroed differential visibilities)
  • corrected bugs relative to NaNs and zero values
  • corrected UV coordinates bugs
  • now avoid to write temporary files
  • amdlibAppendOiData and amdlibAppendAllOiData have been added to the list of amdlib functions
  • update of amdlibShowOiData behavior
• updated documentation

V2.0 -> V2.1

• This version (2.1) corrects a problem wrong estimate of spectral resolution elements invalidated by the spectral displacement between the photometric and interferometric beams. This caused a of loss of valid spectral channels during the P2VM computation, only noticeable in Low resolution JHK mode, when the spectral displacement was large, which, to our knowledge, occured only between September 2007 and January 2008. However, this bug correction may be beneficial for all observations, providing a few (1 to 2) more wavelengths available in the reduced datasets, in the long-wavelength end of LR and HR modes, and short-wavelength end in MR modes.
• zeroed values of VISAMP and VISAMPERR until computation of Differential Visibility Amplitude has been accepted.
• Fix yorick plugin compilation on Linux-x86_64

Beta 1 -> Beta 2

• Set default frame selection criterion to 'fringe SNR'
• Change message when selecting directory to perform all frame selection
• Update for gcc 4.x
• Fix bug in amdlibPerformFrameSelection (yorick plugin only) which invoked flux selection when calling snr selection.
• Fix bug in amdlibLoadSelection which crashed, now called amdlibLoadFrameSelection
• Implement amdlibSaveFrameSelection to create your own selection file
• Automatic check of the log integrity when available and recompute the missing parts if necessary
• Change as much as possible parameter names in yorick-plugin according to document of G.Zins (second proposition) (jblebou).
• Remove as much as possible 'verbose=' parameter in yorick-plugin (jblebou)
• Add PRO CATG keyword in frame selection file header
• Add procedures to handle logged messages ; i.e. message formatting, verbosity control, ...
• Implement these Logging procedure in amdlibOiData.i, amdlibP2vm.i, amdlibDataQuality.i, amdlibGraphics.i (jblebou)
• Fix bug in amdlibShowData when OI file contains 'NaN' values
• Change format of OI fits files in the AMBER_DATA table for ESO compliance. amdlib-2.0-Beta-1-generated OI files should normally still work but it is highly recommended to recompute them.
• Corrected a graphical bug in amdlibShowP2vm about Vk plots : now plotted by telescopes and not by baseline.
• Change parameter name 'skyFileName=' into 'inputSkyFile=', to be consistent with new parameter names.
• Fix bug in amdlibAppendOiData

Known bugs

• amdlibLoadSelection does not work (fixed in Beta 2)
• amdlibPerformFrameSelection (amdlib library) seems to not select frames for J-band raw OI files
• amdlibComputeOiData generates 'NaN' values when data quality is very low

• VraimentFlat.fits.gz: A really Flat Flat!

• AMBER_FFM_CHIP-ESO-test.fits.gz: The Original Flat Field Map of the Old AMBER Deector (ESO CHIP ID "ESO-test")

• AMBER_FFM_CHIP-247.fits.gz: The Flat Field Map of the New AMBER Deector (ESO CHIP ID "247")

• Set ALLOWTOPICCHANGE = JmmcGroup