Tags:
create new tag
view all tags

OITools Roadmap

Updated: 08/02/2018

In progress:

  • Project documentation
  • OIFitsValidator enhancements
    • Severity profil
    • Filter
    • Message Viewer

In Study:

  • New features for OITOOLS 2
  • Nom.tam.fits update

Done:

  • Personal presentation
  • New features for OITOOLS 2 : Units
  • Github

Personal presentation

Prepare the presentation of my work on OITools and the explanations of my involvement. A link to this presentation will be available after this one (9/11)

Project documentation

  • Complete the changelog & documentation wiki pages
  • API Usage Tutorial ?
  • Javadoc
  • Complete / create new schema ?

New features for OITOOLS 2

Units: [DONE]

Description:

  • Units handling: In OIFITS V2, the physical units for few columns are 'user-defined' i.e. given in the column description (TUnit keyword). OITools already parse column keywords but does not store such units in the column meta data
  • Impacted columns are:
    • OI_VIS table: RVIS (RVIS_ERR), IVIS (IVIS_ERR) and VISDATA (VISERR)
    • OI_FLUX table: FLUXDATA

Study:

  • Currently units are implemented by an Enumeration that is not extendable !
  • Possible choices:
    • Realize an enum units more permissive, in this case it will be necessary to fix the validation of the units (OIFitsLoader.parseColumn ()), choose where are store these 'special' units because this one will serve again. They are reused in the data display under OIFitsExplorer as well as retrieve them for writing files with WriteOIFits.
    • or proceed to a rewrite of the enumeration units in class which would allow us to be directly more permissive and to directly manage the case of these units which can vary. They will directly store and manage here.

Impacts:

  • unit parsing & validation : OIFitsLoader
  • OIFitsExplorer : Display of column units
  • writing or rewriting files

Feedback:

  • The choice that has been made is to change the operation of Units (creating subclass) and to treat the case of special units in the loading of the file and the validation. The units are recover directly from OIFits files.
  • FitUnit reporting units for images, Merge Units and FitsUnit ?

Checksum/Datasum

TODO: description & study

Management of dimensions (NWaves)

TODO: description & study

Extra table/column

OI_INSPOL rules

OIFitsValidator enhancements: [IN PROGRESS]

Description:

  1. An error message more provided of information and more explicit. As well as a clearer display and more exploitable for users (filter).
  2. Need to determine severity with profiles
  3. Setting up a filter, which would allow us not to display redundant or not important messages (and to group them together)
  4. The ability to re-use error messages (xml)
  5. Update test to compare the validation results.

Study:

  • JAVA object creation for the message: Rules, Origine, Severity. These objects contain all the information useful for the construction of the message.
    • Creation of an error object, containing all the necessary information for the different uses of the error display. (GUI, XML, ..)

  • M.V.C. idea:
    • Experiment: nosql appreach (Object/SQL) => granule.
    • DataSelection => visual. OIXP (colors / select).
    • JTable
      • Process validation messages as a FITS table: with keyword and column.
    • Export: CSV, votable, FITS export.
    • Import CSV => plot custom
    • TOMCAT (integral)

  • FileRef and HDURef: In order to retrieve informations (fileName, extName, extNb) between our classes to be able to use them in the construction of our error. The idea was to create objects listing these informations FileRef and HDURef.
    • FileRef: FileRef is an object that retrieves informations from the file: fileName and absoluteFilePath. These information was originally contained in FitsImageFiles, which now uses FileRef to retrieve these informations.
    • HDURef: HDURef is an object that retrieves informations from the file: extName and extNb. These informations was originally contained in FitsHDU, which now uses HDURef to retrieve these informations.
  • The creation of these two objects was intended to link all these informations and make them available throughout the structure. Indeed, the idea was that HDURef itself calls FileRef and FitsHDU and all classes derived from it have this access.
  • The main problem is that FitsHDU could not directly reference FitsImageFile because it would have been too heavy (in the reuse of objects created later using FitsHDU).
  • After consultation it turned out that the object HDURef would be too restrictive to set up (work and call in the constructor of all chrild classes) and reference to the FileRef complicated to set up (recovery from the loader). This object has been abandoned for the moment.
  • In the end for the link in the retrieval of all the information it was chosen to make a reference to FileRef at the time of checking the file. FileRef of the current file is retrieved and registered in the validation of the OIFits structure, it is here that the error will be built.

  • Format for the RuleFailure: Request for discussion when choosing the format of the error.
    • ruleFailed(RuleFail, DataLocation) : RuleFail is somehow the key. This is the part that will be checked and compared in order to avoid duplication in creating the error. DataLocation is the part of datas, it is the information that may be redundant depending on the case and repeating itself several times.
    • ruleFailed(Rule, FileRef, extName, extNb, member).addValueAt(datas). Datas contains here value, row, and colunm.
      • In this case, the most would be not to reduce for nothing the messages which have very different and variable values. (where there is almost never a duplicate)
    • ruleFailed(Rule, FileRef, extName, extNb, member, value).addValueAt(datas). Datas contains here row, and colunm. And value is part of the key.
      • This case is very interesting for errors that contain many duplicate values (example the name of a star)
    • There is also the possibility of using one or the other depending on the case!
  • It was also important to define all the terms of the message and avoid the constants:
    • Each term given in parameter allows us to refine the display but also to define the severity!

  • Format RuleFailed:
    • ruleFailed(rule); (7)
    • ruleFailed(rule, HDU); (5, contains 1 with datas)
    • ruleFailed(rule, HDU, member); (78, contains 67 with datas)
    • ruleFailed(rule, extName, extNb) (1)
      • Special case: no hdu information but extname and extNB possible

  • Format DataLocation: (68)
    • addFixedValue(Value) (4)
      • It's more an informative error format
    • addKeywordValue(Value) (15)
    • addKeywordValue(Value, Expected) (13)
    • addValueAt(Value, Row) (15)
    • addValueAt(Value, Expected, Row) (2)
    • addValuesAt(Value, Limit, Row) (5)
    • addValueAtRows(Value, Row1, Row2) (4)
    • addColValueAt(Value, Row, Col) (7)
    • addColValueAt(Value, Expected, Row, Col) (1)
    • addColValueAt(Value, Row, Col1, Col2) (1)
    • addColValueAt(Value, Row, Col, Detail) (1)

  • Format Informative:
    • Severity => INFO

  • Inventory of the list of messages (String)
    • An inventory of messages (String) has been developed to bring together similar ones and to try to build a consortium.
    • The message inventory allowed the development of the format for RuleFailed and Datalocation

Impacts:

  • OIFitsChecker
    • in creating error messages and validation
    • to define error profiles
  • getCheckReport() is used in OITools, Aspro2, LITpro, OImaging, OIFitsExplorer for the error message display
  • OIFitsValidatorTest test save & compare validation results
  • OIVal & OIDB to the re-use of xml error messages

Feedback:

1: Format messages (DONE)

  • checker.sever delete and create new message: pointer to file, row/column, table, value, severity, declare the rule bound... (error list with full info)
  • Creation of an error object with JAVA object: Rules, Origine (HDU, member), Severity and datas (row/colunm, values)
  • ERROR (Rule, FileRef, extName, extNb, member, datas)
  • Ongoing discussion about the format of the error and its contents
    • Reflection on the type of object to use for storing the error. Management of duplications.
    • Reflection on all future uses of this message in order to really have all the information that will be useful.
    • At the same time thinking about methods and message retrieval rules. getErrors(filtre, tri)
    • Datas would be a storage object for values, rows, and columns when this information is available. First, define an inventory of messages (String) to gather those who are alike. And define the most convenient storage object to use using these cases.
  • The question that also arises is the management of the message (String) itself. Usefulness of this message and if necessary the creation of this one, is questioned.
    • An inventory of messages (String) has been developed to bring together similar ones and to try to build a consortium.

    • Check:
      • checker.severe("Invalid value for column '" + this.getName() + "' line " + rowNb + ", found '" + values[r] + "' should be >= 0"); [position 472:33]
      • V2:
      • rule.template = "Invalid value for column '{{MEMBER}}' line {{ROW}}, found '{{VALUE}}' should be >= 0" (constante)

      • toString() {
        • str = rule.template;

        • str = str.replaceAll("{{MEMBER}}", member);
        • str = str.replaceAll("{{ROW}}", row);
        • str = str.replaceAll("{{VALUE}}", value);

      • return str;
      • }

Feedback: Format message DONE.

    • Creating an error message for each failure or each data (if present)
    • A pattern for the messages has been created for each Rule (Use {{}} for values can be changes)
    • Use replaceAll(), to clean all text contained in '{{}}' by the true values of failures

2: Define error profiles

Ideas:

  • The profiles are lenient, strict, or software.
  • The severity (Off, Info, Warning, Error) should be defined using profiles. This profiles can also be modified at the request of the user.
  • Change in severity depending on the use case:
    • example: If we are on a file whose instrument is AMBER, then the severity changes from warning to error.
    • It would be wise to have a user feedback on this, to have their needs.
  • The user could also define his choice of severity, but this would be more than one filtering function (display, GUI).
  • User could also create his own profile (his own config)
  • The idea would be the creation of a config file in which is defined if a violation of this or that rule is severe or not.

Proposal: Define the logical rules how to correctly set the severity of any failure:

  • Questions:
    • Is the keyword EXTVER MANDATORY ? (Warning)
    • How to fill missing columns ? use the first accepted value (FWHM for FOV_TYPE, CAL for CATEGORY) or specify which is the default one (not always the first one !)
    • Software ? how to adapt severity depending on the software ? use a final profile per software ?
    • OFF or INFO ? Laurent prefers INFO to show it as informative but can be hidden if show only ERROR / WARNING (display filter)

  • Profile configuration:
    • We propose to define by default the severity to ERROR.
    • A severity profile is a configuration file that is loaded and analyzed to define the severity of the failures.
    • Possibily use multiple configuration files (1 per OIFITS standard, software ...) to refine iteratively the severity of failures (multi-pass)
    • File format (ASCII, CSV, XML, ...) ?
    • TODO: which fields will be present in this configuration file ?

  • JMMC Profile (Gilles):
    • Case "RULE_ID | SEVERITY":
      • (DEFAULT) * : ERROR
      • COL_UNKNOWN : OFF
      • GENERIC_DATE_OBS_STANDARD : INFO
      • GENERIC_DATE_OBS_RANGE : WARNING
      • OI_ARRAY_XYZ : WARNING
      • OI_INSPOL_MJD_RANGE : WARNING
      • OI_TARGET_COORD : WARNING (ideally if software = 'OIDB, should be SEVERE ?? OH No !!)

    • Case "RULE_ID | EXT_NAME | SEVERITY":
      • Ignore any rule applied to OI_CORR table:
        • CORRNAME_* (CORRNAME_REF, CORRNAME_UNIQ)
        • OI_CORR_* (OI_CORR_CORRNAME, OI_CORR_IINDEX_MIN, OI_CORR_JINDEX_SUP, OI_CORR_IJINDEX_MAX)
        • GENERIC_COL_*, GENERIC_KEYWORD_*
        • Solution: New Case "RULE_ID = * | EXT_NAME = OI_CORR | SEVERITY = OFF"
      • OI_TARGET cases:
        • "RULE_ID = GENERIC_COL_VAL_ACCEPTED_STR | EXT_NAME = OI_TARGET | SEVERITY = WARNING"
        • "RULE_ID = GENERIC_COL_VAL_ACCEPTED_STR | EXT_NAME = OI_TARGET | MEMBER = CATEGORY | SEVERITY = SEVERE"

    • Case "RULE_ID | EXT_NAME | MEMBER | SEVERITY":
      • Ignore missing columns OI_ARRAY.FOV* (FOV et FOVTYPE):
        • New Case "RULE_ID = COL_MANDATORY | EXT_NAME = OI_ARRAY | MEMBER = FOV* | SEVERITY = WARNING"
      • GENERIC_COL_UNIT / GENERIC_COL_UNIT_EXIST:
        • "RULE_ID = GENERIC_COL_UNIT | SEVERITY = INFO"
        • "RULE_ID = GENERIC_COL_UNIT | EXT_NAME = OI_VIS | MEMBER = VISAMP* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT | EXT_NAME = OI_VIS | MEMBER = RVIS* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT | EXT_NAME = OI_VIS | MEMBER = IVIS* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT | EXT_NAME = OI_FLUX | MEMBER = FLUX* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT_EXIST | SEVERITY = INFO"
        • "RULE_ID = GENERIC_COL_UNIT_EXIST | EXT_NAME = OI_VIS | MEMBER = VISAMP* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT_EXIST | EXT_NAME = OI_VIS | MEMBER = RVIS* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT_EXIST | EXT_NAME = OI_VIS | MEMBER = IVIS* | SEVERITY = ERROR"
        • "RULE_ID = GENERIC_COL_UNIT_EXIST | EXT_NAME = OI_FLUX | MEMBER = FLUX* | SEVERITY = ERROR"

    • Case "RULE_ID | MEMBER | STANDARD | SEVERITY":
      • Need: Ignore missing OI_REV in OIFITS 1
      • New Case "RULE_ID = GENERIC_KEYWORD_MANDATORY | MEMBER = OI_REV | STANDARD = 1 | SEVERITY = WARNING"
      • TODO: give the OIFITS standard into the SeverityProfile.defineSeverity() method (missing info)
      • TODO: flagging cases with their corresponding STANDARD = 1 / 2 / * may be interesting to skip useless cases

  • Actions: Rules to review
    • GENERIC_DATE_OBS_STANDARD:
      • Problem: too many failures for GRAVITY: "The keyword DATE-OBS '2016-06-23T03:10:23' is not in a standard format 'YYYY-MM-DD' "
      • Solution: check that DATE-OBS starts with a valid 'YYYY-MM-DD' date: DONE

    • GENERIC_COL_UNIT / GENERIC_COL_UNIT_EXIST:
      • TODO: fix inspectRules for this rule, Rules.xml reports all table columns !
      • RVIS, IVIS, RVISERR, IVISERR, FLUXDATA, FLUXERR columns have custom units (any allowed value by the user): TODO test validation
      • Special case AMPTYP <> "correlated flux": VISAMP and VISAMPERR columns should not have units

    • GENERIC_COL_VAL_ACCEPTED_STR:
      • if the column is missing in the file, then OITools creates anyway an empty column (with default values).
      • For String values: the default is '', then the GENERIC_COL_VAL_ACCEPTED_STR rule fails for all column rows !
      • Examples: OI_ARRAY.FOV_TYPE and OI_TARGET.CATEGORY
      • Proposals:
        • Solution 1: skip the GENERIC_COL_VAL_ACCEPTED_STR rule for missing columns. However, it will only works ONCE at loading phase, any future validation will fail as the table column is still empty !
        • Solution 2: fill the empty column with the first accepted value (FWHM for FOV_TYPE, CAL for CATEGORY); maybe reorder the array of accepted values or specify which is the default one (not always the first one !)

3: Report enhancements:

  • collect, gather & filter all validation failures
  • generate an "User" report

4: Xml serialization (DONE)

  • Update the xml Output to incorporate messages as xml elements to handle & filter them after in the OIVal & OIDB tools
  • Fix OIVal / OIDB message parsing

FeedBack:

    • Creation of an Xml output with display of all possible failures.
    • Display and processing of these outputs by OIVal and OIDB

5: Message Viewer

  • graphical panel displaying the messages (sort, filter, severity) = new Widget (GUI) (litpro/oixp)
  • User view ideas:
    • (Conteneur)
    • File: blabla.oifits
      • On HDU:
        • On member:
          • Errors list...
          • Errors list...
          • Errors list...

    • Or rule display
    • Rule 1:
      • Fail on File blabla.oifits, on HDU, on member at row ...
      • Fail on File blabla.oifits, on HDU, on member at row ...
    • Rule 2:
      • Fail on File blabla.oifits, on HDU, on member at row ...

  • The ideal would be to be able to go from one to the other.

Nom.tam.fits update

TODO: description & study

  • The version of the library that we use at have evolved
    • retrieve and compare versions of GitHub (the one we use and the last one)
    • feasibility study: see at which level the changes made will impact our code.
    • delete our library and add the latest version

GitHub

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt ErrorsClasser.txt r1 manage 15.5 K 2017-12-01 - 16:25 CharleenKemps Inventory of rules and progress
Edit | Attach | Watch | Print version | History: r32 < r31 < r30 < r29 < r28 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r32 - 2018-02-08 - CharleenKemps
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback