Understanding AstrOmatic metadata files

Posted by Emmanuel in MissFITS, SCAMP, SExtractor, SWarp, using software, WeightWatcher on October 5, 2009 – 0:38
Post a comment
report Several AstrOmatic software packages (MissFITS, PSFEx, SCAMP, SExtractor, STIFF, SWarp and WeightWatcher) offer the possibility to generate metadata in XML-VOTable format at the end of an execution run. For this to happen, the configuration parameter WRITE_XML must be set to Y. The XML-VOTable format was chosen  to facilitate integration in Virtual Observatory services, although it is somewhat cumbersome to use.

Content of a metadata file

A metadata file consists essentially of a VOTable RESOURCE named “MetaData” containing several sections:

  • a set of PARAM elements describing the execution run (date/time, user, path, etc.),
  • a series of TABLEs containing various tool-dependent results, such as measurements, calibration results, statistics, …
  • a “WarningsTABLE listing all warnings issued during processing,with date and time of occurence,
  • a “Config” sub-RESOURCE with all the configuration parameters and their values for the run, as well as the full command-line.
Here is an example of a metadata VOTable produced by MissFITS:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="file:///usr/share/missfits/missfits.xsl"?>
<VOTABLE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/v1.1">
<DESCRIPTION>produced by MissFITS</DESCRIPTION>
<!– VOTable description at http://www.ivoa.net/Documents/latest/VOT.html –>
<RESOURCE ID="MissFITS" name="MissFITS">
 <DESCRIPTION>Data related to MissFITS</DESCRIPTION>
 <INFO name="QUERY_STATUS" value="OK" />
 <RESOURCE ID="MetaData" name="MetaData">
  <DESCRIPTION>MissFITS meta-data</DESCRIPTION>
  <INFO name="QUERY_STATUS" value="OK" />
  <PARAM name="Software" datatype="char" arraysize="*" ucd="meta.title;meta.software" value="MissFITS"/>
  <PARAM name="Version" datatype="char" arraysize="*" ucd="meta.version;meta.software" value="2.2.11"/>
  <PARAM name="Soft_URL" datatype="char" arraysize="*" ucd="meta.ref.url;meta.software" value="http://astromatic.iap.fr/software/missfits"/>
  <PARAM name="Soft_Auth" datatype="char" arraysize="*" ucd="meta.bib.author;meta.software" value="Chiara Marmo,Emmanuel Bertin"/>
  <PARAM name="Soft_Ref" datatype="char" arraysize="*" ucd="meta.bib.bibcode;meta.software" value="2006ASPC..351..112B"/>
  <PARAM name="NThreads" datatype="int" ucd="meta.number;meta.software" value="1"/>
  <PARAM name="Date" datatype="char" arraysize="*" ucd="time.event.end;meta.software" value="2009-09-25"/>
  <PARAM name="Time" datatype="char" arraysize="*" ucd="time.event.end;meta.software" value="14:52:19"/>
  <PARAM name="Duration" datatype="float" ucd="time.event;meta.software" value="1" unit="s"/>
  <PARAM name="User" datatype="char" arraysize="*" ucd="meta.curation" value="bertin"/>
  <PARAM name="Host" datatype="char" arraysize="*" ucd="meta.curation" value="kiravix.iap.fr"/>
  <PARAM name="Path" datatype="char" arraysize="*" ucd="meta.dataset" value="/disk2/marquett"/>
  <TABLE ID="OutFields" name="OutFields">
   <DESCRIPTION>Metadata about the MissFITS</DESCRIPTION>  output images
   <PARAM name="NImages" datatype="int" ucd="meta.number;meta.dataset" value="1"/>
   <FIELD name="Output_Image_Name" datatype="char" ucd="meta.id;meta.file;meta.fits"/>
   <FIELD name="Input_Image_Type" datatype="char" ucd="meta;meta.code.class"/>
   <FIELD name="Input_Extensions" datatype="int" ucd="meta.number;meta.dataset"/>
   <FIELD name="Input_Naxis3" datatype="int" ucd="meta.number;meta.dataset"/>
   <FIELD name="Output_Image_Type" datatype="char" ucd="meta;meta.code.class"/>
   <FIELD name="Output_Extensions" datatype="int" ucd="meta.number;meta.dataset"/>
   <FIELD name="Output_Naxis3" datatype="int" ucd="meta.number;meta.dataset"/>
   <FIELD name="HeadFlag" datatype="boolean" ucd="meta.code;obs.param"/>
   <FIELD name="OBJECT" datatype="char" ucd="meta;meta.note"/>
   <DATA><TABLEDATA>
    <TR>
     <TD>temp.miss.fits</TD><TD>SPLIT</TD><TD>1</TD><TD>1</TD>
     <TD>MULTI</TD><TD>8</TD><TD>1</TD><TD>F</TD>
     <TD>centre gal</TD>
    </TR>
   </TABLEDATA></DATA>
  </TABLE>
  <TABLE ID="Warnings" name="Warnings">
   <DESCRIPTION>MissFITS warnings (limited to the last 1000)</DESCRIPTION>
   <FIELD name="Date" datatype="char" arraysize="*" ucd="meta;time.event.end"/>
   <FIELD name="Time" datatype="char" arraysize="*" ucd="meta;time.event.end"/>
   <FIELD name="Msg" datatype="char" arraysize="*" ucd="meta"/>
   <DATA><TABLEDATA>
    <TR><TD>2009-09-25</TD><TD>14:52:18</TD><TD>default.missfits not found, using internal defaults</TD></TR>
   </TABLEDATA></DATA>
  </TABLE>
  <RESOURCE ID="Config" name="Config">
   <DESCRIPTION>MissFITS configuration</DESCRIPTION>
   <PARAM name="Command_Line" datatype="char" arraysize="*" ucd="obs.param" value="missfits -OUTFILE_TYPE MULTI -SAVE_TYPE NEW temp"/>
   <PARAM name="Prefs_Name" datatype="char" arraysize="*" ucd="obs.param;meta.file" value="default.missfits"/>
   <PARAM name="Remove_Keyword" datatype="char" arraysize="*" ucd="  " value=""/>
   <PARAM name="Replace_Keyword" datatype="char" arraysize="*" ucd="  " value=""/>
   <PARAM name="Slice_Keyword" datatype="char" arraysize="*" ucd="  " value=""/>
   <PARAM name="SliceKey_Format" datatype="char" arraysize="*" ucd="  " value="%02d"/>
   <PARAM name="Display_Keyword" datatype="char" arraysize="*" ucd="  " value="OBJECT"/>
   <PARAM name="Header_Suffix" datatype="char" arraysize="*" ucd="  " value=".head"/>
   <PARAM name="NExtensions_Min" datatype="int" ucd="  " value="0"/>
   <PARAM name="OutFile_Type" datatype="char" arraysize="*" ucd="  " value="MULTI"/>
   <PARAM name="Split_Suffix" datatype="char" arraysize="*" ucd="  " value=".%02d.fits"/>
   <PARAM name="Slice_Suffix" datatype="char" arraysize="*" ucd="  " value=".%02d.fits"/>
   <PARAM name="Process_Type" datatype="char" arraysize="*" ucd="meta.id;meta.code" value="NONE"/>
   <PARAM name="Checksum_Type" datatype="char" arraysize="*" ucd="meta.id;meta.code" value="NONE"/>
   <PARAM name="Save_Type" datatype="char" arraysize="*" ucd="meta.id;meta.code" value="NEW"/>
   <PARAM name="New_Suffix" datatype="char" arraysize="*" ucd="meta.id;meta.code" value=".miss"/>
   <PARAM name="Verbose_Type" datatype="char" arraysize="*" ucd="meta.code" value="NORMAL"/>
   <PARAM name="Write_XML" datatype="boolean" ucd="meta.code" value="T"/>
   <PARAM name="NThreads" datatype="int" ucd="meta.number;meta.software" value="1"/>
  </RESOURCE>
 </RESOURCE>
</RESOURCE>

Note that the medata file is written even if an error occurs, in which case a PARAM element named “Error_Msg” is added in the first section. In the case of SExtractor, the metadata is included in the output catalogue, together with the full list of extracted source measurements,  if CATALOG_TYPE is set to ASCII_VOTABLE.

Accessing the metadata file content

The content of a metadata file can be parsed and read by VOTable-compliant VO tools and libraries, such as TOPCAT or the libVOTable. But as any other XML document, it can also be “translated” into another (more user-friendly) format using an XSLT stylesheet. The URL of an XSLT stylesheet must be specified at the beginning of XML file using the ?xml-stylesheet start-tag.

Every AstrOmatic software package that writes metadata VOTables comes with its own XSLT stylesheet, installed by default in /usr/share/<software_name>/ or /usr/local/share/<software_name>/ (the URL in the example above is file:///usr/share/missfits/missfits.xsl, it can be changed using the XSL_URL software configuration parameter, or obviously by hand directly in the metadata file). This XSLT stylesheet contains all the information needed to convert the metadata VOTable to plain old, eye-friendly XHTML. The result can be visualised by pointing your favorite web browser to the metadata file, either locally (e.g. in the example above file://missfits.xml) or remotely (e.g. http://astromatic.net/missfits.xml). Tables are expandable and rows are sortable by clicking on the header cells, thanks to a small JavaScript code (see below).

scamp_small An XML-VOTable metadata file generated with SCAMP and visualised in the Firefox web browser. Click on the image to view a real example.

Be warned that since its version 3, the Firefox browser blocks by default access to XSLT stylesheets not located in the same directory as the XML file (among other things). A consequence on Linux is that the shell command firefox scamp.xml will return only an unformatted page with “flat” text, by default. This behaviour may be overridden in Firefox by changing the flag security.fileuri.strict_origin_policy to false in the about:config preference settings, as shown below.

about-config


References

Bertin, E. & Tissier, G. 2007: VOTables in TERAPIX Software, ASP Conference Series, Vol. 376, 2007, Richard A. Shaw, Frank Hill and David J. Bell., eds, p.507 [PDF] [BibTeX entry].


This entry was written by Emmanuel, filed under MissFITS, SCAMP, SExtractor, SWarp, using software, WeightWatcher.
Bookmark the permalink or follow any comments here with the RSS feed for this post.
Post a comment or leave a trackback: Trackback URL.

Post a Comment

You must be logged in to post a comment.