In the summer of 1998 The ASCA Data Processing Team began working on the code. At first the goal was to make the code y2k compliant, but to otherwise leave the code unchanged. However, y2k compliance lead to replacing the out-of-date version of FITSIO which accompanied the code, which lead to re-writing the output buffering system in C, which lead to finding several bugs in the code, which lead to finding even more bugs in the code, which lead to re-writing a substantial fraction of the code. This document describes these changes in detail.
By looking at the code at this late stage of the mission, we have several advantages over the original authors. In particular the input and output file formats are now fixed, we have a large archive of telemetry for testing, and we now have a better understanding of how ASCA and its software are used. Therefore, although this document describes many problems with the code, there is no insult to the hard work the original authors put into it.
Even though there was a large number of bugs in the code, these bugs did not generally have a large effect on commonly used data. This explains why the bugs went unnoticed for so long. It is unlikely that any scientific results will be invalidated by the problems described here.
We wish to thank Tadayasu Dotani of the SIS team and Yoshitaka Ishisaki of the GIS team, as well as the members of the ASCA GOF for much helpful advice and explanations.
We have eliminated Frfread's ability to extract products such as spectra, light curves, and images. This is better done with the xselect and extractor FTOOLS. If the user wishes to have a quick look at the data, they can use the data products extracted in the US processing, since these files are now available to both US and Japanese observers. Frfread now only produces event files.
We have also eliminated the frfread GUI. This more complicated interface was unnecessary after eliminating the quick look functions. It may have also developed some machine dependent installation problems. There have been several examples of people not being able to install the GUI version of frfread on various platforms.
We have dropped several options in the code including the following:
Frfread 4.0 is y2k compliant. The DATE-OBS, DATE-END and DATE keywords are now all written in "yyyy-mm-dd" format. The routine for calculating the DATE-OBS and DATE-END keywords from the TSTART and TSTOP values has been verified to work correctly through the turn of the century and well past the reasonable end of the mission. Note that the routine does not take leap seconds into account, so DATE-OBS and DATE-END are only accurate to within a few seconds. TSTART and TSTOP should be used if greater accuracy is desired. The ORBITBEG and ORBITEND keywords which were copied from the ORBIT0 and ORBIT1 keywords of the input file have been dropped. They were almost completely unused and had the non-compliant format "yymmddnn", where "nn" was the orbit number.
Frfread 4.0 is more verbose. It reports all telemetry gaps and corrupted super frames and lists all dropped events.
Modal keyword values can be incorrect if a value changes while it is not relevant to the current instrument or mode. This can produce spurious SIS event files with Sn_DESTA=1 when the other instrument is in frame mode. Also GIS event files can have incorrect values of CPUn=STOP associated with periods of memory check mode. Taken at face value, these erroneous keyword values would indicate a serious problem with the data, but in practice they are usually ignored.
The old frfread made a number of checks to determine if the data in a given super frame was corrupted. Frfread 4.0 retains these checks and adds a check in low and medium bit rate for whether SIS0 and SIS1 telemetry alternate correctly. This catches corrupted instrument IDs and continuation (inheritance) flags as well as the rare cases where the bit rate indicator has been corrupted to read low or medium bit rate. Such super frames are rare, but they can produce erroneous results.
Both old and new versions of frfread check the time tags of super frames to see if they are an even multiple of 2 seconds apart. Super frames failing this test are skipped. The tolerance used for this test (to account for round off error) was too small. For a few files after ascatime 1e8 (March 1996) ~10% of the telemetry was rejected unnecessarily. This has been fixed in frfread 4.0.
Frfread did not previously write HK saturation flags, Sn_SATFm, in fast mode. However this is of little consequence, since the saturation threshold for FAST mode is very high and mkfilter2 recalculates the saturation flags in order to compensate for an earlier bug (since fixed). Nevertheless, frfread 4.0 writes saturation flags in all modes.
The method for calculating the saturation flags has also changed. Previously they were calculated by comparing the number of telemetered events for one CCD exposure to the maximum number of events which can normally fit in the telemetry. This has been changed to a method where the saturation flag is set "on" at the beginning of a chip exposure, and then turned "off" if a null event is encountered in the telemetry. This method is more robust and takes into account situations where one SIS sensor is not functioning and the other uses its space in the telemetry, thereby doubling the telemetry capacity. The version of mkfilter2 to be released with FTOOLS 4.2 has been modified to read the saturation flags from the HK files to take advantage of this.
The HK values Sn_HOVRm, Sn_VOVRm, and SnLSLNm (horizontal and vertical over-clocked pixel values and last line number) were sometimes assigned to the wrong chip in the old frfread. These values are synchronized with the telemetry super frames, not the CCD chip exposures. Their instrument assignments can be determined by their location in the telemetry frame (SIS0 and SIS1 values are output in parallel), but the chip assignment must be derived by looking at the current chip in the telemetry. In high bit rate, these HK values are over-sampled, such that the same value will appear in adjacent SIS0 and SIS1 super frames. The old frfread read the HK value for each instrument every time it occurred in the telemetry. When it read an SIS0/1 value in an SIS1/0 frame it would look ahead in the telemetry to find the super frame with the correct instrument, and read the chip ID from that super frame. However, this gave erroneous results when super frames were missing or corrupted. Frfread 4.0 ignores these HK values when they occur in super frames with the incorrect instrument ID. This ensures correct chip assignment without loss of information.
Since many SIS HK values are redundant, there were times when a given value could be repeated with the same value and time. Frfread 4.0 only reads one occurrence of these redundant values, so this can no longer occur.
The HK value Sn_CDBS1 ("bias adjustment on analog board for CCD1") was written with the name "Sn_CDBS0". This has been corrected.
The SIS RBM-related values Sn_RBM_A, Sn_RBMFL, Sn_RBM_F all reported an entire 8-bit telemetry word, even though the value of interest was only contained in some of those bits. Other bits in the word contained unrelated GIS quantities. This has been fixed.
Similarly, the NSASXDAT and NSASYDAT values, contained four extraneous bits at the least significant end. These are copies of values from the common HK file which mkfilter2 reads to calculate "NSAS". Mkfilter2 contains a work-around for this bug which divides values greater than 2^12 by 16. The corrected values from frfread 4.0 will always be treated correctly by mkfilter2, but the old, incorrect values could give incorrect results for low values of NSASXDAT or NSASYDAT. NSAS is not used in normal data reduction.
The Sn_DRDLm HK value (dark level update lower discriminator level) was interpreted as an unsigned quantity, instead of the signed quantity, "-40". Frfread 4.0 fixes this by interpreting all 16 bit SIS HK quantities as signed integers. This value has been constant throughout the mission.
The old frfread had a complicated mechanism for inserting gaps in the GTIs to mark telemetry gaps and dummy frames. Frfread 4.0 keeps track of the individual chip exposure times and inserts GTI breaks when adjacent exposures are separated by more than 4 seconds. This new method is more robust and may fix bugs which were not specifically located. We have also fixed the following specific GTI bugs:
Frfread 4.0 does not open event files during a string of dummy frames. This eliminates the event files with no GTIs which the old frfread sometimes produced.
A 1/8 second processing time lag between CCD exposure and telemetry was ignored in the old frfread, but is taken into account in Frfread 4.0. This lag is much smaller than the time resolution of the instruments.
Typically, when a change in clocking mode (e.g. 4 CCD -> 1 CCD ) appears in the HK telemetry, there is a lag of a few CCD exposures before the data from the new mode appear in the telemetry. The old frfread did not take this into account, resulting in event files with data from inactive chips. This caused errors when applying RDD correction.
To fix this, frfread 4.0 delays the change in clocking mode until the current and previous CCD IDs are inconsistent with the old clocking mode. If there is a gap in the telemetry, only the current CCD ID is used.
Frfread 4.0 also deletes the data from the first readout of each chip after a clocking mode change. This is because the first readout is needed to clear charge accumulated during the previous mode. Often the pixels of the first readout will be filled with dark current, and the onboard software will reject all the resulting grade 7 events. In this case no data are lost and frfread 4.0 merely adjusts the exposure time slightly.
GIS events come in groups of up to 16 events transmitted in a pair of telemetry frames. Timing bits indicate the temporal position of an event within a frame pair. For example, if two bits are assigned to timing, a timing value of "2" would indicate an event which occurred half-way through the frame pair.
Two things complicate this scheme. First, an event occurring late in one frame pair may not appear in the telemetry until the next frame pair. Its timing bits therefore refer to the frame pair before the one in which the event appears. The second complication is that telemetry corruption can change the value of the timing bits or make dummy events (all bits equal to zero) appear to be real. Both complications may cause the events in the output file to be out of time order.
The old frfread detected "leftover" events by testing if the timing bits indicate the event occurred after the telemetry was sent. This works because leftover events occur late in one frame pair and appear early in the next. Frfread 4.0 retains this test.
The old frfread had an additional step which forced the output event files to be time ordered. It did this by examining all the events in a frame pair for a given instrument. If a pair of events was out of order, the time for the first event was set back into the previous frame pair. This was done recursively so that if this adjustment made the event out of order with the event before it, the procedure was repeated. Finally, after all adjustments were made, events which occurred before the last event in the previous pair of frames were deleted and a gap in the GTIs was marked. However, this caused more harm than good, since the out-of-order events remaining at this stage are invariably caused by telemetry corruption. Treating them as leftover events tended to delete or alter the times of "good" events, and to leave the "bad" event in the output.
Frfread 4.0 retains the first check for leftover events (apparent event time after telemetry time), but discards the second "sorting" step. This means that GIS event files with non-zero timing bits may have some events out of time order. This indicates that the time value for at least one event has been corrupted in the telemetry. Note that despite the incorrect time tag, the events appear in the order in which they occurred, so simply sorting the event file is not a correct fix. The safest practice is to screen out a few-second time interval surrounding the out-of-order events.
Frfread 4.0 has also added a check for null events which have had one or more bits switched to "1" by telemetry corruption. Frfread now deletes all events which have RAWX=0 or RAWY=0 and have fewer than 5 out of the total 32 event bits equal to "1". Note, such events are removed in normal data reduction anyway.
When the GIS modal parameters (and output file) change, "leftover" events which occur in the first frame pair of the new file actually belong in the previous file. The old frfread ignored this situation, but frfread 4.0 handles it correctly.
Another problem comes when the telemetry bit assignment changes. Bit assignment is output to the telemetry once per super frame. However, the actual change is not synchronized to the super frame boundary, so the change could have occurred at any point in the super frame previous to the one where it appears. This makes it impossible to know for certain how to decode the events in the super frame where the change actually occurred. Frfread 4.0 gives a warning when this occurs. Users may wish to screen out these questionable events.
If you suspect an error from frfread, please contact ascahelp@athena.gsfc.nasa.gov.