VSA - Known Issues
Please pay particular attention to the following outstanding problems:
VVVXDR1 is missing Quality Bit flags for OB frames, i.e. vvvxDetection.ppErrBits=0 for all non-deep science frames, which means that deblends, saturation, bad pixels, regions in under exposed and from detector 16 are not flagged. We are currently fixing this and will update the release in stages, as we progress. Firstly we will update the Detection tables, which will mean that the light-curve statistics (vvvxVariability) and band-merged table (vvvxSource) will be out of sync. Then we will update the band-merged columns e.g. j_1ppErrBits and primary/secondary flags in vvvxSource and finally the variability tables.
From VMC releases VMCv20160311 onwards including VMCDR4 and VMCDR5, the ordering of the bands in the source table was reversed (YJKs --> KsJY), so that astrometric offsets would be computed from Ks, which had a deeper image. However, it has just been discovered that this lead to Y attributes in the Ks columns in vmcVariability and vmcVarFrameSetInfo since the code was not updated to use the same band pass class as the source merging. We understand which columns are affected now and will fix this as soon as possible starting with the latest team release and public releases and working backwards. FIXED 20201224
The vvvSourceXDetectionBestMatch is unavailable in VVVDR5 while it is being updated (2020-10-01). This is taking longer than expected, but should be completed soon. FIXED.
The MapApertureIDsultravistaMapLc table contains the list of apertures used for light-curves and should be selected using the specification in RequiredMatchedApertureProduct for programmeID=160 and mapID=2, but actually used the specification for programmeID=160 and mapID=1, the dual image mode selection. This leads to 451631 apertures selected from the Ks-band deep mosaic, rather than 576890 selected from a merged-band catalogue of YJHKs+NB118 deep mosaics. The code has been fixed, and this will be corrected in the next release.
The averaging in time intervals is usually done in pawprint pointings, so an object in a VISTA tile/mosaic will probably be on 2 or more pointings, so will have overlapping intervals. In some cases we have specified to sum over overlaps (see RequiredMapAverages, but usually in special cases with limited numbers of objects. This means that objects will not have the full signal to noise that they would have if averaged over all pointings. However, it averaging over the overlaps has the downside of keeping track of the inputs - the MapProvenance would not be suitable, so the table structure may have to change.
20190115. In some of the Value Added Catalogues (team generated catalogues), where primary/secondary sources were flagged across the tile overlaps e.g. vvvPsfDophotZYJHKsSource, vvvPsfDaophotJKsSource and vmcPsfSource, the priOrSec flagging was incorrect and flagged many objects within the main tile away from the boundaries. The code has been fixed and the tables will be updated shortly. The VMC table has already been fixed. All tables have been fixed (2019-04-15)
The vvvPsfDophotZYJHKsSource is unavailable in VVVDR4 while it is being updated (2019-03-19).
20181101. Galactic coordinates (l,b) calculated for value-added catalogues (e.g. vvvDophotZYJHKsSource) are incorrect, because they used a Python function that assumed an input in B1950 coordinates. This may also affect some recent external catalogues which do not calculate l,b themselves. The main pipeline tables \ (vhsSource, vvvSource etc) are not affected because their l,b coordinates were calculated from fast C/C++ code. Over the course of the next few days, we will list all the affected tables, and within a few weeks we expect to fix this problem. All other coordinates: ra,dec, Cartesian are unaffected. Neighbour tables, that are created using ra,dec are equally unaffected. All fixed, (2019-04-15) 20190115. All of the code has been fixed and tested and fixer code is in place to update existing tables. The tables that need updating are vvvDophotZYJHKsSource (also has other issues), vmcPsfDetections/Source (fixed), vvvProperMotionCatalogue, vvvParallaxCatalogue. All fixed (2019-04-15)
Problem with grouting of OB tiles between Feb 2015 and April 2016
Incorrect band-merging in vvvSynopticSource
We have corrected the Ks-band assignment logic and recreated the vvvSynopticMergeLog and vvvSynopticSource tables and also the vvvSourceXSynopticSource linking table. These will shortly be added back into the release. Fixed 20151217.
Effects of Astrometric Library update May 2014 - Oct 2014
New Features in AST
Support for Distorted Projections:
* SCAMP -TAN: This is the same as the TPV projection, but uses a CTYPE
code of -TAN instead of -TPV. AST differentiates between SCAMP (Bertin
2006) TAN headers and standard TAN headers by looking for PV keywords
attached to the latitude axis (a standard TAN projection should have no
such latitude PV keywords).
Support for Distorted Projections: * SCAMP -TAN: This is the same as the TPV projection, but uses a CTYPE code of -TAN instead of -TPV. AST differentiates between SCAMP (Bertin 2006) TAN headers and standard TAN headers by looking for PV keywords attached to the latitude axis (a standard TAN projection should have no such latitude PV keywords).
When we run these codes from data in the data base we had supplied all the WCS keywords whether they existed or not (with default values if not), and in the past the PV keywords were ignored from images with the TAN projection. Unfortunately, to deal with the two types of TAN projection, this is no longer the case. While it seems dangerous to use the same CTYPE for two different projections, we should have been more careful about what we supplied to the AST code.
We have looked through the code and worked out the places that will suffer from this problem. The ppErrBits flag will be incorrect or missing for bit=12 and bit=24, which match the position of a pawprint onto the tile. The ppErrBits has a knock on effect on the primary/secondary selection in the Source tables, so there will be sub-optimal determination of primary sources for the parts of tile that overlap with detector 16 from one or more of the pawprints or where a pawprint detector has been deprecated. Similarly the data in theAstrometricInfo tables will be unreliable in places. This is necessary for completing the SourceXDetectionBestMatch tables, which links unique sources in the Source table to epochs. Specifically epochs where there are no-detections may be incorrectly set or flagged, so if you are interested in the parts of light-curves where there isn't a detection be careful.
The following released databases are affected:
Poor photometry in some VMC deep tiles
Looking at the inputs, we found they were exactly the same as those for the 1_1 Y tile (multiframeID=1403990) in the VMCv20130304 release. Just in case, we did a couple of experiments removing some of the later inputs, particularly one odd pawprint (20110916_v1.2/v20110916_00513_st.fit) that is the only component of an incomplete tile. However, this did not make any difference.
Further experiments demonstrated that older versions of the CASUTOOLS software (e.g. casutools-1.0.21) produced tiles like multiframeID=1403990, and newer versions gave tiles like multiframeID=2768124. Experiments narrowed the difference down to the nebuliser or mosaic package, and a change in casutools-1.0.22 or casutools-1.0.23.
It should be pointed out that the data that went into this tile did have quite a range of seeing, from 0.85" to 1.53", so in the future, eliminating data with seeing>1.1" or seeing>1.2" may reduce this issue. While changes in the mosaicing code may have made this particular tile with highly variable seeing worse, the same changes have improved the quality of many other tiles.
Zeropoint miscalcuation issue
The problem occurred in the ZP calculation of deep stacks - we compare the new deep stack/tile catalogue photometry to the components - OB stacks in the case of deep stacks, or deep stacks in the case of deep tiles. Originally we took the OB stack photometry from the archive, but found in the case of the VVV this took all the archive resources, so we switched to taking the OB stack photometry from the FITS catalogues, as we already did for the deeps. However some of the corrections, which are set to 0 for deeps were not applied for these OB frames. This was realised shortly afterwards and corrected, but we forgot to check all the existing releases, including VMCv20121128. The ZP offset is typically 0.02 mag. The current code uses the archive to get the OB stack photometry except in the case of the VVV. Differences between using the FITS catalogues and the archive are now <0.001 mag, and these difference come from stricter cuts in the archive selection, which makes use of the ppErrBits.
We also discovered that some of the Ks-band OB frames had been recalibrated at the time of processing data for VMCv20120126, by a similar amount. I think that the ZP of the deep tile was miscalculated by the process above and then the recalibration procedure, which was designed to make small adjustments to improve light-curves fed the ZP corrections back into the OB level stacks and tiles. We turned off the OB recalibration stage over a year ago when we found it was making the VVV photometry worse and not making a significant improvement to the VMC photometry. At the time, I thought this was to do with the flatness of the tiles, but now I think it might have been because of some mistakes we made in the past. I will do some tests with the 1.3 data and then turn it back on again in future releases, if I am convinced that there are no problems and after I have introduced a couple of tests which will pick up the above problems early.
Unfortunately VMCv20130304 will be partially affected by these problems. We used existing deep stacks and tiles where possible and any pointings where there were no new data will be affected. Any with at least one new epoch should be correct, as the ZP will have been recalculated using all the corrections.
VHS does not have any deep components, so none of these issues affect it. VIDEO
produce there own deep mosaics, so these issues do not affect it either.
Some of these effects had already been discovered in the VVV originally,
leading to the code changes - the OB zeropoint issue was reverted in VVVDR1 and
the deep stacks and tiles for the next release are unaffected. VIKINGv20121205
and VIKINGv20130417 are also affected.
The problem has been fixed in the VMC, with a new version
Corrupted files on disk38
Poor photometry in some VVVDR1 Ks deep tiles (vvvSource Ks band)
As a result of not ordering frames of differing exposure times prior to
stacking, the resultant tileDeepStacks suffer from incorrect photometry in
the various pawprint offsets. Magnitudes can be off by around 0.6 mags in the
worse cases. Note the Ks photometry in the vvvSource table often comes from
the tileDeepStacks. The disk fields are most affected as typically the initial
observations had different exposure time to the repeats. Bulge fields
have been observed using the same exposure times and are OK.
Of the 336 framesets in vvvMergeLog with Ks frames, 114 are likely to show
the problem to varying degrees,
list of affected frames
Deep tiles not available
Accurate mjds not available
Difficult to use tile-pawprint table
Default mjd values in the SynopticSource table
Missing detections in VMC Ks Deep Tiles
Flagging of saturation in tiles.
Incorrectly deprecated Ks band VVV tiles in VVV-DR1.
These 191 tiles were not included in the vvvSourceXDetection and vvvSourceXDetectionBestMatch tables, the latter is used for creating light-curves, nor were they included when calculating the variability statistics in vvvVariability.
This problem should not occur in the future. The recalibration code has been turned off for tiles in the VSA, and the deprecated frames have now been undeprecated. In agreement with the PIs, this problem will not be fixed in the VVVDR1 release (since this will take a couple of months to fix), but will be fixed in subsequent releases.
Home | Overview | Browser | Access | Login | Cookbook
Listing | FreeSQL
Links | Credits
WFAU, Institute for Astronomy, email@example.com