ENR CTIO Blanco 4-m 2014-03-29

At about 1:20 AM we began having problems with DECam. Alfredo, Alberto, Manuel, and later Javier all worked hard to resolve the difficulties, and succeeded by 4:03 AM.  During this time we obtained only two usable science images, of 90 second exposure.  Slightly more than 2.5 hours of science were lost.  After 4:03 AM the instrument once again performed well.

I will attempt to describe the problems, with the understanding that others were closer to the process of resolving them, and can give more detail.

The issue at first was that the interlocks broke and DECam would not take images.  Resetting, configuring, and re-starting SISPI allowed images to be taken again, but some images came up with about 1/3 of the detectors missing, and we could not successfully run science scripts.  For a while the problem appeared to be with the filter wheel: the filter display indicated hat the filter mechanism was not inserting the correct filter when commanded.  Ultimately it was traced to two computers having gone down -- a more serious problem than any of us had expected at first.  At this point I thought the night was probably lost, very unfortunately.  However, shortly after Javier Rojas arrived, the problem was fully resolved, and I took almost three more hours of good science data -- although with some intermittent clouds.