Continuous. When Umberto Orrego came in this morning, he reported that all the servers were hung. Day staff claim they spent entire day trying to reboot various combinations of machines. Eventually that stopped misbehaving and we were able to bring up observer1/2 computers around 12:30 am, after Anne Elliott disabled BCAMs in software and we power cycled FCM and hexapod. Both observer1 and observer2 machines were unusably slow after that, and system performance eventually degraded to the point where we couldn't keep SISPI interlock. Klaus intervened at 2:00 am and turned off BCAMs again. At this point we had a functional SISPI interface until about 2:45 am, when we ran into continual SISPI configuration problems at the DECAL stage. Also, OCS kept throwing an "invalid prop id" alarm with no numerical reference number. At one particularly ignominious juncture, we couldn't even shut down SISPI cleanly. Also, QuickReduce on observer2 didn't work tonight (though that is probably a quick fix).
| Total Times | Time Observed | Time Engineering | Time lost technical | Time lost weather | Time lost other | Total Program Time |
|---|---|---|---|---|---|---|
| 2014-04-01 | 2.0 | 0.0 | 8.5 | 0.0 | 0.0 | 10.5 |
Printer-friendly version
Only way we got SISPI to work at end of night is by initializing it with Klaus' DES_noBCAM.ini script and then entering a "remove_component DECAL" command in OCS role of architect console. This was crowd-sourced by Alfredo who was helping us all night. It is unfortunate that basically the entire night was lost to technical problems, as the night was photometric with good seeing conditions. Given the solutions that were eventually instituted, these computer/SISPI problems probably could have been resolved earlier by experts.