SISPI failure during morning standards for third night in a row:
On the third standard star set, the gui fell behind by about 5 minutes. Different than last night: we turned obstac off (7:05), let the queue become empty (7:11), and filled the queue with 1 standard star set, let it go down to 1 queue entry, then did the third standard star set.
The turning off of obstac is correlated with the first peak in the max_queue_size. (See first figure). The peaks after 10:25 are at the same time as our debugging efforts described below, the refreshes and the reset.
We watched the images appear on observer2; the gui ran 5 minutes behind. We put in the block; the log reported instantly; the guy ran 5 minutes behind. We turned off vsub, the log reported instantly; the gui, including the telemetry viewer ran 5 minutes behind. A lap top browser telemetry viewer time series ran on time. We conclude that SISPI proper is functional, receives commands from the gui promptly, and executes, while the GUI runs 5 minutes behind.
During this, after the block was put in, we tried refreshing the ics window and restarting the OCS using the architect console. No effect.
We monitored the network traffic using a site provided by Alfredo:
http://nwinfo.ctio.noao.edu/cacti/graph_view.php?action=tree&tree_id=1&l...
I'm attaching the plots from that page: our events start at 8:05 on these plots.
There is traffic correlated with our times on the DECam subnet traffic plot and the SMARTS BW usage plot, maybe on the Prompt BW usages.
We attempted to run a SISPI gui on a laptop, but none of us knew which configuration file to feed the command:
[sispi@observer1 ~]$ setup SISPI
[sispi@observer1 ~]$ architect -c
Tomorrow night we can try it again, this time hopefully with a configuration file.
| Total Times | Time Observed | Time Engineering | Time lost technical | Time lost weather | Time lost other | Total Program Time |
|---|---|---|---|---|---|---|
| 2015-08-18 | 10.75 | 0.0 | 0.0 | 0.0 | 0.0 | 10.75 |
Links
[1] mailto:Sarah.Rice@umit.maine.edu
[2] http://www.ctio.noao.edu/noao/field-collection/field-nr-prog/2531/delete?destination=print/content/enr-ctio-blanco-4-m-2015-08-18
[3] http://www.ctio.noao.edu/noao/field-collection/field-nr-prog/2531/edit?destination=print/content/enr-ctio-blanco-4-m-2015-08-18
[4] mailto:frieman@fnal.gov
[5] http://www.ctio.noao.edu/noao/field-collection/field-nr-prog/2532/delete?destination=print/content/enr-ctio-blanco-4-m-2015-08-18
[6] http://www.ctio.noao.edu/noao/field-collection/field-nr-prog/2532/edit?destination=print/content/enr-ctio-blanco-4-m-2015-08-18
[7] http://www.ctio.noao.edu/noao/field-collection/field-nr-prog/add/node/8298?destination=print/content/enr-ctio-blanco-4-m-2015-08-18
*Observing Team*
_DES Observers:_ Michael Baumer, Jim Annis
_Telops:_ Alberto Alvarez
_Remote Support:_
*Observing Plan*
# Community observer first half: S. Rice and D. Batuski: supercluster cores
# Mid-night standards
# Turn on ObsTac -- all SN fields done recently, so should do mostly main survey
# Morning standards (while watching SISPI closely!)
*Conditions*
# Clear, light winds at twilight increased to ~20-25mph at handoff. Humidity ~25%.
# Better than last night, but still not great
# Seeing was as good as .95" with teff~0.9, but deteriorated after 3am local (~1.3", teff~0.5).
*Accomplishments*
# First sub-arcsecond image of Y3 (exp 465255)!
# 1 ndsflat field sequences (now have 9/10)
# mid-night and morning standards
# main survey hexes
# SN-C3 deep z band
# SN-X2 grizz
# SN-X3 deep gri
# SN-S1 grizz
*Narrative*
# Uneventful. Clear all night, but windy (15-25 mph).
# Good seeing at first (~1"); obstac took main survey hexes.
# Seeing deteriorated; obstac reverted to mainly supernova fields after 3am local.
# 8 flat sequences as of yesterday, 1 today. 90% to our needed 10.
# Standards at midnight and twilight. SISPI failure during 3rd morning standard for third night in a row. See details in eLog.