Troubleshooting

The DECam observer system is quite stable. This section lists several of the most common problemas that can appear during a night, and its possible solutions. If a problem appears find help from your Telescope Operator/ Observer Support Person/ TelOps Electronics-Data support person /Support Astronomer.

There is a skype window in observer1 where many of the experts are usually connected. See if somebody is online to get help.

The best way of get a diagnosis of a possible problem is to look at the Alarm History GUI. The different levels of alarms are:

Warning - The system is still running, but something isn't quite right.
Alert - Something's gone wrong enough that image taking is compromised. An example of this level of alarm is one of the software components going down.
Critical - Hardware failure.

These alarms will produce a distinctive sound to make the observer/operator of a potential problem.

The following is a list of common errors and its solutions:

Unresponsive GUIs
No Guider Stars Found
Broken Interlock
DTS Stops

 

 

UNRESPONSIVE GUIs

It is pretty common that some GUIs stop respondind during the night. Sometimes they will turn out black. Before panicking, just reload the GUIs and they will come back to life. If you cannot see the refresh button on the top part of the windows, just press the F5 key to refresh.

 

NO GUIDER STARS FOUND

First, check the sky. You may not have guider stars because of clouds.
 

The default parameters of the guider worked very well in most conditions. However, sometimes it won't find suitable stars for guiding in any of the CCDs. If this is the case, a warning will appear (which you will see in the Alarm History GUI) and it will make a distinctive sound. The exposure will continue without guiding. You may want to repeat the observation and help the Guider to find guider stars using any (or both) of these options:

  1. Increase/Decrease the exposure time of the guider. The default exposure time is 600ns. For example, during twilight the sky is still bright and you may need to increase the exposure time. On the other hand, in very crowded fields you may want to reduce the exposure time. The procedure to change the exposure time is the following:

     

    • Go to the Architect Console
    • Find the System Control Box
    • Replace the boxes that says:
Component     Device    Command: Press enter when complete

 

with

GCS   GCS   set guideexptime nnn

where nnn is the exposure time in ms.

 

2. Change the search and isolation range (in arcsec) when observing in very crowded fields. To do this go to the Architecture Console and find the GUIDER role. Make the following change:


GUIDER set search_range 5

GUIDER set isolation_range 7


The defaults values are:
GUIDER set search_range 6

GUIDER set isolation_range 12
 

BROKEN INTERLOCK
 

A broken interlock will produce and Alert and you will see a red light in the System Status. Observations cannot continue until the failure is solved. These types of failure can be very simple or quite complex. Simple solutions should be tried first. Complex ones will require an expert. This is the protocol to follow in case of a broken interlock:

1. try Reset (Observer Console, System Control Tab) -> if successful, continue  (takes only a few seconds). Many problems are solved just with this.

2. if you can identify the role with the problem, try to restart (also only a few seconds but some roles depend on others which coould cause problems in rare occasions). To identify the role with the problem go to the "Interlock Viewer" page, which is one of the GUIs up on the observer1 console. Click on the SISPI light blue box in the upper half of the page.  This should fill the lower half of the page with SISPI Interlocks. Make a note of which of the Interlocks is highlighted in red.

2.1. OCS Status -- if the OCS Status is the *only* one that is red, go to the Observer Console GUI, click on the System Control tab, and hit the "Reset" button.

 

2.2. If the OCS Status + Another Process (eg., FCS_STATUS) shows up in red, go to the "Alarm History" page, which is one of the GUIs up on the observer1 console. Identify what the specific alarm is (for example, check the time of the alarm to isolate the alarm that caused the interlock to break).  In most cases, the Alarm History page will provide more details than the Interlock Viewer page. 

Once you have identified the component that has failed, go to the "Architecht Console" page, which is one of the GUIs up on the observer1 console.

Find the component that has failed on the left side of the page, and highlight it.  Click on the "Full Log" button on the middle, right part of the page.  A terminal will pop up with a log history.  Scroll all the way to the bottom of the terminal and verify that there is indeed a problem with the component.

Then, hit the "restart" button under "Role Control" in the top, right part of the Architect Console.  Make sure that a red cross appears before the component that was just restarted--this means that the restart worked.  Wait a few seconds for the red cross to become a green check mark.

Lastly, go to the Observer Console GUI, click on the System Control tab, and hit the "Configure" button.

3. try configure (if you restarted a role or not)   -> if successful, continue (takes about 20 seconds)

None of this 3 options above will affect the queue and the setup.

4. restart SISPI -> if successful, continue. Your telescope operator or support astronomer can help you with this task (takes about 4 min, requires reloading the setup information, configuring and load again the exposure scripts). In a few occasions SISPI has to be restarted 2 or 3 times before the system gets to work again.

5. If restarting SISPI does not work, and expert must be called. One way to do it is by locating people online in the skype account open in observer1.

6. if for some reason the Telescope Control System (TCS) has to be completely restarted then SISPI ought to be restarted afterwards.
 

DTS STOPS

Images are built but not delivered (last buttom never gets green). A Warning will be issued but observations continue. Continue observations and report it in the morning so the people at the NOAO archive recovers your data.