CTIO
Published on CTIO (http://www.ctio.noao.edu/noao)

CTIO Home > ISPI > ISPI Troubleshooting & known problems

ISPI Troubleshooting & known problems

The most common problem when observing with ISPI, is that the ISPI GUI will loose connection with the TCS. As a result the ISPI GUI might hang. If this happens, close the GUI, stop the ArcVIEW main application and then restart the GUI by clicking on the ArcVIEW icon in the VNC window. Detailed instructions can be found under Simple Restart on the Re-starting ISPI page [1].

If simply restarting the GUI does not bring back up the connection with the TCS, ask the night assistant to re-start the TCS router. See also Telescope & TCS related problems [2]

Limit the number of processes running on ctioa9 to a minimum.
This will improve overall stability of the system.

Further information on problems one might encounter when observing with ISPI, can be found in the following pages:
    ISPI PC & ISPI GUI related problems
    Telescope & TCS related problems
    Re-starting ISPI
    Noise & noise patterns
    Condensation on entrance window

February 21, 2007

ISPI Noise & noise patterns

A simple receipe to measure the noise:

  1. Collect 10 darks at shortest exposure (3.227s)
     
  2. Subtract last 3-4 pairs from each other
    Dark10-Dark9, Dark9-Dark8, etc
     
  3. imstat with lo=-100 hi=100, to get rid of outlayers
     
  4. Divide average std number by sqrt(2) and multiply by 4.25e-/ADU (gain)

    • If number is less than 20e-rms, the noise is within specs.
    • If the number is more than 20e-rms, ask moutain support to

      • check that the cables and ground strap are well-tighed and perhaps ask them to move the cables.
      • Switching the motors for the filterwheels OFF & ON might help too.
  1. Measure the noise again, and again...

 

Site under construction

February 21,2007

ISPI Problems related to the TCS

TCS Time Outs

Occasionally, the connection between the TCS and ISPI will time out. The yellow "busy" light on the TCS section of the main GUI will light up (normal), but not turn off, indicating that the TCS command was sent but no response returned to ISPI. In this case, several things can happen. Some intervention by the user may be required.

  • The GUI will try and restart the connection automatically. This is only done if the check box "TCS auto recovery" is checked.
  • If auto recovery is off or does not respond, click on the "TCS GUI" button in the upper left area of the main GUI. This will bring up the separate TCS GUI. Note any error messages in the TCS GUI and add these to your night report. To reconnect, click on "Stop" in the upper right of the GUI. It may be necessary to click on "Stop" a second time. The GUI should flash and a white arrow appear in the upper left corner. Click on this arrow. The green TCS light should come on. The response field should indicate that the connection is back up. Click on "hide" to close the TCS GUI.
  • It may take 10 seconds or so for the "TCS auto recovery" to execute. If one does not want to wait, the TCS GUI can be restarted manually.
  • If there is ever any doubt about the status of the TCS connection, bring up the TCS GUI and click on "info." A current string of TCS info should come over the link. If the normal TCS info is reported in the GUI response window, then communication is ok.
  • As with all the popup GUI's, always open and close with the "show" and "hide" buttons. These buttons can either be clicked on the popup or the main GUI.

Other TCS Errors

Sometimes there will be a TCS error which is not a time out. That is, for whatever reason, a TCS command sent from ISPI will not be completed. In this case, ISPI will also stop, for example, during a script. The TCS error must first be solved, then the ISPI command or script restarted. Have the night assistant note any errors which may have appeared on the TCS console or on the TCS router.

TELESCOPE PROBLEMS

If you have other problems during the night, please have the night assitant call for assitance.

Detail any problems in the nightly report form [3], and ask the night assistant to fill in a trouble report (GNATS [4]) for serious problems.

It is good practice to put in the DARK filters in the filter wheel in case telops needs to turn on the lights in the dome to fix the problem. You can do this either by moving to the DARK position on the filter wheels or just clicking on the END OF NIGHT button.

February 15, 2007

Problems related to the ISPI GUI

GUI hangs because connection with TCS is lost

The most common problem when observing with ISPI, is that the ISPI GUI will loose connection with the TCS. As a result the ISPI GUI might hang. If this happens, close the GUI, stop the ArcVIEW main application and then restart the GUI. Detailed instructions can be found under Simple Restart on the Re-starting ISPI [5]page.

If simply restarting the GUI does not bring back up the connection with the TCS, ask the night assistent to re-start the TCS router. See also the pages on Telescope & TCS related problems.

Overloading ctioa9

The GUI might also hang as a result of an overload on ctioa9, the ISPI computer. Avoid opening too many terminal windows on the VNCs running on ctioa9. It is recommended to limit the number of windows to the minimum, e.g. one to run IRAF and one to edit script. Use either of these windows to check the temperature.

An apparent problem, where the GUI hangs when using too many co-adds, is most likely related to an overload on ctioa9. We have not been able to reproduce this problem. Hence, we advice to limit the number of processes run on ctioa9, rather than limiting the number of coadds.

Along the same line of thought: when transferring data to another computer, it is recommended to start the transfer at the other end, rather than running the command, e.g. scp, on ctioa9.

Other problems with ISPI GUI & ctioa9

....

NOTE: Site under construction

February 21, 2007

Re-starting ISPI

If the GUI hangs, you will need to restart ISPI, you should start from the simplest restart, building to a more full restart as neccessary.

Simple Restart:

  • Single click the red 'EXIT' button on the ISPI Control GUI (with the really cool cross section view of ISPI).
  • Quit the  'ArcVIEW Main Application' window by first clicking on the 'EXIT' button in the window and then pulling down the 'File' menu in the top, left corner of the window, and going to 'Quit'.
  • Single click on the 'ArcVIEW' button in the vnc session xload box. This will pop up the ISPI Control GUI, and an 'ArcVIEW Main Application' window.  The Application window should have two round buttons, labelled 'Connected?' that appear red.
  • On the ISPI Control GUI, there is a white arrow icon in the upper left corner of the window. Single click on the white arrow. This arrow should now turn black. The two buttons in the Application window should turn green.
  • Minimize the Application window by clicking on the underscore '_' icon on the blue bar, in the upper right corner of each window. You are now ready to operate ISPI.

Less simple:

  • Quit as above for the simple restart. Now check to see if there are any remaining labview processes. To do this, from a terminal window, connected to ctioa9 type the command "ps -ef | grep labview". Note the process# attached to the labview processes.
  • Kill all the remaining labview processes with the command "kill -9 process#" from the same terminal window.  You really only need to kill the parent process in order to kill all the subprocesses.  The parent is the process with a 1 next to the process#.
  • Verify with ps -ef | grep labview that all the labview processes are killed.
  • Now continue to re-start the GUI as above in the simple restart.

Frustration level rising, deep breath, restart

  • Do as in the less simple re-start above. Kill the "TightVNC: ispi's X desktop (ctio:9)"  by clicking on  the X in the upper, right corner of the window. THIS IS THE ONLY X IN A BOX THAT YOU SHOULD EVER CLICK ON IN  ANY PART OF THE VNC DESKTOP!
  • Click the ISPI VNC icon in the upper left of the ctioa8 desktop.
  • Single click on the 'ArcVIEW' button in the vnc session xload box. This will pop up the ISPI Control GUI, and an 'ArcVIEW Main Application' window.  The Application window should have two round buttons, labelled 'Connected?' that appear red.
  • On the  ISPI Control GUI, there is a white arrow icon in the upper left corner of the window. Single click on the white arrow. This arrow should now turn black. The two buttons in the Application window should turn green.
  • Minimize the Application window by clicking on the underscore '_' icon on the blue bar, in the upper right corner of each window. You are now ready to operate ISPI.

And finally:

  • Call the mountain electronics support to reboot ctioa9 and/or power cycle the instrument at the telescope.
  • After rebooting, follow the normal startup [6].

If in doubt, or really stuck, phone the instrument scientists:

  • Contact scientist cell phone: 09-499-6216
  • Nicole van der Bliek: home: 211869; office: 205
  • Malcom Smith: office: 217
     

February 15, 2007


Source URL (modified on 09/27/2011 - 10:31): http://www.ctio.noao.edu/noao/content/ISPI-Troubleshooting-known-problems

Links
[1] http://www.ctio.noao.edu/noao/content/re-starting-ispi
[2] http://www.ctio.noao.edu/noao/content/ISPI-Problems-related-TCS
[3] http://www.ctio.noao.edu/new/Tools/Forms/EON/Form.php?telescope=Blanco%204-m
[4] http://www3.ctio.noao.edu/cgi-bin/gnatsweb.pl
[5] http://www.ctio.noao.edu/noao/content/Re-starting-ISPI
[6] http://www.ctio.noao.edu/noao/content/ISPI-Start-Procedure