CTIO DECam Support Pages

Contents of this page:

-------------------------------------------------------------------------------
SISPI GUIs INTERFACE

Here: http://system1.ctio.noao.edu:7001/apps/

Only from within the CTIO network
 

-------------------------------------------------------------------------------
EXPERT CHECK LIST

-- These instructions are meant to be done before the observer takes flats.  Verify that telops has done this.

-------------------------------------------------------------------------------
RESTARTING AND SHUTTING DOWN A SISPI INSTANCE (Experts only, call CTIO support astronomer)

 

--  These instructions detail how to restart and shutdown a SISPI instance

--  The current *ini file and current SISPI configuration is also posted here.
 

-------------------------------------------------------------------------------
INCREASING THE GUIDER EXPOSURE TIME

1.  Go to the Architect console:   http://system1.ctio.noao.edu:7001/apps/architect_console/
2.  Find the System Control Box.  Replace the boxes that say:

Component         Device           Command: Press Enter When Complete

with:

GCS     GCS          set guideexptime nnnn

nnnn is the  time in ms. The default is 600 ms.

NOTE: this can now be done directly from the menu in the guider GUI

 

-------------------------------------------------------------------------------
HELPING THE GUIDER FIND GUIDE STARS IN CROWDED FIELDS

You can change the frame search and isolation parameters (these are arcsec) from the architect console

to:

GUIDER set search_range 5
GUIDER set isolation_range 7

the values below are the default at the moment:

GUIDER set search_range 6
GUIDER set isolation_range 12
 

 

-------------------------------------------------------------------------------
RESTARTING THE COMFORT DISPLAY

1.  Follow instructions here (bottom of page):  Instance Start-up For Experts
2.  If there is no "comfort1" node and no "DISPIB" role to restart, first go to the architect console and replace

Component         Device           Command: Press Enter When Complete

with:

ARCHITECT     MANAGER          start_node comfort1

2a.  Connect to the VNC server on system1, display 2. Find the OCS Console and the Manager Console:

OCS Console: select ARCHITECT
MANAGER Console: start_role DISPIB comfort1
MANAGER Console: start_role DISPIS comfort1
MANAGER Console: select OCS

2b.   Go to the comfort display (comfort1) and make sure a desktop session is running. If not, start one using the DECamObserver account.  Open a terminal and enter the command

xhost +localhost

-------------------------------------------------------------------------------
RESTARTING THE DECAL COMPUTER

If SISPI loses contact with DECal (for example, error message "DECal not ready"), the DECal computer must be re-initialized:

  1. Go to the DECal room (floor MZ) and power-cycle the computer. There is no password in this computer and a straight-forward power-cycle will start automatically all three DECal servers. You may need to do a SISPI "Restart" (and maybe a "Configure").

        Alternatively, this procedure can be performed remotely by VNC connection into the DECalS machine:                                              
        vncviewer -Shared 139.229.13.61   (or vncviewer -Shared DECalS.ctio.noao.edu)
 
           (password available in the Blanco 4m control room)

If this fails (it shouldn't) there are two other options you can try which involve unlinking DECal from SISPI. Notice that these options will not allow to take calibration frames. 

  1. Go to the architecture console and replace 

Component         Device           Command: Press Enter When Complete

with:

OCS                      OCS          remove_component DECAL

to add it back in:

OCS                       OCS         add_component DECAL,DECAL,DECAL

      2. A more drastic option is to re-start SISPI with no DECal role by using the following initialization file:

 

architect -c DES-nodecal.ini

Additional instructions on the DECal computer are found here: DECal Server Emergency Restart Procedure

 

-------------------------------------------------------------------------------
RESTARTING THE GUIDER GUI

Open a terminal on observer 1 and start a vnc viewer that connects to readout2.ctio.noao.edu:15

> vncviewer readout2.ctio.noao.edu:15

(password is on Andreas updated list, come to office 14 to pick up a copy)

If you vnc into the guider fui from home, please use this only when needed and then disconnect - sending the image data across the network uses resources.

 

-------------------------------------------------------------------------------
RESTARTING CHECKING THE STATE OF VSUB (IE, IF VSUB BUTTON IS YELLOW)

Typically vsub is either on or off for the entire system (all 6 Monsoon crates, counting each split backplane as a crate). Sometimes the crates are in a mixed state – some on, some off. In this case the vsub indicator on the ObserverConsole GUI turns yellow. To find out which crate and which module (slot) in each crate has vsub on, use the command get vsub details.

Go to the Architect Console, select the OCS role and enter the command in the control section. This returns a string like this:
<pre>
FPANS:
SL4_CCD12_VSUB_ENBL 0.000000
SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkpf
SL4_CCD12_VSUB_ENBL 0.000000
SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkp4
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp5
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp1
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp3 GPAN: 0.000000 </pre> GPAN is the guider backplane (bkpg).

A 0.000000 indicates that vsub is off for this module.

A 1.000000 indicates that vsub is on.

-------------------------------------------------------------------------------
 ACTIVATING A MASTER CONSOLE

Sometimes the GUIs get slow and we restart them, or we've restarted the SISPI instance, logged in to the GUIs, then remember to restart them because that's easier than refreshing each one. When this happens, the new GUIs have a new session, but the old session still has Master Console. To reset this deadlock and to restart the GUIServer:

(a) open up an xterm, and log into system1.ctio.noao.edu as the "sispi" user.

xterm> ssh sispi@system1

(b) setup a SISPI-specific set of commands

[sisp@system1 ~]$ setup SISPI

(c) execute the releaseMasterConsole command

[sisp@system1 ~]$ releaseMasterConsole DECam_YYYYMMDD

(where YYYY = year, MM = numerical month, DD = numerical day)

(d) log out-of the GUIs (or open them-up again)

 

-------------------------------------------------------------------------------
STOPPING DTS

To observe without DTS use this command in the Arquitecture Console:

OCS OCS set dtsqueue None

Alternatively, Tell the OCS “ib set call_dts false” to disconnect the IBs from the DTS

The dtsqueue and all other exposure parameters are set when the request is entered in the queue and not when the exposure is started. If using this command, the change will be effective for every new request that is loaded. To inmediately stop DTS use the second parameter. To resume DTS, use

OCS OCS ib set call_dts true

Always send a message to sdmops@noao.edu when this happens so they can transfer the images later.
 

-------------------------------------------------------------------------------
MONITORING DTS

you can always check

http://dts.noao.edu/status

to check on DTS at all nodes.  The "Pend" column is the number of pending transfers out of that node and is usually near 1, but can be much higher during fast cadence programs or calibration sequences.  

-------------------------------------------------------------------------------
TURNING AOS ON/OFF (updated ARW 20180712)

 AOS (Adaptive Optics System) is on by default. The active optics system corrects both the focus and alignment of DECam, and the figure of the primary mirror, automatically using information obtained from the eight focus and alignment sensors. Currently SISPI will initialize DECam to control all 5 terms in the hexapod (xshift, yshift, zshift=focus, tip, tilt). In some ocassions, the observer may want to turn this feature off. For example, recently we had a case in which the observer wanted to do consecutive images, with no dithering, hoping their objects will lie in the same pixel (or about) from image to image. This usually does not happen due to shifts an tip/tilt movements of the hexapod, so in this case you want to only allow adjustment of the focus.  

To change the default (full control of hexapod + control of the primary) you need to do the following: In the architect console write the following in the three command boxes (or scroll down to find AOS then it will fill in the first two for you).

FOCUS AOS init_pid aos1        (This will only then adjust the focus using the AOS)

FOCUS AOS init_pid aos5        (This allows control of the focus, shifts, and tilts of the hexapod)

FOCUS AOS init_pid aos8        (This is the same as AOS5 plus it corrects the figure of the primary mirror.  THIS IS THE DEFAULT)

The system will get turned back to AOS8 by a "Configure" or by a SISPI restart.

There are also:

FOCUS AOS init_pid aos_off       (This will turn off all AOS corrections, inclusing focus)

FOCUS AOS init_pid aos_init      (I have never had to use this)

NOTE:  (i)   There is a "tweaks on" and a "tweaks off" in the TCS control.   This is refering only to the primary mirror.   The TCS has no control over the hexapod.  But obviously, when you are using AOS8 ie the default mode, you do want "tweaks on"  on the TCS side.   That is the default there too.

(ii) The hexapod is also contolled by a Look Up Table (LUT) with different values across the sky, to compensate for telescope flexure.   You interact with this via the ICS GUI,  this will only be needed for engineering purposes.   You can turn off the use of the LUT by unclicking "telescope"  You can also turn off tweaks, and zero out trims, or enter values.   You can force the focus to be something "wrong" by for example entering 1500 (microns) for the z trim and zeros for all the other trims, with the AOS off.  Be careful, and note where the "STOP" button is.   You do not want the hexapod to run away and hit a limit.  It is not easy to recover (needs the standalone program on the Michigan laptop).

 

-------------------------------------------------------------------------------
QUICK REDUCE

The Reset button can be used to recover from unexpected errors.  If this does not work, restart the server as follows:

ssh quickreduce@quick1.ctio.noao.edu
(the password is on Tim's list)
[quickreduce@quick1 ~]$ bash
[quickreduce@quick1 ~]$ ctio_portal_server.sh restart
 

If you are missing the browser display,  open the following page in observer2:
 

http://quick1.ctio.noao.edu:8080/# 

and login as DECamObserver (password in Tim's list). Then, hit QuickReduce and Observing History

If the problem remains send email to:  helpdesk@linea.gov.br

See https://cdcvs.fnal.gov/redmine/projects/des-sci-verification/wiki/Qr and reference manual  DES-doc 6705 

-------------------------------------------------------------------------------
GOING THROUGH THE DECam LOGS

DECam writes logs for every action of every component, for every instance, for ever.   It is all on disk.  Some useful tips to look for information in the logs:

ssh into system1.ctio.noao.edu as sispi   (as always when logging as sispi, BE CAREFUL, tread lightly)
cd decam/instances

Every instance since September 2012 is there as a directory.  Change to the one you want.  e.g.:
cd DECam_20140322

then go to the logs:
cd logs
ls   (terrifying, isn’t it!)

What you want usually are the OCS logs:
ls -l *OCS*

If you have restarted SISPI more than once in the 24 hour period there will be multiple logs.  The one in use has a link to it, which is convenient (this is a general feature). The others you can identify by the time stamp on last access, which is why you do “ls -l”. Now you can use your favorite unix tools to parse the log file.    Go to the end, see why it stopped. I usually just search for “ERROR”. If you identify a component giving trouble you can go and look at the corresponding log for that component, but often just looking at the OCS logs can tell you what is causing a problem.

-------------------------------------------------------------------------------
FROZEN OBSERVER2

If observer2 gets frozen, the first thing to determine is if it is really hanged. Try to login from another machine (via ssh for example) and check if images are still transferred. If that is the case, maybe the best bet for the night will be to continue observing and use observer2's tools remotely. This option has zero downtime. On the other hand, should observer2 completely stop, SISPI would hang trying to copy an image to it. At that point you have no choice but to restart and to reboot. 

If rebooting is necessary, this is the correct order: (1) Stop SISPI, (2) reboot observer2, (3) restart SISPI

 If observer2 is rebooted while SISPI is taking data, SISPI will hang, because it copies images over to observer2. Don't want to hang SISPI. 

 

-------------------------------------------------------------------------------
COPYING IMAGES MANUALLY TO OBSERVER2

If for any reason images are not transferred automatically to observer2, there is an option to transfer them manually (one by one or in blocks). The procedure is the following:

1) login in observer2 as sispi

2) cd to the diretory where the images should appear (for community observers this is /home4/images/fits/XXXX  where XXXX is the Prop-ID)

3) setup DDS

4) setup Scripts

5) get_images --help

to obtain the options for getting images. A particularly useful one is: get_images -v '[id>408065]'

 

-------------------------------------------------------------------------------
OBSERVER3 DEAD (update ARW 20180712)

If there is a real failure in observer3, observer2 can be used as a backup following the following instructions:

1. open a vnc session from observer2 into system1, as sispi
2.  cd ~/decam/architectures/ctio
3. setup SISPI
4. shutdown SISPI  (if it was still running, or just in case)
5. architect -c   DES-observer2.ini

Notice that: Our old original machine (observer1) is still sitting under the desk.   That could be bought up to be used, but not without cable swapping, ini file change, etc.

 

-------------------------------------------------------------------------------
TELEMETRY VIEWER

For debugging and finding problems is useful to make your own plots and find correlations among the many quantities of the Telemetry database.

http://system1.ctio.noao.edu:8080/TV/app/index

Works from within the CTIO network.

-------------------------------------------------------------------------------
RESTART Chrome session (if pulldown buttons on desktop don't work):

[observer3 ~]$ ssh -l sispi system1.ctio.noao.edu
Password: [the system1/sispi password, ask TelOps]
[system1 ~]$ setup SISPI
[system1 ~]$ releaseMasterConsole DECam_YYYYMMDD

-------------------------------------------------------------------------------

CHECK_HARDWARE

a command that tests the following DECam hardware for basic functionality:  hexapod, filter unit, shutter, DECal.    For safety, only the filters and the hexapod are actually moved - not the shutter.

Note that this command does not necessarily FIX a problem, it is meant to be used for instance in situations  like we have had in the last month with the power fails, to check that the hardware is still OK  (or not) after the power is back on, but when we had the DECam crates off.

Prequisites

SISPI has to be running. It can be configured but this is not required (for example when the CCD crates are off).
The components to be tested have to be included in the configuration, i.e. in the .ini file.
The hardware controllers have to be on and connected to the network (of course)

Usage

From the either the text console or the Architect Console enter the command (OCS) check_hardware

Tests

The routines checks the hardware components in turn. It configures them individually and then performs a set of predefined operations to check if the device is functioning properly.
Here is a description of the individual commands used:

HEXAPOD -- configure, check for errors (check_ready function), move relative in z +20/-20  
FCM -- configure, get current filter, if not block, move in block else move in and out z filter  
SHUTTER -- configure, call status function and check if connected  
DECAL -- configure, get led

-------------------------------------------------------------------------------

How to turn off Vsub  after a sispi crash

There is in fact a low level script that can turn on/off the vsubs with or
without sispi being present. Now, the VSUB voltage is handled exclusively
through the monsoon controllers, so the ICS has no way of knowing or
handling it, which in turn means that the panviews should be running for
this script to work.
So if sispi is not running, but the panviews are, you can type this on a
terminal on server1 or any of the readout machines:

<i>vsub_all off</i>

or if you run it without argument it should return the status of vsub on
each of the backplanes. Note that you can run this script even is sispi is
running -it goes trough another instance of the socket server.

If something major happened -like a server1 crash, so nothing is running-,
unfortunatelly there is nothing to do about the vsubs other than indeed
turning the crates off.

-------------------------------------------------------------------------------
STAR FLATS COORDINATES

13:27:00 -48:45:00 (preferable)
19:00:00 -50:00:00
20:40:00 -35:00:00
06:40:00 -34:00:00
07:30:00 -50:00:00

-------------------------------------------------------------------------------
RASICAM MAINTENANCE TASK LIST

The rasicam desktop

Thirteen main icons are identified. The two programs run each night are centered and marked with asterisks.

Run nightly are: 

  1. the main rasicam labview VI (G:\PRECAM Version\LabVIEW VI's\RASICAM CONTROLLER.vi)
  2. a background python application started by RUN.bat on the desktop (starts cygwin bash shell and runs bash --login /home/plewis/bin/rasicam_server.sh

 

Turning on the FLIR camera and using the FLIR camera player

The FLIR camera player can be powered on and used independently of rasicam. To do so open the labview VI called "Manual DAQ Controller" which is shortcut number 3 on the desktop at top of page.

Rasicam focus (fix defocus)

Occasionally, rasicam will become defocussed. When this occurs, the secondary mirror and three support spiders become blurry and will "spill out" from behind their masks. Examples of what this looks like are are shown in the following YouTube videos.

The solution is to use the "FLIR Camera Player" and click the "auto focus button". This can be done day or night and with the dome open or closed.

20160529 images of "FLIR Camera Player" when camera was defocused, autofocus button was clicked, secondary came into focus, done.

becomes

Note that these two photos were taken on a rainy night when rasicam was approximately in thermal equilibrium. Much better contrast will be seen on future attempts (replace or add to the above images.

becomes

New flat field

If the previous night was clear (check rasicam movie) and not high humidity (check telemetry) then a new flat field can/should be installed. This should happen about once a month (2 week cadence max). Reil will create an icon on the rasicam desktop that does this.

New field of view mask

When/if alignment changes a new field of view mask is needed. There is a labview vi that creates new masks. Once new mask is created a desktop icon/script will be run to make mask backups, make mask copies for python code, etc needs to be run.

New black body mask

When/if alignment changes a new black body mask is also needed. There is a labview vi that creates new masks. Once new mask is created a desktop icon/script will be run to make mask backups, make mask copies for python code, etc needs to be run.

Nightly cron jobs

  1.  Youtube movie, night summary emails
  2.  Data backup/transfer to SLAC, etc

----------------------------------------------------------------------------------------------

RASICAM Data and Movies  (from Kevin Reil)

1) Data collection (raw data)
*.txt files are collected via labview on rasicam.ctio.noao.edu
 
2) Transfer of raw data to observer3
on rasicam.ctio.noao.edu there is a cron job running (~plewis/bin/cron2.sh) that every hour (at 45 minutes) it copies all of the data on the local hard drive to the raid backup drive, to SLAC national lab and to DECamObserver@observer3:rasicam/raw (I've disabled SLAC as that machine is offline permanently)
3) On observer3.ctio.noao.edu (as DECamObserver) a cron script (mid morning) ($HOME/rasicam/cron/cron1.sh (cron2.sh)
a) Processes raw data into a single fits file with one header per exposure via $HOME/rasicam/scripts/process_all.sh --maxdays=31 (or $HOME/rasicam/cron/cron2.sh --force2)
b) Creates an mp4 file and posts it to youtube (build_movie.sh) (includes youtube upload).
c) Creates and emails a nightly summary image (month view).  (build_summary.sh)
 
4) If a movie is missing then 
~/rasicam/bin/build_movie.sh 2020.03.27 (replace date with YYYY.MM.YY this requires that fits files exist for that date
or
>> cd rasicam/cron
>> ./cron2.sh 2020.03.27 --force
(this will do everything that is done daily including sending the email.)
 
5) You can only upload about 5 missing movies per day until you hit the quota from google.
 
6) youtube and emails are rasicam.ctio google account. 

---------------------------------------------------------------------------------------------------------------------------------------

DECam Optical Metrology

From the solid works model of DECam -
  • From the nearest filter (back surface) to the focal plane: 497mm  (z,g)
  • Between every filter (back surface to back surface): 40mm 
  • So, from the second nearest filter: 537mm (i,Y)
  • Third: 577mm (u,VR)
  • Fourth: 617mm (r,N964/N662)
  • Filters are approx 13 mm thick

-------------------------------------------------------------------------------
USEFUL LINKS

1.  SISPI NEWS

2.  Instance Start-up For Experts

3.  Get-Ready Check List

4.  Ann Elliot's Debugging throug the GUIs page

5.  DECam Operations

6.  Current SISPI Configuration