Status at 2001 Jun 29

Roger Smith

(action items in yellow)

Requirements:

I have invested considerable effort in control of extraneous light sources. Read noise in the SITe 2Kx4K CCD in the Hydra Camera is low enough (3.4 e-) that it can easily be overwhelmed dark current, spurious charge, phosphorescence or light leaks. Recall that

Noise_Floor = SQRT{ read_noise² + Xbin*Ybin*(Spurious_charge + Dark_Current*time) }

Read Noise (in the overscan) is 3.0 e- for the high gain setting. If the longest exposure time is half an hour, and highest binning factor is 2x2, then we require:

and,

(All figures below are quoted for unbinned pixels.....)

Detector performance

• Spurious charge = ~ 0.25 e- / pixel on the high gain setting.
• When gain is changed, parallel clock voltages are manipulated to acheive this low spurious charge at the cost of well capacity. Note however that the well capacity for high gain, 15000 e- / (0.84e-/ADU) = 18000 ADU, when binned 2x2 exceeds the ADC range by about 10% so the loss in well capacity will not ofetn be noticed.
• Dark current < ~0.7 e-/pix/hr, when detector protected from bright lights.

Thus...

Spectrograph

• The lens cap and camera baffle/cover are necessary to avoid phosphorescence or image remnance effects. Values of ~5e-/pix/hr from the detector (window?) are easily achieved by exposing the detector to a flashlight briefly. I have not tested this aggressively since I didn't want to compromise the observer.
• The detector cap plus baffle are sufficient ot make it safe to adjust the spectrograph using flashlight for illumination without risk of compromising the observing. The curtain gives an extra margin of safety for refills.
• However the Schmidt corrector (or surrounding cable?) is mildly phosphorescent. If hit with a flashlight will increase dark current to 3-10 e-/pix/hr decaying over an hour or so to negligible levels. The plan is to divide the camera baffle into two overalpping parts, one serving to protect the corrector from accidental illumination.
• The test has not been performed to see if doing flats is enough to cause significant phosphorescence bt the corrector. I suspect this wont be a problem given based on my flashlight test: a series of images alternating between darks and flats should be done to test for this.
• The LEDs in the filter "windmill" which illuminate the slit were shown to turn off fully now that a relay had been installed.
• A conventional dark frame is an exposure with the shutter closed. Since the shutter is immediately in front of the spectrograph camera, the dark frame does not reveal light leaking into the optical path. A cap is needed over the slit to permit measurement of dark current plus light leaks. (Even when the dome lights are off, a significant amount of light comes down the fibers.)

The room

• Detector limited dark current is achieved whether or not the dome lights are on provided that the baffle is installed and provided that shutter is closed. (Detector uncapped and schmidt corrector not exposused to light for hours).
• If the slit is wrapped in Aluminum foil and black cloth, one can acheive detector limited dark current even with the shutter open, but only with done lights off. When dome lights are on the "dark current" increases about 10 fold. Hence there is a leak into the spectrograph room.
• Removing the baffle over the spectrograph camera increases "dark current". This test was done before I had determined that light leaked into the room from the dome so it is not clear whether the light blocked by the baffle was coming from the dome or the Arcon.
• The black tape which seals up the doors and various feedthroughs from the dome is actually translucent. The room needs to be imaged with Arcon plus camera lens needs to be used to image the room to see whether the black tape needs to be replaced or augmented.

Arcon

• The communication fibers produce a huge amount of light (~850nm) where they enter the Arcon front panel. A rough metal box with holes for the cables to pass though was made, but this requires large amounts of black tape and black cloth to seal it. A more precisely made cover is needed to provide an adequate seal while being easily demounted to provide access for maintenance.
• There are 6 LEDs inside the detector cap, controlled by the preflash parameter in the setdetector menu. These allow one to verify CCD perfromance. Since it is also useful to be able to check dark current with the cap on, the fact that there is leakage current through the LEDs when off is a problem. A transistor needs to be added to the LED drive circuit in the Arcon power supply to shunt the current past the LEDs when off. This is preferable to a relay since the LED pulse must be very short.
• The white plastic diffusor in front of these LEDs was removed since it was found to phosphoresce (tens of e-/pix/hr) with seemingly infinite decay time. This effect is clearly distinct to the LED leakage current problem as it iss present when the LED cable was unplugged.
• The dark current is seen to rise after the Nitrogen is refilled. This is because the addition of liquid Nitrogen to the pumped (solid) Nitrogen raises the temperature of the heat sink for the detector, for an hour or so, until the pumping (and consequent accelerated boil off) cools and re-solidified the Nitrogen. It should be noted that dark current is so critical in this application that the CCD is being operated at the lowest temperature achievable (about 150K): CCD temperature is not actively stabilized. The extra 16 K or so of cooling proved by pumped Nitrogen, is possibly due to poor thermal contact between the CCD and its supporting plate due to differential contraction between the Al plate and the stainless steel bolts which pass through it. Spring washers may be difficult ot retrofit due to the cramped space, and may be insufficient. Since the pumping and refill process are working well it has been decided to continue with this solution. Therefore, vacuum lines need to be routed to the large mechanical pump located in the corridor behind the telescope, instead of using the small pump currently located on the cass elevator platform directly under the telescope.
• Not related to darks but important: I need to figure out why the combination of gain=1 and Xbin=2 causes a strong pattern in the images.
   

2001 August 31
Roger Smith
(action items in green)

Signal to noise in the Hydra Bench Spectrograph is often limited by the noise floor of the detector, which is critically dependent on dark current. Binned by 2 in one or both dimensions, is commonly used to improve the signal to noise ratio or the read time. One must remember that while this helps overcome electronic noise it doesn't help overcome the dark current: binning the signal also bins the dark current.

We have to concern ourselves with all sources of "unwanted signal". Noise in the absence of any astronomical signal is determined by the summation in quadrature of the noise associated with

• CCD and electronics
• Thermal dark current
• Charge injected into CCD
• Spurious charge generated by clocking
• and light leaks.

Definition of Terms

Thermal Dark Current

This is the charge which accumulates per unit time as some electrons in the valence band gain enough energy to overcome the bandgap and jump up into the conduction band. As the detector is cooled the probability that a valence band electron acquires enough energy will drop so that the dark current drops exponentially with temperature. At least this is the case for electrons in a perfect silicon crystal.

In a real device there are imperfections, impurities and crystal dislocations which provide intermediate energy states so that a valence electron can get to the conduction band in two smaller energy steps, increasing the probability of a "thermally generated" conduction band electron. These "hot pixels" produce less dark current with temperature too.

The density of crystal imperfections is greatest at the interface with the oxide layer. Dark current here is suppressed by taking the electrodes negative enough to pull "holes" out of the channel stops (ion implants that create potential barriers between columns). These holes populate the surface of the CCD just under the oxide layer, while the signal charge is repelled to a layer deeper in the CCD. The holes recombine with electrons generated thermally at the surface so that they never reach the storage well.

So, dark current is controlled by lowering the CCD temperature and by maintaining all clock phases negative during integration (MPP mode). The minimum temperature reached by the CCD mount is about 160-165K depending on ambient temperature (radiant load). This should be low enough to ensure good dark current performance. However it was found that dark current always exceeded that expected for the mount temperature measured. The most likely explanation for this is that the CCD is not making good enough thermal contact with the mount and that its temperature is being increased by the radiant load. Fixing this appeared to involve building a new detector mount. Rather than do this the Nitrogen flask is evacuated to promote evaporation/boiling which continues until the remaining liquid solidifies. The temperature drops by 15 degrees in the process and reduces the dark current to an acceptably low level.

Charge injection

Charge injection is the leakage of charge into the image area from the periphery of the CCD during the erase cycle. It can't be erased away since it is smeared across the chip by the erase process. Instead we can clock in reverse at the end of the erase cycle to push it back towards the edge where it originates. I mention it here because it is a serious problem in some of the CCDs in Mosaic which are of the same type, though much less so in this CCD.

Spurious Charge

Spurious charge is the small amount of charge generated by clocking the CCD. The mechanism is impact ionization: the surface just under the oxide separating the electrodes from the depletion region are populated by holes (due to the negative level on the clocks). This is good since these holes recombine with dark current generated as valence band electrons are excited to conduction band more easily by hopping to the intermediate energy levels produced by interface defects. However as these holes flow back into the channel stops during the rising edges of the clocks they have a low but finite probability of delivering energy to valence electrons which once liberated are collected like signal electrons. At the clock voltages needed to achieve full well, one sees about 2 electrons of spurious charge per pixel implying the probability of generating the spurious electron is about 1 in 2000 per shift.

When high gain is sleeted in the setdetector menu, the waveform compiler embeds commands in the new waveform macro which reduces the high levels of the parallel clocks. This reduces the spurious charge to an acceptable level at the expense of full well. When the high gain setting is selected the well capacity is reduced to 15000 e-. At 0.84e-/ADU this represents 18000 ADU, but when binned 2x2 exceeds the ADC range by about 10% so the loss in well capacity will often not be noticed.

Light Leaks

Obviously light leaking into the spectrograph masquerades as dark current and has exactly the same effect. While light leaks are in principal very simple to understand there is an extensive set of tips in the section on Eliminating Light Leaks, which you should read if you detect a problem.

Check for light leaks (etc) before each observing block !....

It is most likely that to maintain full sensitivity of the spectrograph it will be necessary to verify that all of these "dark signals" are within normal bounds prior to each observing block. To do this take the following full frame exposures with binning set to 2x2.

• 3 zeroes with CCD cap on; quadproc (or ccdproc) these to overscan subtract and trim; median filter (imcombine)
• 600 second "object" exposure with slit cap interlude and the detector cap removed, all lights in dome and surrounding rooms off. Subtract overscan and trim (quadproc). Note that you should not use a "dark" exposure since the camera will not see light leaks when the camera shutter is closed, since the camera has shielding to protect it from stray lights and reduce phosphorescence in the corrector during Nitrogen refills.
• Subtract the combined zero from the object; use implot and the "s" key to measure dark current, dodging cosmic rays. If you have time and want to increase the sensitivity of this measurement, you can take a sequence of 3 darks; ccdproc and median filter them. With cosmic rays thus removed you can average many lines in implot before measuring the mean using the "s" key or even "fly blind" and imstat the whole frame. If you use imstat, be careful not to include the 20 or so edge columns which can contain line start transients.
• Look at the difference image with Ximtool, to make sure there are no hot edges or corners.
• Repeat the object frame(s) with the slit still capped but lights on in the dome, corridor outside the bench spectrograph room, and in the neighboring rooms to ensure that it is safe to take darks when people are working in the dome. Process and review identically.

If the dark signal is abnormally high, even when dome and surrounding room lights are off, check that no abnormal or unauthorized objects have been left in the room which may be phosphorescing. For example the paint on some nitrogen dewars may glow in the dark, too faintly to see by eye but enough to affect the CCD. Unless you find a simple explanation like this, don't waste time guessing about the cause: you should use a cooled CCD and camera lens to image the bench spectrograph room.

What is an acceptable dark current and light leak?

Recall that

Read Noise (in the overscan) is 3.0 e- for the high gain setting. Assuming that the highest binning factor is 2x2, then we require:

and if the longest exposure time is half and hour, then we require:

Actual performance

• Spurious charge is only ~ 0.25 e- / unbinned_pixel thanks to the lowered upper voltage rail for the parallel clocks.
• Dark current < ~0.7 e-/pix/hr, when detector has been protected from bright lights for at least a day.
• Light leak should be no greater than dark current, and ideally be undetectable. At present (Aug 2001), this is only achieved with the lights in the dome and surrounding rooms off. We are currently working on fixing this both for robustness and the ability to work in the dome while darks are being taken.

When all is working correctly,

How to Eliminate Light Leaks 

This has not been as simple as one might think! Here are some tips....

True leaks:

• We are susceptible to leaks at such a low level that significant sources will be impossible to see by eye, so intuition is often wrong!
• The black cloth around the spectrograph attentuates the light but it is not completely opaque.
• Adhesive tape may dry out and/or come loose, so you can't assume that the previous fix will be permanent.
• Many types of tape including our favorite high quality black tape is in fact translucent in the visible and/or become transparent beyond the visible range where the CCD is still very sensitive.
• Metal backed tape is opaque for sure but the adhesive is sometimes not strong enough to overcome the intrinsic stiffness of the tape itself.
• The metal baffles around the camera stop scattered light and light leaks from reaching the CCD directly but of course light can still enter through the optical path.
• Use "object" frames, not "dark" frames for testing since you can only see stray light in the optical path when the shutter is open, since the shutter is just in front of the camera corrector.
• Never leave even the red lights on in the vestibule are or behind the sealed door that leads to the back of the telescope.
• Always close the door to the corridor.
• Plug the hole where the cable tray passes from the vestibule to the elevator entrance ("M" floor).
• There are currently leaks under the access doors, so that the warning light (for the detector cap) in the corridor can be easily seen by the CCD camera even when both the inner and outer door are closed.
• There is what appears to be a leak at roof level into the dome or mezzanine room above the entrance door.

False leaks:

• Light coming down the fibers during testing must be blocked. It will overwhelm the dark current even with dome lights off. A manually installed metal box (felt lined) has been made to cover the slit assembly. Remember to remove it when you finish!

Light sources to eliminate within the spectrograph:

• LEDs in Motor Controller (done)
• LEDs in control box for detector cap (done)
• LED in new Quicksilver shutter motor ...to be checked.
• Smoke detector (electronics emit light, not just its LED)
• Arcon data fiber connectors must be encased and sealed at the Arcon and the cable breakout box. This is probably the weakest link since any work on Arcon will potential disturb this work. Imaging these fibers while live requires that a second working fiber be connected to the Arcon which is taking the pictures. To do this one has to feed the Arcon on the hydra bench from a dummy source, such as the Mosaic Trambox. The Arcon doesn't have to be loaded with code and functioning. It just needs to be powered on and have a live fiber connected.
• Arcon fiber leakage. Only use black fibers. All other colors leak a lot of light. Check that the fiber cladding is in good shape along its whole length. A damaged cladding can leak copious amounts of light. (I've seen this happen.)
• There are preflash LEDs in the detector cap which are very useful for checking that the CCD is functioning without needing a working spectrograph. If there is leakage current while they are off, this can appear as excess dark current. The power supply has been modified so the preflash circuit has a shorting switch across the LEDs to shunt the current source to ground as well as a switch in series, to eliminate this effect. Beware: if an unmodified supply is installed the problem will occur again.....one will see dark current increase when the detector is capped!

Phosphorescence

Many materials absorb light then re-emit it over a long period of time. This can cause a dark current which decays over time and is thus harder to pin down. It is therefore important to not leave any item inside the room unless it is essential and has been proven to be free from phosphorescence. These are some examples which have been addressed:

• Fluorescent tubes (removed)
• White paint on fixtures for fluorescent tubes (removed)
• Smoke detector (Removed)
• Gray paint on metal power strip (over-painted black)
• White paint on bulkhead near base of telescope (painted black)
• Linoleum floor ? .... maybe not, could be reflected light
• Oscilloscope screen ...of course.
• White paint on some dewars. (Store dewar and all tools in anteroom)
• Corrector glass (shield built to minimize illumination by flashlights.) Need to verify that bright comparison lamp or quartz spectra don't cause a remnant signal.

Finding unwanted light sources

If no obvious source is found after consulting the above checklists, don't waste time guessing. Use a cooled CCD camera fitted with a camera lens and integrating for 2-5 minutes to identify light sources. The following setup is very effective:

• SITe2K camera, e.g. T2K5 or T2K6.
• Wide angle lens such as Zeiss distagon 40mm f/4, with the custom Hasselblad lens adapter for standard dewar.
• Or if you happen to be able to borrow a 16mm fisheye, with the custom 35mm lens adapter.
• Little Gray Box for controlling shutter.
• Use lab jacks to adjust tilt if necessary. Or one could use the optoliner cart from La Serena as a "tripod".
• Single channel readout, binned 4x4, 3-5 minute exposure

Sometimes it can be difficult to figure out exactly what you are looking at in the noisy dark images. In this case it helps to take a short exposure without moving the camera leaving a low light source on in the room. Use the smallest aperture and low light such as leaving the door open into the anteroom. Be careful not to be confused by reflections from the metallic surfaces or by the light emitted by the arcon you are using. Its data fibers need to be taped over and it needs to be wrapped in black cloth. To avoid spending all day in the elevator, get an assistant at the keyboard downstairs or setup a laptop on the network to provide a local terminal for controlling Arcon.

COMMISSIONING NOTES

The following notes were made during commissioning
and are left here for completeness
though some of the information is duplicated.

Spectrograph

• The lens cap and camera baffle/cover are necessary to avoid phosphorescence or image remnance effects in the CCD or its window. Values of ~5e-/pix/hr from the detector (window?) are easily achieved by exposing the detector to a flashlight briefly. I have not tested this aggressively since I didn't want to compromise the observer.
• The detector cap plus baffle are sufficient to make it safe to adjust the spectrograph using flashlight for illumination without risk of compromising the observing. The curtain gives an extra margin of safety for cryogen refills.
• However the Schmidt corrector (or surrounding cable?) is mildly phosphorescent. If hit with a flashlight will increase dark current to 3-10 e-/pix/hr decaying over an hour or so to negligible levels. The plan is to divide the camera baffle into two overlapping parts, one serving to protect the corrector from accidental illumination. [Done]
• The test has not been performed to see if doing flats is enough to cause significant phosphorescence by the corrector. I suspect this wont be a problem given based on my flashlight test: a series of images alternating between darks and flats should be done to test for this.
• The LEDs in the filter "windmill" which illuminate the slit were shown to turn off fully now that a relay had been installed.
• A conventional dark frame is an exposure with the shutter closed. Since the shutter is immediately in front of the spectrograph camera, the dark frame does not reveal light leaking into the optical path. A cap is needed over the slit to permit measurement of dark current plus light leaks. [Manually installed cap has been made] (Even when the dome lights are off, a significant amount of light comes down the fibers.)

The room

• Detector limited dark current is achieved whether or not the dome lights are on provided that the baffle is installed and provided that shutter is closed. (Detector uncapped and schmidt corrector not exposed to light for hours).
• If the slit is wrapped in Aluminum foil and black cloth, one can achieve detector limited dark current even with the shutter open, but only with done lights off. When dome lights are on the "dark current" increases about 10 fold. Hence there is a leak into the spectrograph room.
• Removing the baffle over the spectrograph camera increases "dark current". This test was done before I had determined that light leaked into the room from the dome so it is not clear whether the light blocked by the baffle was coming from the dome or the Arcon.
• The black tape which seals up the doors and various feedthroughs from the dome is actually translucent. The room needs to be imaged with Arcon plus camera lens needs to be used to image the room to see whether the black tape needs to be replaced or augmented.

Arcon

• The communication fibers produce a huge amount of light (~850nm) where they enter the Arcon front panel. A rough metal box with holes for the cables to pass though was made, but this requires large amounts of black tape and black cloth to seal it. A more precisely made cover is needed to provide an adequate seal while being easily de-mounted to provide access for maintenance.
• There are 6 LEDs inside the detector cap, controlled by the preflash parameter in the setdetector menu. These allow one to verify CCD performance. Since it is also useful to be able to check dark current with the cap on, the fact that there is leakage current through the LEDs when off is a problem. A transistor needs to be added to the LED drive circuit in the Arcon power supply to shunt the current past the LEDs when off. [In progress] This is preferable to a relay since the LED pulse must be very short.
• The white plastic diffuser in front of these LEDs was removed since it was found to phosphoresce (tens of e-/pix/hr) with seemingly infinite decay time. This effect is clearly distinct to the LED leakage current problem as it is present when the LED cable was unplugged.
• The dark current is seen to rise after the Nitrogen is refilled. This is because the addition of liquid Nitrogen to the pumped (solid) Nitrogen raises the temperature of the heat sink for the detector, for an hour or so, until the pumping (and consequent accelerated boil off) cools and re-solidified the Nitrogen. It should be noted that dark current is so critical in this application that the CCD is being operated at the lowest temperature achievable (about 150K): CCD temperature is not actively stabilized. The extra 16 K or so of cooling proved by pumped Nitrogen, is possibly due to poor thermal contact between the CCD and its supporting plate due to differential contraction between the Al plate and the stainless steel bolts which pass through it. Spring washers may be difficult ot retrofit due to the cramped space, and may be insufficient. Since the pumping and refill process are working well it has been decided to continue with this solution. Therefore, vacuum lines need to be routed to the large mechanical pump located in the corridor behind the telescope, instead of using the small pump currently located on the cass elevator platform directly under the telescope. [Done]
• Not related to darks but important: I need to figure out why the combination of gain=1 and Xbin=2 causes a strong pattern in the images.
   

November 2001

Knut Olsen, on behalf of the people who work on Hydra: Rolando Cantarutti, Rodolfo Cardemil, Manuel Martinez, Daniel Maturana, Andres Montane, Javier Rojas, Oscar Saa, Nick Suntzeff, Ricardo Venegas

The Hydra spectrograph and positioner have had a history of problems since its commissioning. In the months July-September 2001, good progress was made towards fixing the worst problems. Below are summaries of the problems we have encountered and the work done to fix them.

PROBLEMS W/ HYDRA-CTIO REPORTED THROUGH October 2001

POSITIONER:

1. Z-motion limit switches frequently tripped, loss of time ~0.5 hour per occurrence.
    
2. Frequent trips to the cage to remove fibers stuck in the gripper jaws or crossed with other fibers, loss of time ~0.5 hour per occurrence.
    
3. During June/July 2001 block, once per field GUI would report that it could not pick up a button, even though the TV camera showed the gripper to be positioned immediately above the button. Restarting the configuration would solve the problem, so minimal loss of time.
    
4. During early part of October 2001 block, many fields generated "transition" errors (sometimes several per field), which required parking fibers in the setup file and reconfiguring. Loss of time a few minutes per occurrence.
    
5. Prior to June/July 2001 block, frequent reports that guiding appeared to drive the guide stars to the edges of FOPS fibers. This problem may in the worst cases have reduced the efficiency by ~50%.
    
6. Occasional errors of "xy motion failure: stages not at destination". Restarting the configuration generally succeeded.
    
7. Many different versions of Hydra simulator and concentricities file on various mountain computers.

The problems with the positioner resulted in the loss of often several hours per night.

COMPARISON SYSTEM:

8. Frequent burnout of Ne penray lamp.
    
9. Penray line ratios not well balanced.
    
10. In echelle mode, many reports of lines being weak and/or difficult to identify. Nick noted that HeNe lamp contained neither He or Ne, but instead Ar.

BENCH SPECTROGRAPH:

11. Frequent problems with shutter moving to wrong position.
     
12. Many light sources in the coude room, as measured both through Hydra dark frames and images taken by Roger & Ramon with a cooled CCD camera placed in the room.
     
13. Region spanning ~100 pixels at one end of the CCD rolls off sharply in intensity, suggesting vignetting in the spectrograph.
     
14. Bias jumps of several ADU, sometimes occurring during night but most frequent right after dewar refill.
     
15. No RTD with 1x1 binning.
     
16. Solid N2 needed to cool CCD sufficiently, requires constant pumping.
     
17. Fiber mount can't be tilted far enough to straighten spectra in echelle mode.

FIXES AND MAINTENANCE

Many thanks to Sam Barden, Phil Massey, Dave Sawyer, and Bezhad Aredeshi for their advice on fixing Hydra's troubles.

POSITIONER:

Our top priority for Hydra before the October 2001 observing block was to reduce the failure rate of the gripper. Many of the problems experienced in previous runs, listed above, have now been fixed:

1. At first, the problem of the Z-limit switch tripping inadvertently was thought to mainly be due to a short circuit between closely packed wires inside the switch. In 2000, Javier Rojas and Esteban Parkes partially disassembled the switch, then wrapped and separated the wires to prevent their contact. After this work, the problem appeared solved, but then reappeared several days later. In April(?) 2001, Ricardo Venegas (and others?) noticed that the travel allowed by the switch in the positive Z direction was very small. Since their tuning of the switch to increase the range of possible Z motion, the problem has disappeared.
    
2. & 3. Early discussions of the problem of occasional crossed fibers came to the conclusion that the buttons were being gripped too tightly during transport; under normal operation, the gripper holds the button in a "relaxed grip" while moving fibers (aided by the magnets in each of the gripper jaws), only closing tightly while picking up or placing buttons on the plate. This conclusion appeared corroborated by the observation that fibers moved at large angles from the plate radius were bent into increasingly severe S-curves. The immediate solution was to limit the angle through which the fibers could bend to 3 - 5 degrees, which solved most, but not all, of the problems with crossed/colliding fibers.

However, with the appearance of another problem with the gripper (#3), it became clear that a complete overhaul of the gripper was necessary. Andres Montane (hardware) and Rolando Cantarruti (software) jointly took on this work, with Ricardo Venegas supporting.

[4]
Fig. 1. Close-up of gripper assembly

In order to uncover the cause of the gripper troubles, Andres & Co. needed to understand how the gripper behaves during the sequence of picking up a button, moving it, and putting it back onto the plate. What is supposed to happen? From Sam Barden:

• The gripper closes to sense that it does not have a button.
• The stages move over the position of the button. The gripper opens.
• The gripper is lowered over the button. The gripper closes.
• The gripper is raised to cruise height. A test is made to see if the button was picked up.
• The gripper moves to the relaxed position.
• After the stages have moved to the new position, the gripper goes to the closed position.
• The gripper lowers button to plate. Gripper jaws open.
• The gripper raises to cruise height. Gripper jaws close, to check that the button was released from the gripper jaws.

The gripper thus cycles through three states while placing a button: open, closed, and relaxed. (A fourth state, "open wide", is also available, but is used mainly when the gripper is initialized). How does the gripper achieve these states? In Fig. 1 above, the three teeth of the gripper are mounted on the black wheel with red LEDs attached. When this wheel rotates, the teeth tilt about their mounting pins and the jaw opens or closes. The motor which provides the power to rotate the wheel is located immediately above the label "C". "A"-"D" indicate four mechanical switches. From our observations of the gripper in April 2001 and conversation with Sam Barden, we established that:

• A is activated when the gripper is closed.
• B is activated when the gripper opens "wide".
• C adjusts the gripper tension for the relaxed position.
• D defines the open position of the gripper.

Andres&Co. independently established these functions of the switches. Their real insight, however, came from Andres' understanding of the mechanics of the system. Andres first observed that the turning of the gripper motor compresses the spring in switch C and opens the gripper. The closing of the gripper is then entirely driven by the stored tension in the spring. Second, given the distance by which the teeth are supposed to move in order to reach the open position, Andres calculated the distance through which the motor needs to turn. He found that the motor was stopping before it reached the full open position, held back by excessive force from the spring and by a spring in switch B. After he adjusted the positions of switches B and C, the gripper was able to complete the opening of the jaws. (Incidentally, Rolando discovered that the setting of switch B is never used in the software control, so that it actually serves no purpose). Third, Andres noted that when closing the gripper with a button in the jaw, the teeth continued to rotate even after they were fully closed around the gripper. By tuning switch A, he was able to stop the closing of the jaw before the teeth imparted rotation to the button.

Following Andres', Rolando's, and Ricardo's work on the gripper, the October observing block passed with no trips to the cage to untangle problems with the fibers, and virtually zero problems with the gripper. Their work leaves us with two valuable lessons:

• The gripper motor and the springs in limit switches B and C are in a delicate balance, and need to be looked at prior to each Hydra block
• Our earlier observation that fibers were often twisted into S-curves is likely not due to their failure to rotate with the jaw in its relaxed position, but instead to switch A being out of tune, which caused the buttons to rotate as the jaw closed

Andes will refine the process of tuning the gripper before the next Hydra block.

4. As an additional preventative measure to avoid crossed fibers, Rolando rewrote the Hydra positioning software to compute fiber trajectories as straight, rather than curved, lines. Observers who configured their fields with the old simulator thus experienced a number of "transition errors" when moves which were allowed by the old version of the software were found to be illegal by the new version. The errors were also seen in fields which were observed at significantly different sidereal times from the ones for which they were configured (as is to be expected). In fields with a number of transition errors, recovery was time-consuming, since the problem fibers had to be parked manually and the errors popped up one at time. Rolando has modified the software so that it checks for ALL problem fibers in a single step, reports the errors, and offers the option to leave the problem fibers parked while configuring the rest. This modification saves a great deal of time when transition errors are encountered. We are also taking steps to maintain the mountain computers with up-to-date simulators and to advertise the latest versions visibly.
    
5. In April 2001, Ricardo, Rolando, and Knut remeasured the centroids of the FOPS fibers and their boundary boxes. For many of the fibers, they found significantly different centroids from those used since the commissioning of Hydra. Fig. 2 demonstrates what happened. At some point, the guide camera was rotated with respect to the FOPS fiber set, causing the guider to guide off-center with respect to the guide star. Indeed, Sam Barden recalls having to rotate the camera slightly in order to mount it properly.

[5]
Fig. 2. Heads of arrows are current FOPS fiber positions, tails are positions before commissioning

6. This problem has been traced to a problem with the Galil box which controls the motion of the gripper stage. Unfortunately, the spare Galil box also has a problem which renders it unusable. The spare has been sent back to the manufacturer for repairs.
    
7. Software and concentricities files are now being kept up-to-date. See 4.

COMPARISON SYSTEM:

8. Javier has checked the Ne penray power supply, and found it supplies the same voltage as does the Xe. He thus finds it unlikely that the power supply is burning out the lamp. A more likely possibility is that the lamps have a gas leak. Needs more work. Investigating the comparison system will be given high priority during the next engineering run.
    
9. Nick and Knut will decide on what filters could be placed over the penrays to balance the strengths of their lines.
    
10. Javier and Ricardo have extensively tested the ThAr lamp. They have compared the effects of placing it where the Argus lamp used to sit and of using the Argus lamp power supply. Although some differences appear when using different power supplies, qualitatively the features do not change. The major difference is the much lower brightness of the Hydra system, but it is exactly as expected considering that the light is diffused over a 40 arcminute field. The HeNe lamp system in the box in the cage remains to be tested in similar fashion.

BENCH SPECTROGRAPH:

8. Prior to the October block, we experienced frequent problems with the SMC shutter control losing its fiducial, which would result in the shutter going to the wrong position. This would generally happen at the beginning of each observing block, although sometimes in the middle, and required Rodolfo Cardemil to adjust the system. Rodolfo has now disabled shutter control from the SMC, and Manuel Martinez has designed and installed a simple system using a motor with two mechanical positions. The new shutter system worked flawlessly throughout the October observing block.
    
9. Roger & Ramon have expended great energy in reducing stray light inside the Coude. They have imaged the room with a cooled CCD camera, and identified many light sources and light leaks in the room. As a result of their efforts, a number of steps have been taken to get the CCD dark current below 1 e-/pix/hour--see Hydra Dark Current II [6] for a summary. However, this dark current is only achieved at night with the fibers covered. More effort is needed to allow astronomers to measure the dark current during the day. In particular:

• Seal the two entrance doors to the Coude, as light leaks in underneath both.
• Continue imaging the room with the CCD camera, to identify the source of the light leaking in at roof level.
• Measure the impact of fluorescence from the corrector on nighttime observations.
• Replace the back cloth covering the spectrograph with non-translucent material.

8. Ricardo has examined the entire optical path through the spectrograph, but found no source of vignetting. It remains possible that the vignetting could arise in the CCD housing. An electronic effect has also not been completely ruled out.
    
9. After Ramon replaced the video card in the Arcon, the bias jumps have been seen less frequently during nighttime exposures. However, the jumps still occur immediately following dewar refills.
    
10. Marco has traced the lack of RTD with 1x1 binning as likely arising because of limited video memory in the Sun acquisition computer, which means we'll have to live with it.
     
11. Roger supposes that the need for solid N2 to keep the CCD's thermal current acceptably low arises from poor thermal contact in the CCD mounting. We have decided to pump the dewar constantly to maintain the low temperature rather than perform the risky undertaking of opening the dewar and identifying and correcting the problem. Originally, a small pump in a room next to the Coude was used. Before the October run, Oscar and the mountain staff rerouted pumping to a larger pump further from the Coude. Before this new system can be used, however, we need to install a valve to control the flow, and perhaps turn off pumping while filling the dewar. We found during the engineering run that LN2 poured into the flask ended up in the tube leading to the pump rather than inside the dewar; the tubing consequently froze and broke.
     
12. There is a software limit that prevents the fiber mount from tilting as far as it needs to in echelle mode. The limit was originally installed to prevent the fiber mount from colliding with the camera baffle. However, since the baffle has been modified, the limit is unnecessary. Rodolfo is working on this.

Page in Spanish

1. Dejar cerrados shutters, superior e inferior de camara de tv del gripper al estar instrumento fuera de telescopio. Abrirlas antes de montar el instrumento para su proximo uso.
    
2. Colocar tapas de cabezales de fibras, dos en instrumento y dos en caja de cass cage, al estar afuera (solo los de fibras de 300u). Retirarlas al conectar cables de fibras, previa limpieza con alcohol y poner optical coupling gel.
    
3. Al terminar un turno no olvidar cubrir la optica en el banco, tapar y extraer grating, para evitar el polvo que anda flotando en el ambiente del Coude.
    
4. Limpiar plato donde se sientan los imanes de las fibras. Lo ideal seria ayudarse con una aspiradora,  y/o alcohol.
    
5. Limpiar alrededores, sobre todo en rincones, donde van a dar suciedades.
    
6. Limpiar con tape y alcohol, las bases de los imanes.
    
7. Ejercitar movimientos de gripper.
    
8. Ejercitar movimiento de fibras. Llevarlas, manualmente, a la posicion stow (apegadas a semicirculo metalico). Llevarlas a posicion park, con programa.
    
9. Ejercitar movimiento de fibras entre posicion park y circulo.
    
10. Ejercitar movimientos de fibras entre posicion park y alguna configuracion de campo.
     
11. Medir concentricidades de las fibras, o sea chequear su zero point.
     
12. En futuro lejano: Calibrar posiciones de fibras en ambas direcciones, utilizando la recien aparecida grilla con cruces, y en varias posiciones. Esto habria que madurarlo y ver como funciona aqui, con el Hydra. (Para futuro, cuando se nos acabe el material para hacer adobes).

Detector, dewar and Arcon work, Ramon Galvez H., June 2002

CCD Dewar

1. Leak tested.
    
2. Determine gasification rate = 1.8 SCFH
    
3. Determine As Is Thermal Radiation Load = 3.8 Watts
    
4. New vacuum valve installed.

CCD & Arcon

1. Dark Current = 1.43 e/pix/hr ; Conditions = Bin 2x2, Gain 1, all sorrounding rooms and dome lights ON.
    
2. Dark Current = 0.88 e/pix/hr ; Conditions = Bin 2x2, Gain 1, all lights OFF.
    
3. New ADC bd prepared + tested ( s/n4 new batch). Got no more missing codes around 15k ADU.
    
4. Video original and spare bd. Noise comparison : both at 3e.
    
5. Not a single crash. Neither during exposure tests nor while downloading.

Dewar Cryogenics

1. A new LN2 refill and solidification device was designed. I had the good advices from Brooke on this. Tested in the lab and, along with Daniel, on the telescope.
    
2. Cryo Status : 24hrs hold time, one daily refill. Cryo system ‘tuned’; as a result the refill is done without necessity of vacuum valve control.

More detector, dewar and Arcon work - Ramon Galvez , August 16th, 2002

Done:

1. Tested the new macro thresholds set for cryo alarms. Working ok.
    
2. With Rolo C. we made changes in the Hydra option of ‘SetFiles’. The waveforms and sequencer code are automatically installed now when Hydra + Arcon3.7 are selected.
    
3. Tests were done also when switching back and forth for other ccds.
    
4. With Ricardo V., we smoothly went through the LN2 solidification using the new device:

Normal operation vac sensors readouts

To be done after this run (minor pending details) :

1. Improve the ‘solid N2’ valve setpoint lock device.
    
2. Install the new adjustable security ‘pop-up’ pressure valve.

1. CCD dewar seems to require many refills before reaching correct temperature. Vacuum indicator appears normal, but perhaps we don't have a fine enough instrument to indicate that there is outgassing.

   Ramon will check out the vacuum and prepare a new molecular sieve during the week of 21/3
    

2. Observers reported problems with the rotation for their fields near the pole (see email from Peter Frinchaboy and Bill Kunkel [7]). The Hydra rotation was measured on the engineering night at the equator, but x_offset and y_offset were left at 0.

   On the next engineering night, we will measure the rotation near the equator and adjust x_offset and y_offset by going to the pole.

   Rolando is looking at exactly how big the axial alignment error is.

   Can Roberto check the optical alignment soon?
    

3. Minor software problem: Locked FOPS 120 always activated when turn on guider.

   Rolando will add a fix to the code

4. Lots of trouble setting up echelle, caused basically by very weak and hard to identify ThAr lines and the fact that observers were operating in UV

   First, Hernan, Ricardo, Knut, Daniel, and Patricio will write up a detailed set of procedures, hints, and tricks for doing the echelle setup

   We want to investigate getting replacement ThAr lamps--Oscar will contact Kitt Peak to find out what they are using

    

5. The Ar penray is actually Hg, the Ne penray appears weak

   Observer support will check out what is currently in the chimney.

   Oscar and Knut will contact Kitt Peak about what lamps they have and who supplies them

   Hernan, Oscar, Ricardo, Knut, Patricio will also make a web page listing all of our comparison lamp supplies

    

6. Some gripper problems: short circuit in z-limit switch, and one instance where the status of the limit switches (seen through swchart and inputchart commands) were changing erratically

   Andres and Ricardo will check out the z-limit switch and the general mechanical tuning of the gripper during the week of 21/3. Andres will also see whether a new z-limit switch design may be implemented before April 16, 2004.

   The trouble with the erratic limit switch signals was solved by replacing the spare Galil with the original Galil. Trouble is that the original Galil had some problems of its own, such as getting lost and producing "x-y stage not at destination" errors. The Galil boxes have been sent back to the company twice for repair, but have been returned with the statement that there is nothing wrong with them. Gale will ask Galil for a quote on a new replacement.
    

7. At the start of the Hydra block, the guider parameters were set incorrectly

   FIXED after David changed the values and Rolando incorporated the fix into automated software.

Hydra Status Report, Notes from the meeting April 2000 - Updates 2002, 2003

Knut Olsen, 4/3/2002

Updated 5/24/2002

Updated 7/14/2003

People and jobs assigned, in order of priority. See list below for job descriptions. Updates in light blue.

• Andres: 10, 17, 3
• Daniel: 3, 15, 16
• Javier: 1, 4, 5
• Knut: 3, 18, 20
• Manuel M.: 17
• Nick: 14, 20
• Oscar: 2, 14 (started), 19, 20
• Ramon: 11, 12, 13
• Ricardo: 3, 15, 16
• Rodolfo C.: 17
• Rolando: 4, 9, 7
• Tim: 2, 14
• Everyone: 1, 2, 6

SAFETY ISSUES:

1. A UPS power supply mounted on Hydra has failed. A consequence of this is that the red "panic button" is no longer functional. Moreover, when the Hydra carriage is dropped, the gripper is still able to move. Javier will notify us of the part that we have to re-order. Until it is installed, nobody should work on the gripper without Javier present. Javier points out: dropping the carriage or hitting the red button will, in fact, halt the gripper. However, this is done by activating the interlocks and applying the brakes to the gripper, rather than by cutting power. One might thus imagine that the gripper could move if the command to apply the brakes somehow fails. A redesign of the circuit is needed to have power cut. Done (7/2003) 
    
2. With the room being sealed against light leaks, it is potentially a fatal hazard if nitrogen, or another toxic substance, were spilled in the room. The entrance door should always be left open while filling the nitrogen flask. Tim, Oscar, and Ramon will locate and install a nitrogen detector in the room, to provide warning should nitrogen levels increase (and oxygen decrease) inside. (4/2003) Daniel's  reports installed: See Daniel's email [8].

POSITIONER:

3. A summary of the maintenance work to be performed regularly between observing blocks is needed. Will be done jointly by Ricardo, Daniel, Andres (gripper mechanics), and Knut. See the latest Hydra maintenance list. [9]
    
4. We saw several errors of "x-y stage not at destination". This problem has been traced to the Galil box. The spare box has been returned from Galil, but is still not functioning properly. Javier and Rolando have a good idea of which part is faulty, and are continuing to work on isolating the problem. While this work continues, the gripper and its associated electronics should not be touched without the knowledge of Javier and Rolando. FIXED. Problem was traced to bad EPROMs in the Galil boxes, have been reprogrammed. See Javier's e-mail [10] summarizing the work.
    
5. Most of one night was lost when the gripper was unable to move. The reported error was a tripped X limit switch. Javier was able to trace the problem to a faulty optocouple inside the Galil. Javier has temporarily fixed the problem, but a new part needs to be ordered. FIXED, see 4.
    
6. Fiber 74 was broken after it was misidentified as fiber 7 through the manual command "ThisIs 7". To prevent this from happening again, we should always use the command "thisis" instead of "ThisIs fiber_number" when recovering a lost fiber.
    
7. When Hydra is put on the telescope, the stowed positions of the fibers change slightly. This appears to be the result of a slight flexure between Hydra's off-telescope and on-telescope mounts. Rolando will correct the problem in software by implementing a lookup table to adjust the stowed positions. Done. Remember to also measure rotation of Hydra field at night.
    
8. After recovering from an "x-y stage not at destination" or similar error, we saw several instances in which the gripper would remove a fiber from the field, then replace it again immediately. Rolando describes this as a software "feature", which would require rewriting significant amounts of the Hydra software to fix. Not a high priority at the moment. Done. Rolando's new software is a huge improvement.

BENCH SPECTROGRAPH:

9. At the start of the run, we had a problem with the sequencer code and a capability missing from SetFiles. A temporary fix was put in place by Ramon with long-distance help from Marco Bonati. The problem appears to have been caused by unaccounted dependencies in a recent Arcon software upgrade. Rolando and Francisco are working together to implement a correction to SetFiles. Done.
    
10. Ramon and Javier had trouble with CommNode errors at the start of the run. The communication fibers apparently get bent at their entry point(?), and need a redesigned cover to prevent this from happening. Andres will look at it. Done.
     
11. One of Arcon's testpoint voltages was out of the allowed range, and produced frequent error messages. Ramon did not have a sufficient block of time to isolate the problem. As the images appeared to be unaffected, the problem was left alone. To work on the problem, Ramon needs to be able to illuminate the fibers while Hydra is off the telescope. Uncapping the fiber plug and using the dome lights will probably do the trick.
     
12. We saw occasional problems with a misbehaving bit somewhere in the Arcon acquisition system, which produced images that looked like they had bands of bad pixels. Ramon discovered that redoing the video calibration in Arcon fixed the problem temporarily. He is working on tracking down the cause of the problem. Done.
     
13. Ramon is continuing to work on why Hydra's original video board produces bias jumps of several ADU. It would be nice to have the board working, both to have a reliable backup and because the read noise was slightly lower with it installed.
     
14. Ramon and Knut spent the morning of the engineering night hunting for light leaks in the bench room. They found several, which they plugged with black tape. Oscar, Nick, and Tim are in charge of overseeing the significant safety issue (both to people and to the optical surfaces) involved with plugging the leaks permanently with black silicone. See also 16. Leaks plugged and room cleaned, awaiting reimaging to verify success. Done (7/2003).
     
15. Vacuum pumping of the nitrogen flask (necessary so as to make the nitrogen solid) continues to be a headache. While it would be wonderful not to have to continually pump the dewar, improving the thermal contact would likely need a complete redesign of the system. The limiting factor is undoubtedly the small volume and large surface area of the tube joining the CCD and the dewar; increasing the volume of this tube to improve the cooling would increase the obscuration of the beam and lower the spectrograph throughput. We thus need to become accustomed to using the vaccum system. Ricardo has written instructions for using the new, larger pump. The smaller pump should be used only as a backup. Ramon and Brooke have designed and fabricated new plumbing for the nitrogen filling and pumping unit, will be installed in May.
     
16. The bench spectrograph room is getting dirty, and the optical surfaces are suffering. The aluminum coating on the reflective mirror looks like it is degrading, and will probably need to realuminized. The glass camera corrector is very dirty.
    Part of the problem is that every time the curtains are opened, small fibers are released. If we can plug the remaining light leaks and seal all but one of the doors, then we could remove the curtains. Plugging the leaks completely is preferable to replacing the curtains with a solid structure, since this would limit access to the spectrograph. Sealing the room unfortunately brings a significant safety risk, if nitrogen or another toxic substance is spilled in the room, or if a fire consumes the oxygen inside. Room has been cleaned, but removal of curtains will wait.
     
17. The fiber mount doesn't reach high enough inclinations to provide compensation for the azimuthal angle of the echelle grating, because the encoder runs out of range. Andres will design an extension for the for the inclination encoder (time: 2days - 1 week), Manuel Martinez will connect the new encoder to the Smart Motor Controller (time: 1 day?), and Rodolfo will modify the SMC software to accept the new set of encoder values (time: 2 weeks). This will be done in May 2002. Mechanical work finished, Rodolfo working hard on software. Done (7/2003).
     
18. /ua13 data disk is filling up, Knut will track down a second disk. Adding another disk to ctioa0, an ancient SunOS machine, is difficult and was decided is too risky. Ron and Eduardo proceeded to repartition one of a0's existing disk to create a larger block of space, but this interrupted Ramon's progress with the CCD and has been put aside.

CALIBRATION SYSTEM:

19. The ThAr lamp was found burned out in March. Daniel replaced it, and reports that we have one spare remaining. Oscar will look into buying more. Daniel turned up three spare lamps, although two need modified housings.
     
20. Nick and Knut decided that a BG38 filter placed over the Ar penray lamp and ND1 filters placed over the He and Ne penrays would give balance to the penray wavelength calibration spectra. Oscar thinks that sleeves exist, but we have to make sure that the lamps won't overheat. We should test it off the telescope before installing the sleeves permanently on the chimney penrays.

Here is a quick summary of the topics discussed at the Hydra meeting on August 23, 2004, and the conclusions we reached:

1. Motorizing the bench camera focus

There are two basic solutions. One is to buy a motor and external encoder and use the existing, now-defunct SMC connection for the shutter operation to operate the motor. May be a significant mechanical design effort, however. The second is to replace the existing manual micrometer with a motorized version. This is mechanically easy, but communication with the SMC requires study.

--> Andres, Manuel, Enrique, Javier, Ricardo, Rodolfo will study the problem

2. Plans for the pellicle

Need to ensure that the pellicle may be replaced without damage.

--> Ricardo and Andres will discuss

3. Z-limit switch issues

Several apparent z-limit switch failures even after redesign and installation by Andres.

--> Javier and Enrique will try to reproduce the problem on the next turno, and write a proposal for a solution

4. What spare parts do we need?

--> send Knut any items you can think of

5. GNATS 457+500+503, 499+502, 320

The first three refer to the z-axis failures referred to above.

The next two refer to failure of the guider, the solution of which was reported in #502. But we need either spare VME cards, or perhaps Rolando's software patch.

--> Enrique and Javier wil discuss with Rolando

The last (320) refers to the broken echelle filter #6.

--> Knut will discuss finding a replacement with Roberto

6. Hydra autofocus

Exists, but doesn't work. Why?

--> Knut will look into it.

By Charles Corson, February 2010.