So, you found something in the Hubble archive you want to process…now what?
Click here for part one
Herein I explain a little more about getting files and what to expect those files to contain aside from the obvious, which are glorious pictures of our universe. Yeah, Hubble is awesome, but it takes a lot of work to go from raw science data to a pretty picture for all to wonder at. Knowing what you are looking at and becoming adept at identifying artifacts and defects will allow you to more quickly process an image without worrying if you’re deleting some real part of a galaxy or star. Moreover, thinking you may have discovered something amazing and new that turns out to be a simple artifact can be disappointing.
Know which files to download
There are a few ways to go wrong when downloading your data. Well, not seriously wrong, but not quite optimal, either.
The two obvious options are the FITS-Science and FITS-MEF. From the cart, you can tell these two apart because the FITS-Science filename ends in _sci and the FITS-MEF (multi-extension FITS) ends in _drz. Both contain an identical image in the first plane. The FITS-MEF contains two additional image planes, one containing a bunch of cosmic rays and other oddities, the other a weight file. I personally have found no use for either of those, so I always choose the FITS-Science file, which can be found under the thumbnails when browsing results from the Image tab.
The big advantage to this is that the FITS-Science file is about one-third of the FITS-MEF file in size. If you have a sluggish internet connection or bandwidth caps, the FITS-Science file is the best because it is the smallest. If you are curious about the extra data contained in the FITS-MEF file, you should definitely take a look at least once. They may contain useful information for more experienced processors and amateur astronomers.
Learn how to use DADS
At some point, you may find that a particular dataset is not immediately available for download from the cart. Some of these are proprietary, which means it’s been less than a year since they were observed, so they’re locked away from the public until that year has elapsed. Others require a DADS (Data Archive and Distribution System) request, which requires some extra work, time, and knowing how to use FTP (File Transfer Protocol).
If you’ve never done any FTPing before, this may be frustrating. I promise it’s not that hard, though. I learned how to do it when I was twelve because I wanted to download MP3s back before people were getting in trouble for that sort of thing. Anyway, it’s a worthwhile skill for any avid user of the Internet. Embrace your inner nerd.
- Add the DADS request to the cart
- Go to the cart, and click the Submit DADS Request button. A pop-up window should appear.
- Note that if you have a pop-up blocker, you may have to add an exception
- As it says, please click Submit Query
- Type your email address into the Email box, it’s light green
- Select DRC (best*) or DRZ (next best) extensions at the lower right. This will ensure only the FITS files you need are retrieved.
- Click the Send retrieval request to ST-DADS button.
- Check your email and wait. Depending on how busy the service is, it could take a while, 15 minutes is typical. I’ve waited hours, though.
- While you’re waiting, get your FTP program of choice set up. I use Filezilla. Here is a handy guide for new users.
- FTP anonymously to archive.stsci.edu and enter the /stage/anonymous directory.
- Your files will be located in the directory that was emailed to you, eg anonymous12345
* Why DRC? This is a drizzled image with an extra step in the pipeline to try to get rid of the long, comet-like tails seen near bright sources. Amazing! See THE NEW CALACS: Putting electrons back where they belong
How to deal with chip gaps
Credit: NASA/ESA and the ACS Science Team
Both the ACS/WFC and WFC3/UVIS have chip gaps. These nefarious little things have caused me more grief than necessary.
What is a chip gap?
Nothing more than a physical gap present between the CCDs. You can see in the image it is actually fairly small; presumably the smallest distance apart that the two CCDs could be. I read some design paper suggesting the gap could be used as a kind of occulting bar for bright sources, but I don’t know how often it is actually used like that.
What should you do about it?
I used to agonize over what to do with it, wondering if the engineer who came up with this imagery torture device had any idea what suffering they had caused me to endure. These days, I’ve learned to accept its inevitability. If there is any old data lying around to fill it in, even if it is not the same filter, I will drop that in there. If not, I clone nearby data into it. Either way, I leave a note in the description so that anyone looking is aware that I did this.
Another thing I do is to not make it perfect. I leave it slightly obvious that there is something going on there within the image so that if anyone examines an image closely enough, they know something is up even if they don’t read the description.
What will you do with the chip gap? It’s entirely up to you. Leave it, crop it, collapse it, or fill it with cloned data. Just don’t let it stop you.
Working with artifacts and other little gremlins
An astute processor will deal with as many of these as possible. Not only will doing so make one’s image more aesthetically pleasing, it will also greatly reduce the confusion any viewers may experience. The universe itself is challenging enough to understand, so getting caught up on some odd little thing the instrument left behind can be an impediment to learning and to the enjoyment of astronomical imagery.
A paper which goes into more technical detail than I do is available here: ACS CCD Image Anomalies in the Hubble Legacy Archive (pdf)
Cosmic rays
As it turns out, a lot of the things Hubble captures on its CCDs are interstellar photo bombers known as cosmic rays, or CRs for short. It’s hard to say where exactly they come from, but these highly energetic particles are constantly striking Space Telescope. If they land in just the right spot, a shower of secondary particles come crashing down onto our hapless detectors, and this ends up looking something like a constant snow of white specks and lines. A dataset with any particular observation composed of three or less exposures will have some CRs to deal with.
Cleaning CRs can be time-consuming. It is easy enough for the human brain to discern cosmic rays from stars, but more difficult for computers to detect this pattern. When you’re putting your image together, CRs will turn up in very bright colors. One way to get rid of them is by selecting pixels of the purest colors and then running a filter such as Photoshop’s Dust & Scratches over that selection to get rid of them. I have found that this invariably results on quite a few false positives since some objects are also very red or very blue. I don’t like blotting out real stars, so I have spent many hours masking off CRs by hand with nothing but my Wacom tablet and some nice music to help me.
If you end up with an image with a lot of CRs, please at least attempt to do something about them. Cosmic rays are one of the most confusing things for lay folk to interpret. They have no idea what they are, so they will tend to assume wrongly that they are stars and background galaxies at best or something like aliens at worst. Besides the confusion they cause, they’re also absurdly ugly. Don’t be that person who leaves them in the image!
Optical ghosts
There are many different types of ghosts or undesirable optical artifacts which appear as a natural result of the behavior of light passing through a telescope. These come in a few different shapes and sizes, but always present themselves when a bright source is in the field. Sometimes they look like donuts, other times like infinity symbols or figure eights. Some others may look smeared or like blobs. Many of them are faint and not noticeable, but the brighter ones can be difficult to deal with.
A. A tiny but infamous ghost in the original WFPC2 rendition of Hubble’s Eagle Nebula by Jeff Hester and Paul Scowen (Arizona State University), and NASA/ESA
B. Typical annular or circular feature next to a bright source
C. More annular features stretched out across the detector
Why deal with them?
Once again, if not for aesthetics, then at least for the sake of anyone who might be confused.
I once watched a rather banal, conspiratorial video created by someone who thought a large annular feature caused by an internal reflection was a solar prominence. This is a novice mistake, but one even people who don’t think NASA is hiding aliens can make.
Every star in the galaxy—even the biggest, closest stars—occupy less than a fraction of a single pixel on Hubble’s detectors. This is very important to remember. You’re never actually seeing stars. You’re seeing their light spread out on a detector. (And no, the Sun doesn’t count. If Hubble looked at old Sol, its detectors would be instantly damaged, probably beyond further use.)
Charge bleeds
In the unfortunate event that the CCD becomes saturated, electrons start to spill out up and down the detector forming something referred to in the instrument handbooks as charge bleed. They are frequently referred to as blooming by astrophotographers, but I like the term charge bleed because it is less ambiguous and is exactly what is happening. Blooming can also more generally refer to a kind of hazy glow around a bright light source for any sort of photography.
Anyway, these are easy to take care of, and I use two methods.
The first was originally detailed to me by Lisa Frattare and Zolt Levay in a useful handout (pdf) given during some online tutorial sessions. It involves taking data perpendicular to the charge bleed, copying it to a new layer, rotating it 90°, and then using it to cover the bleed up. Some of the more egregious bleeds are so large and cause such a loss of data that an alternate source, such as a ground-based observatory, may be necessary if available. An example of this method is presented here (pdf), in the form of an AAS (American Astronomical Society) poster. You can also see some interesting tidbits about what went into the making of the giant Carina mosaic.
A second method, which I use sparingly, is useful for scenes rich in stars with many minor bleeds only a few pixels long. For those, I have found that one may simply apply a blur to the bleeds, effectively camouflaging them against the star’s point spread function.
The occulting finger of the ACS/HRC
Not a major factor in most images, but it will make an appearance from time to time. The occulting finger is sometimes used to block out bright sources so that nearby dim sources can be observed. Unfortunately, the finger is also stationary, so it’s there whether it’s being used or not. Depending on the observing strategy, sometimes the finger can be made to disappear partially or fully. Other times, it will simply exist as an area of lost data that you’ll either have to cover up with cloned data or leave it there. I personally tend to leave it there, but I do make sure to fill it with a color that closely matches its surroundings so that it is less distracting. Whatever you choose, it is good practice to explain what it is and what you’ve done with it, just in case anyone is wondering.
B. Partial data coverage, lots of cosmic rays left behind
C. Complete coverage using dithering strategy, only a small notch left behind.
Some WFC3/IR idiosyncrasies
If you delve into infrared observations, some new and interesting cosmetic features are immediately apparent. Some of them have been given quaint names. One hard, circular defect was dubbed the Death Star. Another somewhat softer one resembling its namesake has been called Wagon Wheel. Numerous other blemishes dubbed blobs began appearing after the detector went into orbit. The number of blobs has increased over the years, and they are caused by small particles of dirt landing on the Channel Select Mechanism (CSM) mirror. A paper describing the discovery of the nature of the blobs is here: The WFC3 IR “Blobs” (pdf)
Many of these defects are overcome during observation by dithering, which means that several observations are made slightly offset from one another. This helps ensure that any data loss gets recovered through redundancy. Hubble time is very valuable, however, and not everyone gets enough time to overcome these difficulties. In many cases we all have to deal with it as well as we can.
Right: A piece of Horsehead mosaic at the same orientation illustrates defects and hard blobs automatically dropped out by the pipeline.
This is not a comprehensive list!
Many other artifacts and defects may be found in Hubble’s archive. Most or all of them have been described in one of the instrument handbooks somewhere. If you find something you are unfamiliar with, try searching the handbook. You can also ask in the comments here and I may be able to help find an answer for you.
Good luck!
Confused?
Read part one of my Hubble Tutorial Series: How to find Hubble Data.
Add Comment