Create master flats

[detoma]

The flats in KCor (and COMP) are taken with the occulter in. Science data and flats do not have the flats necessarily in the same position as the occulter moves over the course of the day. As a result, there is a region of pixels around the occulter for which we cannot performa a proper flat-fielding. I think an average flat value is used for that area in the KCor pipeline.

A way to improve this and avoid artifacts is to create a master flat. The idea is that by combining many flats together which do not have the occulter in the same location, we can cover a larger area of the detector and can flat-field every pixels in a science image.

The way to do this is to use all the good flats available for each cal_epoch and create a 3D array. Flats must be dark-corrected and scaled using vdimref. We will then compute the median flat value for each pixel in the detector that is not under the occulter using all flats. We can also compute the sigma to test flat stability over time.

Since flats tend to be taken around the same time every day, I am not sure that the occulter moves enough from one flat to the next to cover all the area of the detector that we need. I will take a look at the period 20220316.103744 to 20220830.123347 to test this. If the flats we already have do not cover enough area of the detector, when we reopen MLSO, we can take flats at different times of the day to acquire the data we need. Another option is to take flats without the occulter, which I think it is possible for KCor, but stray light could be different in the two configurations, so we need to test how they compare.

The project is pretty simple. The time consuming part is to collect the flats and darks to process for each epoch.

[detoma]

Joan can you please double check if the steps below look right to you to create a master flat:

• stay within a calibration epoch (we have identified a long one in 2022 for testing)
• loop through all the flats
  1. remove dark
  2. divide by the SGS vdimref value at the time of the flat
  3. identify the occulter and mark it as NaN
  4. store flats into a 3D array
• use the 3D array to compute the median and sigma for each pixel in the detector excluding values that are NaN.
• store the median, sigma, and number of images used for each pixel in an array

Is there any other correction, e.g. camera non-linearity correction, that I missed?

If this works, since the flats are already normalized by vdimref, when we apply it to the dark corrected data in the crescent moon, we simply need to divide those pixels by vdimref at the time of the image and for the master flat.

[mgalloy]

Here is a better histogram of when the KCor flats (cover="out" and darkshut="out" and diffuser="in" and calpol="out") were taken:

Found using the query:

select date_obs, cover, darkshut, diffuser, calpol from kcor_cal where cover="out" 
  and darkshut="out" and diffuser="in" and calpol="out" order by date_obs;

[detoma]

Thank you Mike for the histogram. This is encouraging. We may have enough images to cover more of the detector.

[mgalloy]

Only use the master flat for the pixels in the "crescent moon", i.e., the difference in the change in the location of the occulters of the flat vs. the science image.

[detoma]

Add a label that records the number of pixels in the "crescent moon" so we know how many pixels do not have the closest flat.

[mgalloy]

I can combined gains in a cal epoch. Below, I combine the gains from the cal files I had available from cal epoch 10 (bold indicates the 5 latest versions of the files that were used):

• 20150319_183957_kcor_cal_v10_2.1.11-dev_1.000ms.ncdf
• 20150319_183957_kcor_cal_v10_2.1.26-dev_1.000ms.ncdf
• 20150428_193507_kcor_cal_v10_2.1.11-dev_1.000ms.ncdf
• 20150428_193507_kcor_cal_v10_2.1.26-dev_1.000ms.ncdf
• 20150622_195045_kcor_cal_v10_2.0.62_1.000ms.ncdf
• 20150622_195045_kcor_cal_v10_2.1.13-dev_1.000ms.ncdf
• 20150622_195045_kcor_cal_v10_2.1.26-dev_1.000ms.ncdf
• 20160612_190014_kcor_cal_v10_2.1.10-dev_1.000ms.ncdf
• 20160612_190014_kcor_cal_v10_2.1.26-dev_1.000ms.ncdf
• 20160618_181550_kcor_cal_v10_2.0.64_1.000ms.ncdf
• 20160618_181550_kcor_cal_v10_2.1.26-dev_1.000ms.ncdf

The number of images used by pixel:

The master gain:

And the standard deviation:

From Joan:
I checked the calibration files above. They are all comprised of only 1 calibration sequence per day that is collected in just over 14 minutes:

20150319 18:39:57 to 18:52:05
20150428 19:35:07 to 19:49:17
20150622 19:50:45 to 20:04:55
20160612 19:00:14 to 19:14:23
20160618 18:15:50 to 18:29:59

The biggest motion is coming from the moving camera stages. These were replaced Jun 18-19, 2015.

[mgalloy]

The above were for RCAM. The corresponding TCAM images are below, starting with number of images per pixel:

The TCAM master gain:

And the TCAM std deviation:

[detoma]

The master gain does not look right. Something went wrong. Let's go over the logic tomorrow, we are doing something wrong.

update from January 21 2025 KCor discussion:
I suggested to over occult by 1 pixel to make sure that we do not label as good a pixel that is partly under the occulter. This is to account for uncertainty in the occulter edge.

Joan also noted that we are currently averaging flats taken the same day even if they are hours apart. This is source of a separate problem that also needs to be addressed.

[jburkepile]

Mike - it would be helpful if we had access to the master fits array so we can look at the numbers ourselves. Do you have a master flat on one of the MLSO servers that we can look at?
thanks!

[mgalloy]

It's kcor_master_gain_v10_2.1.26-dev.ncdf in calib_files.latest on our normal data server.

[mgalloy]

Try to overoccult by 1 pixel.