diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 21779d9..547b827 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -43,4 +43,4 @@ jobs: LTD_PASSWORD: ${{ secrets.LTD_PASSWORD }} LTD_USERNAME: ${{ secrets.LTD_USERNAME }} run: | - lander --upload --pdf RTN-117.pdf --ltd-product rtn-117 --title "Image Calibration and Instrument Signal Removal for the First Year of Rubin LSSTCam" --handle "RTN-117" --lsstdoc "RTN-117.tex" + lander --upload --pdf RTN-117.pdf --ltd-product rtn-117 --title "Image Calibration and Instrument Signature Removal for the First Year of Rubin LSSTCam" --handle "RTN-117" --lsstdoc "RTN-117.tex" diff --git a/README.rst b/README.rst index 968905d..de7a577 100644 --- a/README.rst +++ b/README.rst @@ -4,7 +4,7 @@ :target: https://github.com/lsst/rtn-117/actions/ ################################################################################### -Image Calibration and Instrument Signal Removal for the First Year of Rubin LSSTCam +Image Calibration and Instrument Signature Removal for the First Year of Rubin LSSTCam ################################################################################### RTN-117 diff --git a/RTN-117.tex b/RTN-117.tex index 7ff93a2..8de1ecc 100644 --- a/RTN-117.tex +++ b/RTN-117.tex @@ -55,12 +55,12 @@ \input{authors} \date{\today} \doi{10.71929/rubin/3488007} -\title{Image Calibration and Instrument Signal Removal for the First Year of the LSST} +\title{Image Calibration and Instrument Signature Removal for the First Year of the LSST} % This can write metadata into the PDF. % Update keywords and author information as necessary. \hypersetup{ - pdftitle={Image Calibration and Instrument Signal Removal for the First Year of the LSST}, + pdftitle={Image Calibration and Instrument Signature Removal for the First Year of the LSST}, pdfauthor={broughtona}, pdfkeywords={} } diff --git a/abstract.tex b/abstract.tex index 1939aa3..273f8b1 100644 --- a/abstract.tex +++ b/abstract.tex @@ -1,9 +1,9 @@ \begin{abstract} -The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. +The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2-gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. -We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signal removal (ISR) data processing pipelines for DP2.\todo{Quote the headline numbers in the abstract itself (the two or three key residual-systematics figures and the dataset size, e.g.\ deg$^2$ and number of visits) rather than only promising that metrics are shown.} +We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1). \end{abstract} diff --git a/body.tex b/body.tex index 9a4bdba..4f34f53 100644 --- a/body.tex +++ b/body.tex @@ -1,20 +1,16 @@ -\input{sections/1_intro} +\input{sections/intro} -\input{sections/2_lsstcam} +\input{sections/lsstcam} -\input{sections/3_data} +\input{sections/data} -\input{sections/4_processing/41_isr-science} -\input{sections/4_processing/42_calibration-production-and-application} +\input{sections/processing} +\input{sections/discussion} -\input{sections/5_discussion} +\input{sections/verification-certification} - -\input{sections/6_verification-certification} - - -\input{sections/7_conclusion} +\input{sections/conclusion} diff --git a/figures/calib-dependency.pdf b/figures/calib-dependency.pdf index 178939a..9de683c 100644 Binary files a/figures/calib-dependency.pdf and b/figures/calib-dependency.pdf differ diff --git a/figures/calibration-epochs-timeline.pdf b/figures/calibration-epochs-timeline.pdf index 3ffc20d..e85c859 100644 Binary files a/figures/calibration-epochs-timeline.pdf and b/figures/calibration-epochs-timeline.pdf differ diff --git a/figures/calibration_boxes_detector_model.pdf b/figures/calibration_boxes_detector_model.pdf index 8779a8f..e3881c2 100644 Binary files a/figures/calibration_boxes_detector_model.pdf and b/figures/calibration_boxes_detector_model.pdf differ diff --git a/figures/isr-pipeline.pdf b/figures/isr-pipeline.pdf index 7e9ea47..42a7c21 100644 Binary files a/figures/isr-pipeline.pdf and b/figures/isr-pipeline.pdf differ diff --git a/figures/lsstcam_focal_plane.pdf b/figures/lsstcam_focal_plane.pdf new file mode 100644 index 0000000..bd888cd Binary files /dev/null and b/figures/lsstcam_focal_plane.pdf differ diff --git a/figures/src/to-make-calibration-boxes-detector-model.tex b/figures/src/to-make-calibration-boxes-detector-model.tex index b9565f9..42b1e55 100644 --- a/figures/src/to-make-calibration-boxes-detector-model.tex +++ b/figures/src/to-make-calibration-boxes-detector-model.tex @@ -5,7 +5,7 @@ % to-make-calibration-boxes-detector-model.tex) % % Text is left at the document's default \normalsize so that, once placed -% in the two-column figure* in sections/2_lsstcam.tex, labels read at the +% in the two-column figure* in sections/lsstcam.tex, labels read at the % same size as the body text. \documentclass[tikz, border=4pt]{standalone} \usepackage{times} @@ -58,8 +58,12 @@ \node[gaincap, text width=2.0cm, opacity=0, text opacity=0] (g1box) at (g1-home) {Gain, $g_1$\\(On-chip)\\($\sim5\,\mu$V/e$^-$)}; % Brighter-Fatter and Dark Current as separate, floating, disconnected boxes. -\node[effect, text width=2.35cm] (bf) at (0.34,0.65) {Brighter-Fatter Effect}; -\node[effect, text width=2.35cm] (dc) at (0.34,-0.40) {Dark Current}; +% Sensor (pixel-array) effects, stacked inside the amplifier segment and +% centred between its title and the serial register. +\node[effect, text width=2.35cm, anchor=north] (bf) at (0.34,1.72) {Brighter-Fatter Effect}; +\node[effect, text width=2.35cm, below=0.16cm of bf] (tr) {Tree Rings}; +\node[effect, text width=2.35cm, below=0.16cm of tr] (fr) {Fringing}; +\node[effect, text width=2.35cm, below=0.16cm of fr] (dc) {Dark Current}; % Pin keeps the detector outline where it was when this box sat below the amplifier. \coordinate (bf-pin) at (0,-3.828); diff --git a/local.bib b/local.bib index d8f0c5c..1153129 100644 --- a/local.bib +++ b/local.bib @@ -175,3 +175,29 @@ @ARTICLE{2024arXiv241113386B adsnote = {Provided by the SAO/NASA Astrophysics Data System} } + +@ARTICLE{2026arXiv260631806M, + author = {{MacBride}, Sean Patrick and others}, + title = "{The On-Sky Performance of the LSST Camera CCD Array}", + journal = {arXiv e-prints}, + year = 2026, + month = jun, + eid = {arXiv:2606.31806}, + pages = {arXiv:2606.31806}, +archivePrefix = {arXiv}, + eprint = {2606.31806}, + primaryClass = {astro-ph.IM} +} + +@ARTICLE{2026arXiv260631822A, + author = {{Amouroux}, Nathan and {Fagrelius}, Parker and {Guillemin}, Thibault and {Mueller}, Fritz and {Neveu}, J{\'e}r{\'e}my and {Rykoff}, Eli and {Souverin}, Thierry and {Stubbs}, Christopher W. and {Urbach}, Elana}, + title = "{Early Telescope Throughput Results from the Collimated Beam Projector at the Vera C. Rubin Observatory}", + journal = {arXiv e-prints}, + year = 2026, + month = jun, + eid = {arXiv:2606.31822}, + pages = {arXiv:2606.31822}, +archivePrefix = {arXiv}, + eprint = {2606.31822}, + primaryClass = {astro-ph.IM} +} diff --git a/lsst-texmf b/lsst-texmf index a5c83dc..e07d4f2 160000 --- a/lsst-texmf +++ b/lsst-texmf @@ -1 +1 @@ -Subproject commit a5c83dcbe15b553d9339442e062be62276b22067 +Subproject commit e07d4f2afa37c44052bfc1afb97f1ea0609662a1 diff --git a/sections/2_lsstcam.tex b/sections/2_lsstcam.tex deleted file mode 100644 index 3a83614..0000000 --- a/sections/2_lsstcam.tex +++ /dev/null @@ -1,59 +0,0 @@ -\section{The LSST Camera (LSSTCam)} \label{sec:lsstcam} - -The Vera C. Rubin Observatory's camera LSSTCam \cite{10.71929/rubin/2571927} is the largest digital camera in the world for astronomy. -Its design, and the technical progress it demanded, were developed in order to meet the requirements set by the ambitious goals of LSST, as described in \cite{2024SPIE13096E..1SR}. - -% Focal plane -The LSSTCam focal plane is a mosaic centrally composed of 189 CCDs used for science imaging, organized into 3$\times$3 rafts, in a focal plane mosaic presented in \citet{CTN-001}. -Each raft is composed of three Readout Electronics Boards (REB) providing the power and electronics of three neighboring CCDs in a row. -The corners of the mosaic contain two CCDs for guiding and one split-wavefront CCD offset intra- and extra-focally relative to the science CCDs. -\todo{Add focal plane layout plot} - -% CCDs -LSSTCam CCDs are read out in 16 channels, each with their own readout electronics and amplifiers. -Each channel is 2048 pixels high and 512 pixels wide for E2V and 2048 pixels high and 576 pixels wide for ITL sensors. -Figure \ref{fig:isr-before-after} shows one E2V and one ITL detector from the same exposure before and after instrument signal removal. -The raw frames retain the serial and parallel overscan, which are virtual pixels read to estimate charge transfer inefficiency (CTI), see \ref{sec:processing:calibration:cti} for more details. -%here add more about electron transportation and size of ovescan region and serial register. - -\begin{figure*}[t] - \centering - \gridline{\fig{figures/isr_2026010700038_det45_raw.pdf}{0.52\textwidth}{(a) E2V, raw} - \fig{figures/isr_2026010700038_det45_afterisr.pdf}{0.48\textwidth}{(b) E2V, post-ISR}} - \gridline{\fig{figures/isr_2026010700038_det79_raw.pdf}{0.52\textwidth}{(c) ITL, raw} - \fig{figures/isr_2026010700038_det79_afterisr.pdf}{0.48\textwidth}{(d) ITL, post-ISR}} - \caption{One LSSTCam exposure (2026010700038) before and after instrument signal removal (ISR), shown for one detector from each vendor: E2V detector 45 (R12\_S00, top row) and ITL detector 79 (R20\_S21, bottom row). - The left panels (a, c) show the raw readout in analog-to-digital units (ADU), with the serial (solid outline) and parallel (dashed outline) overscan regions of each of the 16 amplifiers marked. - The right panels (b, d) show the same detectors after ISR, in electrons, with the overscan trimmed and the mask planes set during processing overlaid in color (see legend). - Each panel is displayed with its own grayscale z-scale stretch, and all panels share the same pixel scale, so the post-ISR images appear smaller because the overscan has been removed.} - \label{fig:isr-before-after} -\end{figure*} - -\subsection{Physical Model of Photon Transport in LSSTCam CCDs} \label{sec:lsstcam:detector-model}\label{sec:processing} - - -Calibration and ISR tasks are organized as a pipeline based on our current model of the detector physics since instrument artifacts are removed in the reverse chronological order in which they compound along the signal chain. - -The calibration and instrument signal removal (ISR) therefore needs a clear picture of the signal chain, from the point of conversion of photons in the detector to the serialized digital records, as well as -physical models of each systematic transformation that applies to the signal in each step. - -Figure \ref{fig:detector-boxes} depicts the different steps in this chain and their physical ordering from the detector -to the Readout Board (REB). - -The algorithms and processing steps required for LSSTCam calibration and ISR were developed from measurements and analysis during electro-optical testing on the laboratory test stand at SLAC National Accelerator Laboratory, at the summit, and on the telescope in-dome during commissioning. -\citet{2024SPIE13096E..1SR} gives a detailed account of system characterization work and results from the laboratory campaigns. -Figure~\ref{fig:detector-boxes} summarizes the instrumental effects identified in the photon transfer chain and their order. - -\begin{figure*}[t] - \centering - \includegraphics{figures/calibration_boxes_detector_model.pdf} - \caption{Photon transfer and data-acquisition model for the LSSTCam detectors and electronic readout board (REB), containing the 18-bit analog-to-digital converter (ADC) at the end of the readout chain \citep{SITCOMTN-086}. - Instrumental effects seen in images are shown as dashed boxes; arrows indicate where each effect enters the chain. - Low-signal non-linearities in LSSTCam's detector response have been observed, but their origin is not yet established; that contribution appears as a disconnected box.} - \label{fig:detector-boxes} -\end{figure*} - - -%Figure~\ref{fig:detector-boxes} summarizes the instrumental effects identified in the photon transfer chain and their order. - -%The ISR pipeline implemented in the Rubin data processing aims at correcting each of these effects in the order opposite to which they happen. diff --git a/sections/3_data.tex b/sections/3_data.tex deleted file mode 100644 index 8769141..0000000 --- a/sections/3_data.tex +++ /dev/null @@ -1,70 +0,0 @@ -\section{Data}\label{sec:data} -Here we discuss the calibration system hardware and performance available for generating the calibration datasets needed as input to the ISR pipeline. - -\subsection{Calibration System (CalSys)}\label{sec:data:calsys} - -LSST uses four calibration systems to construct calibrations. Three of these (the flat-field illuminator, the collimated beam projector, and the Auxiliary Telescope) will be used continuously throughout the LSST, while one calibration system (the Camera Calibration Optical Bench) was used to characterize and qualify the LSSTCam CCDs before integration to the Simonyi Survey Telescope at Rubin Observatory. -\textcolor{red}{add citation to Seans SPIE paper.} - -Here we discuss the calibration system hardware and performance available for generating the calibration datasets needed as input to the ISR pipeline. - -\subsubsection{Camera Calibration Optical Bench}\label{sec:data:calsys:ccob} - -To create optical scenes for characterization, the Camera is placed in-line with an optical scene projector called the Camera Calibration Optical Bench (CCOB) \citep{2024SPIE13103E..0WU}. -The CCOB-wide beam is designed to project a uniform monochromatic light source onto the focal plane, thanks to one of six LEDs, each corresponding to a central wavelength in the LSST filter bandpass.\todo{Can we quantify ``uniform'' and ``monochromatic'' here?, e.g. the achieved illumination uniformity across the focal plane (percent peak-to-peak or RMS), and the LED central wavelengths with their FWHM.} -In-lab tests with the CCOB are described in \textcolor{red}{here add citation}, the so-called ``Run 7'' was the final campaign of testing of LSSTCam at SLAC National Accelerator Laboratory, -calibrations produced during that run were used as the initial LSSTCam on-sky observations. -%Light is produced by , is controlled by a custom LED driver and fed into an integrating sphere. The light from the integrating sphere is projected onto the focal plane with a minimal ND filter (10\%) attached to a C-mount lens with an adjustable f/stop. The CCOB-wide beam was used in run 6 and run 7. -%\textbf{I mention run 6 and run 7 here - it is not properly introduced. Is it within scope to mention the EO-testing run at the Rubin Observatory L3 CR?} -%One difference between run 6 and run 7 was that the f/stop of the lens was changed from 2.6 to 1.6 (fully open). This was done to try to reduce the effect of the `weather' and the `CMB pattern' two effects that we found in Run 6 at SLAC, and were found to be due to our projection setup (\citet{2024arXiv241113386B}). While changing the f/stop did reduce the weather pattern, it also caused a much steeper illumination roll-off across the focal plane. We evaluated the weather pattern and illumination roll-off relative to Run 6. -%To reduce the effects of the `weather' and `CMB' but retain uniform illumination across the focal plane, we installed a Lambertian diffuser to the CCOB-wide beam for Run 7. The diffuser used is a $60^o$ diffusing angle unmounted sheet from Edmunds Optics. We found that the diffuser greatly reduced the `weather', eliminated the CMB pattern, more uniformly illuminated the focal plane, with a penalty of decreasing the overall illumination by roughly 35\%. The diffuser was installed for all run 7 record runs using the CCOB Wide Beam. - -\textbf{I am not sure how much we want to talk about the CCOB here. If there is a specific focus (qualifying the CCDs before integration), I can revisit this} - -\subsubsection{Flat-Field Illuminator}\label{sec:data:calsys:flat_field_illuminator} - -The Flat-Field Illuminator is designed to provide a stable, uniform, and well-characterized illumination required to measure the absolute and relative instrumental responses of the LSST Camera.\todo{Do we have \emph{as-built, measured} stability and uniformity achieved during DP2 alongside the design requirement? the section currently states only the design target.} Operating in tandem with the CBP and the Auxiliary Telescope (AuxTel), this system aims to satisfy a photometric precision of 5 mmag in repeatability and 10 mmag in spatial uniformity across the field-of-view. \todo{Cite a flatfield calib system paper} The physical architecture of the flat-field system is driven by the fast ($f/1.2$) optical configuration of the Simonyi Survey Telescope, which demands exceptional control over both the angular and spatial uniformity of the calibration wavefront.\todo{Can we quantify this?} The system directs light toward a large 10.27-meter diameter Lambertian calibration screen that is tilted to 23 degrees to align precisely with the optical axis of the telescope. Because the calibration hardware must co-rotate dynamically with the observatory dome during operations, the primary illumination sources, environmental enclosures, and projection optics are mounted directly onto the dome structures rather than a static mount. - -To deliver both broadband and monochromatic illumination during the LSST, the system integrates two independent illumination subsystems within a single centralized optical projector, positioned at the geometric center of the calibration screen structure. Broadband ``whitelight'' illumination is provided by a specialized module featuring a collection of 10 high-power LEDs selected to span the complete $ugrizy$ filter passbands, with one LED within the $u$ passband with the other five passbands each getting coverage from two LEDs. For monochromatic calibration, the system incorporates a tunable laser system that provides continuous wavelength coverage from 300 to 2600 nm at a 1 kHz pulse rate. This laser can deliver narrow tuning steps between 0.05 and 1 nm and a spectral line width under 1 nm FWHM.\textbf{Cite a paper by Parker on this} -%The hardware is housed in a dedicated 590 kg environmentally controlled enclosure on the dome platform to minimize transmission losses across its 20-meter optical fiber runs. Both illumination sources feed into the projector optics to emit an $\sim f/4$ beam at a working distance of 3.2 meters, filling a 74.1 cm diameter monolithic aluminum aspheric reflector mounted via an adjustable hexapod at the top end of the telescope to cast a flat pupil image onto the screen. The output beam's intensity and spectral content are continuously tracked by an in-dome monitoring suite consisting of a NIST-calibrated silicon photodiode read out by an electrometer alongside two fiber-fed spectrographs covering a combined range of 315-1150 nm. - -The data products generated by these dual operating modes support separate but complementary calibration goals within the LSST DM Science Pipelines, to ensure uniformity across the LSSTCam focal plane. In the DP2 dataset, the flat fields used during ISR are constructed with a weighted combination of the two LEDs within the passband (or the single LED in the case of $u$-band). These data products enable the correction of detector-level systematics, including gain variations, charge transfer inefficiencies, pixel-level defects, and sensor distortions induced by the brighter-fatter effect (BFE). The monochromatic flat-field data products from the tunable laser are not used in calibration production for DP2, but will be utilized in future data releases to map the system's chromatic throughput as a function of focal plane position. - -\subsubsection{Collimated Beam Projector}\label{sec:data:calsys:cbp} - -The Collimated Beam Projector (CBP) is a dedicated, in-dome photometric calibration instrument to measure the system throughput of focused light to millimag precision \textbf{Cite a CBP paper}. The CBP consists of a modified Schmidt reflector telescope fed by the tunable laser to produce a collimated beam of known flux. By co-moving the CBP and Simonyi Survey Telescope, we are able to place this beam anywhere on the primary mirror and have a focused spot anywhere on the focal plane. The CBP can switch between five different masks at the focal plane, allowing for different spot patterns to be projected onto the focal plane. For DP2, a mask that projects one spot per CCD was used to construct the crosstalk correction. However, the CBP is also capable of measuring quantum efficiency (QE) variations across detectors and filter transmission curves, and will be used for these purposes in future DR1.\todo{cite CBP SPIE paper} - -%Because the CBP beam only partially illuminates the Simonyi Survey Telescope's 8.4-meter primary mirror aperture at any single pointing, a sequence of spatially distinct pointings is executed to sample the full pupil. -%This targeted partial-pupil approach effectively bypasses the atmosphere and eliminates the severe stray light and ghosting artifacts associated with traditional diffuse flat-screen illumination. - -% One of the masks is a spot per detector to characterize the cross-talk, and one mask per amplifiers. -% \textcolor{red}{Ask Elana} - -%The optical assembly is mounted on an alt-azimuth mechanism on a dedicated platform installed approximately 10 meters above the Rubin Observatory dome floor, which allows the CBP to pivot explicitly about its exit pupil to ensure that beams projected at varying angles map along identical paths into the telescope pupil. The laser beam is fiber-coupled into an integrating sphere equipped with an interior coating to ensure absolute Lambertian diffuse uniformity and eliminate optical memory. At the primary exit port of the integrating sphere (coincident with the CBP telescope's focal plane), a motorized mask wheel supports five interchangeable masks---such as a single-spot-per-CCD pinhole configuration---to generate precise illumination patterns across the camera. Absolute flux monitoring is performed at another port of the integrating sphere using a NIST-calibrated silicon photodiode, read out via an electrometer to determine the exact delivered photon dose. \textbf{Could cut for brevity - not sure if we want the nitty-gritty instrumental information here} - -%The data products generated by the CBP are integrated into Rubin DM via specialized processing workflows to enable precise throughput characterization, and contributions from individual optical components (telescope response, filter throughput, and CCD relative quantum efficiency). By cross-referencing the absolute incident flux recorded by the photodiode with the signal detected across the LSSTCam focal plane, the system can characterize the telescope's response. The data products are highly effective at mapping relative variations across the focal plane, yielding relative quantum efficiency curves that have revealed percent-level systematic offsets from factory-tabulated sensor data. \textbf{mention this as an input to FGCM here?} Furthermore, by taking the ratio of telescope signals captured with and without a filter in the optical path, the CBP provides a self-contained measurement of the LSSTCam filter transmission profiles. This is critical for characterizing the chromatic shift of the filter bandpass edges, which vary across the focal plane by several nanometers due to changing angles of incidence across the LSSTCam filters. These data products also allow for the assessment of optical ghosts, electronic crosstalk, and infrared fringing patterns in the z and y bands, while guaranteeing robust calibration coverage during daytime or overcast periods when on-sky observations are not possible. - -\subsection{Calibration Epochs}\label{sec:data:selection} -Describe the data selection process for the DP2 dataset and calibration epoch. See \ref{fig:calibepoch}. - -Describe any system-level constraints that influenced calibration strategy. - -Describe the LSST Camera and any technical limitations (bad detectors/amps like det 122, CCD power cycling and its affect on existing calibs). - -Define/Describe the sequencer changes (to do in LSSTCam section) - -Analyzed with updated sequencer (s3v2). (see calibration epoch section) - - -\begin{figure*}[t] - \centering - \includegraphics[trim=200bp 15bp 203bp 8bp, clip, angle=-90, width=\textwidth]{figures/calibration-epochs-timeline.pdf} - \caption{Timeline of the observing periods and calibration epochs for LSSTCam during commissioning and pre-survey operations included in DP2. - The timeline begins on the day of LSSTCam first photon, soon after being mounted on the telescope. The first science data included in DP2 was taken on April 24th, 2025. Science data that was released as part of Data Preview 2 (DP2) is separated by calibration epochs, determined by changes to the CCD operating mode (sequencer) or power-cycles for maintenance. Colored segments denote the intervals covered by each calibration epoch: epoch~1 (from 2025 April 24) - sequencer with calibrated defects derived from in-lab calibrations, epoch~2 (from 2025 - July 3, \texttt{3s\_v1}), and epoch~3 (from 2025 October 24, \texttt{3s\_v1}). - Calibration epoch~4 (\texttt{3s\_v2}) follows the DP2 data cutoff on January 6, 2026. The calibrations shown in this paper correspond to the most recent (\texttt{3s\_v2}) set of calibrations, despite the fact that no data from this period was released in DP2. - Dashed segments indicate intervals not covered by a calibration epoch, including an engineering and maintenance downtime between September 21-October 24, 2025. The code freeze for v30 of the LSST Data Management (DM) Science Pipelines software was December 3, 2025.} - \label{fig:calibepoch} -\end{figure*} - diff --git a/sections/4_processing/41_isr-science.tex b/sections/4_processing/41_isr-science.tex deleted file mode 100644 index 295a114..0000000 --- a/sections/4_processing/41_isr-science.tex +++ /dev/null @@ -1,124 +0,0 @@ -\section{Instrument Signal Removal for DP2 Science Data} \label{sec:isr-science} - -Instrument signal removal (ISR) is defined by the steps that transform the raw integer image into a floating-point, fully-calibrated image in electron units with an associated variance plane, pixel masks, provenance, and metadata used in downstream data release processing tasks. -Since this process undoes instrument artifacts that enter into the image in the photon transfer chain, it is crucial for the calibration steps to match the physics of the detectors. - -Corrections are applied in reverse chronological order relative to the photon transfer chain (Figure~\ref{fig:detector-boxes} in \S\ref{sec:lsstcam:detector-model}). -For example, steps that undo readout-electronics effects therefore appear before steps that undo detector-level effects and additive corrections precede multiplicative ones such as flat-fielding. - -Running the processing pipeline with misplaced steps would either apply a correction in the wrong physical units or leave residual structure that invalidates downstream calibrations. -Figure~\ref{fig:isr-pipeline} shows the resulting sequence of corrections as applied to LSSTCam science images for DP2. - -\begin{figure*}[t] - \centering - \includegraphics[width=0.9\textwidth]{figures/isr-pipeline.pdf} - \caption{Outline of the instrument signal removal (ISR) pipeline applied to LSSTCam science images for DP2. - The steps in the pipeline follow a very specific sequence determined by the physics of the instrument signature and where it enters in the photon transfer chain. - Blue boxes denote static ISR steps that apply pre-computed calibration inputs; orange boxes denote steps whose application depends on the science exposure itself (for example, overscan subtraction, dithering, or saturation-footprint growth); white boxes mark the raw input and post-ISR output. - A dashed border marks a computation rather than an artifact correction (integer-to-float conversion, dithering, overscan statistics, and variance plane estimation). - The amplifier-offset step remains optional and applies pre-computed per-amplifier offsets when enabled.} - \label{fig:isr-pipeline} -\end{figure*} - -The following outline follows Figure~\ref{fig:isr-pipeline} step by step. -Each item names the correction applied, and the reason it sits at this location in the processing chain. - -\begin{enumerate} - \item \textbf{Convert integer to float.} - Raw images arrive as integer arrays. This first step promotes pixel values to 64-bit floating point precision. - - \item \textbf{Dither counts.} - A uniform random offset in $[-0.5,\,0.5)$~ADU is added to every pixel. - This breaks the correlation between true signal and integer quantization before continuous corrections are applied, reducing systematic bias when later steps interpolate, convolve, or divide the image.\todo{Quantify the bias that dithering removes: the size of the quantization-induced bias in ADU (or in mmag at the relevant sky levels) with and without dithering.} - This step is essentially a marginalization step before a full low-signal, differential non-linearity correction can be measured and implemented. - - \item \textbf{Serial overscan correction.} - An electronic, clock-injected offset plus smaller row-to-row variations are derived from the serial overscan and subtracted from the imaging pixels while the data are still in native analog-to-digital units (ADU). - Overscan correction must precede gain conversion because the overscan bias is measured and removed in the same units as the raw readout. - At this stage we also calculate overscan statistics, including an estimate of the read noise from the scatter in overscan pixels and the amount of deferred charge from extended pixel edge response (EPER) statistics.\todo{Give the typical magnitude of the clock-injected offset and of the row-to-row variations (in ADU), plus the typical per-amplifier read noise measured this way, with its spread across the focal plane.} - - \item \textbf{Mask saturated and suspect pixels.} - Pixels above the photon transfer curve (PTC) turnoff saturation threshold (\S\ref{sec:processing:calibration:ptc}) get flagged as both saturated (``SAT'') and suspect (``SUSPECT'') in the pixel bit mask before further signal corrections. - Early masking prevents corrupted or non-physical pixel values from biasing downstream estimates that assume a well-behaved signal. - - \item \textbf{Apply gain.} - Per-amplifier gains derived from the PTC (\S\ref{sec:processing:calibration:ptc}) convert pixel values from ADU to electrons. - Electron units are the native domain for deferred-charge modeling, BFE correction, and variance plane estimation, all of which were calibrated assuming a linear conversion from measured counts to stored charge. - - \item \textbf{Crosstalk correction.} - Electrical coupling between adjacent amplifier outputs is removed using the crosstalk matrix (\S\ref{sec:processing:calibration:xtalk}). - Crosstalk is applied after gain conversion so that coupled signals are corrected in electron units, consistent with how the matrix was measured. - - \item \textbf{Parallel overscan correction.} - Analogous to the serial overscan step, the parallel overscan region provides a column-direction estimate of readout offset that is subtracted from the imaging area. - It follows crosstalk correction because amplifier coupling can inject signal into overscan regions, and removing crosstalk first yields a cleaner overscan bias estimate. - - \item \textbf{Non-linearity correction.} - The double-spline linearizer (\S\ref{sec:processing:calibration:linearity}) maps measured counts to a linearized electron count by subtracting absolute and amplifier-relative spline corrections. - This step must precede deferred-charge correction because the deferred-charge model and its calibration were fit assuming that signal-dependent readout non-linearities have already been removed (\S\ref{sec:processing:calibration:cti}). - - \item \textbf{Deferred charge correction.} - Residual charge left in pixels during serial readout---from global charge transfer inefficiency, local electronic offset in the output amplifier, and serial charge traps on ITL sensors---is corrected using the parameters fit in \S\ref{sec:processing:calibration:cti}. - This step sits late in the readout-electronics portion of the chain because deferred charge is imprinted during serial transfer, after non-linearities in the output path have acted on the signal. - We do not correct deferred charge on E2V sensors because the deferred charge level is already below the level defined by \citet{LPM-17}.\todo{State both numbers: the LPM-17 threshold ($5\times10^{-6}$, per \S\ref{sec:processing:calibration:cti}) and the measured E2V deferred-charge level, so ``already below'' is demonstrated rather than asserted.} - - \item \textbf{E2V edge-bleed mask.} - We flag the ``SAT'' mask plane bit for pixels in E2V amplifiers that show uncorrectable charge spill from bright regions at the serial-register edge. - - \item \textbf{ITL dip, SatSag, and edge-bleed masks.} - We flag the ``ITL\_DIP'' mask plane bit for pixels in ITL amplifiers that exhibit a pixel column depression in response near certain amplifier boundaries (the ``ITL dip''), a sag in apparent signal after saturation (``SatSag''), and edge bleeding analogous to the E2V case. - These vendor-specific masks follow deferred-charge correction because the artifacts they mark are readout- and saturation-related structures that should be evaluated on linearized, CTI-corrected data. - - \item \textbf{Bias correction.} - A combined bias frame (\S\ref{sec:processing:calibration:bias}) is subtracted to remove the fixed-pattern electronic offset that remains after overscan subtraction. - Overscan correction handles the spatially static electronic structure imprinted in the images before charge readout. - - \item \textbf{Dark correction.} - The combined dark frame (\S\ref{sec:processing:calibration:dark}), scaled by the science exposure time, is subtracted to remove thermal dark current accumulated during exposure integration. - Dark correction follows bias subtraction because both are additive offsets, and the dark frame is constructed from bias-subtracted dark exposures. - - \item \textbf{Defect masking.} - Persistent bad pixels, columns, and other static defects identified during calibration (\S\ref{sec:processing:calibration:defects}) are merged into the pixel mask. - Applying the defect mask after deferred-charge, bias, and dark correction places it once readout-electronics effects and integration-period offsets have been removed, matching the processing state in which the defect catalog was built and ensuring that later defect footprint widening and interpolation operate on the full static bad-pixel map. - - \item \textbf{Widen saturation footprints.} - The saturation mask is expanded spatially to include pixels affected by charge spill and trails from saturated sources.\todo{Specify the growth radius in pixels used for DP2 and, if possible, the resulting typical masked-pixel fraction per detector.} - This step follows defect masking so that all static bad regions are known before the saturation footprint is grown into neighboring pixels. - - \item \textbf{Brighter-fatter correction.} - The electrostatic BFE (\S\ref{sec:processing:calibration:bf}) redistributes flux among neighboring pixels to undo charge-cloud repulsion in the silicon. - In the photon transfer model (Figure~\ref{fig:detector-boxes}), the BFE is the earliest detector-level effect on incident photons, so its correction appears correspondingly late in ISR. - It is applied after the image has been converted to linearized electron units and after additive calibration-frame corrections. - The correction also needs to match the boundary-shifts, which were also calibrated from PTC covariances measured after these corrections. - - \item \textbf{Variance plane estimation.} - An expected variance is assigned to each pixel using the PTC-derived gain and read noise (\S\ref{sec:processing:calibration:ptc}). - The variance plane is computed after gain application and BFE correction so that it reflects the electron units and pixel correlations relevant to source detection and weighting in downstream analyses. - - At this stage the expected per-pixel variance follows the Poisson-plus-read-noise model - \begin{equation} - \sigma^2_{ij} = \frac{F_{ij}}{g_a} + \left(\frac{\sigma_n(a)}{g_a}\right)^2, - \label{eq:isr-variance} - \end{equation} - where $F_{ij}$ is the science image value in the current ISR units, $g_a$ is the PTC gain for amplifier~$a$ (electrons per ADU), and $\sigma_n(a)$ is the read noise in electrons. - For DP2 science processing the image is already in electron units, so $g_a = 1$ and Eq.~\ref{eq:isr-variance} reduces to $\sigma^2_{ij} = F_{ij} + \sigma_n(a)^2$. - Pixels with non-positive variance are flagged in the mask. - - Since the variance plane is used to estimate the uncertainty on just the charge accumulated in the image, it needs to be computed before flat-field correcting. - - \item \textbf{Flat correction.} - The normalized flat field (\S\ref{sec:processing:calibration:flats}) is divided into the image to correct pixel-to-pixel variations in quantum efficiency and optical throughput. - Flat-fielding is intentionally late in the pipeline: it is a multiplicative correction that must be applied only after all additive electronic and dark offsets have been removed, so that the flat field encodes relative sensitivity rather than residual bias structure. - - \item \textbf{Interpolate masked pixels.} - Pixels in the ``SAT'', ``BAD'', and ``UNMASKEDNAN'' mask planes are filled by interpolation from neighboring unmasked values; ``SUSPECT'' pixels are flagged but not interpolated by default. - Interpolation is performed after flat-fielding so that both the science signal and the replacement values are in the same calibrated flux units expected by algorithms that assume a continuous image (for example, background estimation and certain astrometric measurements). - - \item \textbf{Amplifier offsets.} - Small per-amplifier additive adjustments derived from flat-field residuals may be applied as a final optional step to reduce residual flux discontinuities at amplifier boundaries.\todo{Quantify ``small'' and ``residual'': the typical amplitude of the amplifier-boundary discontinuity before and after this step, in electrons or as a fraction of sky, and state whether it was enabled for DP2.} - This step is optional because the PTC gain solution already includes average amplifier matching (\S\ref{sec:processing:calibration:ptc}); the offset correction addresses any remaining channel-to-channel offsets visible after the full ISR chain. -\end{enumerate} - -The output exposure's metadata records the calibration epoch, the identifier of the curated collection used, and a per-step record of which ISR corrections were applied or skipped. -This provenance makes it possible to reproduce the processing of any released image, compare products generated across software or calibration versions, and audit the configurations enabled for a given dataset. -The verification, acceptance, and distribution context for these metadata fields is described in \S\ref{sec:verify-accept-certify:data-products}. \ No newline at end of file diff --git a/sections/4_processing/42_calibration-production-and-application.tex b/sections/4_processing/42_calibration-production-and-application.tex deleted file mode 100644 index 601657f..0000000 --- a/sections/4_processing/42_calibration-production-and-application.tex +++ /dev/null @@ -1,558 +0,0 @@ -\section{Calibration Production and Application} \label{sec:calibration-production} -% former label: sec:processing:calibration - -Some artifacts removed in the ISR sequence described in \S\ref{sec:isr-science} require a calibration data product that characterizes the instrumental effect being removed, measured in advance from dedicated calibration exposures rather than from the science image itself. -In this section, we describe in detail the algorithms used to produce calibrations and apply them to an image in ISR. -Figure~\ref{fig:calib-dependency} summarizes calibration-product dependencies. - -\begin{figure}[t!] - \centering - \includegraphics[width=\columnwidth]{figures/calib-dependency.pdf} - \caption{Calibration production dependency graph. - Arrows indicate data or dependency flow (``x $\rightarrow$ y'' means ``y depends on x''). - Solid outlined boxes represent datasets that are deployed/released with data products while dashed boxes represent internal or intermediate datasets that are not formally released to users.} - \label{fig:calib-dependency} -\end{figure} - -% NOTE: order follows calib-dependency figure - -\subsection{Curated Calibrations} \label{sec:processing:calibration:curated-calibs} - -\subsection{Auxiliary Data Products} \label{sec:processing:calibration:auxiliary-data-products} - -\subsection{Crosstalk} \label{sec:processing:calibration:xtalk} -Large astronomical sensors are read out through multiple amplifiers in parallel (see Section \ref{sec:lsstcam}), reducing the time needed to completely read the sensor. -One downside of this design strategy is the possibility of inducing crosstalk between the different amplifier channels. -In LSSTCam, this crosstalk is produced by electronic coupling that occurs in at least two different places in the readout hardware: (1) between the channels in the cable connecting the detectors to the REB and (2) across opposite sides of each detector with neighboring analog signal-processing integrated circuits (ASPICs). -The result of these couplings is that the data for a particular amplifier shows a scaled copy of neighboring amplifiers added to the image.\todo{Give the typical and maximum crosstalk coefficient magnitudes measured for LSSTCam (e.g.\ $\sim10^{-4}$ nearest-neighbour, $<10^{-5}$ elsewhere), and how the two coupling mechanisms differ in strength.} -As all amplifiers are clocked through the readout process simultaneously, this scaled copy is aligned with the location of the readout corners. -This may make the crosstalk signal visible in the target amplifier flipped relative to the source amplifier. -These two sources cannot be disentangled from one another, so we measure and correct both forms of crosstalk as a single artifact. - -Correcting for this additional signal is easy with a known set of crosstalk coefficients between a given source and target amplifier. Each source amplifier that has a non-zero coefficient is flipped to match the target amplifier, scaled by that coefficient, and added to the results from all other source amplifiers to create a subtrahend image that contains all of the crosstalk signal observed on the target amplifier. -By repeating this for all amplifiers, we can build up a full-detector subtrahend image that can then be subtracted. -As this correction may be imperfect, we set a CROSSTALK mask plane for all locations where the subtrahend image is greater than 1 electron.\todo{Quantify ``may be imperfect'': the residual crosstalk after correction (as a fraction of the uncorrected signal, cf.\ Figure~\ref{fig:crosstalk-linear-nonlinear-fit}) and the typical fraction of pixels flagged CROSSTALK in a science image.} -This provides a way for the downstream object detection and measurement code to identify sources that may have been significantly altered by the crosstalk correction. - -As this crosstalk occurs in the cable connecting the detector to the REB, prior to being digitized into ADU, the images need to be gain corrected prior to the crosstalk correction to ensure they are in electron units. -This naturally places the crosstalk correction early in the ISR processing. This is also important because very bright sources can create bleed trails that deposit charge that is not fully clocked out, and that slowly leaks out as subsequent rows are read out. -These bleed trails can extend into the parallel overscan region, which can in turn create crosstalk signal in neighboring amplifiers. -Performing crosstalk correction prior to overscan correction prevents these amplifiers from oversubtracting the affected columns. - -The crosstalk coefficients used during commissioning were calibrated from lab measurements at SLAC prior to the camera being shipped to the summit and were quadratic with the source flux, as discussed in \citet{2021JATIS...7d48002S} (CZW: do we need to introduce the measurement process first?). -However, the switch from 2s readouts to 3s readouts via a camera sequencer change invalidated these coefficients, as the change in clocking also changed the scale of the crosstalk signal, requiring this calibration to be remeasured.\todo{State by how much the coefficients changed between the 2\,s and 3\,s sequencers (a typical ratio or percent shift), which is what justifies the remeasurement.} - -This measurement used three nights (CZW: check numbers in this paragraph) of spot data taken with the CBP system. -Each exposure placed one spot on each detector, with ten linearly spaced flux levels (CZW: add figure or describe levels) per pointing. -After the flux sequence was completed, the CBP spots were shifted to land on a different amplifier, with the process repeated in order to provide each amplifier a bright source to use to look for crosstalk signals. -A total of XX exposures were used to generate these updated crosstalk coefficients, with the number of pixels used as ``source pixels'' (CZW: which you haven't explained yet) shown in Figure~\ref{fig:crosstalk-source-pixels}. - -\begin{figure}[t] - \centering - % TODO: add exposure-time vs.\ mean-signal figure asset - \fbox{\parbox{0.75\columnwidth}{\centering\vspace{1.5in}\small\textit{Figure pending}\vspace{1.5in}}} - \caption{Number of source pixels used to measure the crosstalk coefficients for each amplifier. - Note that this number drops toward the edge of the focal plane, due to difficulties properly illuminating CBP spots in these locations.\todo{Quantify the drop-off: typical source-pixel counts at the focal-plane centre vs.\ the edge, and the minimum count below which a coefficient was considered unconstrained.}} - \label{fig:crosstalk-source-pixels} -\end{figure} - -As discussed above (CZW: ???), each of these exposures was minimally processed to correct the serial overscan, apply the gains to convert the units to electrons, and to trim and assemble the data segments. -All pixels above a threshold of 30,000 electrons were then identified and used to construct crosstalk ratios $r_{ij} = T(\hat(x), \hat(y))_j / S(x, y)_i$, where $S(x, y)_i$ is the flux value in a pixel at $(x, y)$ in amplifier $i$, and $T(\hat(x), \hat(y))_j$ is the flux value at the same ``readout coordinate'' $(\hat(x), \hat(y))$ in amplifier $j \ne i$. -These ratios therefore show the measured response to the source signal $S$ at the expected location of a crosstalk source at a target location $T$. -These fluxes and ratios are accumulated for the entire set of exposures and detectors. -Prior to any further study, the ratios are clipped to ensure contaminating features, such as stray light and reflections of other spots, are removed.\todo{Report what fraction of ratio measurements the clipping removes, so the reader can judge how aggressive the cleaning is.} -Any pixel with a measured ratio larger than 1.0 are excluded, as these would represent an unphysical situation in which the crosstalk signal was brighter than the source signal. -The remaining signals were XX-sigma clipped based on the inter-quartile distance, under the assumption that the ratio values are normally distributed. - -If the crosstalk response was perfectly linear, the average of these clipped ratios would directly yield the crosstalk coefficient between the source and target amplifiers. -However, as shown in the lab study of the crosstalk, and demonstrated in Figure~\ref{fig:crosstalk-linear-nonlinear-fit}, such a simple model fails to fully remove the crosstalk, as the source brightness changes over the profile of the spot.\todo{Quantify the improvement from the non-linear model: residual amplitude (in electrons or as a fraction of source flux) for the linear vs.\ non-linear coefficients shown in the two panels of the figure.} - -\begin{figure}[t] - \centering - % TODO: add exposure-time vs.\ mean-signal figure asset - \fbox{\parbox{0.75\columnwidth}{\centering\vspace{1.5in}\small\textit{Figure pending}\vspace{1.5in}}} - \caption{An example crosstalk residual on LSSTCam detector XX after correction with a linear crosstalk coefficient. - The oversubtraction in the core and undersubtraction in the wings of the source illustrates that the strength of the crosstalk signal is dependent on the flux of the source pixel. - The right panel shows the same spot, as corrected with the final non-linear crosstalk coefficients.} - \label{fig:crosstalk-linear-nonlinear-fit} -\end{figure} - -To provide the required crosstalk coefficients that are non-linear in source flux, we quantified the evolution of crosstalk coefficients with a linear term and a non-linear term in the fit to the flux ratios. -The intercept of this fit provides the term that is linear in the source flux, and the slope of the fit provides a term quadratic in the source flux. - -\subsection{Defects} \label{sec:processing:calibration:defects} - -We build a \textsc{defects} calibration product based on defects manually defined and automatically detected. -They correspond to pixels whose output is not well corrected by our ISR approach.\todo{Quantify the defect population for DP2: total number of masked pixels/columns, the fraction of the focal plane they represent, the split between manual and automatic detections, and how that fraction varies between E2V and ITL and across epochs.} -These defect pixels are masked and marked as \textsc{BAD} for the downstream analysis and interpolated over for visual inspection. - -The \textsc{defects} calibration product is constructed by measuring defects on bias, dark, flats and then merged together. -The defect detection consists of the following steps: -\begin{itemize} - \item correct for the gradient across the focal plane, by applying a gradient correction to each amplifier - from the gradient measured in a binned image. - \item a background estimate is done and removed, - \item hot pixels are defined as bright pixels with more than 5~electrons/s (scaled by the exposure time) of dark current, above 1000 ADU in bias are measured, - \item cold pixels are defined as pixels with photoresponse less than $80\%$ of the flat mean are measured, - \item pixels along the midline break (the interface between the top and bottom amplifiers) in E2V detectors are - set to the median of the flats (so they are not found by the defect code), - \item bad columns masking. -\end{itemize} - -Afterwards, these measured defects are merged together, and the edges of the detectors are masked, to -mask what we refer to as the ``picture frame'' effect corresponding to the presence of distorted electric fields at the edges of the CCDs.\todo{Give the edge-mask width in pixels and the measured size of the ``picture frame'' response distortion (percent deviation from the detector interior) that motivates that width.} - -For DP2, these defects were produced by using ``bootstrap'' versions of the bias, dark, and flat, as these calibration products need the defects themselves to be defined. -This will probably be updated in future iteration of calibration products. - -Additionally manual defects are large features present in LSSTCam detectors, such as the large defect in detector 0 -which induces large bleeds.\todo{Quantify ``large'': the size of the detector 0 defect and the typical extent of the bleeds it induces, plus how many detectors carry manual defects of this scale.} LSSTCam manual defects are defined in \url{https://github.com/lsst/obs_lsst_data/tree/main/lsstCam/manual_defects}. -%We describe how defect masks are generated (bad pixels/columns, saturation trails, and other artifacts), the thresholds used, and how masks are combined across exposures. - -Further defect statistical analyses, description of the defect monitoring, masking optimization are postponed to a future paper. -Additional masking is applied to an exposure in the ISR processing, as described in Sec ??. -% TODO(ref): literal "Sec ??" -- point this at the ISR masking steps in \S\ref{sec:isr-science}. - -\subsection{Non-Linearity} \label{sec:processing:calibration:linearity} - -Non-linearity (NL) describes a deviation from simple proportionality when mapping amplified voltage integrated during readout to the number of analog-to-digital units (ADU) or counts. -Non-linearity is signal-dependent and is often not a simple function of signal level; it can change suddenly above a large signal level that is characteristic to each readout amplifier; and it can drift with sensor temperature and with time, as slow electronic shifts in the bias level accumulate over a long calibration campaign.\todo{Quantify the drifts: the fitted $\alpha$ (per K) and $\beta$ (per day) sensitivities and the resulting non-linearity change over a typical campaign, plus the range of per-amplifier turnoff levels implied by ``characteristic to each readout amplifier''.} -In practice it introduces pixel-level measurement bias (up to x\% from the true signal) and invalidates Poisson counting statistics and can interfere with measurement of gain (which shouldn't change or drift with signal level).\todo{Fill in $x$ with the measured uncorrected non-linearity amplitude, and say over what signal range it applies.} -Correcting NL to better than y\% is therefore a prerequisite for reliable downstream calibration products and science analyses.\todo{Fill in $y$ with the requirement, cite its source, and state the residual level actually achieved by the double-spline model for DP2.} - -We can measure non-linearity by measuring the response of pixel values to a change in illumination level. -The in-dome calibration illumination system can deliver an approximately uniform flux of light across the LSSTCam focal plane.\todo{Say how uniform ``approximately uniform'' is (percent variation across the focal plane), since the relative-spline step exists specifically to absorb this non-uniformity.} -Given a known change in some input illumination level, we should be able predict a change in the number of counts in an image, which we can then compare to the measured change counts. - -We have independent measurements of the photodiode current produced by our calibration illumination system that we can integrate with the exposure time and use as a baseline truth of the amount of incident light while taking an exposure, -however we found these measurements to be noisy and unreliable.\todo{Quantify ``noisy and unreliable'': the fractional scatter of the integrated photodiode current relative to exposure time, compared with the shutter-timing accuracy, to justify the substitution.} Instead we use the exposure time, which is precisely measured and makes a natural proxy for photodiode measurements. -We therefore calibrate NL by fitting a relation between exposure time ($\Delta t$, abscissa) and the mean measured flat-field signal ($\mu$, ordinate). - -We measured $543\times2$ flat-field exposures at evenly spaced illumination levels. We went as low as 100~analog-to-digital units (ADU) and continued beyond pixel saturation.\todo{State the amplitude and signal range of the observed low-signal non-linearity below this 100~ADU floor (percent deviation below what ADU level), so the disconnected box in Figure~\ref{fig:detector-boxes} and the missing DNL correction flagged in \S\ref{sec:limitations} are backed by a number.} -At each illumination level, a pair of flats was acquired by varying both LED current and exposure time to achieve the targeted levels, measuring the mean signal level as $\mu_i = \mathrm{avg}(F_i^{(1)},F_i^{(2)})$~(in ADU). -These data were acquired with illumination levels decided in random order to decouple signal-dependent non-linearity and time-dependent drifts in illumination. - -\begin{figure}[t] - \centering - \includegraphics[width=0.95\columnwidth]{figures/reference-amp-non-linearity_det94.pdf} - \caption{Absolute non-linearity fit for the reference amplifier~C07 on LSSTCam detector~94. - \textit{Left}---mean flat-field signal $\mu_{\mathrm{ref}}$ as a function of exposure time $\Delta t$, colored by LED illumination group ($k_1$, $k_2$, $k_3$). - Each group has its own ratio of signal to exposure time, which produces the discontinuities between groups. - The hatched region marks signal above the linearity turnoff ($\mu > 1.05 \times 10^5$~ADU), which is excluded from the fit. - \textit{Right}---fractional residuals of the spline fit, $(r_i - 1) \times 10^4$ (Equation~\ref{eq:spline-nl-fit}), as a function of $\mu_{\mathrm{ref}}$, with the same group colors.} - \label{fig:linearity-exptime-vs-signal} -\end{figure} - -Figure~\ref{fig:linearity-exptime-vs-signal} shows the resulting relation for a representative amplifier. -Discontinuities appear at the boundaries between LED current settings, producing different signal levels for the same exposure time. -These data are partitioned into groups associated with distinct LED illumination settings. -Grouping is determined either from exposure metadata or by computationally searching for jumps in the ratio of mean detector signal to exposure time. -Within each group we allow a distinct proportionality constant $k_j$ to capture the size of the jumps. - -To first order, the trend between exposure time and the mean measured flat-field signal over properly scaled groups is linear and related to the intrinsic gain in each readout channel. -We can fit a simple slope and intercept to this trend to the low signal points; the highest signal level in each channel above which this approximation break is the ``linearity turnoff'' (as indicated in Figure~\ref{fig:linearity-exptime-vs-signal}). -Points above the turnoff are excluded from the NL fit because there is a maximum signal above which the measured counts can no longer be trusted due to hardware limitations at the output of the detector (Figure~\ref{fig:detector-boxes}). - -\begin{figure}[t] - \centering - \includegraphics[width=0.95\columnwidth]{figures/abs-rel-non-linearity_det94.pdf} - \caption{Double-spline non-linearity model for LSSTCam detector~94. - \textit{Left}---absolute non-linearity $S_{\mathrm{abs}}(\mu)$ of the reference amplifier~C07. - Gray points are per-exposure measurements (spline value plus fit residual), the black line is the Akima spline, and open circles mark the spline nodes. - \textit{Right}---relative non-linearity $S_{\mathrm{rel}}(\mu)$ of the remaining amplifiers with respect to C07, with nodes shown as filled points. - Blue shades are amplifiers C10--C17 and orange shades are C00--C06; darker shades correspond to higher segment numbers.} - \label{fig:linearity-abs-rel} -\end{figure} - -The NL model we fit to this relation is an extension of the formalism developed by \citet{2019A&A...629A..36A}. -We represent the deviation from linearity by an Akima spline $S(\mu)$ of the measured mean signal $\mu$. - -For each exposure $i$ in group $j$ we define the residual ratio -\begin{equation} - r_i = \frac{\mu_i - S(\mu_i) - O}{k_j\, D_{i}} - \label{eq:spline-nl-fit} -\end{equation} -where $D_{i}$ is the data used for the abscissa (exposure time), $O$ is an optional constant offset that absorbs scattered light or other signal not proportional to $D_{i}$, and $w_i$ are weights. - -The fit minimizes a uniformly weighted ($w_i = 1$) least-squares objective over all accepted points: -\begin{equation} - \chi^2 = \sum_{i} w_i \left(r_i - 1\right)^2. \label{eq:linearity-chisq} -\end{equation} -The fit is subject to the boundary conditions $S(0) = 0$ and $\langle S(\mu)\rangle = 0$ over the fit range. -The first constraint fixes the origin such that zero illumination is always measured as zero signal; the second ensures that the spline captures deviations from linearity independently from knowing the true amplifier gain. - -In addition, any hardware sensitivities are absorbed by nuisance parameters that we parameterize as first order linear corrections to $\mu$ before the spline is evaluated: -\begin{equation} -\begin{aligned} - \mu &\rightarrow \mu \left(1 + \alpha\, T_i\right), \\ - \mu &\rightarrow \mu \left(1 + \beta\, t_i\right), -\end{aligned} -\label{eq:linearity-nuisance} -\end{equation} -where $T_i$ is a REB temperature (referenced to the median over the campaign), $t_i$ is the Modified Julian Date (MJD) of the midpoint of the shutter open time, and $\alpha$ and $\beta$ are fitted coefficients that describe each sensitivity. -For LSSTCam DP2 calibration production we fit only the temporal coefficients as well as the scattered-light offset $O$ on the spline. -For DP2, we found it to be unnecessary to fit the temperature coefficient since the temperature of the REB proved to be quite stable.\todo{Quantify ``quite stable'': the REB temperature range over the calibration campaign and the non-linearity error that neglecting $\alpha$ therefore incurs.} - -We could perform this fit independently for every amplifier on every detector. However, most of the non-linearity is common to all amplifiers due to similar readout architectures, with only small amplifier-to-amplifier variations on top of that.\todo{Put numbers on ``most'' and ``small'': the amplitude of the absolute spline vs.\ the RMS of the relative splines (both readable off Figure~\ref{fig:linearity-abs-rel}), which is the quantitative justification for the double-spline decomposition.} -In addition, each independent fit to every amplifier would pick up not just the intrinsic non-linearity in amplifiers, but also exposure-to-exposure spatial illumination variation that occurs during the calibration campaign. -To better characterize this, it makes sense to perform a spline fit on each amplifier relative to both a single reference exposure and a reference amplifier on the same detector. - -We can distinguish an \emph{absolute} non-linearity in a single reference amplifier on a detector and the \emph{relative} non-linearity of other amplifiers to that reference amplifier, the process for which is summarized in Algorithm~\ref{alg:linearity-double-spline} and illustrated in Figure~\ref{fig:linearity-abs-rel}. -After choosing a reference amplifier and a reference exposure, we do (1) a relative spline fit on every other amplifier using the variation of its mean flat-field signal $\mu$ relative to the reference amplifier and to the $\mu$ in a single reference exposure, and then (2) -an absolute spline fit on the reference amplifier using the usual from $\mu$ versus $\Delta t$. -Every amplifier on a detector has a shared absolute spline fit based on the reference amplifier for that detector and a relative spline fit computed relative to a reference amplifier and a reference exposure ($S_{\mathrm{rel}}(\mu) = 0$ for the reference amplifier). - -\begin{algorithm}[H] -\caption{Double-spline linearity fit (single detector)} -\label{alg:linearity-double-spline} -\begin{algorithmic}[1] -\For{each amplifier $a$} - \State $k_j \leftarrow$ \Call{Find Groups}{$\mu_a$, $\Delta t$} - \State \Call{Compute linearity turnoff}{$\mu_a$, $\Delta t$} -\EndFor -\State choose reference amplifier $\mathrm{ref}$ with the highest linearity turnoff -\State choose reference exposure $i_0$, the flat whose mean signal $\mu_{\mathrm{ref}} = 10^4$ on amplifier $\mathrm{ref}$ is nearest the target signal level -\For{each exposure $i$} - % \State compare flat $i$ to the reference exposure $i_0$ - % \State $\mu_{\mathrm{ref},i} \leftarrow$ $\mu_{\mathrm{ref}}$ / $\mu_{\mathrm{ref}, i_0}$ % mean signal on amplifier $\mathrm{ref}$ in exposure $i$, normalized to $i_0$ - \For{each amplifier $a \neq \mathrm{ref}$} - \State $D^{\mathrm{rel}}_{i} \leftarrow \mu_{\mathrm{ref}, i} \times (\mu_{a,i_0} / \mu_{\mathrm{ref}, i_0})$ (Relative Abscissa) % mean signal on amplifier $a$ in exposure $i$, normalized to $i_0$ and expressed relative to $\mu_{\mathrm{ref},i}$ - \State $\mu^{\mathrm{rel}}_{i} \leftarrow \mu_{a,i}$ (Relative Ordinate) - \EndFor - - \State $D^{\mathrm{abs}}_{i} \leftarrow \Delta t_i$ (exposure time) - \State $\mu^{\mathrm{abs}}_{i} \leftarrow \mu_{\mathrm{ref},i}$ (measured counts) -\EndFor - -\Statex \Comment{Spline fit} -\State $S_{\mathrm{abs}}(\mu), O, \alpha, \beta \leftarrow$ \Call{Minimize}{Equation~\ref{eq:linearity-chisq}, $D^{\mathrm{abs}}_{i}$, $\mu^{\mathrm{abs}}_{i}$, $k_j$} -\State $S_{\mathrm{rel}}(\mu) \leftarrow$ \Call{Minimize}{Equation~\ref{eq:linearity-chisq}, $D^{\mathrm{rel}}_{i}$, $\mu^{\mathrm{rel}}_{i}$, $k_j$} -\end{algorithmic} -\end{algorithm} - -To determine the corrected mean signal level, we can interpolate the Akima splines to compute a correction: $\mu' = \mu - S_{\mathrm{rel}}(\mu) - S_{\mathrm{abs}}(\mu)$. - -If each amplifier is properly linearized around its true gain, we should observe in every exposure that the illumination level is coherent between amplifiers and detectors. -If the linearization is miscalibrated or biased (either because exposure time is not a good proxy for the true integrated photodiode or if constraining $\langle S(\mu)\rangle = 0$ is biased), the amplifiers and detectors will show signal-dependent fractional offsets in mean signal levels in each flat similar to a mis-estimation of gain. - -We can check how the linearized mean signal levels deviate from an overall linear relation over some set of detectors in a given exposure: -For each exposure $i$, we form the linearized signal, $\mu'_i$, on the reference amplifier on every detector and compute the fractional residual: -\begin{equation} - r_i = \frac{\mu'_i - m_i D_{i}}{y_i}, -\end{equation} -where $m_i$ is the slope of a linear fit to $\mu'_i$ versus $D_{i}$. -We take the median $\langle r \rangle_i$ over a subset of focal-plane detectors (in the center of the focal plane) in that exposure and derive a multiplicative correction -\begin{equation} - N_i = \frac{1}{1 - \langle r \rangle_i}, -\end{equation} that we use to rescale the abscissa ($D_{i} \rightarrow N_i D_{i}$) and enforce that the linearization produces a coherent illumination across all detectors. - -We can then re-compute the double-spline fit with the normalized abscissa on the reference amplifier. We can repeat the cycle (normalize, spline, normalize, etc.) recursively several times, each time computing the normalization as \begin{equation} - N_i = \frac{N_i^{\mathrm{prev}}}{1 - \langle r \rangle_i}. -\end{equation} - -A single iteration improves the precision of the linearity solution by a factor of \textcolor{red}{$N$}, with little further gain from additional iterations.\todo{Fill in $N$ and quantify ``little further gain'' --- e.g.\ the residual RMS after 1, 2, and 3 iterations --- and state how many iterations were used for DP2.} Because the normalization is derived from all the reference amplifiers, our measurement also benefits from the added statistical power which comes from combining measurements across the focal plane to calibrate each amplifier.\todo{Quantify the statistical gain from combining across the focal plane (number of reference amplifiers used and the resulting reduction in normalization uncertainty).} - -In ISR, given an image with pixel values measured in counts ($F_{ij}$), the final spline model can be interpolated, providing a continuous map from measured counts to the corrected counts for any pixel value within the range of 0 signal to the linearity turnoff, computed as: -$F_{ij}' = F_{ij} - S_{\mathrm{rel}}(F_{ij}) - S_{\mathrm{abs}}(F_{ij})$. - -% During science ISR, the linearizer is applied at the non-linearity correction step in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. - - -\subsection{Deferred Charge} \label{sec:processing:calibration:cti} - -Several distinct effects in the readout chain displace charge from its pixel of origin and leave correlated structure in the serial overscan: (1) a pixel-to-pixel fractional loss during charge transfer referred to as ``global CTI'', (2) a local electronic-offset hysteresis in the output amplifier that mimics deferred charge in the first few overscan columns, and (3) localized capture and delayed release of charge by isolated trap sites in the serial register. -Strictly, only global CTI is ``charge transfer inefficiency'' in the textbook sense, but because all three enter at the same point in the photon transfer chain (Figure~\ref{fig:detector-boxes}) and produce overlapping signatures in the serial overscan, we model them together and produce a single calibration data product that we loosely refer to as ``CTI'' in Figure~\ref{fig:calib-dependency}. -These effects are measured, modeled, and corrected together using the extended pixel edge response (EPER) estimator developed for LSSTCam by \citet{2021JATIS...7d48002S}. - -Calibrating deferred charge uses the same input data as the linearity dataset (\S\ref{sec:processing:calibration:linearity}), estimating the residual charge in electrons left in the overscan region after integrating and reading out flat-field images over a wide range of signal levels. -The global CTI is quantified by a unitless fraction, but the trap size and the electronic drift-scale all carry signal-dependent physical units, so an imperfect conversion from counts back to physical electrons biases the eventual correction. -This calculation therefore needs to happen in native electron units, so we need to remove signal-dependent non-linearities that occur when the signal is measured as counts and estimate a gain conversion from counts to electrons. -For each input exposure, we apply an intermediate pass of instrument signal removal that applies defect masking and non-linearity correction, and converts pixel values to electrons, and also measures the overscan EPER statistics for each readout channel. - -The natural way to convert from counts to electrons is to use the gain inferred from the PTC (\S\ref{sec:processing:calibration:ptc}), but the PTC needs to be CTI-corrected to get the most accurate inferred gains, creating what seems like a circular dependency between CTI and PTC calibration production. -We break the cycle by bootstrapping. We create a temporary (intermediate) PTC that is not CTI-corrected, which we fit (see \S\ref{sec:processing:calibration:ptc}) to derive an initial rough estimate of the per-amplifier gain. -That estimate is stored and applied during the ISR of the flat-field images before calculating the EPER statistics. -We find that the gains from a PTC without CTI correction are close enough to the final gains measured from a fully calibrated PTC (typically within 1\% of the true gains). -At this level, the rough gains are accurate enough for CTI calibration. The global-CTI fit is unit-free and so insensitive to any gain error, and the trap and local-offset amplitudes have only weak sensitivity to the few-percent gain bias that remains in the bootstrap estimate.\todo{Quantify ``weak sensitivity'': the propagated change in fitted trap size and drift scale for a 1\% gain error, which is what makes the bootstrap defensible.} -% TODO(number): bootstrap gains are "typically within 1\%" of the final gains two sentences earlier, but here a "few-percent gain bias" remains; the PTC gain adjustments are also quoted as < 0.03. Make the gain-accuracy numbers agree. - -An example of the measured serial EPER statistics as a function of signal level for a single channel readout is shown in Figure~\ref{fig:cti-serial-eper}. -The global CTI produces an overall offset of this curve at all signal levels, the electronic offset produces a linear slope in the curve as a function of signal level, and the serial traps produce non-linear jumps that ``turn on'' above some signal level after a trap fills up and begins releasing charge with a characteristic timescale.\todo{Give typical numbers for the three components: the global CTI offset, the local-offset slope, and the trap turn-on signal level, size, and decay timescale, so the qualitative decomposition is anchored to measured values.} -There is a point at high signal levels at which we can no longer effectively transfer charge pixel to pixel, which we define as the ``serial CTI turnoff'' as indicated in the figure. -We calculate this point as the highest mean signal for which the serial EPER remains consistent with a sigma-clipped linear fit to the low-signal points. For the majority of readout channels, the image region flat-field level was not high enough to observe a serial CTI turnoff, which we then set as the maximum observed signal level.\todo{Replace ``the majority'' with the actual fraction/number of channels for which no turnoff was measured, and give the typical turnoff level for those where it was.} - -\begin{figure}[t] - \centering - \includegraphics[width=\columnwidth]{figures/cti-det93-ampC04-serial-eper.pdf} - \caption{Serial EPER statistics as a function of flat-field signal level for detector/amplifier R22-S10/C04. - Vertical markers indicate the PTC rolloff and turnoff thresholds (described in \S\ref{sec:processing:calibration:ptc}) and the serial CTI turnoff derived from the low-signal trend. - The global CTI fit produces the overall offset of the measured curve at all signal levels.} - \label{fig:cti-serial-eper} -\end{figure} - -The CTI solver simulates charge readout and each of the three effects at the amplifier image level for a given set of parameters. -The model in the simulation is parameterized by a global serial CTI parameter $b$ that is a fractional charge loss per pixel transfer, a local electronic offset described by drift scale $A_L$ and decay time $\tau_L$, and a spline-based serial charge-trap decay model whose size is allowed to depend on the signal in the last imaging column. -We perform a least squares fit that finds the optimal set of parameters that can reproduce the measured overscan statistics trend as a function of signal level. -Then the solver fits these parameters to the overscan image in stages. -First, it varies only the local offset parameters, then the global CTI parameter, and then the trap capture curve from residuals between model and data. -Exposures above configured flux limits are excluded from each stage (typically $10^4$~electrons for the global CTI fit and $1.5\times10^5$~electrons for trap fitting). - -The LSST Science Requirements Document requires that total deferred charge measured by EPER be kept under a total of $5\times10^{-6}$ for signal levels below saturation level on each amplifier (citation needed). -Within the population of amplifiers in active science detectors, only 95 amplifiers do not meet this requirement without ISR correction.\todo{Express this as a fraction of the total amplifier population (95 out of how many?) and by how much those amplifiers exceed the requirement, rather than a bare count; also reconcile this number with the 26 quoted in the next sentence, since it is unclear whether they are subsets of the same population.} -Figure~\ref{fig:cti-global-histogram} shows the distribution of inferred global CTI across all amplifiers on the LSSTCam focal plane, and we show that 26 have a global CTI offset that is already above the maximum allowed threshold. -The majority of these (19) are bad amplifiers that are in the detector corners (C00, C07, C10, C17) positioned near the serial register, and it is possible that what is inferred as global CTI could be a serial trap that turns on at very low signal levels (below the $10^4$ electron level) and appears as an overall offset that is mistaken for global CTI.\todo{Support this hypothesis quantitatively: the trap size that would be needed to mimic the observed offsets, or the measured EPER behaviour of these corner amplifiers below $10^4$\,electrons.} - -\begin{figure}[t] - \centering - \includegraphics[width=\columnwidth]{figures/cti-det93-ampC04-global-cti-histogram.pdf} - \caption{Distribution of inferred global serial CTI across all science sensor amplifiers on the LSSTCam focal plane. - \textit{Blue}---Teledyne e2v (E2V) detectors; \textit{Orange}---Imaging Technology Laboratory (ITL) detectors. - The fit to the deferred charge model allows global CTI to vary up to $1.0 \times 10^{-5}$.} - \label{fig:cti-global-histogram} -\end{figure} -% TODO: Add CTI-corrected versions of Figure~\ref{fig:cti-global-histogram}. - -When correcting deferred charge from an image during ISR, the deferred-charge correction takes in these fit parameters and applies the inverse operators from the model. -The global CTI and local-offset fits, and the inverse correction operators applied during ISR, are the same for both ITL and E2V sensors, but serial traps are fitted only on ITL since it was observed that E2V sensors had small trap size that was difficult to measure beyond noise fluctuations and already negligible enough not to require correction.\todo{Give the upper limit on E2V trap size (in electrons) and the measurement noise floor, so ``negligible'' is a stated bound rather than a judgement; likewise give the typical ITL trap size for contrast.} -After applying a deferred charge correction, we can re-compute the EPER statistics on the corrected dataset and find that all amplifiers meet the science requirements.\todo{Report the post-correction residual deferred charge (median and worst-case) against the $5\times10^{-6}$ requirement --- ``all amplifiers meet'' is much stronger with the margin attached.} - -While there can be deferred charge from a charge transfer in the parallel direction (shift row-to-row), in principle this is a difficult quantity to estimate accurately from parallel overscan statistics. -EPER statistics computed in the parallel overscan region are noisy and reveal no statistically significant charge residuals, so we do not apply any parallel deferred charge correction to science images.\todo{Quote the upper limit this non-detection places on parallel CTI (value and confidence level) and compare it to the requirement, instead of only stating that nothing was significant.} - -% During science ISR, this product is applied at the deferred-charge correction step in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. - - -\subsection{Photon Transfer Curve (PTC)} \label{sec:processing:calibration:ptc} - -The photon transfer curve (PTC) characterizes how pixels respond to varying signal levels in a given wavelength band by measuring covariances between pixels in flat-field exposures across a range of illumination levels. -This dataset provides critical calibration information including amplifier-level read noise and gain for variance plane estimation, saturation thresholds for masking, and parameters needed to correct for non-linearity, charge transfer inefficiency (CTI), and the brighter-fatter effect. -All of these parameters can differ at the amplifier level.\todo{Quantify the amplifier-to-amplifier spread in the key PTC outputs for DP2 (gain, read noise, PTC turnoff): median and range, ideally split by vendor.} - -% The photon transfer curve used to calibrate LSSTCam for DP2 is measured from $543\times2$ flat field exposures at evenly spaced illumination levels as low as 100~analog-to-digital units (ADU) and reaching at least 10\% beyond saturation. -% At each illumination level, a pair of flats was acquired by varying both LED output and exposure time to achieve the target signal. -% We measure the covariance between pixels on the difference image of the two exposures to remove variations sourced by static, fixed-pattern artifacts. - -Let $F_1$ and $F_2$ denote the two flats at a given illumination level, and let $\mu = \mathrm{avg}(F_1,F_2)$ denote the mean signal level (in ADU) over the image and away from the detector edges. -The covariance between a pixel $\mathbf{x}=(0,0)$ and a neighbor at pixel $\mathbf{x}'=(i,j)$ is: -\begin{equation} - C_{ij} \;=\; \tfrac{1}{2}\, - \mathrm{Cov}\!\left(F_1(\mathbf{x})-F_2(\mathbf{x}),\, - F_1(\mathbf{x}')-F_2(\mathbf{x}')\right). -\end{equation} - -% Following the standard front-illuminated CCD convention, serial (row) readout proceeds along $(1,0)$ and parallel (column) readout along $(0,1)$. - -\paragraph{Pre-model fit} -Preliminarily, we fit only the variance to an approximate exponential relation derived in \citet{2019A&A...629A..36A} (their Eq.~16). -\begin{equation} - C_{00}^{\mathrm{exp.\,approx}}(\mu) = \frac{1}{2 g^2 a_{00}}\bigl[\exp(2 a_{00} \mu g) - 1\bigr] + \frac{n_{00}}{g^2}\ . -\end{equation} -In this expression, covariances are contributed by a gain $g$ (electrons$/\mathrm{ADU}$), noise pedestal $n_{00}$ (electrons$^2$), and a single brighter-fatter coefficient $a_{00}$ ($1/$electrons) which has the effect of decreasing the image variance. - -Near saturation, the variance deviates from this functional form, exhibiting a gradual rollover rather than a sharp cutoff due to electrostatic effects that are difficult to model analytically.\todo{Give typical values for $\mu_0$ and $\tau$, and the typical separation between the rolloff and turnoff levels, so ``gradual'' is quantified.} -On top of the exponential approximation, we introduce an empirical saturation term: -\begin{equation} - C_{00}^{\mathrm{sat}}(\mu) = \left\{ - \begin{array}{ll} - -e^{-(\mu-\mu_0)/\tau} & \mu \ge \mu_0 \\ - 0 & \mu < \mu_0 - \end{array} - \right., -\end{equation} -where $\mu_0$ marks the onset of the PTC ``rolloff'' effects and $\tau$ controls the shape of the rollover. -The total preliminary fit model is $C_{00}^{\mathrm{prelim.}}(\mu) = C_{00}^{\mathrm{exp.\,approx}}(\mu) + C_{00}^{\mathrm{sat}}(\mu)$ to the unweighted points. This fit is performed iteratively with additional 3~sigma clipping until convergence. -The ``PTC turnoff'' is defined as the signal level of the last unmasked point following the fit to only the exponential approximation, typically corresponding to the maximum of the variance curve and generally higher than the rolloff threshold. -The ``PTC rolloff'' is the signal level where $C_{00}^{\mathrm{prelim.}}(\mu)$ and $C_{00}^{\mathrm{exp.\,approx}}$ deviate by more than 1\%. - -\paragraph{Final model fit} -With the remaining points below the PTC rolloff, we fit the complete covariance expansion from \citet{2019A&A...629A..36A} (their Eq.~20): -\begin{equation} - \begin{aligned} - C_{ij}^{\mathrm{model}}(\mu) = \frac{\mu}{g} \Bigl[ \delta_{i0}\delta_{j0} - &+ a_{ij}\mu g \\ - &+ \frac{2}{3}\left([\mathbf{a}\otimes\mathbf{a} + \mathbf{ab}]_{ij}\right)(\mu g)^{2} \\ - &+ \frac{1}{6}\left(2\mathbf{a}\otimes\mathbf{a}\otimes\mathbf{a} + 5\mathbf{a}\otimes\mathbf{ab}\right)_{ij}(\mu g)^{3} \\ - &+ \mathcal{O}[(\mu g)^4] - \Bigr] + \frac{n_{ij}}{g^2}, - \end{aligned} - \label{eq:cov_model} -\end{equation} -where $a_{ij}$ (electrons)$^{-1}$ represents the fractional change in pixel area per stored charge due to boundary motion from the brighter-fatter effect, $b_{ij}$ (electrons)$^{-1}$ captures weaker time-dependent corrections, $g$ is the gain (electrons$/\mathrm{ADU}$), and $n_{ij}$ is the constant noise contribution (electrons)$^2$ (e.g., $n_{00}\sim (5.7\,\mathrm{electrons})^{2}$ on the diagonal, with off-diagonal terms determined by read electronics and overscan). -The symbol $\mathbf{ab}$ denotes element-wise matrix multiplication, and $\otimes$ represents discrete convolution. -In practice, the $b_{ij}$ parameter is inferred to be just noise around a value of zero.\todo{State the constraint: the measured $b_{ij}$ scatter and the corresponding upper limit, and the size of $b_{ij}$ relative to $a_{ij}$ that would matter for the correction.} The size of the covariance matrix ($n \times n$) is determined by the range of the brighter-fatter effect (see \S\ref{sec:processing:calibration:bf}). -Figure~\ref{fig:ptc-fit-example} shows an example fit for amplifier~C03 on detector~94, illustrating the measured $C_{00}$ points, the preliminary exponential approximation with saturation roll-off, and the final variance from Eq.~\ref{eq:cov_model}. - -\begin{figure*}[t] - \centering - \includegraphics[width=\textwidth]{figures/ptc-triangle-det94-ampC03.pdf} - \caption{Example PTC fit for amplifier~C03 on LSSTCam detector~94. - Measured $C_{00}$ covariance points are compared to the preliminary fit (exponential approximation plus saturation roll-off) and the final full-covariance model variance. - Vertical markers indicate the PTC rolloff and turnoff thresholds.} - \label{fig:ptc-fit-example} -\end{figure*} - -Because the gains are fit independently per amplifier, small fitting residuals can leave flux discontinuities at amplifier boundaries within a detector.\todo{Quantify the discontinuity before adjustment (percent flux step across an amplifier boundary) and the statistical uncertainty on the per-amplifier gain fit that produces it.} -We add a subsequent step that measures the relative offsets between amplifiers in flat fields after gain correction and computes small adjustments to the gain to make the detector image continuous while preserving the average gain across the detector. -We compute the average gain adjustment in flat fields across a range of signal levels, and use these to produce overall ``adjusted gains''. -The adjustments to the gain are always small (less than 0.03), and uncorrelated between detectors.\todo{Clarify the units of the 0.03 (electrons/ADU or fractional?) and give the distribution --- median and 95th percentile --- rather than only a bound; also quantify ``uncorrelated'' with a correlation coefficient if that claim is load-bearing.} -These final gains are used for conversion of science images to electron units and construction of the variance plane for source detection. - -The PTC calibration products are applied during instrument signal removal of science images in three ways. -First, the gain converts pixel values from ADU to electrons. -Second, the gain and read noise inform variance plane estimation. -Third, the saturated pixel mask flag (denoted ``SAT'') is set for all pixels with values greater than or equal to the PTC turnoff threshold. - -% During science ISR, PTC-derived gains, variance-plane inputs, and saturation masking are applied at the steps shown in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. - - -\subsection{Brighter-Fatter Effect (BFE)} \label{sec:processing:calibration:bf} - -The brighter-fatter effect (BFE) is the first influence the detector has on an incident signal (Figure~\ref{fig:detector-boxes}). -Accumulated electrons in a pixel repel incoming charges into neighboring pixels. -Brighter point sources therefore appear wider in the recorded images. -This electrostatic effect reduces to an effective change in pixel area and introduces correlations between nearby pixel values. -The effect is intrinsic to thick CCDs and can significantly bias point-spread-function (PSF) photometry and shape measurement.\todo{Quantify ``significantly'': the uncorrected PSF size growth with source flux (e.g.\ percent change in second moments per decade of flux) and the resulting photometric/shape bias, relative to the LSST requirement.} - -The BFE has been extensively characterized for LSSTCam by laboratory and in-dome commissioning measurements \citep{2024PASP..136d5003B}. -The strength of the effect is anisotropic ($a_{01}/a_{10} \sim 3$), long-ranged ($\sim 10$~px), and wavelength dependent. -It is also about twice as strong in the E2V-family detectors as in ITL-family detectors. - -Several BFE correction methods have been proposed and implemented in the LSST Science Pipelines \citep{2015JInst..10C5032G,2018AJ....155..258C,2023A&A...670A.118A,2024PASP..136d5003B}, and they differ widely in their underlying assumptions. -\citet{2024PASP..136d5003B} found that modeling individual boundary shifts for every pixel (N, S, E, W) is required for adequate correction at the level needed for LSST Y10 science requirements. -Flat-field pixel correlations can measure only the net per-pixel area change, so determining these boundary shifts requires additional electrostatic modeling. -For processing raw images from DP2, we use the \citet{2023A&A...670A.118A} algorithm developed for Hyper Suprime-Cam, which derives from electrostatic physics and treats each pixel boundary individually. - -% The correction assumes three simplifying conditions. -% First, the average charge current into a pixel is approximately constant during a long exposure, even when the PSF varies with time. -% Second, fourth-order terms in Eq.~\ref{eq:cov_model} are negligible; at the scale of the BFE, the corresponding coefficients are of order $10^{-24}$. -% Third, photons convert to electrons quickly after entering the detector. -% Simulations and empirical modeling indicate that most photons in $u$ through $i$ bands convert within the top few microns of silicon, while the BFE influence on charge drifting to the pixel bottom occurs primarily in the last several microns. -% Chromatic effects are therefore likely negligible for photons with wavelengths across the LSST bandpasses. - -\begin{deluxetable}{ll}[t!] - \tablecaption{Parameter Priors for the Electrostatic BFE Model Fit\label{tab:bf-electrostatic-fit-priors}} - \tablehead{ - \colhead{Parameter} & \colhead{Prior} - } - \startdata - Si thickness, $t$ ($\mu$m) & $100$ (fixed) \\ - Pixel size, $p$ ($\mu$m) & $10$ (fixed) \\ - Charge cloud height, $z_q$ ($\mu$m) & $0 \leq z_q \leq t/2$ \\ - Charge cloud length, $a$ ($\mu$m) & $10^{-5} \leq a \leq 0.35\,p$ \\ - Charge cloud width, $b$ ($\mu$m) & $10^{-5} \leq b \leq 0.35\,p$ \\ - Potential well vertex height,\\horizontal $z_{sh}$ ($\mu$m) & $0 \leq z_{sh} \leq t/2$ \\ - Potential well vertex height,\\vertical $z_{sv}$ ($\mu$m) & $0 \leq z_{sv} \leq t/2$ \\ - \enddata - \tablecomments{Priors on the fit parameters are informed by pixel-level simulations of the electric fields in these pixels \citep{2017JInst..12C3091L}. The potential well vertices refer to the heights above the front-side of the CCD to the saddlepoints in the electric field that defines the pixel boundaries in the vertical (0, 1) and horizontal (1, 0) directions. The electrostatic model assumes that the stored charge cloud in the sensor takes the form of a flat, rectangular sheet of charge with length ($a$) and width ($b$). The inequality constraints $z_{sh} > z_q$, $z_{sv} > z_q$ are enforced during optimization, which is true in the unsaturated signal level regime.} -\end{deluxetable} - -Calibration of the BFE using this algorithm takes the form of modeling a charge cloud at the bottom of a pixel, computing -how the effective areas of nearby pixels would be expected to change electrostatically in response. It adjusts some geometric parameters (summarized in Table~\ref{tab:bf-electrostatic-fit-priors}), and solves -Gauss's law to compute the shift of pixel boundaries and the relative change in pixel area from each shift ($\mathbf{a}^{\mathrm{N}}$, -$\mathbf{a}^{\mathrm{S}}$, $\mathbf{a}^{\mathrm{E}}$, $\mathbf{a}^{\mathrm{W}}$). -We optimize these parameters to best fit the -measured net pixel area changes ($\mathbf{a}$ from Eq.~\ref{eq:cov_model}): -\begin{equation} - \mathbf{a} = \mathbf{a}^{\mathrm{N}} + \mathbf{a}^{\mathrm{S}} + \mathbf{a}^{\mathrm{E}} + \mathbf{a}^{\mathrm{W}}. -\end{equation} - -In principle, the BFE can vary amplifier-to-amplifier, but we have observed only noise-level variations in the pixel distortion matrices across amplifiers and even detector families.\todo{Give the amplifier-to-amplifier scatter in $a_{01}$/$a_{10}$ against the per-amplifier measurement uncertainty, which is the quantitative basis for fitting at the detector level; note this also appears to sit in tension with the statement above that the effect is twice as strong in E2V as in ITL.} -For this reason, and to avoid discontinuous amplifier boundaries, the fit is performed at the detector level. -All $a_{ij}$ matrices from well-behaved amplifiers are 3$\sigma$-clipped elementwise and averaged into a detector-mean $a_{ij}$; the scatter becomes the per-element uncertainty, and a single fit is run against this average pixel distortion matrix. -The output calibration data product is the set of model parameters and a model that can be evaluated per-detector. - -Applying this calibration to remove the effect during ISR uses the boundary-shift matrices $\mathbf{a}^{\mathrm{N}}$, $\mathbf{a}^{\mathrm{S}}$, $\mathbf{a}^{\mathrm{E}}$, and $\mathbf{a}^{\mathrm{W}}$ to redistribute flux accordingly. -We tile these into two, $(2n+1)\times(2n+1)$, convolution kernels $K^{H}$ and $K^{V}$ for the horizontal and vertical shifts, respectively; we apply sign flips and edge adjustments so each matrix sums to zero. -These matrices need not be symmetric, which is more realistic per \citet{2024PASP..136d5003B}. -Figure~\ref{fig:ebf-kernels} shows an example pair of these kernels for amplifier~C03 on detector~94, illustrating the dipole-like, charge-conserving structure that redistributes flux across the vertical and horizontal pixel boundaries. - -\begin{figure}[t] - \centering - \includegraphics[width=\columnwidth]{figures/ebf-det94-ampC03-kVH.pdf} - \caption{Example BFE correction kernels $K^{V}$ (vertical) and $K^{H}$ (horizontal) derived from the electrostatic BFE model for amplifier~C03 on LSSTCam detector~94. - The diverging color scale shows the fractional charge redistributed across pixel boundaries; each kernel sums to zero to conserve charge and exhibits the antisymmetric dipole structure that moves flux from the brighter pixel to its neighbors.} - \label{fig:ebf-kernels} -\end{figure} -For an image $Q_{ij}$ in natural gain-corrected units of electrons, the amount of charge lost over the vertical and horizontal boundaries is: -\begin{align} - \delta^{H}_{ij} &= \tfrac{1}{2}\!\!\sum_{kl}\!\!K^{H}_{kl}\,Q_{i-k,\,j-l}, \notag \\ - \delta^{V}_{ij} &= \tfrac{1}{2}\!\!\sum_{kl}\!\!K^{V}_{kl}\,Q_{i-k,\,j-l}, - \label{eq:bf-shift} -\end{align} -The $1/2$ prefactor accounts for the average BFE assuming a constant rate of charge accumulation during an exposure. -The corrected image is assembled by removing the charge each pixel lost across its $N$/$E$ boundaries and adding what it gained from $S$/$W$ neighbors, guaranteeing charge conservation by construction, $\sum_{ij}Q^{\mathrm{corr}}_{ij}=\sum_{ij}Q_{ij}$. -This correction does not require iterative convolutions. Unlike the \citet{2018AJ....155..258C} algorithm used for DP1 \citep{2026AJ....171..360V}, which requires multiple convolutions to reach convergence, we perform exactly two convolutions with the horizontal and vertical boundary-shift matrices.\todo{Quantify the gain from this: the typical number of iterations the DP1 algorithm required, and the resulting speedup in per-detector ISR wall-clock time --- and, if available, whether correction quality changed.} - -% During science ISR, the BFE correction is applied at the step shown in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. - - -\subsection{Combined Bias} \label{sec:processing:calibration:bias} - -A bias image is a zero-second exposure read out with the shutter closed, and so records the electronic pedestal of the readout chain in the absence of any accumulated charge. -Overscan correction removes the component of that pedestal that varies from exposure to exposure, while the combined bias measures the static two-dimensional electronic structure that remains and is subtracted from every science image. - -To remove this remnant signal that is not already removed by the previous calibrations, a combined bias frame is subtracted from the exposure. -This combined bias is constructed from the per-pixel clipped mean of a large number of input zero-second bias frames.\todo{Quantify the residual static structure the combined bias removes: its peak-to-peak and RMS amplitude in electrons, so the reader knows the size of the signal being corrected.} -These input bias frames have all of the corrections discussed up until this point applied. -For LSSTCam, these combined biases have 400 input exposures, which reduces the noise in the correction.\todo{State the resulting noise: with 400 frames the combined-bias noise is read noise$/\sqrt{400}$ --- give that value in electrons and compare it to the single-frame read noise, so the choice of 400 is justified numerically.} -CZW: quick study to see if we have any with fewer than 400 exposures. - -\subsection{Combined Dark} \label{sec:processing:calibration:dark} - -A dark image is an exposure taken with the shutter closed, and measures the signal that accumulates in proportion to integration time rather than to incident light. It is dominated by absorption from thermally generated infrared photons, called dark current, together with any glow from the readout electronics.\todo{Give the measured dark current level (electrons/pixel/s, median and range across the focal plane) and the amplitude of the readout glow, plus how much signal that amounts to in a 30\,s science exposure.} -These artifacts are removed by scaling the dark to the exposure time of the science image and subtracting it. - -Similar to the bias, we construct a combined dark frame from a large number of 30-second exposures, with that time chosen to match the standard exposure time of a science exposure. -As before, all of the previous calibrations are applied to these dark frames, including the bias correction. -These exposures have an additional cosmic ray rejection step applied after the ISR processing, as these longer exposures may have more of these features than are found on the bias frames.\todo{Quantify the cosmic-ray rate (events or affected pixels per detector per 30\,s exposure) and the fraction of pixels rejected, instead of ``may have more''.} -The combined dark is again created from the per-pixel clipped mean, and for LSSTCam we have a target of 400 input frames. -CZW: I know we don't always hit that, put words here. - -\subsection{Combined Flats} \label{sec:processing:calibration:flats} - -A flat field measures the relative response of the system to uniform illumination, capturing both pixel-to-pixel variations in quantum efficiency and larger-scale structure imposed by the optics, the filter transmission, and vignetting of the focal plane. -Dividing a science exposure by the combined flat normalizes this response, so that a source of a given flux produces the same measured signal regardless of where it lands on the focal plane. - -Although flat-field images are a common astronomical correction, the large field-of-view of LSSTCam makes generating this calibration a particular challenge. -In addition to ensuring consistent illumination across this field-of-view, there is an added complication that the ITL and E2V detectors have different responses as a function of wavelength.\todo{Quantify the ITL/E2V response difference (the fitted ratio per band, which the six-node spline pass already measures) rather than only noting that it exists.} -One final complication comes from the finite choices of LEDs in the flat-field illumination system described in Section \ref{sec:data:calsys:flat_field_illuminator}. -We anticipate that we will be able to construct higher quality synthetic flats from sequences of laser-illuminated monochromatic flats in the future, for DP2 we have chosen to construct ``anaglyph'' flats for most bands.\todo{Quantify what the two-LED anaglyph approximation costs relative to a true synthetic flat: the residual chromatic error expected (in mmag across the focal plane), which is what ``higher quality'' should mean here.} - -These anaglyph flats superimpose a red-side and a blue-side flat to construct a single flat across a full LSST bandpass. For a given filter, we combine a pair of flat-field images that have each been observed with a single active LED illuminating the flat-field screen. -For the subsequent discussion, these two sets of input flat-field images are referred to as the ``red'' and ``blue'' exposures, depending on the wavelength of the two LEDs within the filter bandpass that final flat field will be used for. -These anaglyph flats have been constructed for each of the $grizy$ filters. -This process is unavailable for the $u$ filter since a sufficiently blue ultraviolet LED was not available, nor was there space to fit one inside the projection system, and so flats for this filter are constructed from only a single set of flats (with an LED that is at the red end of the bandpass). -A bluer ultraviolet LED will be available to be periodically swapped in and out for DP1 calibrations. - -To begin the construction of new flats, the input exposures are processed through the ISR steps that are applied prior to flat correction, including the bias, dark, crosstalk, gain application, linearization, CTI, and overscan (CZW: reorder to be in order). -Statistics are measured on these processed exposures, determining the median value and dispersion on each detector. -Normalization factors are then measured for the red and blue inputs separately based on these statistics, which should account for any differences in the absolute flux received in each exposure over all detectors. -These normalization factors are used to properly scale the inputs before they are combined using a clipped mean per pixel into separate intermediate red and blue flats. - -These intermediate flats include gradients related both to how well the flat-field illumination and screen were oriented and due to the vignetting of the focal plane itself.\todo{Give the amplitude of each gradient separately (percent across the focal plane for the planar illumination gradient, and the vignetting depth at the outer radius), so the reader can see which dominates.} -Both types of gradients are measured next, so that contribution can be removed from the final flat. -An initial six-node radial spline model is fit to the full binned focal plane image of the intermediate flats. -This initial model is used to constrain both the centroid of the focal plane and the observed ratio of the ITL to E2V detectors, to account for their possibly different response. -These parameters are then used for a 15-node radial spline model that is more careful about fitting the observed vignetting at high focal plane radii.\todo{Quantify what the 15-node model buys over the 6-node one (residual RMS for each), and state the radius beyond which vignetting becomes significant.} -This second fit uses a fixed ITL/E2V ratio measured during the first pass, to reduce the degrees of freedom in the final fit. -An additional planar gradient is also included in this second fit, to account for simple illumination differences. - -The measured gradients are then removed from the intermediate flats. -The planar gradients are simply divided from the per-detector intermediate flats. -The radial gradient is scaled by a reference gradient to reduce the effects of the vignetting, and to match the vignetting observed in on-sky data (CZW: I need to explain reference gradient better).\todo{State how well the flat-derived and on-sky vignetting agree after this scaling (residual percent as a function of focal-plane radius) --- this is the key quantitative test of the flats.} -A final normalization factor is applied to scale the entire flat to match the intensity level observed within the central 25mm of the focal plane. -With these gradient-corrected intermediate flats generated, a final flat is constructed by combining the red and blue intermediates with a filter-specific mixture weight, as listed in Table \ref{tab:anaglyph_ratios}. - -\begin{table}[] - \begin{tabular}{lll} - Band & Blue weight & Red weight \\ - $g$ & 0.63 & 0.37 \\ - $r$ & 0.50 & 0.50 \\ - $i$ & 0.476 & 0.524 \\ - $z$ & 0.65 & 0.35 \\ - $y$ & 0.52 & 0.48 \\ - \end{tabular} - % TODO(ref): \label comes before \caption, so \ref{tab:anaglyph_ratios} resolves to the section number (5.11 in the .aux) instead of the table number; move the \label after the \caption. - \label{tab:anaglyph_ratios} - \caption{Mixture weights for anaglyph flat construction.\todo{Say how these weights were derived (matching the LED-weighted effective wavelength to the band's effective wavelength?) and give the uncertainty on each weight plus the sensitivity of the resulting flat to a weight error.}} -\end{table} diff --git a/sections/5_discussion.tex b/sections/5_discussion.tex deleted file mode 100644 index 30408be..0000000 --- a/sections/5_discussion.tex +++ /dev/null @@ -1,7 +0,0 @@ -\section{Discussion} \label{sec:discussion} -Describe how calibrations/ISR have changed across different calibration epochs (sequencer changes). - -List the set of criteria that prompt a new round of calibrations. - -% The datasets discussed in this paper were collected before the dome was completed, before stabilization of thermal regulation systems, and before finalization of the precise detector operating mode. - diff --git a/sections/7_conclusion.tex b/sections/conclusion.tex similarity index 85% rename from sections/7_conclusion.tex rename to sections/conclusion.tex index c96a01c..a9faa1d 100644 --- a/sections/7_conclusion.tex +++ b/sections/conclusion.tex @@ -1,19 +1,19 @@ \section{Conclusion} \label{sec:conclusion} -This paper documents the calibration-production algorithms, instrument signal removal (ISR) workflow, and quality-assurance framework used to generate DP2 calibration products for LSSTCam. -DP2 is the first end-to-end demonstration of data processing with an on-sky LSSTCam dataset, and it establishes a reproducible reference for the first stage of processing in Data Release Processing (DRP) pipeline. +This paper documents the calibration-production algorithms, instrument signature removal (ISR) workflow, and quality-assurance framework used to generate DP2 calibration products for LSSTCam. +DP2 is the first end-to-end demonstration of data processing with an on-sky LSSTCam dataset, and it establishes a reproducible reference for the first stage of processing in the Data Release Processing (DRP) pipeline. We organize calibration production and ISR around the electronic detector model introduced in \S\ref{sec:lsstcam:detector-model}, where each instrument artifact gets its own associated calibration dataset and correction algorithm. Calibrations are built on top of other calibrations (summarized in Figure~\ref{fig:calib-dependency}), requiring intermediate applications of ISR (summarized in Figure~\ref{fig:isr-pipeline}). In general, the pipelines are derived and applied in the reverse chronological order of the photon transfer chain. For DP2, we present the science ISR pipeline (Figure~\ref{fig:isr-pipeline}, \S\ref{sec:isr-science}) and the production algorithms for crosstalk, defect masking, non-linearity, deferred charge (CTI), the photon transfer curve (PTC), the BFE, and the classical bias, dark, and flat-field calibrations (\S\ref{sec:calibration-production}). Where detailed validation is already in place, DP2 calibrations meet or approach the requirements needed for commissioning-era science. -These products are accompanied by a verification, acceptance, and certification workflow (\S\ref{sec:verify-accept-certify}) intended to make calibration quality auditable, and we record the relevant provenance in exposure dataset metadata for the science community (\S\ref{sec:isr-science:provenance}). +These products are accompanied by a verification, acceptance, and certification workflow (\S\ref{sec:verify-accept-certify}) intended to make calibration quality auditable, and we record the relevant provenance in exposure dataset metadata for the science community (\S\ref{sec:isr-science}). -DP2 nevertheless reflects commissioning-era conditions: calibration data were collected before dome completion and full thermal stabilization (\S\ref{sec:intro}), detector operating modes and sequencer configurations evolved across epochs (\S\ref{sec:discussion}), and calibrations and correction tools for some artifacts remain under active development (\S\ref{sec:limitations}). +DP2 nevertheless reflects commissioning-era conditions: calibration data were collected before dome completion and full thermal stabilization (\S\ref{sec:intro}), detector operating modes and sequencer configurations evolved across epochs (\S\ref{sec:discussion:sequencer}), and calibrations and correction tools for some artifacts remain under active development (\S\ref{sec:discussion:limitations}). Users should treat residual focal-plane non-uniformities, epoch-dependent systematics, and incomplete coverage effects as known limitations when interpreting DP2 data products and deciding paths/configurations for science analyses. -Looking toward Data Release~1 (DR1), we expect iterative improvements in three areas: \emph{algorithms} (refined CTI and trap modeling, expanded color-dependent BFE treatment), \emph{operations} (more stable dome and calibration-system performance, clearer calibration-epoch boundaries, and expanded automation through verification and acceptance pipelines), and \emph{diagnostics} (trending analyses, regression and alerts for calibration changes against baselines, and quality metrics on science-images for downstream pipeline acceptance criteria). +Looking toward Data Release~1 (DR1), we expect iterative improvements in three areas: \emph{algorithms} (refined CTI and trap modeling, expanded color-dependent BFE treatment), \emph{operations} (more stable dome and calibration-system performance, clearer calibration-epoch boundaries, and expanded automation through verification and acceptance pipelines), and \emph{diagnostics} (trending analyses, regression and alerts for calibration changes against baselines, and quality metrics on science images for downstream pipeline acceptance criteria). The DP2 workflow documented here is the baseline from which those improvements will be measured; maintaining versioned, reproducible calibration production will remain essential as LSSTCam transitions from data previews during commissioning to data releases over the full 10-year survey. \vskip 0.4in diff --git a/sections/data.tex b/sections/data.tex new file mode 100644 index 0000000..ba4ad3b --- /dev/null +++ b/sections/data.tex @@ -0,0 +1,48 @@ +\section{Calibration System (CalSys)}\label{sec:calsys} +Each instrumental signature imprinted on the data along the photon transfer chain (Figure~\ref{fig:detector-boxes}) is characterized by a dedicated set of calibration exposures, which we use to build the calibration products that ISR applies to remove it. +Calibration campaigns produce zero-second bias frames, dark exposures, and flat fields to characterize spatially static image artifacts and a set of flat fields taken over a wide range of illumination levels to characterize signal-dependent effects. +We also produce images with projected spot patterns to characterize the image response to focused and high-contrast light. +The quality and reproducibility of these products depend both on the illumination hardware and on our control of the camera and calibration systems. + +In this section we describe the calibration systems used to acquire these data in the dome during commissioning. +There are two calibration systems that will be used continuously throughout the LSST: the Flat-Field Illuminator and the Collimated Beam Projector. + +\subsection{Flat-Field Illuminator}\label{sec:calsys:flat-field-illuminator} + +The Flat-Field Illuminator is designed to provide a stable, uniform, and well-characterized +illumination required to measure the absolute and relative instrumental responses of the LSST Camera. + +The physical architecture of the flat-field system is driven by the fast ($f/1.2$) optical configuration +of the Simonyi Survey Telescope, which demands exceptional control over both the angular and spatial +uniformity of the calibration wavefront \citep{TSTN-060}. This system aims to satisfy a photometric +precision of 5 mmag in repeatability and 10 mmag in spatial +uniformity across the field of view. The system directs light toward a +large 10.27-meter-diameter Lambertian calibration screen that is tilted to 23 +degrees to align precisely with the optical axis of the telescope. Because the calibration hardware must +co-rotate dynamically with the observatory dome during operations, the primary illumination +sources, environmental enclosures, and projection optics are mounted directly onto the dome structures +rather than onto a static mount \citep{TSTN-057}. + +To deliver both broadband and monochromatic illumination during the LSST, the system integrates two +independent illumination subsystems within a single centralized optical projector, positioned at the +geometric center of the calibration screen structure. Broadband ``white-light'' illumination is provided +by a specialized module featuring a collection of 10 high-power LEDs selected to span the complete +$ugrizy$ filter passbands, with one LED within the $u$ passband and two LEDs within each of the +other five passbands. For monochromatic calibration, the system incorporates a tunable laser +system that provides continuous wavelength coverage from 300 to 2600 nm at a 1 kHz pulse rate. This +laser can deliver narrow tuning steps between 0.05 and 1 nm and a spectral line width under 1 nm FWHM \citep{TSTN-065}. + +% The data products generated by these dual operating modes support separate but complementary calibration goals within the LSST DM Science Pipelines, to ensure uniformity across the LSSTCam focal plane. +% In the DP2 dataset, the flat fields used during ISR are constructed with a weighted combination of the two LEDs within the passband (or the single LED in the case of $u$-band). +% These data products enable the correction of detector-level systematics, including gain variations, charge transfer inefficiencies, pixel-level defects, and sensor distortions induced by the brighter-fatter effect (BFE). +% The monochromatic flat-field data products from the tunable laser are not used in calibration production for DP2, but will be utilized in future data releases to map the system's chromatic throughput as a function of focal plane position. + +\subsection{Collimated Beam Projector}\label{sec:calsys:cbp} + +The Collimated Beam Projector (CBP) is a separate, in-dome photometric calibration instrument to +measure the system throughput of focused light to $\mathcal{O}(\text{mmag})$ precision \citep{2026arXiv260631822A}. +The CBP consists of a modified Schmidt reflector telescope fed by the tunable laser to produce a +collimated beam of known flux. By co-moving the CBP and Simonyi Survey Telescope, we are able to +place this beam anywhere on the primary mirror and have a focused spot anywhere on the focal plane. +The CBP can switch between five different masks at the focal plane, allowing for different spot +patterns to be projected onto the focal plane \citep{2026arXiv260631822A}. \ No newline at end of file diff --git a/sections/discussion.tex b/sections/discussion.tex new file mode 100644 index 0000000..fdb0211 --- /dev/null +++ b/sections/discussion.tex @@ -0,0 +1,67 @@ +\section{Discussion} \label{sec:discussion} + +\subsection{Limitations \& Known Issues} \label{sec:discussion:limitations} + +The ISR pipeline developed for LSSTCam removes the dominant instrumental signatures, but several effects remain partially corrected, unmodeled, or unmasked. +The paragraphs below summarize known limitations (instrumental systematics not fully corrected, edge cases, validity constraints, etc.) for interpreting DP2 data, our recommendations for handling them in the context of DP2 science, and plans for near-term improvements expected before Data Release~1 (DR1). + +\paragraph{Flares} +Some science images contain localized, bright flare-like structures. +They appear as elongated trails that begin at a detector edge and fade with a wavelength-dependent power-law profile consistent with attenuation of light in silicon at that wavelength. +The light is likely scattered from bright stars: focused starlight may reflect from the beveled edge of the silicon mount and enter a neighboring detector through its side, although this mechanism is not yet confirmed. +Similar structures appear in some flat-field images as many small edge flares (referred to as a ``comb'' pattern), possibly linked to the rough-cut silicon edge. +So far, these features have been seen only in $y$-band, with both focused and unfocused illumination, and they are transient, consistent with a dependence on the chance alignment of bright stars with the focal-plane array. +They are not yet fully characterized or masked in DP2 processing. +Science users should treat anomalously bright detections/extended objects near detector edges with caution in $y$-band single-visit and difference images, and verify suspect sources by visual inspection. + +\paragraph{Residual non-linearity wiggles} +The double-spline non-linearity model (\S\ref{sec:calibration-production:linearity}) removes the dominant signal-dependent departure from linear response, but small quasi-oscillatory residuals remain as a function of signal level. +They appear as low-amplitude, signal-dependent structure in linearized images. +These ``wiggles'' are irregular, clearly observed beyond noise fluctuations, and they do not correlate with measured properties of the input flat-field images (e.g., time, temperature) or with channel on a detector or across the focal plane. + +\paragraph{Differential non-linearity (DNL)} +The DP2 linearizer corrects average amplifier non-linearity but does not yet apply a full low-signal differential non-linearity correction. +This limitation reflects the difficulty of delivering flat-field illumination below 100 ADU/pixel with the in-dome calibration system. +Users should exercise caution when relying on photometric or astrometric measurements at low signal levels, in particular when relying on $u$-band sky-background estimates from later processing steps, where backgrounds are typically at or below 100 ADU/pixel. +Count dithering in ISR (\S\ref{sec:isr-science}) provides interim mitigation until a dedicated DNL correction is measured and deployed. + +\paragraph{Detector vignetting models} +Flat-field calibrations measure relative pixel and filter quantum efficiencies and throughputs under uniform illumination, but they require an accurate model of optical vignetting across the focal plane. +Improved vignetting models, informed by larger calibration datasets and cross-checks against on-sky stellar photometry, are planned for future calibration epochs to reduce focal-plane non-uniformity. + +\paragraph{Color-dependent BFE corrections} +The BFE is color dependent because its amplitude depends on photon conversion depth in silicon and on the path length over which the transverse electric field acts on drifting electrons. +Laboratory and commissioning measurements show that the effect is chromatic \citep[$a_{00}^{y-band} = 0.94 \times a_{00}^{r-band}$, see][]{2024PASP..136d5003B}. +The electrostatic BFE model used in DP2 (\S\ref{sec:calibration-production:bf}) is fit from PTC covariances and applies a single wavelength-averaged correction per detector, derived in the band of the PTC input flats. +Because the effect is strongest near the bottom of the potential well and the front of the chip, calibrations for $u$ through $i$ bands are approximately consistent, as most light in those bands converts to photoelectrons within the first few microns. +Separate calibrations are appropriate for $z$- and $y$-band light, which typically convert far deeper into the silicon. +Boundary-shift calibrations can be color-corrected by assuming a flat incident spectral energy distribution in each filter band and a depth-conversion model in silicon. +These corrections were not enabled in DP2 because validation was incomplete, but they are expected for DR1. +Until then, PSF photometry and shape measurements of bright sources, especially in redder bands, may retain small flux- and filter-dependent biases. + +\paragraph{Fringing} +Fringing is an interference pattern produced when narrow-band emission reflects within the thin, back-illuminated silicon of LSSTCam CCDs. +It is driven primarily by night-sky emission lines and becomes appreciable in the $z$- and $y$-bands, where the silicon is increasingly transmissive and OH and other atmospheric lines contribute strongly to the background. +In-dome flat-field exposures (\S\ref{sec:calibration-production:flats}) imprint and partially divide out a fringe pattern, but the fringing amplitude and phase in science images depend on the time-varying on-sky spectrum and therefore are not fully canceled by flat-fielding alone. +The DP2 ISR pipeline (\S\ref{sec:isr-science}) does not include a dedicated fringe-subtraction step. +Residual fringe structure is visible at low surface brightness in $z$- and $y$-band visit images and can bias wide-aperture photometry. +\citet{2023PASP..135c4503G} predict a fringing amplitude of 0.04\% to 0.2\% in LSST images, depending on the location of a CCD on the focal plane, and a corresponding surface-brightness variation of about 0.6~$\mu$Jy\,arcsec$^{-2}$ in $y$-band sky-background images. +Corrections based on their fringing models are under development, so users analyzing faint extended emission or precision surface-brightness measurements in redder bands should inspect visit images for quasi-periodic fringe patterns and treat affected regions with appropriate caution. + +\subsection{Calibration Sensitivity to Camera Sequencer} \label{sec:discussion:sequencer} +Because many instrument signatures depend on the electronic configuration of the sensor and the timing of the readout, a change to the sequencer can invalidate calibration products built under the previous one. +\citet{2026arXiv260631806M} describe how the LSSTCam sequencer has evolved since the camera arrived at the summit. +The most significant on-sky change was the move to a more relaxed total focal plane readout time from 2\,s to 3\,s. +The extra time was spent on a longer per-row readout interval, while the other sequencer parameters tuned during laboratory testing were left unchanged. +This lowered the read noise of the E2V sensors by about 0.5\,e$^-$, while the ITL sensors were largely unaffected \citep{2026arXiv260631806M}. + +DP2 includes science data taken under two sequencers: the original 2\,s \texttt{2s\_v30} sequencer (epoch~1) and the \texttt{3s\_v1} sequencer (epochs~2 and 3; Figure~\ref{fig:calibepoch}). +Each sequencer required its own set of calibrations. +The move to 3\,s readout changed the read noise and bias structure measured in the overscan, and it also changed the size of the crosstalk signal, so the crosstalk coefficients had to be measured again (\S\ref{sec:calibration-production:xtalk}). +A further update, \texttt{3s\_v2}, was introduced after the DP2 data cutoff on 2026 January 6. +Throughout DP2, detector~101 (R23\_S02) showed signal-dependent correlated noise. +This noise interfered with the photon transfer curve calibration (\S\ref{sec:calibration-production:ptc}) and with the measurement of both gain and read noise for that detector. +The correlated noise disappeared with the move to the \texttt{3s\_v2} sequencer. +No \texttt{3s\_v2} data or calibrations are included in the DP2 release, but \texttt{3s\_v2} is the sequencer used for survey operations from now on. +For that reason, the calibration products shown in this paper are the \texttt{3s\_v2} products. +The algorithms are the same for all sequencers; only the input exposures and the resulting calibration products differ. diff --git a/sections/1_intro.tex b/sections/intro.tex similarity index 63% rename from sections/1_intro.tex rename to sections/intro.tex index 8198c51..49c49d3 100644 --- a/sections/1_intro.tex +++ b/sections/intro.tex @@ -1,21 +1,20 @@ \section{Introduction} \label{sec:intro} -The Vera C.\ Rubin Observatory will conduct the 10-year Legacy Survey of Space and Time \citep[LSST,][]{2019ApJ...873..111I}, imaging roughly 18,000~deg$^{2}$ of the southern sky in six broadband, optical--near-infrared (NIR) filters ($ugrizy$). -Achieving the LSST science goals requires calibrating the instrument with high precision and uniformity so that raw detector readouts can be transformed into science-ready images and catalogs. +The Vera C.\ Rubin Observatory will conduct the 10-year Legacy Survey of Space and Time \citep[LSST,][]{2019ApJ...873..111I}, imaging roughly $18{,}000~\mathrm{deg}^{2}$ of the southern sky in six broadband, optical--near-infrared (NIR) filters ($ugrizy$). +Achieving the LSST science goals \citep{LPM-17} requires calibrating the instrument with high precision and uniformity so that raw detector readouts can be transformed into science-ready images and catalogs. Errors or non-uniformities in image calibration can introduce survey-scale photometric and astrometric systematics that propagate into downstream measurements and may be degenerate with astrophysical or cosmological signals \citep{2023PASP..135k5003E}. -Key challenges to delivering this level of calibration include: +Key challenges to delivering the necessary level of calibration include: \begin{enumerate} \item \textbf{Instrument complexity:} The LSST Camera (LSSTCam) is the largest digital camera built for astronomy. Its 3.2-gigapixel focal plane consists of 205 fully depleted, backside-illuminated charge-coupled devices (CCDs) from two vendors (Teledyne e2v and Imaging Technology Laboratory, hereafter E2V and ITL). - % TODO(number): 205 CCDs here vs. 189 science CCDs plus 3 corner CCDs x 4 corners = 201 in \S\ref{sec:lsstcam}; also, 3.2 Gpix matches the 189 science CCDs (189 x 4096^2 ~ 3.2 Gpix), not 205. Detector and amplifier variations require calibration and removal algorithms robust to heterogeneous sensor behavior. - \item \textbf{Data volume and cadence:} LSSTCam typically acquires 30~s exposures with 3~s readout, yielding about 1000 exposures per night and nightly data rates of approximately 20~TB. The need for fast, efficient processing speed is driven by the need for prompt alert production and the active optics system. - \item \textbf{Calibration system and optics:} The flat-field illuminator (flat-field screen, LEDs, tunable laser) needs to provide uniform and stable illumination over the large focal plane, and the Collimated Beam Projector (CBP) needs to provide a capability to measure total system throughput at any given location of the focal plane in a fast converging ($f/1.2$) optical system. - \item \textbf{Coupled instrumental effects:} Instrument signal removal (ISR) must correct artifacts in the reverse chronological order in which they imprint in the photon transfer chain. + \item \textbf{Data volume and cadence:} LSSTCam typically acquires 30~s exposures with 3~s readout, yielding about 1000 exposures per night and nightly data rates of approximately 20~TB. The need for fast, efficient processing is driven by prompt alert production and the active optics system. + \item \textbf{Calibration system and optics:} The Flat-Field Illuminator (flat-field screen, LEDs, tunable laser) needs to provide uniform and stable illumination over the large focal plane, and the Collimated Beam Projector (CBP) needs to provide a capability to measure total system throughput at any given location of the focal plane in a fast converging ($f/1.2$) optical system. + \item \textbf{Coupled instrumental effects:} Instrument signature removal (ISR) must correct artifacts in the reverse chronological order in which they imprint in the photon transfer chain. Calibration derivation follows the same ordering, since characterizing a given artifact requires that all effects imprinted before it have already been calibrated and removed. This is further complicated by partial degeneracies among instrument artifacts, which necessitate iterative or ``bootstrapped'' approaches to converge on self-consistent calibration products. - \item \textbf{Evolving conditions and cadence:} Changing environmental conditions, hardware and operation mode updates, and evolving observing strategies requires detailed daily tracking of calibration quality and well-defined conditions for determining new calibration epochs across all sensors. + \item \textbf{Evolving conditions and cadence:} Changing environmental conditions, hardware and operation mode updates, and evolving observing strategies require detailed daily tracking of calibration quality and well-defined conditions for determining new calibration epochs across all sensors. \item \textbf{Verification, provenance, and operational reproducibility:} Calibration products must be accompanied by quantitative diagnostics for rapid data quality assessment, cross-time and cross-version comparisons, and clear acceptance and certification decisions for downstream processing. Products must be reproducible, versioned, and continuously validated as instruments and algorithms evolve. \end{enumerate} @@ -23,17 +22,17 @@ \section{Introduction} \label{sec:intro} The LSST Data Management (DM) Science Pipelines \citep{PSTN-019} implement this functionality using production pipelines and supporting tools to generate, verify, and publish calibration products for routine processing. The design benefits from lessons learned in LSST Data Preview 1 \citep[DP1;][]{2026AJ....171..360V} and precursor surveys, including the Hyper Suprime-Cam Subaru Strategic Program \citep{2012SPIE.8446E..0ZM,2022PASJ...74..247A,2018PASJ...70S...5B} and the Dark Energy Survey \citep{2005astro.ph.10346T,2012PhPro..37.1332T,2018PASP..130g4501M}. -The first end-to-end demonstration of Rubin data processing on a substantial, internally consistent LSSTCam dataset is \emph{Data Preview 2} \citep[DP2;[]{RTN-117}. +The first end-to-end demonstration of Rubin data processing on a substantial, internally consistent LSSTCam dataset is \emph{Data Preview 2} \citep[DP2;][]{RTN-115}. DP2 covers about $2{,}500~\mathrm{deg}^{2}$ in all six filters and includes commissioning and pre-survey operations data. This paper describes the calibration-production algorithms and the operational approach used to generate DP2 calibration products. Our goals are to (1) make the DP2 calibration products interpretable to science users, (2) quantify our confidence in the quality of released data products, (3) make the processing reproducible for pipeline developers and operators, and (4) provide a reference point for expected improvements on the path to the first LSST public data release. This paper is organized as follows. -Section \S\ref{sec:lsstcam} introduces LSSTCam, its CCD readout architecture, and the electronic detector model that organizes calibration production and ISR. -Section \S\ref{sec:data} describes the DP2 calibration datasets, the calibration system (CalSys), and the criteria used to select input exposures and define calibration epochs. -Section \S\ref{sec:isr-science} describes how those products compose during instrument signal removal for DP2 science data. -Section \S\ref{sec:calibration-production} describes calibration product production and application, including the algorithms and data products used in DP2. -Section \S\ref{sec:discussion} discusses calibration-epoch differences and residual systematics relevant to DP2. -Section \S\ref{sec:verify-accept-certify} describes how calibration products are verified, accepted, certified, and deployed, and \S\ref{sec:limitations} summarizes known limitations for DP2. -Section \S\ref{sec:conclusion} summarizes the DP2 calibration quality and outlines expected improvements toward Data Release~1 (DR1). +Section~\ref{sec:lsstcam} introduces LSSTCam, its CCD readout architecture, and the electronic detector model that organizes calibration production and ISR. +Section~\ref{sec:calsys} describes the DP2 calibration datasets, the calibration system (CalSys), and the criteria used to select input exposures and define calibration epochs, including the sequencer changes that separate them. +Section~\ref{sec:isr-science} describes how those products compose during instrument signature removal for DP2 science data. +Section~\ref{sec:calibration-production} describes calibration product production and application, including the algorithms and data products used in DP2. +Section~\ref{sec:verify-accept-certify} describes how calibration products are verified, accepted, certified, and deployed, and what triggers a new set of calibrations. +Section~\ref{sec:discussion:limitations} summarizes known limitations for DP2. +Section~\ref{sec:conclusion} summarizes the DP2 calibration quality and outlines expected improvements toward Data Release~1 (DR1). diff --git a/sections/lsstcam.tex b/sections/lsstcam.tex new file mode 100644 index 0000000..e5badd5 --- /dev/null +++ b/sections/lsstcam.tex @@ -0,0 +1,106 @@ +\section{The LSST Camera (LSSTCam)} \label{sec:lsstcam} + +The design of the LSSTCam \cite{10.71929/rubin/2571927} was developed in order to meet the strict requirements set by the ambitious goals of LSST \citep{2019ApJ...873..111I, RTN-011}, as described in \cite{2024SPIE13096E..1SR}. + +% Focal plane +The LSSTCam focal plane (Figure~\ref{fig:focal-plane}) is a mosaic centrally composed of 189 thick, fully depleted charge-coupled devices (CCDs) used for science imaging, organized into 3$\times$3 rafts (described in more detail in \citet{CTN-001}). +Each raft is composed of three Readout Electronics Boards (REB) providing the power and electronics of three neighboring CCDs in a row. +The corners of the mosaic contain two CCDs for guiding and one split-wavefront CCD offset intra- and extra-focally relative to the science CCDs. + +\begin{figure*}[t] + \centering + \includegraphics[width=\textwidth]{figures/lsstcam_focal_plane.pdf} + \caption{LSSTCam focal plane layout in focal-plane coordinates and field angle, following \citet{CTN-001}. + Each sensor is labeled with its sensor slot (S\#\#) and detector number, and each raft with its designation (R\#\#); the thin lines within each sensor delimit its readout channels. + Detector 122 (R30\_S12) is not operational (cross-hatched) and was not turned on for the duration of the commissioning and pre-survey operation periods.} + \label{fig:focal-plane} +\end{figure*} + +% CCDs +LSSTCam CCDs are read out in 16 channels, each with its own readout electronics and signal amplifiers. +The imaging region of each channel is 2002 pixels high and 512 pixels wide for E2V sensors and 2000 pixels high and 509 pixels wide for ITL sensors, giving a full imaging area of $4096 \times 4004$ pixels (E2V) and $4072 \times 4000$ pixels (ITL). +Charge is shifted down the columns (the \emph{parallel} direction) into a serial register, which carries each row to the amplifier (the \emph{serial} direction). +During readout, each row begins with serial prescan columns and ends with serial overscan columns, and the image ends with parallel overscan rows, so that the raw readout of every channel is 2048 rows $\times$ 576 columns for both vendors (Table~\ref{tab:amp-geometry}). + +\begin{table}[t] + \centering + \small + \caption{Readout geometry of one LSSTCam amplifier channel, in pixels. + Sizes are given as rows (parallel direction) $\times$ columns (serial direction).} + \label{tab:amp-geometry} + \begin{tabular}{lcc} + \hline + Region & ITL & E2V \\ + \hline + Image area & $2000 \times 509$ & $2002 \times 512$ \\ + Serial prescan (columns) & 3 & 10 \\ + Serial overscan (columns) & 64 & 54 \\ + Parallel overscan (rows) & 48 & 46 \\ + \hline + Raw channel & $2048 \times 576$ & $2048 \times 576$ \\ + \hline + \end{tabular} +\end{table} + +The CCD clocking pattern, or sequencer, sets the timing of every parallel transfer, serial transfer, and sampling step during readout. +In order to minimize survey overhead time, the full focal plane is read out in under 3 seconds. However, data collected during DP2 +include science data taken under two different sequencers, an earlier 2\,s readout version (\texttt{2s\_v30}) and a later 3\,s readout version (\texttt{3s\_v1}), as shown in Figure~\ref{fig:calibepoch}. +After the DP2 data taking period, we switched to a slightly different sequencer (\texttt{3s\_v2}). Despite the fact that no data in DP2 were collected with the camera operating under the \texttt{3s\_v2} +sequencer, all the calibrations and metrics shown in this paper were derived with the \texttt{3s\_v2} sequencer since it shows the current on-sky performance of the LSSTCam and the current state of calibration and ISR. +We will discuss the differences between these sequencers and their impact on the camera performance in the Discussion (Section~\ref{sec:discussion:sequencer}). + +\begin{figure*}[t] + \centering + \includegraphics[width=0.75\textwidth]{figures/calibration-epochs-timeline.pdf} + \caption{Timeline of the observing periods and calibration epochs for LSSTCam during commissioning and pre-survey operations included in DP2. + The timeline begins with LSSTCam going on-sky on 2025 April 15; the first science data included in DP2 were taken on 2025 April 24. + DP2 science data are separated into calibration epochs, which are set by changes to the CCD operating mode (sequencer) or by power cycles for maintenance. + Blue boxes mark the interval covered by each calibration epoch and are labeled with the sequencer in use: epoch~1 (from 2025 April 24, \texttt{2s\_v30}), epoch~2 (2025 July 3 to September 21, \texttt{3s\_v1}), and epoch~3 (from 2025 October 24, \texttt{3s\_v1}). + Epoch~4 (from 2026 January 26, \texttt{3s\_v2}) follows the DP2 data cutoff on 2026 January 6. + The calibrations shown in this paper are the most recent (\texttt{3s\_v2}) set, even though no data from this epoch were released in DP2. + The dotted segment marks an engineering and maintenance downtime between 2025 September 21 and October 24 that is not covered by a calibration epoch. + The feature freeze for v30 of the LSST Data Management (DM) Science Pipelines software was 2025 December 3.} + \label{fig:calibepoch} +\end{figure*} + +\subsection{Physical Model of Photon Transport in LSSTCam CCDs} \label{sec:lsstcam:detector-model} + +Instrumental signatures imprinted by the detectors and readout electronics cause each recorded pixel value to depart from the incident signal. +Removing these signatures requires a model of the full signal chain, which describes how photons are converted to charge, how that charge is transferred and amplified, and how the resulting voltage is digitized. +We built this model through electro-optical characterization on the laboratory test stand at SLAC National Accelerator Laboratory, at the summit, and on the telescope during commissioning. +\citet{2024SPIE13096E..1SR} give a detailed account of the laboratory campaigns and their results. + +The model also sets the structure of the calibration and instrument signature removal (ISR) pipeline. +Each artifact enters at a specific point along the signal chain, so ISR removes them in the reverse order. +It starts with the last effect imposed, at digitization, and ends with the effects that act on the charge in the silicon. +Each step therefore relies on a physical model of the effect itself. +Figure~\ref{fig:detector-boxes} shows where each artifact enters the chain, from the detector to the REB. + +\begin{figure*}[t] + \centering + \includegraphics{figures/calibration_boxes_detector_model.pdf} + \caption{Photon transfer and data-acquisition model for the LSSTCam detectors and readout electronics board (REB), containing the 18-bit analog-to-digital converter (ADC) at the end of the readout chain \citep{SITCOMTN-086}. + Instrumental effects seen in images are shown as dashed boxes; arrows indicate where each effect enters the chain. + Low-signal non-linearities in LSSTCam's detector response have been observed, but their origin is not yet established; that contribution appears as a disconnected box.} + \label{fig:detector-boxes} +\end{figure*} + +We consider effects that enter the signal chain as soon as the photon enters the silicon detector, where incident photons are converted to electrons that collect in the potential wells of pixels. +During the integration of an exposure, dark current adds charge that is thermally generated rather than photo-generated \citep{2024SPIE13096E..1SR}, and the brighter-fatter effect redistributes incoming charge to neighboring pixels as the wells fill \citep{2014JInst...9C3048A, 2024PASP..136d5003B}. +Additionally, changes in dopant concentration throughout the silicon bulk cause lateral electric fields that deflect the path of electrons as they drift to the potential wells, producing a fixed pattern of astrometric distortion called ``tree rings''. +At the reddest wavelengths, mainly in the $z$- and $y$-bands, light reflects off the surfaces of the 100\,$\mu$m-thick sensors before it is absorbed, and the resulting thin-film interference produces a quasi-periodic pattern of bright and dark fringes whose amplitude and phase depend on the spectrum of the incident light \citep{2023PASP..135c4503G}. + +When the integration stops, the readout process begins, and the charge packets are then shifted down the columns by parallel transfer and along the serial register toward the amplifier, where a fraction of each packet is left behind as charge transfer inefficiency (CTI) or caught in serial traps in the output register and released at a later time \citep{2021JATIS...7d48002S}. + +At the output node of the detector's chip, the on-chip amplifier converts charge to voltage with gain $g_1$, and from this point the signal is carried as a voltage rather than electrons. +The signal then passes through cabling to the REB, where crosstalk couples signals between neighboring amplifiers and across the detector midline \citep{2015JInst..10C5010O}, after which the analog signal processing integrated circuit (ASPIC) applies a second, temperature-dependent gain ($g_2$). +The dual-slope integrator then samples each pixel twice, once at the reset level and once at the signal level, and outputs their difference scaled by a third gain, $g_3$ \citep{SITCOMTN-086}. +Two distinct electronic offsets are added to the signal along this chain, and we keep them separate (Figure~\ref{fig:detector-boxes}). +The clock-injected offset is an unintended signal: transitions of the parallel and serial clocks couple into the video signal near the output amplifier, adding an offset of up to $\pm10$k ADU that depends on position in the readout sequence and produces the 2D bias structure and the bias ``wave'' \citep{2026arXiv260631806M}. +Bias injection, by contrast, is a deliberate stage of the REB: after the dual-slope integrator, a fixed DC pedestal of $\sim$10--20k ADU is added so that the signal, including negative excursions from noise and the clock-injected offset, stays within the input range of the ADC. +Finally, the analog-to-digital converter (ADC) maps the analog signal to digitized counts, but it introduces non-linearities in the response at high, mid, and low signal levels \citep{RTN-055}. + +The overall gain $g$ describes the net conversion from electrons to digital ADU. +Altogether, the signal processing chain can be summarized as a change of units of the image from electrons to counts. +Detector artifacts along the chain make this conversion ill-defined. +The goal of ISR is to recover the true incident signal by undoing these effects in reverse order, each in its native units. diff --git a/sections/processing.tex b/sections/processing.tex index b0884d3..68e6350 100644 --- a/sections/processing.tex +++ b/sections/processing.tex @@ -1 +1,729 @@ -\input{sections/processing/detector-model} +\section{Instrument Signature Removal for DP2 Science Data} \label{sec:isr-science} + +Instrument signature removal (ISR) is defined by the steps that transform the raw integer image into a floating-point, fully calibrated image in electron units with an associated variance plane, pixel masks, provenance, and metadata used in downstream data release processing tasks. +% Since this process undoes instrument artifacts that enter into the image in the photon transfer chain, it is crucial for the calibration steps to match the physics of the detectors. + +Corrections are applied in reverse chronological order relative to the photon transfer chain (Figure~\ref{fig:detector-boxes} in \S\ref{sec:lsstcam:detector-model}). +For example, steps that undo readout-electronics effects appear before steps that undo detector-level effects, and additive corrections precede multiplicative ones such as flat-fielding. + +Figure \ref{fig:isr-before-after} shows one E2V and one ITL detector from the same exposure before and after instrument signature removal. +The raw frames retain the serial and parallel overscan, which are virtual pixels read to estimate charge transfer inefficiency (CTI); see \S\ref{sec:calibration-production:cti} for more details. +%here add more about electron transportation and size of overscan region and serial register. + +\begin{figure*}[t] + \centering + \gridline{\fig{figures/isr_2026010700038_det45_raw.pdf}{0.52\textwidth}{(a) E2V, raw} + \fig{figures/isr_2026010700038_det45_afterisr.pdf}{0.48\textwidth}{(b) E2V, post-ISR}} + \gridline{\fig{figures/isr_2026010700038_det79_raw.pdf}{0.52\textwidth}{(c) ITL, raw} + \fig{figures/isr_2026010700038_det79_afterisr.pdf}{0.48\textwidth}{(d) ITL, post-ISR}} + \caption{One LSSTCam exposure (2026010700038) before and after instrument signature removal (ISR), shown for one detector from each vendor: E2V detector 45 (R12\_S00, top row) and ITL detector 79 (R20\_S21, bottom row). + The left panels (a, c) show the raw readout in analog-to-digital units (ADU), with the serial (solid outline) and parallel (dashed outline) overscan regions of each of the 16 amplifiers marked. + The right panels (b, d) show the same detectors after ISR, in electrons, with the overscan trimmed and the mask planes set during processing overlaid in color (see legend). + All panels share the same pixel scale.} + \label{fig:isr-before-after} +\end{figure*} + +Running the processing pipeline with misplaced steps would either apply a correction in the wrong physical units or leave residual structure that invalidates downstream calibrations. +Figure~\ref{fig:isr-pipeline} shows the resulting sequence of corrections as applied to LSSTCam science images for DP2. + +\begin{figure*}[t] + \centering + \includegraphics[width=0.9\textwidth]{figures/isr-pipeline.pdf} + \caption{Outline of the instrument signature removal (ISR) pipeline applied to LSSTCam science images for DP2. + The steps in the pipeline follow a very specific sequence determined by the physics of the instrument signature and where it enters in the photon transfer chain. + Blue boxes denote static ISR steps that apply pre-computed calibration inputs; orange boxes denote steps whose application depends on the science exposure itself (for example, overscan subtraction, dithering, or saturation-footprint growth); white boxes mark the raw input and post-ISR output. + A dashed border marks a computation rather than an artifact correction (integer-to-float conversion, dithering, overscan statistics, and variance plane estimation). + The amplifier-offset step remains optional and applies pre-computed per-amplifier offsets when enabled.} + \label{fig:isr-pipeline} +\end{figure*} + +The following outline follows Figure~\ref{fig:isr-pipeline} step by step. +Each item names the correction applied and the reason it sits at this location in the processing chain. + +\begin{enumerate} + \item \textbf{Convert integer to float.} + Raw images arrive as integer arrays. This first step promotes pixel values to 64-bit floating-point precision. + + \item \textbf{Dither counts.} + A uniform random offset in $[-0.5,\,0.5)$~ADU is added to every pixel. The variance added by a uniform distribution of width 1~ADU is $1/12$~ADU$^2$. + This breaks the correlation between true signal and integer quantization before continuous corrections are applied, reducing systematic bias when later steps interpolate, convolve, or divide the image. + This step is essentially a marginalization step before a full low-signal differential non-linearity correction can be measured and implemented. + + \item \textbf{Serial overscan correction.} + The serial overscan records the total electronic offset of each row, which combines two contributions that originate differently (\S\ref{sec:lsstcam:detector-model}): the deliberate DC pedestal added by the REB's bias injection stage, and the unintended clock-injected offset from clock transitions coupling into the video signal. + This combined offset, plus smaller row-to-row variations, is derived from the serial overscan and subtracted from the imaging pixels while the data are still in native analog-to-digital units (ADU). + The spatially static part of the clock-injected structure that the overscan does not capture is removed later by the combined bias. + Overscan correction must precede gain conversion because the overscan bias is measured and removed in the same units as the raw readout. + At this stage we also calculate overscan statistics, including an estimate of the read noise from the scatter in overscan pixels and the amount of deferred charge from extended pixel edge response (EPER) statistics (\S\ref{sec:calibration-production:cti}). + + \item \textbf{Mask saturated and suspect pixels.} + Pixels above the photon transfer curve (PTC) turnoff saturation threshold (\S\ref{sec:calibration-production:ptc}) are flagged as both saturated (``SAT'') and suspect (``SUSPECT'') in the pixel bit mask before further signal corrections. + Early masking prevents corrupted or non-physical pixel values from biasing downstream estimates that assume a well-behaved signal. + + \item \textbf{Apply gain.} + Per-amplifier gains derived from the PTC (\S\ref{sec:calibration-production:ptc}) convert pixel values from ADU to electrons. + Electron units are the native domain for deferred-charge modeling, BFE correction, and variance plane estimation, all of which were calibrated assuming a linear conversion from measured counts to stored charge. + + \item \textbf{Crosstalk correction.} + Electrical coupling between adjacent amplifier outputs is removed using the crosstalk matrix (\S\ref{sec:calibration-production:xtalk}). + Crosstalk is applied after gain conversion so that coupled signals are corrected in electron units, consistent with how the matrix was measured. + + \item \textbf{Parallel overscan correction.} + For each column, the parallel overscan pixels are averaged over the overscan rows, and this offset is subtracted from every imaging pixel in that column. + It follows crosstalk correction because amplifier coupling can inject signal into overscan regions, and removing crosstalk first yields a cleaner overscan bias estimate. + + \item \textbf{Non-linearity correction.} + The double-spline linearizer (\S\ref{sec:calibration-production:linearity}) maps measured counts to a linearized electron count by subtracting absolute and amplifier-relative spline corrections. + This step must precede deferred-charge correction (\S\ref{sec:calibration-production:cti}) because the deferred-charge model and its calibration were fit assuming that signal-dependent readout non-linearities have already been removed. + + \item \textbf{Deferred charge correction.} + Residual charge left in pixels during serial readout---from global charge transfer inefficiency, local electronic offset in the output amplifier, and serial charge traps on ITL sensors---is corrected using the parameters fit in \S\ref{sec:calibration-production:cti}. + This step sits late in the readout-electronics portion of the chain because deferred charge is imprinted during serial transfer, after non-linearities in the output path have acted on the signal. + We do not correct deferred charge on E2V sensors because the deferred charge level is already below the $5\times10^{-6}$ threshold required by the LSST Science Requirements Document \citep{LPM-17}, as described in \S\ref{sec:calibration-production:cti}. + + \item \textbf{Assembly.} + At this point, we no longer need the overscan regions, so the final detector is assembled by trimming the prescan and overscan regions and mosaicking the trimmed amplifiers into a contiguous pixel array. + + \item \textbf{E2V edge-bleed mask.} + We flag the ``SAT'' mask plane bit for pixels in E2V amplifiers that show uncorrectable charge spill from bright regions at the serial-register edge. + + \item \textbf{ITL dip, SatSag, and edge-bleed masks.} + We flag the ``ITL\_DIP'' mask plane bit for pixels in ITL amplifiers that exhibit a pixel column depression in response near certain amplifier boundaries (the ``ITL dip''), a sag in apparent signal after saturation (``SatSag''), and edge bleeding analogous to the E2V case. + These vendor-specific masks follow deferred-charge correction because the artifacts they mark are readout- and saturation-related structures that should be evaluated on linearized, CTI-corrected data. + + \item \textbf{Bias correction.} + A combined bias frame (\S\ref{sec:calibration-production:bias}) is subtracted to remove the fixed-pattern electronic offset that remains after overscan subtraction. + Overscan correction handles the spatially static electronic structure imprinted in the images before charge readout. + + \item \textbf{Dark correction.} + The combined dark frame (\S\ref{sec:calibration-production:dark}), scaled by the dark time of the science exposure, is subtracted to remove thermal dark current accumulated while charge is collected. + The dark time is slightly longer than the exposure time (by nearly one second for typical exposures) because the detector also accumulates dark current during the interval between the end of the clear and the start of readout, beyond the time the shutter is open. + Dark correction follows bias subtraction because both are additive offsets, and the dark frame is constructed from bias-subtracted dark exposures. + + \item \textbf{Defect masking.} + Persistent bad pixels, columns, and other static defects identified during calibration (\S\ref{sec:calibration-production:defects}) are merged into the pixel mask. + Applying the defect mask after deferred-charge, bias, and dark correction places it once readout-electronics effects and integration-period offsets have been removed, matching the processing state in which the defect catalog was built and ensuring that later defect footprint widening and interpolation operate on the full static bad-pixel map. + + \item \textbf{Widen saturation footprints.} + The saturation mask is expanded spatially to include pixels affected by charge spill and trails from saturated sources. + For DP2, the amount of widening depends on the width of the saturated region. + Regions narrower than 6 pixels are left unchanged, regions 6 or 7 pixels wide are expanded by 1 pixel on each side, regions 8 or 9 pixels wide are expanded by 2 pixels on each side, and any wider region is expanded by 4 pixels on each side. + This step follows defect masking so that all static bad regions are known before the saturation footprint is grown into neighboring pixels. + + \item \textbf{Brighter-fatter correction.} + The electrostatic BFE (\S\ref{sec:calibration-production:bf}) redistributes flux among neighboring pixels to undo charge-cloud repulsion in the silicon. + In the photon transfer model (Figure~\ref{fig:detector-boxes}), the BFE is the earliest detector-level effect on incident photons, so its correction appears correspondingly late in ISR. + It is applied after the image has been converted to linearized electron units and after additive calibration-frame corrections. + The correction also needs to match the boundary shifts, which were also calibrated from PTC covariances measured after these corrections. + + \item \textbf{Variance plane estimation.} + An expected variance is assigned to each pixel using the PTC-derived gain and read noise (\S\ref{sec:calibration-production:ptc}). + The variance plane is computed after gain application and BFE correction so that it reflects the electron units and pixel correlations relevant to source detection and weighting in downstream analyses. + + At this stage the expected per-pixel variance follows the Poisson-plus-read-noise model + \begin{equation} + \sigma^2_{ij} = \frac{F_{ij}}{g_a} + \left(\frac{\sigma_n(a)}{g_a}\right)^2, + \label{eq:isr-variance} + \end{equation} + where $F_{ij}$ is the science image value in the current ISR units, $g_a$ is the PTC gain for amplifier~$a$ (electrons per ADU), and $\sigma_n(a)$ is the read noise in electrons. + For DP2 science processing, the image is already in electron units, so $g_a = 1$ and Eq.~\ref{eq:isr-variance} reduces to $\sigma^2_{ij} = F_{ij} + \sigma_n(a)^2$. + Pixels with non-positive variance are flagged in the mask. + + Since the variance plane is used to estimate the uncertainty on just the charge accumulated in the image, it needs to be computed before flat-field correction. + + \item \textbf{Flat correction.} + The normalized flat field (\S\ref{sec:calibration-production:flats}) is divided into the image to correct pixel-to-pixel variations in quantum efficiency and optical throughput. + Flat-fielding is a multiplicative correction that must be applied only after all additive electronic and dark offsets have been removed, so that the flat field encodes relative sensitivity rather than residual bias structure. + + \item \textbf{Interpolate masked pixels.} + Pixels in the ``SAT'', ``BAD'', and ``UNMASKEDNAN'' mask planes are filled row by row, interpolating from the nearest unmasked values in the same row; ``SUSPECT'' pixels are flagged but not interpolated by default. + Interpolation is performed after flat-fielding so that both the science signal and the replacement values are in the same calibrated flux units expected by algorithms that assume a continuous image (for example, background estimation and certain astrometric measurements). + + \item \textbf{Amplifier offsets.} + Small per-amplifier additive adjustments derived from flat-field residuals are applied as a final step to reduce residual flux discontinuities at amplifier boundaries. + The additive correction is typically less than 1 electron, and the code caps it at a maximum of 5 electrons. + The offset correction addresses any remaining channel-to-channel offsets visible after the full ISR chain. +\end{enumerate} + +Several of these steps flag pixels in the mask plane rather than changing their values. +Table~\ref{tab:isr-mask-planes} summarizes the mask planes set during ISR, which downstream tasks use to exclude or down-weight affected pixels in source detection, measurement, and image coaddition. + +\begin{deluxetable*}{ll}[t!] + \tablecaption{Mask Planes Set During ISR\label{tab:isr-mask-planes}} + \tablehead{ + \colhead{Mask Plane} & \colhead{Description} + } + \startdata + BAD & Static defects (\S\ref{sec:calibration-production:defects}), fully defective amplifiers, and pixels with non-positive variance \\ + SAT & Pixels above the PTC turnoff (\S\ref{sec:calibration-production:ptc}), plus widened saturation trails, edge bleeds, and ITL saturation sag \\ + SUSPECT & Pixels above the suspect level, set to the PTC turnoff (\S\ref{sec:calibration-production:ptc}), plus ITL dip pixels \\ + ITL\_DIP & Columns with depressed response on ITL sensors (the ``ITL dip'') \\ + CROSSTALK & Pixels whose crosstalk source pixel (\S\ref{sec:calibration-production:xtalk}) exceeds $45{,}000$ electrons \\ + UNMASKEDNAN & Non-finite pixel values not already masked \\ + EDGE & The ``picture frame'' of pixels around the detector perimeter \\ + INTRP & Pixels whose values were replaced by interpolation \\ + \enddata +\end{deluxetable*} + +The output exposure's metadata records the calibration epoch, the identifier of the collection used, and a per-step record of which ISR corrections were applied or skipped. +This provenance makes it possible to reproduce the processing of any released image, compare products generated across software or calibration versions, and audit the configurations enabled for a given dataset. +The verification, acceptance, and distribution context for these metadata fields is described in \S\ref{sec:verify-accept-certify:data-products}. + +\section{Calibration Production and Application} \label{sec:calibration-production} +% former label: sec:processing:calibration + +Some artifacts removed in the ISR sequence described in \S\ref{sec:isr-science} require a calibration data product that characterizes the instrumental effect being removed, measured in advance from dedicated calibration exposures rather than from the science image itself. +In this section, we describe in detail the algorithms used to produce calibrations and apply them to an image in ISR. +Figure~\ref{fig:calib-dependency} summarizes calibration-product dependencies. + +\begin{figure}[t!] + \centering + \includegraphics[width=\columnwidth]{figures/calib-dependency.pdf} + \caption{Calibration production dependency graph. + Arrows indicate data or dependency flow (``x $\rightarrow$ y'' means ``y depends on x''). + Solid outlined boxes represent datasets that are deployed/released with data products, while dashed boxes represent internal or intermediate datasets that are not formally released to users.} + \label{fig:calib-dependency} +\end{figure} + +% NOTE: order follows calib-dependency figure + +\subsection{Curated Calibrations} \label{sec:calibration-production:curated-calibs} + +\subsection{Auxiliary Data Products} \label{sec:calibration-production:auxiliary-data-products} + +\subsection{Crosstalk} \label{sec:calibration-production:xtalk} +Large astronomical sensors are read out through multiple amplifiers in parallel (see Section~\ref{sec:lsstcam}), reducing the time needed to completely read the sensor. +One downside of this design strategy is the possibility of inducing crosstalk between the different amplifier channels. +In LSSTCam, this crosstalk is produced by electronic coupling that occurs in at least two different places in the readout hardware: (1) between the channels in the cable connecting the detectors to the REB and (2) across opposite sides of each detector with neighboring analog signal-processing integrated circuits (ASPICs). +The result of these couplings is that the data for a particular amplifier show a scaled copy of neighboring amplifiers added to the image.\todo{Give the typical and maximum crosstalk coefficient magnitudes measured for LSSTCam (e.g.\ $\sim10^{-4}$ nearest-neighbour, $<10^{-5}$ elsewhere), and how the two coupling mechanisms differ in strength.} +As all amplifiers are clocked through the readout process simultaneously, this scaled copy is aligned with the location of the readout corners. +This may make the crosstalk signal visible in the target amplifier flipped relative to the source amplifier. +These two sources cannot be disentangled from one another, so we measure and correct both forms of crosstalk as a single artifact. + +Correcting for this additional signal is easy with a known set of crosstalk coefficients between a given source and target amplifier. Each source amplifier that has a non-zero coefficient is flipped to match the target amplifier, scaled by that coefficient, and added to the results from all other source amplifiers to create a subtrahend image that contains all of the crosstalk signal observed on the target amplifier. +By repeating this for all amplifiers, we can build up a full-detector subtrahend image that can then be subtracted. +As this correction may be imperfect, we set a CROSSTALK mask plane for all locations where the subtrahend image is greater than 1 electron.\todo{Quantify ``may be imperfect'': the residual crosstalk after correction (as a fraction of the uncorrected signal, cf.\ Figure~\ref{fig:crosstalk-linear-nonlinear-fit}) and the typical fraction of pixels flagged CROSSTALK in a science image.} +This provides a way for the downstream object detection and measurement code to identify sources that may have been significantly altered by the crosstalk correction. + +As this crosstalk occurs in the cable connecting the detector to the REB, prior to being digitized into ADU, the images need to be gain corrected prior to the crosstalk correction to ensure they are in electron units. +This naturally places the crosstalk correction early in the ISR processing. This is also important because very bright sources can create bleed trails that deposit charge that is not fully clocked out, and that slowly leaks out as subsequent rows are read out. +These bleed trails can extend into the parallel overscan region, which can in turn create crosstalk signal in neighboring amplifiers. +Performing crosstalk correction prior to overscan correction prevents these amplifiers from oversubtracting the affected columns. + +The crosstalk coefficients used during commissioning were calibrated from lab measurements at SLAC prior to the camera being shipped to the summit and were quadratic with the source flux, as discussed in \citet{2021JATIS...7d48002S} (CZW: do we need to introduce the measurement process first?). +However, the switch from 2\,s readouts to 3\,s readouts via a camera sequencer change invalidated these coefficients, as the change in clocking also changed the scale of the crosstalk signal, requiring this calibration to be remeasured.\todo{State by how much the coefficients changed between the 2\,s and 3\,s sequencers (a typical ratio or percent shift), which is what justifies the remeasurement.} + +This measurement used three nights (CZW: check numbers in this paragraph) of spot data taken with the CBP system. +Each exposure placed one spot on each detector, with ten linearly spaced flux levels (CZW: add figure or describe levels) per pointing. +After the flux sequence was completed, the CBP spots were shifted to land on a different amplifier, with the process repeated in order to provide each amplifier a bright source to use to look for crosstalk signals. +A total of XX exposures were used to generate these updated crosstalk coefficients, with the number of pixels used as ``source pixels'' (CZW: which you haven't explained yet) shown in Figure~\ref{fig:crosstalk-source-pixels}. + +\begin{figure}[t] + \centering + % TODO: add exposure-time vs.\ mean-signal figure asset + \fbox{\parbox{0.75\columnwidth}{\centering\vspace{1.5in}\small\textit{Figure pending}\vspace{1.5in}}} + \caption{Number of source pixels used to measure the crosstalk coefficients for each amplifier. + Note that this number drops toward the edge of the focal plane, due to difficulties properly illuminating CBP spots in these locations.\todo{Quantify the drop-off: typical source-pixel counts at the focal-plane centre vs.\ the edge, and the minimum count below which a coefficient was considered unconstrained.}} + \label{fig:crosstalk-source-pixels} +\end{figure} + +As discussed above (CZW: ???), each of these exposures was minimally processed to correct the serial overscan, apply the gains to convert the units to electrons, and trim and assemble the data segments. +All pixels above a threshold of $30{,}000$ electrons were then identified and used to construct crosstalk ratios $r_{ij} = T(\hat{x}, \hat{y})_j / S(x, y)_i$, where $S(x, y)_i$ is the flux value in a pixel at $(x, y)$ in amplifier $i$, and $T(\hat{x}, \hat{y})_j$ is the flux value at the same ``readout coordinate'' $(\hat{x}, \hat{y})$ in amplifier $j \ne i$. +These ratios therefore show the measured response to the source signal $S$ at the expected location of a crosstalk source at a target location $T$. +These fluxes and ratios are accumulated for the entire set of exposures and detectors. +Prior to any further study, the ratios are clipped to ensure contaminating features, such as stray light and reflections of other spots, are removed.\todo{Report what fraction of ratio measurements the clipping removes, so the reader can judge how aggressive the cleaning is.} +Any pixel with a measured ratio larger than 1.0 is excluded, as these would represent an unphysical situation in which the crosstalk signal was brighter than the source signal. +The remaining signals were XX-sigma clipped based on the interquartile distance, under the assumption that the ratio values are normally distributed. + +If the crosstalk response were perfectly linear, the average of these clipped ratios would directly yield the crosstalk coefficient between the source and target amplifiers. +However, as shown in the lab study of the crosstalk, and demonstrated in Figure~\ref{fig:crosstalk-linear-nonlinear-fit}, such a simple model fails to fully remove the crosstalk, as the source brightness changes over the profile of the spot.\todo{Quantify the improvement from the non-linear model: residual amplitude (in electrons or as a fraction of source flux) for the linear vs.\ non-linear coefficients shown in the two panels of the figure.} + +\begin{figure}[t] + \centering + % TODO: add exposure-time vs.\ mean-signal figure asset + \fbox{\parbox{0.75\columnwidth}{\centering\vspace{1.5in}\small\textit{Figure pending}\vspace{1.5in}}} + \caption{An example crosstalk residual on LSSTCam detector XX after correction with a linear crosstalk coefficient. + The oversubtraction in the core and undersubtraction in the wings of the source illustrate that the strength of the crosstalk signal is dependent on the flux of the source pixel. + The right panel shows the same spot, as corrected with the final non-linear crosstalk coefficients.} + \label{fig:crosstalk-linear-nonlinear-fit} +\end{figure} + +To provide the required crosstalk coefficients that are non-linear in source flux, we quantified the evolution of crosstalk coefficients with a linear term and a non-linear term in the fit to the flux ratios. +The intercept of this fit provides the term that is linear in the source flux, and the slope of the fit provides a term quadratic in the source flux. + +\subsection{Defects} \label{sec:calibration-production:defects} + +Defects are defined as pixel regions with anomalously high or low response compared to their neighbors, which cannot be corrected through the ISR process. +Defects can be persistent and isolated to specific pixel areas, or they can be transient and appear in different locations across exposures. +Defects are identified through a combination of automated detection in science images and automatic and manual inspection of calibration frames. +These defect pixels are masked and marked as \textsc{BAD} for the downstream analysis and interpolated over for visual inspection. +\todo{Quantify the defect population for DP2: total number of masked pixels/columns, the fraction of the focal plane they represent, the split between manual and automatic detections, and how that fraction varies between E2V and ITL and across epochs.} +%We describe how defect masks are generated (bad pixels/columns, saturation trails, and other artifacts), the thresholds used, and how masks are combined across exposures. + +Static defects are detected automatically in the combined bias, dark, and flat frames and then merged. +Before searching the flats, we remove the focal-plane illumination gradient, which imprints large-scale structure through non-uniform illumination. +The flat gradient is measured empirically by binning the image by a factor of 8, then fitting a four-node radial spline within each amplifier, and the full-resolution flat is divided by this model. +We then subtract the sigma-clipped mean of the amplifier, so that each threshold is measured relative to the typical pixel level. +On E2V detectors, the pixel row that defines the midline of the detector contains pixels roughly 2/3 the size of a typical pixel. +In normal science frames this is effectively a quantum efficiency change, but for defect detection, the boundary between the top and bottom amplifiers is set to the amplifier median before the search so that it is never flagged. + +Cold pixels are found in the combined flat as pixels whose response is below 90\% of the amplifier mean. +Hot pixels are found in the combined dark as pixels whose dark current is more than 3~e$^-$\,s$^{-1}$ above the amplifier mean, and in the combined bias as pixels more than 1000~ADU above the amplifier mean. +In the bias and dark, each hot pixel mask is widened by 2 pixels along its row to cover glow from the saturated column. +In any image, an amplifier column with at least 1000 flagged pixels, or at least 20 saturated pixels, is masked in full. + +Some defects cover large features in LSSTCam detectors that the automated search does not reliably capture. +These defects are identified by eye and masked manually by defining distinct pixel bounding boxes. +They include \textit{vampire pixels}, which are bright pixels that draw charge away from their neighbors, resulting in bright central pixels surrounded by an extended ring of dark pixels. +They also include scratches, which are complicated, irregular features of mixed response caused by physical damage to the sensor surface. +For example, an approximately 500~pixel-wide horizontal scratch in detector 0 (R01\_S00) affects multiple entire pixel columns. + +Finally, we mask a border of pixels around the edge of each detector, which we call the \textit{picture frame}. +Electric fields are distorted near the edges of the CCDs, which changes the pixel response there. +These edges are also difficult to calibrate, because corrections that rely on convolution, such as the brighter-fatter correction, are poorly defined near the image boundary. +The picture frame mask is therefore approximately twice the range of the brighter-fatter effect (\S\ref{sec:calibration-production:bf}), corresponding to 7 rows at the top and bottom and 8 columns at the left and right of each detector. + +Additional masking is applied to an exposure in the ISR processing, as described in \S\ref{sec:isr-science}. +Further statistical analyses of the defects and a description of the defect monitoring over long time baselines are given in \S\ref{sec:verify-accept-certify:verification}. + +\subsection{Non-Linearity} \label{sec:calibration-production:linearity} + +Non-linearity (NL) describes a deviation from simple proportionality when mapping amplified voltage integrated during readout to the number of analog-to-digital units (ADU) or counts. +Non-linearity is signal-dependent and is often not a simple function of signal level; it can change suddenly above a large signal level that is characteristic of each readout amplifier; and it can drift with sensor temperature and with time, as slow electronic shifts in the bias level accumulate over a long calibration campaign.\todo{Quantify the drifts: the fitted $\alpha$ (per K) and $\beta$ (per day) sensitivities and the resulting non-linearity change over a typical campaign, plus the range of per-amplifier turnoff levels implied by ``characteristic to each readout amplifier''.} +In practice it introduces pixel-level measurement bias (up to 0.5\% from the true signal), invalidates Poisson counting statistics, and can interfere with measurement of gain (which shouldn't change or drift with signal level). +Correcting NL to better than 0.01\% is therefore a prerequisite for reliable downstream calibration products and science analyses. + +We can measure non-linearity by measuring the response of pixel values to a change in illumination level. +The in-dome calibration illumination system can deliver an approximately uniform flux of light across the LSSTCam focal plane. +Given a known change in some input illumination level, we should be able to predict a change in the number of counts in an image, which we can then compare to the measured change in counts. + +We have independent measurements of the photodiode current produced by our calibration illumination system that we can integrate with the exposure time and use as a baseline truth of the amount of incident light while taking an exposure; +however, we found these measurements to be noisy and unreliable. Instead, we use the exposure time, which is precisely measured and makes a natural proxy for photodiode measurements. +We therefore calibrate NL by fitting a relation between exposure time ($\Delta t$, abscissa) and the mean measured flat-field signal ($\mu$, ordinate). + +We measured $543\times2$ flat-field exposures at evenly spaced illumination levels. We went as low as 100~analog-to-digital units (ADU) and continued beyond pixel saturation. +At each illumination level, a pair of flats was acquired by varying both LED current and exposure time to achieve the targeted levels, measuring the mean signal level as $\mu_i = \mathrm{avg}(F_i^{(1)},F_i^{(2)})$~(in ADU). +These data were acquired with illumination levels decided in random order to decouple signal-dependent non-linearity and time-dependent drifts in illumination. + +\begin{figure}[t] + \centering + \includegraphics[width=0.95\columnwidth]{figures/reference-amp-non-linearity_det94.pdf} + \caption{Absolute non-linearity fit for the reference amplifier~C07 on LSSTCam detector~94. + \textit{Left}---mean flat-field signal $\mu_{\mathrm{ref}}$ as a function of exposure time $\Delta t$, colored by LED illumination group ($k_1$, $k_2$, $k_3$). + Each group has its own ratio of signal to exposure time, which produces the discontinuities between groups. + The hatched region marks signal above the linearity turnoff ($\mu > 1.05 \times 10^5$~ADU), which is excluded from the fit. + \textit{Right}---fractional residuals of the spline fit, $(r_i - 1) \times 10^4$ (Equation~\ref{eq:spline-nl-fit}), as a function of $\mu_{\mathrm{ref}}$, with the same group colors.} + \label{fig:linearity-exptime-vs-signal} +\end{figure} + +Figure~\ref{fig:linearity-exptime-vs-signal} shows the resulting relation for a representative amplifier. +Discontinuities appear at the boundaries between LED current settings, producing different signal levels for the same exposure time. +These data are partitioned into groups associated with distinct LED illumination settings. +Grouping is determined either from exposure metadata or by computationally searching for jumps in the ratio of mean detector signal to exposure time. +Within each group we allow a distinct proportionality constant $k_j$ to capture the size of the jumps. + +To first order, the trend between exposure time and the mean measured flat-field signal over properly scaled groups is linear and related to the intrinsic gain in each readout channel. +We can fit a simple slope and intercept to the low-signal points of this trend; the highest signal level in each channel above which this approximation breaks is the ``linearity turnoff'' (as indicated in Figure~\ref{fig:linearity-exptime-vs-signal}). +Points above the turnoff are excluded from the NL fit because there is a maximum signal above which the measured counts can no longer be trusted due to hardware limitations at the output of the detector (Figure~\ref{fig:detector-boxes}). + +\begin{figure}[t] + \centering + \includegraphics[width=0.95\columnwidth]{figures/abs-rel-non-linearity_det94.pdf} + \caption{Double-spline non-linearity model for LSSTCam detector~94. + \textit{Left}---absolute non-linearity $S_{\mathrm{abs}}(\mu)$ of the reference amplifier~C07. + Gray points are per-exposure measurements (spline value plus fit residual), the black line is the Akima spline, and open circles mark the spline nodes. + \textit{Right}---relative non-linearity $S_{\mathrm{rel}}(\mu)$ of the remaining amplifiers with respect to C07, with nodes shown as filled points. + Blue shades are amplifiers C10--C17 and orange shades are C00--C06; darker shades correspond to higher segment numbers.} + \label{fig:linearity-abs-rel} +\end{figure} + +The NL model we fit to this relation is an extension of the formalism developed by \citet{2019A&A...629A..36A}. +We represent the deviation from linearity by an Akima spline $S(\mu)$ of the measured mean signal $\mu$. + +For each exposure $i$ in group $j$ we define the residual ratio +\begin{equation} + r_i = \frac{\mu_i - S(\mu_i) - O}{k_j\, D_{i}} + \label{eq:spline-nl-fit} +\end{equation} +where $D_{i}$ is the data used for the abscissa (exposure time), $O$ is an optional constant offset that absorbs scattered light or other signal not proportional to $D_{i}$, and $w_i$ are weights. + +The fit minimizes a uniformly weighted ($w_i = 1$) least-squares objective over all accepted points: +\begin{equation} + \chi^2 = \sum_{i} w_i \left(r_i - 1\right)^2. \label{eq:linearity-chisq} +\end{equation} +The fit is subject to the boundary conditions $S(0) = 0$ and $\langle S(\mu)\rangle = 0$ over the fit range. +The first constraint fixes the origin such that zero illumination is always measured as zero signal; the second ensures that the spline captures deviations from linearity independently from knowing the true amplifier gain. + +In addition, any hardware sensitivities are absorbed by nuisance parameters that we parameterize as first-order linear corrections to $\mu$ before the spline is evaluated: +\begin{equation} +\begin{aligned} + \mu &\rightarrow \mu \left(1 + \alpha\, T_i\right), \\ + \mu &\rightarrow \mu \left(1 + \beta\, t_i\right), +\end{aligned} +\label{eq:linearity-nuisance} +\end{equation} +where $T_i$ is a REB temperature (referenced to the median over the campaign), $t_i$ is the Modified Julian Date (MJD) of the midpoint of the shutter open time, and $\alpha$ and $\beta$ are fitted coefficients that describe each sensitivity. +For LSSTCam DP2 calibration production we fit only the temporal coefficients as well as the scattered-light offset $O$ on the spline. +For DP2, we found it to be unnecessary to fit the temperature coefficient since the temperature of the REB proved to be stable enough that no temperature dependence was observed. + +We could perform this fit independently for every amplifier on every detector. However, most of the non-linearity is common to all amplifiers due to similar readout architectures, with only small amplifier-to-amplifier variations on top of that (see Figure~\ref{fig:linearity-abs-rel}). +In addition, each independent fit to every amplifier would pick up not just the intrinsic non-linearity in amplifiers, but also exposure-to-exposure spatial illumination variation that occurs during the calibration campaign. +To better characterize this, it makes sense to perform a spline fit on each amplifier relative to both a single reference exposure and a reference amplifier on the same detector. + +We can distinguish an \emph{absolute} non-linearity in a single reference amplifier on a detector and the \emph{relative} non-linearity of other amplifiers to that reference amplifier, the process for which is summarized in Algorithm~\ref{alg:linearity-double-spline} and illustrated in Figure~\ref{fig:linearity-abs-rel}. +After choosing a reference amplifier and a reference exposure, we do (1) a relative spline fit on every other amplifier using the variation of its mean flat-field signal $\mu$ relative to the reference amplifier and to the $\mu$ in a single reference exposure, and then (2) +an absolute spline fit on the reference amplifier using a fit of measured $\mu$ versus $\Delta t$. +Every amplifier on a detector has a shared absolute spline fit based on the reference amplifier for that detector and a relative spline fit computed relative to a reference amplifier and a reference exposure ($S_{\mathrm{rel}}(\mu) = 0$ for the reference amplifier). + +\begin{algorithm}[H] +\caption{Double-spline linearity fit (single detector)} +\label{alg:linearity-double-spline} +\begin{algorithmic}[1] +\For{each amplifier $a$} + \State $k_j \leftarrow$ \Call{Find Groups}{$\mu_a$, $\Delta t$} + \State \Call{Compute linearity turnoff}{$\mu_a$, $\Delta t$} +\EndFor +\State choose reference amplifier $\mathrm{ref}$ with the highest linearity turnoff +\State choose reference exposure $i_0$, the flat whose mean signal $\mu_{\mathrm{ref}} = 10^4$ on amplifier $\mathrm{ref}$ is nearest the target signal level +\For{each exposure $i$} + % \State compare flat $i$ to the reference exposure $i_0$ + % \State $\mu_{\mathrm{ref},i} \leftarrow$ $\mu_{\mathrm{ref}}$ / $\mu_{\mathrm{ref}, i_0}$ % mean signal on amplifier $\mathrm{ref}$ in exposure $i$, normalized to $i_0$ + \For{each amplifier $a \neq \mathrm{ref}$} + \State $D^{\mathrm{rel}}_{i} \leftarrow \mu_{\mathrm{ref}, i} \times (\mu_{a,i_0} / \mu_{\mathrm{ref}, i_0})$ (Relative Abscissa) % mean signal on amplifier $a$ in exposure $i$, normalized to $i_0$ and expressed relative to $\mu_{\mathrm{ref},i}$ + \State $\mu^{\mathrm{rel}}_{i} \leftarrow \mu_{a,i}$ (Relative Ordinate) + \EndFor + + \State $D^{\mathrm{abs}}_{i} \leftarrow \Delta t_i$ (exposure time) + \State $\mu^{\mathrm{abs}}_{i} \leftarrow \mu_{\mathrm{ref},i}$ (measured counts) +\EndFor + +\Statex \Comment{Spline fit} +\State $S_{\mathrm{abs}}(\mu), O, \alpha, \beta \leftarrow$ \Call{Minimize}{Equation~\ref{eq:linearity-chisq}, $D^{\mathrm{abs}}_{i}$, $\mu^{\mathrm{abs}}_{i}$, $k_j$} +\State $S_{\mathrm{rel}}(\mu) \leftarrow$ \Call{Minimize}{Equation~\ref{eq:linearity-chisq}, $D^{\mathrm{rel}}_{i}$, $\mu^{\mathrm{rel}}_{i}$, $k_j$} +\end{algorithmic} +\end{algorithm} + +To determine the corrected mean signal level, we can interpolate the Akima splines to compute a correction: $\mu' = \mu - S_{\mathrm{rel}}(\mu) - S_{\mathrm{abs}}(\mu)$. + +If each amplifier is properly linearized around its true gain, we should observe in every exposure that the illumination level is coherent between amplifiers and detectors. +If the linearization is miscalibrated or biased (either because exposure time is not a good proxy for the true integrated photodiode signal or because constraining $\langle S(\mu)\rangle = 0$ is biased), the amplifiers and detectors will show signal-dependent fractional offsets in mean signal levels in each flat similar to a mis-estimation of gain. + +We can check how the linearized mean signal levels deviate from an overall linear relation over some set of detectors in a given exposure. +For each exposure $i$, we form the linearized signal, $\mu'_i$, on the reference amplifier on every detector and compute the fractional residual: +\begin{equation} + r_i = \frac{\mu'_i - m_i D_{i}}{y_i}, +\end{equation} +where $m_i$ is the slope of a linear fit to $\mu'_i$ versus $D_{i}$. +We take the median $\langle r \rangle_i$ over a subset of focal-plane detectors (in the center of the focal plane) in that exposure and derive a multiplicative correction +\begin{equation} + N_i = \frac{1}{1 - \langle r \rangle_i}, +\end{equation} that we use to rescale the abscissa ($D_{i} \rightarrow N_i D_{i}$) and enforce that the linearization produces a coherent illumination across all detectors. + +We can then re-compute the double-spline fit with the normalized abscissa on the reference amplifier. We can repeat the cycle (normalize, spline, normalize, etc.) recursively several times, each time computing the normalization as \begin{equation} + N_i = \frac{N_i^{\mathrm{prev}}}{1 - \langle r \rangle_i}. +\end{equation} + +A single iteration improves the precision of the linearity solution by a factor of 10, with little further gain from additional iterations. + +In ISR, given an image with pixel values measured in counts ($F_{ij}$), the final spline model can be interpolated, providing a continuous map from measured counts to the corrected counts for any pixel value from zero signal to the linearity turnoff, computed as: +$F_{ij}' = F_{ij} - S_{\mathrm{rel}}(F_{ij}) - S_{\mathrm{abs}}(F_{ij})$. + +% During science ISR, the linearizer is applied at the non-linearity correction step in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. + + +\subsection{Deferred Charge} \label{sec:calibration-production:cti} + +Several distinct effects in the readout chain displace charge from its pixel of origin and leave correlated structure in the serial overscan: (1) a pixel-to-pixel fractional loss during charge transfer referred to as ``global CTI'', (2) a local electronic-offset hysteresis in the output amplifier that mimics deferred charge in the first few overscan columns, and (3) localized capture and delayed release of charge by isolated trap sites in the serial register. +Strictly, only global CTI is ``charge transfer inefficiency'' in the textbook sense, but because all three enter at the same point in the photon transfer chain (Figure~\ref{fig:detector-boxes}) and produce overlapping signatures in the serial overscan, we model them together and produce a single calibration data product that we loosely refer to as ``CTI'' in Figure~\ref{fig:calib-dependency}. +These effects are measured, modeled, and corrected together using the extended pixel edge response (EPER) estimator developed for LSSTCam by \citet{2021JATIS...7d48002S}. + +Calibrating deferred charge uses the same input data as the linearity dataset (\S\ref{sec:calibration-production:linearity}), estimating the residual charge in electrons left in the overscan region after integrating and reading out flat-field images over a wide range of signal levels. +The global CTI is quantified by a unitless fraction, but the trap size and the electronic drift scale both carry signal-dependent physical units, so an imperfect conversion from counts back to physical electrons biases the eventual correction. +This calculation therefore needs to happen in native electron units, so we need to remove signal-dependent non-linearities that occur when the signal is measured as counts and estimate a gain conversion from counts to electrons. +For each input exposure, we apply an intermediate pass of instrument signature removal that applies defect masking and non-linearity correction, converts pixel values to electrons, and measures the overscan EPER statistics for each readout channel. + +The natural way to convert from counts to electrons is to use the gain inferred from the PTC (\S\ref{sec:calibration-production:ptc}), but the PTC needs to be CTI-corrected to get the most accurate inferred gains, creating what seems like a circular dependency between CTI and PTC calibration production. +We break the cycle by bootstrapping. We create a temporary (intermediate) PTC that is not CTI-corrected, which we fit (see \S\ref{sec:calibration-production:ptc}) to derive an initial rough estimate of the per-amplifier gain. +That estimate is stored and applied during the ISR of the flat-field images before calculating the EPER statistics. +We find that the gains from a PTC without CTI correction are close enough to the final gains measured from a fully calibrated PTC (typically within 1\% of the true gains). +At this level, the rough gains are accurate enough for CTI calibration. The global-CTI fit is unit-free and so insensitive to any gain error, and the trap and local-offset amplitudes have only weak sensitivity to the gain bias that remains in the bootstrap estimate. + +An example of the measured serial EPER statistics as a function of signal level for a single channel readout is shown in Figure~\ref{fig:cti-serial-eper}. +The global CTI produces an overall offset of this curve at all signal levels, the electronic offset produces a linear slope in the curve as a function of signal level, and the serial traps produce non-linear jumps that ``turn on'' above some signal level after a trap fills up and begins releasing charge with a characteristic timescale. +There is a point at high signal levels at which we can no longer effectively transfer charge pixel to pixel, which we define as the ``serial CTI turnoff'' as indicated in the figure. +We calculate this point as the highest mean signal for which the serial EPER remains consistent with a sigma-clipped linear fit to the low-signal points. +In practice, the flat-field signal levels in our data never reached the serial CTI turnoff in any readout channel, so we set the turnoff to the maximum observed signal level. + +\begin{figure}[t] + \centering + \includegraphics[width=\columnwidth]{figures/cti-det93-ampC04-serial-eper.pdf} + \caption{Serial EPER statistics as a function of flat-field signal level for detector/amplifier R22\_S10/C04. + Vertical markers indicate the PTC rolloff and turnoff thresholds (described in \S\ref{sec:calibration-production:ptc}) and the serial CTI turnoff derived from the low-signal trend. + The global CTI fit produces the overall offset of the measured curve at all signal levels.} + \label{fig:cti-serial-eper} +\end{figure} + +The CTI solver simulates charge readout and each of the three effects at the amplifier image level for a given set of parameters. +The model in the simulation is parameterized by a global serial CTI parameter $b$ that is a fractional charge loss per pixel transfer, a local electronic offset described by drift scale $A_L$ and decay time $\tau_L$, and a spline-based serial charge-trap decay model whose size is allowed to depend on the signal in the last imaging column. +We perform a least-squares fit that finds the optimal set of parameters that can reproduce the measured overscan statistics trend as a function of signal level. +Then the solver fits these parameters to the overscan image in stages. +First, it varies only the local offset parameters, then the global CTI parameter, and then the trap capture curve from residuals between model and data. +Exposures above configured flux limits are excluded from each stage (typically $10^4$~electrons for the global CTI fit and $1.5\times10^5$~electrons for trap fitting). + +The LSST Science Requirements Document requires that total deferred charge measured by EPER be kept under a total of $5\times10^{-6}$ for signal levels below saturation level on each amplifier (citation needed). +Within the population of amplifiers in active science detectors, only 95 amplifiers do not meet this requirement without ISR correction. +Figure~\ref{fig:cti-global-histogram} shows the distribution of inferred global CTI across all amplifiers on the LSSTCam focal plane, and we show that 26 have a global CTI offset that is already above the maximum allowed threshold. +The majority of these (19) are bad amplifiers that are in the detector corners (C00, C07, C10, C17) positioned near the serial register, and it is possible that what is inferred as global CTI could be a serial trap that turns on at very low signal levels (below the $10^4$ electron level) and appears as an overall offset that is mistaken for global CTI. + +\begin{figure}[t] + \centering + \includegraphics[width=\columnwidth]{figures/cti-det93-ampC04-global-cti-histogram.pdf} + \caption{Distribution of inferred global serial CTI across all science sensor amplifiers on the LSSTCam focal plane. + \textit{Blue}---Teledyne e2v (E2V) detectors; \textit{Orange}---Imaging Technology Laboratory (ITL) detectors. + The fit to the deferred charge model allows global CTI to vary up to $1.0 \times 10^{-5}$.} + \label{fig:cti-global-histogram} +\end{figure} + +When correcting deferred charge from an image during ISR, the deferred-charge correction takes in these fit parameters and applies the inverse operators from the model. +The global CTI and local-offset fits, and the inverse correction operators applied during ISR, are the same for both ITL and E2V sensors, but serial traps are fitted only on ITL since it was observed that E2V sensors had small trap sizes that was difficult to measure beyond noise fluctuations and already negligible enough not to require correction. +After applying a deferred charge correction, we can re-compute the EPER statistics on the corrected dataset and find that all amplifiers meet the science requirements. + +While there can be deferred charge from a charge transfer in the parallel direction (shift row-to-row), in principle this is a difficult quantity to estimate accurately from parallel overscan statistics. +EPER statistics computed in the parallel overscan region are noisy and reveal no statistically significant charge residuals, so we do not apply any parallel deferred charge correction to science images. + +% During science ISR, this product is applied at the deferred-charge correction step in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. + + +\subsection{Photon Transfer Curve (PTC)} \label{sec:calibration-production:ptc} + +The photon transfer curve (PTC) characterizes how pixels respond to varying signal levels in a given wavelength band by measuring covariances between pixels in flat-field exposures across a range of illumination levels. +This dataset provides critical calibration information including amplifier-level read noise and gain for variance plane estimation, saturation thresholds for masking, and parameters needed to correct for non-linearity, charge transfer inefficiency (CTI), and the brighter-fatter effect. +All of these parameters can differ at the amplifier level. \todo{Add histogram plots of the distributions of these parameters across the focal plane.} + +% The photon transfer curve used to calibrate LSSTCam for DP2 is measured from $543\times2$ flat field exposures at evenly spaced illumination levels as low as 100~analog-to-digital units (ADU) and reaching at least 10\% beyond saturation. +% At each illumination level, a pair of flats was acquired by varying both LED output and exposure time to achieve the target signal. +% We measure the covariance between pixels on the difference image of the two exposures to remove variations sourced by static, fixed-pattern artifacts. + +Let $F_1$ and $F_2$ denote the two flats at a given illumination level, and let $\mu = \mathrm{avg}(F_1,F_2)$ denote the mean signal level (in ADU) over the image and away from the detector edges. +The covariance between a pixel $\mathbf{x}=(0,0)$ and a neighbor at pixel $\mathbf{x}'=(i,j)$ is: +\begin{equation} + C_{ij} \;=\; \tfrac{1}{2}\, + \mathrm{Cov}\!\left(F_1(\mathbf{x})-F_2(\mathbf{x}),\, + F_1(\mathbf{x}')-F_2(\mathbf{x}')\right). +\end{equation} + +% Following the standard front-illuminated CCD convention, serial (row) readout proceeds along $(1,0)$ and parallel (column) readout along $(0,1)$. + +\paragraph{Pre-model fit} +Preliminarily, we fit only the variance to an approximate exponential relation derived in \citet{2019A&A...629A..36A} (their Eq.~16). +\begin{equation} + C_{00}^{\mathrm{exp.\,approx}}(\mu) = \frac{1}{2 g^2 a_{00}}\bigl[\exp(2 a_{00} \mu g) - 1\bigr] + \frac{n_{00}}{g^2}\ . +\end{equation} +In this expression, covariances are contributed by a gain $g$ (electrons$/\mathrm{ADU}$), noise pedestal $n_{00}$ (electrons$^2$), and a single brighter-fatter coefficient $a_{00}$ ($1/$electrons) which has the effect of decreasing the image variance. + +Near saturation, the variance deviates from this functional form, exhibiting a gradual rollover rather than a sharp cutoff due to electrostatic effects that are difficult to model analytically. +On top of the exponential approximation, we introduce an empirical saturation term: +\begin{equation} + C_{00}^{\mathrm{sat}}(\mu) = \left\{ + \begin{array}{ll} + -e^{-(\mu-\mu_0)/\tau} & \mu \ge \mu_0 \\ + 0 & \mu < \mu_0 + \end{array} + \right., +\end{equation} +where $\mu_0$ marks the onset of the PTC ``rolloff'' effects and $\tau$ controls the shape of the rollover. +The total preliminary fit model is $C_{00}^{\mathrm{prelim.}}(\mu) = C_{00}^{\mathrm{exp.\,approx}}(\mu) + C_{00}^{\mathrm{sat}}(\mu)$ to the unweighted points. This fit is performed iteratively with additional 3~sigma clipping until convergence. +The ``PTC turnoff'' is defined as the signal level of the last unmasked point following the fit to only the exponential approximation, typically corresponding to the maximum of the variance curve and generally higher than the rolloff threshold. +The ``PTC rolloff'' is the signal level where $C_{00}^{\mathrm{prelim.}}(\mu)$ and $C_{00}^{\mathrm{exp.\,approx}}$ deviate by more than 1\%. + +\paragraph{Final model fit} +With the remaining points below the PTC rolloff, we fit the complete covariance expansion from \citet{2019A&A...629A..36A} (their Eq.~20): +\begin{equation} + \begin{aligned} + C_{ij}^{\mathrm{model}}(\mu) = \frac{\mu}{g} \Bigl[ \delta_{i0}\delta_{j0} + &+ a_{ij}\mu g \\ + &+ \frac{2}{3}\left([\mathbf{a}\otimes\mathbf{a} + \mathbf{ab}]_{ij}\right)(\mu g)^{2} \\ + &+ \frac{1}{6}\left(2\mathbf{a}\otimes\mathbf{a}\otimes\mathbf{a} + 5\mathbf{a}\otimes\mathbf{ab}\right)_{ij}(\mu g)^{3} \\ + &+ \mathcal{O}[(\mu g)^4] + \Bigr] + \frac{n_{ij}}{g^2}, + \end{aligned} + \label{eq:cov_model} +\end{equation} +where $a_{ij}$ (electrons)$^{-1}$ represents the fractional change in pixel area per stored charge due to boundary motion from the brighter-fatter effect, $b_{ij}$ (electrons)$^{-1}$ captures weaker time-dependent corrections, $g$ is the gain (electrons$/\mathrm{ADU}$), and $n_{ij}$ is the constant noise contribution (electrons)$^2$ (e.g., $n_{00}\sim (5.7\,\mathrm{electrons})^{2}$ on the diagonal, with off-diagonal terms determined by read electronics and overscan). +The symbol $\mathbf{ab}$ denotes element-wise matrix multiplication, and $\otimes$ represents discrete convolution. +In practice, the $b_{ij}$ parameter is inferred to be just noise around a value of zero. +The size of the covariance matrix ($n \times n$) is determined by the range of the brighter-fatter effect (see \S\ref{sec:calibration-production:bf}). +Figure~\ref{fig:ptc-fit-example} shows an example fit for amplifier~C03 on detector~94, illustrating the measured $C_{00}$ points, the preliminary exponential approximation with saturation rolloff, and the final variance from Eq.~\ref{eq:cov_model}. + +\begin{figure*}[t] + \centering + \includegraphics[width=\textwidth]{figures/ptc-triangle-det94-ampC03.pdf} + \caption{Example PTC fit for amplifier~C03 on LSSTCam detector~94. + Measured $C_{00}$ covariance points are compared to the preliminary fit (exponential approximation plus saturation rolloff) and the final full-covariance model variance. + Vertical markers indicate the PTC rolloff and turnoff thresholds.} + \label{fig:ptc-fit-example} +\end{figure*} + +Because the gains are fit independently per amplifier, small fitting residuals can leave flux discontinuities at amplifier boundaries within a detector. +We add a subsequent step that measures the relative offsets between amplifiers in flat fields after gain correction and computes small adjustments to the gain to make the detector image continuous while preserving the average gain across the detector. +We compute the average gain adjustment in flat fields across a range of signal levels and use these to produce overall ``adjusted gains''. +The adjustments to the gain are always small (less than 0.03 electrons/ADU) and uncorrelated between detectors. +These final gains are used for conversion of science images to electron units and construction of the variance plane for source detection. + +The PTC calibration products are applied during instrument signature removal of science images in three ways. +First, the gain converts pixel values from ADU to electrons. +Second, the gain and read noise inform variance plane estimation. +Third, the saturated pixel mask flag (denoted ``SAT'') is set for all pixels with values greater than or equal to the PTC turnoff threshold. + +% During science ISR, PTC-derived gains, variance-plane inputs, and saturation masking are applied at the steps shown in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. + + +\subsection{Brighter-Fatter Effect (BFE)} \label{sec:calibration-production:bf} + +The brighter-fatter effect (BFE) is the first influence the detector has on an incident signal (Figure~\ref{fig:detector-boxes}). +Accumulated electrons in a pixel repel incoming charges into neighboring pixels. +Brighter point sources therefore appear wider in the recorded images. +This electrostatic effect reduces to an effective change in pixel area and introduces correlations between nearby pixel values. +The effect is intrinsic to thick CCDs and can significantly bias point-spread-function (PSF) photometry and shape measurement \citep{2024PASP..136d5003B}. + +The BFE has been extensively characterized for LSSTCam by laboratory and in-dome commissioning measurements \citep{2024PASP..136d5003B}. +The strength of the effect is anisotropic ($a_{01}/a_{10} \sim 3$), long-ranged ($\sim 10$~px), and wavelength dependent. +It is also about twice as strong in the E2V-family detectors as in ITL-family detectors. + +Several BFE correction methods have been proposed and implemented in the LSST Science Pipelines \citep{2015JInst..10C5032G,2018AJ....155..258C,2023A&A...670A.118A,2024PASP..136d5003B}, and they differ widely in their underlying assumptions. +\citet{2024PASP..136d5003B} found that modeling individual boundary shifts for every pixel (N, S, E, W) is required for adequate correction at the level needed for LSST Y10 science requirements. +Flat-field pixel correlations can measure only the net per-pixel area change, so determining these boundary shifts requires additional electrostatic modeling. +For processing raw images from DP2, we use the \citet{2023A&A...670A.118A} algorithm developed for Hyper Suprime-Cam, which derives from electrostatic physics and treats each pixel boundary individually. + +% The correction assumes three simplifying conditions. +% First, the average charge current into a pixel is approximately constant during a long exposure, even when the PSF varies with time. +% Second, fourth-order terms in Eq.~\ref{eq:cov_model} are negligible; at the scale of the BFE, the corresponding coefficients are of order $10^{-24}$. +% Third, photons convert to electrons quickly after entering the detector. +% Simulations and empirical modeling indicate that most photons in $u$ through $i$ bands convert within the top few microns of silicon, while the BFE influence on charge drifting to the pixel bottom occurs primarily in the last several microns. +% Chromatic effects are therefore likely negligible for photons with wavelengths across the LSST bandpasses. + +\begin{deluxetable}{ll}[t!] + \tablecaption{Parameter Priors for the Electrostatic BFE Model Fit\label{tab:bf-electrostatic-fit-priors}} + \tablehead{ + \colhead{Parameter} & \colhead{Prior} + } + \startdata + Si thickness, $t$ ($\mu$m) & $100$ (fixed) \\ + Pixel size, $p$ ($\mu$m) & $10$ (fixed) \\ + Charge cloud height, $z_q$ ($\mu$m) & $0 \leq z_q \leq t/2$ \\ + Charge cloud length, $a$ ($\mu$m) & $10^{-5} \leq a \leq 0.35\,p$ \\ + Charge cloud width, $b$ ($\mu$m) & $10^{-5} \leq b \leq 0.35\,p$ \\ + Potential well vertex height,\\horizontal $z_{sh}$ ($\mu$m) & $0 \leq z_{sh} \leq t/2$ \\ + Potential well vertex height,\\vertical $z_{sv}$ ($\mu$m) & $0 \leq z_{sv} \leq t/2$ \\ + \enddata + \tablecomments{Priors on the fit parameters are informed by pixel-level simulations of the electric fields in these pixels \citep{2017JInst..12C3091L}. The potential well vertices refer to the heights above the front side of the CCD of the saddle points in the electric field that define the pixel boundaries in the vertical (0, 1) and horizontal (1, 0) directions. The electrostatic model assumes that the stored charge cloud in the sensor takes the form of a flat, rectangular sheet of charge with length ($a$) and width ($b$). The inequality constraints $z_{sh} > z_q$, $z_{sv} > z_q$ are enforced during optimization, which is true in the unsaturated signal level regime.} +\end{deluxetable} + +Calibration of the BFE using this algorithm takes the form of modeling a charge cloud at the bottom of a pixel and computing +how the effective areas of nearby pixels would be expected to change electrostatically in response. It adjusts some geometric parameters (summarized in Table~\ref{tab:bf-electrostatic-fit-priors}) and solves +Gauss's law to compute the shift of pixel boundaries and the relative change in pixel area from each shift ($\mathbf{a}^{\mathrm{N}}$, +$\mathbf{a}^{\mathrm{S}}$, $\mathbf{a}^{\mathrm{E}}$, $\mathbf{a}^{\mathrm{W}}$). +We optimize these parameters to best fit the +measured net pixel area changes ($\mathbf{a}$ from Eq.~\ref{eq:cov_model}): +\begin{equation} + \mathbf{a} = \mathbf{a}^{\mathrm{N}} + \mathbf{a}^{\mathrm{S}} + \mathbf{a}^{\mathrm{E}} + \mathbf{a}^{\mathrm{W}}. +\end{equation} + +In principle, the BFE can vary amplifier-to-amplifier, but we have observed only noise-level variations in the pixel distortion matrices across amplifiers within detector families ($a_{00}^{(\mathrm{E2V})} \approx 2 \times a_{00}^{(\mathrm{ITL})}$). +For this reason, and to avoid discontinuous amplifier boundaries, the fit is performed at the detector level. +All $a_{ij}$ matrices from well-behaved amplifiers are 3$\sigma$-clipped element-wise and averaged into a detector-mean $a_{ij}$; the scatter becomes the per-element uncertainty, and a single fit is run against this average pixel distortion matrix. +The output calibration data product is the set of model parameters and a model that can be evaluated per-detector. + +Applying this calibration to remove the effect during ISR uses the boundary-shift matrices $\mathbf{a}^{\mathrm{N}}$, $\mathbf{a}^{\mathrm{S}}$, $\mathbf{a}^{\mathrm{E}}$, and $\mathbf{a}^{\mathrm{W}}$ to redistribute flux accordingly. +We tile these into two $(2n)\times(2n)$ convolution kernels $K^{H}$ and $K^{V}$ for the horizontal and vertical shifts, respectively; we apply sign flips and edge adjustments so each matrix sums to zero. +These matrices need not be symmetric, which is more realistic per \citet{2024PASP..136d5003B}. +Figure~\ref{fig:ebf-kernels} shows an example pair of these kernels for amplifier~C03 on detector~94, illustrating the dipole-like, charge-conserving structure that redistributes flux across the vertical and horizontal pixel boundaries. + +\begin{figure}[t] + \centering + \includegraphics[width=\columnwidth]{figures/ebf-det94-ampC03-kVH.pdf} + \caption{Example BFE correction kernels $K^{V}$ (vertical) and $K^{H}$ (horizontal) derived from the electrostatic BFE model for amplifier~C03 on LSSTCam detector~94. + The diverging color scale shows the fractional charge redistributed across pixel boundaries; each kernel sums to zero to conserve charge and exhibits the antisymmetric dipole structure that moves flux from the brighter pixel to its neighbors.} + \label{fig:ebf-kernels} +\end{figure} +For an image $Q_{ij}$ in natural gain-corrected units of electrons, the amount of charge lost over the vertical and horizontal boundaries is: +\begin{align} + \delta^{H}_{ij} &= \tfrac{1}{2}\!\!\sum_{kl}\!\!K^{H}_{kl}\,Q_{i-k,\,j-l}, \notag \\ + \delta^{V}_{ij} &= \tfrac{1}{2}\!\!\sum_{kl}\!\!K^{V}_{kl}\,Q_{i-k,\,j-l}, + \label{eq:bf-shift} +\end{align} +The $1/2$ prefactor accounts for the average BFE assuming a constant rate of charge accumulation during an exposure. +The corrected image is assembled by removing the charge each pixel lost across its $N$/$E$ boundaries and adding what it gained from $S$/$W$ neighbors, guaranteeing charge conservation by construction, $\sum_{ij}Q^{\mathrm{corr}}_{ij}=\sum_{ij}Q_{ij}$. +This correction does not require iterative convolutions. Unlike the \citet{2018AJ....155..258C} algorithm used for DP1 \citep{2026AJ....171..360V}, which requires multiple convolutions (typically 3 or more) to reach convergence, we perform exactly two convolutions with the horizontal and vertical boundary-shift matrices. + +% During science ISR, the BFE correction is applied at the step shown in Figure~\ref{fig:isr-pipeline}; see \S\ref{sec:isr-science}. + + +\subsection{Combined Bias} \label{sec:calibration-production:bias} + +A bias image is a zero-second exposure read out with the shutter closed, and so records the electronic pedestal of the readout chain in the absence of any accumulated charge. +Overscan correction removes the component of that pedestal that varies from exposure to exposure, while the combined bias measures the static two-dimensional electronic structure that remains and is subtracted from every science image. + +To remove this remnant signal that is not already removed by the previous calibrations, a combined bias frame is subtracted from the exposure. +This combined bias is constructed from the per-pixel clipped mean of a large number of input zero-second bias frames. +These input bias frames have all of the corrections discussed up until this point applied. +For LSSTCam, these combined biases have 400 input exposures, which reduces the noise in the correction. + +\subsection{Combined Dark} \label{sec:calibration-production:dark} + +A dark image is an exposure taken with the shutter closed, and measures the signal that accumulates in proportion to integration time rather than to incident light. It is dominated by thermally generated electrons, called dark current, together with any glow from the readout electronics, resulting in approximately $0.02$~e$^-$\,pixel$^{-1}$\,s$^{-1}$. +These artifacts are removed by scaling the dark to the exposure time of the science image and subtracting it. + +Similar to the bias, we construct a combined dark frame from a large number of 30-second exposures, with that time chosen to match the standard exposure time of a science exposure. +As before, all of the previous calibrations are applied to these dark frames, including the bias correction. +These exposures have an additional cosmic ray rejection step applied after the ISR processing, as these longer exposures may have more of these features than are found on the bias frames. + +\subsection{Combined Flats} \label{sec:calibration-production:flats} + +A flat field measures the relative response of the system to uniform illumination, capturing both pixel-to-pixel variations in quantum efficiency and larger-scale structure imposed by the optics, the filter transmission, and vignetting of the focal plane. +Dividing a science exposure by the combined flat normalizes this response, so that a source of a given flux produces the same measured signal regardless of where it lands on the focal plane. + +Although flat-field images are a common astronomical correction, the large field of view of LSSTCam makes generating this calibration a particular challenge. +In addition to ensuring consistent illumination across this field of view, there is an added complication that the ITL and E2V detectors have different responses as a function of wavelength. +One final complication comes from the finite choices of LEDs in the flat-field illumination system described in Section \ref{sec:calsys:flat-field-illuminator}. +Although we anticipate that we will be able to construct higher-quality synthetic flats from sequences of laser-illuminated monochromatic flats in the future, for DP2 we have chosen to construct ``anaglyph'' flats for most bands. + +These anaglyph flats superimpose a red-side and a blue-side flat to construct a single flat across a full LSST bandpass. For a given filter, we combine a pair of flat-field images that have each been observed with a single active LED illuminating the flat-field screen. +For the subsequent discussion, these two sets of input flat-field images are referred to as the ``red'' and ``blue'' exposures, depending on the wavelength of the two LEDs within the filter bandpass for which the final flat field will be used. +These anaglyph flats have been constructed for each of the $grizy$ filters. +This process is unavailable for the $u$ filter since a sufficiently blue ultraviolet LED was not available, nor was there space to fit one inside the projection system, and so flats for this filter are constructed from only a single set of flats (with an LED that is at the red end of the bandpass). +A bluer ultraviolet LED will be available to be periodically swapped in and out for DR1 calibrations. + +To begin the construction of new flats, the input exposures are processed through the ISR steps that are applied prior to flat correction, including the bias, dark, crosstalk, gain application, linearization, CTI, and overscan (CZW: reorder to be in order). +Statistics are measured on these processed exposures, determining the median value and dispersion on each detector. +Normalization factors are then measured for the red and blue inputs separately based on these statistics, which should account for any differences in the absolute flux received in each exposure over all detectors. +These normalization factors are used to properly scale the inputs before they are combined using a clipped mean per pixel into separate intermediate red and blue flats. + +These intermediate flats include gradients related both to how well the flat-field illumination and screen were oriented and to the vignetting of the focal plane itself. \todo{Plot the flat field gradient?} +Both types of gradients are measured next, so that their contribution can be removed from the final flat. +An initial six-node radial spline model is fit to the full binned focal plane image of the intermediate flats. +This initial model is used to constrain both the centroid of the focal plane and the observed ratio of the ITL to E2V detectors, to account for their possibly different response. +These parameters are then used for a 15-node radial spline model that is more careful about fitting the observed vignetting at high focal plane radii. +This second fit uses a fixed ITL/E2V ratio measured during the first pass, to reduce the degrees of freedom in the final fit. +An additional planar gradient is also included in this second fit, to account for simple illumination differences. + +The measured gradients are then removed from the intermediate flats. +The planar gradients are simply divided from the per-detector intermediate flats. +The radial gradient is scaled by a reference gradient to reduce the effects of the vignetting, and to match the vignetting observed in on-sky data (CZW: I need to explain reference gradient better). +A final normalization factor is applied to scale the entire flat to match the intensity level observed within the central 25~mm of the focal plane. +With these gradient-corrected intermediate flats generated, a final flat is constructed by combining the red and blue intermediates with a filter-specific mixture weight, as listed in Table~\ref{tab:anaglyph_ratios}. + +\begin{deluxetable}{lcc}[t!] + \tablecaption{Mixture Weights for Anaglyph Flat Construction\label{tab:anaglyph_ratios}} + \tablehead{ + \colhead{Band} & \colhead{Blue Weight} & \colhead{Red Weight} + } + \startdata + $g$ & 0.63 & 0.37 \\ + $r$ & 0.50 & 0.50 \\ + $i$ & 0.476 & 0.524 \\ + $z$ & 0.65 & 0.35 \\ + $y$ & 0.52 & 0.48 \\ + \enddata + \tablecomments{The final flat in each band is the weighted sum of the gradient-corrected blue and red intermediate flats, $F = w_{\rm blue}\,F_{\rm blue} + w_{\rm red}\,F_{\rm red}$, where the weights in each row sum to unity. The $u$ band is not listed because its flat is constructed from a single LED at the red end of the bandpass.} +\end{deluxetable} diff --git a/sections/6_verification-certification.tex b/sections/verification-certification.tex similarity index 50% rename from sections/6_verification-certification.tex rename to sections/verification-certification.tex index c2d96e5..b195553 100644 --- a/sections/6_verification-certification.tex +++ b/sections/verification-certification.tex @@ -1,14 +1,14 @@ \section{Verification, Acceptance, and Certification} \label{sec:verify-accept-certify} -Calibrations applied to survey images undergo an organized official process to verify their quality, approve them, and deploy them to the community for processing. The need for an open, controlled, and documented procedure is designed to meet the needs of the rapid data rates and survey cadence of the LSST and the importance of calibration to all downstream science with LSST. +Calibrations applied to survey images undergo an organized official process to verify their quality, approve them, and deploy them to the community for processing. This open, controlled, and documented procedure is designed to meet the needs of the rapid data rates and survey cadence of the LSST and the importance of calibration to all downstream science with LSST. As these calibration products are used for all parts of the survey, approving updates is done via an internal review that is open to any member of the project, to ensure that all views are heard before processing uses them. All data products, including calibrations, are managed by the data \texttt{butler}, an abstraction middleware layer that mediates all access to data. It contains a registry database that records the metadata and provenance of every dataset, allowing pipelines to retrieve and store data by dataset type and identifying labels rather than by file path or storage format. Datasets in a butler repository are organized into named collections, and it is through these collections that calibrations are associated with the exposures they apply to, made publicly available, and tracked over the length of the survey \citep[see][for details]{2022SPIE12189E..11J}. This section details the steps used to verify, accept, and deploy calibration products for use as part of the survey. -These steps do not preclude users from constructing and using their own set of calibrations, but they ensure that the default set of calibrations are suitable for use. -Full details about the commands used for these steps, and suggestions for a consistent collection naming system are presented in DMTN-222 \citep{DMTN-222}. +These steps do not preclude users from constructing and using their own set of calibrations, but they ensure that the default set of calibrations is suitable for use. +Full details about the commands used for these steps and suggestions for a consistent collection naming system are presented in DMTN-222 \citep{DMTN-222}. % This section defines the quality metrics for calibration products and ISR outputs. % Here we define the criteria used to promote products to ``accepted'' and ``certified''. @@ -18,13 +18,13 @@ \subsection{Verification} \label{sec:verify-accept-certify:verification} \todo{The verification criteria in this subsection are qualitative throughout. They need quantitative pass/fail thresholds and acceptance criteria, along with the values measured for DP2 --- possibly collected in a table.} After new calibrations have been constructed, they are used to process some set of exposures to show that they correct the various instrument features as expected. -For simpler calibrations, such as bias and darks, it is usually sufficient to apply the combined calibration to their input bias and dark frames, to confirm that the residuals are consistent with only the read noise remaining. +For simpler calibrations, such as biases and darks, it is usually sufficient to apply the combined calibration to their input bias and dark frames to confirm that the residuals are consistent with only the read noise remaining. Other calibrations require more analysis, such as applying the BFE correction to a series of on-sky images and confirming that the distribution of source sizes is no longer a function of the source flux. Calibrations that do not pass some tests are not necessarily rejected outright, although those failures should be raised and considered during the acceptance process. -This is particularly true of tests that examine a property at the amplifier segment level; rejecting a calibration due to a small number of problematic segments is impractical. +This is particularly true of tests that examine a property at the amplifier segment level; rejecting a calibration due to a small number of problematic segments is impractical. Where calibration construction does not produce a result that passes all of the requirements, we can add to our defect list to exclude regions that have serious issues from further processing. -The current set of tests used, and the criteria used for acceptance are detailed in DMTN-101, with a brief summary listed in Table \ref{tab:verification_tests}. +The current set of tests and the criteria used for acceptance are detailed in DMTN-101, with a brief summary listed in Table~\ref{tab:verification_tests}. It is likely that we will expand the list of verification tests calculated for calibrations as our familiarity with the camera and data improves. \begin{deluxetable*}{llll}[t!] @@ -46,7 +46,7 @@ \subsection{Acceptance} \label{sec:verify-accept-certify:acceptance} Once a set of calibrations has been generated and verified, a meeting of the Telescope and Auxiliary Instrumentation Calibration Acceptance Board (TAXICAB) is called to decide if those calibrations should be deployed for use. The TAXICAB does not have a fixed membership, and welcomes any member of the project to join to offer opinions and comments on the calibrations presented. -The goal of this board to ensure that the decision making for calibration approval is public and accessible, with the final decisions made by consensus agreement. +The goal of this board is to ensure that the decision-making for calibration approval is public and accessible, with the final decisions made by consensus agreement. Associated with each TAXICAB meeting is a ``hailing ticket,'' which is used to provide a location for details and artifacts from the generation and verification process. These artifacts of the generation and verification process are presented at the TAXICAB meeting, where the attendees can decide whether or not the calibration is sufficient, and whether further tests are needed to determine if the calibration performs as expected. A calibration with failures of some of the verification tests is not necessarily excluded, and may be allowed for use based on the significance of the failures, with the TAXICAB making that final decision. @@ -62,7 +62,7 @@ \subsection{Certification} \label{sec:verify-accept-certify:certification} \subsection{Distribution} \label{sec:verify-accept-certify:distribution} Distribution of LSSTCam calibrations is also a butler database operation. The certified calibration collections are gathered into a single CHAINED collection named \texttt{LSSTCam/calib}, which chains the calibrations from the successive calibration epochs along with their associated validity ranges. This calibration collection is synchronized across all data repositories and data facilities used for data production, including the summit facility, the SLAC Shared Scientific Data Facility (S3DF) at the United States Data Facility (USDF), the French Data Facility (FrDF) at CC-IN2P3 in Lyon, and the United Kingdom Data Facility (UKDF). -This ensures that the same calibrations are applied to the same images regardless of where data production is run and data is served, and it ensures that the Prompt Products data pipeline, which produces the live alert stream always uses the most up-to-date calibrations. +This ensures that the same calibrations are applied to the same images regardless of where data production is run and data are served, and it ensures that the Prompt Products data pipeline, which produces the live alert stream, always uses the most up-to-date calibrations. Processing data with \texttt{LSSTCam/calib} therefore always applies the calibrations appropriate to each science image, without any manual selection on the part of the user. \subsection{Data Monitoring} \label{sec:verify-accept-certify:data-monitoring} @@ -70,69 +70,22 @@ \subsection{Data Monitoring} \label{sec:verify-accept-certify:data-monitoring} One additional advantage of the verification system is that it provides a way to monitor the state of the camera, and the continuing appropriateness of the calibration products. By running the verification tasks on new calibration exposures as they are taken, we can check how well the existing calibrations correct these new exposures. Although not running during the DP2 observations and analysis, this verification will be added to the processing performed at the summit, with the metrics calculated from this processing passed to the Sasquatch and Chronograf monitoring system that is used to track other performance quantities. -This monitoring will allow the health of the camera to be tracked, and for the generation of new calibrations to be prompted by any observed changes in the metrics over time. +This monitoring will allow the health of the camera to be tracked and the generation of new calibrations to be prompted by any observed changes in the metrics over time. -\subsection{Data products, Distribution, and User Access} \label{sec:verify-accept-certify:data-products} -Describe the output schema, distribution process (exporting/importing, rucio, TAXICAB tracking), and user access. +A drift in these metrics showing that the current calibrations no longer describe the instrument within tolerance is one of several triggers for a new calibration epoch (\S\ref{sec:discussion:sequencer}). +A new epoch, with a complete new set of calibration products, is also started whenever any of the following happens: +\begin{itemize} + \item a change to the sequencer or to any other part of the CCD operating mode (e.g.\ clock voltages or timing); + \item a power cycle of the focal plane or the readout electronics, such as during maintenance, which can shift bias levels, gains, and defect populations; + \item a hardware intervention on the camera, such as recovering or replacing a detector or readout board. +\end{itemize} +Each new set of calibrations then passes through the same verification, acceptance, and certification steps described above. + +\subsection{Data Products, Distribution, and User Access} \label{sec:verify-accept-certify:data-products} +Describe the output schema, distribution process (exporting/importing, Rucio, and TAXICAB tracking), and user access. We enumerate the primary data products (calibration frames, ISR outputs, masks, and metadata tables), describe naming/versioning conventions, and summarize the provenance tracked for each product (inputs, software versions, and configuration). Briefly describe how users discover and retrieve products (e.g., via a data butler/repository interface) and recommended query patterns. Be sure to mention that header metadata contains information about whether a particular ISR step was applied/not applied and which calibration dataset was used. - -\section{Limitations \& Known Issues} \label{sec:limitations} - -The ISR pipeline developed for LSSTCam removes the dominant instrumental signatures, but several effects remain partially corrected, unmodeled, or unmasked. -The paragraphs below summarize known limitations (instrumental systematics not fully corrected, edge cases, validity constraints, etc.) for interpreting DP2 data, our recommendations for handling them in the context of DP2 science, and plans for near-term improvements expected before Data Release~1 (DR1). - -\paragraph{Flares} -Some science images contain localized, bright flare-like structures.\todo{Quantify?} -They appear as elongated trails that begin at a detector edge and fade with a wavelength-dependent power-law profile consistent with attenuation of light in silicon at that wavelength. -The light is likely scattered from bright stars: focused starlight may reflect from the beveled edge of the silicon mount and enter a neighboring detector through its side, although this mechanism is not yet confirmed. -Similar structures appear in some flat-field images as many small edge flares (referred to as a ``comb'' pattern), possibly linked to the rough-cut silicon edge. -So far, these features have been seen only in $y$-band, with both focused and unfocused illumination, and they are transient, consistent with a dependence on the chance alignment of bright stars with the focal-plane array. -They are not yet fully characterized or masked in DP2 processing. -Science users should treat anomalously bright detections/extended objects near detector edges with caution in $y$-band single-visit and difference images, and verify suspect sources by visual inspection. - -\paragraph{Residual non-linearity wiggles} -The double-spline non-linearity model (\S\ref{sec:processing:calibration:linearity}) removes the dominant signal-dependent departure from linear response, but small quasi-oscillatory residuals remain as a function of signal level. -They appear as low-amplitude, signal-dependent structure in linearized images. -These ``wiggles'' are irregular, clearly observed beyond noise fluctuations, and they do not correlate with measured properties of the input flat-field images (e.g. time, temperature) or with channel on a detector or across the focal plane. -These residuals are well below current science requirements and are left uncorrected in DP2, but further characterization is in progress.\todo{Quantify?} - -\paragraph{Differential non-linearity (DNL)} -The DP2 linearizer corrects average amplifier non-linearity but does not yet apply a full low-signal differential non-linearity correction. -This limitation reflects the difficulty of delivering flat-field illumination below 100 ADU/pixel with the in-dome calibration system. -Users should exercise caution when relying on photometric or astrometric measurements at low signal levels, in particular when relying on $u$-band sky-background estimates from later processing steps, where backgrounds are typically at or below 100 ADU/pixel. -Count dithering in ISR (\S\ref{sec:isr-science:overview}) provides interim mitigation until a dedicated DNL correction is measured and deployed. - -\paragraph{Detector vignetting models} -Flat-field calibrations measure relative pixel and filter quantum efficiencies and throughputs under uniform illumination, but they require an accurate model of optical vignetting across the focal plane. -Improved vignetting models, informed by larger calibration datasets and cross-checks against on-sky stellar photometry, are planned for future calibration epochs to reduce focal-plane non-uniformity.\todo{Quantify?} - -\paragraph{Color-dependent BFE corrections} -The BFE is color dependent because its amplitude depends on photon conversion depth in silicon and on the path length over which the transverse electric field acts on drifting electrons. -Laboratory and commissioning measurements show that the effect is chromatic \citep[$a_{00}^{y-band} = 0.94 \times a_{00}^{r-band}$, see][]{2024PASP..136d5003B}. -The electrostatic BFE model used in DP2 (\S\ref{sec:processing:calibration:bf}) is fit from PTC covariances and applies a single wavelength-averaged correction per detector, derived in the band of the PTC input flats. -Because the effect is strongest near the bottom of the potential well and the front of the chip, calibrations for $u$ through $i$ bands are approximately consistent as most light in those bands converts to photoelectrons within the first few microns. -Separate calibrations are appropriate for $z$- and $y$-band light, which typically convert far deeper into the silicon. -Boundary-shift calibrations can be color-corrected by assuming a flat incident spectral energy distribution in each filter band and a depth-conversion model in silicon. -These corrections were not enabled in DP2 because validation was incomplete, but they are expected for DR1. -Until then, PSF photometry and shape measurements of bright sources, especially in redder bands, may retain small flux- and filter-dependent biases.\todo{Quantify?} - -\paragraph{Fringing} -Fringing is an interference pattern produced when narrow-band emission reflects within the thin, back-illuminated silicon of LSSTCam CCDs. -It is driven primarily by night-sky emission lines and becomes appreciable in the $z$- and $y$-bands, where the silicon is increasingly transmissive and OH and other atmospheric lines contribute strongly to the background. -In-dome flat-field exposures (\S\ref{sec:processing:calibration:flats}) imprint and partially divide out a fringe pattern, but the fringing amplitude and phase in science images depend on the time-varying on-sky spectrum and therefore are not fully canceled by flat-fielding alone. -The DP2 ISR pipeline (\S\ref{sec:isr-science}) does not include a dedicated fringe-subtraction step. -Residual fringe structure is visible at low surface brightness in $z$- and $y$-band visit images and can bias wide-aperture photometry.\todo{Quantify?} -Fringing models developed by \citet{2023PASP..135c4503G} and corresponding corrections are under development and expected for DR1; until then, users analyzing faint extended emission or precision surface-brightness measurements in redder bands should inspect visit images for quasi-periodic fringe patterns and treat affected regions with appropriate caution. - -\paragraph{Masking} -Several limitations for the defects exist for DP2: -\begin{itemize} -\item vampire pixels -\item edge bleeds -\end{itemize}\todo{Quantify?} -Further developments will provide better support and masking for LSST data releases. \ No newline at end of file