Skip to content

DM-56808: add LSDB to 101_3, 201_1, and 201_4 #231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MelissaGraham merged 2 commits into from
Oct 23, 2026
Merged

DM-56808: add LSDB to 101_3, 201_1, and 201_4 #231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
31fa762
add lsdb to 101 3
MelissaGraham Oct 23, 2026
c84599c
add lsdb to 201 01 and 04
MelissaGraham Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 104 additions & 3 deletions DP2/100_How_to_use_RSP_tools/101_Get_started/101_3_Data_services_overview.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"Data Release: [Data Preview 2](https://dp2.lsst.io/)\\\n",
"Container Size: Large\\\n",
"LSST Science Pipelines version: r30.0.11\\\n",
"Last verified to run: 2026-08-18\\\n",
"Last verified to run: 2026-10-23\\\n",
"Repository: [github.com/lsst/tutorial-notebooks](https://github.com/lsst/tutorial-notebooks)\\\n",
"DOI: [10.11578/rubin/dc.20250909.20](https://doi.org/10.11578/rubin/dc.20250909.20)"
]
Expand Down Expand Up @@ -73,6 +73,7 @@
"* TAP and SIA queries:\n",
" * `lsst.rsp.RSPDiscovery`\n",
" * `lsst.images.serialization.read_archive`\n",
"* LSDB: `lsdb`\n",
"* Image display: `lsst.afw.display`"
]
},
Expand All @@ -86,6 +87,8 @@
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import io\n",
"from upath import UPath\n",
"import lsdb\n",
"from lsst.daf.butler import Butler\n",
"from lsst.rsp import RSPDiscovery\n",
"from lsst.images.serialization import read_archive\n",
Expand Down Expand Up @@ -444,6 +447,104 @@
"del query, refs, column_names, tbl"
]
},
{
"cell_type": "markdown",
"id": "c3bf0ced-5db1-4941-8111-70977cd2751e",
"metadata": {},
"source": [
"### 2.3. LSDB\n",
"\n",
"[LSDB](https://docs.lsdb.io/) is a python tool for scalable analyses that operate on most or all of a large catalog.\n",
"Built on top of [Dask](https://docs.dask.org/), LSDB uses the [HATS](https://hats.readthedocs.io/) (Hierarchical Adaptive Tiling Scheme) data format ([HEALPix](https://healpix.sourceforge.io/documentation.php)-sharded [Parquet](https://parquet.apache.org/docs/)) to efficiently perform spatial operations.\n",
"\n",
"For DP2, the `Object` and `DiaObject` tables are provided in LSDB format, and photometric redshift estimates are *only* provided in LSDB format.\n",
"The LSDB files are located in the directory `/rubin/lsdb_data`.\n",
"\n",
"**Why use LSDB?** \n",
"Catalogs are opened lazily: only the schema is read at first, and data are read on demand. Computations are parallelized with Dask, enabling queries and column-based filtering over the full catalogs. LSDB efficiently cross-matches the DP2 catalogs against external catalogs that are also distributed in HATS format (e.g., Gaia, ZTF, Pan-STARRS). Lightcurves are stored as nested columns alongside each object, so per-object time-series computations (e.g., variability statistics, period finding) can be applied across the full catalog without expensive joins. [Additional LSDB information and tutorials for DP2 data.](https://dp2.lsst.io/products/user/lsdb.html)\n",
"\n",
"**Related tutorials for catalogs with LSDB:**\n",
"* The 100-level series \"Catalog access\" contains an LSDB tutorial.\n",
"\n",
"**Repeat the TAP query above with LSDB.**\n",
"This simple demo does not demonstrate the power of LSDB's special design - that is left to other tutorials."
]
},
{
"cell_type": "markdown",
"id": "f8b6cead-e809-4ee7-a8c7-cc6c58b981cf",
"metadata": {},
"source": [
"Set the path to the LSDB data, define the columns to use, and execute the query."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88097754-9fd4-4d1b-acc0-495854ce189f",
"metadata": {},
"outputs": [],
"source": [
"base_path = UPath(\"/rubin/lsdb_data/dp2\")\n",
"use_columns = ['objectId', 'coord_ra', 'coord_dec', 'g_cModelMag', 'r_cModelMag']\n",
"object_cat = lsdb.open_catalog(base_path / \"object_collection\", columns=use_columns)\n",
"object_cat_ecdfs = object_cat.cone_search(ra=targ_ra, dec=targ_dec,\n",
" radius_arcsec=targ_rad * 3600.0)\n",
"query = \"g_cModelMag < 23 and r_cModelMag < 23 and g_cModelMag > 17 and r_cModelMag > 17\"\n",
"object_cat_ecdfs_query = object_cat_ecdfs.query(query)\n",
"table = object_cat_ecdfs_query.compute()"
]
},
{
"cell_type": "markdown",
"id": "c1e18d67-ed15-47aa-be1c-301f4f4cfc59",
"metadata": {},
"source": [
"Recreate Figure 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "40637d4b-3d9e-4c5a-85fa-a4b1a3e7fcf9",
"metadata": {},
"outputs": [],
"source": [
"fig = plt.figure(figsize=(4, 3))\n",
"plt.plot(table['g_cModelMag']-table['r_cModelMag'],\n",
" table['r_cModelMag'], 'o', ms=3, mew=0, alpha=0.2, color='black')\n",
"plt.xlabel('g-r')\n",
"plt.ylabel('r')\n",
"plt.ylim(23, 17)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "4cc862c3-78ef-4b02-9dbd-82c7d18e6f48",
"metadata": {},
"source": [
">**Figure 3:** The same as Figure 1, but the data came from the LSDB-formatted `Object` catalog."
]
},
{
"cell_type": "markdown",
"id": "9897bed7-257a-4a3c-b356-85728a160b3d",
"metadata": {},
"source": [
"Delete the query and the results."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d2ae2c6-82e1-4bec-9cc6-babdfa8ff3d8",
"metadata": {},
"outputs": [],
"source": [
"del base_path, use_columns, object_cat, object_cat_ecdfs, query, object_cat_ecdfs_query, table"
]
},
{
"cell_type": "markdown",
"id": "90848d44-d59d-4dc2-b43e-451a9d7c623d",
Expand Down Expand Up @@ -520,7 +621,7 @@
"\n",
"</div>\n",
"\n",
"> **Figure 3:** The $r$-band deep coadd image displayed in the Firefly window."
"> **Figure 4:** The $r$-band deep coadd image displayed in the Firefly window."
]
},
{
Expand Down Expand Up @@ -918,7 +1019,7 @@
"\n",
"</div>\n",
"\n",
"> **Figure 4:** The cutout as displayed in the Firefly tab."
"> **Figure 5:** The cutout as displayed in the Firefly tab."
]
},
{
Expand Down
87 changes: 85 additions & 2 deletions DP2/200_Data_products/201_Catalogs/201_01_Object_table.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"Data Release: [Data Preview 2](https://dp2.lsst.io)\\\n",
"Container Size: Large\\\n",
"LSST Science Pipelines version: r30.0.11\\\n",
"Last verified to run: 2026-08-18\\\n",
"Last verified to run: 2026-10-23\\\n",
"Repository: [github.com/lsst/tutorial-notebooks](https://github.com/lsst/tutorial-notebooks)\\\n",
"DOI: [10.11578/rubin/dc.20250909.20](https://doi.org/10.11578/rubin/dc.20250909.20)"
]
Expand Down Expand Up @@ -78,7 +78,8 @@
"\n",
"Import standard python packages `re`, `numpy`, `matplotlib`, and `astropy`.\n",
"\n",
"From the `lsst` package, import modules for the TAP service and the butler."
"From the `lsst` package, import modules for the TAP service and the Butler.\n",
"Also import the `lsdb` package to read an LSDB-formatted version of the `Object` table."
]
},
{
Expand All @@ -91,6 +92,8 @@
"import re\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from upath import UPath\n",
"import lsdb\n",
"from lsst.rsp import RSPDiscovery\n",
"from lsst.daf.butler import Butler"
]
Expand Down Expand Up @@ -985,6 +988,86 @@
"source": [
"del query, refs, col_list, results, tx"
]
},
{
"cell_type": "markdown",
"id": "bf7c77ce-1c5a-4422-8cf1-002a1894128b",
"metadata": {},
"source": [
"### 3.4. LSDB\n",
"\n",
"[LSDB](https://docs.lsdb.io/) is a python tool for scalable analyses that operate on most or all of a large catalog.\n",
"The `Object` table is one of the three DP2 tables provided in LSDB format, along with `DiaObject` and photometric redshift estimates.\n",
"[Additional LSDB information and tutorials for DP2 data.](https://dp2.lsst.io/products/user/lsdb.html)\n",
" \n",
"As a simple demo of accessing LSDB data, recreate Figure 1 using data obtained from the LSDB-formatted `Object` table.\n",
"First, set the path to the LSDB data, define the columns to use, and execute a query that matches Section 3.2.1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "744acac4-9a00-4846-a830-4a8e1f181e06",
"metadata": {},
"outputs": [],
"source": [
"base_path = UPath(\"/rubin/lsdb_data/dp2\")\n",
"use_columns = ['objectId', 'coord_ra', 'coord_dec', 'r_cModelMag', 'r_cModelMagErr']\n",
"object_cat = lsdb.open_catalog(base_path / \"object_collection\", columns=use_columns)\n",
"object_cat_ecdfs = object_cat.cone_search(ra=53.13, dec=-28.10,\n",
" radius_arcsec=0.2 * 3600.0)\n",
"query = \"r_cModelMag < 25\"\n",
"object_cat_ecdfs_query = object_cat_ecdfs.query(query)\n",
"table = object_cat_ecdfs_query.compute()"
]
},
{
"cell_type": "markdown",
"id": "84e2ab7f-09b6-4b42-a5aa-857dc00304c3",
"metadata": {},
"source": [
"Create the plot."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49080735-196d-4b87-874a-d26ee68b1f34",
"metadata": {},
"outputs": [],
"source": [
"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(7, 3))\n",
"ax1.plot(table['coord_ra'], table['coord_dec'],\n",
" 'o', ms=2, mew=0, alpha=0.4, color='grey')\n",
"ax1.set_xlabel('Right Ascension')\n",
"ax1.set_ylabel('Declination')\n",
"ax2.plot(table['r_cModelMag'], table['r_cModelMagErr'],\n",
" 'o', ms=2, mew=0, alpha=0.4, color='grey')\n",
"ax2.set_xlabel('Apparent Magnitude')\n",
"ax2.set_ylabel('Error')\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "152f1976-d91f-4cf2-9f1f-2fedd9bd5741",
"metadata": {},
"source": [
">**Figure 3:** A recreation of Figure 1, using the LSDB-formatted `Object` table.\n",
"\n",
"Clean up."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f96f4813-be3a-48ba-9105-7cb500d4169a",
"metadata": {},
"outputs": [],
"source": [
"del base_path, use_columns, object_cat, object_cat_ecdfs, query, object_cat_ecdfs_query, table"
]
}
],
"metadata": {
Expand Down
78 changes: 74 additions & 4 deletions DP2/200_Data_products/201_Catalogs/201_04_DiaObject_table.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"Data Release: [Data Preview 2](https://dp2.lsst.io/)\\\n",
"Container Size: Large\\\n",
"LSST Science Pipelines version: r30.0.11\\\n",
"Last verified to run: 2026-08-18\\\n",
"Last verified to run: 2026-10-23\\\n",
"Repository: [github.com/lsst/tutorial-notebooks](https://github.com/lsst/tutorial-notebooks)\\\n",
"DOI: [10.11578/rubin/dc.20250909.20](https://doi.org/10.11578/rubin/dc.20250909.20)"
]
Expand Down Expand Up @@ -78,7 +78,8 @@
"\n",
"Import standard python packages `numpy`, `matplotlib`, and `astropy`.\n",
"\n",
"From the `lsst` package, import modules for the TAP service and the butler."
"From the `lsst` package, import modules for the TAP service and the Butler.\n",
"Also import the `lsdb` package to read an LSDB-formatted version of the `Object` table."
]
},
{
Expand All @@ -91,6 +92,8 @@
"import re\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from upath import UPath\n",
"import lsdb\n",
"from lsst.rsp import RSPDiscovery\n",
"from lsst.daf.butler import Butler"
]
Expand Down Expand Up @@ -790,13 +793,80 @@
"del query, refs, col_list, results, tx"
]
},
{
"cell_type": "markdown",
"id": "9b21e2ca-a7ec-4645-b7be-682f8d191d24",
"metadata": {},
"source": [
"### 3.4. LSDB\n",
"\n",
"[LSDB](https://docs.lsdb.io/) is a python tool for scalable analyses that operate on most or all of a large catalog.\n",
"The `DiaObject` table is one of the three DP2 tables provided in LSDB format, along with `Object` and photometric redshift estimates.\n",
"[Additional LSDB information and tutorials for DP2 data.](https://dp2.lsst.io/products/user/lsdb.html)\n",
" \n",
"As a simple demo of accessing LSDB data, recreate Figure 1 using data obtained from the LSDB-formatted `DiaObject` table.\n",
"First, set the path to the LSDB data, define the columns to use, and execute a query that matches Section 3.2.1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9b21e2ca-a7ec-4645-b7be-682f8d191d24",
"id": "16b3b975-4d75-48b7-8ba3-22d1f1dfbaec",
"metadata": {},
"outputs": [],
"source": [
"base_path = UPath(\"/rubin/lsdb_data/dp2\")\n",
"use_columns = ['diaObjectId', 'nDiaSources', 'r_psfFluxNdata']\n",
"dia_object_cat = lsdb.open_catalog(base_path / \"dia_object_collection\", columns=use_columns)\n",
"dia_object_cat_ecdfs = dia_object_cat.cone_search(ra=53.13, dec=-28.10, radius_arcsec=0.2 * 3600.0)\n",
"query = \"r_psfFluxNdata > 5\"\n",
"dia_object_cat_ecdfs_query = dia_object_cat_ecdfs.query(query)\n",
"table = dia_object_cat_ecdfs_query.compute()"
]
},
{
"cell_type": "markdown",
"id": "67c890b7-b375-4363-8e2f-20ef8f076bd1",
"metadata": {},
"source": [
"Create the plot."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4da7c230-c155-4e9e-838e-652800cc5bdb",
"metadata": {},
"outputs": [],
"source": []
"source": [
"plt.figure(figsize=(4, 3))\n",
"plt.hist(table['nDiaSources'], bins=20, log=True, color='gray')\n",
"plt.xlabel('nDiaSources')\n",
"plt.ylabel('Counts of diaObjects')\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cd940c00-8c45-409a-adec-f6a1c9b6d3bb",
"metadata": {},
"source": [
">**Figure 4:** A recreation of Figure 1, using the LSDB-formatted `DiaObject` table.\n",
"\n",
"Clean up."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c93fb9f0-8cbe-44bb-aead-5d55014be90c",
"metadata": {},
"outputs": [],
"source": [
"del base_path, use_columns, dia_object_cat, dia_object_cat_ecdfs\n",
"del query, dia_object_cat_ecdfs_query, table"
]
}
],
"metadata": {
Expand Down
Loading