2tops2mintpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare ISCE topsApp interferograms for processing with MintPy\n",
    "\n",
    "We often use `topsApp.py` to process sentinel-1 data. However, MintPy can only read the results of `topsStack` directly at present. We need some post-processing to make MintPy can produce the  result of `topsApp.py` ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the structured folders\n",
    "\n",
    "The folder structure required by Minpy to handle tops is as follows [ISCE / topsStack](https://mintpy.readthedocs.io/en/latest/dir_structure/):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "```bash\n",
    "$DATA_DIR/GalapagosSenDT128\n",
    "├── baselines\n",
    "│   ├── 20141213_20141225\n",
    "│   │   └── 20141213_20141225.txt\n",
    "│   └── ...\n",
    "├── reference\n",
    "│   ├── data.rsc  # generated by prep_isce.py\n",
    "│   ├── IW1\n",
    "│   ├── IW1.xml\n",
    "│   ├── IW2\n",
    "│   └── IW2.xml\n",
    "├── merged\n",
    "│   ├── geom_reference\n",
    "│   │   ├── hgt.rdr\n",
    "│   │   ├── hgt.rdr.full.vrt\n",
    "│   │   ├── hgt.rdr.full.xml\n",
    "│   │   ├── hgt.rdr.vrt\n",
    "│   │   ├── hgt.rdr.xml\n",
    "│   │   ├── lat.rdr\n",
    "│   │   ├── lat.rdr.full.vrt\n",
    "│   │   ├── lat.rdr.full.xml\n",
    "│   │   ├── lat.rdr.vrt\n",
    "│   │   ├── lat.rdr.xml\n",
    "│   │   ├── lon.rdr\n",
    "│   │   ├── lon.rdr.full.vrt\n",
    "│   │   ├── lon.rdr.full.xml\n",
    "│   │   ├── lon.rdr.vrt\n",
    "│   │   ├── lon.rdr.xml\n",
    "│   │   ├── los.rdr\n",
    "│   │   ├── los.rdr.full.vrt\n",
    "│   │   ├── los.rdr.full.xml\n",
    "│   │   ├── los.rdr.vrt\n",
    "│   │   ├── los.rdr.xml\n",
    "│   │   ├── shadowMask.rdr\n",
    "│   │   ├── shadowMask.rdr.full.vrt\n",
    "│   │   ├── shadowMask.rdr.full.xml\n",
    "│   │   ├── shadowMask.rdr.vrt\n",
    "│   │   └── shadowMask.rdr.xml\n",
    "│   └── interferograms\n",
    "│       ├── 20141213_20141225\n",
    "│       │   ├── filt_fine.cor\n",
    "│       │   ├── filt_fine.cor.vrt\n",
    "│       │   ├── filt_fine.cor.xml\n",
    "│       │   ├── filt_fine.unw\n",
    "│       │   ├── filt_fine.unw.conncomp\n",
    "│       │   ├── filt_fine.unw.conncomp.vrt\n",
    "│       │   ├── filt_fine.unw.conncomp.xml\n",
    "│       │   ├── filt_fine.unw.vrt\n",
    "│       │   ├── filt_fine.unw.xml\n",
    "│       │   ├── ...\n",
    "│       ├── 20141213_20150307\n",
    "│       └── ...\n",
    "├── secondarys\n",
    "│   ├── 20141225\n",
    "│   │   ├── IW1\n",
    "│   │   ├── IW1.xml\n",
    "│   │   ├── IW2\n",
    "│   │   └── IW2.xml\n",
    "│   ├── 20150307\n",
    "│   └── ...\n",
    "└── mintpy\n",
    "└── GalapagosSenDT128.txt\n",
    "```\n",
    "Suppose you process all interferograms in one folder, which has the folder structure as follows:\n",
    "\n",
    "```bash\n",
    ".\n",
    "|-- 20210917_20210929\n",
    "|-- 20210917_20211011\n",
    "|-- 20210917_20211023\n",
    "|-- 20210917_20211210\n",
    "|-- 20210929_20211011\n",
    "|-- 20210929_20211023\n",
    "|-- 20210929_20211104\n",
    "|-- ...\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, import packages and specify the home directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm\n",
    "\n",
    "# the folder that contains the interferograms processed by topsApp.py\n",
    "ifg_home = Path('/data/sentinel1/result/interferogams')  \n",
    "\n",
    "# the folder that used to run MintPy\n",
    "stack_home = Path('/data/sentinel1/result/stacks')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's create the folder structure that MintPy required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "############  make dirs #############\n",
    "def ensure_folder(folder):\n",
    "    if not folder.is_dir():\n",
    "        folder.mkdir(parents=True)\n",
    "\n",
    "baseline_dir = stack_home / 'baselines'\n",
    "reference_dir = stack_home / 'reference'\n",
    "secondarys_dir = stack_home / 'secondarys'\n",
    "mintpy_dir = stack_home / 'mintpy'\n",
    "\n",
    "merged_dir = stack_home / 'merged'\n",
    "geom_reference_dir = merged_dir / 'geom_reference'\n",
    "interferograms_dir = merged_dir / 'interferograms'\n",
    "\n",
    "stack_fld_list = [baseline_dir, reference_dir,\n",
    "                  secondarys_dir, mintpy_dir,\n",
    "                  merged_dir, geom_reference_dir,\n",
    "                  interferograms_dir]\n",
    "\n",
    "for fld in stack_fld_list:\n",
    "    ensure_folder(fld)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Link files \n",
    "\n",
    "We have created the folders that MintPy required. But there are no files in those folders. We need to link or create files from ISCE processed into those folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def looks(vrt_file, rdr_file, nlook_a, nlook_r):\n",
    "    os.system(f'looks.py -i {vrt_file} -o {rdr_file} -a {nlook_a} -r {nlook_r}')\n",
    "\n",
    "ifgs_files = sorted(ifg_home.iterdir())\n",
    "for file in tqdm(ifgs_files, desc='link files'):\n",
    "    name = file.name\n",
    "    ref_name, sec_name = name.split('_')\n",
    "    merged = file / 'merged'\n",
    "    i = 0\n",
    "    if merged.exists():\n",
    "        os.chdir(file)\n",
    "        if i == 0:\n",
    "            ####################################################################\n",
    "            # For the folder 'reference' and 'merged/geom_reference', you only \n",
    "            # need to get it from the first interferogram            \n",
    "            #####################################################################\n",
    "            i += 1\n",
    "            \n",
    "            ######### link to 'reference' folder ######\n",
    "            ref_dir = file / ref_name\n",
    "            ref_cmd = f'ln -sf {ref_dir}/* {reference_dir}/'\n",
    "            os.system(ref_cmd)\n",
    "\n",
    "            ####################################################################\n",
    "            #     multi-looking geo_info files into 'geom_reference' folder\n",
    "            #####################################################################\n",
    "            # For the files that contain the geo info of ifgs, only 'los' was \n",
    "            # processed well (multi-looked and with '.rdr' format). We need \n",
    "            # to convert the full resolution geo info files into multi-looked \n",
    "            # files with ISCE format('.rdr'). In this script, azimuth/range looks\n",
    "            # are: 4*20.\n",
    "            #####################################################################\n",
    "            looks(merged.joinpath('z.rdr.full.xml'),\n",
    "                    geom_reference_dir.joinpath('hgt.rdr'),\n",
    "                    4,20)\n",
    "            looks(merged.joinpath('lat.rdr.full.xml'),\n",
    "                    geom_reference_dir.joinpath('lat.rdr'),\n",
    "                    4,20)\n",
    "            looks(merged.joinpath('lon.rdr.full.xml'),\n",
    "                    geom_reference_dir.joinpath('lon.rdr'),\n",
    "                    4,20)\n",
    "            looks(merged.joinpath('los.rdr.full.xml'),\n",
    "                    geom_reference_dir.joinpath('los.rdr'),\n",
    "                    4,20)\n",
    "\n",
    "\n",
    "        # ########## link ifgs  ################\n",
    "        ifg_out = interferograms_dir / name\n",
    "        ensure_folder(ifg_out)\n",
    "\n",
    "        # ifgs\n",
    "        ln_cmd1 = f'ln -sf {merged}/filt_* {ifg_out}/'\n",
    "        os.system(ln_cmd1)\n",
    "\n",
    "        # cor\n",
    "        ln_cmd2 = f'ln -sf {merged}/phsig.cor* {ifg_out}/'\n",
    "        os.system(ln_cmd2)\n",
    "        # ########## link secondarys  ################\n",
    "        sec_dir = file / sec_name\n",
    "        sec_cmd = f'ln -sf {sec_dir} {secondarys_dir}/'\n",
    "        os.system(sec_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generate baseline\n",
    "We can use `isce2/contrib/stack/topsStack/computeBaseline.py` to generate baselines. This script was not installed by default, for example, installed by `conda`. You need to find its location from the source code of [ISCE](https://github.com/isce-framework/isce2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "########## generate_baseline  #############\n",
    "def get_dates_from_ifgs(ifgs):\n",
    "    dates_all = []\n",
    "    for ifg in ifgs:\n",
    "        dates_all.extend(ifg.split('_'))\n",
    "    dates = sorted(set(dates_all))\n",
    "    return dates\n",
    "\n",
    "\n",
    "ifgs = [i.name for i in ifgs_files]\n",
    "dates = get_dates_from_ifgs(ifgs)\n",
    "\n",
    "ref_name = dates[0]\n",
    "ref_dir = ifgs_files[0] / ref_name\n",
    "for sec_name in tqdm(dates[1:], desc='generate baselines'):\n",
    "    sec_dir = secondarys_dir / sec_name\n",
    "\n",
    "    bl_name = f'{ref_name}_{sec_name}'\n",
    "    bl_dir = baseline_dir / bl_name\n",
    "    ensure_folder(bl_dir)\n",
    "\n",
    "    bl_file = bl_dir / f'{bl_name}.txt'\n",
    "    bl_cmd = f'/data/isce2/contrib/stack/topsStack/computeBaseline.py -m {ref_dir}/ -s {sec_dir}/ -b {bl_file}'\n",
    "    os.system(bl_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## prep_isce.py\n",
    "\n",
    "For the folder `reference`, we need to use `prep_isce.py` to generate the `data.rsc` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(reference_dir)\n",
    "prep_cmd = ('/root/miniconda3/bin/prep_isce.py'\n",
    "            f' -f \"{interferograms_dir}/*/filt_*.unw\"'\n",
    "            f' -m {reference_dir}/IW2.xml'\n",
    "            f' -b {baseline_dir}/'\n",
    "            f' -g {geom_reference_dir}/')\n",
    "print(prep_cmd)\n",
    "os.system(prep_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything is ok, no errors will be reported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## write options file for MintPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg_file = mintpy_dir / 'mintpy_data_options.cfg'\n",
    "\n",
    "cfg_info = f'''mintpy.load.processor        = isce\n",
    "##---------for ISCE only:\n",
    "mintpy.load.metaFile         = {reference_dir}/IW*.xml\n",
    "mintpy.load.baselineDir      = {baseline_dir}\n",
    "##---------interferogram datasets:\n",
    "mintpy.load.unwFile          = {interferograms_dir}/*/filt_*.unw\n",
    "mintpy.load.corFile          = {interferograms_dir}/*/phsig.cor\n",
    "mintpy.load.connCompFile     = {interferograms_dir}/*/filt_*.unw.conncomp\n",
    "##---------geometry datasets:\n",
    "mintpy.load.demFile          = {geom_reference_dir}/hgt.rdr\n",
    "mintpy.load.lookupYFile      = {geom_reference_dir}/lat.rdr\n",
    "mintpy.load.lookupXFile      = {geom_reference_dir}/lon.rdr\n",
    "mintpy.load.incAngleFile     = {geom_reference_dir}/los.rdr\n",
    "mintpy.load.azAngleFile      = {geom_reference_dir}/los.rdr\n",
    "# mintpy.load.shadowMaskFile   = {geom_reference_dir}/shadowMask.rdr\n",
    "'''\n",
    "\n",
    "with open(cfg_file, 'w') as f:\n",
    "    f.write(cfg_info)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.13 ('geo')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "630114ded1cb6136faf7350823e03790fe4102d781074abef124d6f86b9fe410"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}