Merge pull request #19 from ljubicicrobert/develop

develop -> master @ v0.3.3.0

.gitignore

@@ -1,2 +0,0 @@
-
-README.md

README.md

@@ -14,7 +14,7 @@ Complete preprocessing tool for UAV image velocimetry - unpacking, filtering, st
     5.2. [Camera calibration](#camera-calibration)<br/>
     5.3. [Selecting features for tracking](#selecting-features-for-tracking)<br/>
     5.4. [Orthorectification](#orthorectification)<br/>
-    5.5. [Image filtering](#image-filtering)
+    5.5. [Image filtering / image enhancement](#image-filtering--image-enhancement)
 6. [Future features](#future-features)
 7. [Acknowledgements](#acknowledgements)
 8. [References](#references)
@@ -208,24 +208,36 @@ The GUI also offers a simple orthorectification to be performed by estimating th
 Users can also set a ground sampling distance (GSD, in px/m) to rescale the image and help with the postprocessing - use this feature carefully as it will always introduce additional errors/noise in the transformed images. **It's best to keep this ratio as close as possible to the original GSD!**. For these reasons user can click **Measure** button to quickly measure in-image distances and to compare them with real-world data in order to help determine appropriate GSD value.
 
 
-### Image filtering
+### Image filtering / image enhancement
 
 <img align="right" width="320" src="https://github.com/ljubicicrobert/SSIMS/blob/master/screenshots/filter_form.png">
 
+Image filtering/enhancement is often a crucial part of the image velocimetry workflow. This tools offers a form in which such filtering can easily be performed. For those users unfamiliar with the image filtering/enhancement process, it is advisable to visit the [Image enhancement for UAV velocimetry](https://github.com/ljubicicrobert/Image-enhancement-for-UAV-velocimetry) repository. In this repository, a detailed overview of different aspects of image enhancement is provided through a series of Jupyter notebooks, along with a detailed PDF instructions on how to perform [filtering with SSIMS tool](https://github.com/ljubicicrobert/Image-enhancement-for-UAV-velocimetry/blob/main/using_ssims.pdf).
+
 Users can perform filtering of images in a folder using the [Filter frames](https://github.com/ljubicicrobert/SSIMS/blob/master/screenshots/filter_form.png) form. In the form, user should first define the _frame folder path_ and the _extension_ of images in the folder. Available filters are listed on left side of the form, and the user can add them to the **filtering stack** on the right side by clicking on the desired filter. Only one filter of a given type can be selected at a time.
 
-Once the filters have been added to the filtering stack, user can edit the filter by clicking on the filter button in the filtering stack, which opens the [Filter parameters](https://github.com/ljubicicrobert/SSIMS/blob/master/screenshots/filter_parameters.png) form, or reorder the filters by dragging and dropping them at the desired position. IMPORTANT: Filters will be applied in the top-down order from the filtering stack.
+Before filtering, it is highly advisable to inspect the different colorspace models of the frames. This can be done by clicking on the **Explore colorspaces** button at the top-left of the **Filter frames** form.
+
+Once the filters have been added to the filtering stack, user can edit the filter by clicking on the filter button in the filtering stack, which opens the [Filter parameters](https://github.com/ljubicicrobert/SSIMS/blob/master/screenshots/filter_parameters.png) form, or reorder the filters by dragging and dropping them at the desired position.
 
 In the **Filter parameters** form, the user can remove the filter from the stack using the **Remove** button, or adjust the parameters of the filter (using the trackbar of using the neighboring numerical box) and finally apply the parameters using the **Apply** button.
 
 Once the filters have been chosen, user can preview the filtering results using the **Preview results** button in the bottom-middle section of the **Filter frames** form. In the opened window, user can toggle between the filtered and original image using the **SPACE** key.
 
+Some of the available filter are just colorspace model conversions (titled _Convert to..._). These will transform the image from the previous colorspace to the chosen one. **Default colorspace model**, which is active when the image is loaded for filtering, is **RGB**.
+
+> SSIMS will keep track of the colorspace conversions during filtering. For example, if _Convert to L\*a\*b\*_ is in the stack, followed by the _Single image channel_, by choosing the channel in the **Filter parameters** form, the user will effectively be selecting a channel of the image from its L\*a\*b\* colorspace model.
+
 Clicking **Apply filters** will initiate filtering on all frames in the selected folder.
 
+> **IMPORTANT**: Filters will be applied in the top-down order from the filtering stack.
+
 
 ### Future features
 
-&#9744; Additional filters for preprocessing
+&#9746; Additional filters for preprocessing
+
+&#9746; Colorspace model inspection and conversion
 
 &#9746; Complete camera calibration form (as of v0.3.1.0)
 
@@ -236,11 +248,13 @@ Clicking **Apply filters** will initiate filtering on all frames in the selected
 
 I wish to express my gratitude to the following people (in no particular order):
 
+[Mrs. Sophie Pierce](https://www.worcester.ac.uk/about/profiles/sophie-pearce) - for motivating me to start the work in the first place;
+
 [Dr Budo Zindović](https://www.grf.bg.ac.rs/fakultet/pro/e?nid=153) - for helping me with many implementational details and extensive testing of the tool;
 
-[Mrs. Sophie Pierce](https://www.worcester.ac.uk/about/profiles/sophie-pearce) - for motivating me to start the work in the first place;
+[Mrs. Dariia Strelnikova](https://www.fh-kaernten.at/en/en/faculty-and-staff-details?personId=4298793872) for supporting and reviewing the work related to the image enhancement;
 
-[Mrs. Dariia Strelnikova](https://www.fh-kaernten.at/en/en/faculty-and-staff-details?personId=4298793872) and [Dr Anette Eltner](https://tu-dresden.de/bu/umwelt/geo/ipf/photogrammetrie/die-professur/beschaeftigte/Anette_Eltner?set_language=en) - for testing the software and allowing me to learn from their own work;
+[Mrs. Dariia Strelnikova](https://www.fh-kaernten.at/en/en/faculty-and-staff-details?personId=4298793872) (again) and [Dr Anette Eltner](https://tu-dresden.de/bu/umwelt/geo/ipf/photogrammetrie/die-professur/beschaeftigte/Anette_Eltner?set_language=en) - for testing the software and allowing me to learn from their own work;
 
 [Dr Alonso Pizarro](https://www.researchgate.net/profile/Alonso_Pizarro) and [Dr Salvador Peña‐Haro](https://www.researchgate.net/profile/Salvador_Pena-Haro) - for providing me with valuable insights into their own work;
 
@@ -253,6 +267,8 @@ I wish to express my gratitude to the following people (in no particular order):
 
 Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004) *Image Quality Assessment: From Error Visibility to Structural Similarity*, IEEE Trans. Image Process., 13(4), 600–612, [https://doi.org/10.1109/TIP.2003.819861](https://doi.org/10.1109/TIP.2003.819861)
 
+Fast-SSIM by Chen Yu ([https://github.com/chinue/Fast-SSIM](https://github.com/chinue/Fast-SSIM))
+
 ### How to cite
 Ljubicic, R. (2021) *SSIMS: Preprocessing tool for UAV image velocimetry*, [https://github.com/ljubicicrobert/SSIMS](https://github.com/ljubicicrobert/SSIMS)
 

config_templates/config_tracking.txt

@@ -16,11 +16,6 @@ ImageExtension = jpg
 SearchAreaSize = 21
 InterrogationAreaSize = 11
 
-;Subpixel estimation kernel size, must be odd integer. If ==0 then no
-;subpixel estimation of the feature positions will be performed
-;
-Subpixel = 3
-
 
 [Advanced]
 

gui/SSIM Stabilization GUI.application

@@ -14,7 +14,7 @@
           <dsig:Transform Algorithm="urn:schemas-microsoft-com:HashTransforms.Identity" />
         </dsig:Transforms>
         <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha256" />
-        <dsig:DigestValue>3RPsOZN78HtDmcHYz9R2vEDeiPdk86kuVOnX2XWb9DA=</dsig:DigestValue>
+        <dsig:DigestValue>xTsGTwXf6YUDKRjOiQyQGplLaH1x8QeC7mrVI6jIOoE=</dsig:DigestValue>
       </hash>
     </dependentAssembly>
   </dependency>

gui/SSIM Stabilization GUI.exe.manifest

@@ -111,7 +111,7 @@
           <dsig:Transform Algorithm="urn:schemas-microsoft-com:HashTransforms.Identity" />
         </dsig:Transforms>
         <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha256" />
-        <dsig:DigestValue>1SUVMnjos5qUF8w1wPSg2INKvq5fIY0VWlQgF7N6+pQ=</dsig:DigestValue>
+        <dsig:DigestValue>2Zsn0rpzN0BWBml9ErP+r0j7IQnpG7B9Y4Z4fuNWVRM=</dsig:DigestValue>
       </hash>
     </dependentAssembly>
   </dependency>

gui/update.ini

@@ -1,8 +1,8 @@
 [Updates]
-CurrentReleaseTag = v0.3.2.0
-CurrentReleaseName = SSIMS_v0.3.2.0
-CurrentReleaseDate = 2022-04-05 17:57:43.422532
-LastCheckDate = 2022-04-05 17:57:43.422532
+CurrentReleaseTag = v0.3.3.0
+CurrentReleaseName = SSIMS_v0.3.3.0
+CurrentReleaseDate = 2022-04-11 18:18:09.384151
+LastCheckDate = 2022-04-11 18:18:09.384151
 DisableUpdateCheck = 0
 PauseDays = 0
 

help/HelpBasicTracking.help

@@ -6,9 +6,3 @@ Width/height (in pixels) of the feature area. Must be lower than the search area
 
 --- Search area size ---
 Width/height (in pixels) of the area used for feature detection/tracking. Must be higher than the interrogation area size. Must be odd integer.
-
---- Use subpixel estimator ---
-Whether to try to improve the feature position estimation accuracy using 2D Gaussian distribution.
-
---- Subpixel estimator ---
-Size of the 2D Gaussian kernel used for subpixel feature position estimation. Optimally 3 or 5 pixels. Must be odd integer.

release_notes.txt

@@ -1,3 +1,17 @@
+SSIMS_v0.3.3.0 ----------------------------------------------------------------
+
+Major changes:
+- Huge speed improvement for feature tracking by using https://github.com/chinue/Fast-SSIM
+- Significantly reduced computational complexity for feature tracking step
+- Removed option to choose subpixel estimator size, Gaussian 2x3 fit now default
+
+Minor changes:
+- Changes to console printing format, added progress bar
+- Bugfixes in filter_frames.py
+- Minor bugfixes in GUI
+- Added docstrings and type hints in Python files
+
+
 SSIMS_v0.3.2.0 ----------------------------------------------------------------
 
 Major changes:

scripts/FastSSIM/ssim.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""
+Fast SSIM implementation from https://github.com/chinue/Fast-SSIM.
+"""
+
+from os import path, name
+from skimage.metrics import structural_similarity as skimage_ssim
+from re import search
+
+import numpy as np
+import ctypes
+
+ssim_dll_path = path.split(path.realpath(__file__))[0]
+ssim_dll_name = 'ssim.dll' if name=='nt' else 'libssim.so'
+
+
+class Loader:
+    if(path.exists(path.join(ssim_dll_path, ssim_dll_name))):
+        dll = np.ctypeslib.load_library(ssim_dll_name, ssim_dll_path)
+
+    type_dict = {'int':ctypes.c_int, 'float':ctypes.c_float, 'double':ctypes.c_double, 'void':None,
+                 'int32':ctypes.c_int32, 'uint32':ctypes.c_uint32, 'int16':ctypes.c_int16, 'uint16':ctypes.c_uint16,
+                 'int8':ctypes.c_int8, 'uint8':ctypes.c_uint8, 'byte':ctypes.c_uint8,
+                 'char*':ctypes.c_char_p,
+                 'float*':np.ctypeslib.ndpointer(dtype='float32', ndim=1, flags='CONTIGUOUS'),
+                 'int*':np.ctypeslib.ndpointer(dtype='int32', ndim=1, flags='CONTIGUOUS'),
+                 'byte*':np.ctypeslib.ndpointer(dtype='uint8', ndim=1, flags='CONTIGUOUS')}
+
+    @staticmethod
+    def get_function(res_type='float', func_name='PSNR_Byte', arg_types=['Byte*', 'int', 'int', 'int', 'Byte*']):
+        func = Loader.dll.__getattr__(func_name)
+        func.restype = Loader.type_dict[res_type]
+        func.argtypes = [Loader.type_dict[str.lower(x).replace(' ', '')] for x in arg_types]
+        return func
+
+    @staticmethod
+    def get_function2(c_define='DLL_API float PSNR_Byte(const Byte* pSrcData, int step, int width, int height, OUT Byte* pDstData);'):
+        r = search(r'(\w+)\s+(\w+)\s*\((.+)\)', c_define)
+        assert(r!=None)
+        r = r.groups()
+        arg_list = r[2].split(',')
+        arg_types=[]
+
+        for a in arg_list:
+            a_list = a.split()
+
+            if('*' in a_list[-1]):
+                arg = a_list[-1].split('*')[0]+'*' if(a_list[-1][0]!='*') else a_list[-2]+'*'
+            else:
+                arg = a_list[-3]+'*' if(a_list[-2]=='*') else a_list[-2]
+
+            arg_types.append(arg)
+
+        return Loader.get_function(r[0], r[1], arg_types)
+
+    @staticmethod
+    def had_member(name='dll'):
+        return (name in Loader.__dict__.keys())
+
+
+class DLL:
+    @staticmethod
+    def had_function(name='PSNR_Byte'):
+        return (name in DLL.__dict__.keys())
+
+    if(Loader.had_member('dll')):
+        # float SSIM_Byte(Byte* pDataX, Byte* pDataY, int step, int width, int height, int win_size, int maxVal);
+        SSIM_Byte = Loader.get_function('float', 'SSIM_Byte', ['Byte*', 'Byte*', 'int', 'int', 'int', 'int', 'int'])
+
+
+def SSIM(x, y, win_size=7):
+    h, w = x.shape
+
+    if(DLL.had_function('SSIM_Byte') and x.dtype=='uint8' and y.dtype=='uint8'):
+        return DLL.SSIM_Byte(x.reshape([-1]), y.reshape([-1]), w, w, h, win_size, 255)
+    else:
+        return skimage_ssim(x,y, win_size=win_size, data_range=255)

scripts/__init__.py

@@ -4,13 +4,14 @@
 import cv2
 import numpy as np
 import configparser
+
 from argparse import ArgumentParser
 
 __package_name__ = 'SSIMS: Preprocessing tool for UAV image velocimetry'
 __description__ = 'Preprocessing and video stabilization tool for UAS/UAV image velocimetry based on Structural Similarity (SSIM) Index metric'
-__version__ = '0.3.2.0'
+__version__ = '0.3.3.0'
 __status__ = 'beta'
-__date_deployed__ = '2022-04-05'
+__date_deployed__ = '2022-04-11'
 
 __author__ = 'Robert Ljubicic, University of Belgrade - Civil Engineering Faculty'
 __author_email__ = '[email protected]'

scripts/camera_calibration.py

@@ -20,16 +20,18 @@
 """
 
 try:
-	import os
-	import glob
-	import matplotlib.pyplot as plt
-
 	from __init__ import *
+	from os import path, mkdir
+	from glob import glob
+	from class_console_printer import tag_string
+
+	import matplotlib.pyplot as plt
 
 except Exception as ex:
-	print('\n[EXCEPTION] Import failed: \n\n'
-	      '  {}'.format(ex))
-	input('\nPress ENTER/RETURN to exit...')
+	print()
+	print(tag_string('exception', 'Import failed: \n'))
+	print('  {}'.format(ex))
+	input('\nPress ENTER/RETURN key to exit...')
 	exit()
 
 
@@ -50,8 +52,8 @@
 		extension = args.ext
 		camera_model = 'camera_parameters' if args.model == '' else args.model
 
-		if not os.path.exists(output_folder):
-			os.mkdir(output_folder)
+		if not path.exists(output_folder):
+			mkdir(output_folder)
 
 		cheq_w = int(args.w) - 1
 		cheq_h = int(args.h) - 1
@@ -68,8 +70,8 @@
 
 		fig, ax = plt.subplots()
 
-		images = glob.glob('{}/*.{}'.format(input_folder, extension))
-		image_names = [os.path.basename(x) for x in images]
+		images = glob('{}/*.{}'.format(input_folder, extension))
+		image_names = [path.basename(x) for x in images]
 		num_images = len(images)
 		ret_list = [None] * num_images
 
@@ -81,37 +83,37 @@
 		try:
 			mng = plt.get_current_fig_manager()
 			mng.set_window_title('Chequerboard corners detection')
-		except:
+		except Exception:
 			pass
 
-		print('[START] Starting camera calibration using images in folder [{}]\n'.format(input_folder))
+		print(tag_string('start', 'Starting camera calibration using images in folder [{}]\n'.format(input_folder)))
 
 		for i, fname in enumerate(images):
 			img_bgr = cv2.imread(fname)
 			img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
 
 			if img_gray.shape != (h, w):
 				if img_gray.shape == (w, h):
-					print('   [INFO] Rotating image {}/{}'.format(i+1, num_images))
+					print(tag_string('info', 'Rotating image {}/{}'.format(i+1, num_images)))
 					img_bgr = cv2.rotate(img_bgr, cv2.ROTATE_90_CLOCKWISE)
 					img_gray = cv2.rotate(img_gray, cv2.ROTATE_90_CLOCKWISE)
 					rotations[i] = 1
 				else:
-					print('[ERROR] All images in the folder [{}] must be of the same size!'.format(input_folder))
-					input(input('\nPress ENTER/RETURN to exit...'))
+					print(tag_string('error', 'All images in the folder [{}] must be of the same size!'.format(input_folder)))
+					input('\nPress ENTER/RETURN to exit...')
 					exit()
 
 			ret, corners = cv2.findChessboardCorners(img_gray, board_size)
 			plt.cla()
 
 			if ret:
-				print('[SUCCESS] Detected corners in image {}/{}'.format(i+1, num_images))
+				print(tag_string('success', 'Detected corners in image {}/{}'.format(i+1, num_images)))
 				ret_list[i] = ret
 				objpoints.append(objp)
 				corners_subpixel = cv2.cornerSubPix(img_gray, corners, (11, 11), (-1, -1), (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001))
 				imgpoints.append(corners_subpixel)
 			else:
-				print(' [FAILED] Corner detection failed in image {}/{}'.format(i+1, num_images))
+				print(tag_string('failed', 'Corner detection failed in image {}/{}'.format(i+1, num_images)))
 
 			if ret:
 				xs = corners_subpixel[:, 0, 0].tolist()
@@ -127,9 +129,10 @@
 		plt.pause(1.0)
 		plt.close()
 
-		print('\n   [INFO] Calculating camera intrinsics and distortion coefficients...', end='')
+		print()
+		print(tag_string('info', 'Calculating camera intrinsics and distortion coefficients... '), end='')
 		ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, (w, h), None, None, None, None, flags=calibration_flags)
-		print(' DONE!\n')
+		print(tag_string('info', 'DONE!\n'))
 
 		mean_img_errors = []
 
@@ -147,7 +150,7 @@
 
 		np.savetxt('{}/ret_list.txt'.format(input_folder), mean_img_errors, fmt='%.4f')
 
-		print('   [INFO] Mean reprojection error = {:.3f} px'.format(mean_error))
+		print(tag_string('info',  'Mean reprojection error = {:.3f} px'.format(mean_error)))
 
 		mtx_scaled = mtx / w
 
@@ -213,10 +216,12 @@
 		with open('{}/{}.cpf'.format(input_folder, camera_model), 'w') as configfile:
 			cfg.write(configfile)
 
-		print('\n [END] Camera calibration complete!')
+		print()
+		print(tag_string('end', 'Camera calibration complete!'))
 		input('\nPress ENTER/RETURN to exit...')
 
 	except Exception as ex:
-		print('\n[EXCEPTION] The following exception has occurred: \n\n'
-		      '  {}'.format(ex))
-		input('\nPress ENTER/RETURN to exit...')
+		print()
+		print(tag_string('exception', 'The following exception has occurred: \n'))
+		print('  {}'.format(ex))
+		input('\nPress ENTER/RETURN key to exit...')