dev_log: log encryption update

combined .ulge file instead of separate .ulgc/ulgk files

Author: jnippula

en/dev_log/log_encryption.md

@@ -25,21 +25,45 @@ If another [SDLOG_ALGORITHM](../advanced_config/parameter_reference.md#SDLOG_ALG
 
 The encryption process for each new ULog is:
 
-1. A ULog file is created and opened for writing on the SD card.
-   This is named with the file extension `.ulogc`(ulog cipher).
-2. A XChaCha20 symmetric key is generated and encrypted using an RSA2048 public key.
-   This encrypted/wrapped key is stored on the SD card in a file that has the suffix `.ulgk` (ulog wrapped key).
-3. The unencrypted symmetric key is used to encrypt ULog data blocks before they are written to disk (the `.ulogc` file).
+1. A XChaCha20 symmetric key is generated and encrypted using an RSA2048 public key.
+   This encrypted/wrapped key is stored on the SD card in the beginning of a file that has the suffix `.ulge` (ulog encrypted).
+2. A ULog data is encrypted with unwrapped symmetric key and the resulting data is appended into the end of the `.ulge` file immediately after the stored key data.
 
-After the flight, there are two files on the SD card:
-
-- `.ulogc` (ulog cipher): the encrypted log file data.
-- `.ulogk` (ulog wrapped key): the symmetric key used to encrypt the data, encrypted with an RSA public key.
+After the flight, the `.ulge` file containing both symmetric key and the encrypted log data can be found on the SD card.
 
 In order to extract the log file, a user must first decrypt the wrapped symmetric key, which can then be used to decrypt the log.
 Note that decrypting the symmetric key file is only possible if the user has the appropriate RSA private key (corresponding to the public key that was used to wrap it).
 This process is covered in [Download & Decrypt Log Files](#download-decrypt-log-files) below.
 
+## File structure
+
+Encrypted `.ulge` file contains following sections:
+```
+-------------------------
+| Header                |
+-------------------------
+| Wrapped symmetric key |
+-------------------------
+| Encrypted ulog data   |
+-------------------------
+
+```
+
+Header section (22 bytes) contains following fields:
+
+| Bytes  | Field                  |
+|--------|------------------------|
+| 0..6   | File magic identifier  |
+| 7      | Header version         |
+| 8..15  | Timestamp              |
+| 16     | exchange algorithm     |
+| 17     | exchange key index     |
+| 18..19 | key size               |
+| 20..21 | nonce size             |
+
+The header part begins with magic string: `"ULogEnc"`, which identifies this is encrypted ulog file.
+File offset of the symmetric key section is 22 and file offset of the log data section is 22+key_size+nonce_size (key_size and nonce_size are taken from header section).
+
 ## Custom PX4 Firmware with Log Encryption
 
 You will need to build custom firmware that contains your own public RSA key and the required Crypto API modules to support log encryption.
@@ -101,7 +125,7 @@ The stub keystore is a keystore implementation that can store up to four keys.
 The initial values of these keys are set in the locations defined by `CONFIG_PUBLIC_KEY0` to `CONFIG_PUBLIC_KEY3`.
 The keys can be used for different cryptographic purposes, which are determined by parameters.
 
-The _exchange key_, which is the public key used for encrypting the symmetric key stored in the `.ulgk` file, is specified using [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY) as an index value into the key store.
+The _exchange key_, which is the public key used for encrypting the symmetric key stored in the beginning of `.ulge` file, is specified using [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY) as an index value into the key store.
 The value is `1` by default, which maps to the key defined in `CONFIG_PUBLIC_KEY1`.
 
 The _logging key_ is the unencrypted symmetric key.
@@ -111,7 +135,7 @@ Note that the value is generated fresh for each log, and any value specified in
 You can use choose different locations for your keys as long as they aren't used by anything else.
 :::
 
-The key in `CONFIG_PUBLIC_KEY1` is the public key used to wrap the symmetric key in the `.ulgk` file (by default: see [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY)).
+The key in `CONFIG_PUBLIC_KEY1` is the public key used to wrap the symmetric key in the the beginning of `.ulge` file (by default: see [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY)).
 You can use the `rsa2048.pub` key for testing, or replace it with the path to your own public key in the file (see [Generate RSA Public & Private Keys](#generate-rsa-public-private-keys)).
 
 Build the firmware like this:
@@ -160,19 +184,8 @@ You can now build and test.
 ## Download & Decrypt Log Files
 
 Encrypted log files are downloaded using the QGroundControl [Log Download](https://docs.qgroundcontrol.com/master/en/qgc-user-guide/analyze_view/log_download.html) view (**Analyze Tools > Log Download**) just like ordinary log files.
-The only difference is that for each flight you will need to download both the encrypted log file, and the file containing the encrypted symmetric key.
-
-The encrypted log file and encrypted symmetric key file are displayed with a timestamp (but no filename) in QGroundControl, as shown below.
-You can determine which files are associated based on their timestamps.
-
-![QGroundControl ULog Download](../../assets/qgc/analyze/encrypted_log.png)
-
-Select and download both files.
-
-Note that both files will be downloaded with the `.ulg` suffix.
-You can identify the symmetric key file, as it is usually much smaller than the log file (about 300 bytes)
 
-For convenience in the decryption step, you might rename the file extensions to add back the `.ulgc` (log) and `.ulgk` (key) file extensions.
+Note that the encrypted files will be downloaded with the `.ulg` suffix, instead of `.ulge`.
 
 ### Decrypt ULogs
 
@@ -181,18 +194,18 @@ There is a Python script that can be used to decrypt logs in `Tools/decrypt_ulog
 
 `decrypt_ulog.py` takes 3 arguments:
 
-1. The encrypted `.ulogc` file.
-2. The symmetric key `.ulogk` file.
-3. The decryption key (the RSA2048 `.pem` private key which is used to unwrap the `.ulogk` file).
+1. The encrypted ulog file. Supporting both `.ulge` and the old legacy `.ulgc` file format.
+2. Optional symmetric key `.ulgk` file. Give empty string  `''` for decrypting `.ulge`. This is for supporting legacy `.ulgc/.ulgk` log files.
+3. The decryption key (the RSA2048 `.pem` private key which is used to unwrap the symmetric key).
 
 ```sh
 usage: decrypt_ulog.py [-h] [ulog_file] [ulog_key] [rsa_key]
 
 CLI tool to decrypt an ulog file
 
 positional arguments:
-  ulog_file   .ulog file
-  ulog_key    .ulogk, encrypted key
+  ulog_file   .ulge/.ulgc, encrypted log file
+  ulog_key    .ulgk, legacy encrypted key (give empty string '' to ignore for .ulge)
   rsa_key     .pem format key for decrypting the ulog key
 
 optional arguments:
@@ -204,13 +217,11 @@ As an example:
 
 ```sh
 python3 decrypt_ulog.py \
-/home/john/Downloads/log_24_2024-10-6-23-39-50.ulgc \
-/home/john/Downloads/log_23_2024-10-6-23-39-48.ulgk \
+/home/john/Downloads/log_24_2024-10-6-23-39-50.ulg '' \
 new_keys/private_key.pem
 ```
 
-On success the decrypted log file is created with the `.ul` suffix instead of `.ulg`.
-Rename the file back to `.ulg` and it is now ready for flight review.
+On success the decrypted log file is created with the `.ulog` suffix.
 
 ## Generate RSA Public & Private Keys