Skip to content

Commit 773fd31

Browse files
jnippulahamishwillee
jnippula
and
hamishwillee
authored
dev_log: log encryption update (#3468)
* dev_log: log encryption update combined .ulge file instead of separate .ulgc/ulgk files * Minor subedit * update vitepress version * Simplify the instructions a little * Add a release note * Update en/releases/main.md * log_encryption.md: fix file extension from uloge to ulge --------- Co-authored-by: Hamish Willee <[email protected]>
1 parent f464c52 commit 773fd31

File tree

5 files changed

+2583
-1076
lines changed

5 files changed

+2583
-1076
lines changed

‎en/dev_log/log_encryption.md

+81-59
Original file line numberDiff line numberDiff line change
@@ -11,34 +11,64 @@ Log encryption is not enabled by default in PX4 firmware builds.
1111
To use it you will need to build firmware with this feature enabled and then upload it to the flight controller (see instructions below).
1212
:::
1313

14-
::: info
15-
The encryption algorithm used is set in [SDLOG_ALGORITHM](../advanced_config/parameter_reference.md#SDLOG_ALGORITHM).
16-
At time of writing, only `XChaCha20` is supported (AES can be selected, but there is no implementation).
14+
::: tip
15+
Log encryption was has been improved in PX4 main (v1.16+) to generate a single encrypted log file that contains both encrypted log data, and an encrypted symmetric key that you can use to decrypt it (provided you can decrypt the symmetric key).
16+
17+
In earlier versions the encrypted symmetric key was stored in a separate file.
18+
For more information see the [Log Encryption (PX4 v1.15)](https://docs.px4.io/v1.15/en/dev_log/log_encryption.html).
1719
:::
1820

1921
## How ULog Encryption Works
2022

2123
::: info
22-
This process assumes the default XChaCha20 algorithm is used.
23-
If another [SDLOG_ALGORITHM](../advanced_config/parameter_reference.md#SDLOG_ALGORITHM) is used, the process is _likely_ to remain the same.
24+
The encryption algorithm used is set in [SDLOG_ALGORITHM](../advanced_config/parameter_reference.md#SDLOG_ALGORITHM).
25+
At time of writing, only `XChaCha20` is supported (AES can be selected, but there is no implementation).
26+
27+
If another algorithm is supported in future, the process is _likely_ to remain the same as documented here.
2428
:::
2529

2630
The encryption process for each new ULog is:
2731

28-
1. A ULog file is created and opened for writing on the SD card.
29-
This is named with the file extension `.ulogc`(ulog cipher).
30-
2. A XChaCha20 symmetric key is generated and encrypted using an RSA2048 public key.
31-
This encrypted/wrapped key is stored on the SD card in a file that has the suffix `.ulgk` (ulog wrapped key).
32-
3. The unencrypted symmetric key is used to encrypt ULog data blocks before they are written to disk (the `.ulogc` file).
32+
1. A XChaCha20 symmetric key is generated and encrypted using an RSA2048 public key.
33+
This wrapped (encrypted) key is stored on the SD card in the beginning of a file that has the suffix `.ulge` ("ulog encrypted").
34+
2. When a log is captured, the ULog data is encrypted with the unwrapped symmetric key and the resulting data is appended into the end of the `.ulge` file immediately after the wrapped key data.
3335

34-
After the flight, there are two files on the SD card:
35-
36-
- `.ulogc` (ulog cipher): the encrypted log file data.
37-
- `.ulogk` (ulog wrapped key): the symmetric key used to encrypt the data, encrypted with an RSA public key.
36+
After the flight, the `.ulge` file containing both the wrapped symmetric key and the encrypted log data can be found on the SD card.
3837

3938
In order to extract the log file, a user must first decrypt the wrapped symmetric key, which can then be used to decrypt the log.
40-
Note that decrypting the symmetric key file is only possible if the user has the appropriate RSA private key (corresponding to the public key that was used to wrap it).
41-
This process is covered in [Download & Decrypt Log Files](#download-decrypt-log-files) below.
39+
Decrypting the wrapped symmetric key file is only possible if the user has the corresponding RSA private key for the public key that was used to wrap it.
40+
41+
This process is covered in more detail in [Download & Decrypt Log Files](#download-decrypt-log-files) below.
42+
43+
## File Structure
44+
45+
Encrypted `.ulge` file contains following sections:
46+
47+
```plain
48+
-------------------------
49+
| Header |
50+
-------------------------
51+
| Wrapped symmetric key |
52+
-------------------------
53+
| Encrypted ulog data |
54+
-------------------------
55+
56+
```
57+
58+
Header section (22 bytes) contains following fields:
59+
60+
| Bytes | Field |
61+
| ------ | --------------------- |
62+
| 0..6 | File magic identifier |
63+
| 7 | Header version |
64+
| 8..15 | Timestamp |
65+
| 16 | exchange algorithm |
66+
| 17 | exchange key index |
67+
| 18..19 | key size |
68+
| 20..21 | nonce size |
69+
70+
The header part begins with magic string: `"ULogEnc"`, which identifies this is encrypted ulog file.
71+
The file offset of the symmetric key section is `22` and the file offset of the log data section is `22 + key_size + nonce_size` (`key_size` and `nonce_size` are taken from the header section).
4272

4373
## Custom PX4 Firmware with Log Encryption
4474

@@ -60,7 +90,7 @@ Crypto uses large amounts of flash memory, and is therefore not included in the
6090
The easiest way to add support for encrypted logs is to define a custom `make` target that includes the required modules and your public RSA keys.
6191

6292
::: warning
63-
Crypto uses a lot of flash memory, and many builds are close to their maximum capacity.
93+
Many builds are close to their maximum capacity.
6494
If you run into a build error telling you that you have gone above the maximum flash memory, you will need to disable other features in the `.px4board` file you are working on, or in the `default.px4board` file.
6595
Be careful not to disable something you need.
6696

@@ -87,21 +117,21 @@ This is not used in the current PX4 implementation and can be ignored.
87117

88118
::: details Overview of crypto-relevant keys
89119

90-
| Argument | Description |
91-
| ---------------------------- | --------------------------------------------------------------------------------------------------------- |
92-
| CONFIG_BOARD_CRYPTO | Include crypto module in firmware.<br />= `y`: Enable log encryption.<br />= `n`: Disable log encryption. |
93-
| CONFIG_DRIVERS_SW_CRYPTO | Include the PX4 crypto backend library (used by above library).<br />= `y`: Enable<br />= `n`: Disable |
94-
| CONFIG_DRIVERS_STUB_KEYSTORE | Includes the PX4 stub keystore driver.<br />= `y`: Enable<br />= `n`: Disable |
95-
| CONFIG_PUBLIC_KEY0 | Location of public key for keystore index 0. |
96-
| CONFIG_PUBLIC_KEY1 | Location of public key for keystore index 1.<br />= `{path to key1}` |
97-
| CONFIG_PUBLIC_KEY2 | Location of public key for keystore index 2.<br />= `{path to key2}` |
98-
| CONFIG_PUBLIC_KEY3 | Location of public key for keystore index 3.<br />= `{path to key3}` |
120+
| Argument | Description |
121+
| ---------------------------- | ----------------------------------------------------------------------------------------------------- |
122+
| CONFIG_BOARD_CRYPTO | Include crypto module in firmware.<br>= `y`: Enable log encryption.<br>= `n`: Disable log encryption. |
123+
| CONFIG_DRIVERS_SW_CRYPTO | Include the PX4 crypto backend library (used by above library).<br>= `y`: Enable<br>= `n`: Disable |
124+
| CONFIG_DRIVERS_STUB_KEYSTORE | Includes the PX4 stub keystore driver.<br>= `y`: Enable<br>= `n`: Disable |
125+
| CONFIG_PUBLIC_KEY0 | Location of public key for keystore index 0. |
126+
| CONFIG_PUBLIC_KEY1 | Location of public key for keystore index 1.<br>= `{path to key1}` |
127+
| CONFIG_PUBLIC_KEY2 | Location of public key for keystore index 2.<br>= `{path to key2}` |
128+
| CONFIG_PUBLIC_KEY3 | Location of public key for keystore index 3.<br>= `{path to key3}` |
99129

100130
The stub keystore is a keystore implementation that can store up to four keys.
101131
The initial values of these keys are set in the locations defined by `CONFIG_PUBLIC_KEY0` to `CONFIG_PUBLIC_KEY3`.
102132
The keys can be used for different cryptographic purposes, which are determined by parameters.
103133

104-
The _exchange key_, which is the public key used for encrypting the symmetric key stored in the `.ulgk` file, is specified using [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY) as an index value into the key store.
134+
The _exchange key_, which is the public key used for encrypting the symmetric key stored in the beginning of the `.ulge` file, is specified using [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY) as an index value into the key store.
105135
The value is `1` by default, which maps to the key defined in `CONFIG_PUBLIC_KEY1`.
106136

107137
The _logging key_ is the unencrypted symmetric key.
@@ -111,7 +141,7 @@ Note that the value is generated fresh for each log, and any value specified in
111141
You can use choose different locations for your keys as long as they aren't used by anything else.
112142
:::
113143

114-
The key in `CONFIG_PUBLIC_KEY1` is the public key used to wrap the symmetric key in the `.ulgk` file (by default: see [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY)).
144+
The key in `CONFIG_PUBLIC_KEY1` is the public key used to wrap the symmetric key in the the beginning of `.ulge` file (by default: see [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY)).
115145
You can use the `rsa2048.pub` key for testing, or replace it with the path to your own public key in the file (see [Generate RSA Public & Private Keys](#generate-rsa-public-private-keys)).
116146

117147
Build the firmware like this:
@@ -160,61 +190,53 @@ You can now build and test.
160190
## Download & Decrypt Log Files
161191

162192
Encrypted log files are downloaded using the QGroundControl [Log Download](https://docs.qgroundcontrol.com/master/en/qgc-user-guide/analyze_view/log_download.html) view (**Analyze Tools > Log Download**) just like ordinary log files.
163-
The only difference is that for each flight you will need to download both the encrypted log file, and the file containing the encrypted symmetric key.
164193

165-
The encrypted log file and encrypted symmetric key file are displayed with a timestamp (but no filename) in QGroundControl, as shown below.
166-
You can determine which files are associated based on their timestamps.
194+
Note that the encrypted files will be downloaded with the `.ulg` suffix, instead of `.ulge`.
167195

168-
![QGroundControl ULog Download](../../assets/qgc/analyze/encrypted_log.png)
196+
### Decrypt ULogs
169197

170-
Select and download both files.
198+
Before you can analyze your encrypted logs, you will need to decrypt them.
199+
There is a Python script that can be used to decrypt logs in `Tools/decrypt_ulog.py`.
171200

172-
Note that both files will be downloaded with the `.ulg` suffix.
173-
You can identify the symmetric key file, as it is usually much smaller than the log file (about 300 bytes)
201+
When decrypting a `.ulge` file the script takes 3 arguments:
174202

175-
For convenience in the decryption step, you might rename the file extensions to add back the `.ulgc` (log) and `.ulgk` (key) file extensions.
203+
1. The encrypted log file.
204+
2. An empty string `''`.
205+
3. The decryption key (the RSA2048 `.pem` private key which is used to unwrap the symmetric key).
176206

177-
### Decrypt ULogs
207+
For example:
178208

179-
Before you can analyze your encrypted logs, you will need to decrypt them.
180-
There is a Python script that can be used to decrypt logs in `Tools/decrypt_ulog.py`.
209+
```sh
210+
python3 decrypt_ulog.py \
211+
/home/john/Downloads/log_24_2024-10-6-23-39-50.ulg '' \
212+
new_keys/private_key.pem
213+
```
181214

182-
`decrypt_ulog.py` takes 3 arguments:
215+
On success the decrypted log file is created with the `.ulog` suffix.
183216

184-
1. The encrypted `.ulogc` file.
185-
2. The symmetric key `.ulogk` file.
186-
3. The decryption key (the RSA2048 `.pem` private key which is used to unwrap the `.ulogk` file).
217+
::: info
218+
The script can be used with both `.ulge` logs and the `.ulgc`/`.ulgk` files used in [PX4 v1.15 Log Encryption](https://docs.px4.io/v1.15/en/dev_log/log_encryption.html).
219+
The full command line syntax is given below:
187220

188221
```sh
189222
usage: decrypt_ulog.py [-h] [ulog_file] [ulog_key] [rsa_key]
190223

191224
CLI tool to decrypt an ulog file
192225

193226
positional arguments:
194-
ulog_file .ulog file
195-
ulog_key .ulogk, encrypted key
227+
ulog_file .ulge/.ulgc, encrypted log file
228+
ulog_key .ulgk, legacy encrypted key (give empty string '' to ignore for .ulge)
196229
rsa_key .pem format key for decrypting the ulog key
197230

198231
optional arguments:
199232
-h, --help show this help message and exit
200-
201233
```
202234

203-
As an example:
204-
205-
```sh
206-
python3 decrypt_ulog.py \
207-
/home/john/Downloads/log_24_2024-10-6-23-39-50.ulgc \
208-
/home/john/Downloads/log_23_2024-10-6-23-39-48.ulgk \
209-
new_keys/private_key.pem
210-
```
211-
212-
On success the decrypted log file is created with the `.ul` suffix instead of `.ulg`.
213-
Rename the file back to `.ulg` and it is now ready for flight review.
235+
:::
214236

215237
## Generate RSA Public & Private Keys
216238

217-
To generate a rsa2048 private and public key, you can use OpenSSL:
239+
To generate a RSA2048 private and public key, you can use OpenSSL:
218240

219241
```sh
220242
openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048
@@ -225,7 +247,7 @@ Then you can create a public key from this private key:
225247
```sh
226248
# Convert private_key.pem to a DER file
227249
openssl rsa -pubout -in private_key.pem -outform DER -out public_key.der
228-
# From the DER file, generate a public key in hex format, seperated by commas
250+
# From the DER file, generate a public key in hex format, separated by commas
229251
xxd -p public_key.der | tr -d '\n' | sed 's/\(..\)/0x\1, /g' > public_key.pub
230252
```
231253

‎en/releases/main.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,10 @@ Please continue reading for [upgrade instructions](#upgrade-guide).
3232
- [Battery level estimation improvements](../config/battery.md) ([PX4-Autopilot#23205](https://github.com/PX4/PX4-Autopilot/pull/23205)).
3333
- [Voltage-based estimation with load compensation](../config/battery.md#voltage-based-estimation-with-load-compensation) now uses a real-time estimate of the internal resistance of the battery to compensate voltage drops under load (with increased current), providing a better capacity estimate than with the raw measured voltage.
3434
- Thrust-based load compensation has been removed (along with the `BATn_V_LOAD_DROP` parameters, where `n` is the battery number).
35-
35+
- [Log Encryption](../dev_log/log_encryption.md) now generates an encrypted log that contains the public-key-encrypted symmetric key that can be used to decrypt it, instead of putting the key into a separate file.
36+
This makes log decryption much easier, as there is no need to download or identify a separate key file.
37+
([PX4-Autopilot#24024](https://github.com/PX4/PX4-Autopilot/pull/24024)).
38+
3639
### Control
3740

3841
- TBD

0 commit comments

Comments
 (0)