Worked on Apple Unified Logging format support

Author: joachimmetz

documentation/Apple Unified Logging and Activity Tracing formats.asciidoc

@@ -298,10 +298,10 @@ The header (chunk) is 224 bytes of size and consists of:
 | 4 | 4 | 0x0011 | Chunk sub tag (subtag)
 | 8 | 8 | 208 | Chunk data size (length)
 4+| _Chunk data (tracev3_chunk_header)_
-| 16 | 4 | | Mach timebase numerator (first number in timebase # / #)
-| 20 | 4 | | Mach timebase denominator (second number in timebase # / #)
+| 16 | 4 | | Timebase numerator (first number in timebase # / #)
+| 20 | 4 | | Timebase denominator (second number in timebase # / #)
 | 24 | 8 | | Start time +
-Contains a continuous time
+Contains a Mach continuous timestamp
 | 32 | 4 | | [yellow-background]*Unknown time (related to starttime?)* +
 Contains number of seconds since January 1, 1970 00:00:00 UTC (POSIX epoch), disregarding leap seconds
 | 36 | 4 | | [yellow-background]*Unknown*
@@ -341,7 +341,8 @@ The header continuous time sub chunk is 16 bytes of size and consist of:
 | 4 | 4 | | Sub chunk data size +
 The size value does not include the 8 bytes of the sub chunk tag and data size
 4+| _The continuous time sub chunk data (tracev3_subchunk_continuous)_
-| 8 | 8 | | Continuous time (value data related to starttime?)
+| 8 | 8 | | Continuous time +
+Contains a Mach continuous timestamp
 |===
 
 ==== [[header_system_information_sub_chunk]]Header system information sub chunk
@@ -423,7 +424,7 @@ The offset is relative to the start of the catalog UUIDs
 | 24 | 2 | | Number of sub chunks
 | 26 | 6 | | [yellow-background]*Unknown (Reserved or Padding)*
 | 32 | 8 | | Earliest firehose timestamp +
-Contains continuous time
+Contains a Mach continuous timestamp
 | 40 | 16 x ... | | Catalog UUIDs +
 Contains an array of UUIDs stored in big-endian
 | ... | ... | | Catalog sub system strings +
@@ -501,8 +502,10 @@ The catalog sub chunk describes metadata for the chunk to follow and consists of
 [cols="1,1,1,5",options="header"]
 |===
 | Offset | Size | Value | Description
-| 0 | 8 | | Start (earliest) continuous time
-| 8 | 8 | | End (latest) continuous time
+| 0 | 8 | | Start (earliest) continuous time +
+Contains a Mach continuous timestamp
+| 8 | 8 | | End (latest) continuous time +
+Contains a Mach continuous timestamp
 | 16 | 4 | | Uncompressed size of chunk
 | 20 | 4 | 0x100 | Compression algorithm used (0x100 = LZ4)
 | 24 | 4 | | Number of indexes (num_indexes or procinfos)
@@ -599,7 +602,8 @@ Contains 4096 (0x1000) if there is no private data and `-(4096 - offset)` as off
 | 36 | 2 | | [yellow-background]*Unknown*
 | 38 | 1 | | Stream type
 | 39 | 1 | | [yellow-background]*Unknown3*
-| 40 | 8 | | Base continuous time for events in the firehose chunk
+| 40 | 8 | | Base continuous time for events in the firehose chunk +
+Contains a Mach continuous timestamp
 4+|
 | 48 | data size | | data +
 Contains one or more <<tracev3_firehose_tracepoint,Firehose tracepoints>>
@@ -792,11 +796,11 @@ Contains inline data or a value data range
 [cols="1,1,5",options="header"]
 |===
 | Value | Identifier | Description
-| 0x00 | | [yellow-background]*Unknown (floating-point value?)* +
+| 0x00 | | [yellow-background]*Unknown (integer or floating-point value)* +
 Contains a 32-bit or 64-bit value
 | 0x01 | | [yellow-background]*Unknown (private value)* +
 Contains a 32-bit value, formatted as "<private>"
-| 0x02 | | [yellow-background]*Unknown (integer)* +
+| 0x02 | | [yellow-background]*Unknown (integer or floating-point value)* +
 Contains a 8-bit, 16-bit, 32-bit or 64-bit value
 3+|
 | 0x10 | | [yellow-background]*Unknown (integer format precision)* +
@@ -918,6 +922,8 @@ Contains the delta relative to the base continuous time in the <<tracev3_firehos
 Note that "has private data range flag (0x0100)" has been observed to be set
 but without any obvious changes to the activity firehose tracepoint structure.
 
+[yellow-background]*TODO: determine meaning of MSB of UUID entry load address (lower 32-bit)*
+
 ==== [[tracev3_firehose_tracepoint_trace]]Trace firehose tracepoint
 
 A trace firehose tracepoint is variable of size and consists of:
@@ -989,6 +995,8 @@ Where the range offset is a virtual private strings offset in the <<tracev3_fire
 | ... | ... | | 64-bit alignment padding
 |===
 
+[yellow-background]*TODO: determine meaning of MSB of UUID entry load address (lower 32-bit)*
+
 ==== [[tracev3_firehose_tracepoint_singpost]]Signpost firehose tracepoint
 
 A signpost firehose tracepoint is variable of size and consists of:
@@ -1039,6 +1047,8 @@ Where the range offset is a virtual private strings offset in the <<tracev3_fire
 | ... | ... | | 64-bit alignment padding
 |===
 
+[yellow-background]*TODO: determine meaning of MSB of UUID entry load address (lower 32-bit)*
+
 ==== [[tracev3_firehose_tracepoint_loss]]Loss firehose tracepoint
 
 A loss firehose tracepoint is variable of size and consists of:
@@ -1135,6 +1145,7 @@ Other observerd value type decoders are:
 | "mdnsresponder:domain_name" | |
 | "mdnsresponder:ip_addr" | | Formatted as a <<mdnsresponder_ip_address,mDNSResponder IP address>>
 | "mdnsresponder:mac_addr" | | Formatted as a <<mdnsresponder_mac_address,mDNSResponder MAC address>>
+| "name=NAME" | | Name formatting argument, where NAME is the name of the value, which has no additional formatting
 | "network:in_addr" | | Formatted as an IPv4 address, for example "127.0.0.1"
 | "network:in6_addr" | | Formatted as an IPv6 address, for example "fe80::f:86ff:fee9:5c16"
 | "network:sockaddr" | |
@@ -1144,9 +1155,14 @@ Other observerd value type decoders are:
 | "odtypes:mbr_details" | |
 | "odtypes:mbridtype" | |
 | "odtypes:nt_sid_t" | |
-| "sensitive" | | Sensitive log argument, which is formatted as "<private>"
-| "private" | | Private log argument, which is formatted as "<private>"
-| "public" | | Public log argument, which has no additional formatting
+| "sensitive" | | Sensitive formatting argument, which is formatted as "<private>"
+| "signpost.description:attribute" | | Formatted as a signpost description attribute, for example `__##__signpost.description#____#attribute#_##_#efilogin-helper##__##`
+| "signpost.description:begin_time" | | Formatted as a signpost description begin time, for example `__##__signpost.description#____#begin_time#_##_#2180300470618##__##`
+| "signpost.description:end_time" | | Formatted as a signpost description end time, for example `__##__signpost.description#____#end_time#_##_#1005756624719##__##`
+| "signpost.telemetry:number1" | | For example `__##__signpost.telemetry#____#number1#_##_#5.8203125##__##`
+| "signpost.telemetry:string1" | | For example `__##__signpost.telemetry#____#string1#_##_#executeQueryBegin##__##`
+| "private" | | Private formatting argument, which is formatted as "<private>"
+| "public" | | Public formatting argument, which has no additional formatting
 |===
 
 [NOTE]
@@ -1238,7 +1254,8 @@ The oversize chunk is variable of size and consists of:
 | 24 | 4 | | Second number in proc_id #@#
 | 28 | 1 | | TTL
 | 29 | 3 | 0 | [yellow-background]*Unknown (Reserved?)*
-| 32 | 8 | | Continuous time
+| 32 | 8 | | Timestamp +
+Contains a Mach continuous timestamp
 | 40 | 4 | | Data reference index
 | 44 | 2 | | Size of public data
 | 46 | 2 | | Size of private data
@@ -1265,16 +1282,17 @@ The StateDump chunk is variable of size and consists of:
 | 24 | 4 | | Second number in proc_id #@#
 | 28 | 1 | | TTL
 | 29 | 3 | 0 | [yellow-background]*Unknown (Reserved?)*
-| 32 | 8 | | Continuous time
+| 32 | 8 | | Timestamp +
+Contains a Mach continuous timestamp
 | 40 | 8 | | Activity identifier
 | 48 | 16 | | [yellow-background]*Unknown identifier* +
 Contains a UUID stored in big-endian
 | 64 | 4 | | Data type
 | 68 | 4 | | Data size
 | 72 | 64 | | [yellow-background]*Unknown* +
-[yellow-background]*Oncly used when data type is 3?*
+[yellow-background]*Only used when data type is 3?*
 | 136 | 64 | | [yellow-background]*Unknown* +
-[yellow-background]*Oncly used when data type is 3?*
+[yellow-background]*Only used when data type is 3?*
 | 200 | 64 | | Name +
 Contains an UTF-8 formatted string with an end-of-string character
 | 264 | data size | | Data
@@ -1308,7 +1326,7 @@ The SimpleDump chunk is variable of size and consists of:
 | 29 | 1 | | Type
 | 29 | 2 | 0 | [yellow-background]*Unknown (Reserved?)*
 | 32 | 8 | | Timestamp +
-Contains a continuous time
+Contains a Mach continuous timestamp
 | 40 | 8 | | Thread identifier
 | 48 | 8 | | Offset
 | 56 | 16 | | Sender identifier +
@@ -1356,14 +1374,14 @@ The timesync boot record is 48 bytes of size and consists of:
 | Offset | Size | Value | Description
 | 0 | 2 | "\xb0\xbb" | Signature
 | 2 | 2 | 48 | Size of record
-| 4 | 4 | | [yellow-background]*Unknown*
+| 4 | 4 | | [yellow-background]*Unknown (Seen: 0)*
 | 8 | 16 | | Boot identifier (boot UUID)
-| 24 | 4 | | Timebase numerator (first number in timebase # / #)
-| 28 | 4 | | Timebase denominator (second number in timebase # / #)
+| 24 | 4 | | (Mach) Timebase numerator (first number in timebase # / #)
+| 28 | 4 | | (Mach) Timebase denominator (second number in timebase # / #)
 | 32 | 8 | | Timestamp (or boot time) +
 Signed integer that contains the number of nanoseconds since January 1, 1970 00:00:00 UTC or 0 if not set
 | 40 | 4 | | Time zone offset in minutes +
-Contains a signed integer
+Contains a signed integer that contains the number of minutes relative from UTC, for example -60 represents UTC+1
 | 44 | 4 | | Daylight savings (DST) flag (0 = no DST, 1 = DST)
 |===
 
@@ -1383,14 +1401,13 @@ The timesync sync record is 32 bytes of size and consists of:
 | Offset | Size | Value | Description
 | 0 | 2 | "Ts" | Signature
 | 2 | 2 | 32 | Size of record
-| 4 | 4 | | [yellow-background]*Unknown (flags?)* +
-Seen: 0 and 1
+| 4 | 4 | | [yellow-background]*Unknown (Seen: 0 and 1)*
 | 8 | 8 | | Kernel time +
 Contains a Mach continuous timestamp
 | 16 | 8 | | Timestamp (or wall time) +
 Signed integer that contains the number of nanoseconds since January 1, 1970 00:00:00 UTC or 0 if not set
 | 24 | 4 | | Time zone offset in minutes +
-Contains a signed integer
+Contains a signed integer that contains the number of minutes relative from UTC, for example -60 represents UTC+1
 | 28 | 4 | | Daylight savings (DST) flag (0 = no DST, 1 = DST)
 |===
 
@@ -1492,10 +1509,9 @@ and consist of:
 | Offset | Size | Value | Description
 | 0 | 4 | | (dsc) text offset
 | 4 | 4 | | (dsc) text size
-| 8 | 16 | | Sender (process or library) identifier +
-Contains a UUID stored in big-endian +
-[yellow-background]*Does this reference to the uuidtext file?*
-| 24 | 4 | | Path offset +
+| 8 | 16 | | Image (process or library) identifier +
+Contains a UUID stored in big-endian
+| 24 | 4 | | Image (process or library) path offset +
 The offset is relative to the start of the file
 |===
 
@@ -1509,10 +1525,9 @@ and consist of:
 | Offset | Size | Value | Description
 | 0 | 8 | | (dsc) text offset
 | 8 | 4 | | (dsc) text size
-| 12 | 16 | | Sender (process or library) identifier +
-Contains a UUID stored in big-endian +
-[yellow-background]*Does this reference to the uuidtext file?*
-| 28 | 4 | | Path offset +
+| 12 | 16 | | Image (process or library) identifier +
+Contains a UUID stored in big-endian
+| 28 | 4 | | Image (process or library) path offset +
 The offset is relative to the start of the file
 |===
 
@@ -1566,7 +1581,7 @@ The UUID text (uuidtext) entry descriptor is 8 bytes of size and consists of:
 [cols="1,1,1,5",options="header"]
 |===
 | Offset | Size | Value | Description
-| 0 | ... | | Path of sender process/library +
+| 0 | ... | | Image (process or library) path +
 Contains an UTF-8 formatted string with an end-of-string character
 |===
 
@@ -1854,6 +1869,9 @@ Output starts with *.tracev3 files under "Signpost" followed by
 }]
 ....
 
+traceID consists of:
+( fmt lower 32-bit << 32 ) | ( tp flags << 16 ) | ( tp log type << 8 ) | ( tp record type )
+
 :numbered!:
 [appendix]
 == References

dtformats/aul_dsc.yaml

@@ -91,7 +91,7 @@ members:
   data_type: uint32
 - name: text_size
   data_type: uint32
-- name: sender_identifier
+- name: image_identifier
   data_type: uuid_be
 - name: path_offset
   data_type: uint32
@@ -106,7 +106,7 @@ members:
   data_type: uint64
 - name: text_size
   data_type: uint32
-- name: sender_identifier
+- name: image_identifier
   data_type: uuid_be
 - name: path_offset
   data_type: uint32

dtformats/aul_tracev3.yaml

@@ -75,18 +75,6 @@ attributes:
   size: 8
   units: bytes
 ---
-name: float32
-type: floating-point
-attributes:
-  size: 4
-  units: bytes
----
-name: float64
-type: floating-point
-attributes:
-  size: 8
-  units: bytes
----
 name: cstring
 type: string
 encoding: utf8
@@ -141,6 +129,20 @@ attributes:
   size: 8
   units: bytes
 ---
+name: float32le
+type: floating-point
+attributes:
+  byte_order: little-endian
+  size: 4
+  units: bytes
+---
+name: float64le
+type: floating-point
+attributes:
+  byte_order: little-endian
+  size: 8
+  units: bytes
+---
 name: uuid_be
 type: uuid
 attributes:
@@ -771,7 +773,7 @@ members:
   data_type: uint8
 - name: unknown2
   data_type: uint16
-- name: continuous_time
+- name: timestamp
   data_type: uint64
 - name: data_reference_index
   data_type: uint32
@@ -845,7 +847,7 @@ members:
   data_type: uint8
 - name: unknown2
   data_type: uint16
-- name: continuous_time
+- name: timestamp
   data_type: uint64
 - name: activity_identifier
   data_type: uint64

dtformats/aul_uuidtext.yaml

@@ -71,7 +71,7 @@ description: UUIDtext file footer.
 attributes:
   byte_order: little-endian
 members:
-- name: library_path
+- name: image_path
   type: string
   encoding: utf8
   element_data_type: char