Compliance with TS 26.517 and TS 26.346 (FLUTE v1, RFC 3986): wide TSI, FDT Instance ID wraparound, Close Session/Object flags, EXT_FTI bootstrap - #62
Conversation
52a3ed8 to
a9b4650
Compare
a9b4650 to
d3b8792
Compare
Spec-compliance verificationEach item below was checked directly against the primary RFC text (not inferred from existing code comments or secondary sources):
|
|
3GPP TS 26.517 (v18.5.1)
3GPP TS 26.346 (v18.1.0)
Therefore the FLUTE session for 3GPP applications is based on RFC3926, which is the FLUTE v1 specification, and states:
Since we use this library with 3GPP applications it needs to support version 1 of FLUTE. It could support version 2 as well, but changes to make this version 2 only will break 3GPP based applications. |
davidjwbbc
left a comment
davidjwbbc
left a comment
There was a problem hiding this comment.
I have no problem with FLUET v2 being added, but this library needs to support FLUTE v1 for compliance with 3GPP specifications. Please do not make changes that remove FLUTE v1 compliance.
|
Thanks @davidjwbbc. Having discussed this now wth @dsilhavy, I'm going to defer v2 to another branch and leave that code out of this PR. Then I will check that all v1 is consistent with 3GPP and update this PR accordingly. |
AlcPacket's transmit constructor took a uint16_t TSI while Transmitter itself stores a uint64_t, silently truncating any TSI above 65535 on every packet. Widened the constructor to uint64_t and now sets tsi_flag (the LCT header's S flag, adding a second 32-bit TSI word per RFC 5651 SS4.2's S/H flag mechanism -- unchanged since RFC 3451, so this applies identically to FLUTE v1) whenever the value exceeds 16 bits, giving the full 48-bit field width this wire scheme supports. Values that fit in 16 bits keep the original single-half-word encoding unchanged. Dropped this commit's original FLUTE-version-2 change: RFC 6726 SS3.1/3.4.1 does require version 2, but 3GPP TS 26.517 cl.6.2.1 -> TS 26.346 Annex L mandates the FLUTE v1 profile (RFC 3926) for MBMS/5G-MAG applications, and this library's primary consumers are exactly those applications -- per davidjwbbc's review on 5G-MAG#62. Kept on a separate future/flute-v2-support branch in case genuine (non-3GPP) FLUTE v2 support is wanted later.
_instance_id was a plain uint32_t incremented with no bound, ignoring the wire format's actual 20-bit field width (0..0xFFFFF) and the spec's requirement that a sender MUST NOT reuse an FDT Instance ID until every receiver can be assumed to have received or discarded all packets carrying the previous instance that used it. Added a small history of (instance ID -> expiry) for every ID that has been sent, recorded right before each advance. Below the 20-bit ceiling this is a plain increment as before; once exhausted, wraps to the smallest ID whose recorded expiry has already passed, and throws if none has (which would require well over a million content changes inside one FDT expiry window).
FDT-Instance's optional Complete attribute (RFC 6726) was never read or written -- added FileDeliveryTable::complete()/set_complete(), parsed on receive and serialized on send. The LCT Close Session and Close Object flags (RFC 5651 SS3.4) were parsed into private bitfields on receive but never exposed, and never settable on transmit at all: - AlcPacket: added close_session_flag()/close_object_flag() accessors, and extended the transmit constructor with matching optional parameters to actually set the bits. - Transmitter: added close_session()/close_object(toi) -- close_session() also marks the FDT Complete (the two are the same real-world event: no more files are coming), and both are applied to every subsequent outgoing packet in send_next_packet(). - Receiver: added a close-notification callback so a caller can learn a sender signalled session/object termination, mirroring the existing file-completion callback.
…try exists yet RFC 6726 allows a sender to carry EXT_FTI on individual TOI>0 packets, not just the FDT, specifically so a receiver isn't forced to wait for (or never receive) that object's <File> entry before it can start reassembly. The receiver ignored this: any packet for a TOI with no existing File object was silently discarded, discovery only ever came from iterating the FDT's file_entries() once the FDT itself completed. - AlcPacket: added has_fec_oti(), set when the packet's own header actually carried an EXT_FTI extension (distinct from the field just being zero-initialised). - Receiver: bootstrap a File for the TOI directly from the packet's own FEC OTI when none exists yet, mirroring the existing TOI-0 bootstrap pattern just above it. - Receiver: when the FDT eventually does arrive and describes that same TOI, adopt its metadata into the already-in-progress File (added File::adopt_fdt_metadata()) instead of discarding it as stale -- a bootstrapped File's content_location is empty until the FDT supplies it, so without this it would look identical to a genuinely stale reused TOI and get thrown away and restarted the moment the FDT arrived, defeating the whole point of starting early. Note on scope: this also reinstates the pre-existing stale-TOI-reuse discard check (a File still incomplete under a TOI that a new FDT instance now describes with a *different* content_location is abandoned reception state, not the current transfer -- discard and start clean) alongside the new bootstrap-adopt branch above. That check isn't new -- it predates this branch -- but the version of development this PR is now based on doesn't yet have it, and the new bootstrap-adopt logic has to be ordered against it (adopt only when content_location is empty; otherwise fall through to the existing discard-if-different check) so the two don't misfire against each other.
d3b8792 to
6c7137d
Compare
|
Pushed a corrected version, per this thread:
Tests/build unaffected — only the one line (FLUTE version nibble) and its commit message changed; the other three commits are untouched. |
|
@jordijoangimenez: Now the scope of this PR has changed to be compliance with TS 26.517 and TS 26.346, the title should also be changed accordingly. Given that the target is now compliance with FLUTE v1, we are talking about RFC 3986 rather than RFC 6726. Rather than burying the update in a supplementary comment, it would be better to integrate the correct scope into the main issue description to avoid confusing future readers. |
RFC 6726 SS3.1/SS3.4.1 requires version 2 in EXT_FDT; the decoder already tolerates up to version 2 (unchanged by this commit), only transmit lagged. Split out of 5G-MAG#62: 3GPP TS 26.517 cl.6.2.1 -> TS 26.346 Annex L mandates the RFC 3926 (v1) profile for MBMS/5G-MAG applications, this library's primary consumers, so this must not be the default. Kept as a separate branch for a genuine non-3GPP FLUTE v2 deployment, or in case 3GPP adopts v2 later.
jordijoangimenez
changed the title
jordijoangimenez
changed the title
|
Suggestions applied, @rjb1000 |
I agree with the principle that a generic FLUTE library should allow for full RFC compliance, so allowing TSI values greater than 65535 to be encoded with the relevant ALC flags (S and H) set to signal the longer field length. What this means is that it is the responsibility of the controlling application to achieve compliance with more restrictive profiles such as those in TS 26.346. |
rjb1000
left a comment
rjb1000
left a comment
There was a problem hiding this comment.
Now that the library is targeting FLUTE v1, we think the RFC references should be RFC 3926 and, for the corresponding LCT v1, RFC 3451.
| * @param max_size Maximum payload size | ||
| * @param fdt_instance_id FDT instance ID (only relevant for FDT with TOI=0) | ||
| * @param close_session_flag Set the LCT Close Session flag (RFC 5651 SS3.4) on this packet | ||
| * @param close_object_flag Set the LCT Close Object flag (RFC 5651 SS3.4) on this packet |
There was a problem hiding this comment.
Better to reference the LCT specification reference by the older FLUTE v1 specification:
| * @param close_object_flag Set the LCT Close Object flag (RFC 5651 SS3.4) on this packet | |
| * @param close_session_flag Set the LCT Close Session flag (RFC 3451 Section 5.1) on this packet | |
| * @param close_object_flag Set the LCT Close Object flag (RFC 3451 Section 5.1) on this packet |
| * Whether the sender set the LCT Close Session flag on this packet, signalling that no | ||
| * further objects will be sent in this session (RFC 5651 SS3.4). |
There was a problem hiding this comment.
| * Whether the sender set the LCT Close Session flag on this packet, signalling that no | |
| * further objects will be sent in this session (RFC 5651 SS3.4). | |
| * Whether the sender set the LCT Close Session flag on this packet, signalling that no | |
| * further objects will be sent in this session (RFC 3451 Section 5.1). |
|
|
||
| /** | ||
| * Whether the sender set the LCT Close Object flag on this packet, signalling that this | ||
| * is the last packet for this TOI (RFC 5651 SS3.4). |
There was a problem hiding this comment.
| * is the last packet for this TOI (RFC 5651 SS3.4). | |
| * is the last packet for this TOI (RFC 3451 Section 5.1). |
| FecScheme fec_scheme() const { return _fec_oti.encoding_id; }; | ||
|
|
||
| /** | ||
| * Whether this packet carried its own EXT_FTI header extension. RFC 6726 allows a sender |
There was a problem hiding this comment.
We think AL-FEC signalling on non-FDT objects is also allowed by FLUTE v1 (RFC 3926 Section 5.1). Is that right? Do the relevant profiles of TS 26.346 also allow this?
| * Whether this packet carried its own EXT_FTI header extension. RFC 6726 allows a sender | |
| * Whether this packet carried its own EXT_FTI header extension. RFC 3926 allows a sender |
| void set_expires(uint64_t exp) { _expires = exp; }; | ||
|
|
||
| /** | ||
| * Set the RFC 6726 Complete attribute: true once this FDT Instance describes the full, |
There was a problem hiding this comment.
| * Set the RFC 6726 Complete attribute: true once this FDT Instance describes the full, | |
| * Set the Complete attribute (RFC 3926 Section 3.4.2): true once this FDT Instance describes the full, |
|
|
||
| private: | ||
| /** | ||
| * Advance _instance_id to a value that is safe to reuse, per RFC 6726 SS3.4.1: the FDT |
There was a problem hiding this comment.
| * Advance _instance_id to a value that is safe to reuse, per RFC 6726 SS3.4.1: the FDT | |
| * Advance _instance_id to a value that is safe to reuse, per RFC 3926 Section 3.4.1: the FDT |
|
|
||
| /** | ||
| * Definition of a callback invoked whenever an incoming packet carries the LCT Close | ||
| * Session and/or Close Object flag (RFC 5651 SS3.4), registered through |
There was a problem hiding this comment.
| * Session and/or Close Object flag (RFC 5651 SS3.4), registered through | |
| * Session and/or Close Object flag (RFC 3451 Section 5.1), registered through |
| * as Complete (RFC 6726, so receivers know the file set is final) and sets the LCT Close | ||
| * Session flag (RFC 5651 SS3.4) on every packet sent from this point on, including for |
There was a problem hiding this comment.
| * as Complete (RFC 6726, so receivers know the file set is final) and sets the LCT Close | |
| * Session flag (RFC 5651 SS3.4) on every packet sent from this point on, including for | |
| * as Complete (RFC 3926, so receivers know the file set is final) and sets the LCT Close | |
| * Session flag (RFC 3451 Section 5.1) on every packet sent from this point on, including for |
|
|
||
| /** | ||
| * Signal that no further data will be sent for a specific TOI. Sets the LCT Close Object | ||
| * flag (RFC 5651 SS3.4) on subsequent packets carrying that TOI. |
There was a problem hiding this comment.
| * flag (RFC 5651 SS3.4) on subsequent packets carrying that TOI. | |
| * flag (RFC 3451 Section 5.1) on subsequent packets carrying that TOI. |
|
|
||
| if (alc.toi() != 0 && _files.find(alc.toi()) == _files.end() && alc.has_fec_oti()) { | ||
| // No <File> entry for this TOI yet (the FDT describing it hasn't arrived, or won't -- | ||
| // RFC 6726 allows a sender to carry EXT_FTI on individual object packets precisely so |
There was a problem hiding this comment.
| // RFC 6726 allows a sender to carry EXT_FTI on individual object packets precisely so | |
| // RFC 3926 allows a sender to carry EXT_FTI on individual object packets precisely so |
4 participants
Fixes five gaps identified in a spec-compliance audit, each independently built and tested (unit tests + a live two-process transmitter/receiver regression check).
All five verified directly against TS 26.346 V18.2.0, not just the IETF RFCs -- see the note on FLUTE version below and the per-item notes.
Changes
AlcPacket's transmit constructor now takes the fulluint64_tTSITransmitteralready stores, and setstsi_flag(the LCT header's S flag, adding a second 32-bit TSI word per RFC 5651 §4.2 -- unchanged since RFC 3451, so it applies identically to FLUTE v1) when the value exceeds 16 bits, giving the full 48-bit field width this wire scheme supports. Values that fit in 16 bits keep the original encoding unchanged.FileDeliveryTable.cpp) -- implements the RFC 3926 §3.4.1 algorithm: track each sent ID's expiry, wrap to the smallest expired ID once the 20-bit space (0..0xFFFFF) is exhausted, never reuse a still-live ID.Completeattribute -- read/write support, so a genuinely-complete FDT Instance can be marked as such.AlcPacket, aTransmitter::close_session()/close_object(toi)API, and aReceiverclose-notification callback.<File>entry; the receiver discarded such packets outright. Now bootstraps aFilefrom the packet's own FEC OTI, and reconciles metadata in-place if the FDT later arrives for the same TOI, instead of discarding the in-progress reception as stale. Receiver-side only -- the transmitter still emits EXT_FTI only on TOI=0 packets per cl.7.2.8's mandate, unchanged.Items 2-4 were re-checked directly against TS 26.346 v18.2.0: the FDT Instance ID wraparound rule, the
Completeattribute, and the Close Session/Object flags are all explicitly present in 3GPP's own FDT schema and clause 7.2.7 -- none of these are FLUTE-v2-only additions, all confirmed compliant with the v1 profile below.Note on FLUTE version
An earlier version of this PR also hardcoded the FLUTE version nibble to 2 (RFC 6726 §3.1/§3.4.1 requires version 2 on the wire). Per @davidjwbbc's review: 3GPP TS 26.517 cl.6.2.1 → TS 26.346 Annex L mandates the RFC 3926 (v1) profile for MBMS/5G-MAG applications, this library's primary consumers, so that change has been dropped from this PR.
The decoder is unaffected (it already tolerated up to version 2 on receive; still does). The dropped change is preserved on its own branch and PR (#65) for a genuine non-3GPP FLUTE v2 deployment, or in case 3GPP adopts v2 later.
Testing
flute-transmitter/flute-receiverexamples with a wide TSI (>16 bits, exercising fix FEC support for FLUTE #1) across multiple retransmit rounds (exercising FDT instance advances, fix Development #2); received files were byte-identical to the source.Note on integration
This branch is rebased onto
development(which now includes #56). It does not include the Raptor FEC work from #61 or the FDT-growth fix from #60 -- those are separate, in-flight PRs with no overlap with this one's changes. No further reconciliation needed on this PR's account; #61 already carries this PR's accessor pattern and IPv4/IPv6 parity forward for when both land.