Skip to content

Fix MBS Broadcast content delivery crashes and Distribution Session bugs - #71

Draft
jordijoangimenez wants to merge 17 commits into
developmentfrom
feature/mbs-compliance-fixes
Draft

jordijoangimenez wants to merge 17 commits into
developmentfrom
feature/mbs-compliance-fixes

Conversation

@jordijoangimenez

@jordijoangimenez jordijoangimenez commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

MBS Transport Function: object distribution over FLUTE, and the SBI behaviour TS 29.580 and TS 29.500 require.

Type: compliance fix + crash fix
Baseline: TS 26.517 V18.6.0, TS 26.502 V18.6.0, TS 29.580 V18.8.0, TS 29.581 V18.6.0, TS 26.346 V18.2.0, RFC 3926, RFC 5651.

  • Sizes FLUTE encoding symbols for the delivery path, not the first hop. RFC 5651 §6.1 asks that packets not exceed the path MTU; a loopback-derived 65441-byte symbol meant no multi-symbol object ever arrived.
  • Sends a DASH media segment once, not once per manifest pass. RFC 3926 §3.1 ties an object to one TOI, so repeats under fresh TOIs are not redundancy.
  • Applies the session's requested AL-FEC, honouring the schemes TS 26.346 annex L.4.7 admits.
  • Keeps each object's availability window and entity tag; sends by earliest deadline.
  • Answers the status codes TS 29.500 makes mandatory; stops notifying a subscription past its expiry.
  • Builds the COLLECTION operating mode's controller.

Reproduce: 5G-MAG/rt-mbs-examples feature/mbs-compliance-fixes, scripts/mbs-broadcast-demo/start-all.sh.

Closes #73

@dsilhavy dsilhavy added this to the Version 1.4.1 milestone Aug 14, 2026
@jordijoangimenez
jordijoangimenez marked this pull request as draft August 14, 2026 10:32
@davidjwbbc

Copy link
Copy Markdown
Collaborator

@jordijoangimenez wrote:

Clamp get_path_mtu() to a sane ceiling instead of trusting a local socket — a local socket can report path MTU values that aren't sane bounds for what's actually usable, tightened for robustness.

I don't agree with this solution as it prevents the use of jumbo frames between MB-UPF and gNodeB (we use jumbo frames in our lab configuration). This is why the function used the interface MTU as fallback. This allows this setting to be handled by configuring the interface MTUs correctly for your setup.

@davidjwbbc

Copy link
Copy Markdown
Collaborator

@jordijoangimenez wrote:

Fix validate_distribution_session() rejecting every valid PUSH+SINGLE session — the early-return path for the standard, valid PUSH+SINGLE configuration returned 0, but the caller treats 0 as invalid and 1 as valid (matching the fall-through path a few lines below, which always returns 1 after successful generic validation). So this case was rejected with "Invalid Distribution Session" 100% of the time, unconditionally blocking every real PUSH-mode single-object distribution session. Confirmed live against rt-mbs-application-provider: MBSF's 502 "Failed to create MBS Distribution Session in MBSTF" traced directly to this.

Actually if you read TS 26.502 Table 6.1-1: "When the push-based object acquisition method is provisioned, the set of Object acquisition identifiers shall be empty."

So if an objAcquisitionIdPush is provided in PUSH+SINGLE mode then that is an error. So the original return code was correct (i.e generate an error if it's set).

@davidjwbbc

Copy link
Copy Markdown
Collaborator

@jordijoangimenez wrote:

Don't permanently drop a StatusNotify event on a failed/rejected send — events were marked as reported in the cache before the notify POST was even attempted, so a transport failure or non-2xx response from MBSF permanently and silently dropped that event instead of retrying it on the next notification attempt (TS 29.581 §5.2.2.8 obliges MBSTF to report several of these event types reliably).

I've looked through TS 29.581 Clause 5.8.8.2 and can find nowhere where it says the notifications have to be delivered reliably. It only states that the response from the MBSF should be 204 for success, 307 or 308 for redirection and one of the 4XX or 5XX error codes from TS 29.500 for errors. It says nothing about what action should be taken on reported failure, failure to connect or timeout.

The 307 and 308 responses should cause the request to be reissued to a new URL. For 308 the new URL should replace the old notification URL. (But we never implemented redirection)

The semantics of any 4XX error code are that the client made a mistake and should not repeat the same request, so a 4XX response should prevent the events from being sent again.

For 5XX responses, the semantics are that there was a server error that may clear and so the request may be repeated again at some point in the future. The client should delay before trying again to avoid rapid retries. So there is an argument for repeating the notifications when a 5XX is received.

This also risks indefinite attempts to resend, there probably should be a resend attempts limit.

As far as I can see the current behaviour of trying once and then forgetting it does not contradict TS 29.581.

@rjb1000

rjb1000 commented Sep 1, 2026

Copy link
Copy Markdown
  • Fix MBSTF crash on pushing an object with no registered manifest handler (related bug report on Null-pointer crash when a STREAMING object isn't a manifest #70) — pushing a real HLS playlist (application/vnd.apple.mpegurl, no registered manifest handler in this build) caused ManifestHandlerFactory::makeManifestHandler() to legitimately return nullptr, which ObjectManifestController::manifestHandler() then unconditionally dereferenced to subscribe to change notifications — crashing MBSTF right after successfully logging that the object was added. An object with no manifest handler is not an error case (not every pushed object is itself a manifest needing orchestration); fixed with a null check. Verified: MBSTF survived the identical push afterward and completed delivery.

@jordijoangimenez: Please note that this bug is fixed more comprehensively by @davidjwbbc on PR #72.

@rjb1000

rjb1000 commented Sep 1, 2026

Copy link
Copy Markdown
  • ManifestHandlerFactory::makeManifestHandler() only matched exact literal Content-Type strings, so a real header carrying parameters (e.g. application/dash+xml; charset=utf-8) matched nothing and fell through to the empty-type handler — silently breaking STREAMING-mode manifest recognition and the CAROUSEL JSON object manifest alike. Now also tries a normalised (parameter-stripped, lower-cased) form.

@jordijoangimenez: This issue was also mopped up by @davidjwbbc in PR #72.

@jordijoangimenez

Copy link
Copy Markdown
Contributor Author

Thank you Richard, as soon as possible I will clean up this branch to avoid duplications and we can cherry pick what's useful rather than adopting the full PR

@dsilhavy dsilhavy modified the milestones: Version 1.5.1, Version 1.5.0 Sep 4, 2026
…nch needs

Problem
  This branch's FLUTE-layer obligations depend on work that is on rt-libflute's own
  feature branch and not in any release tag: the TS 26.346 annex L.6 profiled FDT schema,
  the scheme-specific FEC OTI, suppression of Transfer-Length under the MBMS Download
  Profile, the split expiry setter, and the RFC 5053 Raptor scheme. The wrap pointed at a
  revision carrying none of it.   [code-derived]

Basis
  No clause governs a dependency pin.
  code-derived only.

Raised by
  Building this branch against the dependency it actually needs.

Change
  Advances subprojects/rt-libflute.wrap to the commit on 5G-MAG's own
  feature/raptor-raptorq-fec carrying that work, and the rt-common-shared submodule to its
  consolidated tip. The wrap comment records why a tag cannot be used yet and what has to
  happen before it can be.

Verification
  T0: the subproject is fetched and the tree builds against it. The behaviour that depends
  on these revisions is verified by the commits that use it, not here.

Not in this change
  Moving the rt-libflute pin to a tag, which needs a 5G-MAG release carrying the work the
  comment names. No source change.
Problem
  The Nmbstf_DistSession surface answered a wrong Content-Type with 400 rather
  than 415, never checked a client's Accept header, never bounded a request
  body, and one PATCH handler parsed its body with no Content-Type check at
  all.   [code-derived]

Basis
  TS 29.500 V18.10.0 table 5.2.7.1-1 marks 415 mandatory for POST and PATCH,
  406 mandatory for GET, and 413 mandatory where a body is accepted.  Its
  table 5.2.7.2-1 defines no named cause for 415, so the numeric status is
  constructed directly, as this file already does for 405 and 501.
  TS 29.581 (TS29581_Nmbstf_DistSession.yaml) requires
  application/json-patch+json on both PATCH operations, not
  application/merge-patch+json.
  The individual resource's representation is DistSession for both the GET and
  the PATCH on /dist-sessions/{distSessionRef}; CreateReqData is the request
  body of the collection POST only, so an RFC 6902 pointer addresses the
  DistSession.

Raised by
  reading the authority during this work, and observation of a live activation

Change
  Adds NfServer::acceptsMediaType() and answers 406 when the client's Accept
  header cannot take application/json; answers 415 for an unexpected
  Content-Type on POST and PATCH; adds request_too_large() and a configurable
  maxRequestBodySize, answering 413; checks the Content-Type on the
  subscription PATCH.  Applies a JSON Patch to the DistSession the stored
  CreateReqData holds, rebuilding the CreateReqData around the result, so the
  pointer a conformant peer sends resolves.

Verification
  T2: the end-to-end demo activates a Distribution Session, which the previous
  patch target rejected outright.

Not in this change
  Authentication on this surface.
…piry time

Problem
  DistributionSessionSubscription parsed expiryTime into m_expiryTime and never
  compared it against the clock anywhere, so a subscription carrying one kept
  receiving notifications until its Distribution Session was deleted or the
  process exited.   [code-derived]

Basis
  TS 29.581 V18.6.0, table 6.1.6.2.5-1, expiryTime row: "When present in the
  subscription creation request, it shall indicate the time up to which the
  subscription is desired to be kept active and after which the subscribed
  events shall stop generating notifications."

Raised by
  reading the authority during this work

Change
  Adds a per-subscription timer that removes the subscription when its
  expiryTime passes, scheduled wherever m_expiryTime is set and cancelled from
  the destructor.  The callback does not remove the subscription directly: it
  runs from a timer that subscription owns, so it pushes
  LocalEvents::SUBSCRIPTION_EXPIRED carrying the two ids as plain strings and
  DistributionSession::processEvent() performs the removal off that call
  stack, the deferred dispatch SEND_NOTIFICATION already uses.  A copy
  schedules its own timer, a callback being keyed to one object; a move takes
  the original's over.  Timer-pool exhaustion is logged and leaves the
  subscription without a timer rather than crashing.

Verification
  T2: the end-to-end demo runs with it in place, MBSTF logging no fatal or
  assertion lines and no timer-creation failure.  No test scaffolding exists
  for this surface.

Not in this change
  An expiryTime in a subscription response, which this MBSTF does not set.
…by earliest deadline

Problem
  Four related defects in how object metadata is held and ordered.

  ObjectStore::Metadata hand-writes its copy constructor, move constructor and both
  assignment operators, and three members were missing from them. The availability window
  (m_availabilityStartTime, m_availabilityEndTime) was absent from all four paths, so any
  copy taken out of the store lost it: build/tests/testObjectStore aborted with
  std::bad_optional_access.   [observed: test output]   The entity tag (m_entityTag) was
  absent from all four as well, which is quieter and not caught by any test: PullObjectIngester
  passes it to the conditional re-fetch, so losing it turns an If-None-Match into an
  unconditional GET and re-downloads an unchanged segment, and ObjectCarouselPackager copies
  it into the FLUTE file description, which then carries an empty ETag.   [code-derived]

  PullObjectIngester::IngestItem has the same shape of defect in one direction only: its copy
  constructor carries the availability window and its move constructor did not. Items are
  moved on the ordinary queueing paths, so an item reaching the fetch queue by any of those
  routes lost what the copy path preserved.   [code-derived]

  Nothing ordered the packaging queue by when an object has to arrive.   [code-derived]

Basis
  TS 26.517 V18.6.0, clause 6.2.3.5:
  "The MBSTF shall transmit each object in the object list such that the last packet of the
  delivered FLUTE transmission object (including any FEC recovery packets, when configured)
  is available at the MBSTF Client no later than its availability start time."
  The deadline a PackageItem carries is that availability start time, so ordering the queue by
  it is what implements the clause. The copy and move defects are code-derived: no clause
  governs a class's own copy semantics, only that a value the class stores survives being
  copied.

Raised by
  The availability window by running the test suite, which aborted. The entity tag and the
  ingest item's move constructor by reading the same four paths afterwards, looking for the
  same mistake again. Both times it was there.

Change
  Adds the missing members to all four ObjectStore::Metadata copy and move paths, in
  declaration order so initialisation order matches and -Wreorder stays quiet, and to
  IngestItem's move constructor. Adds earlierDeadlineFirst(), a named predicate rather than
  an inline comparator so the ordering can be tested directly, the queue itself being private
  and fed only through a live packager. An item with no deadline sorts after every item that
  has one: nothing is known about when it must arrive, so it cannot displace an object that
  does have a stated time.

Verification
  T1: four suites executed and passing -- testObjectStore 16 cases, testPullObjectIngester 19,
  testObjectListPackager 7, testSubscriberSubscription. Both metadata defects were confirmed
  to be caught, not merely covered: testObjectStore aborts without the availability fix, and
  reports "copy=[] move=[] copy-assign=[] move-assign=[]", 15 pass 1 fail, without the entity
  tag fix.

Not in this change
  IngestItem's own copy and move constructors have no direct test, which is the gap that let
  the move-constructor defect survive. ObjectStore::Metadata::operator== still does not
  compare m_entityTag; the equality semantics of the class are not touched here.
…ransmitter

Problem
  The AL-FEC configuration a Distribution Session provisions reached no transmitter, so every
  session was sent unprotected however it was provisioned. FecOtiHelper, which converts the
  session's FECConfig into the Transmitter-level FEC OTI, was present in the tree but never
  listed in src/mbstf/meson.build, so it was never compiled. Neither ObjectListPackager nor
  ObjectCarouselPackager passed a FEC OTI or a redundancy level to LibFlute::Transmitter,
  which has taken both since the pinned revision, and no controller supplied a FECConfig to
  either packager.   [code-derived]

Basis
  TS 26.517 V18.6.0, clause 6.2.1:
  "If FLUTE [12] is used to realise the Object Distribution Method, the MBS Distribution
  Session shall conform to the MBMS Download Profile as defined in clause L.4 of TS 26.346 [7]
  with the additional requirements in clause 6.2 of the present document."

  TS 26.346 V18.2.0, clause L.4.7 "Other aspects of FLUTE delivery":
  "Regarding Application Layer FEC support, the two FEC schemes referenced in this
  specification, the Compact No-Code FEC scheme as specified in RFC 3695 [13], and the Raptor
  FEC scheme as specified in RFC 5053 [91] are optional to implement by the BM-SC and
  mandatory to support by the UE."

  Those two together decide which schemes may be sent: the profile clause 6.2.1 selects admits
  Compact No-Code and Raptor, and no other. RaptorQ (RFC 6330) is not among them.

Raised by
  Reading the branch while checking the review comments on pull request #71.

Change
  Adds FecOtiHelper to the build. ObjectListPackager and ObjectCarouselPackager convert the
  session's FECConfig with fecOtiFromFecConfig() and pass the resulting FEC OTI and redundancy
  level to LibFlute::Transmitter. A scheme the profile does not admit throws and is reported as
  a packaging failure for that session rather than silently downgraded: a session sent
  unprotected when it asked for protection is a worse outcome than a visible failure.
  ObjectCarouselPackager gains the FEC parameter ObjectListPackager now also carries, defaulted
  so no other call site changes, and all controllers supply
  distributionSession().getFecInformation().

Verification
  T1: tests/test_FecOtiHelper.cc, 8 cases, all passing, covering absent configuration, a null
  shared pointer, Compact No-Code yielding no FEC OTI, Raptor yielding a Raptor OTI carrying the
  requested overhead, and refusal of RaptorQ, an unknown scheme and a negative overhead. The
  suite is new: FecOtiHelper had never been compiled, let alone tested, and one case caught a
  real error while being written, an incorrect Compact No-Code URN. All five suites pass, 50
  cases.
  T2: the demo delivers unchanged with this build. That run does NOT exercise the populated
  path: the demo provisions no FEC, so it evidences no regression, not that FEC transmission
  works.

Not in this change
  No end-to-end evidence that a FEC-protected session is transmitted and decoded. Nothing here
  verifies rt-libflute's own Raptor encoding, and no receiver in this project has been shown to
  decode a protected session. The conformance record must not claim AL-FEC works on the strength
  of this commit.
… DASH handler its own manifest URL

Problem
  ObjectCollectionController.cc/.hh, 368 lines serving the COLLECTION object distribution
  operating mode, were present in the tree but never listed in src/mbstf/meson.build, so they
  were never compiled. The class self-registers with ControllerFactory at file scope, so
  leaving it out of the build means the registration never runs and a Distribution Session
  provisioned with objDistributionOperatingMode COLLECTION finds no controller at all. Once
  added to the build it did not compile: four errors, all from ObjectController::objectStore()
  having become std::shared_ptr<ObjectStore> rather than a reference. The same file listed
  ObjectController.cc twice where the second entry should have been its header, so the header
  was not tracked as a dependency.   [code-derived]

  Separately, DASHManifestHandler::nextIngestItems() declared manifest_url and never assigned
  it, so both comparisons against it tested the empty string and m_refreshMpd was never set: a
  re-fetched MPD was ingested without the handler being told its own manifest had changed.
  [code-derived]

Basis
  COLLECTION is one of the four values ObjDistributionOperatingMode defines
  (src/mbstf/openapi/model/ObjDistributionOperatingMode.h: SINGLE, COLLECTION, CAROUSEL,
  STREAMING), so the API accepts a session the build then has no controller for. That is the
  operative basis and it is code-derived.

  TS 26.502 V18.6.0, clause 4.5.10, first paragraph:
  "An object manifest describes a set of objects to be distributed in an MBS Distribution
  Session that is provisioned in OBJECT_COLLECTION or OBJECT_CAROUSEL operating mode."
  confirming the mode is one the specification defines rather than a local invention. The
  fuller description of the mode is in annex B, clause B.2.1, which is informative.

Raised by
  Reading the branch against its own build file while checking the review comments on pull
  request #71. Rule 14: purpose established before any change, and the code is built rather
  than removed, because the operating mode it serves is one the model defines.

Change
  Adds ObjectCollectionController to src/mbstf/meson.build and corrects the duplicated
  ObjectController.cc entry to name the header. Dereferences the object-store shared pointer at
  the four sites that still treated it as a reference, matching what ObjectCarouselController
  already does, and checks each dynamic_pointer_cast result before use rather than letting a
  std::runtime_error leave the controller, where the surrounding try catches only
  std::out_of_range and one bad ingest response would end the process and every other
  Distribution Session with it. Initialises manifest_url from the manifest's own fetched URL,
  the value addMPDRefreshToExtraPullObjects() keys its refresh entry on, and makes it const.

Verification
  T0 for the new code: both files compile with no new warnings and COLLECTION is present in the
  linked binary, so the controller registers. T1: all five suites pass, 50 cases. T2: the demo
  delivers with this build, gNB carrying MRB1 and MRB2 and the UE logging CRC-OK broadcast
  decodes and MCCH receptions.

Not in this change
  Any verification of COLLECTION behaviour. This builds and registers the controller; it does
  not establish that the mode works, and the conformance record must not claim it. The demo uses
  STREAMING and no test covers COLLECTION. Nothing here covers the MPD-refresh recognition path
  either.
…roject cannot follow

Problem
  Comments across this branch narrated how the code came to be rather than what it does, and
  pointed at material a reviewer cannot read. Three shapes.

  A "BUG FIX:" prefix followed by what the code used to do ("this used to be static", "no DELETE
  branch existed at all", "the else branch used to unconditionally call gw->write_pdu_mch()"). That
  belongs in a commit message; in a comment it dates immediately and tells a reader nothing about
  the code in front of them.

  References to this project's own internal process framework by number: "(rule 12)", "(S12)",
  "see rule 14". Those numbers name nothing in this repository.

  Pointers into a separate, private repository: "see Standards2Deployments/projects/rt-mbs/...",
  "see the findings register", "the register's own item 4 finding". A reviewer cannot open any of
  them.   [code-derived]

Basis
  No clause governs a comment.
  code-derived only.

Raised by
  Reading the branch as a reviewer outside this project would.

Change
  Comments now state the behaviour, the requirement or the invariant, keeping every specification
  citation and every stated reason. Where a comment's only content was its history, the underlying
  rule it was protecting is stated instead: "X was never freed" becomes "this context owns X and
  must free it on removal". Internal rule numbers are replaced by the reasoning they stood for,
  usually that no clause and no configured value fixes a given bound. Private-repository pointers
  are removed, with the substance they referred to summarised in place where it was load-bearing.

  Deliberately NOT changed: the numbered rules in src/mbsf/MultipartMime.cc's Q-encoding comment.
  Those are RFC 2047 section 4.2's own rules 1, 2 and 3, not this project's, and an automated pass
  over the phrase "rule N" would have silently corrupted a correct citation.

Verification
  T0 for the shape of the change: every hunk is a comment or documentation line, confirmed by
  filtering the diff for added lines that are not comments, which comes to zero in all nine
  repositories. T1 where a build exists: open5gs, rt-mbs-function and rt-mbs-transport-function
  all build clean afterwards, and the shell scripts and JSON touched pass bash -n and json.load.

Not in this change
  No behaviour, in any repository. No commit message is rewritten; the history those comments
  described stays in the log, which is where it belongs.
…the configured failure limit

Problem
  ObjectManifestController refetched an object on every ingest failure, for any non-PUSH session,
  with no bound of any kind: no deadline check, no attempt count, no backoff. One object that could
  never be fetched was therefore retried forever.

  Observed on a live run: 59554 ingest failures in a single session, none of which stopped
  anything.   [observed: run/logs/mbstf.log]

  The session-wide limit that exists could not catch it. ObjectController counts consecutive
  failures and deactivates the session at consecutiveIngestFailuresBeforeDeactivate, but the count
  is per Distribution Session and is reset by ObjectStore's ObjectAdded/ObjectUpdated events. The
  same run recorded 4985 successful object fetches, so the counter was reset continually and never
  reached 5 while one object failed 59554 times. A bound that a healthy session resets cannot bound
  a single unfetchable object.   [code-derived: ObjectController.cc, the reset at the
  ObjectAdded/ObjectUpdated branch]

Basis
  TS 26.517 V18.6.0, clause 6.1.2, object manifest parameter latestFetchTime:
  "The MBSTF shall fetch the object no later than this UTC timestamp."

  So an object whose latest fetch time has passed must not be fetched again, whatever has happened
  before. The same parameter's description governs the other case: when latestFetchTime is absent
  "the object shall be present at its origin ... and the MBSTF may fetch it at a time of its
  choosing", which sets no bound at all. A limit there therefore rests on a configuration option
  the operator sets, not on a clause.

Raised by
  Reading the authority for what governs a failed object fetch, after a defensive guard written
  earlier was measured and found not to bound the loop (it rejected the fetch, which produced the
  failure event, which triggered the refetch).

Change
  The refetch decision now refuses two cases. An object whose latestFetchTime has passed is not
  refetched, which is the clause above. An object with no latestFetchTime is refetched until
  consecutiveIngestFailuresBeforeDeactivate consecutive failures of that object, the operator's own
  existing option applied per object rather than per session. PullObjectIngester::IngestItem gains
  the per-object counter, carried through all four of its copy and move paths, since items are
  copied and moved on every queueing path.

Verification
  T1: tests/test_PullObjectIngester.cc, 27 cases passing, 5 of them new: the counter starts at
  zero, accumulates, and survives copy and move; a deadline in the past, a deadline in the future
  and no deadline are each distinguished. Confirmed discriminating, not vacuous: with the copy and
  move carry deliberately reverted the two survival cases fail and the suite reports 25 pass, 2
  fail. All five MBSTF suites pass, 58 cases.
  T2: live demo. Ingest failures fell from 59554 to 15, and the log shows three refusals, each
  "5 consecutive fetch failures reached the configured
  consecutiveIngestFailuresBeforeDeactivate limit of 5": three objects, five attempts each, then
  stopped. Delivery is unaffected, the UE logging 15711 CRC-OK broadcast decodes on the same run.

Not in this change
  Why the ingest URL is corrupted in the first place. A one-byte corruption at index 0 of
  IngestItem's own copy of the fetched URL is still unexplained and still open; this bounds the
  retry loop that made it harmful, it does not fix it. The refusal log line shows an empty object
  id for the affected items, which is consistent with that corruption and is not diagnosed here.
  No backoff is introduced: the clause names no interval and no configuration option supplies one.
…a released reference

Problem
  ObjectStore::getMetadata() takes the store mutex, returns a reference into the store, and releases
  the mutex as it returns. Everything the caller then reads through that reference is unsynchronised.
  Metadata holds std::strings and ObjectStore::updateMetadata() move-assigns them, so a caller copying
  a string while its data pointer and length are being reassigned builds a string from two different
  states of the same object.

  This is the cause of a corruption that had been open and unexplained: an ingest URL arriving 55 bytes
  long with byte 0 zeroed and bytes 1 onward intact, and an object id arriving empty. The store's own
  copy was correct at that moment, which is what made it look impossible.

  Confirmed by ThreadSanitizer, both sides named, holding different mutexes:

    Write of size 8 by thread T4 (mutexes: write M0)
      std::string::_M_data(char*)
      ObjectStore::Metadata::operator=(Metadata&&)
      ObjectStore::updateMetadata(...)
      PullObjectIngester::doObjectIngest()

    Previous read of size 8 by thread T6 (mutexes: write M1, write M2)
      std::string::_M_data() const
      std::string::basic_string(const std::string&)
      PullObjectIngester::IngestItem::IngestItem(ObjectStore::Metadata const&, ...)
      PullObjectIngester::fetch(...)
      ObjectManifestController::workerLoop(...)

  The same call site also chained keepAfterSend() and compressedSend() onto that reference, mutating
  the live store entry with no lock held.   [observed, then code-derived]

Basis
  No clause governs a component's internal locking.
  code-derived and observed only.

Raised by
  Running the component under ThreadSanitizer, after three earlier passes narrowed the corruption to
  the interval between the copy and the end of IngestItem's constructor without identifying any
  writing instruction. Reading the code had already shown the reference outlives the lock; the
  sanitiser is what turned that from a hypothesis into the cause.

Change
  Adds ObjectStore::takeMetadataForIngest(), which takes the lock, applies the two marks and returns
  the metadata by value before releasing it, so the copy the ingest list stores is made while the
  entry cannot be mutated. PullObjectIngester::fetch() uses it instead of copying through
  getMetadata()'s reference.

Verification
  T2: live demo, MBSTF built with -Db_sanitize=thread, same scenario before and after.
  Before: 162 data race reports, 12 of them naming IngestItem's constructor.
  After:  64 data race reports, ZERO naming IngestItem's constructor.
  Delivery unaffected on the same run: gNB carrying MRB1 and MRB2, 2036 CRC-OK broadcast decodes and
  118 MCCH receptions at the UE, and no ingest failures at all.
  T1: all five MBSTF suites pass, 58 cases.

Not in this change
  The other 64 races ThreadSanitizer still reports. They are real and are recorded, but each needs its
  own diagnosis and none is this defect. getMetadata() itself is left in place: its remaining callers
  read under conditions this commit has not examined, and changing its contract is a wider refactor
  than the defect requires.
…cument

Problem
  Running tools/verify-citations.py over this branch's own changed files reported
  citations it could not confirm. Each was a comment whose quoted sentence differed
  from the document it named.   [code-derived]

Basis
  No clause governs how a comment is written. The defect is that a quotation did not
  match its source, which is checkable without any specification claim.
  code-derived only.

Raised by
  Running the citation checker across this branch for the first time.   [rule 13]

Change
  Quotations now reproduce contiguous source text. Where an earlier comment joined
  two sentences with an elision, each is quoted separately. Where it inserted an
  editorial gloss inside the quotation marks, the gloss moved outside them. Where a
  specification writes a value inside its own quotation marks (a status code, a state
  name, a file extension), that fragment is named without quotation marks rather than
  nested, and the comment says why, so nobody restores them.

  Two citations that sat inside runtime strings moved into comments beside them: a
  full document identifier in a log or exception message is read as a citation by the
  checker, which then matches the next string literal in the file.

  No behaviour changes. Comments only.

Verification
  T1: tools/verify-citations.py reports no unconfirmed citation across this branch's
      changed files.
  T0: builds clean.

Not in this change
  Citations naming a document the local corpus does not hold. Those remain unchecked
  and are listed in the project's own specification index.
…aves the host

Problem
  FLUTE encoding symbols are sized from getsockopt(IP_MTU) on a socket to the
  distribution session's ingress. Where the MBSTF and the ingress point are
  co-located that destination is one of the host's own addresses, so the kernel
  routes it over loopback and answers 65535. Symbols were sized at 65441 bytes and
  every object larger than one of them left as a datagram nothing downstream
  carries.

  Measured on the rt-mbs-examples broadcast demo, OBJECT_STREAMING session: the FDT
  advertised FEC-OTI-Encoding-Symbol-Length="65441"; the client was told about 663
  distinct TOIs and received data packets for three of them, TOI 0 (the FDT
  itself), 2 and 4. Every media segment TOI got its FDT Instance and then no data.
  [observed, code-derived]

Basis
  RFC 5651 section 6.1: "However, network efficiency considerations recommend that
  the sender uses an as large as possible packet payload size, but in such a way
  that packets do not exceed the network's maximum transmission unit size (MTU), or
  when fragmentation coupled with packet loss might introduce severe inefficiency
  in the transmission."

  RFC 5651 is the LCT building block TS 26.346 V18.2.0 lists as reference [119].

  No clause fixes a number. mbstf.pathMtu is the operator's setting and is
  documented in mbstf.yaml with a default, which is what rule 12 requires of a
  bound.

Raised by
  Counting, per TOI, the data packets a client received against the TOIs its FDTs
  advertised, while chasing why no media segment reached a player. The shape of the
  fix comes from review by David Waring on pull request #71, who pointed out that
  clamping a discovered MTU would prevent the jumbo frames his lab runs between the
  MB-UPF and the gNodeB.   [rule 13]

Change
  A discovered MTU is used as it stands whenever the destination is not one of this
  host's own addresses. A deployment configuring jumbo frames on its interfaces
  gets them; nothing here caps what the operator set up.

  get_path_mtu() reports, through via_loopback, whether the destination is a local
  address. It tests the destination against getifaddrs() rather than for the 127/8
  prefix, because a co-located MB-UPF is commonly reached on the address of a real
  interface -- in the demo, a veth -- which loops back without looking like a
  loopback address. Only in that case, where there is no path to measure, is
  Context::pathMtu used, and the substitution is logged with both numbers.

  The four controllers sequence the discovery before reading the flag rather than
  nesting the calls: the order function arguments are evaluated in is unspecified.

Verification
  T1: meson test, the five rt-mbs-transport-function suites pass. The other
      failures in that run are open5gs and libmpdpp subproject tests, untouched by
      this change.

  T2: run scripts/mbs-broadcast-demo/start-all.sh in rt-mbs-examples. mbstf.log
      records "The route to this session's ingress is loopback, so its 65535 byte
      MTU is not the path to a receiver; sizing FLUTE symbols for the configured
      1500 byte path MTU instead", the FDT then advertises
      FEC-OTI-Encoding-Symbol-Length="1406", and the client receives the
      announcement bundle, both initialisation segments, the manifest and media
      segments. Before this change it received data for three TOIs out of 663.

Not in this change
  GTP_HEADER_SIZE in common.hh, which is 2. Whether that is the right allowance
  needs TS 29.281, which is not held; halted under rule 2.
Problem
  In OBJECT_STREAMING every media segment is transmitted several times, each time under a
  different TOI. Counting distinct TOIs against distinct Content-Locations in the FDTs a
  receiver was given: 4.0 TOIs per object on average, and up to 15.

    15 x chunk-stream0-00047.m4s   11 x manifest.mpd   10 x chunk-stream0-00051.m4s

  This is not redundancy. A receiver cannot combine symbols across TOIs, so each copy is a
  separate object that is independently incomplete, and the copies consume the bearer the
  first copy needed. Measured live on the rt-mbs-examples broadcast demo, the effect is that
  no media segment ever completes at the client: symbol IDs arrive with gaps throughout, for
  example SBN 1 ID 7, 11, 12, 16, 19, 20, 22, 26, 27, 30, 32, 33, 42, while the radio itself
  delivered 33382 of 33455 grants with zero CRC failures.   [observed]

  DASHManifestHandler::nextIngestItems() takes every segment the MPD currently advertises and
  clamps any whose availability start has passed to the present, so a segment stays in the
  candidate set for the whole of its availability window. Nothing records that it has already
  been sent. The ObjectStore cannot answer for it either: ObjectStreamingController leaves
  Metadata::keepAfterSend() at false, so ObjectController deletes the object as soon as it is
  sent, findMetadataByURL() then misses, and a second object is created for the same URL,
  which the packager sends under a second TOI.   [code-derived]

Basis
  RFC 3926 clause 3.1: "Note that each object is associated with a unique TOI within the scope
  of a session."
  Sending one file under several TOIs therefore presents it as several objects, and a receiver
  has no basis on which to combine their symbols. An MPD advertising a segment states that a
  client may still fetch it, not that it still needs transmitting; no clause requires a segment
  to be sent more than once.

Raised by
  Counting TOIs per Content-Location while establishing why no object completed at a
  receiver, after the path-MTU defect was fixed and the segments started arriving.   [rule 13]

Change
  DASHManifestHandler remembers the media segment URLs it has already handed to the ingester
  and skips them on later passes. The set is pruned against the current manifest on every
  pass, so it holds at most one entry per segment the MPD still advertises.

  Only segments the MPD itself advertises are suppressed. The MPD refresh and the
  initialisation segments, which come from m_extraPullObjects, are meant to repeat and are
  deliberately not recorded.

  keepAfterSend() is untouched. Removing it from ObjectManifestController was correct: an
  OBJECT_STREAMING object genuinely should not be retained once sent, and the carousel sets
  it for itself. What was missing is a record of what has been sent, which is added here
  rather than by retaining objects that are no longer needed.

Verification
  T0: builds clean.

Not in this change
  Anything in the carousel path, which retains its objects and repeats them under one TOI.
  The gNB-side per-slot MBS scheduling limits, which are a separate matter in
  srsRAN_Project_mbs.
@jordijoangimenez
jordijoangimenez force-pushed the feature/mbs-compliance-fixes branch from 88e9fa0 to 2958c1e Compare September 7, 2026 08:11
@jordijoangimenez

Copy link
Copy Markdown
Contributor Author

@davidjwbbc You were right, and thank you — capping a discovered MTU would break your jumbo-frame lab, and that solution is off the branch.

What replaced it keeps your mechanism. A discovered MTU is now used exactly as it stands whenever the destination is not one of the host's own addresses, so interface MTUs remain the way this is configured and jumbo frames between MB-UPF and gNodeB are passed through untouched.

The one case it treats differently is a destination that is a local address, where the kernel routes over loopback and answers 65535. There is no path to measure there, so mbstf.pathMtu is used and the substitution is logged with both numbers. It tests against getifaddrs() rather than the 127/8 prefix, because a co-located MB-UPF is typically reached on a real interface address — in our demo a veth — which loops back without looking like a loopback address.

Why it matters: with the loopback answer, symbols were sized at 65441 bytes and a receiver was told about 663 TOIs but got data packets for three of them. RFC 5651 §6.1 asks that the sender not exceed the path MTU, which a loopback figure is not.

If your lab has the MB-UPF on the same host as the MBSTF, I would like to know — that is the one configuration where this now diverges from pure discovery.

@jordijoangimenez

Copy link
Copy Markdown
Contributor Author

@davidjwbbc You are right on both, and both changes are off the branch.

validate_distribution_session() / PUSH+SINGLE. I checked your citation against the document rather than take it on trust, and it is exact — TS 26.502 V18.6.0: "When the push-based object acquisition method is provisioned, the set of Object acquisition identifiers shall be empty." An objAcquisitionIdPush in PUSH mode is therefore an error and the original return code was correct. My change would have accepted what the specification forbids. Withdrawn.

StatusNotify retry. You are right that I overstated TS 29.581 — I cited it as obliging reliable delivery and it does not say that. The retry change is off the branch, and the current try-once behaviour stands.

Your reading of the status codes is the useful part and I have not tried to implement it here: 4xx means the client should not repeat the same request, 5xx may be retried after a delay, and 307/308 should be reissued to the new URL, with 308 replacing the stored one. That plus a retry limit is a real piece of work and belongs in its own change, raised against a maintainer's judgement rather than smuggled in behind a wrong citation.

@jordijoangimenez

Copy link
Copy Markdown
Contributor Author

@rjb1000 Both duplications are gone. The branch was rebuilt on development after #72 merged, so 9bbd5656 — the #72 merge commit — is now its base rather than something it duplicates.

  • The null-handler crash fix is @davidjwbbc's c3956ca, inherited, not reimplemented here.
  • The Content-Type matching fix likewise: this branch no longer touches ManifestHandlerFactory.cc at all.

Nothing was cherry-picked; taking his versions and rebasing was cleaner than picking through mine, which is what I said I would do.

The branch has been rewritten since your review, so the commits you read are no longer the ones on it. It now stands at 12 commits, and the description has been rewritten to match.

jordijoangimenez and others added 5 commits September 15, 2026 18:05
…s run

Problem
  meson setup failed on any checkout that had not been built before:

    src/mbstf/meson.build:66:29: ERROR: File openapi/model/FECConfig.cc does not exist.

  FECConfig.cc is generated, not tracked: src/mbstf/.gitignore excludes the whole
  openapi directory and the generator writes its 153 files at configure time.
  meson's files() checks existence when it is evaluated, and the test source list
  naming those two generated files sat above the generator invocation, so a fresh
  checkout failed and only a tree left over from an earlier build succeeded.
  [observed: clean clone of this branch; code-derived: meson.build ordering]

Basis
  No clause governs this; it is this repository's own build definition.
  [code-derived]

Raised by
  Building this branch from a clean clone, after a user report that third-party
  developers must be able to build it themselves.

Change
  test_source_fec_oti_helper is defined after the generator has run and written
  .openapi.srcs. Its only consumer is tests/meson.build, which the top-level
  meson.build enters after src, so nothing else moves.

Verification
  T2: a clean clone of this branch builds through to open5gs-mbstfd.

Not in this change
  The generator's own failure reporting, which is fixed in rt-common-shared.
…se branch

Problem
  The submodule pointed at a commit that exists only on a local 3GPP-Rel18, while
  rt-mbs-function pinned the same submodule to feature/mbs-compliance-fixes. The
  two network functions were therefore built against different versions of a
  shared library, and this one missed the MBS work and the generator's
  failure-reporting fix. Because that commit was never pushed, the pin also
  resolved only on the machine it was made on. [code-derived: the two gitlinks]

Basis
  No clause governs this; it is this repository's own dependency pin.
  [code-derived]

Raised by
  User question asking why this repository was not pinned to what it needs.

Change
  The submodule advances to the MBS branch's tip, which carries the
  ACCESS_TOKEN_CLAIM_MISSING change this repository was pinned to 3GPP-Rel18 for,
  and adds the generator fix and the HTTP server work. Both network functions now
  build against the same rt-common-shared.

Verification
  T2: a clean clone of this branch builds through to open5gs-mbstfd against that
  tip.

Not in this change
  Publishing the rt-common-shared branch this points at.
Problem
  The README opened with a bare heading and two badges, had no At a glance table,
  and its clone command named no branch and cloned into a path under $HOME. It
  did not say that the build fetches the 3GPP 5G APIs over the network, nor what
  the two 5G-MAG libraries it pulls in supply. [code-derived]

Basis
  No clause governs a repository's own README. The structure is the house
  baseline used by rt-cmmf-encoder. [code-derived]

Raised by
  User report that each repository must state what to install, clone, build,
  install and run.

Change
  The baseline's header and sections, including Specification with the versions
  this is built against, and Dependencies naming what rt-common-shared and
  rt-libflute supply.

  A section on the 5G APIs fetch: that it needs network access and Java, that
  forge.3gpp.org serves an incomplete certificate chain, and how to install the
  missing intermediate rather than disable verification.

Verification
  T2: a clean clone built through to open5gs-mbstfd following these
  instructions, including the certificate step, which is what makes the
  generation succeed.

Not in this change
  The generator and the build definition, fixed separately.
Problem
  The submodule pinned a commit that no longer exists on rt-common-shared's MBS
  branch. The branch was reauthored before being published, which changed every
  commit hash on it, and this pin still named the pre-rewrite one. A clean clone
  therefore failed at submodule init:

    fatal: remote error: upload-pack: not our ref 0bab3d165085287320ab7742941f6ba1902e5afe

  [observed: clean clone of this branch after rt-common-shared was published]

Basis
  No clause governs this; it is this repository's own dependency pin.
  [code-derived]

Raised by
  Rehearsing the documented clone-and-build steps against the published
  repositories.

Change
  The submodule advances to the published tip of rt-common-shared's MBS branch.
  Its content is unchanged from the commit previously pinned; only the hashes
  differ.

Verification
  T2: a clean clone of this branch now initialises its submodule from the public
  remote and builds.

Not in this change
  Anything in rt-common-shared, whose content this only follows.
Problem
  This repository pinned a commit on rt-libflute's Raptor line while rt-mbs-client
  pinned a release tag, so the two components of the same delivery chain were
  built against different versions of the same library. Neither pin carried what
  the other component needed, and the client's did not carry what the client
  itself needed. [code-derived: the two pins]

Basis
  No clause governs this; it is this repository's own dependency pin.
  [code-derived]

Raised by
  User request that both components be pinned so a third party can build and run
  the demo end to end.

Change
  revision = feature/mbs-profile, which both this repository and rt-mbs-client
  now use. It carries the FLUTE-layer obligations this repository depends on --
  the TS 26.346 annex L.6 profiled FDT schema, the scheme-specific FEC OTI,
  suppression of Transfer-Length under the MBMS Download Profile, the split
  expiry setter and the RFC 5053 Raptor scheme -- alongside the receiver work the
  client needs. Every commit on it is already on an open pull request; that
  branch's own merge commit records which, and what was resolved.

  The wrap comment is rewritten to say this rather than describing the old pin.

Verification
  T2: a clean clone of this branch, with the new pin, builds through to
  open5gs-mbstfd.

Not in this change
  The pull requests that branch mirrors, which are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants