"Received a packet with an invalid Magic Value" due to huge message

[scs-b]

Description

This bug has only happened a few times in the past 6 months, and we're not sure what's causing it.

We are letting the game run on repeat for hours or days, and this happened after successfully running the game for over 24 hours.

The log says that the client received a seemingly huge message, but the server is not complaining about trying to write too much to the FastBufferWriter. This is how we send the message;

public void SendInitState(InitState message) {
    using var writer = new FastBufferWriter(4096, Allocator.Temp);
    writer.WriteValueSafe(message);
    _sharedData.NetworkManager.CustomMessagingManager.SendNamedMessageToAll(Constants.InitStateMessage, writer, NetworkDelivery.ReliableFragmentedSequenced);
}

Error message

[Netcode] Received a packet with an invalid Magic Value. Please report this to the Netcode for GameObjects team at https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/issues and include the following data: Offset: 4, Size: 6357069, Full receive array: (25mb of data here)
[Netcode] Received a packet too small to contain a BatchHeader. Ignoring it.
[Netcode] Received a packet too small to contain a BatchHeader. Ignoring it.
ArgumentOutOfRangeException: ArgumentOutOfRange_NeedNonNegNum
Parameter name: count
  at System.ArraySegment`1[T]..ctor (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x00011] in <467a840a914a47078e4ae9b0b1e8779e>:0 
  at Unity.Netcode.Transports.UTP.BatchedReceiveQueue.PopMessage () [0x0003a] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.ReceiveMessages (System.UInt64 clientId, Unity.Networking.Transport.NetworkPipeline pipeline, Unity.Networking.Transport.DataStreamReader dataReader) [0x00046] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.ProcessEvent () [0x000c4] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.Update () [0x000c3] in <cb73a45f761942c68e8e5de2a8859e77>:0 

Environment

• OS: Windows 11
• Unity Version: 2022.3.30f1
• Netcode Version: 1.11.0
• Unity Transport Version: 1.4.1

[NoelStephensUnity]

Are you using the default UnityTransport or a 3rd party transport?

[scs-b]

We're using the default UnityTransport, installed automatically as a NGO 1.11.0 dependency.

[NoelStephensUnity]

@scs-b
I have the same question as #3153 about using relay and whether it is a pure server or host you are using. This will help me narrow down where the issue is coming from.

For both this issue and #3153, if you are using Unity relay then my next question is whether you have you run the same kind of burn in tests locally without the use of relay or any services and if so do you get the same issue ?

[scs-b]

@NoelStephensUnity
We were able to fix this by replacing one of our network message calls with an RPC.

Old implementation:

// Server
private void SynchronizeState(AppState message) {
    SetLocalState(message).Forget();
    using var writer = new FastBufferWriter(1024, Allocator.Temp);
    writer.WriteValueSafe(message);
    NetworkManager.CustomMessagingManager.SendNamedMessageToAll(Constants.AppStateMessage, writer, NetworkDelivery.Reliable);
}

// Client
private void ReceiveMessage(ulong senderClientId, FastBufferReader messagePayload) {
    messagePayload.ReadValueSafe(out AppState newState);
    SetLocalState(newState).Forget();
}

New implementation:

[Rpc(SendTo.Everyone)]
private void SynchronizeStateRpc(AppState message) {
    SetLocalState(message).Forget();
}

We haven't had issues with larger network messages that contain the entire game state, so I thought the issue might be that this message was too small. I changed this method to an RPC as well just to be sure.

private void SendMessage() {
    using (var writer = new FastBufferWriter(1024, Allocator.Temp)) {
        _appManager.NetworkManager.CustomMessagingManager?.SendNamedMessageToAll(Constants.RecentreMessage, writer, NetworkDelivery.Reliable);
    }
}

We successfully ran our burn in test for up to 3 days without any issues.

We are not using relay, and we are using a pure server - it does not double as a client.

Hope this helps!

[NoelStephensUnity]

@scs-b
Ahhh.... that does help indeed. So, converting over from a custom named message to an RPC ended up solving the issue.... 🤔
It would be nice to see the InitState serialization code (did it implement INetworkSerializable?) in order to determine how the buffers could get into a bad state like that (to try and replicate on our end to see if we can catch the scenario before it happens and/or provide more detailed information about the issue).

Glad you got past the issue...but if/when you have time it would help us track down where the buffer corruption was happening.
(will also look through our custom named message code a bit more too)

[scs-b]

@NoelStephensUnity
I'm honestly not sure if the initstate message was the culprit since we did get the Received a packet too small to contain a BatchHeader. Ignoring it. message right after, and that has been the consistent symptom throughout the issues we've had. Here how we serialize InitState though;

public struct InitState : INetworkSerializable, IDisposable {
    public LevelType Level;
    public NativeList<SyncedViewType> ViewTypes;
    public NativeList<SyncedPosition> Positions;
    public NativeList<SyncedRotation> Rotations;
    public NativeList<SyncedOwner> Owners;

    public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter {
        serializer.SerializeValue(ref Level);
        serializer.SerializeValue(ref ViewTypes);
        serializer.SerializeValue(ref Positions);
        serializer.SerializeValue(ref Rotations);
        serializer.SerializeValue(ref Owners);
    }

The message that we replaced with an RPC was just sending an AppState enum that looks something like this;

public enum AppState {
    Boot,
    Connecting,
    Lobby,
    Game,
}

#3153 ended up being a bigger issue for us since it was happening consistently. This magic number thing was a lot more rare. Since this change we haven't ran into any of these issues though!