Clarification regarding snapshot building and interpreting.

[natilivni]

I think I need some clarification regarding the SnapshotBuilder pattern.

Here is how I understand it right now:

1. The snapshot is built by a call to public override ValueTask WriteToAsync<TWriter>(TWriter writer, CancellationToken token). All logic needed to persist a snapshot should be placed in there.
2. Restoring a snapshot state is done when a snapshot LogEntry is passed to the ApplyAsync(LogEntry entry) method of the SnapshotBuilder. The SnapshotBuilder then restores the state using the passed in entry.
3. The ApplyAsync(LogEntry entry) method of the PersistentState is never called with a LogEntry of type IsSnapshot, those only get passed to the SnapshotBuilder.
4. Further, it is quite hard to create a Blittable snapshot type. State, by its nature is quite dynamic. So I assume it is a legitimate to serialize state as as string.

Are these assumptions correct?

[sakno]

1. WriteToAsync is a method from IDataTransferObject and it's used for serialization. The serialized state will be persisted by PersistentState automatically.
2. Not exactly. PersistentState.CreateSnapshotBuilder must be overridden by the derived class. This method is called by PersistentState automatically when it decides to build a snapshot. SnapshotBuilder.ApplyAsync is calling sequentially for each log entry in the partition to be squashed. So the builder is responsible for constructing a single squashed log entry from multiple log entries. As you can see, SnapshotBuilder itself implements IRaftLogEntry interface.
3. It's called on application startup if PersistentState.Options.ReplayOnInitialize was set to true which is default behavior. During the runtime, the snapshot will never be passed to PersistentState.ApplyAsync method, there is no need in this.
4. Log entry is not restricted to blittable type. The structure of serialized log entry can be of any form. IAsyncBinaryWriter provides wide range of the methods for serialization that can be called sequentially.

As stated previously, log entry is based on DTO concept represented by IDataTransferObject interface. This interface provides two key methods:

1. WriteToAsync method for serialization
2. GetObjectDataAsync for transforming from the current representation of DTO to another

Let me provide a basic example of the log entry which holds a dictionary where keys and values are represented by strings.

public readonly struct MyLogEntry : IRaftLogEntry
{
  public const short Prefix = 1;

  private readonly Dictionary<string, string> payload;

  public MyLogEntry(Dictionary<string, string> payload, long term)
  {
    Term = term;
    Timestamp = DateTimeOffset.Now;
    this.payload = payload;
  }

  public long Term { get; }
  public DateTimeOffset Timestamp { get; }

  long? IDataTransferObject.Length // optional implementation, may return null
  {
    get
    {
      // compute length of the serialized data, in bytes
      long result = Count; // 4 bytes for count
      for (var (key, value) in payload)
      {
        result += Encoding.UTF8,GetByteCount(key) + Encoding.UTF8.GetByteCount(value);
      }

      return result;
    }
  }

  public async ValueTask WriteToAsync<TWriter>(TWriter writer, CancellationToken token)
    where TWriter : notnull, IAsyncBinaryWriter
  {
    // write prefix to recognize this type of the log entry
    await wrtier.WriteInt16Async(Prefix, true, token);
    // write the number of entries
   await writer.WriteInt32Async(payload.Count, true, token);
   // write the entries
   var context = new EncodingContext(Encoding.UTF8, true);
   foreach (var (key, value) in payload)
   {
     await writer.WriteAsync(key.AsMemory(), context, StringLengthEncoding.Plain, token);
     await writer.WriteAsync(key.AsMemory(), context, StringLengthEncoding.Plain, token);
   }
  }
}

Now we have custom log entry that wraps the dictionary and serialization logic for it. Each custom log entry must be distinguishable for other types. In this example I'm using a prefix represented by 16-bit signed integer. The custom log entry can be instantiated and added to the log using one of the PersistentState.AppendAsync overloads.

PersistentState.LogEntry is a generic representation of the log entry. The internal structure of the log entry is unknown to the log. But it's known to you. However, this type has GetObjectDataAsync method that can be used to transform the unstructured representation back to structured (deserialized) representation. Therefore, we need to have a deserialization logic for our log entry:

public async ValueTask<Dictionary<string, string>> DecodeAsync<TReader>(TReader reader, CancellationToken token);
    where TReader : notnull, IAsyncBinaryReader
  {
    var count = await reader.ReadInt32Async(true, token);
    var result = new Dictionary<string, string>(count);
    // deserialize entries
    var context = new DecodingContext(Encoding.UTF8, true);
    while (count-- > 0)
    {
       string key = await reader.ReadStringAsync(StringLengthEncoding.Plain, context, token);
       string value = await reader.ReadStringAsync(StringLengthEncoding.Plain, context, token);
       result.Add(key, value);
    }
    return result;
}

Log entry metadata such as Term and Timestamp should not be included in serialization/deserialization logic. The metadata is stored automatically by the infrastructure.

Note that the entry ID is not included in the deserialization logic. This is happening because reading of the entry prefix is a responsibility of the interpreter:

public class MyInterpreter : IDataTransferObject.IDecoder<int>
{
  public async ValueTask<int> DecodeAsync<TReader>(TReader reader, CancellationToken token)
    where TReader : notnull, IAsyncBinaryReader
  {
   var prefix = await reader.ReadInt16Async(true, token);
   switch (prefix)
   {
    case MyLogEntry.Prefix:
     var dictionary = await MyLogEntry.DecodeAsync(reader, token);
     // interpretation logic here
    default:
      throw new Exception($"Unknown command {prefix}");
   }
   return prefix;
  }
}

Now it's possible to use this interpreter inside of the class derived from PersistentState:

public class MyPersistentState : PersistentState
{
  private readonly MyInterpreter interpreter = new MyInterpreter();

  private ValueTask<int> InterpretAsync<TEntry>(TEntry entry)
    where TEntry : struct, IRaftLogEntry
    => entry.GetObjectDataAsync(interpreter, CancellationToken.None);

  protected override async ValueTask ApplyAsync(LogEntry entry)
  {
    if (entry.IsSnapshot)
    {
      // interpret snapshot
    }
    else
    {
      await InterpretAsync(entry);
    }
  }
}

[natilivni]

Thank you for the detailed response. So if I understand correctly, When the log is compacted, a snapshotbuilder is created and then the previous snapshot as well as all subsequent log entries until the compaction point are passed into the builder. After this is done, the builder is asked to write the content of the snapshot entry.

Isn't this a bit heavy? Since I am already keeping the state of the node in my implementation of persistent state, can I not simply know that this state is identical to the one that would be generated by the application of the entire in the snapshotbuilder? And if so, can I not simply use that to generate the content of the next snapshot?

[sakno]

Yes, you can. The default implementation of PersistentState is for general applicability when the actual implementation is now known to me. Therefore, I cannot rely on some specific use cases.

From other side, typically you don't need to persist the internal state separately from PersistentState. Your state can be reproduced as a replay of all committed log entries including the snapshot.

[natilivni]

Thank you! Regarding the binary encoding. I wonder what the performance boost is over simply serializing the dictionary as a json string and storing that? Is the inefficiency introduced by this method really that significant?

[sakno]

JSON is a text, binary encoding is not. JSON is good for wiring and debugging. Files produced by PersistentState are binary by design. There is no benefit to use JSON as a format for log entries. However, you're not limited to any format. Use JSON, MessagePack or even standard binary serialization from .NET if you want.

You can use BenchmarkDotNet to measure the performance in your use case.