Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
7c8f5af
An untested concept for json-based persistence
Elesbaan70 Mar 29, 2022
6556ac3
Merge branch 'master' into json-persistence
Elesbaan70 Mar 29, 2022
21818ae
Untested demonstration code will certainly have issues like this
Elesbaan70 Mar 29, 2022
8ebe2e1
Move manager containers to their own stream so old serialization data…
Elesbaan70 Apr 2, 2022
3e80477
UTF-8 will pretty much cut the size of the serialized data in half, s…
Elesbaan70 Apr 2, 2022
1f3b84d
merge 'master' into 'json-persistence'
Elesbaan70 Apr 5, 2022
c179156
Basic Linq to XML based framework
Elesbaan70 Apr 7, 2022
0fa9a85
Account for persistent object dependencies
Elesbaan70 Apr 8, 2022
9800020
Merge branch 'master' into xml-persistence
Elesbaan70 Apr 8, 2022
f6cc9fd
Persistence namespace
Elesbaan70 Apr 8, 2022
5b69eaa
Clean up persistence framework and stub out new ttl persistence
Elesbaan70 Apr 9, 2022
58d5120
initial draft of save code
Elesbaan70 Apr 9, 2022
1b53ed2
Merge branch 'master' into xml-persistence
Elesbaan70 Apr 9, 2022
d04a570
TTL save code reworked, plus partial implementation of load
Elesbaan70 Apr 10, 2022
6d7cf3b
Remove extra layer originally intended for per-node versioning, direc…
Elesbaan70 Apr 13, 2022
b519407
Merge branch 'master' into xml-persistence
Elesbaan70 Apr 13, 2022
fcaca36
Further breaking down of large methods for clarity
Elesbaan70 Apr 14, 2022
43fa12d
half-baked traffic light model
Elesbaan70 Apr 28, 2022
fa6f141
Merge branch 'master' into xml-persistence
Elesbaan70 Apr 29, 2022
762b217
Some XML DOM bugs
Elesbaan70 May 4, 2022
40306ba
Workaround for Mono bug
Elesbaan70 May 5, 2022
fc2d3ea
Typos and logging
Elesbaan70 May 6, 2022
f1d735a
Fix all the little problems that kept this from working
Elesbaan70 May 7, 2022
2c7e2f9
Remove the DEBUGLOAD symbol
Elesbaan70 May 7, 2022
df5f375
Remove some unused stuff
Elesbaan70 May 7, 2022
f61d990
Renaming
Elesbaan70 May 7, 2022
51caa14
Unused usings didn't work in release build
Elesbaan70 May 7, 2022
3ceee80
Include System.Xml.Linq.dll in deployment
Elesbaan70 May 7, 2022
eb0cd5d
Merge branch 'master' into xml-persistence
Elesbaan70 May 8, 2022
639e4b7
Fully persistable model for JunctionRestrictionsManager
Elesbaan70 May 8, 2022
8aad2fa
Load/Save Junction Restrictions in DOM
Elesbaan70 May 14, 2022
02254d6
Merge branch 'master' into xml-persistence
Elesbaan70 May 14, 2022
4b2ef27
Remove setters from read-only traffic light model
Elesbaan70 May 14, 2022
dc98ac1
Compression
Elesbaan70 May 14, 2022
5f03eeb
Separate container for each manager's save data
Elesbaan70 May 21, 2022
71595b0
make ReferenceEqualityComparer<T> a singleton
Elesbaan70 May 21, 2022
a6a8ce1
Some better logging
Elesbaan70 May 26, 2022
ed6f077
Merge branch 'master' into xml-persistence
Elesbaan70 May 26, 2022
0772a52
Use SharpZipLib that is bundled with CS
Elesbaan70 May 31, 2022
d53e13f
Merge branch 'master' into xml-persistence
Elesbaan70 May 31, 2022
0788894
Fix DLL reference
Elesbaan70 May 31, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
385 changes: 248 additions & 137 deletions TLM/TLM/Lifecycle/SerializableDataExtension.cs

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions TLM/TLM/Manager/Impl/JunctionRestrictionsManager.Persistence.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Linq;
using TrafficManager.Manager.Model;
using TrafficManager.Persistence;

namespace TrafficManager.Manager.Impl {
partial class JunctionRestrictionsManager {

internal class Persistence : PersistentObject<Persistence.JunctionRestrictionsFeature> {
public override XName ElementName => "JunctionRestrictions";

public static readonly XName segmentElementName = "Segment";

public override Type DependencyTarget => typeof(JunctionRestrictionsManager);

public override IEnumerable<Type> GetDependencies() {
yield return typeof(LaneConnection.LaneConnectionManager);
}

protected override PersistenceResult OnLoadData(XElement element, ICollection<JunctionRestrictionsFeature> featuresRequired, PersistenceContext context) {

foreach (var segmentElement in element.Elements(segmentElementName)) {
Instance.AddSegmentModel(segmentElement.XDeserialize<SegmentEndPair<JunctionRestrictionsModel>>());
}

return PersistenceResult.Success;
}

protected override PersistenceResult OnSaveData(XElement element, ICollection<JunctionRestrictionsFeature> featuresRequired, ICollection<JunctionRestrictionsFeature> featuresForbidden, PersistenceContext context) {
var result = PersistenceResult.Success;

foreach (var segment in Instance.EnumerateSegmentModels()) {
element.Add(segment.XSerialize(segmentElementName));
}

return result;
}

public enum JunctionRestrictionsFeature {

None = 0,
}

}
}
}
207 changes: 129 additions & 78 deletions TLM/TLM/Manager/Impl/JunctionRestrictionsManager.cs

Large diffs are not rendered by default.

409 changes: 409 additions & 0 deletions TLM/TLM/Manager/Impl/TrafficLightSimulationManager.Persistence.cs

Large diffs are not rendered by default.

14 changes: 13 additions & 1 deletion TLM/TLM/Manager/Impl/TrafficLightSimulationManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ namespace TrafficManager.Manager.Impl {
using TrafficManager.Util;
using ColossalFramework;
using TrafficManager.Util.Extensions;
using TrafficManager.Persistence;

public class TrafficLightSimulationManager
public partial class TrafficLightSimulationManager
: AbstractGeometryObservingManager,
ICustomDataManager<List<Configuration.TimedTrafficLights>>,
ITrafficLightSimulationManager
Expand All @@ -29,11 +30,22 @@ private TrafficLightSimulationManager() {
for (int i = 0; i < TrafficLightSimulations.Length; ++i) {
TrafficLightSimulations[i] = new TrafficLightSimulation((ushort)i);
}

GlobalPersistence.PersistentObjects.Add(new Persistence());
}

public static readonly TrafficLightSimulationManager Instance =
new TrafficLightSimulationManager();

private IEnumerable<TimedTrafficLights> EnumerateTimedTrafficLights() {

for (int i = 0; i < NetManager.MAX_NODE_COUNT; i++) {
if (TrafficLightSimulations[i].IsTimedLight()) {
yield return TrafficLightSimulations[i].timedLight;
}
}
}

/// <summary>
/// For each node id: traffic light simulation assigned to the node
/// </summary>
Expand Down
14 changes: 14 additions & 0 deletions TLM/TLM/Manager/Model/JunctionRestrictionsModel.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using TrafficManager.API.Traffic.Enums;

namespace TrafficManager.Manager.Model {
internal struct JunctionRestrictionsModel {

public JunctionRestrictionsFlags values;

public JunctionRestrictionsFlags mask;
}
}
15 changes: 15 additions & 0 deletions TLM/TLM/Manager/Model/SegmentEndPair.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace TrafficManager.Manager.Model {
internal struct SegmentEndPair<T> {

public ushort segmentId;

public T start;

public T end;
}
}
198 changes: 198 additions & 0 deletions TLM/TLM/Persistence/FeatureFilter.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Linq;

namespace TrafficManager.Persistence {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading save file with features blocked, will it skip loading the incompatible sections?
When you save the game will the game lose that skipped data?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no attempt to preserve unsupported data through a load and re-save. That data is completely skipped.

I would oppose adding this as a requirement. Putting it into practice in a reliable way is far more complicated than it might appear on the surface.

When saving XML data, the save code has the option of marking its data as requiring a specific feature identified in an enum. It gets added to a featuresRequired collection.

When featuresRequired comes back non-empty, the framework transfers the required feature with the highest numeric value (which would normally mean the newest feature) to the featuresForbidden collection, clears featuresRequired, and repeats the save.

The above step is repeated until the save code either (1) is no longer adding items to featuresRequired, or (2) indicates that it is no longer able to save the data. In the latter case, this means that some older versions will simply lose this data, due to either a fundamental incompatibility or an intentional choice on our part.

Outside of dev and alpha builds, we will normally choose not to write old data, since we do not support older versions. We could decide to keep writing old data for some period of time, in case an emergency rollback is necessary.

internal sealed class FeatureFilter<TFeature>
where TFeature : struct {

private const string featuresRequiredAttributeName = "featuresRequired";
private const string featuresForbiddenAttributeName = "featuresForbidden";

private enum FeatureStatus {
Forbidden,
Required,
}

static FeatureFilter() {
Type tFeature = typeof(TFeature);
if (!tFeature.IsEnum)
throw new PersistenceException($"FeatureSet<{tFeature.FullName}>: Type {tFeature.Name} is not an enum");
}

private readonly Dictionary<TFeature, FeatureStatus> filter;

public static bool CanLoad(XElement element, out FeatureFilter<TFeature> result) {
return CanLoad(element.Attribute(featuresRequiredAttributeName)?.Value,
element.Attribute(featuresForbiddenAttributeName)?.Value,
out result);
}

private static bool CanLoad(string requiredAttribute, string forbiddenAttribute, out FeatureFilter<TFeature> result) {

result = null;

if (!string.IsNullOrEmpty(forbiddenAttribute)) {
foreach (var featureName in forbiddenAttribute.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries)) {
try {
Enum.Parse(typeof(TFeature), featureName);
return false;
}
catch {
}
}
}
var filter = new Dictionary<TFeature, FeatureStatus>();
if (!string.IsNullOrEmpty(requiredAttribute)) {
foreach (var featureName in requiredAttribute.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries)) {
try {
filter[(TFeature)Enum.Parse(typeof(TFeature), featureName)] = FeatureStatus.Required;
}
catch {
return false;
}
}
}
result = new FeatureFilter<TFeature>(filter);
return true;
}

public FeatureFilter() => filter = new Dictionary<TFeature, FeatureStatus>();

public FeatureFilter(IEnumerable<TFeature> required, IEnumerable<TFeature> forbidden)
: this()
{

if (required != null) {
if (forbidden != null) {
IEnumerable<TFeature> conflictingFeatures = required.Intersect(forbidden);
if (conflictingFeatures.Any()) {
throw new ArgumentException($"{conflictingFeatures.Count()} features including {conflictingFeatures.First()} were found in both the required and forbidden collections");
}
}

foreach (var f in required) {
Require(f);
}
}

if (forbidden != null) {
foreach (var f in forbidden) {
Forbid(f);
}
}
}

private FeatureFilter(Dictionary<TFeature, FeatureStatus> filter) => this.filter = filter;

public void Clear() => filter.Clear();

public void Require(TFeature feature) => filter[feature] = FeatureStatus.Required;

public void Forbid(TFeature feature) => filter[feature] = FeatureStatus.Forbidden;

public void Disregard(TFeature feature) => filter.Remove(feature);

public bool IsRequired(TFeature feature) {
return filter.TryGetValue(feature, out var status)
&& status == FeatureStatus.Required;
}

public bool IsForbidden(TFeature feature) {
return filter.TryGetValue(feature, out var status)
&& status == FeatureStatus.Forbidden;
}

public bool IsAnyRequired() => filter.ContainsValue(FeatureStatus.Required);

public bool IsAnyForbidden() => filter.ContainsValue(FeatureStatus.Forbidden);

private IEnumerable<TFeature> Enumerate(FeatureStatus status)
=> filter.Where(e => e.Value == status).Select(e => e.Key).OrderByDescending(f => f);

public IEnumerable<TFeature> GetRequired()
=> Enumerate(FeatureStatus.Required);

public IEnumerable<TFeature> GetForbidden()
=> Enumerate(FeatureStatus.Forbidden);

public ICollection<TFeature> GetRequiredCollection(bool readOnly)
=> new FeatureCollection(this, FeatureStatus.Required, readOnly);

public ICollection<TFeature> GetForbiddenCollection()
=> new FeatureCollection(this, FeatureStatus.Forbidden, true);

private class FeatureCollection : ICollection<TFeature> {
private readonly FeatureFilter<TFeature> featureFilter;
private readonly FeatureStatus collectionFeatureStatus;
private readonly bool isReadOnly;

public FeatureCollection(FeatureFilter<TFeature> featureFilter, FeatureStatus collectionFeatureStatus, bool isReadOnly) {
this.featureFilter = featureFilter;
this.collectionFeatureStatus = collectionFeatureStatus;
this.isReadOnly = isReadOnly;
}

public int Count => featureFilter.filter.Values.Where(v => v == collectionFeatureStatus).Count();

public bool IsReadOnly => isReadOnly;

public void Add(TFeature item) {
if (isReadOnly)
throw new NotSupportedException();
if (featureFilter.filter.TryGetValue(item, out var existingStatus) && existingStatus != collectionFeatureStatus)
throw new PersistenceException($"{item} could not be added to the required collection because it is already forbidden.");
featureFilter.filter[item] = collectionFeatureStatus;
}

public void Clear() {
if (isReadOnly)
throw new NotSupportedException();
foreach (var e in featureFilter.filter.Where(e => e.Value == collectionFeatureStatus).ToArray())
featureFilter.filter.Remove(e.Key);
}

public bool Contains(TFeature item) {
return featureFilter.filter.TryGetValue(item, out var status)
&& status == collectionFeatureStatus;
}

public void CopyTo(TFeature[] array, int arrayIndex) {
throw new NotImplementedException();
}

public IEnumerator<TFeature> GetEnumerator()
=> featureFilter.Enumerate(collectionFeatureStatus).GetEnumerator();

public bool Remove(TFeature item) {
if (isReadOnly)
throw new NotSupportedException();

return featureFilter.filter.TryGetValue(item, out var status)
&& status == collectionFeatureStatus
&& featureFilter.filter.Remove(item);
}

IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

public object[] GetAttributes() {
var result = new ArrayList();

if (IsAnyRequired()) {
result.Add(new XAttribute(featuresRequiredAttributeName,
string.Join(" ", GetRequired().Select(f => f.ToString()).ToArray())));
}

if (IsAnyForbidden()) {
result.Add(new XAttribute(featuresForbiddenAttributeName,
string.Join(" ", GetForbidden().Select(f => f.ToString()).ToArray())));
}

return result.ToArray();
}
}
}
11 changes: 11 additions & 0 deletions TLM/TLM/Persistence/GlobalPersistence.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace TrafficManager.Persistence {
internal static class GlobalPersistence {

public static List<IPersistentObject> PersistentObjects { get; } = new List<IPersistentObject>();
}
}
22 changes: 22 additions & 0 deletions TLM/TLM/Persistence/IPersistentObject.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Linq;

namespace TrafficManager.Persistence {
internal interface IPersistentObject : IComparable, IComparable<IPersistentObject> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of new code has no xmldoc comments. Is this some copy pasted pattern code?
Not everyone knows the meaning of this new added stuff, me included, a lot of magical new interfaces and fields and properties, no idea what they all do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will do this. (It's not copy/pasted code, it's all new.)

I just felt a little urgency to get people's eyes on this, because @krzychu124 is waiting on it for Airports DLC enhancements.

btw, I will probably be cleaning up this framework a little more and publishing it separately as an open-source library, which I'll then use in other mod work. Whether TMPE switches over to that or continues to use its own version, is a decision that can be made when the time comes.


Type DependencyTarget { get; }

IEnumerable<Type> GetDependencies();

XName ElementName { get; }

bool CanLoad(XElement element);

PersistenceResult LoadData(XElement element, PersistenceContext context);

PersistenceResult SaveData(XElement container, PersistenceContext context);
}
}
12 changes: 12 additions & 0 deletions TLM/TLM/Persistence/PersistenceContext.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Linq;

namespace TrafficManager.Persistence {
internal class PersistenceContext {

public int Version { get; set; }
}
}
11 changes: 11 additions & 0 deletions TLM/TLM/Persistence/PersistenceException.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace TrafficManager.Persistence {
internal class PersistenceException : Exception {
public PersistenceException(string message)
: base(message) { }
}
}
Loading