JSON format as an alternative to Markdown

[osaxma]

This is a general idea that I'd like to discuss and hopefully get some feedback on it. I'm currently experimenting with it in an app.

Motivation

I find Markdown limiting as a mean of storage. For instance, if an app needs to store a BlurHash of an image or metadata about a URL, they cannot be easily embedded in markdown without some clever tricks or using HTML (even then, there could be limitations and it would introduce extra complexity to the parsing process).

Idea

Using pure JSON as a format to store SuperEditor's Document(s). The closest example to such approach would be Contentful's Rich Text:

Rich Text is a field type that enables authors to create rich text content, similar to traditional "What you see is what you get" (wysiwyg) editors. The key difference here is that the Contentful Rich Text field response is returned as pure JSON rather than HTML.

Example

The following is an experimental JSON format that I've been using to store the document. As one can observe, the JSON structure resembles the structure of SuperEditor's Document. For nodes with AttributedText, each attribution is stored as a key where the value is a list of ranges indicating where that attribution is applied.

{
    "version": "0.0.1",
    "nodes": [
        {
            "node": "paragraph",
            "text": {
                "text": "Example Document"
            },
            "metadata": {
                "blockType": "header1",
                "textAlign": "left"
            }
        },
        {
            "node": "blockQuote",
            "text": {
                "text": "Example Document"
            },
            "metadata": {
                "textAlign": "left",
                "blockType": "header3"
            }
        },
        {
            "node": "horizontalRule"
        },
        {
            "node": "paragraph",
            "text": {
                "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus sed sagittis urna. Aenean mattis ante justo, quis sollicitudin metus interdum id. Aenean ornare urna ac enim consequat mollis. In aliquet convallis efficitur. Phasellus convallis purus in fringilla scelerisque. Ut ac orci a turpis egestas lobortis. Morbi aliquam dapibus sem, vitae sodales arcu ultrices eu. Duis vulputate mauris quam, eleifend pulvinar quam blandit eget.",
                "attributions": {
                    "bold": [
                        [
                            0,
                            147
                        ],
                        [
                            315,
                            438
                        ]
                    ],
                    "italics": [
                        [
                            149,
                            223
                        ],
                        [
                            315,
                            438
                        ]
                    ],
                    "strikethrough": [
                        [
                            225,
                            313
                        ],
                        [
                            315,
                            438
                        ]
                    ]
                }
            },
            "metadata": {
                "textAlign": "left",
                "blockType": "header3"
            }
        },
        {
            "node": "listItem",
            "text": {
                "text": "This is an unordered list item"
            },
            "metadata": {
                "textAlign": "left",
                "blockType": "header3"
            },
            "indent": 0,
            "itemType": "unordered"
        },
        {
            "node": "paragraph",
            "text": {
                "text": "Cras vitae sodales nisi. Vivamus dignissim vel purus vel aliquet. Sed viverra diam vel nisi rhoncus pharetra. Donec gravida ut ligula euismod pharetra. Etiam sed urna scelerisque, efficitur mauris vel, semper arcu. Nullam sed vehicula sapien. Donec id tellus volutpat, eleifend nulla eget, rutrum mauris."
            },
            "metadata": {
                "textAlign": "left",
                "blockType": "header3"
            }
        },
        {
            "node": "listItem",
            "text": {
                "text": "First thing to do"
            },
            "metadata": {
                "textAlign": "left",
                "blockType": "header3"
            },
            "indent": 0,
            "itemType": "ordered"
        },
        {
            "node": "paragraph",
            "text": {
                "text": "Nam hendrerit vitae elit ut placerat. Maecenas nec congue neque. Fusce eget tortor pulvinar, cursus neque vitae, sagittis lectus. Duis mollis libero eu scelerisque ullamcorper. Pellentesque eleifend arcu nec augue molestie, at iaculis dui rutrum. Etiam lobortis magna at magna pellentesque ornare. Sed accumsan, libero vel porta molestie, tortor lorem eleifend ante, at egestas leo felis sed nunc. Quisque mi neque, molestie vel dolor a, eleifend tempor odio."
            },
            "metadata": {
                "blockType": "header2",
                "textAlign": "left"
            }
        }
    ]
}

Discussion

So far, I haven't seen any drawback of using pure JSON to represent the document instead of Markdown. The only thing is that the document size could be larger, but I found that to be insignificant.

The main advantage is being able to easily store additional information specific to each node or the document as a whole. I also assume such format would facilitate the implementation of a collaborative editing feature (I haven't looked much into this tbh).

Note: I don't have the implementation in a public repo yet, but I simply created extension methods for json-serialization/deserialization for all the nodes, attributed text and up to the document itself (i.e., Document.fromJson and Document.toJson).

Please let know what do you think about the idea. Any suggestions would help.

WIP -- I wrote this a bit quickly, I'll polish the idea later.

[Jethro87]

I'm also interested in toJson / fromJson capabilities. In my own explorations into mobile text editing (prior to learning about super_editor), I had come up with a storage format similar to @osaxma's. I got inspiration from how Notion structures their data.

If more advanced nodes (e.g., checkboxes, radio buttons, complex groups) and node capabilities (e.g., references to own/other nodes) are developed, I think the case for not storing in markdown becomes stronger.

[Jethro87]

@osaxma I'm currently beginning to implement a json serializer in my project, and I'm curious if you have your implementation in a public repo I could reference.

[osaxma]

@Jethro87

I don't have anything in a public repo but I'll extract some code from a proof of concept that I had before. I believe it should still work -- maybe with some modification. I also believe some parts can be improved because I haven't changed much of it once I got it to work.

The following are 5 steps that I took to make a MutableDocument serializable:

1. Create a custom document

   This will simply allow calling toJson or fromJson for an entire document. This may also be done using an extension method over MutableDocument. I created a custom document because I had more properties (not shown here) such as title and whatnot.

   class MyCustomDocument extends MutableDocument {
     // Having a version is useful for handling breaking changes in case they occur 
     static const documentVersion = '0.0.1';
   
     MyCustomDocument({List<DocumentNode>? nodes}) : super(nodes: nodes);
   
     List<Map<String, dynamic>> _serializeNodes() {
       // `toJson()` here is from an extension method over 'DocumentNode' which is shown below
       return nodes.map((node) => node.toJson()).toList(growable: false);
     }
   
     Map<String, dynamic> toJson() {
       return {
         'version': documentVersion,
         'nodes': _serializeNodes(),
       };
     }
   
     static MyCustomDocument fromJson(Map<String, dynamic> map) {
       final newNodes = List.generate(map['nodes'].length, (index) {
         return DocumentNodeEx.fromJson(map['nodes'][index])!;
       });
   
       return MyCustomDocument(
         nodes: newNodes,
       );
     }
   
   }

2. Add Extension to DocumentNode
   During de/serialization, we only know that we have a List<DocumentNode>. We need such nested if-statements or similar approach to de/serialize the correct DocumentNode. Also given that we are limited to extension methods when it comes to the default nodes (e.g. to have ParagraphNode.toJson or ListItemNode.fromJson), this was the easiest approach that I came with at the time:

   // See discussion here if this looks weird:
   // https://www.reddit.com/r/dartlang/comments/msca7o/why_extension_methods_are_not_called_based_on/
   extension DocumentNodeEx on DocumentNode {
    
     Map<String, dynamic> toJson() {
       // NOTE: THE ORDER HERE MATTER! 
       //             e.g. a `BlockQuoteNode` is also a `ParagraphNode` but not vice versa
       //              hence `BlockQuoteNode` must be placed before `ParagraphNode` and so on. 
       if (this is BlockQuoteNode) {
         return BlockQuoteNodeEx(this as BlockQuoteNode).toJson(); // `toJson` is defined in step 4 for each DocumentNode
       }
   
       if (this is ListItemNode) {
         return ListItemNodeEx(this as ListItemNode).toJson();
       }
   
       // must be below BlockQuoteNode since both are ParagraphNode
       if (this is ParagraphNode) {
         return ParagraphNodeEx(this as ParagraphNode).toJson();
       }
      
       // This is a custom node
       if (this is ImageAssetNode) {
         return ImageAssetNodeEx(this as ImageAssetNode).toJson();
       }
   
       if (this is HorizontalRuleNode) {
         return HorizontalRuleNodeEx(this as HorizontalRuleNode).toJson();
       }
   
       // this is also a custom node 
       if (this is LinkNode) {
         return LinkNodeEx(this as LinkNode).toJson();
       }
   
      //  You must return something here to satisfy the return type.
      //  Since this is a serialization step, the user should only able to insert nodes that available to them.
      //  Tests should cover that each type of node is serializable. 
      //
      //  On other hand, during deserialization, it's important to return something meaningful (empty node or plain text)
      //  especially when the user is running an older version but a new node was introduced which wouldn't be available to them.
       throw UnimplementedError();
     }
   
     DocumentNode copyWith() => throw UnimplementedError();
   
     static DocumentNode? fromJson(Map<String, dynamic> map) {
       final nodeType = map['node'];
   
       if (nodeType == null) {
         return null;
       }
       // This can be improved to be strongly typed using enums, for example. 
       switch (nodeType) {
         case 'imageAsset':
           return ImageAssetNodeEx.fromJson(map);
         case 'paragraph':
           return ParagraphNodeEx.fromJson(map);
         case 'listItem':
           return ListItemNodeEx.fromJson(map);
         case 'blockQuote':
           return BlockQuoteNodeEx.fromJson(map);
         case 'horizontalRule':
           return HorizontalRuleNodeEx.fromJson(map);
         case 'link':
           return LinkNodeEx.fromJson(map);
         default:
           // This can be handled externally to show something meaningful (e.g. new node not available in the version used by the user).
           return null;
       }
     }
   }

3. Serialize Attributed Text

   /// For reference from `AttributedText` documentation:
   /// ------------------------------------------------------
   /// Bold    :  {xxxx}                      {xxxxx}
   /// Italics :             {xxxxxxxx}
   /// Link    :                              {xxxxx}
   /// ------------------------------------------------------
   extension AttributedTextEx on AttributedText {
     // this will output a map that contains attributions as keys and their spans as values where the values are List of
     // ranges (i.e., List<int>) so the output will be like this:
     // {strikethrough: [[7, 182]], italics: [[7, 182]], bold: [[7, 182], [267, 275], [415, 425]]}
     Map<String, dynamic> toJson() {
       final attributions = <String, List<List<int>>>{};
       for (var span in spans.collapseSpans(contentLength: text.length)) {
         for (var attr in span.attributions) {
           final ranges = attributions.putIfAbsent(attr.id, () => <List<int>>[]);
           ranges.add([span.start, span.end]);
         }
       }
       return {
         'text': text,
         if (attributions.isNotEmpty) 'attributions': attributions,
       };
     }
   
     static AttributedText fromJson(Map<String, dynamic> map) {
       final text = AttributedText(text: map['text'] ?? '');
       if (text.text.isEmpty || map['attributions'] == null) {
         return text;
       }
   
       try {
         // casting as Map<String, List<List<int>>> does not work here -- it'll throw:
         // `error while parsing attributions type '_InternalLinkedHashMap<String, dynamic>' is not a subtype
         // of type 'Map<String, List<List<int>>>' in type cast
         //
         // The worst case scenario is that the attributed style is lost and a plain text is shown instead
         final attrs = map['attributions'];
         attrs.forEach((key, value) {
           for (var range in value) {
             final attr = attributionFromJson(key);
             if (attr == null) {
               continue;
             }
             text.addAttribution(attr, SpanRange(start: range.first, end: range.last));
           }
         });
         return text;
       } catch (e) {
         log('error while parsing attributions $e');
         // return plain AttributedText in case of an error
         return text;
       }
     }
   }
   
   // this is from `super_editor/lib/src/default_editor/attributions.dart` 
   const supportedAttributions = [
     boldAttribution,
     strikethroughAttribution,
     italicsAttribution,
   ];
   
   Attribution? attributionFromJson(dynamic value) {
     if (value is! String) {
       return null;
     }
     return supportedAttributions.firstWhereOrNull((element) => element.id == value);
   }

4. Add Extension to each node
   As mentioned earlier, for the default nodes of SuperEditor, extension methods seems to be an easy approach. For custom nodes, you can add your serialization directly to the class itself. The following are some examples:

   extension ParagraphNodeEx on ParagraphNode {
     ParagraphNode copyWith({
       String? id,
       AttributedText? text,
       Map<String, dynamic>? metadata,
     }) {
       return ParagraphNode(
         id: id ?? this.id,
         text: text ?? this.text,
         metadata: metadata ?? this.metadata,
       );
     }
   
     Map<String, dynamic> toJson() {
       return {
         'nodeId': id,
         'node': 'paragraph',
         'text': text.toJson(),
         if (metadata.isNotEmpty) 'metadata': metaDatatoJson(metadata),
       };
     }
   
     static ParagraphNode fromJson(Map<String, dynamic> map) {
       return ParagraphNode(
         id: map['id'] ?? generateId(),
         text: AttributedTextEx.fromJson(map['text']),
         metadata: metaDatafromJson(map['metadata']),
       );
     }
   }
   
   extension BlockQuoteNodeEx on BlockQuoteNode {
     BlockQuoteNode copyWith({
       String? id,
       AttributedText? text,
       Map<String, dynamic>? metadata,
     }) {
       return BlockQuoteNode(
         id: id ?? this.id,
         text: text ?? this.text,
         metadata: metadata ?? this.metadata,
       );
     }
   
     Map<String, dynamic> toJson() {
       return {
         'nodeId': id,
         'node': 'blockQuote',
         'text': text.toJson(),
         if (metadata.isNotEmpty) 'metadata': metaDatatoJson(metadata),
       };
     }
   
     static BlockQuoteNode fromJson(Map<String, dynamic> map) {
       return BlockQuoteNode(
         id: map['id'] ?? generateId(),
         text: AttributedTextEx.fromJson(map['text']),
         metadata: metaDatafromJson(map['metadata']),
       );
     }
   }
   
   extension ListItemNodeEx on ListItemNode {
     ListItemNode copyWith({
       String? id,
       AttributedText? text,
       ListItemType? type,
       Map<String, dynamic>? metadata,
       int? indent,
     }) {
       return ListItemNode(
         id: id ?? this.id,
         text: text ?? this.text,
         itemType: type ?? this.type,
         indent: indent ?? this.indent,
         metadata: metadata ?? this.metadata,
       );
     }
   
     Map<String, dynamic> toJson() {
       return {
         'nodeId': id,
         'node': 'listItem',
         'text': text.toJson(),
         if (metadata.isNotEmpty) 'metadata': metaDatatoJson(metadata),
         'indent': indent,
         'itemType': type == ListItemType.ordered ? 'ordered' : 'unordered',
       };
     }
   
     static ListItemNode fromJson(Map<String, dynamic> map) {
       return ListItemNode(
         id: map['id'] ?? generateId(),
         text: AttributedTextEx.fromJson(map['text']),
         metadata: metaDatafromJson(map['metadata']),
         indent: map['indent'] ?? 0,
         itemType: map['itemType'] == 'ordered' ? ListItemType.ordered : ListItemType.unordered,
       );
     }
   }

5. Serialize metadata

   Map<String, dynamic>? metaDataFromJson(dynamic map) {
     if (map == null) return map;
     if (map is Map) {
       if (map.containsKey('blockType')) {
         // the provided map can be immutable, so we need to create our own copy.
         map = Map<String, dynamic>.from(map);
         map['blockType'] = NamedAttribution(map['blockType'] ?? '');
       }
     }
     return map;
   }
   
   Map<String, dynamic> metaDataToJson(dynamic metaData) {
     if (metaData == null || metaData.isEmpty) return const {};
   
     return (metaData as Map).map(
       (key, value) => value is NamedAttribution ? MapEntry(key, value.id) : MapEntry(key, value),
     );
   }

---

I hope you and others find this helpful.

p.s. sorry if something does not make sense above since I extracted this code and edited manually here.

[Jethro87]

Thanks so much for sharing @osaxma! That's really helpful. Much appreciated.

[Dboy233]

I have developed a JSON conversion library called 'super_editor_json' on my own. After seeing everyone's discussion, I feel that my program design is somewhat primitive. Where can I find publicly available source code for JSON conversion? I would like to learn from it.