Discussion: What data formats / structure best facilitate collaborative documentation

[DominicWatson]

This is more of a space for discussion / input gathering than an "issue" as such. What I think the considerations are:

• One source of documentation
• Ability to export, with as little effort as possible, to multiple formats (PDF, HTML, Dash, ePub, MOBI, raw JSON...)
• Ability to import from Lucee source to build the reference base of the docs
• Ability for contributers to easily contribute to the wording and code samples of the documentation, these edits should not be overwritten by reference imports

I already kicked off my own thoughts on here https://github.com/DominicWatson/luceedocsbuilder but there wasn't really a sensible space for feedback, hence this GH issue.

My preference is for tags and functions to have their own editorial source folder. In it, could be a number of things:

/editorial
    /functions
        /abs
            /examples
            /arguments
            description.md
            references.json
            README.md <-- auto built contribution guidelines describing the structure of the folder and where to look to edit stuff

I'm in two minds re "description.md", and I understand your preference for a .json source (that could contain both description of the function/tag + its arguments + other things).

Either way, I think the folder approach works well as it makes potential future enhancements easier to manage (thinking of multiple contexts of tags and functions for example).

Feel free to close if you think there is a better space for this.

D

[cybersonic]

This space is fine for this.
I think we should think of this as a more generic documentation format. I like the folder approach and we should remember that there are two types of data:

• structured data
• textual data

Structured data can be versions (as the root) with various DocBase instances. It can also be a class for example, or a function, or a tag or whatever. This base metadata is required for filtering, tagging (such as deprecated marks) etc.

Textual data is stuff like descriptions, be it if arguments/attributes, and maybe anything. So I would say that the JSON all it needs to do is have some way of saying that the description should be included from a markdown file and that is what should be edited in an editor (in comparison to editing the other data in a simple field)

[DominicWatson]

100% agreed.

In terms of the output, a full JSON based output per version could be generated by combining the imported reference and editorial content. This would not need to be human editble friendly.

[cybersonic]

Right, so now to the "DocBase" format as we shall call it (unless other votes come in?)

/Documents
/versions.json
/versionx
/versiony

Agreed?

[cybersonic]

With each version you can have sections, so you have a index.json which has metadata about the category and an an array of links to sub categories (since you might want to sort your sections differently than alphabetical.
If there is no index.json, you would get just a list of stuff.

In our case we would have:

index.json 
/language 
/tags
/functions
/objects
/components

Within each section you can have sub sections

An example JSON for the above would be
{
name: "Lucee Documentation",
description: "include:readme.md",
version: "4.5.0.1"
sections: [
"language ",
"tags",
"functions",
"objects",
"components"
]
}

notice the include:readme.me ? This is the pattern to say what you want to use in a section as a long text form

[cybersonic]

This gives great flexibility as you can have sections within sections, you can organise sub sections, you have an easy way to parse stuff and you have a generic way to include (using include:readme.md) longer text descriptions in things such as arguments and attribute descriptions.

[cybersonic]

This also leaves it open for longer formats, such as one requirement to create generic docs, that are not tied to metadata. So you just have a index.md as the root and any sections underneath.

[DominicWatson]

Nice. With you.

Another consideration is how cross referencing can work. So in a markdown file for description or whatever else, we might want to do something like:

See also {{reference:/language/templating/conditionalLogic}}

Or

See also {{reference:tag:abort}}

Etc. It would have to be up to each individual docs builder to render those references (the output could/would be entirely different for each).

Would be interesting to see what other documentation systems are using syntax wise for cross referencing. I've been using ReadTheDocs for preside which uses ReStructuredText as a format. Not sure I like it per se, but does do referencing pretty well.

[cybersonic]

I think I get you. I like the {{reference:}} format. I am wondering if the version becomes a "magic" thing... so in your docs if you do {{reference:tag:abort}} it would automatically be the current tag/abort, 4.5.0.1/tag/abort but if you put reference:4.3.0.0:tag:abort it would go to that specific version.

Edit: I think reference should essentially create a url of /section/section/section... so that is how you reference stuff. The latest version will be the current version and therefore no need for a path.

To explain myself better:

{{reference:a:b:c}} --> http://website.com/a/b/c (using the current version)
{{reference:4.3.2.1:a:b:c}} - > http://website.com/4.3.2.1/a/b/c

Also now thinking about it, the include should be {{include:readme.md}} so to keep the functions similar.

[DominicWatson]

Absolutely - the build system should make the version that is being built available as a variable, and reference renderers could get the path to the reference passed to it pre-built with the version in it. So a particular docs builder would implement:

// path might look like: /latest/tags/abort, or /4.5.1/tags/abort, or /4.5.1/language/components/syntax
public string function renderReference( required string path ) {
    return _myRenderLogic( arguments.path ):
}

[cybersonic]

Makes sense so far. I think making this a simple format to generate and to be modular means we can
a) build different renderers in different languages as long as they read JSON and MD
b) for Luceedocs, separate the content from the presentation engine.
c) allow for other editors, i.e Github to be the markdown editor for this
d) allow for NEW editors i.e. build one to create these doc formats.

we should see if there are other doc formats out there but I am guessing there are not ?

[DominicWatson]

http://sphinx-doc.org/ is what readthedocs use. It's a little too gnarly though IMO for general use. Python based and well, gnarly.

It has good concepts though and build systems built in for HTML, ePub, etc. Theming it is gnarly and there don't appear to be any decent themes out there.

Stealing concepts and building something using the language that is being documented I think is a positive move (so long as done a little at a time so as not to be overwhelming).

[DominicWatson]

Here's what we're doing for docs in Preside. It has a mixture of auto generated documentation (built using a hacky cfm build script triggered using CommandBox), and free form documentation:

https://github.com/pixl8/Preside-CMS/tree/master/support/docs

[DominicWatson]

There are massive benefits to using ReadTheDocs if you can get over the initial RST and theming barriers.

• Versioning is awesome and can run simply off branches in your repo.
• The docs get built for you when you commit (including PDF, ePub)
• They host the docs for you. For free. (and you can use custom domains with CNAME records to point to it)

Perhaps we'd be insane not to be using this? IDK. Struggle between losing ultimate control + free swag.

I could certainly have a go at exporting to their format so that I could demo it. Should not take much.

[cybersonic]

I did look into this but the theming was all over the place and I wasted far too much time with Sphinx that it made me hate it.

Mark Drew

• Sent by typing with my thumbs.

On 27 Feb 2015, at 11:55, Dominic Watson [email protected] wrote:

There are massive benefits to using ReadTheDocs if you can get over the initial RST and theming barriers.

Versioning is awesome and can run simply off branches in your repo.
The docs get built for you when you commit (including PDF, ePub)
They host the docs for you. For free. (and you can use custom domains with CNAME records to point to it)
Perhaps we'd be insane not to be using this? IDK.

—
Reply to this email directly or view it on GitHub.

[DominicWatson]

Yup ;)

I wonder can we learn from the things that are good about it instead. So imagine a folder structure like this:

/functions
    index.md
    bycategory.md
/tags
/language
     /templating
index.md

Those index.md files could then just be:

# I am the page title

Blah blah blah blah blah.

* {{reference:tag:abort}}
* ...

[cybersonic]

I do like the branches as versions feature though. I am sure the doc reader can do that

Mark Drew

• Sent by typing with my thumbs.

On 27 Feb 2015, at 11:55, Dominic Watson [email protected] wrote:

There are massive benefits to using ReadTheDocs if you can get over the initial RST and theming barriers.

Versioning is awesome and can run simply off branches in your repo.
The docs get built for you when you commit (including PDF, ePub)
They host the docs for you. For free. (and you can use custom domains with CNAME records to point to it)
Perhaps we'd be insane not to be using this? IDK.

—
Reply to this email directly or view it on GitHub.

[DominicWatson]

I do like the branches as versions feature though. I am sure the doc reader can do that

Agreed.

[cybersonic]

So, if we forget about versions, we create the format, say that it is in git and then luceedocs (the reader) just checks it out and generates the docs for each branch that would be good enough.
There has to be a model (how I see it in my head) that would parse all the json files, create a big index of content and then output it in the right format using the templates.

[cybersonic]

Here is my first pass at converting tags and functions to the new model, now I need to do that for object functions (member functions)
https://github.com/cybersonic/lucee-docpack/tree/4.5.1.000

[DominicWatson]

Nice. Not sure this is quite right though:

https://github.com/cybersonic/lucee-docpack/blob/4.5.1.000/tags/continue/description.md

Reckon that should be the imported description in there...?

[cybersonic]

I did say it was a first pass

[nodoherty]

This would warrant its own thread for documentation / learning resources but...

Just a side note on best facilitating documentation resources for Lucee, especially code examples, but would it be worth implementing Koans?

I was involved in the below project with Ryan and believe it's a great resource for documenting & learning a language by writing unit tests. The code used for these tests could become part of the documentation to enrich the content.

Obviously testing framework, MXUnit / TestBox + MockBox TBC but thought I'd throw this in here to get your opinion.

See: https://github.com/bittersweetryan/ColdFusion-Koans

[DominicWatson]

+1 @nodoherty, though defo worth it's own issue / thread I think. Or better still, once the structure of the docs is clearer, some kinda pull request to have a Koans section of the docs. That would rock all kinds of awesome. Reckon Ryan would be up for porting that project over? I'd say it'd have to be testbox given that MXUnit has been taken over by Ortus and Testbox is the way of the future.

[DominicWatson]

@cybersonic re the doc-pack repo - loving the form its taking. I personally feel though that all the code should be in the one place. Import logic, source data + target build logic. This creates less of a barrier to entry for people wanting to contribute in different ways. So the form of the project might be like:

/data <--- under which all the doc-pack stuff lives
/build
    /import
    /export
        /html
        /pdf
    <!--- + other stuff for various bits of documentation building logic --->

Could create generic helpers / API for the building of targets and do the main site + at least one other (pdf?) to get people started.

That's my preference anyways.

[DominicWatson]

Actually, nomlicature-wise, either /docs or /docs-src would be better than /data.

[cybersonic]

Yeah, I think docs as a root and build as how you generate it automatically. I think if we get this right the way we present it will be more important than the way we generate it.

Another side note, I want to add gists as id's or user/id or something like that so that each page can have some mechanism to have gists and ids. So the the JSON should reflect that (since it's metadata)

[cybersonic]

@nodoherty That is a great idea but I think start a new thread :) maybe in the doc-pack repo? https://github.com/cybersonic/lucee-docpack/tree/4.5.1.000

[DominicWatson]

Yeah agreed on both points, the json format definitely useful for the easy expansion of the dataset:

{
    "argMax": -1,
    "keywords": [ "ajax" ],
    ...
    "gists" : [ "id1", "id2" ]
}

However, if everything is hosted in this manner on GitHub, will a Gist really be that much better than just referencing a file in the repo (keeping all the documentation in one place):

{
    "argMax": -1,
    "keywords": [ "ajax" ],
    ...
    "examples" : [ { title="Some example", body="{{include:examples/someExample.md}}" } ]
}

The advantage with having in the repo is that non web-based outputs of the documentation can more easily include the examples in their output. Advantage of gists is comments, syntax highlighted embed code and ability to directly run in TryCF and the like.

Jury is out for me.

[DominicWatson]

The index.json at the root of the docpack interests me:

{
    name: "Lucee Server Documentation",
    description: "{{include:README.md}}",
    version: "4.5.1.000",
    sections: [
        "tags",
        "functions",
        "objects"
    ]

}

This suggests a kind of tree based CMS who's data storage is these json files. A process that is taking this storage and outputting a finalized product, might do something like (very rough, and not the best nomlicature):

function processPage( page, depth ) {
     switch( page.type ){
         case "function": someFunctionSpecificProcessing( page, depth ); break;
         // ... etc.
     }

     for( var section in page.sections ?: [] ){
         processPage( getPage( section ), depth+1 );
     }
}

The file structure might then look like:

/functions
    /abs
        index.json
        description.md
        ...
    ...
    index.json
/tags
/objects
/language
index.json

So that index.json represents the root page of a given folder section. Having a "type":"function", or "type":"page", etc. entry in the index.json can then be used to inform the build.

Thoughts?

[DominicWatson]

Actually, it might better be thought of as:

public void function processTree( required DocTree tree ) {
    // here arguments.tree is a DocTree object provided by the generic build system and built from reading the json files
    // you could traverse it's nodes and do whatever recursion you like to build your documentation output
}

[nodoherty]

FYI on Koans - thanks @DominicWatson @cybersonic.

Agreed Dom - format / structure needs finalising before integrating this kind of content. I'll speak to Ryan about porting the Koans we've already written. Testbox FTW too.

As suggested Mark, I'll create a new issues in the doc-pack repo for the Koans stuff.

Sorry to clutter this thread but just an FYI that I'll also create an issue for a DashDocs / Zeal / DevDocs.io generator too so these awesome docs can go viral :-)

[DominicWatson]

Don't create an issue, create a pull request :p

[DominicWatson]

@cybersonic so I've been playing a bit more and have a working prototype of the concepts talked about.

A very rough static output of the docs can be found here: http://prototype-luceedocs.surge.sh/ <- should be available some time soon - CDN just 'propagating'

I've basically been playing with how the source might be laid out and how we could go about building the output docs (the page templates and layouts would need plenty more work). Stuff I think is cool:

• It can be for regular editorial pages, or indeed any structured or non-structured page types we can come up with (i.e. Koans pages...)
• Its written in Lucee, so the generators and output can more readily be editable by Lucee community members - might be good to use it to show off some Lucee 5 features
• {{include:somefile}} is implemented
• Cross referencing. Checkout the DateFormat() docs, in particular the 'mask' argument. This cross reference can be marked up anywhere simply with {{ref:/path/to/page}}, in this case, {{ref:/reference/misc/dateTimeMasks}} (see source: https://github.com/DominicWatson/luceedocsbuilder/blob/master/docs/reference/functions/dateformat/arguments/mask/description.md).

Stuff that's not there:

• vision for how versions would be handled, and how version aware builds could be triggered
• loads of specifics in tag/function pages
• docs

The repo is here if you're interested to see where it's at: https://github.com/DominicWatson/luceedocsbuilder/. Could be free to Skype or whatever, but only once sprogs are in bed ;)

[DominicWatson]

FYI, I've been continuing this discussion and scatting with the LAS - getting close to a point where we'll have an official repo in Bitbucket. I'd like to get to a point v soon where we have some sort of documentation committee who can work together on the docs and can be its guardians. Most importantly, it needs a strong + official lead first.

The current state of the stuff I've been doing is taking it's lead from "Grav", a php based static file CMS that uses Markdown. What's good about it is that it uses very familiar concepts to most contributers. So pages are just markdown files with any meta data being driven by a little YAML at the top of the file. Reference metadata is coming directly from source. See the docs source here: https://github.com/DominicWatson/luceedocsbuilder/tree/master/docs

[DominicWatson]

There is now an official repo on BitBucket (https://bitbucket.org/lucee/documentation) which I think is important. Given that it is significantly based on your large efforts, I feel a bit cuntish working in a slightly separate direction and under the 'Lucee' name rather than your own. Would be great to have you involved if you're up for it.