Slugification of wikilinks

[tusharjoshi]

• Foam version: v0.10.0
• Platform: Mac (Big Sur 11.2.1)
• Issue occur on the foam template repo: Yes

Summary
Foam Graph shows two entries for same note. One entry is shown orphan which is active. The actual link is shown as inactive.

Steps to reproduce

1. I created a daily journal entry, and types a wiki link to create it
2. Click on the wiki link to create the desired page
3. As the page got created in same journal folder I moved it to root folder

Additional information
I have attached a demo project with only the actions I have taken and the foam graph has that issue
The project started from Foam Template and I have added some extension suggestions which I am already using

foam-graph-test.zip

[gxingh]

Same issue faced by me.
The grey links on the graph are not clickable but the white and the grey links both are the same nodes.

Is the issue that we can't have spaces while creating a new note?
Like [[Redux Actions]] is invalid and [[Redux-Actions]] is valid?

[riccardoferretti]

This is happening because in 0.10.0 we are less lenient towards slugified links.
In the attached repo, the problem is that the file is called personal-knowledge-mastery.md, but the link is Personal Knowledge Mastery.
You can fix the issue in one of these two ways:

• rename the file Personal Knowledge Mastery.md
• rename the link personal-knowledge-mastery

Basically the two have to match.

The reason why they appear to match in the graph is that we show the note title (# Personal Knowledge Mastery) in there, if you remove the title from the note you will better see what's happening.

Can you confirm the same is true for the you @iamtherealgd ?

[zomars]

This is not working anymore tho

[riccardoferretti]

Sorry I am not sure what you are referring to here, can you explain what's not working?

[zomars]

[[This link]] would correctly link to this-link.md correctly in previous versions. Not anymore.

[tusharjoshi]

The comment above about the naming of the wikilink and the file solves my issue.
As the automatic file creation makes all characters small and introduces dash in place of space, I changed all my wikilink references to match the file names as they will be created by clicking on them.

This is issue is resolved at my end by fixing the names of the wikilink references. Before closing this issue documentation shall be made clear to explain this fact that the heading is shown in Graph view whereas the reference is to a file name hence the wikilink reference should match the file name not the title of the page.

[gxingh]

Can you confirm the same is true for the you @iamtherealgd ?

I think what I was doing wrong was the link was in capital case like [[Maintainable]] and its file was maintainable.md.

But I created the maintainable.md file by the shortcut cmd + click on the [[Maintainable]] link and it automatically created the file with lowercase name.
If i create the link like [[maintainable]] in lowercase and then cmd + click on it then everything works fine.

So Foam creates new files in lowercase irrespective of the case how it's mentioned in some other note, is there a way to change this?

[riccardoferretti]

that's a really good thought @iamtherealgd - I had not thought of that.

I think what I was doing wrong was the link was in capital case like [[Maintainable]] and its file was maintainable.md.

my take is that we can be lenient about letter case, even if the implicit tradeoff is that we don't support path/to/file.md and path/to/File.md - I want to think about it a bit more, but that's my first inclination

So Foam creates new files in lowercase irrespective of the case how it's mentioned in some other note, is there a way to change this?

currently the note creation from the link is managed by Markdown Notes, but we'll soon bring this in house for better compatibility

[pranphy]

You can fix the issue in one of these two ways:

rename the file Personal Knowledge Mastery.md
rename the link personal-knowledge-mastery

I prefer my filenames to not have space, and at the same time I prefer that the text in wikilink to not have hyphen. So the first option of having filenames with space is a big no for me. I think [[unhyphenated wiki links]] look much nicer than [[hyphenated-wiki-links]] in general and more so especially when the slugified filename is long.

@riccardoferretti reading your comments above I get an impression that this behaviour is undesirable. I was wondering what would be the argument against having [[wiki link with space]] link to wiki-link-with-space.md file?

[gxingh]

@riccardoferretti reading your comments above I get an impression that this behaviour is undesirable. I was wondering what would be the argument against having [[wiki link with space]] link to wiki-link-with-space.md file?

This is exactly the issue I'm facing.
I'm generally creating new notes using cmd+click/hover on a wiki link which is in capital case with spaces(I feel its more easily readable that way) and the file name it creates is lowercase with dashes, so the graph is off.

[tusharjoshi]

Today I created a wikilink [[active-recall]] exactly as expected by Foam and created the file by Cmd+Click making a file active-recall.md, the Foam Graph is still showing phantom node 'Active Recall' and connected node as 'active-recall' which is gray

[riccardoferretti]

@riccardoferretti reading your comments above I get an impression that this behaviour is undesirable. I was wondering what would be the argument against having [[wiki link with space]] link to wiki-link-with-space.md file?

Adding a level of indirection, a slugification function, between the text of the wikilink and the corresponding file name creates complexity for other features and use cases.

For example we are adding refactoring support, where wikilinks will be updated when a file is renamed.
If multiple wikilink strings can reference the same slug, (e.g. [[my file-name]], [[my-file name]], [[my-file-name]], all become my-file-name.md, [[asp.net]] and [[aspnet]] are slugified to aspent.md, and so on..) trying to de-slugify quickly becomes complex.

Also, slugification gets in the way of other use cases, e.g. you wouldn't be able to easily support spaces in file names: if you have two files, my-file.md and my file.md, the wikilink [[my file]] cannot be resolved, or my file.md would not be linkable.

An additional level of complexity is due to several slugification functions incompatible with each other (we have already dealt with this problem in Foam for markdown-notes). Slugification is also unfriendly towards non-latin alphabets.

Finally, there are workarounds that could be implemented to satisfy the use case without breaking the core contract, which admittedly require features that are in the works, but that are actually coming soon:

• using link aliases [[my-file|My File]] (we are adding this soon)
• using link definitions [[My File]] and [My File]: my-file.md (we already support this, but we don't have a feature to autogenerate a "slugified" definition for links, something that once the API is released (very soon) will be possible for a third party extension leveraging the Foam model)

[riccardoferretti]

@tusharjoshi Today I created a wikilink [[active-recall]] exactly as expected by Foam and created the file by Cmd+Click making a file active-recall.md, the Foam Graph is still showing phantom node 'Active Recall' and connected node as 'active-recall' which is gray

This seems like a different issue, can you please create a reproducible scenario and file an issue for it?

[riccardoferretti]

@iamtherealgd can you also please create another issue for your problem? I think that's something we need to properly support and would like to untangle it from this issue

[krry]

Tangentially, having had similar experiences to many of yall's, I have found joy in massren combing through my filenames, making-them-all-nice-and-uniform.md

[keikhcheung]

If multiple wikilink strings can reference the same slug, (e.g. [[my file-name]], [[my-file name]], [[my-file-name]], all become my-file-name.md, [[asp.net]] and [[aspnet]] are slugified to aspent.md, and so on..) trying to de-slugify quickly becomes complex.

Also, slugification gets in the way of other use cases, e.g. you wouldn't be able to easily support spaces in file names: if you have two files, my-file.md and my file.md, the wikilink [[my file]] cannot be resolved, or my file.md would not be linkable.

@riccardoferretti It sounds like there are a lot of assumptions on what is good for how people should name their files here. I wonder how often the combination of cases mentioned here happens.

Given there are people who would like to just have Foam work with their files of which spaces exist in file names, would it not be more preferable to leave an option for the user to set whether slugification should be turned on or not in Foam core, instead of putting everyone to a one-size-fit-all solution?

[riccardoferretti]

We could make it customizable, but this would add complexity and maintenance cost.
We need to decide what to prioritize from a tactical (how many people care about this? what are the workarounds? ...) and strategic (how does this impact the development? how do we support this feature? ...) POV.

I am not against your suggestion, I feel though that there are other tasks with higher priority, and that this change has the potential of making the development of other features more complex.
Also, there are viable workarounds (from naming conventions, to wikilink alias/references, to using markdown-notes instead) for users that feel strongly about this convention.

For those reasons we are not considering this work ATM. If there were a lot of requests for it I am open to reconsideration - but right now the cost/complexity of adding support for it is not worth the candle IMO.

[njnygaard]

The vscodeMarkdownNotes.newNote functionality automatically turns the title New Note in to new-note.md. I have been using that functionality for a while now. As that is a recommended extension, it makes sense to provide slugification. I stumbled on this thread because I was finally getting arount to configuring templates in Foam and realized that the Title I provided would generate a file in the same case. I would like the ability to match the recommended 'new note' format.

[zomars]

The problem is that markdown notes conflicts with foam as described in #1023

[riccardoferretti]

Thanks for your comment @njnygaard, it doesn't change my thoughts above but it is a good data point.

We have had a few conversations here and in Discord about the slugification of wikilinks and we might end up supporting it again in the future, it's just not a big priority now

[vJourneyman]

Is it intended that Foam works best with slugified lower case, but Foam: Create New Note calls for a Title Case Name for the note, which is explicitly then not lower-cased and slugified?

I've been doing a lot of "Foam: Create New Note" and then having to rename the file.

[riccardoferretti]

Is it intended that Foam works best with slugified lower case

I am not sure what you mean by "works best". it supports it, it's not tailored for it.
One thing I was thinking, but might require additional thoughts as well, is to support a FOAM_TITLE_SLUG variable in templates to better support your use case.

@movermeyer, what do you think of that? I believe it is backwards compatible and doesn't add regressions.
Of course, referencing such note from another one would require using the [[title-slug]], as the wikilink needs to match the filename

[vJourneyman]

I am not sure what you mean by "works best". it supports it, it's not tailored for it.

In my installation, if I create a note/link like [[Three Word Link]], GitHub has problems rendering it.

Maybe the issue is that there's a mix of support from rendering engines for links like [[three-word-link|Three Word Link]] to use slugified names while making the link names look nice when rendered by a git engine.

[lucidBrot]

I sadly don't have the spoons to reproduce with a small example, but I think I'm experiencing the same thing as @vJourneyman mentioned.

I have "imported" a few markdown files (that originally came from OneNote, using an export script) and these files have filenames that contain multiple dashes, dots, apastrophes, and I understand somewhat when they make problems. But even removing them does not seem to help - there still is every node twice in the Foam Graph except for a few. However, it seems that renaming the file to use lowercase-only naming fixes this. At least after deleting and recreating the wikilink.

Notably, the "duplicates appearing in the Foam Graph" problem does not happen on my ubuntu laptop with the files that cause problems synced via syncthing. So it miiight have to do with Windows/NTFS?

Steps for Repairing a Node that appears twice in the Graph

1. Figure out where the wikilinks to it are. In my case, they are luckily only in one single file.
2. Rename the file who is shown duplicated in the graph to use only lowercase characters and dashes, apart from the .md ending. E.g. I renamed Reformulate.md to reformulate.md by selecting it in the explorer panel and pressing F2.
3. change the wikilink that links to this from [[Reformulate]] to [[reformulate]] and save the file
4. The graph instantly after saving adapts. Repeat for all your duplicated Nodes.

I'm automating the renaming for all files (not recursive) in the folder using

# in cmd
for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f")

(explanation here)

and to change the wikilinks to lowercase in the file:

# in cygwin
sed 's/\(\[\[[a-zA-Z-]*\]\]\)/\L\1/g' my-linking-file.md

but that did not seem to do the whole trick. While I first thought it fixed everything, after reopening vscode the duplicates are back.

But now it seems that the duplicates are using as path /n:/Files/syncthing/priv-notes/archive/my-folder/thefile.md and the correct nodes use relative paths. Maybe it is relevant that all these files are not in the project root, but in a subfolder?
The wikilinks that are generated sometimes look like that too, except with ../../../../../.. prefixed.

Now I closed vscode, removed all the wikilinks in Notepad++, reopened vscode, and the Foam graph looks beautiful and correct again - except for the fact that it shows wikilinks that are not there. As soon as I touch and save the file in vscode again, it regenerates the references and the dupe appears in the graph.

After removing the link references once again, but this time on the linux computer, and saving it in vscode again (on linux), then waiting for the sync and reopening vscode on the windows machine seems to have repaired the graph - it shows no dupes and no imaginary connections. However, just hitting ctrl+s in the file on windows is enough to recreate the dupes in the graph view.

[techCarpenter]

[...] One thing I was thinking, but might require additional thoughts as well, is to support a FOAM_TITLE_SLUG variable in templates to better support your use case.

I would appreciate support for a FOAM_TITLE_SLUG variable in the template frontmatter to use with filepath naming 🙌🏼

[riccardoferretti]

Was reading this thread again, and came across this comment.
Thanks again for contributing by adding support for FOAM_TITLE_SLUG (#865)!

[techCarpenter]

No problem, I'm glad I could contribute to the project!

[movermeyer]

I would appreciate support for a FOAM_TITLE_SLUG variable in the template frontmatter to use with filepath naming 🙌🏼

@techCarpenter #692

[riccardoferretti]

Here is a quick update on this:

• aliases (e.g. [[note-title|Note Title]]) are supported
• the FOAM_SLUG template variable is supported
• the alias property in the YAML is supported

[zomars]

Any docs on how does this works/will work?

[riccardoferretti]

This is an interesting point.
Given that several people have been asking about support for slugification, a doc on how to achieve something "as close as possible" with what Foam provides might be helpful. Is this what you are talking about?

Also, adding a reference to #1038 as it might actually be useful in this setup as well

[zomars]

I find it odd that this was previously the default (when foam relied on "markdown notes" for wiki-links). But now we need docs on how to restore previously working functionality.

[riccardoferretti]

After a conversation here I will post also in this discussion as people in it might find it useful.

Let's go back to the reasons why I decided to make the switch:

• slugification is not unique: If multiple wikilink strings can reference the same slug, (e.g. [[my file-name]], [[my-file name]], [[my-file-name]], all become my-file-name.md, [[asp.net]] and [[aspnet]] are slugified to aspent.md, and so on..) trying to de-slugify quickly becomes complex
• some might say this case is not super common, but unfortunately the code has to work in all cases, which means that we would have likely needed to account for it, even if accounting meant being ok with breaking (until a user were to file a bug report...)
• adding a level of indirection, a slugification function, between the text of the wikilink and the corresponding file name creates complexity for other features and use cases
  • e.g. we added refactoring support and I am not sure how much harder it would have been with slugification, but I am sure it would have not been easier and I am glad I didn't have to think of the various edge cases involved
  • we introduced the concept of "Foam identifier", which is not strictly markdown but allows people that have files with the same name across folders to uniquely identify them - again, I am glad I didn't have to think about slugification, or even worse a configuration option to switch between slugified or not links
• An additional level of complexity is due to several slugification functions incompatible with each other (we have already dealt with this problem in Foam for markdown-notes).
  • we actually had a couple of issues due to incompatible slugification back in the days
• Slugification is also unfriendly towards non-latin alphabets, and we have users that use different alphabets

Beside all of those reasons, from a design POV I simply don't believe that slugs are a good foundation, and that's why I chose to get rid of them. I might be wrong, by all means, I have just not been persuaded by the arguments yet.
If we find a clean solution to supporting slugification we might add it, by all means.
What I mean by "clean" is simple: I don't want the core model to have to deal with slugs. In my view slugification is a feature, not a design choice, not a building block. The very conversation in this PR (pre-merge) shows how tricky things can quickly become when slugs are foundational.

At the same time we had some features in the works that would have softened the blow for people that really like to use slugified filenames:

• [[my-file|My File]]: Foam today supports labels
• FOAM_SLUG: Foam today supports this variable to access a slugified version of the title (which you can use in your template as the filepath)
• alias: My File: Foam today supports the alias property which can be used to achieve the [[my-file|My File]] link directly with autocompletion (which can be added in your template to make the file accessible from the title)
• using link definitions [[My File]] and [My File]: my-file.md - not supported today by Foam (and no plans to implement it in Foam core), but it could make it easier for a third party to build on the Foam model and achieve (explicit) slugification, without having labels in the links

With these building blocks we currently have we are getting very close to have an almost seamless support for slugification (if you are happy with all your links looking like [[my-file|My File]], hence "almost seamless").
We are about to add another feature, which I think will make things even easier, currently captured in #1038.

And maybe in the future we'll have link definitions (my last point above) as a great way to support "non-aliased" slugs links in Foam, but that is not a priority in my opinion at this point.

I don't expect everyone to agree with the design choice, and I do appreciate the feedback (when done constructively), slug vs non-slug has been an ongoing topic, and I do want to build a solution that in the end works for the majority of users.