Document paths as byte strings vs char strings

[grr]

Some methods return byte strings, like children because of the underlying call to readdir. But there is no mention in the docs of what is expected or what is returned, in respect to encoding of pathnames. I tested mkdir and touch with with both a byte string and char string (utf8-encoded and decoded) and both resulted in the same path being created, but I'm not sure if this is guaranteed or only OS-specific. I know you need to encode the pathname as bytes if you pass it to an external command. And pathname comparison requires the two paths to both be either bytes or strings.

Some of this is discussed in #48

[karenetheridge]

Filename encodings are architecture-dependent: not just on the operating system but which filesystem you're using. I've seen filesystems that use UTF-8, UTF-16 and some that barf on anything that's not ascii. This is very much a "buyer beware" situation.

The only thing I think we could do, other than the user be vigilant about hwo the filename string is used, is potentially add an attribute that one could store on the file object that indicates how the filename should be encoded when used in a system command (as opposed to included in a string of unicode characters, which might be later encoded as UTF-8 for printing to a terminal or writing as file content).

[ap]

The point of Path::Tiny is that you can use it in place of a filename string, i.e. you can just pass it to anything that would accept a filename, including Perl’s builtin open and friends. That means the stringified value of a Path::Tiny must be suitable for that, which precludes stringifying to a decoded string, and ultimately also precludes automatic encoding when you assemble paths (e.g. via the child method). So filename encoding isn’t and can’t be Path::Tiny’s business. Instead the rules are simply the same as when you directly use Perl’s builtins that deal with filenames.

The only thing I can think of that Path::Tiny could even do is make the encoding/decoding business a bit more convenient by letting the caller explicitly ask for $path->decoded( $encoding ) and Path::Tiny->new_encoded( $encoding, @args_for_new ), and only in that context does an encoding attribute seem feasible to me, solely for the purpose of allowing the parameter of the decoded method to be optional when that attribute is set – i.e. a mechanism for different parts of a program to communicate with each other without an extra sideband, nothing more.

[grr]

The only doc that seems to touch on this is here: https://perldoc.perl.org/perlunicode#When-Unicode-Does-Not-Happen

One reason that Perl does not attempt to resolve the role of Unicode in these situations is that the answers are highly dependent on the operating system and the file system(s). For example, whether filenames can be in Unicode and in exactly what kind of encoding, is not exactly a portable concept. Similarly for qx and system: how well will the "command-line interface" (and which of them?) handle Unicode?

I was just hoping the doc would mention something like this and especially point out that that path comparisons should be similarly encoded. A cookbook as mentioned here would be useful:

I'm inclined to just add a documentation section (or a cookbook) with explaining this and with useful idioms and leave it at that.

[xdg]

Is the request to document in Path::Tiny that paths are bytes?

[grr]

Yes, mostly. But also the caveat that you can seemingly use character strings for paths sometimes with no ill effect, but it will be an issue if you have to compare them at some point.