to_json is double encoding UTF8?

[ambs]

try:

package foo;
use Dancer2;
use utf8;

get '/' => sub {
    return to_json({foo => "Simões"});
};
1;

vs

package foo;
use Dancer2;
use utf8;

get '/' => sub {
    return to_json({foo => "Simões"}, { utf8 => 0} );
};
1;

The second works :-|

When fixing this, take care not to break the automatic serializer.
Also, I think this is a regression.

[DavsX]

Update: OK, I think a got it. The whole thing is in D2::Core::Response.pm .When not using a 'set serialzer => something' (https://metacpan.org/pod/distribution/Dancer2/lib/Dancer2/Manual.pod#Serializers1)
, i.e. when manually serializing the output in a route handler (like you are doing with to_json), the 'is_encoded' flag in the Response object is NOT set, causing it to encode it with UTF-8 in D2::Core::Response->encode_content (line: ~170). That is why setting { utf8 => 0 } is necessary in this case.

Indeed, only setting utf8 to 0 in to_json made this test to pass for me. But I have the same problem with Dancer1, where I too use { utf8 => 0 } everywhere when it comes to serialization.

use utf8;
use strict;
use warnings;

use Test::More;
use Plack::Test;
use HTTP::Request::Common;
use Encode;

use Dancer2::Serializer::JSON;

my $utf8 = 'ľščťžýáí';

{
    package MyApp;

    use Dancer2;
    use utf8;

    get '/serialize'  => sub {
        return to_json({ foo => $utf8 }, { utf8 => 0 });
    };
}

my $app = Dancer2->psgi_app;

test_psgi $app, sub {
    my $cb = shift;

    my $res = $cb->( GET '/serialize' );
    is $res->code, 200;
    is Encode::decode('utf8', $res->content), '{"foo":"'.$utf8.'"}';

    my $json = $res->content;
    my $serializer = Dancer2::Serializer::JSON->new;

    is_deeply $serializer->deserialize($json),
              { foo => "$utf8" };
};

done_testing();

[ambs]

I think (not proved by looking into the code) that Dancer is encoding the return value of the route, even if it is already encoded. You know, that "utf8" vs "perl internal format".

Need to look at it, when I finish my hundred other tasks :-|

[DavsX]

See my updated post above (the first half), I think the problem is described there.

#TODO write a Dancer2::Unicode manual?

[ambs]

No, do not think a manual is what is missing. This should work out of the box.

If to_json is always used by the user (the serializer can use other method by itself), then just switch off the utf8 encoding in that method by default!

[DavsX]

Well the 'auto-serializer' uses the 'serialize' method directly, so yes, turning utf-8 to 0 by default when using to_/from_* methods would be an option.

But there could be problems, when people use Dancer2's to_json method for something else, than returning a response (serializing stuff into/from the DB etc. - in those cases the content won't be encoded to UTF-8 by default), so that could break compatibility/cause problems in existing codebases.

Calling the to_json method teoretically could set the is_encoded flag in the response somehow, but then again, there is no guarantee, that to_json is used to return a response.

[ambs]

Yes, good point.

Would it be possible to ask to_json to return a native Perl string, that will then be encoded accordingly with the filehandle that is used?

[DavsX]

BTW in your case using "set serializer => 'JSON';" solves the problem; so another option is to document this behaviour and state, that to_json should NOT be used to return a response from a route handler, and an auto-serializer should be used in those occasions.

[ambs]

Not really. Imagine you want to generate javascript code:

     template 'index.tt' => { json_obj => to_json($myobj) };

This is what I intend to do. I just used a simpler version that was failing.

[DavsX]

Yeah, good point. I have no idea what would be best in this case.

[ambs]

I confess I am quite ignorant regarding the utf8 handling of Perl.
But if I can say

  use utf8;
 ...

  template 'index.tt' => { name => 'Alberto Simões' };

and it works (it doesn't get double encoded), then it should be possible to make to_json return an encoding agnostic string...

[DavsX]

Yeah. 'use utf8;' only tells Perl, that the source code (the .pm file) is in utf-8. In this case this use statement is important, because you have the 'Alberto Simões' string directly in the source code. I believe (and i may be wrong), that in this stage that string is unicode. Only after getting it through the Response object is it encoded into utf-8 (into actual bytes). Now the problem with to_json now is, that by default it does internally the same, after which comes the Response object and encodes the encoded string one more time.

While your assumption regarding to_json usage is quite reasonable, it can be achieved by telling JSON not-to-encode the data by setting utf8 to 0. By playing with the JSON's utf8 setting we must either manually specifying { utf8 => 1 } whenever we use to_json for not returning a response from a route handler, or we must specify { utf8 => 0 } everytime we use it for returning a response. If i'd have to choose from this two option, i'd say we should not need to specify any utf8 options when we are returning a response.

As far as I know there is no way of telling if a string is utf8 or not. There is a is_utf8 method in Perl, which tells if the UTF-8 flag is turned on in a string, but the documentation states that it should be only used for debugging purposes.

BTW this is still my favourite unicode tutorial: http://taniquetil.com.ar/unicode.png :)

[ambs]

My problem is that it is counter-intuitive to set { utf8 => 0 } when you know it is not.

[DavsX]

Yes, indeed, it is. But if we set utf8 to 0 by default, it would be counter-intuitive to set { utf8 => 1 } elsewhere. There is no one good solution, when the default utf8 config option is involved in the solution.