Fix UnicodeEncodeError in urllib if non-ascii characters are present in Request parameters

[romanbarczynski]

Request.to_postdata was fixed some time ago, but Request.to_url was still unable to create url when unicode characters were present in any of Request parameters (because of UnicodeEncodeError)

See included test.

[8maki]

I have the same problem now.
Please pull this!

[atomatt]

Just hit this problem too, and can confirm that romke's commit fixes things.

[atomatt]

Also, that method seems awfully complicated and can probably be replaced with the following if you prefer:

    def to_url(self):
        """Serialize as a URL for a GET request."""
        scheme, netloc, path, query, fragment = urlparse.urlsplit(self.url.encode('utf-8')) 
        query = parse_qs(query)                                                             
        for k, v in self.iteritems():                                                       
            query.setdefault(k.encode('utf-8'), []).append(to_utf8_optional_iterator(v))    
        query = urllib.urlencode(query, True)                                               
        return urlparse.urlunsplit((scheme, netloc, path, query, fragment))                 

[brunsgaard]

Thank you romke for the patch, works like charm..

Someone should pull this into master, the patch has been available for over a year now..

[marians]

Is there any update on this? I seem to run into this problem.

[bxm156]

Thank you!

[yjlou]

Thanks emgee. Your fix saves my code.

[joestump]

Your patch was half-fixed in a different PR. I modified your PR to use to_utf8() instead, but kept your test as-is, which passes. See 7d0b8d7 for the details.