[http_fuzz] Accept-Encoding header will result in content not being decoded

[laxa]

How to reproduce:

patator http_fuzz url=https://www.google.fr/ header='Accept-Encoding: gzip' -l log ; cat log/1_*

Then, the content of the response (not the headers) is not uncompressed.
This behavior is due to the use of the DEBUGFUNCTION callback rather than the WRITEDATA or WRITEFUNCTION that are not affected by this issue.
I don't actually know if this behavior is intended for the pycurl library...

[lanjelot]

Are you saying you would like Patator to gzip-decode the HTTP response when you are specifically passing Accept-Encoding: gzip ?

[laxa]

I am just saying that the intended behavior is not what's happening in this case.
I simplified the case but I encountered this problem when using raw_request with Accept-Encoding headers. I don't actually know why patator uses the DEBUGFUNCTION but I guess the more straightforward fix would be to use the WRITEDATA or WRITEFUNCTION.
My overall idea would be that this behavior does not happen, I guess stripping this header would also make sense.

[lanjelot]

Patator uses DEBUGFUNCTION in order to log both the request and response, whereas WRITEDATA would only log the response.

Also even if Patator used WRITEDATA, sending Accept-Encoding: gzip will result in the response being gzip-encoded so I'm not sure why you suggest using WRITEDATA at all but maybe I'm missing something.

import pycurl                                              
from StringIO import StringIO                              

buffer = StringIO()                                        
c = pycurl.Curl()                                          
c.setopt(c.URL, 'http://pycurl.io/')                       
c.setopt(c.HTTPHEADER, [('Accept-Encoding: gzip'),])       
c.setopt(c.WRITEDATA, buffer)                              
c.perform()                                                
c.close()                                                  

body = buffer.getvalue()                                                                                     
print('%r' % body)

[lanjelot]

Also, I'm not too sure about stripping the header on behalf of the user. It may confuse him/her too if for example they are surprised not to receive a encoded response. I tend to prefer letting the user understand what they are actually doing instead of doing it for them.

[laxa]

Here is a PR that should fix this issue. Maybe I should add a warning in case the Accept-Encoding contains a value that is not correctly handled by pycurl ?