bin/data-mining-monthly.sh stops with response timeout

[sfendrich]

bin/data-mining-monthly.sh frequently stops during the step osm-vis/osm-tags-word-frequency-wiki with a response timeout. Re-running the script sometimes solves the problem. The full error message is as follows:

Preprocessing executable 'tool' for tool-0.1.0...
Running tool...
tool: HttpExceptionRequest Request {
  host                 = "wiki.openstreetmap.org"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/wiki/de:Map_Features"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ResponseTimeout

[mocnik-science]

Could you try whether increasing the number of seconds before timeout for a curl request solves the issue? see http://hackage.haskell.org/package/curl-1.3.8/docs/Network-Curl-Opts.html#t:CurlOption

That is, something like

downloadRaw = fmap snd . flip curlGetString [CurlTimeout 600] . urlWikiRaw . fromMaybe "" . stripPrefix urlWikiPrefix

in line 109 in file https://github.com/mocnik-science/osm-vis/blob/master/data-mining/osm-tags-word-frequency-wiki/tool.hs

[sfendrich]

Thanks. With a CurlTimeout 600 the manual restart went through now. As it also worked sometimes before, I would like to keep this issue open until the next monthly data mining passed successfully.

[mocnik-science]

Great.

[sfendrich]

Unfortunately this still does not work reliably. Maybe we need to think about a different way of doing this.