Investigate web site outage (Mar 5, 2019)

[sillitoe]

No description provided.

[sillitoe]

There was an web outage which required a reset (reported in #39).

Looks like an issue with memory - needs investigation.

[sillitoe]

For reference - looks like errors start creeping in around 05/Mar/2019:06:26:14 GMT

Log entries around that time:

128.40.46.15 - - [05/Mar/2019:06:26:21 +0000] "POST /solr/select HTTP/1.1" 200 39610
128.40.46.15 - - [05/Mar/2019:06:26:21 +0000] "GET /search/by_text?q=4h87 HTTP/1.1" 200 14225
128.40.46.15 - - [05/Mar/2019:06:26:19 +0000] "GET / HTTP/1.1" 200 36436
128.40.46.15 - - [05/Mar/2019:06:26:14 +0000] "GET /superfamily/3.90.190.10 HTTP/1.1" 302 323
128.40.46.15 - - [05/Mar/2019:06:26:32 +0000] "GET / HTTP/1.1" 200 36436
128.40.46.15 - - [05/Mar/2019:06:26:37 +0000] "GET /superfamily/mda/1.10.275.10-1.20.200.10 HTTP/1.1" 302 339
128.40.46.15 - - [05/Mar/2019:06:26:40 +0000] "GET /cathnode/3.30.70.2940 HTTP/1.1" 302 321
128.40.46.15 - - [05/Mar/2019:06:26:42 +0000] "GET /search/by_sequence HTTP/1.1" 200 45759
128.40.46.15 - - [05/Mar/2019:06:26:16 +0000] "GET /version/v4_2_0/superfamily/3.30.70.2230/classification HTTP/1.1" 500 620
128.40.46.15 - - [05/Mar/2019:06:26:32 +0000] "GET /version/v4_2_0/superfamily/3.40.50.2020/funfam/18420/go HTTP/1.1" 500 620
128.40.46.15 - - [05/Mar/2019:06:26:13 +0000] "GET /pdb/2xnz HTTP/1.1" 500 620
128.40.46.15 - - [05/Mar/2019:06:26:38 +0000] "GET /pdb/3dmn HTTP/1.1" 500 620

Might be several long running queries saturating threads. Certainly need to check /superfamily/mda