Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jenkins module has whacky behavior on SIGINT #181

Open
cburroughs opened this issue Feb 5, 2015 · 0 comments
Open

jenkins module has whacky behavior on SIGINT #181

cburroughs opened this issue Feb 5, 2015 · 0 comments

Comments

@cburroughs
Copy link
Contributor 

saving metadata for metric: tcpext_listendrops host: csb-buildbot-0
Processing a metric value message from csb-buildbot-0
***Allocating value packet for host--csb-buildbot-0-- and metric --tcpext_listendrops-- ****

Processing a metric metadata message from csb-buildbot-0
***Allocating metadata packet for host--csb-buildbot-0-- and metric --tcp_attemptfails-- ****

saving metadata for metric: tcp_attemptfails host: csb-buildbot-0
Processing a metric value message from csb-buildbot-0
***Allocating value packet for host--csb-buildbot-0-- and metric --tcp_attemptfails-- ****

Processing a metric metadata message from csb-buildbot-0
***Allocating metadata packet for host--csb-buildbot-0-- and metric --disk_free_absolute_rootfs-- ****

saving metadata for metric: disk_free_absolute_rootfs host: csb-buildbot-0
Processing a metric value message from csb-buildbot-0
***Allocating value packet for host--csb-buildbot-0-- and metric --disk_free_absolute_rootfs-- ****

Processing a metric metadata message from csb-buildbot-0
***Allocating metadata packet for host--csb-buildbot-0-- and metric --disk_free_percent_rootfs-- ****

saving metadata for metric: disk_free_percent_rootfs host: csb-buildbot-0
Processing a metric value message from csb-buildbot-0
***Allocating value packet for host--csb-buildbot-0-- and metric --disk_free_percent_rootfs-- ****

^Capr_pollset_poll returned unexpected status 4 = Interrupted system call

Traceback (most recent call last):
  File "/usr/lib64/ganglia/python_modules/jenkins.py", line 96, in refresh_metrics
    data = UpdateJenkinsThread._get_jenkins_statistics(self.base_url, self.username, self.apitoken)
  File "/usr/lib64/ganglia/python_modules/jenkins.py", line 66, in _get_jenkins_statistics
    c = urllib2.urlopen(url, None, 2)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 97] Address family not supported by protocol>
Traceback (most recent call last):
  File "/usr/lib64/ganglia/python_modules/jenkins.py", line 104, in refresh_metrics
    for k, v in data.items():
UnboundLocalError: local variable 'data' referenced before assignment
Traceback (most recent call last):
  File "/usr/lib64/ganglia/python_modules/diskstat.py", line 361, in metric_cleanup
    logging.shutdown()
  File "/usr/lib64/python2.6/logging/__init__.py", line 1512, in shutdown
    for h in handlerList[:]:
AttributeError: 'module' object has no attribute 'metric_cleanup'
Traceback (most recent call last):
  File "/usr/lib64/ganglia/python_modules/apache_status.py", line 377, in metric_cleanup
    _Worker_Thread.shutdown()
NameError: global name '_Worker_Thread' is not defined
*** glibc detected *** gmond: double free or corruption (!prev): 0x0000000000f23ee0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x39e1a75e66]
/lib64/libc.so.6[0x39e1a789b3]
/usr/lib64/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x39e2a17f7d]
/usr/lib64/libapr-1.so.0(apr_pool_terminate+0x34)[0x39e2a18b84]
/lib64/libc.so.6(exit+0xe2)[0x39e1a35b22]
/lib64/libc.so.6(__libc_start_main+0x104)[0x39e1a1ed64]
gmond[0x404579]
======= Memory map: ========
00400000-0041c000 r-xp 00000000 fd:00 801322                             /usr/sbin/gmond
0061c000-0061d000 rw-p 0001c000 fd:00 801322                             /usr/sbin/gmond
0061d000-0061e000 rw-p 00000000 00:00 0 
00a93000-01153000 rw-p 00000000 00:00 0                                  [heap]
3394400000-3394483000 r-xp 00000000 fd:00 1310776                        /lib64/libm-2.12.so
3394483000-3394682000 ---p 00083000 fd:00 1310776                        /lib64/libm-2.12.so
3394682000-3394683000 r--p 00082000 fd:00 1310776                        /lib64/libm-2.12.so
3394683000-3394684000 rw-p 00083000 fd:00 1310776                        /lib64/libm-2.12.so
39e1200000-39e1220000 r-xp 00000000 fd:00 1313808                        /lib64/ld-2.12.so
39e141f000-39e1420000 r--p 0001f000 fd:00 1313808                        /lib64/ld-2.12.so
39e1420000-39e1421000 rw-p 00020000 fd:00 1313808                        /lib64/ld-2.12.so
39e1421000-39e1422000 rw-p 00000000 00:00 0 
39e1600000-39e1602000 r-xp 00000000 fd:00 1322010                        /lib64/libdl-2.12.so
39e1602000-39e1802000 ---p 00002000 fd:00 1322010                        /lib64/libdl-2.12.so
39e1802000-39e1803000 r--p 00002000 fd:00 1322010                        /lib64/libdl-2.12.so
39e1803000-39e1804000 rw-p 00003000 fd:00 1322010                        /lib64/libdl-2.12.so
39e1a00000-39e1b8a000 r-xp 00000000 fd:00 1322003                        /lib64/libc-2.12.so
39e1b8a000-39e1d8a000 ---p 0018a000 fd:00 1322003                        /lib64/libc-2.12.so
39e1d8a000-39e1d8e000 r--p 0018a000 fd:00 1322003                        /lib64/libc-2.12.so
39e1d8e000-39e1d8f000 rw-p 0018e000 fd:00 1322003                        /lib64/libc-2.12.so
39e1d8f000-39e1d94000 rw-p 00000000 00:00 0 
39e1e00000-39e1e17000 r-xp 00000000 fd:00 1322008                        /lib64/libpthread-2.12.so
39e1e17000-39e2017000 ---p 00017000 fd:00 1322008                        /lib64/libpthread-2.12.so
39e2017000-39e2018000 r--p 00017000 fd:00 1322008                        /lib64/libpthread-2.12.so
39e2018000-39e2019000 rw-p 00018000 fd:00 1322008                        /lib64/libpthread-2.12.so
39e2019000-39e201d000 rw-p 00000000 00:00 0 
39e2200000-39e222c000 r-xp 00000000 fd:00 1310772                        /lib64/libpcre.so.0.0.1
39e222c000-39e242b000 ---p 0002c000 fd:00 1310772                        /lib64/libpcre.so.0.0.1
39e242b000-39e242c000 rw-p 0002b000 fd:00 1310772                        /lib64/libpcre.so.0.0.1
39e2600000-39e2615000 r-xp 00000000 fd:00 1322005                        /lib64/libz.so.1.2.3
39e2615000-39e2814000 ---p 00015000 fd:00 1322005                        /lib64/libz.so.1.2.3
39e2814000-39e2815000 r--p 00014000 fd:00 1322005                        /lib64/libz.so.1.2.3
39e2815000-39e2816000 rw-p 00015000 fd:00 1322005                        /lib64/libz.so.1.2.3
39e2a00000-39e2a2b000 r-xp 00000000 fd:00 791234                         /usr/lib64/libapr-1.so.0.3.9
39e2a2b000-39e2c2a000 ---p 0002b000 fd:00 791234                         /usr/lib64/libapr-1.so.0.3.9
39e2c2a000-39e2c2c000 rw-p 0002a000 fd:00 791234                         /usr/lib64/libapr-1.so.0.3.9
39e3200000-39e320b000 r-xp 00000000 fd:00 789171                         /usr/lib64/libconfuse.so.0.0.0
39e320b000-39e340b000 ---p 0000b000 fd:00 789171                         /usr/lib64/libconfuse.so.0.0.0
39e340b000-39e340c000 rw-p 0000b000 fd:00 789171                         /usr/lib64/libconfuse.so.0.0.0
39e3600000-39e375d000 r-xp 00000000 fd:00 806218                         /usr/lib64/libpython2.6.so.1.0
39e375d000-39e395c000 ---p 0015d000 fd:00 806218                         /usr/lib64/libpython2.6.so.1.0
39e395c000-39e3998000 rw-p 0015c000 fd:00 806218                         /usr/lib64/libpython2.6.so.1.0
39e3998000-39e39a6000 rw-p 00000000 00:00 0 
39e3a00000-39e3a16000 r-xp 00000000 fd:00 1316044                        /lib64/libresolv-2.12.so
39e3a16000-39e3c16000 ---p 00016000 fd:00 1316044                        /lib64/libresolv-2.12.so
39e3c16000-39e3c17000 r--p 00016000 fd:00 1316044                        /lib64/libresolv-2.12.so
39e3c17000-39e3c18000 rw-p 00017000 fd:00 1316044                        /lib64/libresolv-2.12.so
39e3c18000-39e3c1a000 rw-p 00000000 00:00 0 
39e4200000-39e43b8000 r-xp 00000000 fd:00 808285                         /usr/lib64/libcrypto.so.1.0.1e
39e43b8000-39e45b8000 ---p 001b8000 fd:00 808285                         /usr/lib64/libcrypto.so.1.0.1e
39e45b8000-39e45d3000 r--p 001b8000 fd:00 808285                         /usr/lib64/libcrypto.so.1.0.1e
39e45d3000-39e45df000 rw-p 001d3000 fd:00 808285                         /usr/lib64/libcrypto.so.1.0.1e
39e45df000-39e45e3000 rw-p 00000000 00:00 0 
39e4600000-39e4616000 r-xp 00000000 fd:00 1322045                        /lib64/libnsl-2.12.so
39e4616000-39e4815000 ---p 00016000 fd:00 1322045                        /lib64/libnsl-2.12.so
39e4815000-39e4816000 r--p 00015000 fd:00 1322045                        /lib64/libnsl-2.12.so
39e4816000-39e4817000 rw-p 00016000 fd:00 1322045                        /lib64/libnsl-2.12.so
39e4817000-39e4819000 rw-p 00000000 00:00 0 
39e4e00000-39e4e73000 r-xp 00000000 fd:00 1322011                        /lib64/libfreebl3.so
39e4e73000-39e5072000 ---p 00073000 fd:00 1322011                        /lib64/libfreebl3.so
39e5072000-39e5074000 r--p 00072000 fd:00 1322011                        /lib64/libfreebl3.so
39e5074000-39e5075000 rw-p 00074000 fd:00 1322011                        /lib64/libfreebl3.so
39e5075000-39e5079000 rw-p 00000000 00:00 0 
39e5600000-39e5607000 r-xp 00000000 fd:00 1322012                        /lib64/libcrypt-2.12.so
39e5607000-39e5807000 ---p 00007000 fd:00 1322012                        /lib64/libcrypt-2.12.so
39e5807000-39e5808000 r--p 00007000 fd:00 1322012                        /lib64/libcrypt-2.12.so
39e5808000-39e5809000 rw-p 00008000 fd:00 1322012                        /lib64/libcrypt-2.12.so
39e5809000-39e5837000 rw-p 00000000 00:00 0 
39e5a00000-39e5a26000 r-xp 00000000 fd:00 1322006                        /lib64/libexpat.so.1.5.2
39e5a26000-39e5c25000 ---p 00026000 fd:00 1322006                        /lib64/libexpat.so.1.5.2Aborted

Stack trace between when I hit C-c and the Abort

[root@csb-buildbot-0 ~]# gstack 15725
Thread 2 (Thread 0x7fbc12432700 (LWP 15734)):
#0  0x00000039e1ae1453 in select () from /lib64/libc.so.6
#1  0x00007fbc14d05219 in ?? () from /usr/lib64/python2.6/lib-dynload/timemodule.so
#2  0x00000039e36d59e4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#3  0x00000039e36d6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#4  0x00000039e36d6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#5  0x00000039e36d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#6  0x00000039e366acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0
#7  0x00000039e3643c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#8  0x00000039e36566af in ?? () from /usr/lib64/libpython2.6.so.1.0
#9  0x00000039e3643c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#10 0x00000039e36cfc93 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0
#11 0x00000039e37017ba in ?? () from /usr/lib64/libpython2.6.so.1.0
#12 0x00000039e1e079d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00000039e1ae89dd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbc1c12f740 (LWP 15725)):
#0  0x00000039e1e0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00000039e36fd438 in PyThread_acquire_lock () from /usr/lib64/libpython2.6.so.1.0
#2  0x00000039e3701324 in ?? () from /usr/lib64/libpython2.6.so.1.0
#3  0x00000039e36d59e4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#4  0x00000039e36d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#5  0x00000039e36d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#6  0x00000039e36d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#7  0x00000039e36d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#8  0x00000039e36d6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#9  0x00000039e36d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#10 0x00000039e366acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0
#11 0x00000039e3643c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#12 0x00000039e3643d51 in ?? () from /usr/lib64/libpython2.6.so.1.0
#13 0x00000039e3644972 in PyObject_CallFunction () from /usr/lib64/libpython2.6.so.1.0
#14 0x00007fbc1525b0ec in pyth_metric_cleanup () from /usr/lib64/ganglia/modpython.so
#15 0x00000039e2a1899e in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
#16 0x00000039e2a18975 in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
#17 0x0000000000408a5c in main ()
[root@csb-buildbot-0 ~]# pgrep gmond
18608
You have new mail in /var/spool/mail/root
[root@csb-buildbot-0 ~]# gstack 18608
Thread 1 (Thread 0x7fcefd47c740 (LWP 18608)):
#0  0x00007fcef21798eb in ?? () from /lib64/libgcc_s.so.1
#1  0x00000039e1b266f6 in dl_iterate_phdr () from /lib64/libc.so.6
#2  0x00007fcef217a207 in _Unwind_Find_FDE () from /lib64/libgcc_s.so.1
#3  0x00007fcef2177603 in ?? () from /lib64/libgcc_s.so.1
#4  0x00007fcef2178119 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#5  0x00000039e1afea66 in backtrace () from /lib64/libc.so.6
#6  0x00000039e1a7054b in __libc_message () from /lib64/libc.so.6
#7  0x00000039e1a75e66 in malloc_printerr () from /lib64/libc.so.6
#8  0x00000039e1a789b3 in _int_free () from /lib64/libc.so.6
#9  0x00000039e2a17f7d in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#10 0x00000039e2a18b84 in apr_pool_terminate () from /usr/lib64/libapr-1.so.0
#11 0x00000039e1a35b22 in exit () from /lib64/libc.so.6
#12 0x00000039e1a1ed64 in __libc_start_main () from /lib64/libc.so.6
#13 0x0000000000404579 in _start ()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cburroughs