Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT SUBMIT] Reproduce failure while running with server already using port 8080 #282

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

[DO NOT SUBMIT] Reproduce failure while running with server already using port 8080 #282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions debug.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@

[2020-08-14 16:49:26.448][4107145][critical][assert] [external/envoy/source/common/thread_local/thread_local_impl.cc:121] assert failure: std::this_thread::get_id() == main_thread_id_.
[2020-08-14 16:49:26.448][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x8a965003eab47
[2020-08-14 16:49:26.448][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-14 16:49:26.448][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.15.0-dev//DEBUG/BoringSSL
[2020-08-14 16:49:26.483][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x562ed49bb1ac]
[2020-08-14 16:49:26.483][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7fba89718110]
[2020-08-14 16:49:26.514][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::ThreadLocal::InstanceImpl::Bookkeeper::~Bookkeeper() [0x562ed3c9b395]
[2020-08-14 16:49:26.545][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::ThreadLocal::InstanceImpl::Bookkeeper::~Bookkeeper() [0x562ed3c9b429]
[2020-08-14 16:49:26.576][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::default_delete<>::operator()() [0x562ed3091e2f]
[2020-08-14 16:49:26.607][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: std::unique_ptr<>::~unique_ptr() [0x562ed309086a]
[2020-08-14 16:49:26.640][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Extensions::HttpFilters::JwtAuthn::FilterConfigImpl::~FilterConfigImpl() [0x562ed35d66f4]
[2020-08-14 16:49:26.673][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Extensions::HttpFilters::JwtAuthn::FilterConfigImpl::~FilterConfigImpl() [0x562ed35d6759]
[2020-08-14 16:49:26.705][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: std::_Sp_counted_ptr<>::_M_dispose() [0x562ed35d9b0f]
[2020-08-14 16:49:26.736][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: std::_Sp_counted_base<>::_M_release() [0x562ed2ff9f9c]
[2020-08-14 16:49:26.768][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::__shared_count<>::~__shared_count() [0x562ed2ff9f4a]
[2020-08-14 16:49:26.799][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::__shared_ptr<>::~__shared_ptr() [0x562ed35d9d59]
[2020-08-14 16:49:26.832][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: std::shared_ptr<>::~shared_ptr() [0x562ed35d2bb5]
[2020-08-14 16:49:26.865][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Extensions::HttpFilters::JwtAuthn::FilterConfigImpl::init()::$_0::~init() [0x562ed35dde65]
[2020-08-14 16:49:26.898][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::_Function_base::_Base_manager<>::_M_destroy() [0x562ed35de46f]
[2020-08-14 16:49:26.933][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::_Function_base::_Base_manager<>::_M_manager() [0x562ed35de226]
[2020-08-14 16:49:26.965][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: std::_Function_base::~_Function_base() [0x562ed3022d24]
[2020-08-14 16:49:26.994][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: std::function<>::~function() [0x562ed30901a5]
[2020-08-14 16:49:27.024][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: Envoy::ThreadLocal::InstanceImpl::Bookkeeper::set()::$_7::~set() [0x562ed3c929ec]
[2020-08-14 16:49:27.054][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: std::_Function_base::_Base_manager<>::_M_destroy() [0x562ed3c9794f]
[2020-08-14 16:49:27.087][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: std::_Function_base::_Base_manager<>::_M_manager() [0x562ed3c97756]
[2020-08-14 16:49:27.087][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: std::_Function_base::~_Function_base() [0x562ed3022d24]
[2020-08-14 16:49:27.087][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: std::function<>::~function() [0x562ed30901a5]
[2020-08-14 16:49:27.117][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: Envoy::ThreadLocal::InstanceImpl::SlotImpl::set()::$_13::~set() [0x562ed3c94839]
[2020-08-14 16:49:27.148][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: std::_Function_base::_Base_manager<>::_M_destroy() [0x562ed3c995cf]
[2020-08-14 16:49:27.180][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #25: std::_Function_base::_Base_manager<>::_M_manager() [0x562ed3c99396]
[2020-08-14 16:49:27.180][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: std::_Function_base::~_Function_base() [0x562ed3022d24]
[2020-08-14 16:49:27.211][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #27: std::function<>::~function() [0x562ed3092ce5]
[2020-08-14 16:49:27.242][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #28: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x562ed3e52701]
[2020-08-14 16:49:27.275][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #29: Envoy::Event::DispatcherImpl::run() [0x562ed3e525f5]
[2020-08-14 16:49:27.305][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #30: Envoy::Server::WorkerImpl::threadRoutine() [0x562ed3e2d009]
[2020-08-14 16:49:27.338][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #31: Envoy::Server::WorkerImpl::start()::$_4::operator()() [0x562ed3e2e76f]
[2020-08-14 16:49:27.368][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #32: std::_Function_handler<>::_M_invoke() [0x562ed3e2e61d]
[2020-08-14 16:49:27.403][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #33: std::function<>::operator()() [0x562ed30f167e]
[2020-08-14 16:49:27.432][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #34: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x562ed4a1b8f0]
[2020-08-14 16:49:27.467][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #35: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x562ed4a1b8c5]
[2020-08-14 16:49:27.467][4107145][critical][backtrace] [bazel-out/k8-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #36: start_thread [0x7fba8970cf27]
2 changes: 1 addition & 1 deletion tests/env/env.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ func (e *TestEnv) Setup(confArgs []string) error {
confArgs = append(confArgs, "--service_control_iam_delegates="+e.serviceControlIamDelegates)
}

confArgs = append(confArgs, fmt.Sprintf("--listener_port=%v", e.ports.ListenerPort))
confArgs = append(confArgs, fmt.Sprintf("--listener_port=%v", 8080))
confArgs = append(confArgs, fmt.Sprintf("--discovery_port=%v", e.ports.DiscoveryPort))
confArgs = append(confArgs, fmt.Sprintf("--service=%v", e.fakeServiceConfig.Name))

Expand Down
90 changes: 90 additions & 0 deletions third_party/tools/stack_decode.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#!/usr/bin/env python3

# Call addr2line as needed to resolve addresses in a stack trace. The addresses
# will be replaced if they can be resolved into file and line numbers. The
# executable must include debugging information to get file and line numbers.
#
# Two ways to call:
# 1) Execute binary as a subprocess: stack_decode.py executable_file [args]
# 2) Read log data from stdin: stack_decode.py -s executable_file
#
# In each case this script will add file and line information to any backtrace log
# lines found and echo back all non-Backtrace lines untouched.

import collections
import re
import subprocess
import sys


# Process the log output looking for stacktrace snippets, for each line found to
# contain backtrace output extract the address and call add2line to get the file
# and line information. Output appended to end of original backtrace line. Output
# any nonmatching lines unmodified. End when EOF received.
def decode_stacktrace_log(object_file, input_source):
traces = {}
# Match something like:
# [backtrace] [bazel-out/local-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:84]
backtrace_marker = "\[backtrace\] [^\s]+"
# Match something like:
# ${backtrace_marker} #10: SYMBOL [0xADDR]
# or:
# ${backtrace_marker} #10: [0xADDR]
stackaddr_re = re.compile("%s #\d+:(?: .*)? \[(0x[0-9a-fA-F]+)\]$" % backtrace_marker)
# Match something like:
# #10 0xLOCATION (BINARY+0xADDR)
asan_re = re.compile(" *#\d+ *0x[0-9a-fA-F]+ *\([^+]*\+(0x[0-9a-fA-F]+)\)")

try:
while True:
line = input_source.readline()
if line == "":
return # EOF
stackaddr_match = stackaddr_re.search(line)
if not stackaddr_match:
stackaddr_match = asan_re.search(line)
if stackaddr_match:
address = stackaddr_match.groups()[0]
file_and_line_number = run_addr2line(object_file, address)
file_and_line_number = trim_proc_cwd(file_and_line_number)
sys.stdout.write("%s %s" % (line.strip(), file_and_line_number))
continue
else:
# Pass through print all other log lines:
sys.stdout.write(line)
except KeyboardInterrupt:
return


# Execute addr2line with a particular object file and input string of addresses
# to resolve, one per line.
#
# Returns list of result lines
def run_addr2line(obj_file, addr_to_resolve):
return subprocess.check_output(["addr2line", "-Cpie", obj_file, addr_to_resolve]).decode('utf-8')


# Because of how bazel compiles, addr2line reports file names that begin with
# "/proc/self/cwd/" and sometimes even "/proc/self/cwd/./". This isn't particularly
# useful information, so trim it out and make a perfectly useful relative path.
def trim_proc_cwd(file_and_line_number):
trim_regex = r'/proc/self/cwd/(\./)?'
return re.sub(trim_regex, '', file_and_line_number)


if __name__ == "__main__":
if len(sys.argv) > 2 and sys.argv[1] == '-s':
decode_stacktrace_log(sys.argv[2], sys.stdin)
sys.exit(0)
elif len(sys.argv) > 1:
rununder = subprocess.Popen(sys.argv[1:],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True)
decode_stacktrace_log(sys.argv[1], rununder.stdout)
rununder.wait()
sys.exit(rununder.returncode) # Pass back test pass/fail result
else:
print("Usage (execute subprocess): stack_decode.py executable_file [additional args]")
print("Usage (read from stdin): stack_decode.py -s executable_file")
sys.exit(1)
14 changes: 14 additions & 0 deletions tmp.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[2020-08-14 16:30:38.839][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[2020-08-14 16:30:38.839][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-14 16:30:38.839][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.15.0-dev//RELEASE/BoringSSL
[2020-08-14 16:30:38.839][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7faaa2c01110]
[2020-08-14 16:30:38.848][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: espv2::envoy::http_filters::service_control::ClientCache::ClientCache() [0x55c877763d60]
[2020-08-14 16:30:38.856][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: espv2::envoy::http_filters::service_control::ThreadLocalCache::ThreadLocalCache() [0x55c877763740]
[2020-08-14 16:30:38.860][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::_Function_handler<>::_M_invoke() [0x55c877763383]
[2020-08-14 16:30:38.865][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::_Function_handler<>::_M_invoke() [0x55c877c47f46]
[2020-08-14 16:30:38.870][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: std::_Function_handler<>::_M_invoke() [0x55c877c48eed]
[2020-08-14 16:30:38.874][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x55c877ce451d]
[2020-08-14 16:30:38.879][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Event::DispatcherImpl::run() [0x55c877ce43bc]
[2020-08-14 16:30:38.884][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Server::WorkerImpl::threadRoutine() [0x55c877cd8b40]
[2020-08-14 16:30:38.888][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x55c8782720b5]
[2020-08-14 16:30:38.888][4069482][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: start_thread [0x7faaa2bf5f27]