Segmentation Fault

[gyamxxx]

Getting frequent restarting td-agent at once an hour. this is td-agent.log.

2017-11-15 21:06:08 +0900 [info]: process finished code=6
2017-11-15 21:06:08 +0900 [error]: fluentd main process died unexpectedly. restarting.
2017-11-15 21:06:08 +0900 [info]: starting fluentd-0.12.20

So, I could get entire log using following command like official site
https://docs.fluentd.org/v0.12/articles/trouble-shooting

$ sudo LD_PRELOAD=/opt/td-agent/embedded/lib/libjemalloc.so /usr/sbin/td-agent -c /etc/td-agent/td-agent.conf --user td-agent --group td-agent

And, I got segmentation faults.

  451 /opt/td-agent/embedded/lib/ruby/2.1.0/yaml.rb
  452 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/os-0.9.6/lib/os.rb
  453 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/credentials_loader.rb
  454 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/signet.rb
  455 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/compute_engine.rb
  456 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/service_account.rb
  457 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/scope_util.rb
  458 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/user_refresh.rb
  459 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/client_id.rb
  460 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/user_authorizer.rb
  461 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth/web_user_authorizer.rb
  462 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleauth-0.5.1/lib/googleauth.rb
  463 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-core-1.0.0/lib/google/cloud/credentials.rb
  464 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/credentials.rb
  465 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/version.rb
  466 /opt/td-agent/embedded/lib/ruby/2.1.0/English.rb
  467 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/errors.rb
  468 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/bundling.rb
  469 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/api_callable.rb
  470 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/constants.rb
  471 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/credentials.rb
  472 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/version.rb
  473 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/lex_token.rb
  474 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/lex.rb
  475 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/file_lex.rb
  476 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/production.rb
  477 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/lr_item.rb
  478 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/grammar.rb
  479 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/yacc_production.rb
  480 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/yacc_symbol.rb
  481 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/lr_table.rb
  482 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/ply_dump.rb
  483 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/yacc.rb
  484 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly/parse/rule_parser.rb
  485 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/rly-0.2.3/lib/rly.rb
  486 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/path_template.rb
  487 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/settings.rb
  488 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/util.rb
  489 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/version.rb
  490 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax.rb
  491 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/message_exts.rb
  492 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/2.1/protobuf_c.so
  493 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/repeated_field.rb
  494 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf.rb
  495 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-google-iam-v1-0.6.8/lib/google/iam/v1/policy_pb.rb
  496 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-google-iam-v1-0.6.8/lib/google/iam/v1/iam_policy_pb.rb
  497 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleapis-common-protos-1.3.5/lib/google/api/http_pb.rb
  498 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/googleapis-common-protos-1.3.5/lib/google/api/annotations_pb.rb
  499 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/duration_pb.rb
  500 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/empty_pb.rb
  501 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/field_mask_pb.rb
  502 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-protobuf-3.3.0-x86_64-linux/lib/google/protobuf/timestamp_pb.rb
  503 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/pubsub/v1/pubsub_pb.rb
  504 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/v1/publisher_client.rb
  505 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/v1/subscriber_client.rb
  506 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/v1.rb
  507 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/service.rb
  508 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/topic/publisher.rb
  509 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/topic/list.rb
  510 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/subscription/list.rb
  511 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/message.rb
  512 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/received_message.rb
  513 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/subscription.rb
  514 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/policy.rb
  515 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/topic.rb
  516 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub/project.rb
  517 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/cloud/pubsub.rb
  518 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retryable-2.0.4/lib/retryable/version.rb
  519 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retryable-2.0.4/lib/retryable/configuration.rb
  520 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/retryable-2.0.4/lib/retryable.rb
  521 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-custom-0.4.6/lib/fluent/plugin/gcloud_pubsub/client.rb
  522 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-custom-0.4.6/lib/fluent/plugin/out_gcloud_pubsub.rb
  523 /opt/td-agent/embedded/lib/ruby/2.1.0/x86_64-linux/bigdecimal.so
  524 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/version.rb
  525 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/bag.rb
  526 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/error.rb
  527 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/mimic.rb
  528 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/saj.rb
  529 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/schandler.rb
  530 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj/oj.so
  531 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/oj-2.14.4/lib/oj.rb
  532 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/2.1/grpc_c.so
  533 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/grpc.rb
  534 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/errors.rb
  535 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/logconfig.rb
  536 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/notifier.rb
  537 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/version.rb
  538 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/core/time_consts.rb
  539 /opt/td-agent/embedded/lib/ruby/2.1.0/weakref.rb
  540 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb
  541 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb
  542 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb
  543 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb
  544 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/service.rb
  545 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb
  546 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-1.4.1-x86_64-linux/src/ruby/lib/grpc.rb
  547 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-gax-0.8.5/lib/google/gax/grpc.rb
  548 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/grpc-google-iam-v1-0.6.8/lib/google/iam/v1/iam_policy_services_pb.rb
  549 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.24.0/lib/google/pubsub/v1/pubsub_services_pb.rb
  550 /opt/td-agent/embedded/lib/ruby/2.1.0/x86_64-linux/enc/trans/single_byte.so

Using fluentd 0.12.20 and dependencies are like this.

$ sudo td-agent-gem list | egrep "pubsub|google"
fluent-plugin-gcloud-pubsub-custom (0.4.6)
google-api-client (0.8.6)
google-cloud-core (1.0.0)
google-cloud-env (1.0.1)
google-cloud-pubsub (0.24.0)
google-gax (0.8.5)
google-protobuf (3.3.0 x86_64-linux)
googleapis-common-protos (1.3.5)
googleauth (0.5.1)
grpc-google-iam-v1 (0.6.8)

How can I fix it without update td-agent 0.14?

Full log available here:
full.log

[mia-0032]

Thank you for your reporting.
I'm not sure about this issue, but I think this error caused by grpc or gax gem.
Could you upgrade google-gax and grpc gem and try again?
Perhaps google-gax should be able to upgrade to v0.8.12, grpc up to v1.6.7.

[gyamxxx]

updating fluentd 0.12.20 -> 0.12.40, this problem was not happend...

Could you upgrade google-gax and grpc gem and try again?

I will try it.

[gyamxxx]

Perhaps google-gax should be able to upgrade to v0.8.12,grpc up to v1.6.7.

same problem was happened these library version on fluentd 0.12.20 ;-(

google-gax (0.8.12)
grpc (1.6.7 x86_64-linux)