invalid configuration option `:xray_recorder' on Kinesis

[mt-kelvintaywl]

Hi guys, are there better documentations on what dependencies i'll need to update for working with instrumenting this SDK with Ruby Lambda functions?

I followed the example, and have tried to instrument X-Ray with a Lambda function (with Kinesis data stream as its event source).

require 'aws-sdk-lambda'
require 'aws-xray-sdk/lambda'

require './lib/logger.rb'

xray_config = {
  context_missing: 'LOG_ERROR',
  logger: LOGGER
}
XRay.recorder.configure(xray_config)

def my_function(event:, context:)
  client = Aws::Lambda::Client.new

  # business logic

  client.get_account_settings.account_usage.to_h
end

However, I seem to hit a possible bug(?) :

invalid configuration option `:xray_recorder'

The redacted error trace:

"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/configuration.rb:162:in `block in apply_options'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/configuration.rb:161:in `each'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/configuration.rb:161:in `apply_options'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/configuration.rb:149:in `build!'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/base.rb:62:in `build_config'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/base.rb:19:in `initialize'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-kinesis-1.20.0/lib/aws-sdk-kinesis/client.rb:273:in `initialize'",
"/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.72.0/lib/seahorse/client/base.rb:99:in `new'",

My dependencies are below:

GEM
  remote: https://rubygems.org/
  specs:
    ...
    aws-eventstream (1.1.0)
    aws-partitions (1.343.0)
    aws-record (2.4.0)
      aws-sdk-dynamodb (~> 1.18)
    aws-sdk-core (3.104.1)
      aws-eventstream (~> 1, >= 1.0.2)
      aws-partitions (~> 1, >= 1.239.0)
      aws-sigv4 (~> 1.1)
      jmespath (~> 1.0)
    aws-sdk-dynamodb (1.37.0)
      aws-sdk-core (~> 3, >= 3.71.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-kinesis (1.26.0)
      aws-sdk-core (~> 3, >= 3.99.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-lambda (1.31.0)
      aws-sdk-core (~> 3, >= 3.71.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-sns (1.20.0)
      aws-sdk-core (~> 3, >= 3.71.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-xray (1.4.0)
      aws-sdk-core (~> 3)
      aws-sigv4 (~> 1.0)
    aws-sigv4 (1.2.1)
      aws-eventstream (~> 1, >= 1.0.2)
    aws-xray-sdk (0.11.5)
      aws-sdk-xray (~> 1.4.0)
      multi_json (~> 1)
...

DEPENDENCIES
  aws-record
  aws-sdk-kinesis
  aws-sdk-lambda
  aws-sdk-sns
  aws-xray-sdk
  ...

Not sure if it's related to #26 but the fix mentioned here shows the Kinesis service is whitelisted so I'm not sure what exactly is going wrong?
Would i just need to bump my dependencies on the aws-sdk-* gems?

Thank you! 🙏

[srprash]

Hi @mt-kelvintaywl, apologies for the delay in response. I believe having kinesis as your event source for the lambda function wouldn't cause the error since the X-Ray instrumentation is within the function and the message from kinesis stream is passed in as an event. Are you creating a kinesis client and/or calling kinesis service from within your lambda function?

[mt-kelvintaywl]

Hi @srprash, thank you for reaching out and my apologies for the late response here myself too 🙇

I believe having kinesis as your event source for the lambda function wouldn't cause the error since the X-Ray instrumentation is within the function and the message from kinesis stream is passed in as an event.

Indeed, i would believe so too. Yet, this was happening.
For context:

• this Lambda is setup to be triggered by the Kinesis stream events, and this lambda also sends a payload to another Kinesis stream (hence the aws-sdk-kinesis gem as a dependency).
• We are deploying the Lambda code via Serverless Framework and also updated the Lambda's IAM role to allow for X-ray actions.

in the end, i've reverted and abandoned the instrumentation.
Let me close this for now, and perhaps re-open it once I can re-produce once once 🙇

[srprash]

Hi @mt-kelvintaywl
Thanks for providing more details here. I will try to reproduce this on my end as well. Please feel to re-open if you come across it in the future.

[omenking]

I'm getting this error as well, just trying to do general instrutumentation.
Following what we have here:

https://docs.amazonaws.cn/en_us/lambda/latest/dg/ruby-tracing.html

In JS we have:

var AWSXRay = require('aws-xray-sdk');
var AWS = AWSXRay.captureAWS(require('aws-sdk'));

So its my assumption once things are instrumented I'll be able to see functions passed downstream, Since right now I'm just trying to see my Lambda call SNS in my service map. but how Ruby configures vs NodeJS is confusing me since next to no examples.

[Update] I found this while clicking through on one of the files in the documentation.

# lambda_function.rb
require 'logger'
require 'json'
require 'aws-sdk-lambda'
$client = Aws::Lambda::Client.new()
$client.get_account_settings()

require 'aws-xray-sdk/lambda'

def lambda_handler(event:, context:)
  logger = Logger.new($stdout)
  logger.info('## ENVIRONMENT VARIABLES')
  vars = Hash.new
  ENV.each do |variable|
    vars[variable[0]] = variable[1]
  end
  logger.info(vars.to_json)
  logger.info('## EVENT')
  logger.info(event.to_json)
  logger.info('## CONTEXT')
  logger.info(context)
  $client.get_account_settings().account_usage.to_h
end

I suppose that would let me record subsegments, but I just want to see my SNS topic getting called:

require 'aws-xray-sdk/lambda'
require 'aws-sdk-sns'

def handler(event:,context:)
  results = { hello: 'world' }

  sns = Aws::SNS::Resource.new
  sns.client.publish({
    topic_arn: ENV['TOPIC_ARN'],
    # need a default for SNS
    message: results.merge({default: 'evaluators-ruby'}).to_json,
    message_structure: 'json'
  })

  return {
    "statusCode" => 200,
    "body" => results.to_json
  }
end

Just checking through the full backtrace to see what I can find in thes respective Ruby Gems:

{
    "errorMessage": "invalid configuration option `:xray_recorder'",
    "errorType": "Function<ArgumentError>",
    "stackTrace": [
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/configuration.rb:168:in `rescue in block in apply_options'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/configuration.rb:164:in `block in apply_options'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/configuration.rb:163:in `each'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/configuration.rb:163:in `apply_options'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/configuration.rb:151:in `build!'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/base.rb:65:in `build_config'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/base.rb:22:in `initialize'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sns-1.36.0/lib/aws-sdk-sns/client.rb:325:in `initialize'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.110.0/lib/seahorse/client/base.rb:102:in `new'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sns-1.36.0/lib/aws-sdk-sns/resource.rb:28:in `initialize'",
        "/var/task/function.rb:119:in `new'",
        "/var/task/function.rb:119:in `handler'"
    ]
}

This error is similar to this one over here suppose: #26

[omenking]

I looked into the the gems and noticed something called patching, so I added the following to my configuration and this stopped the error. Still can't see my Lambda calling SNS in my Service Map.

require 'aws-xray-sdk'

config = {
  patch: %I[net_http aws_sdk]
}

XRay.recorder.configure(config)

Carefully reading the docs more. Still don't see my downstream

The X-Ray SDK for Ruby automatically instruments all AWS SDK clients when you patch the aws-sdk library. You cannot instrument individual clients.

[omenking]

What is odd is that it looks like require 'aws-xray-sdk/lambda' should already patch, which is what I'm including...

# Set `XRAY_LAMBDA_PATCH_CONFIG` before requiring `aws-xray-sdk/lambda`
# to configure which libraries (if any) to instrument.
#
# By default, both `net_http` and `aws_sdk` will be instrumented
unless defined? XRAY_LAMBDA_PATCH_CONFIG
  XRAY_LAMBDA_PATCH_CONFIG = %I[net_http aws_sdk]
end

# Configure the XRay.recorder with Lambda specific config.
#
# From here, a lambda may create subsegments manually, or via
# the instrumented libraries setup by XRAY_LAMBDA_PATCH_CONFIG
XRay.recorder.configure(
  patch: XRAY_LAMBDA_PATCH_CONFIG,
  context: XRay::LambdaContext.new,
  streamer: XRay::LambdaStreamer.new
)

[jessedoyle]

I've encountered similar errors a number of times now with this library. It's probably worth it to take some time and document the issue.

When this invalid configuration option error occurs it's generally for one of two reasons:

1. The client/service you're trying to use isn't whitelisted in this library yet.
2. The application code is attempting to patch the AWS SDK before the SDK clients have been loaded. This isn't documented anywhere to my knowledge, but ensure your SDK clients are required prior to loading the Xray patches:

i.e.

require 'aws-sdk-lambda'
require 'aws-sdk-kinesis'
# be sure to require any other AWS SDK clients your app will use here
require 'aws-xray-sdk/lambda'