Support for SQS Fair Queues

[catkins]

AWS just launched a new type of queue for multi tenant apps.

https://aws.amazon.com/blogs/compute/building-resilient-multi-tenant-systems-with-amazon-sqs-fair-queues/

Similar to FIFO queues, it is built around message group ids as the partitioning key.

https://github.com/ruby-shoryuken/shoryuken/wiki/FIFO-Queues

Could Shoryuken build on top of the existing FIFO queue support to detect fair queues and pass in appropriate message group ids?

[phstc]

The fairness capability will be applied automatically in all Amazon SQS standard queues for messages with the MessageGroupId property. It does not require any change in the consumer code, it has no impact on API latency, and it does not come with any throughput limitations.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html

@catkins it seems fairness is enabled to all standard queues by default. Checking this implementation, unless I'm missing something, it should work out of the box. No need for detection.

If you pass a message_group_id that should be added to the message.

Maybe we just need to update the wiki.

[catkins]

Oh nice, I think I misread the announcement post and didn't realise it was enabled by default! That makes things easy 😁

[kkumler]

@catkins it seems fairness is enabled to all standard queues by default. Checking this implementation, unless I'm missing something, it should work out of the box. No need for detection.

Hi. This portion is for FIFO queues only though? So with FIFO queues only, it would pass the :message_group_id option along from the send_message? For non-FIFO queues, this would not happen. Should :message_group_id be processed like this for all cases?

With ActiveJob integration, the only current path would be to use the set method?

[phstc]

The fairness capability will be applied automatically in all Amazon SQS standard queues for messages with the MessageGroupId property. It does not require any change in the consumer code, it has no impact on API latency, and it does not come with any throughput limitations.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html

This portion is for FIFO queues only though?

@kkumler, according to the docs, it's for standard queues. Standard queues now support message_group_id, which is used for fairness.

[kkumler]

@kkumler, according to the docs, it's for standard queues. Standard queues now support message_group_id, which is used for fairness.

@phstc Sorry, I kept the wrong part of my quote reply. I meant to reference the Shoryuken code that places the message_group_id in the right place, but only for FIFO queues?

shoryuken/lib/shoryuken/queue.rb

Lines 133 to 152 in a3af118

	def add_fifo_attributes!(options)
	return unless fifo?
	options[:message_group_id] ||= MESSAGE_GROUP_ID
	options[:message_deduplication_id] ||= Digest::SHA256.hexdigest(options[:message_body].to_s)
	options
	end
	def sanitize_message!(options)
	options = { message_body: options } if options.is_a?(String)
	if (body = options[:message_body]).is_a?(Hash)
	options[:message_body] = JSON.dump(body)
	end
	add_fifo_attributes!(options)
	options
	end

I was using the ActiveJob integration so far, so one abstraction layer out, and have not had the time to experiment further.

[phstc]

Thanks for clarifying it.

That code adds a default message group ID for FIFO if not set already ||=. Correct, for non-FIFO, that needs to be set manually.

For setting the message group ID https://github.com/ruby-shoryuken/shoryuken/wiki/FIFO-Queues#how-does-that-work