Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Batches are executed out of order when using DNS connections #2752

Open
yangbodong22011 opened this issue Jun 28, 2024 · 5 comments
Open

Comments

@yangbodong22011
Copy link

Backgroud

Our user told us that when they use batch on our cloud (AlibabaCloud), their commands came out of order. code is below

namespace RedisTest;

using StackExchange.Redis;
using System;
using System.Collections.Generic;
using System.Linq;

public class RedisClient
{
    class Program
    {
        static void Main(string[] args)
        {
            ConfigurationOptions configurationOptions = ConfigurationOptions.Parse("r-bpxxxxpd.redis.rds.aliyuncs.com:6379,password=xxx,connectTimeout=2000");
            ConnectionMultiplexer redisConn = ConnectionMultiplexer.Connect(configurationOptions);
            var db = redisConn.GetDatabase();
            var batch = db.CreateBatch();
            batch.KeyDeleteAsync("testhash"); // UNLINK
            var keyval = new List<HashEntry>(10);
            for (int i = 0; i < 10; i++) {
                keyval.Add(new HashEntry(i, i));
            }
            batch.HashSetAsync("testhash", keyval.ToArray()); // HMSET
            batch.KeyExpireAsync("testhash", TimeSpan.FromSeconds(100)); // EXPIRE
            batch.Execute();
        }
    }
}

Ideally, they would get the following sequence:

  • UNLINK
  • HMSET
  • EXPIRE
image

But sometimes, the following sequence is produced:

  • UNLINK
  • EXPIRE
  • HMSET
Wireshark 2024-06-28 21 26 46

Going a step further, we found that UNLINK and EXPIRE are sent over one TCP connection, but HMSET uses another TCP connection.

Why are there two links? We found that when connecting through a domain name, there will be two links in ConnectionMultiplexer#ServerSnapshot (one is the domain name, and the other is the IP resolved by the domain name (obtained through cluster nodes)). Therefore, two links will be returned randomly in AnyServer, causing commands to be assigned to different connections.

image

How to reproduction

  1. Find a server with DNS (I can provide Alibaba Cloud test environment for free)
  2. Run the above command

How to fix

During the initialization process, Lettuce will generate the result of the domain name as a URI, but will change the domain name to an alias. Please refer to redis/lettuce@16f9e75

Version

  • StackExchange.Redis: latest
  • .Net: I reproduced this issue under 6.0 and 8.0, but it has nothing to do with the .net version.
@shaofing
Copy link

shaofing commented Jul 2, 2024

@mgravell Can you fix it ?

@mgravell
Copy link
Collaborator

mgravell commented Jul 2, 2024

This is ... odd; short term, switching up to transactions will enforce single connection, but... this is odd; I'd have to investigate, which I can add to the list, but I can't guarantee a "today" thing

@worming004
Copy link

worming004 commented Jul 2, 2024

As both theses lines are not awaited

batch.HashSetAsync("testhash", keyval.ToArray()); // HMSET
batch.KeyExpireAsync("testhash", TimeSpan.FromSeconds(100)); // EXPIRE

There is no garantee of order. Just probability that line 1 would be executed earlier, but not garantee. Right ?

Edit: didn't see it is about batch. My bad, my remark is irrelevant

@mgravell
Copy link
Collaborator

mgravell commented Jul 2, 2024

It is reasonable to expect this to occur in order; what is described is definitely not "working as intended", but it doesn't sound trivial - it will need some investigation

@yangbodong22011
Copy link
Author

yangbodong22011 commented Jul 3, 2024

@mgravell I sent you the account and password of the test instance environment via google mail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants