Issues in getting sctp multi homing instance to work .. #922

megarajan · 2025-01-06T12:37:21Z

megarajan
Jan 6, 2025

I have created a sctp multi homing loxilb instance in fullnat mode with this config.

Ip : 10.77.87.205
sec ips : 10.77.87.201, 10.77.87.202,10.77.87.203
endpoint ip : 10.77.87.181
endpoint port : 3869

root@pn-distributor-3:/# loxicmd get lb -o wide

EXT IP	SEC IPS	PORT	PROTO	NAME	MARK	SEL	MODE	ENDPOINT	EPORT	WEIGHT	STATE	COUNTERS
10.77.87.205	10.77.87.201, 10.77.87.202,	2020	sctp	sctpmh	0	rr	fullnat	10.77.87.181	3869	1	-	3900:277296
	10.77.87.203
root@pn-distributor-3:/#

when we send messages to the 10.77.87.205 ip , the connections are getting aborted. a sample tcpdump is below.

12:30:20.468526 IP 10.77.87.87.33682 > 10.77.87.205.2020: sctp (1) [INIT] [init tag: 196020939] [rwnd: 8388608] [OS: 10] [MIS: 65535] [init TSN: 840558473]
12:30:20.468928 IP 10.77.87.205.2020 > 10.77.87.87.33682: sctp (1) [ABORT]
12:30:25.467253 IP 10.77.87.87.33682 > 10.77.87.205.2020: sctp (1) [INIT] [init tag: 3870108015] [rwnd: 8388608] [OS: 10] [MIS: 65535] [init TSN: 473674365]
12:30:25.467836 IP 10.77.87.205.2020 > 10.77.87.87.33682: sctp (1) [ABORT]
12:30:30.465660 IP 10.77.87.87.33682 > 10.77.87.205.2020: sctp (1) [INIT] [init tag: 1360199288] [rwnd: 8388608] [OS: 10] [MIS: 65535] [init TSN: 2930406781]
12:30:30.466264 IP 10.77.87.205.2020 > 10.77.87.87.33682: sctp (1) [ABORT]

Could you please share any pointers on why this could be happening ?

I see that the counters are getting incremented in the get lb command output.

thanks

Answered by TrekkieCoder

Jan 17, 2025

Thanks @megarajan ,

I quickly tested your scenario. It works fine. My steps are as follows :

Disable ipv6 in client, loxilb and server VMs
Use the following sysctls in client/server (optional for now)

sysctl net.ipv4.conf.<intf-name>.arp_accept=1
sysctl net.ipv4.conf.default.arp_accept=1

Create LB rule in loxilb docker as follows:

loxicmd create lb 10.77.87.205 --name=sctpmh1 --secips=10.77.87.201,10.77.87.202,10.77.87.203 --sctp=2020:9999 --endpoints=10.77.87.181:1 --mode=fullnat

Set "MASTER" instance state of loxilb docker from loxilb VM

curl -X 'POST'   'http://127.0.0.1:11111/netlox/v1/config/cistate'   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '…

View full answer

megarajan · 2025-01-06T12:46:23Z

megarajan
Jan 6, 2025
Author

I created a loxilb in hostmode like below

docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged -dit -v /var/run/docker.sock:/var/run/docker.sock -v /dev/log:/dev/log --name loxilb-host-mode --network host 1a8df5eae3fa98df3e3a45e4c1417c3cbd36ad2d0ad711d8137f0930d50f187f

even though I have created a lb with 2020 port and in host mode, I dont see 2020 port in listening mode ..

cps@pn-distributor-3:$ sudo netstat -anp | grep 2020
cps@pn-distributor-3:$

0 replies

megarajan · 2025-01-07T05:45:18Z

megarajan
Jan 7, 2025
Author

hi any clues or pointers ?

0 replies

megarajan · 2025-01-08T04:45:17Z

megarajan
Jan 8, 2025
Author

hi team , any pointers ?

0 replies

TrekkieCoder · 2025-01-08T04:59:57Z

TrekkieCoder
Jan 8, 2025
Maintainer

Hi @megarajan,

Loxilb does not listen on 2020 port because things are implemented with eBPF. Not sure why your multi-homing setup is not working. Our sctp-mh cicd works fine. I will take a look and get back.

4 replies

megarajan Jan 17, 2025
Author

Hi @megarajan,

Loxilb does not listen on 2020 port because things are implemented with eBPF. Not sure why your multi-homing setup is not working. Our sctp-mh cicd works fine. I will take a look and get back.

were you able to take a look ? JFYI , I went thru the sctp-mh cicd tests and they worked fine for me as well.

TrekkieCoder Jan 17, 2025
Maintainer

I will give it a try soon. Can you please tell what OS/Kernel/Hypervisor you are using ? Any other info which might make the test as close to yours would be appreciated !

megarajan Jan 17, 2025
Author

I will give it a try soon. Can you please tell what OS/Kernel/Hypervisor you are using ? Any other info which might make the test as close to yours would be appreciated !

I run loxilb as a docker container in Ubuntu 20.04.6 LTS

docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged -dit -v /var/run/docker.sock:/var/run/docker.sock -v /dev/log:/dev/log --name loxilb-host-mode --network host 1a8df5eae3fa98df3e3a45e4c1417c3cbd36ad2d0ad711d8137f0930d50f187f

and the docker inspect for reference is

cps@pn-distributor-3:~$ docker inspect 1a8df5eae3fa98df3e3a45e4c1417c3cbd36ad2d0ad711d8137f0930d50f187f
[
{
"Id": "sha256:1a8df5eae3fa98df3e3a45e4c1417c3cbd36ad2d0ad711d8137f0930d50f187f",
"RepoTags": [
"1a8df5eae3fa:httpecho"
],
"RepoDigests": [],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-11-08T02:43:28.060170606Z",
"Container": "",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": null,
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"11111/tcp": {},
"22222/tcp": {},
"3784/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin",
"LD_LIBRARY_PATH=:/usr/lib64/"
],
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/root/loxilb-io/loxilb/loxilb"
],
"OnBuild": null,
"Labels": {
"description": "loxilb official docker image",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.version": "20.04"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 333285767,
"VirtualSize": 333285767,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/4e1439e7d2c01d8829b2000eb773497280119e0f984aaf357c1b37a481ca2654/diff:/var/lib/docker/overlay2/b3150e3ece209d495767912b7b67b5863e07bd7e83a8dcbf95e8bb7ad0333990/diff:/var/lib/docker/overlay2/c038e32ac7cd3b17f4436719b2a6ca0a06120938f3f286b3b1616fd61960df98/diff:/var/lib/docker/overlay2/d15777994736b563497ce0b17d815fd27a5a9a5b5d6793189a17ec9dd8a49d0d/diff:/var/lib/docker/overlay2/0967b8b079d4d5e90a6c3a8da84480c617dddb70669fac5a8249cd1cf5d7c588/diff:/var/lib/docker/overlay2/e14451eefeb841da5d0ff52a473d2d0f6590598b5d71d43222c415bad544bfce/diff:/var/lib/docker/overlay2/d48970b870b328f0995e64a5af2ce51ba9dfefef45e7da68a563aeeee9cb5968/diff:/var/lib/docker/overlay2/3e66f7e6948799c3177dd5d938e70665beeeae09ae273eadbe0b1ded02939467/diff:/var/lib/docker/overlay2/bab1b9606a4d471f9a2a58be5b71e11b047dd45940aaaad6f014483c00795eb1/diff:/var/lib/docker/overlay2/8ba3f25bc44c262ae5826f8b2201303a16fb3fa639afea24e32d950859f2db63/diff:/var/lib/docker/overlay2/60e0e1afc52df2cd28d14478591ed555448dd8d5c80354fff237401ef80fff62/diff:/var/lib/docker/overlay2/8a78852df3fd543268a2c94333334c715ccee24fed68be8087678513ffb190bf/diff:/var/lib/docker/overlay2/30c3904c5c938eb66ba7664cf6e8333b7697a8af7a8e3e0d7434dd1fb260c3cc/diff:/var/lib/docker/overlay2/9cf1b7c1ae0a39adc114964077ad6c70b89a8a4a4831ad66d3cd5bc20ff3dd96/diff",
"MergedDir": "/var/lib/docker/overlay2/7a1355f0adc7acffc199d3d63a88607e335c33e2272ca8f623d7b7cdc33b5135/merged",
"UpperDir": "/var/lib/docker/overlay2/7a1355f0adc7acffc199d3d63a88607e335c33e2272ca8f623d7b7cdc33b5135/diff",
"WorkDir": "/var/lib/docker/overlay2/7a1355f0adc7acffc199d3d63a88607e335c33e2272ca8f623d7b7cdc33b5135/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:fffe76c64ef2dee2d80a8bb3ad13d65d596d04a45510b1956a976a69215dae92",
"sha256:d1a14aef2429e4b4e17ca9654e206cfff71cb8304182307285d2917b91be6098",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
"sha256:9a117eaf0a410ea27d09a97483370388434d6fd29e52bb9eab384346f223c58d",
"sha256:437cb1a37227fabc1568f2bf0a294374907d575c522ba40408789fabb29349ff",
"sha256:33550c6a467b1fadcc3219a088ab664c5bf1cab7c1e499cc0734c4743654459d",
"sha256:a003c9310ad39915bb17c38d3d46a9f9e0fb04057985fcf79b05016fe9f35a47",
"sha256:ec9838fe61bb84c0662190291f9bdb1f7f5a6dd338ebedf82ebb396b9a5ae36b",
"sha256:bf0249f18f664b9200790da52859efde69cd5f7579373969c47d6094e4ecbc17",
"sha256:65073110a00fd7b7ba167925a7a65598ac40b55aa204094bc6f8b2e7d92d11d9",
"sha256:6a0926735bd2558fdc5665520c7a1c2777110eb8ebd9ae06599a322167347c54",
"sha256:7d1755d41733e1f537a9b9fecb45b8b1e0d8b4216dffdac483b86d00b55f6a5a",
"sha256:88c8c1896324348e2d2957aaa1bdbb15e6c4da2e5c8181e525fc84277c6d11d8",
"sha256:90e2d073c6f3d35fd893364f545f875012d34fbd289873496c6384edbe930795",
"sha256:279a49a2cf0c20bd5c7561005b4ffa19f72c3c71ab84fdd94f0e6e92967df08e"
]
},
"Metadata": {
"LastTagTime": "2024-11-18T12:21:56.911787077Z"
}
}
]

TrekkieCoder Jan 17, 2025
Maintainer

Thanks @megarajan ,

I quickly tested your scenario. It works fine. My steps are as follows :

Disable ipv6 in client, loxilb and server VMs
Use the following sysctls in client/server (optional for now)

sysctl net.ipv4.conf.<intf-name>.arp_accept=1
sysctl net.ipv4.conf.default.arp_accept=1

Create LB rule in loxilb docker as follows:

loxicmd create lb 10.77.87.205 --name=sctpmh1 --secips=10.77.87.201,10.77.87.202,10.77.87.203 --sctp=2020:9999 --endpoints=10.77.87.181:1 --mode=fullnat

Set "MASTER" instance state of loxilb docker from loxilb VM

curl -X 'POST'   'http://127.0.0.1:11111/netlox/v1/config/cistate'   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '{
  "instance": "llb-inst0",
  "state": "MASTER",
  "vip": "0.0.0.0"
}'

Instance name should be "default" instead of "llb-inst0" for little older (than a month) versions of loxilb.
This will make sure secondary IPs are added to the system. You can check if they are added using ip addr show dev lo in loxilb VM.

Run server app (using sctp_darn)

sctp_darn -H 0.0.0.0  -P 9999 -l

Run client app (using sctp_darn)

sctp_darn -H 10.77.87.200 -h 10.77.87.205 -p 2020 -s

10.77.87.200 is the client VM's IP address

Answer selected by UltraInstinct14

megarajan · 2025-01-08T05:17:46Z

megarajan
Jan 8, 2025
Author

thanks for the reponse, have sent a request to join the community you pointed .

waiting for your detailed response..

0 replies

megarajan · 2025-01-08T05:27:32Z

megarajan
Jan 8, 2025
Author

to give more background on our test setup, we have 3 linux ubuntu vms

client vm ( running sctp traffic sending app) -> loxilb vm -> endpoint vm ( running app to process to sctp traffic )

0 replies

my102022 · 2025-01-08T05:30:31Z

my102022
Jan 8, 2025

Can you check if ipv6 is enabled. If so please disable

…

On Wed, Jan 8, 2025, 2:27 PM megarajan ***@***.***> wrote: to give more background on our test setup, we have 3 linux ubuntu vms client vm ( running sctp traffic sending app) -> loxilb vm -> endpoint vm ( running app to process to sctp traffic ) — Reply to this email directly, view it on GitHub <#922 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMXPDCB4ZBAYZWNBRVPK6V32JSZNZAVCNFSM6AAAAABUVQ2GM6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZWHA4DKNA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

2 replies

megarajan Jan 8, 2025
Author

ok will check and try

megarajan Jan 17, 2025
Author

JFYI , disabling ipv6 didnt help in my case

megarajan · 2025-01-20T05:13:53Z

megarajan
Jan 20, 2025
Author

@TrekkieCoder ,

Thank you very much for the support , as you requested in the server running our sctp app and is tagged as a endpoint for loxilb , I see this errors

cat /proc/net/sctp/snmp
SctpCurrEstab 0
SctpActiveEstabs 0
SctpPassiveEstabs 0
SctpAborteds 0
SctpShutdowns 0
SctpOutOfBlues 89877
SctpChecksumErrors 0
SctpOutCtrlChunks 89877
SctpOutOrderChunks 0
SctpOutUnorderChunks 0
SctpInCtrlChunks 89877
SctpInOrderChunks 0
SctpInUnorderChunks 0
SctpFragUsrMsgs 0
SctpReasmUsrMsgs 0
SctpOutSCTPPacks 89877
SctpInSCTPPacks 89877
SctpT1InitExpireds 0
SctpT1CookieExpireds 0
SctpT2ShutdownExpireds 0
SctpT3RtxExpireds 0
SctpT4RtoExpireds 0
SctpT5ShutdownGuardExpireds 0
SctpDelaySackExpireds 0
SctpAutocloseExpireds 0
SctpT3Retransmits 0
SctpPmtudRetransmits 0
SctpFastRetransmits 0
SctpInPktSoftirq 89877
SctpInPktBacklog 0
SctpInPktDiscards 89877
SctpInDataChunkDiscards 0

but in other servers which are not loxilb endpoints I dont see these errors at all.

director-1:$ cat /proc/net/sctp/snmp
SctpCurrEstab 0
SctpActiveEstabs 0
SctpPassiveEstabs 0
SctpAborteds 0
SctpShutdowns 0
SctpOutOfBlues 0
SctpChecksumErrors 0
SctpOutCtrlChunks 0
SctpOutOrderChunks 0
SctpOutUnorderChunks 0
SctpInCtrlChunks 0
SctpInOrderChunks 0
SctpInUnorderChunks 0
SctpFragUsrMsgs 0
SctpReasmUsrMsgs 0
SctpOutSCTPPacks 0
SctpInSCTPPacks 0
SctpT1InitExpireds 0
SctpT1CookieExpireds 0
SctpT2ShutdownExpireds 0
SctpT3RtxExpireds 0
SctpT4RtoExpireds 0
SctpT5ShutdownGuardExpireds 0
SctpDelaySackExpireds 0
SctpAutocloseExpireds 0
SctpT3Retransmits 0
SctpPmtudRetransmits 0
SctpFastRetransmits 0
SctpInPktSoftirq 0
SctpInPktBacklog 0
SctpInPktDiscards 0
SctpInDataChunkDiscards 0
cps@pn-director-1:$

0 replies

megarajan · 2025-01-20T05:14:17Z

megarajan
Jan 20, 2025
Author

as you requested, will check our server app and will update you on the result

0 replies

megarajan · 2025-01-20T16:42:22Z

megarajan
Jan 20, 2025
Author

@TrekkieCoder , You were right, the issue was at our server end, and now things are working as expected

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues in getting sctp multi homing instance to work .. #922

{{title}}

Replies: 10 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issues in getting sctp multi homing instance to work .. #922

megarajan Jan 6, 2025

Replies: 10 comments · 6 replies

megarajan Jan 6, 2025 Author

megarajan Jan 7, 2025 Author

megarajan Jan 8, 2025 Author

TrekkieCoder Jan 8, 2025 Maintainer

megarajan Jan 17, 2025 Author

TrekkieCoder Jan 17, 2025 Maintainer

megarajan Jan 17, 2025 Author

TrekkieCoder Jan 17, 2025 Maintainer

megarajan Jan 8, 2025 Author

megarajan Jan 8, 2025 Author

my102022 Jan 8, 2025

megarajan Jan 8, 2025 Author

megarajan Jan 17, 2025 Author

megarajan Jan 20, 2025 Author

megarajan Jan 20, 2025 Author

megarajan Jan 20, 2025 Author

megarajan
Jan 6, 2025

Replies: 10 comments 6 replies

megarajan
Jan 6, 2025
Author

megarajan
Jan 7, 2025
Author

megarajan
Jan 8, 2025
Author

TrekkieCoder
Jan 8, 2025
Maintainer

megarajan Jan 17, 2025
Author

TrekkieCoder Jan 17, 2025
Maintainer

megarajan Jan 17, 2025
Author

TrekkieCoder Jan 17, 2025
Maintainer

megarajan
Jan 8, 2025
Author

megarajan
Jan 8, 2025
Author

my102022
Jan 8, 2025

megarajan Jan 8, 2025
Author

megarajan Jan 17, 2025
Author

megarajan
Jan 20, 2025
Author

megarajan
Jan 20, 2025
Author

megarajan
Jan 20, 2025
Author