[Bug]: [benchmark] Failed to create diskann index with error:`code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported` #34222

elstic · 2024-06-27T03:00:59Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.4-20240626-07a05db5-amd64
- Deployment mode(standalone or cluster): all
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.4.4rc8
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

deploy milvus
insert data
build diskann index

test env: 4am , qa-milvus ns

fouram-disk-sta24800-6-88-6723-etcd-0                             1/1     Running                           0                4m8s    10.104.25.26    4am-node30   <none>           <none>
fouram-disk-sta24800-6-88-6723-milvus-standalone-ff46bbc9csqxtq   1/1     Running                           3 (2m22s ago)    4m8s    10.104.16.12    4am-node21   <none>           <none>
fouram-disk-sta24800-6-88-6723-minio-5477cd5c9d-4hpzh             1/1     Running                           0                4m8s    10.104.16.11    4am-node21   <none>           <none> (base.py:261)

client error log

[2024-06-26 18:15:02,242 -  INFO - fouram]: [Base] Index params of fouram_jDJJmh2l:[{'float_vector': {'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}}}] (base.py:486)
[2024-06-26 18:15:02,242 -  INFO - fouram]: [Base] Start build index of DISKANN for field:float_vector collection:fouram_jDJJmh2l, params:{'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}}, kwargs:{} (base.py:467)
[2024-06-26 18:15:02,753 - ERROR - fouram]: RPC error: [create_index], <MilvusException: (code=1, message=450737097756315587: failed to create index, C Runtime Exception: index not supported
: segcore unsupported error[segcoreCode=2003];)>, <Time:{'RPC start': '2024-06-26 18:15:02.243028', 'RPC error': '2024-06-26 18:15:02.753373'}> (decorators.py:146)
[2024-06-26 18:15:02,755 - ERROR - fouram]: (api_response) : [Index] <MilvusException: (code=1, message=450737097756315587: failed to create index, C Runtime Exception: index not supported
: segcore unsupported error[segcoreCode=2003];)>, [requestId: 0180e52c-33e8-11ef-b9ae-de58703b56af] (api_request.py:57)
[2024-06-26 18:15:02,755 - ERROR - fouram]: [CheckFunc] init_index request check failed, response:<MilvusException: (code=1, message=450737097756315587: failed to create index, C Runtime Exception: index not supported
: segcore unsupported error[segcoreCode=2003];)> (func_check.py:54)

milvus log:

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

The text was updated successfully, but these errors were encountered:

xiaocai2333 · 2024-06-27T04:03:14Z

@elstic
Did you enable diskann when building the image?

elstic · 2024-06-27T04:45:15Z

@elstic Did you enable diskann when building the image?

The image is automatically typed by pipeline, I looked for the log that typed the image.

yanliang567 · 2024-06-27T04:48:28Z

/assign @xiaocai2333
/unassign

wangting0128 · 2024-06-27T05:26:26Z

same error

argo task: multi-vector-corn-1-1719410400
test case name: test_hybrid_search_locust_dml_load_release_cluster
image:2.4-20240626-07a05db5-amd64

server:

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1719410400-3-etcd-0                           1/1     Running                           0               7m25s   10.104.34.252   4am-node37   <none>           <none>
multi-vector-corn-1-1719410400-3-etcd-1                           1/1     Running                           0               7m24s   10.104.25.74    4am-node30   <none>           <none>
multi-vector-corn-1-1719410400-3-etcd-2                           1/1     Running                           0               7m24s   10.104.20.117   4am-node22   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-datacoord-674f9f45jtsqg   1/1     Running                           4 (5m43s ago)   7m25s   10.104.13.115   4am-node16   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-datanode-856995b7dhmcsz   1/1     Running                           4 (5m49s ago)   7m25s   10.104.24.101   4am-node29   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-datanode-856995b7dqvcfh   1/1     Running                           4               7m25s   10.104.25.58    4am-node30   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-indexcoord-564fddbngtvf   1/1     Running                           0               7m25s   10.104.13.114   4am-node16   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-indexnode-555c49d4jj7kr   1/1     Running                           4 (6m1s ago)    7m25s   10.104.14.77    4am-node18   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-proxy-6569f58fc4-4l6ff    1/1     Running                           4 (5m59s ago)   7m25s   10.104.6.106    4am-node13   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-querycoord-79f9f74mltjt   1/1     Running                           4 (5m59s ago)   7m25s   10.104.6.109    4am-node13   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-querynode-55db44b6bk2wk   1/1     Running                           4 (5m45s ago)   7m25s   10.104.34.236   4am-node37   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-querynode-55db44b6sh4mj   1/1     Running                           4 (5m49s ago)   7m25s   10.104.18.224   4am-node25   <none>           <none>
multi-vector-corn-1-1719410400-3-milvus-rootcoord-79749876pn2gr   1/1     Running                           4 (6m ago)      7m25s   10.104.14.78    4am-node18   <none>           <none>
multi-vector-corn-1-1719410400-3-minio-0                          1/1     Running                           0               7m25s   10.104.34.251   4am-node37   <none>           <none>
multi-vector-corn-1-1719410400-3-minio-1                          1/1     Running                           0               7m24s   10.104.20.113   4am-node22   <none>           <none>
multi-vector-corn-1-1719410400-3-minio-2                          1/1     Running                           0               7m24s   10.104.25.75    4am-node30   <none>           <none>
multi-vector-corn-1-1719410400-3-minio-3                          1/1     Running                           0               7m23s   10.104.24.115   4am-node29   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-bookie-0                  1/1     Running                           0               7m24s   10.104.16.95    4am-node21   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-bookie-1                  1/1     Running                           0               7m23s   10.104.25.79    4am-node30   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-bookie-2                  1/1     Running                           0               7m23s   10.104.20.118   4am-node22   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-bookie-init-r2ggt         0/1     Completed                         0               7m25s   10.104.5.246    4am-node12   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-broker-0                  1/1     Running                           0               7m25s   10.104.20.104   4am-node22   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-proxy-0                   1/1     Running                           0               7m24s   10.104.5.247    4am-node12   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-pulsar-init-2cbwc         0/1     Completed                         0               7m25s   10.104.6.107    4am-node13   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-recovery-0                1/1     Running                           0               7m25s   10.104.6.108    4am-node13   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-zookeeper-0               1/1     Running                           0               7m25s   10.104.34.250   4am-node37   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-zookeeper-1               1/1     Running                           0               5m55s   10.104.20.145   4am-node22   <none>           <none>
multi-vector-corn-1-1719410400-3-pulsar-zookeeper-2               1/1     Running                           0               4m20s   10.104.17.28    4am-node23   <none>           <none>

client pod name: multi-vector-corn-1-1719410400-3362665307
client log:

test steps:

        concurrent test and calculation of RT and QPS

        :purpose:  `DML & load -> release collection`
            verify DML & load -> release collection scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 100k data
            4. flush collection
            5. build indexes again using the same params <- build diskann failed

xiaofan-luan · 2024-06-27T08:35:42Z

on build.sh

UBUNTU system build diskann index

if [ "$OS_NAME" == "ubuntu20.04" ] ; then
BUILD_DISK_ANN=ON
fi

since we upgrade to 22.04, that might be issue

wangting0128 · 2024-06-27T08:37:09Z

on build.sh

UBUNTU system build diskann index

if [ "$OS_NAME" == "ubuntu20.04" ] ; then BUILD_DISK_ANN=ON fi

since we upgrade to 22.04, that might be issue

@samhuang-z

please help check it out～

xiaofan-luan · 2024-06-27T08:40:49Z

OS_NAME=$(uname -s | tr '[:upper:]' '[:lower:]')

OS_VERSION="unknown"
DISTRO_NAME="unknown"

if [[ -f /etc/os-release ]]; then
. /etc/os-release
DISTRO_NAME=$ID
OS_VERSION=$VERSION_ID
fi

if [[ ("$DISTRO_NAME" == "ubuntu" && "$OS_VERSION" >= "20.04") || "$DISTRO_NAME" == "rocky" || "$DISTRO_NAME" == "amzn" ]]; then
BUILD_DISK_ANN=ON
fi

@samhuang-z can you help me to verify the code above work?

samhuang-z · 2024-06-27T09:37:59Z

@xiaofan-luan It's quite crazy and challenging to handle float and complex if statement in shell. how about this? make it more easy

issue: #34222 Signed-off-by: Liang Huang <[email protected]>

elstic added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Jun 27, 2024

elstic assigned yanliang567 Jun 27, 2024

kdabbir mentioned this issue Jun 27, 2024

[Bug]: Getting error "index not supported" for GPU CAGRA index, please help #34220

Open

1 task

sre-ci-robot assigned xiaocai2333 and unassigned yanliang567 Jun 27, 2024

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 27, 2024

yanliang567 added this to the 2.4.6 milestone Jun 27, 2024

samhuang-z mentioned this issue Jun 27, 2024

enhance: [skip e2e] use disk ann when os is ubuntu #34236

Merged

sre-ci-robot pushed a commit that referenced this issue Jun 27, 2024

enhance: [skip e2e] use disk ann when os is ubuntu (#34236)

b14482b

issue: #34222 Signed-off-by: Liang Huang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark] Failed to create diskann index with error:`code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported` #34222

[Bug]: [benchmark] Failed to create diskann index with error:`code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported` #34222

elstic commented Jun 27, 2024 •

edited

Loading

xiaocai2333 commented Jun 27, 2024

elstic commented Jun 27, 2024

yanliang567 commented Jun 27, 2024

wangting0128 commented Jun 27, 2024 •

edited

Loading

xiaofan-luan commented Jun 27, 2024

wangting0128 commented Jun 27, 2024 •

edited

Loading

UBUNTU system build diskann index

xiaofan-luan commented Jun 27, 2024

samhuang-z commented Jun 27, 2024

[Bug]: [benchmark] Failed to create diskann index with error:code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported #34222

[Bug]: [benchmark] Failed to create diskann index with error:code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported #34222

Comments

elstic commented Jun 27, 2024 • edited Loading

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

xiaocai2333 commented Jun 27, 2024

elstic commented Jun 27, 2024

yanliang567 commented Jun 27, 2024

wangting0128 commented Jun 27, 2024 • edited Loading

same error

xiaofan-luan commented Jun 27, 2024

UBUNTU system build diskann index

wangting0128 commented Jun 27, 2024 • edited Loading

UBUNTU system build diskann index

xiaofan-luan commented Jun 27, 2024

samhuang-z commented Jun 27, 2024

[Bug]: [benchmark] Failed to create diskann index with error:`code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported` #34222

[Bug]: [benchmark] Failed to create diskann index with error:`code=1, message= 450737097756315587: failed to create index, C Runtime Exception: index not supported` #34222

elstic commented Jun 27, 2024 •

edited

Loading

wangting0128 commented Jun 27, 2024 •

edited

Loading

wangting0128 commented Jun 27, 2024 •

edited

Loading