Plans to support sglang

[ziming-zh]

Hi,

This seems to be a very promising and important feature for large-batch throughput oriented llm inference. Just want to check if there are any plans (or ongoing effort) to support this in sgLang/vLLM? Thanks!

[sleepwalker2017]

also curious about that. @SwiftSeal03

[SwiftSeal03]

We have the plan and contact at vLLM team, but just could not find person to implement it, because it would involve considerable efforts and understanding of vLLM internals. So please let us know if you are interested and want to help integrate and do more research onwards.

[sleepwalker2017]

We have the plan and contact at vLLM team, but just could not find person to implement it, because it would involve considerable efforts and understanding of vLLM internals. So please let us know if you are interested and want to help integrate and do more research onwards.

Hi, I'm very interested in this work. And I'm trying to run this code. But I got a lot of issues.

pacpu# bash build.sh llama2_13b 1

...
In file included from /data/NEO/pacpu/pacpu.cpp:5:
/data/NEO/pacpu/core.h:15:3: error: ‘data_t’ has not been declared
   15 |   data_t k[],
      |   ^~~~~~

Do I need to specify some Macros to compile it?

[sleepwalker2017]

We have the plan and contact at vLLM team, but just could not find person to implement it, because it would involve considerable efforts and understanding of vLLM internals. So please let us know if you are interested and want to help integrate and do more research onwards.

I'm not familiar with ISPC, after I manually add typedef __fp16 data_t;in dtype.h, I get this error:

[ 33%] Building ISPC object CMakeFiles/pacpu-llama2_13b-tp1.dir/pacpu.ispc.o
Warning: No --target specified on command-line. Using default system target "avx2-i32x8".
/data//NEO/pacpu/dtype.h:12:16: Error: syntax error, unexpected identifier, expecting ',' or ';'.
typedef __fp16 data_t;
               ^^^^^^

/data//NEO/pacpu/pacpu.ispc:10:17: Warning: No type specified in declaration. Assuming int32.
  const uniform data_t q[], // [NUM_Q_HEADS, HEAD_DIM]
                ^^^^^^

/data//NEO/pacpu/pacpu.ispc:10:24: Error: syntax error, unexpected identifier, expecting ')'.
  const uniform data_t q[], // [NUM_Q_HEADS, HEAD_DIM]
                       ^

/data//NEO/pacpu/pacpu.ispc:16:22: Error: Undeclared symbol "seq_len".
  uniform int imax = seq_len / BLOCK_SIZE + 1;
                     ^^^^^^^

/data//NEO/pacpu/pacpu.ispc:18:26: Error: syntax error, unexpected '*', expecting ',' or ';'.
    const uniform data_t * k = k_cache +
                         ^

/data//NEO/pacpu/pacpu.ispc:24:34: Error: Undeclared symbol "seq_len".
      uniform int tmax = min(BLOCK_SIZE, seq_len - i * BLOCK_SIZE);
                                 ^^^^^^^

[SwiftSeal03]

May I know the versions of the your g++ and ISPC compilers?

[sleepwalker2017]

May I know the versions of the your g++ and ISPC compilers?

g++ --version
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

ispc --version
Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.21.0 (build commit f925f2563441a36e @ 20230818, LLVM 15.0.7)

Is that ok ?

[SwiftSeal03]

You will need g++-13 to have fp16 support.