BUG: Installing Numpy and Pandas with conda-forge channel may leads to compilation issues

[ChengjieLi28]

Note that the issue tracker is NOT the place for general support. For
discussions about development, questions about usage, or any general questions,
contact us on https://discuss.xorbits.io/.

In a recent xorbits CD, it was found that there were problems compiling xorbits if the numpy version installed was 1.24.3 resolved by conda-forge.