[c++][python-package] add int8 input dtype for pre-discretized features

[kyo219]

Problem

When feature values are small discrete integers (e.g. 0–4), LightGBM's
Python wrapper forces a copy to float32 before passing them to the C++
layer. This quadruples memory usage for the feature matrix even though
LightGBM immediately bins the values into uint8/16/32 indices internally.

Solution

Add C_API_DTYPE_INT8 so that np.int8 arrays flow through to
RowFunctionFromDenseMatrix without any intermediate copy.

Changes

• include/LightGBM/c_api.h — new C_API_DTYPE_INT8 (4) constant
• src/c_api.cpp — int8_t case in RowFunctionFromDenseMatrix
• python-package/lightgbm/basic.py — accept np.int8 in
  _np2d_to_np1d() and _c_float_array()

Benchmark (500 K rows × 2 000 cols, int8 values 0–4)

Metric	int8	float32
Feature array memory	954 MB	3.73 GB
Prediction results	identical (max	diff

Training / prediction speed is unchanged — the saving is in the
Python → C++ data hand-off, not in the tree-building itself.

[jameslamb]

Thanks for your interest in LightGBM.

Before we consider this, please add some unit tests covering this new code.

[kyo219]

Thanks for the review, @jameslamb!

I've added unit tests in tests/python_package_test/test_basic.py to cover the new int8 path:

• test_no_copy_in_dataset_from_numpy_2d: extended the existing parametrization to include int8, verifying that _np2d_to_np1d forwards int8 arrays without a copy (same as float32, unlike int64 which still copies).
• test_c_float_array_accepts_int8: confirms _c_float_array returns _C_API_DTYPE_INT8 for np.int8 input and reuses the underlying buffer (no float conversion).
• test_c_float_array_rejects_unsupported_int_dtypes: parametrized over int16, int32, int64, and uint8 to ensure unsupported integer dtypes still raise the expected TypeError.
• test_dataset_from_int8_matches_float32: end-to-end check showing that a Dataset built from np.int8 produces byte-identical _dump_text output compared to an equivalent np.float32 input, confirming consistent binning via the RowFunctionFromDenseMatrix int8 path.

All 119 tests in test_basic.py pass locally against a freshly built lib_lightgbm.

Let me know if you'd like additional coverage (e.g. CSR/CSC paths or prediction-time int8 inputs).

[jameslamb]

Thanks, I've started a CI run. Let's see how that goes before we review further.

[kyo219]

@BelixRogner
Thanks! Really happy to hear it!
I’ll do my best to help get it merged into LightGBM as well. haha

[kyo219]

@jameslamb Thanks in advance for taking a look!

I pushed a small fix here (3739a04): the int8 input support made _c_float_array return a POINTER(c_int8), but the _ctypes_float_ptr type alias only listed c_float/c_double, so mypy was failing at basic.py:767. I added ctypes._Pointer[ctypes.c_int8] to that Union, and mypy now passes locally with the repo config. The lint check is green again.

The two remaining red checks both look like unrelated infrastructure flakes, not something caused by my one-line type-annotation change:

Python-package / regular (ubuntu22.04, clang-17, Python 3.13)

• All tests passed — 1624 passed, 19 skipped, 12 xfailed, 4 xpassed.
• It failed after the tests, in a post-test step: ModuleNotFoundError: No module named 'conda', alongside ln: failed to create symbolic link '/etc/resolv.conf': Device or resource busy and a dbus socket error — looks like an unhealthy runner.

R-package / r-package (macos-15-intel, clang, R 4.3, cran)

• Failed during TeX setup, before the package was even built: tlmgr: TLPDB::from_file could not get texlive.tlpdb from https://mirror.latigo.net/ctan/.../texlive.tlpdb — a CTAN mirror network failure.

Both of these jobs passed on the previous run of this PR, so they don't seem deterministic. Since I don't have permission to re-run jobs on the upstream repo, would you mind re-running them (or the whole workflow) when you get a chance? Happy to push an empty commit to re-trigger everything if you'd prefer. Thanks!