sophgo · HarmonyHu · Jul 10, 2026 · Jul 9, 2026 · Jul 9, 2026
diff --git a/README.md b/README.md
@@ -31,7 +31,8 @@
 
 | 日期 | 更新内容 |
 | :--- | :--- |
-| 🔥 **2026.06.30** | **MiniCPM-V-4.6** 已支持 BM1684X / BM1688，Python Demo，支持图片与视频 → [查看](./models/MiniCPMV4_6/) |
+| 🔥 **2026.07.09** | **LocateAnything-3B** 已支持 BM1684X / BM1688，Python Demo，视觉定位（box / point） → [查看](./models/LocateAnything/) |
+| **2026.06.30** | **MiniCPM-V-4.6** 已支持 BM1684X / BM1688，Python Demo，支持图片与视频 → [查看](./models/MiniCPMV4_6/) |
 | **2026.05.21** | **Gemma4** 已支持 BM1684X / BM1688，Python Demo，支持图片 / 视频 / 音频 → [查看](./models/Gemma4/) |
 | **2026.04.15** | **Qwen3.5** 已支持 BM1684X / BM1688，提供 Python 与 C++ Demo，支持图片与视频 → [查看](./models/Qwen3_5/) |
 | **2025.10.15** | **Qwen3-VL** 已支持 BM1684X / BM1688，Python / C++ Demo，支持图片与视频 → [查看](./models/Qwen3_VL/) |
@@ -98,6 +99,7 @@ cd LLM-TPU
 
 | 模型 | 支持芯片 | 一键编译 | 备注 |
 | :--- | :---: | :---: | :--- |
+| [LocateAnything-3B](https://huggingface.co/NVIDIA/LocateAnything-3B) | BM1684X / 1688 | — | Python，视觉定位 box / point |
 | [Qwen3.5](https://www.modelscope.cn/collections/Qwen/Qwen35) | BM1684X / 1688 | ✅ | Python + C++，图片 / 视频 |
 | [Qwen3-VL](https://www.modelscope.cn/models/Qwen/Qwen3-VL-4B-Instruct) | BM1684X / 1688 | ✅ | Python + C++，图片 / 视频 |
 | [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct-AWQ) | BM1684X / 1688 | ✅ | Python + C++ |
@@ -158,6 +160,7 @@ cd LLM-TPU
 [InternVL3](./models/InternVL3) ·
 [Janus-Pro](./models/Janus-Pro) ·
 [Llama3_2-Vision](./models/Llama3_2-Vision) ·
+[LocateAnything](./models/LocateAnything) ·
 [MiniCPM-V-2_6](./models/MiniCPM-V-2_6) ·
 [MiniCPMV4](./models/MiniCPMV4) ·
 [MiniCPMV4_6](./models/MiniCPMV4_6) ·

diff --git a/models/LocateAnything/README.md b/models/LocateAnything/README.md
@@ -0,0 +1,175 @@
+# LocateAnything-3B
+
+本工程实现BM1684X/BM1688部署视觉定位大模型[LocateAnything-3B](https://huggingface.co/NVIDIA/LocateAnything-3B)。通过[TPU-MLIR](https://github.com/sophgo/tpu-mlir)编译器将模型转换成bmodel，并将其部署到PCIE环境，或者SoC环境。
+
+该模型支持图片中的目标定位（visual grounding），输入自然语言描述 + 图片，输出目标的 bounding box 坐标或 point 点坐标。
+
+本文包括如何编译bmodel，和如何在BM1684X/BM1688环境运行bmodel。如何编译bmodel环节可以省去，直接用以下链接下载已编译好的bmodel（W4BF16，seq2048，max_input_length 1280，静态文本+动态ViT）：
+
+``` shell
+# BM1684X (PCIe / SoC, 1 dev)
+python3 -m dfss --url=open@sophgo.com:/ext_model_information/LLM/LLM-TPU/locateanything-3b-autoround-w4a16_w4bf16_seq2048_bm1684x_1dev_static_20260709_154503.bmodel
+# BM1688 (PCIe / SoC, 2 core)
+python3 -m dfss --url=open@sophgo.com:/ext_model_information/LLM/LLM-TPU/locateanything-3b-autoround-w4a16_w4bf16_seq2048_bm1688_2core_static_20260709_162252.bmodel
+```
+
+## 编译LLM模型
+
+此处介绍如何将LLM编译成bmodel。
+
+#### 1. 从Huggingface下载模型
+
+``` shell
+git lfs install
+git clone https://huggingface.co/groxaxo/LocateAnything-3B-AutoRound-W4A16
+```
+
+#### 2. 下载docker，启动容器
+
+``` shell
+docker pull sophgo/tpuc_dev:latest
+
+docker run --privileged --name myname1234 -v $PWD:/workspace -it sophgo/tpuc_dev:latest
+```
+后文假定环境都在docker的`/workspace`目录。
+
+#### 3. 下载`TPU-MLIR`代码并编译
+
+``` shell
+cd /workspace
+git clone git@github.com:sophgo/tpu-mlir.git
+cd tpu-mlir
+source ./envsetup.sh
+./build.sh
+```
+
+#### 4. 编译模型生成bmodel
+
+``` shell
+llm_convert.py -m /workspace/LocateAnything-3B-AutoRound-W4A16 \
+  -q w4bf16 -s 2048 --max_input_length 1280 \
+  -c bm1684x --max_pixels 896,896 --num_device 1 \
+  -o /workspace/locateanything_bmodel
+```
+
+编译参数说明：
+- `-q w4bf16`：文本LLM使用W4A16量化（AutoRound），ViT保持BF16
+- `-s 2048`：最大序列长度（KV cache 总长，输入+输出）
+- `--max_input_length 1280`：最大输入长度（图像 token + 文本 prompt），静态编译时 prefill padded 到此值
+- `--max_pixels 896,896`：最大图片分辨率（对应64×64 patch grid，1024 image token）
+- `-c bm1684x` / `-c bm1688`：目标芯片。bm1688 默认使用 2 core
+- ViT 始终动态编译，支持任意尺寸图片输入；文本模型默认静态（prefill padded 到 `max_input_length`）。如需动态文本（prefill 按实际 token 数，小图更快），编译时加 `--dynamic`
+
+## 编译与运行程序(python)
+
+支持 PCIE 和 SoC 环境。SoC 环境下直接在设备上编译库文件（链接设备自带的 libsophon）。
+
+### 1. 环境准备
+
+需要 python3.10 环境。如果不满足，参考[此文档](https://github.com/sophgo/sophon-demo/blob/release/docs/FAQ.md#13-se7%E5%AE%89%E8%A3%85python310)安装。
+
+``` shell
+sudo apt-get update
+sudo apt-get install pybind11-dev
+
+pip3 install torch==2.6.0 torchvision==0.21.0 transformers==5.7.0 \
+            pillow numpy lmdb opencv-python-headless
+```
+
+> 模型的 processor（trust_remote_code）顶层 import 了 `decord`/`lmdb`/`cv2`，transformers 5.7.0 的 `check_imports` 要求全部安装。`lmdb`、`opencv-python-headless` 在 aarch64 有 wheel；`decord` 在 aarch64（SoC）**无 PyPI wheel**：
+> - PCIe（x86）：`pip install decord` 即可。
+> - SoC（aarch64）仅图片部署时，可创建 stub 模块让 import 通过（video 路径不会执行）：
+>   ``` shell
+>   SP=$(python3.10 -c "import site;print(site.getusersitepackages())")
+>   mkdir -p "$SP" && cat > "$SP/decord.py" <<'EOF'
+>   class VideoReader:
+>       def __init__(self,*a,**k):
+>           raise ImportError("decord not installed on aarch64; video unsupported")
+>   EOF
+>   ```
+>   需要视频则从源码编译 decord（依赖 ffmpeg-dev）。
+
+### 2. 编译库文件
+
+编译C++库文件，生成`chat.cpython*.so`：
+
+``` shell
+cd python_demo
+mkdir build
+cd build && cmake .. && make && cp *cpython* .. && cd ..
+```
+
+> SoC 上若 cmake 选中了非 3.10 的默认 python（如 python3.8），显式指定：
+> `cmake -DPython_EXECUTABLE=$(which python3.10) ..`
+
+### 3. 运行demo
+
+``` shell
+python3 pipeline.py -m /path/to/model.bmodel -c ../config
+```
+
+交互模式下输入描述文本和图片路径即可获得 bounding box 坐标。也可用 `-p` 单次推理模式（见下方示例）。
+
+### 运行示例
+
+支持以下任务（prompt 模板与源模型一致）：
+
+**单目标检测 / 短语定位**
+
+``` shell
+python3 pipeline.py -m model.bmodel -c ../config -p "detect bed" --media_path test.jpg
+# token 输出：<ref>bed</ref><box><0><585><627><998></box>
+# 坐标为归一化值 [0,1000]，对应 (x1,y1,x2,y2)
+# 解析输出（像素坐标）：[bed] box (0,283)-(401,482)
+```
+
+**多目标检测 / 密集检测**
+
+对一个 `<ref>` 可输出多个 `<box>`，支持多类别（用 `</c>` 分隔）：
+
+``` shell
+python3 pipeline.py -m model.bmodel -c ../config \
+  -p "Locate all the instances that matches the following description: bed</c>window</c>pillow" \
+  --media_path test.jpg
+# 解析输出（保留 ref 关联，bed 1 框、window 1 框、pillow 16 框）：
+#   [bed] box (0,283)-(401,482)
+#   [window] box (259,36)-(394,226)
+#   [pillow] box (0,283)-(57,317)
+#   ...（共 16 个 pillow 框）
+```
+
+**点定位（Pointing）**
+
+输出 2 个坐标 (x, y)：
+
+``` shell
+python3 pipeline.py -m model.bmodel -c ../config -p "Point to: bed" --media_path test.jpg
+# token 输出：<ref>bed</ref><box><333><811></box>
+# 解析输出（像素坐标）：[bed] point (213,392)
+```
+
+> pipeline 会自动将归一化坐标 [0,1000] 解析为像素坐标（`parse_boxes`/`parse_points`/`parse_result`），并按 `<ref>` 分组打印。token 原文也会实时打印。
+
+## 技术说明
+
+- **量化方案**：文本 LLM（Qwen2.5-3B）使用 AutoRound W4A16 对称量化；ViT（MoonViT-SO-400M）和 MLP1 投影器保持 BF16
+- **推理模式**：当前仅支持 slow 模式（纯自回归），MTP 多 token 预测待后续实现
+- **批量推理**：暂不支持批量推理（源模型提供 `batch_infer.py` + `la_flash` 后端），当前仅单张图片串行推理
+- **ViT 动态编译**：支持任意图片尺寸，pipeline 动态计算 pos_emb（bicubic 插值）、2D RoPE cos/sin、merger_index 并传入 ViT bmodel
+
+### 图片 token 数计算
+
+图片先按 14×14 切 patch，再 2×2 空间合并（merge），因此每 28×28 像素块对应 1 个 image token。processor 会将图片 resize 到满足 `max_pixels` 与每维 28 像素对齐（`merge_kernel_size × patch_size = 2 × 14`）的尺寸，再计算 grid。
+
+- 公式：`image_tokens = (grid_h × grid_w) / 4`，其中 `grid_h = resized_H / 14`、`grid_w = resized_W / 14`
+- 等价地：`image_tokens ≈ 图片像素数 / 784`
+- 另有 2 个包装 token（`<img>`、`</img>`）将 image token 包住
+- `max_pixels 896,896` 时上限为 **1024 个 image token**
+
+| 图片尺寸 | grid (h×w) | image token 数 |
+|---------|-----------|---------------|
+| 640×483 | 36×46 | 414 |
+| 896×896 | 64×64 | 1024 |
+
+序列长度 `-s 2048`（KV cache 总长）需容纳：image token + 文本 prompt + 生成的回答。`--max_input_length 1280` 限制输入上限（图像 1024 + 文本），输出预算 = 2048 − 1280 = 768 token；大图会占用更多输入预算，留给生成回答的空间更少。
+
diff --git a/models/LocateAnything/config/README.md b/models/LocateAnything/config/README.md
@@ -0,0 +1,151 @@
+---
+license: other
+license_name: nvidia-license-agreement
+license_link: https://huggingface.co/nvidia/LocateAnything-3B
+tags:
+  - locateanything
+  - vision-grounding
+  - quant
+  - int4
+  - auto-round
+  - w4a16
+  - text-generation
+  - multimodal
+base_model: nvidia/LocateAnything-3B
+pipeline_tag: image-text-to-text
+---
+
+# LocateAnything-3B-AutoRound-W4A16
+
+**W4A16 (4-bit weight, 16-bit activation) INT4 quantization of [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B)** using
+Intel's [AutoRound](https://github.com/intel/auto-round) v0.13.0 (per-channel symmetric, group_size=128).
+
+This checkpoint is a **drop-in replacement** for the original BF16 model that delivers a 51% size reduction and 54% VRAM reduction with **no measurable accuracy loss** on a 50-image single-object grounding benchmark. The vision encoder (MoonViT), multimodal projector (MLP1), embedding, and `lm_head` are preserved in BF16; only the Qwen2.5-3B text decoder linears are quantized.
+
+## TL;DR
+
+| | BF16 (base) | INT4 (this) | Δ |
+|---|---|---|---|
+| Disk size | 7.3 GB | **3.55 GB** | **−51%** |
+| Runtime VRAM | 7.64 GB | **3.54 GB** | **−54%** |
+| Mean IoU (n=50) | 0.753 | 0.754 | +0.002 |
+| IoU@0.5 | 92% | **96%** | **+4 pts** |
+| Output validity | 100% | 100% | 0 |
+| Latency / call | 2.75 s | 2.61 s | −5% |
+
+**No accuracy drop.** INT4 is statistically tied with BF16 on mean IoU and is +4 points on IoU@0.5 (the "did the model find the object" metric). On letters specifically, BF16 occasionally hallucinates oversized bounding boxes due to MTP-speculation drift; INT4's quantization suppresses this drift and produces tighter, more accurate boxes on the hardest examples.
+
+## How to use
+
+The model uses custom code from the [NVIDIA/Eagle](https://github.com/NVlabs/EAGLE) `LocateAnything` repo, so `trust_remote_code=True` is required. **Use SDPA attention (not magi)** unless you are on Hopper+ — the magi backend is not available on Ampere (RTX 3090) GPUs.
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, AutoTokenizer, AutoProcessor
+from auto_round.inference import convert_hf_model
+
+MODEL = "groxaxo/LocateAnything-3B-AutoRound-W4A16"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
+processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    MODEL,
+    dtype=torch.bfloat16,
+    trust_remote_code=True,
+    attn_implementation="sdpa",
+    device_map={"": "cuda"},
+).eval()
+
+# CRITICAL: swap the standard linears for AutoRound QuantLinear runtime layers.
+convert_hf_model(model, target_device=str(model.device))
+
+image = Image.open("your_image.jpg").convert("RGB")
+question = "Please provide the bounding box of the <ref>red car</ref>."
+
+messages = [{"role": "user", "content": [
+    {"type": "image", "image": image},
+    {"type": "text", "text": question},
+]}]
+text = processor.py_apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+images, videos = processor.process_vision_info(messages)
+inputs = processor(text=[text], images=images, videos=videos, return_tensors="pt").to(model.device)
+inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
+
+with torch.no_grad():
+    out = model.generate(
+        **inputs,
+        max_new_tokens=2048,
+        do_sample=False,
+        custom_generate="locateanything",   # or omit if you use worker.predict()
+    )
+answer = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False)
+print(answer)
+```
+
+Or use the official worker directly:
+
+```python
+import sys
+sys.path.append("Eagle/Embodied")
+from locateanything_worker import LocateAnythingWorker
+worker = LocateAnythingWorker(MODEL, device="cuda", dtype=torch.bfloat16)
+convert_hf_model(worker.model, target_device=str(worker.device))
+result = worker.predict(image, question, generation_mode="hybrid")
+print(result["answer"])
+```
+
+> **Note** if you call `model.generate(...)` directly: the custom `generate` in `modeling_locateanything.py` requires both `use_cache=True` and a `tokenizer=` positional argument (it reads `tokenizer.model_max_length`). Easiest path is to use `LocateAnythingWorker.predict(...)` which sets both for you.
+
+## Quantization recipe
+
+```bash
+# Extract Qwen2.5-3B text decoder weights from the LocateAnything checkpoint
+python scripts/extract_text_decoder.py
+
+# Quantize the text decoder (252 linears → QuantLinear, 200 iters on Pile/CC)
+python scripts/quantize_text_decoder.py --profile final
+
+# Repack quantized Qwen2 + BF16 vision + BF16 projector into a full
+# LocateAnythingForConditionalGeneration checkpoint
+python scripts/repack_locateanything.py
+```
+
+- **Method**: AutoRound 0.13.0, LLM path (the MLLM path is broken for `LocateAnythingForConditionalGeneration` because the custom processor wraps the image list in a way that doesn't compose with AutoRound's hf processor)
+- **Recipe**: W4A16, symmetric, group_size=128, batch_size=8, nsamples=128, seqlen=2048, 200 iters
+- **What stays BF16**: `vision_model.*` (MoonViT, 326 tensors), `mlp1.*` (multimodal projector, 6 tensors), `language_model.lm_head`, `language_model.model.embed_tokens`, `language_model.model.norm`
+- **What gets quantized**: 36 decoder layers × 7 linears (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`) = **252 linears → 756 tensors** in the AutoRound packed format
+- **Triton fix**: downgraded `triton` from 3.3.0 to 3.2.0 to dodge `PY_SSIZE_T_CLEAN` kernel-build failures on Python 3.11
+
+### Why split-and-repack?
+
+AutoRound 0.13.0's MLLM quantization path requires the model class to be pre-registered with `AutoModelForCausalLM` and uses the standard `Processor.__call__` flow. `LocateAnythingForConditionalGeneration`'s custom processor wraps images in a way that breaks `make_list_of_images` (it receives `[[path]]` instead of `[path]`). Working around this in the calibration pipeline is fragile. The robust path is to extract the inner `Qwen2ForCausalLM` text decoder, quantize it as a plain LLM (which AutoRound handles cleanly), then merge the quantized Qwen2 weights back into a full LocateAnything checkpoint alongside the unmodified BF16 vision/projector.
+
+## Evaluation
+
+50-image synthetic benchmark (1024×1024, 25 colored solid geometric shapes + 25 black capital letters). Ground-truth bounding boxes are the model's prompt-aligned object bbox. **Same prompts sent to BF16 and INT4 in parallel across two RTX 3090s (2.76 s/iter avg, 138 s total wall)**.
+
+Per-class breakdown:
+
+| Class | n | BF16 mIoU | INT4 mIoU | BF16 IoU@0.5 | INT4 IoU@0.5 |
+|---|---|---|---|---|---|
+| Shapes | 25 | 0.955 | 0.955 | 100% | 100% |
+| Letters | 25 | 0.550 | 0.554 | 84% | **92%** |
+
+The hard cases are letters: the model often returns a box slightly larger than the tight text bbox because of MTP-speculation drift. INT4 quantization suppresses that drift and yields tighter boxes on the worst examples. See `benchmarks/results/viz/` for 12 side-by-side annotated comparisons (green = ground truth, red/blue = prediction).
+
+## Known caveats
+
+- **vLLM 0.19.1 is not supported.** `LocateAnythingForConditionalGeneration` is not in the vLLM architecture matrix and `--model-impl auto` does not pick it up. The custom mask/MagI generation path is not implemented in vLLM. Use the official Transformers worker (above) or write a vLLM plugin (out of scope here).
+- **Magi attention is not available** on Ampere GPUs (RTX 3090, A100). Use `attn_implementation="sdpa"`. On Hopper/Blackwell you can keep the original magi path for max speed.
+- The synthetic 50-image benchmark measures single-object box grounding. For multi-object detect, point queries, and hybrid generation, run your own evaluation on a real distribution (e.g. RefCOCO, D3, ReasonSeg, etc.). Smoke tests pass cleanly on all modes (`hybrid`, `slow`, `fast`, `detect`, `point`).
+
+## Credits
+
+- **Base model**: [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) — Wang et al., *"Locate Anything, Serve Anything"* ([paper](https://research.nvidia.com/labs/dvl/projects/locate_anything/))
+- **Quantization**: [Intel AutoRound 0.13.0](https://github.com/intel/auto-round)
+- **Reference implementation**: [NVlabs/EAGLE](https://github.com/NVlabs/EAGLE) (the `Eagle/Embodied/locateanything_worker.py` is the canonical inference path)
+
+## License
+
+Inherits the [NVIDIA LocateAnything license](https://huggingface.co/nvidia/LocateAnything-3B). Read the upstream terms before commercial use.