Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull latest main into GOAT #307

Draft
wants to merge 127 commits into
base: goat
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
4ce19a9
add test config paths for remote eval (#287)
yvsriram Jul 17, 2023
156200c
Remove episode argument in agent reset [needs hw test] (#285)
yvsriram Jul 17, 2023
d2e1157
Refactor: Remove episode argument in agent reset (#289)
rpartsey Jul 18, 2023
3b25bdf
SLAP, PerAct baseline and Execution Code on Stretch for longitudinal …
zephirefaith Jul 18, 2023
79732fd
Offline instance mapping (#290)
yvsriram Jul 18, 2023
26f5f9c
updated docs/challenge.md (#291)
rpartsey Jul 19, 2023
e2769aa
update discrete step, rendering for eval (#292)
yvsriram Jul 19, 2023
0dc5079
Add MaskRCNN perception module (#296)
theophilegervet Jul 24, 2023
50824db
add some notes on ROS setup on the workstation (#295)
cpaxton Jul 24, 2023
7c0a4dc
Automatic install script for downloading data (#299)
cpaxton Jul 25, 2023
0b8d6bc
added receptacles data collection script (#293)
anuragajay Jul 26, 2023
8dc5cbf
Baseline agent Dockerfile update (#298)
rpartsey Jul 26, 2023
cb90554
fix a broken readme link in the docs (#302)
cpaxton Jul 26, 2023
95c2f75
Add install script for first-time users (#304)
cpaxton Jul 26, 2023
635534f
Conditionally import habitat dependencies (#303)
cpaxton Jul 27, 2023
a5732be
Fix license headers (#309)
cpaxton Jul 27, 2023
b98bd05
docs: fix link to training DD-PPO (#312)
acmiyaguchi Jul 31, 2023
c12b562
Add Grounded mobile sam for perception (#294)
yvsriram Aug 1, 2023
1ae2f48
Rewrite installation instructions (#308)
cpaxton Aug 1, 2023
e8e3425
Minor readme fixes missed in the last PR (#314)
cpaxton Aug 1, 2023
90bef70
Add specific phase commands since they are misleading (#326)
cpaxton Aug 7, 2023
6174367
Fix bug with agent evaluation (#327)
cpaxton Aug 7, 2023
4e31491
Extra note about python (#331)
cpaxton Aug 9, 2023
647f997
added ubuntu20.04 base challenge Docker image; updated challenge read…
rpartsey Aug 14, 2023
28149f5
add README for installing perception modules (#315)
yvsriram Aug 14, 2023
2972734
minor fix info is not Nnne (#337)
yvsriram Aug 15, 2023
50780f7
Add quality of life checks to make sure robots are homed and initaliz…
keke-220 Aug 16, 2023
b643698
Updates for fixing head tilt + motion in smaller kitchen (#241)
cpaxton Aug 16, 2023
7c6d099
Fix conda cuda incompatibility and upgrade to torch 2.0.1 and cuda 11.8
Aug 18, 2023
744b06f
Instance association using IoU of 3d bounding boxes (#342)
yvsriram Aug 24, 2023
74910f4
ScanNet data creation, dataset, and visualization (#343)
alexsax Aug 24, 2023
cb6c025
allow rl eval to use different discrete step (#345)
yvsriram Aug 25, 2023
aa854b6
Prototype sampling-based motion planning + 3d representations (#323)
cpaxton Aug 25, 2023
f413405
adds latest train/val episodes (#348)
yvsriram Aug 28, 2023
a7663d6
Try 2.0.0
cpaxton Aug 29, 2023
5146dca
Merge branch 'main' into torch201cu118
cpaxton Aug 29, 2023
ab978cc
Add referit3d and scanrefer to scannet dataset (#346)
alexsax Aug 30, 2023
56ccf2e
Updated category mapping in episodes (#359)
yvsriram Sep 1, 2023
76154c9
Merge branch 'main' into torch201cu118
cpaxton Sep 7, 2023
a0e4a24
branch tweaks
cpaxton Sep 7, 2023
fb2255b
cleanup env code
cpaxton Sep 7, 2023
9fcb2f5
add some deps
cpaxton Sep 7, 2023
1ddaa99
remove pytorch
cpaxton Sep 7, 2023
02645f1
fixing some issues installing branch with opencv
cpaxton Sep 7, 2023
ca25dc9
try adding pyg
cpaxton Sep 7, 2023
219b898
remove cluster install
cpaxton Sep 7, 2023
c377e5f
add pin so that we can also download boost
cpaxton Sep 7, 2023
912eb36
Add RRT motion planner which integrates into 3d world representations…
cpaxton Sep 8, 2023
3c2241a
changes for binit
cpaxton Sep 8, 2023
0c9b069
force update ovmm scenes directory (#377)
yvsriram Sep 9, 2023
95477ab
Update docker file to v0.2 (#370)
cpaxton Sep 11, 2023
2ba2761
update code
cpaxton Sep 11, 2023
bdfcca7
Merge branch 'main' into torch201cu118
cpaxton Sep 12, 2023
645a338
use minival episodes for local docker evaluations (#378)
yvsriram Sep 14, 2023
ff707f3
remove unused code
cpaxton Sep 18, 2023
6eebf7a
Add object det and referring expression det on ScanNet (#358)
alexsax Sep 19, 2023
8759cba
Merge branch 'main' of github.com:facebookresearch/home-robot into cp…
cpaxton Sep 19, 2023
ee8cb38
fix in sparse voxel map code
cpaxton Sep 19, 2023
0b74f64
Fix issues with pkl files and add geodesic distance stuff
cpaxton Sep 20, 2023
369d1ed
planner fixes
cpaxton Sep 20, 2023
87346e7
update code to switch to yield loops
cpaxton Sep 20, 2023
cd110b9
need exploration to be a bit better
cpaxton Sep 20, 2023
428943f
update voxel code and move it around a bit
cpaxton Sep 20, 2023
975e3d6
mprove 3d map setup
cpaxton Sep 20, 2023
48de5fe
explore -> run_explore
cpaxton Sep 20, 2023
b164e41
update motion planning code
cpaxton Sep 20, 2023
13c9bfc
add reasons for failure
cpaxton Sep 20, 2023
c01fd73
bug fix when no goals left
cpaxton Sep 20, 2023
09dd598
final fixes
cpaxton Sep 20, 2023
e9e85fd
fix an issue with trajectory following
cpaxton Sep 20, 2023
ff52e10
update fmm planner code
cpaxton Sep 21, 2023
db04ab1
update some things and let us make exploration a tiny bit more careful
cpaxton Sep 21, 2023
8d60180
disable verbose outputs by default
cpaxton Sep 21, 2023
ab9d89b
Merge pull request #380 from facebookresearch/cpaxton/voxel-fmm-front…
cpaxton Sep 21, 2023
9f2f333
scannet dataset can pick the number of classes to use and referring e…
alexsax Sep 22, 2023
7e2251e
Installable project that gives access to the continual stretch enviro…
SamNPowers Sep 26, 2023
23f653e
Merge branch 'main' of github.com:facebookresearch/home-robot into to…
cpaxton Sep 26, 2023
bc5655a
updates to setup
cpaxton Sep 26, 2023
90095e5
update some issues
cpaxton Sep 26, 2023
c14c5df
fix minor typos
yvsriram Sep 26, 2023
a8762d1
add back CLIP dependency
yvsriram Sep 27, 2023
61f18d8
get 2d association for IM working for OVMM, add other fixes from GOAT…
yvsriram Sep 27, 2023
d43b825
update README
cpaxton Sep 27, 2023
c6b1943
Merge branch 'cpaxton/cuda-install' of github.com:facebookresearch/ho…
cpaxton Sep 27, 2023
601debd
Merge pull request #395 from facebookresearch/cpaxton/cuda-install
cpaxton Sep 28, 2023
5f9fcca
Merge branch 'main' of https://github.com/facebookresearch/home-robot…
yvsriram Sep 29, 2023
3ce6e45
move interpolate_image to under utils file
yvsriram Sep 29, 2023
89b212f
Merge pull request #399 from facebookresearch/yvsriram/ovmm_im_fixes
yvsriram Sep 29, 2023
8c38519
correcly drop instances whose class not in subset (#401)
alexsax Oct 3, 2023
77746a7
OVMM v0.2: Upgrade habitat-lab and habitat-sim to v0.2.5 (#392)
yvsriram Oct 3, 2023
bed589f
update circleci config (#408)
cpaxton Oct 4, 2023
4b3c1e1
Fix some issues with angle interpolation (#410)
cpaxton Oct 5, 2023
1f393f3
updates hab-lab commit that fixes camera pose (#443)
yvsriram Nov 2, 2023
f724f3b
[Major update] Spot support, open-world exploration, instance mapping…
cpaxton Dec 21, 2023
9c43fa9
fix: protect against NoneError in heuristic place policy (#456)
villekuosmanen Jan 1, 2024
881059c
remove data and add a downloader script
cpaxton Jan 4, 2024
2b74b4a
update scripts to make sure we dont accidentally download all this data
cpaxton Jan 4, 2024
710499c
Update download_data.sh: update hssd-hab's branch to ovmm
yvsriram Jan 4, 2024
54d3b09
Update .gitmodules: remove entries for ovmm datasets
yvsriram Jan 4, 2024
0c9a099
Update download_data.sh
cpaxton Jan 8, 2024
c4571b5
Update download_data.sh
cpaxton Jan 8, 2024
1bec0c7
Update download_data.sh
cpaxton Jan 8, 2024
000fa10
update scripts for data downloading
cpaxton Jan 8, 2024
fbeb159
code cleanup for linter
cpaxton Jan 8, 2024
7ec07ef
remove lseg
cpaxton Jan 9, 2024
ca661d7
isort cfg
cpaxton Jan 9, 2024
0e99b26
update isort
cpaxton Jan 9, 2024
0eddd30
update cfg to skip contact graspnet server since it's a mess
cpaxton Jan 9, 2024
5456307
velocity control fixes
Jdvakil Jan 9, 2024
ded2781
reformatting manip
cpaxton Jan 11, 2024
93a6a69
reformatting and adding commment
cpaxton Jan 11, 2024
a23b25f
feat: specify detic object recognition confidence threshold in agent …
villekuosmanen Jan 11, 2024
65f5997
Merge pull request #461 from facebookresearch/anygrasp_fixes
Jdvakil Jan 11, 2024
e4149d5
Merge pull request #457 from facebookresearch/no-data
Jdvakil Jan 11, 2024
867b800
add openai client
cpaxton Jan 16, 2024
7022421
Add troubleshooting docs and fix some config issues (#463)
cpaxton Jan 18, 2024
a32a23d
Camera pose sequence length and hssd_demo fixes (#468)
yvsriram Jan 18, 2024
7033d48
update openai client
cpaxton Jan 20, 2024
71f8ad6
Merge pull request #466 from facebookresearch/cpaxton/openai-integration
Jdvakil Jan 25, 2024
cd71ecd
update troubleshooting readme (#474)
Jdvakil Feb 28, 2024
1e3caab
Update sophuspy version
yvsriram Mar 2, 2024
cbbfae0
Update sophuspy version in requirements.txt
yvsriram Mar 2, 2024
5eca396
Merge pull request #478 from yvsriram/main
Jdvakil Mar 5, 2024
5308ee4
Changes for v0.2.5 support for challenge (#488)
yvsriram Apr 29, 2024
cb6fe5a
Spot-Sim2Real submodule fix (#495)
Jdvakil May 15, 2024
825fca8
Bug Fix Pillow Denial of Service vulnerability (#487)
imhunterand May 24, 2024
ede6a67
Patch for issues (#491)
yvsriram May 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
scannet dataset can pick the number of classes to use and referring e… (
#388)

* scannet dataset can pick the number of classes to use and referring expressions respect this choice

* change default to allow eval

* remove comment

---------

Co-authored-by: Sasha Sax <asax@stanford.edu>
alexsax and Sasha Sax authored Sep 22, 2023
commit 9f2f33338b8a4b6d9211bec60ce1675d2e0ede7f
3 changes: 2 additions & 1 deletion projects/scannet_offline_eval/configs/dataset/scannet.yaml
Original file line number Diff line number Diff line change
@@ -12,4 +12,5 @@ referit3d_config:
_target_: home_robot.datasets.scannet.ReferIt3dDataConfig
scanrefer_config:
_target_: home_robot.datasets.scannet.ScanReferDataConfig
show_load_progress: False
show_load_progress: False
n_classes: 100
Original file line number Diff line number Diff line change
@@ -5,6 +5,4 @@ defaults:

_target_: build_sparse_voxel_map.SparseVoxelMapAgent
device: 'cuda:0'
global_nms_thresh: 0.3
instance_box_compression_resolution: 0.01
instance_box_compression_drop_prop: 0.2

Original file line number Diff line number Diff line change
@@ -9,3 +9,6 @@ instance_memory_kwargs:
instance_view_score_aggregation_mode: 'max'
overlap_eps: 1e-6
min_pixels_for_instance_view: 100
global_box_nms_thresh: 0.3
instance_box_compression_resolution: 0.01
instance_box_compression_drop_prop: 0.2
2 changes: 2 additions & 0 deletions src/home_robot/home_robot/datasets/scannet/__init__.py
Original file line number Diff line number Diff line change
@@ -4,5 +4,7 @@
# LICENSE file in the root directory of this source tree.

from .referit3d_data import ReferIt3dDataConfig
from .scannet_constants import CLASS_ID_TO_NAME
from .scannet_constants import NUM_CLASSES as NUM_CLASSES_LONG
from .scannet_dataset import ScanNetDataset
from .scanrefer_data import ScanReferDataConfig
Original file line number Diff line number Diff line change
@@ -32,7 +32,6 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
"""

import argparse
import inspect
import json
@@ -53,9 +52,9 @@ def read_aggregation(filename):
data = json.load(f)
num_objects = len(data["segGroups"])
for i in range(num_objects):
object_id = (
data["segGroups"][i]["objectId"] + 1
) # instance ids should be 1-indexed
object_id = data["segGroups"][i][
"objectId"
] # + 1 # instance ids should be 1-indexed
label = data["segGroups"][i]["label"]
segs = data["segGroups"][i]["segments"]
object_id_to_segs[object_id] = segs
@@ -140,7 +139,7 @@ def export(
"""

label_map = scannet_utils.read_label_mapping(
label_map_file, label_from="raw_category", label_to="nyu40id"
label_map_file, label_from="raw_category", label_to="id" # nyu40id
)
mesh_vertices = scannet_utils.read_mesh_vertices_rgb(mesh_file)

28 changes: 15 additions & 13 deletions src/home_robot/home_robot/datasets/scannet/referit3d_data.py
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import logging
from ast import literal_eval
from dataclasses import dataclass
from pathlib import Path
@@ -35,6 +36,8 @@
import numpy as np
import pandas as pd

logger = logging.getLogger(__name__)


@dataclass
class ReferIt3dDataConfig:
@@ -102,7 +105,7 @@ def load_referit3d_data(
n_original = len(referit_data)
referit_data = referit_data[referit_data["mentions_target_class"]]
referit_data.reset_index(drop=True, inplace=True)
print(
logger.info(
"Dropping utterances without explicit "
"mention to the target class {}->{}".format(n_original, len(referit_data))
)
@@ -132,41 +135,40 @@ def load_referit3d_data(
# train_token_lens = referit_data.tokens[is_train].apply(lambda x: len(x))
train_token_lens = referit_data.tokens.apply(lambda x: len(x))
if len(train_token_lens) == 0:
print(f"No NR3D expressions found for scenes {scans_split['train']}")
logger.info(f"No NR3D expressions found for scenes {scans_split['train']}")
else:
print(
"{}-th percentile of token length for remaining (training) data"
" is: {:.1f}".format(95, np.percentile(train_token_lens, 95))
pctile = 95
logger.info(
f"{pctile}-th percentile of token length for remaining (training) data"
+ " is: {np.percentile(train_token_lens, 95):.1f}"
)
n_original = len(referit_data)
referit_data = referit_data[
referit_data.tokens.apply(lambda x: len(x) <= max_seq_len)
]
referit_data.reset_index(drop=True, inplace=True)
print(
"Dropping utterances with more than {} tokens, {}->{}".format(
max_seq_len, n_original, len(referit_data)
)
logger.info(
f"Dropping utterances with more than {max_seq_len} tokens, {n_original}->{len(referit_data)}"
)

# do this last, so that all the previous actions remain unchanged
if sr3d_csv_fpath is not None:
print("Adding Sr3D as augmentation.")
logger.info("Adding Sr3D as augmentation.")
sr3d = pd.read_csv(sr3d_csv_fpath)
sr3d.tokens = sr3d["tokens"].apply(literal_eval)
is_train = sr3d.scan_id.apply(lambda x: x in scans_split["train"])
sr3d["is_train"] = is_train
sr3d = sr3d[is_train]
sr3d = sr3d[referit_columns]
print("Dataset-size before augmentation:", len(referit_data))
logger.info(f"Dataset-size before augmentation: {len(referit_data)}")
referit_data = pd.concat([referit_data, sr3d], axis=0)
referit_data.reset_index(inplace=True, drop=True)
print("Dataset-size after augmentation:", len(referit_data))
logger.info(f"Dataset-size after augmentation: {len(referit_data)}")

context_size = referit_data.stimulus_id.apply(
lambda x: decode_stimulus_string(x)[2]
)
print(
logger.info(
"(mean) Random guessing among target-class test objects {:.4f}".format(
(1 / context_size).mean()
)
567 changes: 564 additions & 3 deletions src/home_robot/home_robot/datasets/scannet/scannet_constants.py

Large diffs are not rendered by default.

170 changes: 148 additions & 22 deletions src/home_robot/home_robot/datasets/scannet/scannet_dataset.py
Original file line number Diff line number Diff line change
@@ -5,9 +5,11 @@

import copy
import dataclasses
import logging
import os
import warnings
from functools import partial
from numbers import Number
from pathlib import Path
from typing import List, Optional, Tuple, Union

@@ -17,16 +19,20 @@
import torch
from natsort import natsorted
from PIL import Image
from torch import Tensor
from tqdm import tqdm

from .referit3d_data import ReferIt3dDataConfig, load_referit3d_data
from .scannet_constants import (
NUM_CLASSES,
SCANNET_DATASET_CLASS_IDS,
SCANNET_DATASET_CLASS_LABELS,
SCANNET_DATASET_COLOR_MAPS,
)
from .scanrefer_data import ScanReferDataConfig, load_scanrefer_data

logger = logging.getLogger(__name__)


class ScanNetDataset(object):

@@ -104,13 +110,6 @@ def __init__(
assert (
n_classes in SCANNET_DATASET_COLOR_MAPS
), f"{n_classes=} must be in {SCANNET_DATASET_COLOR_MAPS.keys()}"
self.METAINFO = {
"COLOR_MAP": SCANNET_DATASET_COLOR_MAPS[n_classes],
"CLASS_NAMES": SCANNET_DATASET_CLASS_LABELS[n_classes],
"CLASS_IDS": SCANNET_DATASET_CLASS_IDS[n_classes],
}

self.class_ids_ten = torch.tensor(self.METAINFO["CLASS_IDS"])

# Set up directories and metadata
assert split in ["train", "val", "test"]
@@ -120,6 +119,40 @@ def __init__(
self.instance_2d_dir = self.root_dir / "scannet_instance_data"
self.scan_dir = self.root_dir / "scannet_instance_data"

# Metainfo
self.METAINFO = {
"COLOR_MAP": SCANNET_DATASET_COLOR_MAPS[n_classes],
"CLASS_NAMES": SCANNET_DATASET_CLASS_LABELS[n_classes],
"CLASS_IDS": SCANNET_DATASET_CLASS_IDS[n_classes],
}
# Load class names
labels_pd = pd.read_csv(
self.root_dir / "meta_data" / "scannetv2-labels.combined.tsv",
sep="\t",
header=0,
)
labels_pd.loc[labels_pd.raw_category == "stick", ["category"]] = "object"
labels_pd.loc[labels_pd.category == "wardrobe ", ["category"]] = "wardrobe"
self.ALL_CLASS_IDS_TO_CLASS_NAMES = dict(
zip(labels_pd["id"], labels_pd["category"])
)
self.ALL_CLASS_NAMES_TO_CLASS_IDS = dict(
zip(labels_pd["category"], labels_pd["id"])
)
# self.METAINFO['CLASS_NAMES'] = [self.ALL_CLASS_IDS_TO_CLASS_NAMES[k] for k in self.METAINFO['CLASS_IDS']]
self.METAINFO["CLASS_IDS"] = [
self.ALL_CLASS_NAMES_TO_CLASS_IDS[k] for k in self.METAINFO["CLASS_NAMES"]
]
# Create tensor lookup table
self.class_ids_ten = torch.tensor(self.METAINFO["CLASS_IDS"])
self.DROP_CLASS_VAL = -1
self.class_ids_lookup = make_lookup_table(
self.class_ids_ten,
self.class_ids_ten,
missing_key_value=self.DROP_CLASS_VAL,
)

# Image metadata
self.split = split
self.height = height
self.width = width
@@ -133,7 +166,7 @@ def __init__(
if keep_only_scenes is not None:
self.scene_list = [s for s in self.scene_list if s in keep_only_scenes]
self.scene_list = natsorted(self.scene_list)
print(
logger.info(
f"ScanNetDataset: Keeping next {keep_only_first_k_scenes} scenes starting at idx {skip_first_k_scenes}"
)
self.scene_list = self.scene_list[skip_first_k_scenes:][
@@ -171,7 +204,6 @@ def __init__(
/ f"ScanRefer_filtered_{split}.json"
)
self.scanrefer_data = load_scanrefer_data(json_fpath)
# '/private/home/ssax/home-robot/src/home_robot/home_robot/datasets/scannet/data/scanrefer/ScanRefer_filtered_val.json'

def find_data(self, scan_name: str):
# RGBD + pose
@@ -276,9 +308,10 @@ def __getitem__(self, idx: Union[str, int], show_progress: bool = False):
boxes_aligned, box_classes, box_obj_ids = load_3d_bboxes(
data["bboxs_aligned_path"]
)
keep_boxes = (box_classes.unsqueeze(1) == self.class_ids_ten.unsqueeze(0)).any(
dim=1
)
# keep_boxes = (box_classes.unsqueeze(1) == self.class_ids_ten.unsqueeze(0)).any(
# dim=1
# )
keep_boxes = self.class_ids_lookup[box_classes] != self.DROP_CLASS_VAL
boxes_aligned = boxes_aligned[keep_boxes]
box_classes = box_classes[keep_boxes]
box_obj_ids = box_obj_ids[keep_boxes]
@@ -310,6 +343,8 @@ def __getitem__(self, idx: Union[str, int], show_progress: bool = False):
][column_names]
ref_expr_df = pd.concat([scanrefer_expr, r3d_expr])

ref_expr_df = filter_ref_exp_by_class(ref_expr_df, box_obj_ids, box_classes)

# Return as dict
return dict(
# Pose
@@ -334,17 +369,9 @@ def __len__(self):
return len(self.scene_list)


def maybe_show_progress(iterable, description, length, show=False):
if show:
for x in tqdm(iterable, desc=description, total=length):
yield x
else:
for x in iterable:
yield x


##################################
# Load different modalities
#################################
def load_pose_opengl(path):
pose = np.loadtxt(path)
pose = np.array(pose).reshape(4, 4)
@@ -445,6 +472,105 @@ def load_3d_bboxes(path) -> Tuple[torch.Tensor, torch.Tensor]:
return torch.stack([mins, maxs], dim=-1), labels, obj_ids


def filter_ref_exp_by_class(
ref_expr_df: pd.DataFrame, box_target_ids: Tensor, box_classes: Tensor
) -> pd.DataFrame:
"""Keeps only referring expressions where referring expression"""
ref_exp_target_ids = torch.tensor(ref_expr_df.target_id.to_numpy())

# Make lookuptable of lookuptable[target_ids] -> target_class
max_key = max(box_target_ids.max(), ref_exp_target_ids.max()) + 1
ids_to_classes = make_lookup_table(
box_target_ids.long(), box_classes, missing_key_value=-1, key_max=max_key
)

# Keep referring expressions who have targets where class != -1 (i.e. where target is in box_target_ids)
ref_exp_classes = ids_to_classes[ref_exp_target_ids]
df = ref_expr_df.copy()
df["target_class_id"] = ref_exp_classes.cpu().numpy()
keep_exp = ref_exp_classes != -1
df = df.loc[keep_exp.cpu().numpy()]

# # Map to class name with something like:
# df['instance_type2'] = [class_id_to_name[class_idx] for class_idx in df['target_class_id']]
return df


#############################################################
# Utils
#############################################################


def maybe_show_progress(iterable, description, length, show=False):
if show:
for x in tqdm(iterable, desc=description, total=length):
yield x
else:
for x in iterable:
yield x


def make_lookup_table(
keys: Tensor,
values: Tensor,
key_max: Optional[int] = None,
missing_key_value: Number = torch.nan,
) -> Tensor:
"""
Create a lookup table using keys and values tensors.
This function creates a 1D tensor (lookup table) using keys and values.
The length of the lookup table is determined by `key_max`. The `keys` tensor
specifies the indices in the lookup table that will be populated with the corresponding
values from the `values` tensor. Indices not present in `keys` will be filled with
`missing_key_value`.
Parameters:
-----------
keys : torch.Tensor
1D tensor of long integers specifying the indices in the lookup table
where values should be placed. Must have dtype of torch.long.
values : torch.Tensor
1D tensor containing the values to be placed in the lookup table.
Must have the same length as `keys`.
key_max : int, optional
The maximum key value + 1, which determines the length of the lookup table.
If None, it is set to the maximum value in `keys` + 1. Default is None.
missing_key_value : Number, optional
The value to fill in for missing keys in the lookup table. Default is NaN.
Returns:
--------
keys_expanded : torch.Tensor
The populated lookup table. The dtype will match that of `values`.
Raises:
-------
AssertionError
If the dtype of the `keys` is not torch.long.
Example:
--------
>>> keys = torch.tensor([1, 3, 5], dtype=torch.long)
>>> values = torch.tensor([10.0, 30.0, 50.0])
>>> make_lookup_table(keys, values)
tensor([nan, 10.0, nan, 30.0, nan, 50.0])
"""
if key_max is None:
key_max = keys.max().item() + 1
assert (
keys.dtype == torch.long
), f"keys must have dtype torch.long -- not {keys.dtype}"
keys_expanded = torch.full(
[key_max],
fill_value=missing_key_value,
device=values.device,
dtype=values.dtype,
)
keys_expanded.scatter_(dim=0, index=keys, src=values)
return keys_expanded


if __name__ == "__main__":
import open3d

@@ -456,7 +582,7 @@ def load_3d_bboxes(path) -> Tuple[torch.Tensor, torch.Tensor]:
from home_robot.utils.point_cloud_torch import get_xyz_coordinates

data = ScanNetDataset(
root_dir="/private/home/ssax/home-robot/projects/eval_scannet/scannet",
root_dir="./data/",
frame_skip=30,
)
result = data.__getitem__(0, show_progress=True)
57 changes: 45 additions & 12 deletions src/home_robot/home_robot/datasets/scannet/visualize_scannet.ipynb
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@
"import numpy as np\n",
"import torch\n",
"from tqdm import tqdm\n",
"\n"
"import pandas as pd\n"
]
},
{
@@ -31,12 +31,17 @@
"metadata": {},
"outputs": [],
"source": [
"from scannet_dataset import ScanNetDataset\n",
"from referit3d_data import ReferIt3dDataConfig\n",
"from scanrefer_data import ScanReferDataConfig\n",
"# from scannet_dataset import ScanNetDataset\n",
"# from referit3d_data import ReferIt3dDataConfig\n",
"# from scanrefer_data import ScanReferDataConfig\n",
"from pytorch3d.io import IO\n",
"from pytorch3d.structures import Pointclouds\n",
"from home_robot.datasets.scannet import ScanNetDataset, ReferIt3dDataConfig, ScanReferDataConfig, NUM_CLASSES_LONG\n",
"data = ScanNetDataset(\n",
" root_dir = '/private/home/ssax/home-robot/src/home_robot/home_robot/datasets/scannet/data',\n",
" frame_skip = 180,\n",
" n_classes=NUM_CLASSES_LONG,\n",
" # n_classes=50,\n",
" referit3d_config = ReferIt3dDataConfig(),\n",
" scanrefer_config = ScanReferDataConfig(),\n",
")\n",
@@ -51,8 +56,37 @@
"from pytorch3d.io import IO, load_obj, load_ply\n",
"scene_id = scene_obs['scan_name']\n",
"print(\"Loading GT mesh for\", scene_id)\n",
"verts = load_ply(data.root_dir / f'scans/{scene_id}/{scene_id}_vh_clean.ply')\n",
"aligned_verts = torch.cat([verts[0], torch.ones_like(verts[0][:,:1])], dim=-1) @ scene_obs['axis_align_mats'][0].T\n"
"# verts = load_ply(data.root_dir / f'scans/{scene_id}/{scene_id}_vh_clean.ply')\n",
"pc = IO().load_pointcloud(data.root_dir / f'scans/{scene_id}/{scene_id}_vh_clean.ply')\n",
"verts = pc.points_packed()\n",
"aligned_verts = torch.cat([verts, torch.ones_like(verts[:,:1])], dim=-1) @ scene_obs['axis_align_mats'][0].T\n",
"pointcloud_aligned = Pointclouds(points=aligned_verts[...,:3].unsqueeze(0), features=pc.features_packed().unsqueeze(0))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# df[:10]\n",
"scene_obs['ref_expr']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize referring expression\n",
"# Title: Query\n",
"# Trace: Pointcloud \n",
"# Trace: GT bbox\n",
"# Trace: Distractors of same class\n",
"selected = scene_obs['box_target_ids'] == 39\n",
"id_to_name = dict(zip(data.METAINFO['CLASS_IDS'], data.METAINFO['CLASS_NAMES']))\n",
"id_to_name[scene_obs['box_classes'][selected].item()]"
]
},
{
@@ -146,12 +180,9 @@
"\n",
"fig = plot_scene_with_bboxes(\n",
" plots = { f\"{scene_id}\": { \n",
" \"Points\": svm.global_voxel_grid._pcl,\n",
" # \"Boxes\": join_boxes_as_scene(svm.instance_bboxes3d),\n",
" \"All boxes\": global_boxes,\n",
" \"Global boxes\": global_boxes,\n",
"\n",
" \"GT boxes\": gt_boxes,\n",
" \"GT points\": Pointclouds(points=[aligned_verts[:, :3]]),\n",
" \"GT points\": pointcloud_aligned,\n",
" # \"cameras\": cameras,\n",
" }\n",
" },\n",
@@ -163,7 +194,9 @@
" pointcloud_max_points=30_000,\n",
" boxes_wireframe_width=3,\n",
" boxes_add_cross_face_bars=False,\n",
" boxes_name_int_to_display_name_dict = dict(zip([int(i) for i in data.METAINFO['seg_valid_class_ids']], data.METAINFO['classes'])),\n",
" # boxes_name_int_to_display_name_dict = dict(zip([int(i) for i in data.METAINFO['seg_valid_class_ids']], data.METAINFO['classes'])),\n",
" boxes_name_int_to_display_name_dict = dict(zip(data.METAINFO['CLASS_IDS'], data.METAINFO['CLASS_NAMES'])),\n",
"\n",
" boxes_plot_together=False,\n",
" height=1000,\n",
" # width=1000,\n",