Pull latest main into GOAT #307
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add test config paths for remote eval * hide heuristic pick/place behind verbose * reduce if conditions to pass tests
* remove episode argument in agent reset * linting
…task execution (#198) * PointConv->PointNetConv; adding autoregressive training * fixed bugs; overfits 1traj,1waypt to mm; 1traj, 3 waypts to 1cm * time as one-hot appended to proprio; also per-actn cmd * remove time from proprio; add separate mlp * added in GRU unit; NN too simple me feels * clean-up * making network narrower and deeper * added mdn; requires tweaking * starter code * first draft of driver code + agent * drafting agent class * extending home_robot; fixing class methods * return types * bug-fixing * bug-fixing * update recorder to use new StretchClient * set position mode before moving * use velocity to control base * remove debug statement * add task-name in config * store camera pose rotated * revised interaction point detection logic * utility to collect top N episodes from each h5 * lower sleep * persistent pressing for moving base * persistent pressing for moving base * shutdown timers or robot will run away * bug-fix * tweaking vel-scale * tweaking vel-scale, again * diff scale for rot and trans * adding skills as code * 1st version of lang to execution working * changing env and detic to suit lang planner needs * added data for cmd2steps * get codelist from task cmd * small bug-fixes * fix for new client * faster motion * faster motion;slower rot * works for sequence of multiple tasks with detic object change on fly * extended support to be able to search for objects-on-receptacles * pre-commit formatting * pick, place and find all three work; loads of offset though * reverting unnecessary changes * reverting unnecessary changes, debugging * hopefully smoother execution on arm * todo's and prep for using demo-success/fail flag * reset to manip mode per episode for consistency * QoL updates * move to manip posture b4 starting episode * move to pre-demo position; diff from pre-manip * bug-fix * instructions * Slap integration and new actions * unnecessary changes * Merge * habitat-lab as in main * not sure what is going on * cleaning up old agent * SLAP flows through SM; basic YAML for task info; script for adding Detic features * renaming * state-machine works for variable # of keypoints * documentation * can read feats, passes through all methods; need avging * bringing in averaging which also considers semantic-feats * pulled in everything required to process semantic feats during training/validation on h5s * renaming and recfging for new feats/trials * APM, IPM work with semantic feats; utility scripts for H5s * do not read trials with demo_status==0 * todo and skeleton for skill-planner * start-up instructions * updating task-list * weights and gripper arch which works * testing perception for SLAP inference * instructions for env management to lessen pain * obs to IPM input pipeline works; test IPM next; then APM * goto nav mode once episode is done * inference works; can visualize actions with respect to local cropped-PCD * robustifying so you can choose trial by number; Nth from the beginning * robot-side code for closing loop on SLAP prediction for 1 skill * policy-side code for debugging + closing the loop on the robot * one-stop shop for all your H5+detic needs * try blocking motions and all that to see if obs stream makes sense * new scripts and debug * stupid bug; removing brkpoints * debug print-and-stop * eval llm task plans * modify path * remove interactive cmts; * loads of debugging to make coordinate frames work * more visualizer calls and types * can I stop farther from goal? * take out debug * debugging code, lininterpolation, adding back removed param * this version works for e2e for open-object-drawer * handover works too * last-mile changes to make experiments easier * worked for devfair env * updated installatn instructions * changes to execution code * corl-experiment weights and data-massaging * all-skills, make training and testing the same * Experiment structure * bug-fix * leftover debug * execution settings to work with SLAP assumptions * remove debug and fix labeling * training ease for tuning; peract * ignore training stuff * remove stuff with personal paths * peract-loader dbugged * quality of life; labeling bug * good weights, etc * fix mask; fix p_i logic; relabeled data * important rotations for obs from nav mod * consistency in training anf debugging * viz OK; data OK; train * remove abs path * adjustment * Experiment ready * experiment version * other 2 tasks * small adjustments for experiments * trining ready * apm specific change * per-skill experiments * adding per-skill experiments * specific to non-PnP skills * configs for peract * peract agent with end2end predictions + execution * also send gripper information * updates for rolling out with the new version * formatting? * task definition for experiment * peract debugged; better scenebounds for robot setup * include transformations from ROS; consistent quaternion * change training/inference for peract to be per waypoint * adding parameters for specific use-case; adding SLAP config * arm goes higher; handling None val for keys; adding paramaters for specific use-cases * adding details for LLM inferred language * addig parameters from main * adding QoL feature for easier experimentation; porting to new config * modifying semantic-sensor dataflow to respect new responsibilities; code for LLM inferred plans * save top 5% mask not rgb * borrowing get_obs from parent-class; modifying set_goal for specific use-case * verifying dataloaders and integrity of peract input * remove TODOs * correct bounds * remove dead code * shaping for torch tensors to make sense * removing dead code * option to filter in method rather than loader body * removing dead code * simplifying training/eval loop * executing peract * input consumption and output shaping * execution of point-to-point motion * validation on roll-out+simplification * feats is optional; logic for point-to-point motion planning * adding gripper-state as a signal * evaluation loop for PerAct agent * bounds and everything * lint fixes * documentation * docstrings, types, size is param * dead code * remove data splits and references * ignore asset yamls * adding license * cleanup * revert * remove hard-coded arm-height; document ros viz * renaming to avoid confusion * removing dead code and stale TODO/FIXME * fixing * moved saving to helper func, visualize and saving params * config documentation for inference * removing defunct params; docstring, hints in slap_agent * docstring, hints * +docstring, hints * docstring, hints; nice paths * missed out * params, defaults * docstring, hint * docstrings, deleting defunct code * more license, docstrings, corner-case safeguards * removing problematic components (license-wise) * licenses * problematic license * licenses * document everything * documentations * no more nested configs * better docstring * Script docstring * revert changes * resolving duplication * better docstring? * comment * consolidating functions; docstrings; etc * flagging, documentation * why config * documentation and proper usage * deduplication of reset; docstrings, type hints * Documentation for all scripts * helpful documentation * documenting params * comment based doc * docstring and more comments * fixed installation instructions, troubleshoot, etc * comment-based documentation * parameterize PerAct * proper documentation; merging SLAP with LangAgent * last mile changes for proper compilation and deduplication * removing duplication * resolving licenses; parameterizing preferences; comments * config to param; sim-test * bug-fix --------- Co-authored-by: Vidhi Jain <[email protected]> Co-authored-by: keke-220 <[email protected]>
* add category name to viz for objectnav, instance_tracking in offline mapping * update global map update for objectnav * record instance ids while offline mapping
* Swap Detic for MaskRCNN * Swap Detic for MaskRCNN * Swap Detic for MaskRCNN * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Swap Detic for Maskrcnn * Address PR comments
* add some notes on robot configuration * fixing image
* add install script * FIx the install script * update instructions * uppdate scripts * Update projects/habitat_ovmm/README.md Co-authored-by: Arun Ramachandran <[email protected]> --------- Co-authored-by: Arun Ramachandran <[email protected]>
* added receptacles data collection script * fixed formatting issues with pylint; added helper function to avoid repetition; added argument for providing save dir; TODO: fix dataset format * improved error messages * fixed dataset saving format * fixed small bugs; tested it as well; most issues fixed; TODO: what object questions do we want? * fixed a docstring * Fix linter error * Fix linter error --------- Co-authored-by: Anurag Ajay <[email protected]> Co-authored-by: Chris Paxton <[email protected]>
* added step to download pretrained checkpoints * updated docs/challenge.md
* Create an install script * Rename script * Make the script executable * detectron2 * fix installation script * fix auto detectron install * debug python env * make installer download the detic checkpoint * install precommit * add lfs * add case check
facebook-github-bot
added
the
CLA Signed
* Conditionally import habitat dependencies, so that we can run things without sim installed. * fixing some bugs in habitat-free scripts
* Update license header info * Update license headers
* update discrete step, rendering for eval * add grounded mobile sam as new perception module * linting * remove evaluation code from grounded_mobile_sam, refactor * add exception E402 * linting * improve documentation, add depth filtering * add detection to agent config
* update installation guides * add step-by-step workstaion setup instructions * Update README.md Co-authored-by: Arun Ramachandran <[email protected]> * Update README.md Co-authored-by: Mohamed Fazil <[email protected]> * initialize hab-lab repo * update instructions * update some lines in install script * fix a bug in eval_episode * update to pass linter --------- Co-authored-by: Arun Ramachandran <[email protected]> Co-authored-by: Mohamed Fazil <[email protected]>
* add phase example commands * more detailed commands * Update the documentation
* add __init__ * fix agent command because of condition in ovmm perception
* Add extra note about python 3.9 * extra note about python and dependencies
* add README for installing perception modules * Update install_deps.sh * Mention GroundedSAM in README.md
…ed right (#338) * Add quality of life checks for experimenters to make sure robots are homed and working properly * add a note about creating a catkin workspace --------- Co-authored-by: Chris Paxton <[email protected]>
Co-authored-by: Sriram Yenamandra <[email protected]>
Co-authored-by: Sriram Yenamandra <[email protected]>
Co-authored-by: Sriram Yenamandra <[email protected]>
velocity control fixes
Remove "data" directory as submodules
* update docs and scripts * add some timing options and remove a weird bug on my machine caused by matplotlib
* fix errors when detic_viz=True in simulation * use only the goal names from episodes in sim * add sequence length dimension to camera pose in map modules
add simple openai client
* update troubleshooting readme * Update troubleshooting.md * Update troubleshooting.md * typo
Pin sophuspy version to 0.0.8
* Pin trimesh version * Update setup.py * fix tests * use right evaluation_pb2 * Fix to create vis dirs only when self.print_images=True * Update ovmm_agent.py to remove redundant debug statements * precommit * changes to docker * changes to README for challenge * update submission phase names * Update README.md to point to the challenge release branch * Download GSO objects with updated textures * Update README.md
* init * Update spot.md * update spot-sim2real to read from main * try fixing the failing tests * Update README.md --------- Co-authored-by: Sriram Yenamandra <[email protected]>
## Descriptions An issue was discovered in Pillow prior to 10.0.0. It is a Denial of Service that uncontrollably allocates memory to process a given task, potentially causing a service to crash by having it run out of memory. This occurs for truetype in "ImageFont" when textlength in an "ImageDraw" instance operates on a long text argument. ## Impact CVE-2023-44271 CWE-400 CWE-770 `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H` Co-authored-by: Chris Paxton <[email protected]>
* short term goal viz error * Update training config paths * update command --------- Co-authored-by: Chris Paxton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Changelog
How Has This Been Tested
Types of changes
Checklist:
tests/hw_manual_test.pyon hardware and observed that robot behavior looks normal.