Skip to content

Latest commit

 

History

History
97 lines (61 loc) · 2.44 KB

README.md

File metadata and controls

97 lines (61 loc) · 2.44 KB

MotionMap: Transforming Body Movements into Keyboard Actions

Transform real-time body movements — including walking, squatting, hand swinging, and head tilting — into keyboard inputs, enabling a more interactive and intuitive way to control apps and games.

Demo

Control an RPG game (Legend of Zelda) (old version)

Watch the video

Control a racing game (old version)

Watch the video

How to install

Install python3

Create an virtual environment (optional)

python -m venv venv
.\venv\Scripts\activate

Install packages

pip install -r requirements.txt

Run the application

python app.py

Build

Windows

python -m nuitka --disable-console --product-name MotionMap app.py

Run the build file in the root directory

  • Windows: app.exe

Supported body movements

Movements are based on Mediapipe Pose model. The model detects 33 key points of the body, including eyes, ears, nose, shoulders, elbows, wrists, hips, knees, and ankles. From these key points, we can detect the movements by calculating the angles between the points, the distance between the points, and the position of the points.

Body

See details in file src/movements.py

Head

Tilt head in left/right

Hands

  • Swing hands
    • Left: swing hand from left to right
    • Full swing: swing left hand from head to bottom
  • Raise both hands up
  • Hands crossed: Cross 2 hands in Wakanda style

Legs

  • Walking:
    • Up: walking with 2 hands down
    • Left: walking with only left hand up 90 degree
    • Right: walking with only right hand up 90 degree
    • Backwards: : walking with both hands up 90 degree
  • Squat
  • Left/right leg raised

Driving mode (in development)

  • Move 2 hands close to enable steering wheel, tilt left or right to control
  • Move 2 hands inside the green box to enable driving up control

FPS mode (in development)

Roadmap

  • Support more modes: driving, FPS games, controlling mouse
  • Support 2 players mode
  • Support more OS: Mac, Linux
  • Edit movements detection thresholds (advanced mode)
  • Fix the mirror camera issue

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.