Immersive assistant that helps you stay organized and responds with care and concern for your well being. Immersiveness is achieved by creating a room where you are situated in a desk - every time you log on - but are allowed to roam around your room while music plays in the back or, if you would prefer, view a virtual scenery through your windows.
Using the Google Cloud Speech-to-Text API, the user can speak to Mischa, who then responds with Google text-to-speech.
Using the Google DialogFlow API, users can trigger different intents when speaking to Mischa, as well as hold general conversation. These intents include scheduling with Google Calendar, a meditation intent, and a music intent.
Using the Google Cloud Natural Language API, Mischa analyzes the user's language to determine their sentiment and keeps track of this value over time. Based on the user's sentiment, Mischa may suggest that the user uses our stress-reduction activities.
Mischa fully supports seated users with dynamic height tracking and snap turns for an optimal viewing angle. Also features teleportation capabilities to help reduce motion sickness. The selection visualizer (pointer) and teleportation controls are accessible with both controllers.
Users can freely explore different scenes, practice archery, stream videos in a mini theater, and even create 3D sculptures using Mischa's built-in paintbrush. Users can also ask Mischa to play a meditation tape or music of their choice from YouTube.
Unity Editor 2.1.3
Unity Version: 2018.4.9f1 (LTS)
Python 3.7
Python packages: speech_recognition, playsound, gtts, pyaudio, google.cloud, dialogflow, selenium
- Unity - Main Platform, Dependency Management
- SteamVR Plugin - Archery, Painting
- Oculus Integration - Control Mapping, Selection Visualizer
- Virtual Reality Toolkit - Teleportation, Snap Turns, Audio Spatialization
- Blender - Prefab Modeling
- Google Cloud Natural Language API - Sentiment Analysis
- Google DialogFlow API - Conversation and intents
- Google Calendar API - Scheduling Events
- Google Cloud Speech-to-Text - Speech Recognition
- Google Cloud Text-to-Speech - Converting text to speech
- Flask - Used to call Python backend from C# in Unity
- Selenium - Used for getting music from YouTube
Ideas for future implementation:
- Expansion to other platforms (HTC Vive, Valve Index, Windows Mixed Reality) & mobile support (Google Cardboard)
- Advanced music functionality, where users can import songs/playlists, pause, queue, and shuffle songs
- Enhanced voice assistant compatibility (Google Assistant & Amazon Alexa)
- Environment customization, allowing users to adjust the furniture, lighting & import their own 3D models
- Yohan Flores - Project Manager - Pixel-Yohan
- Firstname Lastname - Role - Username
See also the list of contributors who participated in this project.