feat: implement AI-driven behavioral video sentiment analyzer (#1744) - #1749
Open
desireddymohithreddy0925 wants to merge 2 commits into
Open
Conversation
📝 WalkthroughWalkthroughThe frontend adds a webcam sentiment analyzer. It loads face-api.js models, detects facial expressions during a session, displays expression percentages, and reports the latest results when the session ends. ChangesVideo sentiment analysis
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
actor User
participant VideoSentimentAnalyzer
participant Browser
participant FaceApi
User->>VideoSentimentAnalyzer: Start analysis
VideoSentimentAnalyzer->>Browser: Request webcam access
Browser-->>VideoSentimentAnalyzer: Return video stream
VideoSentimentAnalyzer->>FaceApi: Detect facial expressions
FaceApi-->>VideoSentimentAnalyzer: Return expression scores
User->>VideoSentimentAnalyzer: End session
VideoSentimentAnalyzer-->>User: Display final expression results
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontend/src/components/VideoSentimentAnalyzer.jsx`:
- Around line 16-18: Update the model-loading flow in VideoSentimentAnalyzer to
store a user-visible error when either face-api model request fails, instead of
only logging it in the catch block. Render the error state in place of “Loading
AI Models...” with a retry action that re-invokes the existing model-loading
function and resets the error state.
- Around line 23-29: Update startVideo and the component lifecycle effect to
store the acquired stream in a ref, stop all tracks on effect cleanup, and stop
any existing stream before assigning a replacement; preserve the End Session
behavior while ensuring unmount releases the webcam.
- Around line 13-14: Add the Tiny Face Detector and face-expression model
manifests and weight shards under frontend/public/models, ensuring the existing
loadFromUri('/models') calls in the model-loading effect can retrieve them.
Alternatively, update both loadFromUri calls to the deployed URL containing
these assets.
- Around line 43-55: Update handleVideoPlay to store the created interval
identifier in a ref instead of leaving it unmanaged. Clear that interval in
stopVideo and the component’s unmount cleanup, preventing duplicate detection
loops after playback restarts while preserving the existing analysis behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6e35b4cb-f376-496a-9613-3f81b9af78a6
⛔ Files ignored due to path filters (1)
frontend/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
frontend/package.jsonfrontend/src/components/VideoSentimentAnalyzer.jsx
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
frontend/src/components/VideoSentimentAnalyzer.jsx (2)
44-49: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick winPrevent overlapping face detections.
setIntervalinvokes the async callback every second without waiting for the previousdetectAllFacescall. If inference takes longer than one second, concurrent calls can consume excessive CPU and updateexpressionsout of order. Use one in-flight loop or a recursivesetTimeoutscheduled after each detection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/VideoSentimentAnalyzer.jsx` around lines 44 - 49, Update the interval logic around the face detection callback in VideoSentimentAnalyzer so only one detectAllFaces invocation is in flight at a time. Replace setInterval with a recursive setTimeout or equivalent sequential loop that schedules the next run only after detection completes, while preserving the existing videoRef.current and isAnalyzing checks and expression updates.
24-29: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winShow camera-start failures in the UI.
If
getUserMediarejects, the handler logs the error only. The user sees no reason for failure and no recovery state. Track a camera error and render a retry message or action.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/VideoSentimentAnalyzer.jsx` around lines 24 - 29, Update the getUserMedia error handler in VideoSentimentAnalyzer to store the camera-start failure in component state instead of only logging it, reset the analyzing state as needed, and render that error with a retry message or action so users can recover from camera permission or startup failures.
🧹 Nitpick comments (1)
frontend/src/components/VideoSentimentAnalyzer.jsx (1)
76-85: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winPrevent analyzer controls from submitting a parent form.
If this component is rendered inside a
<form>, neither button declarestype, so both default tosubmit. Clicking Start or End can submit the parent form and unmount the analyzer. Addtype="button"to both controls.Proposed button fix
-<button onClick={startVideo} +<button type="button" onClick={startVideo} ... -<button onClick={stopVideo} +<button type="button" onClick={stopVideo}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/VideoSentimentAnalyzer.jsx` around lines 76 - 85, Prevent the analyzer controls from submitting a containing form by adding type="button" to both buttons rendered in the !isAnalyzing conditional around startVideo and stopVideo, while preserving their existing click handlers and styling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontend/src/components/VideoSentimentAnalyzer.jsx`:
- Line 39: Update the expression state handling in the video analysis component:
set expressions to {} when detections.length is zero, and reset it at the start
of startVideo before processing the new stream. Ensure onAnalysisComplete
receives the cleared state when no face is detected and cannot retain the
previous session’s expression.
---
Outside diff comments:
In `@frontend/src/components/VideoSentimentAnalyzer.jsx`:
- Around line 44-49: Update the interval logic around the face detection
callback in VideoSentimentAnalyzer so only one detectAllFaces invocation is in
flight at a time. Replace setInterval with a recursive setTimeout or equivalent
sequential loop that schedules the next run only after detection completes,
while preserving the existing videoRef.current and isAnalyzing checks and
expression updates.
- Around line 24-29: Update the getUserMedia error handler in
VideoSentimentAnalyzer to store the camera-start failure in component state
instead of only logging it, reset the analyzing state as needed, and render that
error with a retry message or action so users can recover from camera permission
or startup failures.
---
Nitpick comments:
In `@frontend/src/components/VideoSentimentAnalyzer.jsx`:
- Around line 76-85: Prevent the analyzer controls from submitting a containing
form by adding type="button" to both buttons rendered in the !isAnalyzing
conditional around startVideo and stopVideo, while preserving their existing
click handlers and styling.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 0b5506ef-11fd-412f-9c47-b0050b6e81de
⛔ Files ignored due to path filters (1)
frontend/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
frontend/package.jsonfrontend/src/components/VideoSentimentAnalyzer.jsx
🚧 Files skipped from review as they are similar to previous changes (1)
- frontend/package.json
| } | ||
| setIsAnalyzing(false); | ||
| if (onAnalysisComplete) { | ||
| onAnalysisComplete(expressions); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Clear expression state between frames and sessions.
expressions is updated only when detections.length > 0, and startVideo never resets it. If the face leaves the frame or a new session has no detection yet, the UI and onAnalysisComplete(expressions) can report the previous session's last expression. Set {} when no face is detected and when a new stream starts.
Proposed state reset
.then((stream) => {
videoRef.current.srcObject = stream;
+ setExpressions({});
setIsAnalyzing(true);
...
- if (detections.length > 0) {
- setExpressions(detections[0].expressions);
- }
+ setExpressions(detections[0]?.expressions ?? {});Also applies to: 51-52
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/src/components/VideoSentimentAnalyzer.jsx` at line 39, Update the
expression state handling in the video analysis component: set expressions to {}
when detections.length is zero, and reset it at the start of startVideo before
processing the new stream. Ensure onAnalysisComplete receives the cleared state
when no face is detected and cannot retain the previous session’s expression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1744
Description
This PR implements the foundation for the AI-Driven Behavioral Interview Video Sentiment Analyzer. It introduces a client-side WebAssembly pipeline using
face-api.jsto process video frames locally, evaluating user sentiment in real-time during mock interviews.Changes Made
VideoSentimentAnalyzer.jsxinfrontend/src/components/.face-api.jsfor facial expression recognition.Checklist
Summary
face-api.jsfor client-side facial expression analysis.VideoSentimentAnalyzerwith webcam controls, local model processing, real-time expression metrics, and completion callbacks.