Sentivue - Emotion-Based GIF Captioning System

An intelligent system that analyzes GIFs to detect emotions, objects, actions, and generates contextually-aware natural language captions.

Overview

SentiVue is a multimodal AI system that combines:

Emotion Detection - Classifies 6 emotion groups using ResNet50
Object Recognition - Multi-frame YOLO detection for robust object identification
Action Recognition - VideoMAE-based activity detection
Scene Understanding - Lighting analysis and content type classification
Caption Generation - Context-aware natural language descriptions

Upload a GIF, and SentiVue generates rich, emotionally-aware captions like:

"a joyful person dancing with a dog in bright lighting"

Features

Core Capabilities

Feature	Technology	Details
Emotion Detection	ResNet50	6 emotion groups: positive_energetic, positive_calm, negative_intense, negative_subdued, surprise, contempt
Object Detection	YOLO v8	Multi-frame voting ensures accurate detection across animation frames
Action Recognition	VideoMAE	Detects activities (dancing, jumping, etc.) with motion-based fallback
Person Counting	YOLO	Counts people in the GIF with confidence thresholds
Lighting Analysis	OpenCV	Classifies brightness as dim, moderate, or bright
Caption Generation	Template-based	Context-aware English descriptions with emotion-specific vocabulary

API Features

RESTful endpoints with FastAPI
Automatic API documentation (Swagger UI)
CORS enabled for cross-domain requests
Error handling and validation
Detailed metadata in responses

Frontend Features

Modern, responsive UI (React + Tailwind CSS)
File upload and URL-based GIF input
Real-time processing feedback
Detailed result visualization
Emotion and metadata display
Dark theme with gradient design

Tech Stack

Backend:

Python 3.11+
FastAPI (REST API framework)
PyTorch (Deep learning)
YOLO v8 (Object detection)
VideoMAE (Action recognition)
OpenCV (Image processing)
Hugging Face Hub (Model distribution)

Frontend:

React
TypeScript
Tailwind CSS (Styling)
Vite (Build tool)
Vercel (Deployment)

Getting Started

Prerequisites

Python 3.11 or higher
Node.js 16+ (for frontend development)
CUDA 11.8+ (optional, for GPU acceleration)
Git

Backend Installation

# Clone the repository
git clone https://github.com/Akindu27/Emotion-Based-GIF-captioning-system.git
cd Prototype_final/GIF\ captioner_v2/project/backend

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Download Models
Before proceeding with installation, you need to download the pre-trained models:

1. Download the models from this [Google Drive link](https://drive.google.com/drive/folders/1cQ8Yaxt1MhlZDCsMcELU3IoT5QeYbEWr?usp=sharing)

2. Extract and place them in the `models` directory:

Emotion-Based-GIF-captioning-system/Prototype_final/GIF captioner_v2/project/backend/models


3. The models directory should contain all downloaded model files.

# Install dependencies
pip install -r requirements.txt

# Run locally (development)
python main_final.py

The backend will start at http://localhost:7860 with:

API: http://localhost:7860/generate (POST)
Docs: http://localhost:7860/docs
Health: http://localhost:7860/health

Frontend Installation

# Navigate to project root
cd Prototype_final/GIF\ captioner_v2/project

# Install dependencies
npm install

# Run development server
npm run dev

The frontend will be available at http://localhost:5173

Environment Variables

Backend (`.env` or Space secrets)

# Model repository (Hugging Face Hub)
HF_MODEL_REPO=Akindu27/sentivue-models

# Local models directory
MODELS_DIR=models

# Optional: HuggingFace token for authenticated downloads
HF_TOKEN=your_hf_token_here

Frontend (`.env.local` or `.env.production`)

# Local development
VITE_API_URL=http://localhost:7860

# Production (HF Space)
VITE_API_URL=https://Akindu27-sentivue-backend.hf.space

API Usage

Generate Caption

Endpoint: POST /generate

Request:

curl -X POST "http://localhost:7860/generate" \
  -F "file=@your_gif.gif"

Response:

{
  "emotion": "positive_energetic",
  "caption": "a joyful person dancing with a dog",
  "confidence": 0.92,
  "objects": ["dog", "person"],
  "action": "dancing",
  "content_type": "real_world",
  "content_warning": null,
  "person_count": 1,
  "lighting": {
    "brightness": 120.5,
    "lighting_label": "moderate"
  }
}

Health Check

Endpoint: GET /health

Response:

{
  "status": "healthy",
  "device": "cpu",
  "models": {
    "emotion": "loaded",
    "objects": "loaded",
    "action": "loaded"
  },
  "emotion_groups": ["contempt", "negative_intense", ...]
}

Root/Info

Endpoint: GET /

Returns API information and available features.

Model Details

Emotion Classifier

Architecture: ResNet50 (feature extractor) + 2-layer classifier
Number of Classes: 6 emotion groups
Input Size: 224x224 RGB
Output: Emotion label + confidence score
Training Data: GIFGIF dataset (grouped emotions)

Object Detector

Architecture: YOLO v8 Nano
Detection Strategy: Multi-frame voting across 8 sampled frames
Confidence Threshold: 0.20
Post-processing: Removes generic "person" labels, keeps specific objects
Top Results: 2 most frequently detected objects

Action Recognizer

Architecture: VideoMAE (vision transformer for video)
Input: 16 evenly sampled frames
Confidence Threshold: 0.15
Fallback: Motion-based heuristics if confidence is low
Outputs: Actions like dancing, jumping, moving, gesturing

Development

Running Tests

# Backend health check
curl http://localhost:7860/health

# Frontend build
cd project && npm run build

# Check for TypeScript errors
npm run type-check

Code Quality

Linting: ESLint configured for React + TypeScript
Formatting: Prettier (run via VS Code or npm run format)
Type Safety: Full TypeScript coverage on frontend
Error Handling: Try-catch blocks with logging in backend

Deployment

Backend Deployment (HuggingFace Spaces)

Create a new Space at https://huggingface.co/new-space
Select Docker SDK
Connect GitHub repository
Add required secrets (HF_TOKEN optional for private models)
Space automatically rebuilds and deploys on push

Deployed at: https://Akindu27-sentivue-backend.hf.space

Frontend Deployment (Vercel)

Push to GitHub repository
Connect to Vercel at https://vercel.com/import
Configure environment variable: VITE_API_URL=https://Akindu27-sentivue-backend.hf.space
Vercel auto-deploys on main branch push

Deployed at: https://sentivue-olive.vercel.app/

Troubleshooting

General Issues

Issue	Solution
Models fail to download	Set HF_TOKEN environment variable
CORS errors	Verify backend and frontend URLs match in .env files
Out of memory	Use CPU mode (default) or reduce GIF size
Slow processing	Ensure backend is running on appropriate device

Backend Issues

# Check if backend is running
curl http://localhost:7860/health

# View logs (development)
python main_final.py

# Verify models are downloaded
ls -la backend/models/

Frontend Issues

# Clear cache and reinstall
rm -rf node_modules package-lock.json
npm install

# Build and test locally
npm run build
npm run preview

Performance

GIF Processing: 2-5 seconds (depends on GIF size and complexity)
Model Inference: Optimized for CPU; GPU recommended for production
Memory Usage: ~2GB RAM for all models
Concurrent Requests: Limited by available resources

Contributing

Contributions are welcome! Areas for improvement:

Add more emotion categories
Improve action recognition accuracy
Optimize model inference speed
Add batch processing capability
Expand caption template diversity
Multi-language support

Development Workflow

# Create feature branch
git checkout -b feature/your-feature

# Make changes and test
npm run dev          # frontend
python main_final.py # backend

# Commit and push
git add .
git commit -m "feat: your feature description"
git push origin feature/your-feature

# Create pull request

Author

Akindu27 - GitHub | HuggingFace Hub

Back to top | SentiVue - Emotion-Based GIF Captioning System

Name		Name	Last commit message	Last commit date
Latest commit History 48 Commits
GIFGIF		GIFGIF
Prototype_final/GIF captioner_v2		Prototype_final/GIF captioner_v2
v1_1k_TGIF		v1_1k_TGIF
v2_10k_TGIF		v2_10k_TGIF
.gitignore		.gitignore
README.md		README.md
models.txt		models.txt

Folders and files

Latest commit

History

Repository files navigation