AcademicSkillHub

中文文档 · Live Demo · MIT License

AcademicSkillHub is an academic and research-oriented directory for AI agent Skills and MCP servers. It discovers, verifies, indexes, and publishes resources that help researchers, students, reviewers, and academic writers work with tools such as Codex, Claude Code, Cursor, Gemini, LangChain/LangGraph, and other agent clients.

The project is designed as a production-ready pnpm monorepo with a public web UI, REST API, scheduled discovery worker, PostgreSQL persistence, Redis queues, Meilisearch indexing, and optional SearxNG/Brave/GitHub-based discovery.

Live Demo

Production site:

https://academic.jaycue.dpdns.org

The admin UI is intentionally hidden from public navigation and is available through /admin/login when an ADMIN_TOKEN is configured.

Features

• Academic Skill and MCP directory: lists verified academic/research Skills and Model Context Protocol servers.
• Verified-only publication policy: public records must resolve to real source content, concrete SKILL.md files, or confirmed MCP directory/registry entries.
• Multi-source discovery:
  • GitHub SKILL.md search
  • GitHub MCP repository search
  • Official MCP Registry
  • Glama, PulseMCP, and MCP.so directory pages
  • skills.sh
  • SkillHub.cn
  • Self-hosted SearxNG search
  • Optional Brave Search API
• Academic relevance scoring for topics such as papers, literature review, peer review, citation, BibTeX, LaTeX, arXiv, PubMed, Zotero, Crossref, OpenAlex, Semantic Scholar, reproducibility, datasets, journal submission, rebuttal, proofreading, and Chinese academic workflows.
• Search and filtering by item kind, category, keyword, client compatibility, rating, and popularity.
• Ratings, reviews, and view counts:
  • Anonymous star ratings
  • GitHub-authenticated text comments
  • Admin approval workflow for comments
  • Same-day duplicate view protection
• Admin console for review, publication, runtime settings, and GitHub OAuth setup.
• Daily scheduled discovery worker with queue-backed crawling.
• Containerized deployment with Docker Compose, Nginx config, and helper scripts.

Architecture

AcademicSkillHub is organized as a TypeScript monorepo managed by pnpm and Turborepo.

academic-skillhub
├── apps
│   ├── web       # Next.js public site, detail pages, admin UI, API proxy routes
│   ├── api       # Fastify REST API, auth, admin, reviews, submissions, stats
│   └── worker    # BullMQ worker and scheduled discovery crawler
├── packages
│   ├── shared    # Shared schemas, scoring, parsing, verification, categories
│   ├── db        # Prisma schema, Prisma client export, seed script
│   └── connectors# Discovery connectors for GitHub, MCP registries/directories, SearxNG, etc.
├── nginx         # Production Nginx virtual host config
├── scripts       # Environment generation and deployment helpers
└── searxng       # Self-hosted SearxNG configuration

Runtime Components

flowchart LR
  Browser[Browser] --> Nginx[Nginx / HTTPS]
  Nginx --> Web[Next.js Web App]
  Web --> API[Fastify API]
  API --> Postgres[(PostgreSQL)]
  API --> Redis[(Redis)]
  API --> Meili[(Meilisearch)]
  Worker[Discovery Worker] --> Redis
  Worker --> Postgres
  Worker --> Meili
  Worker --> GitHub[GitHub / MCP Registries / Directories]
  Worker --> SearxNG[SearxNG]

Loading

Application Responsibilities

• apps/web
  • Renders the public directory, skill/MCP detail pages, filters, ratings, reviews, and admin screens.
  • Uses INTERNAL_API_BASE_URL server-side and /api proxy routes client-side.
• apps/api
  • Exposes REST endpoints under /api/* and Swagger UI under /docs.
  • Handles CORS, rate limiting, GitHub OAuth, sessions, submissions, reviews, admin operations, search indexing, and discovery queue triggers.
• apps/worker
  • Runs scheduled and queued discovery jobs.
  • Calls registered connectors, scores candidates, applies verification rules, writes database records, and updates search indexes.
• packages/shared
  • Provides shared Zod schemas, category definitions, academic scoring, source validation, install-method helpers, and SKILL.md parsing logic.
• packages/db
  • Owns the Prisma schema and database client.
  • Main models include Skill, Candidate, DiscoveryRun, User, Session, Review, RatingVote, Category, Submission, and RuntimeSetting.
• packages/connectors
  • Registers source connectors through packages/connectors/src/registry.ts.
  • New discovery sources should be added here without changing worker orchestration whenever possible.

Technology Stack

• Language: TypeScript / Node.js
• Package manager: pnpm 11.9.0
• Build orchestration: Turborepo
• Web: Next.js, React, Tailwind CSS, lucide-react
• API: Fastify, Zod, Swagger, Fastify CORS/rate limit
• Database: PostgreSQL with Prisma
• Queue/cache: Redis and BullMQ
• Search index: Meilisearch
• Discovery: GitHub APIs, MCP Registry, MCP directories, skills.sh, SkillHub.cn, SearxNG, optional Brave Search
• Deployment: Docker Compose, Nginx, Certbot-compatible domain binding script

Requirements

For local development:

• Node.js 22 or newer recommended
• pnpm 11.9.0 via Corepack
• Docker and Docker Compose for the easiest full-stack setup
• PostgreSQL, Redis, and Meilisearch if you choose to run infrastructure manually

Enable pnpm through Corepack:

corepack enable
pnpm --version

Environment Configuration

Start from the example file:

cp .env.example .env

For production-like deployment, you can generate secure defaults:

./scripts/gen-env.sh

Important variables:

Variable	Purpose
PUBLIC_BASE_URL	Public site URL, for example https://academic.jaycue.dpdns.org
INTERNAL_API_BASE_URL	Server-side web-to-api URL, usually http://api:18121 in Docker
CORS_ORIGINS	Allowed browser origins for API requests
DATABASE_URL	PostgreSQL connection string used by Prisma
REDIS_URL	Redis URL used by API and worker queues
MEILI_HOST / MEILI_MASTER_KEY	Meilisearch endpoint and key
GITHUB_TOKEN	Optional but recommended for GitHub code/repository discovery
GITHUB_OAUTH_CLIENT_ID / GITHUB_OAUTH_CLIENT_SECRET	GitHub login OAuth credentials
AUTH_COOKIE_SECRET	Secret used for user auth cookies
ADMIN_TOKEN	Token for /admin/login and protected admin API access
ADMIN_SESSION_SECRET	Secret used to sign admin sessions
CRAWL_DAILY_CRON	Cron expression for scheduled discovery
RUN_ON_STARTUP	Whether the worker enqueues a discovery job on startup
SEARXNG_URL	Optional self-hosted SearxNG endpoint for web discovery
BRAVE_API_KEY	Optional Brave Search API key

Never commit real .env secrets. The repository keeps .env.example as the reference template.

Installation

1. Install dependencies

pnpm install

2. Generate Prisma client

pnpm db:generate

3. Prepare the database

When using Docker Compose, run the migration/seed helper service:

docker compose --profile tools run --rm migrate

For a manually managed database, use one of:

pnpm db:push      # convenient for development
pnpm db:migrate   # migration deploy flow
pnpm seed

4. Build all packages and apps

pnpm build

Running Locally

Full stack with Docker Compose

cp .env.example .env
# Edit .env values first, especially passwords and tokens.

docker compose build
docker compose --profile tools run --rm migrate
docker compose up -d postgres redis meilisearch searxng api worker web

Local service ports:

Service	URL
Web	http://127.0.0.1:18120
API	http://127.0.0.1:18121
API docs	http://127.0.0.1:18121/docs
Meilisearch	http://127.0.0.1:18122
SearxNG	http://127.0.0.1:18123

View logs:

docker compose logs -f api web worker

Stop services:

docker compose down

Development mode

If PostgreSQL, Redis, and Meilisearch are available through your .env, run:

pnpm dev

This starts the monorepo development tasks in parallel. The web app listens on port 18120 and the API listens on port 18121 by default.

Discovery Worker

The worker defaults to a daily crawl at 08:00 Asia/Shanghai when configured as:

TZ=Asia/Shanghai
CRAWL_DAILY_CRON=0 8 * * *
RUN_ON_STARTUP=false

Run a one-off crawl manually:

pnpm crawl

Common discovery tuning variables:

SEARXNG_URL=http://searxng:8080
SEARXNG_MAX_QUERIES=8
SEARXNG_RESULTS_PER_QUERY=10
SEARXNG_MAX_DISCOVERIES=50
GITHUB_SEARCH_PER_QUERY=8
GITHUB_MAX_DISCOVERIES=80
GITHUB_MCP_MAX_DISCOVERIES=60
MCP_REGISTRY_MAX_DISCOVERIES=120
GLAMA_MCP_MAX_DISCOVERIES=60
PULSEMCP_MAX_DISCOVERIES=60
MCPSO_MAX_DISCOVERIES=60
AUTO_PUBLISH_ACADEMIC_SCORE=0.16
MCP_AUTO_PUBLISH_ACADEMIC_SCORE=0.16
MCP_AUTO_PUBLISH_CONFIDENCE=0.70

Publication and Verification Policy

AcademicSkillHub intentionally avoids publishing placeholders or unverified records:

• Example, demo, placeholder, and watchlist-only records are not public.
• GitHub Skill discoveries must resolve to a concrete SKILL.md and fetch the markdown successfully.
• skills.sh and SkillHub.cn entries are saved only after a concrete upstream skill file is found and verified.
• MCP records are published only after a real registry entry, directory detail page, API response, or GitHub MCP repository signal is fetched.
• Official sites are used as query seeds only; they are not published as skills by themselves.
• Admin publication flows still enforce verification checks before public release.

GitHub Login and Admin Setup

GitHub OAuth is required for text comments and user identity. Anonymous visitors can still submit star ratings.

Create a GitHub OAuth App with callback URL:

https://academic.jaycue.dpdns.org/api/auth/github/callback

Then configure:

GITHUB_OAUTH_CLIENT_ID=
GITHUB_OAUTH_CLIENT_SECRET=
AUTH_COOKIE_SECRET=change_to_a_long_random_value
SESSION_TTL_DAYS=30
ADMIN_GITHUB_LOGINS=your_github_login
ADMIN_SESSION_SECRET=change_to_a_long_random_value
CORS_ORIGINS=https://academic.jaycue.dpdns.org

Recommended admin flow:

1. Open /admin/login.
2. Enter ADMIN_TOKEN.
3. Open /admin.
4. Use the GitHub App Manifest setup button if you prefer automated GitHub OAuth creation.

Admin protection layers:

• /admin/login sets an HttpOnly admin session cookie.
• /admin redirects to login without a valid admin session.
• Browser-side admin calls go through /admin/api/*, which injects ADMIN_TOKEN server-side.
• Backend /api/admin/* endpoints require bearer-token admin authorization.

Production Deployment

A one-shot helper is available for the target domain:

./scripts/one-shot-deploy.sh

Manual production flow:

./scripts/gen-env.sh
pnpm install
pnpm db:generate
docker compose build
docker compose --profile tools run --rm migrate
docker compose up -d api worker web meilisearch redis postgres searxng
sudo ./scripts/bind-domain.sh

The Nginx helper binds:

https://academic.jaycue.dpdns.org -> http://127.0.0.1:18120

The API is consumed by the web app through the configured proxy/internal URL.

Useful Commands

Command	Description
pnpm dev	Run app/package development tasks in parallel
pnpm build	Build all apps and packages
pnpm lint	Run repository lint/type checks configured for each package
pnpm typecheck	Run TypeScript type checks
pnpm test	Run Vitest tests
pnpm format	Check formatting with Prettier
pnpm db:generate	Generate Prisma client
pnpm db:push	Push Prisma schema to the database
pnpm db:migrate	Deploy Prisma migrations
pnpm db:studio	Open Prisma Studio
pnpm seed	Run database seed script
pnpm crawl	Run one discovery crawl

Adding a Discovery Source

1. Add a connector under packages/connectors/src that implements the shared Connector interface.
2. Return normalized RawDiscovery records with source URL, source type, tags, subjects, client compatibility, and metadata.
3. Register the connector in packages/connectors/src/registry.ts.
4. Reuse shared helpers from packages/shared for scoring, URL validation, parsing, and verification.
5. Keep worker orchestration unchanged unless the new source requires a new global workflow.

Contributing

Contributions are welcome. Recommended checks before opening a pull request:

pnpm typecheck
pnpm test
pnpm build
pnpm format

Please keep new public records and discovery connectors aligned with the verified-only publication policy.

License

AcademicSkillHub is released under the MIT License.