Skip to content

feat: add static robots.txt file #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PaiJi merged 1 commit into from
Feb 21, 2026
Merged

feat: add static robots.txt file #300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
User-agent: *

Sitemap: https://www.furrycons.cn/sitemap/index.xml
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use deployment host in robots sitemap URL

The new static robots file hardcodes https://www.furrycons.cn/sitemap/index.xml, which regresses the previous host-aware behavior for any deployment where the hostname is not www.furrycons.cn (for example staging or self-hosted mirrors). The codebase still supports configurable public hosts via NEXT_PUBLIC_WEBSITE_URL (see src/utils/env.ts and URL/canonical helpers), so this fixed value can cause crawlers to fetch the wrong sitemap and miss the current deployment’s URLs.

Useful? React with 👍 / 👎.

Copy link

@coderabbitai coderabbitai bot Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

git ls-files | head -20

Repository: FurryConsChina/website

Length of output: 464

🏁 Script executed:

# Find key config and deployment files
fd -e yml -e yaml -e json -e sh | grep -E "(github|gitlab|jenkins|ci|cd|deploy|build)" | head -20

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Check for robots.txt related files and configs
fd "robots" --type f

Repository: FurryConsChina/website

Length of output: 83

🏁 Script executed:

# Look for environment variable configurations and overrides
fd -e env -e config | head -20

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Read the current robots.txt content
cat public/robots.txt

Repository: FurryConsChina/website

Length of output: 132

🏁 Script executed:

# Check deployment workflow for production
cat .github/workflows/deploy-prod-cn.yml

Repository: FurryConsChina/website

Length of output: 1723

🏁 Script executed:

# List all GitHub workflows to see if there are staging/preview deployments
ls -la .github/workflows/

Repository: FurryConsChina/website

Length of output: 526

🏁 Script executed:

# Check .env.example for environment variables
cat .env.example

Repository: FurryConsChina/website

Length of output: 257

🏁 Script executed:

# Search for any robots-related typescript/javascript files
find . -name "*robots*" -type f

Repository: FurryConsChina/website

Length of output: 85

🏁 Script executed:

# Search for NEXT_PUBLIC_WEBSITE_URL usage
rg "NEXT_PUBLIC_WEBSITE_URL" -A 2 -B 2

Repository: FurryConsChina/website

Length of output: 231

🏁 Script executed:

# Search for any typescript/javascript robots files in src directory
find src -name "*robots*" -type f 2>/dev/null

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Check next.config.js for robots.txt handling
cat next.config.js

Repository: FurryConsChina/website

Length of output: 3630

🏁 Script executed:

# Check Dockerfile for environment-specific handling
cat Dockerfile

Repository: FurryConsChina/website

Length of output: 2441

🏁 Script executed:

# Search for staging, preview, or dev deployment configurations
rg -i "staging|preview|dev" --glob "*.yml" --glob "*.yaml" --glob "*.json" -A 2 -B 2

Repository: FurryConsChina/website

Length of output: 2500

🏁 Script executed:

# Check if there are environment-specific public directory or build steps
rg -rn "public/robots" -A 2 -B 2

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Search for any robots-related environment overrides
rg -rn "robots\.txt" -A 3 -B 3

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Search more broadly for any staging/preview/dev references
rg -i "(staging|preview|dev|development)" . --type-list | head -30

Repository: FurryConsChina/website

Length of output: 455

🏁 Script executed:

# Check README for deployment information
cat README.md | grep -i -A 5 -B 5 "deploy\|staging\|preview"

Repository: FurryConsChina/website

Length of output: 1322

🏁 Script executed:

# Check if there are any build scripts or deployment scripts
find . -name "*.sh" -o -name "Makefile" | head -20

Repository: FurryConsChina/website

Length of output: 48

🏁 Script executed:

# Search for any references to multiple environments or domain configurations
rg "furrycons\.cn" -A 2 -B 2

Repository: FurryConsChina/website

Length of output: 6175

🏁 Script executed:

# Check for any conditional environment handling in next.config.js or build process
rg "process\.env\." next.config.js

Repository: FurryConsChina/website

Length of output: 135

Sitemap URL 硬编码生产域名，非生产环境面临索引风险

robots.txt 已是静态文件，Sitemap 地址被固定为 https://www.furrycons.cn/sitemap/index.xml。更严重的是，该文件完全缺少 Disallow: / 规则，仅包含 User-agent: * 和 Sitemap 指向，意味着搜索引擎将索引该文件所在的任何环境中的所有页面。

存在的风险：

  1. 非生产环境被完整索引：README 提及 PR 提交会触发 Firebase 部署，若这些预览环境对外可访问，搜索引擎会抓取到不带 Disallow 规则的 robots.txt，导致这些临时环境被搜索引擎完整收录。
  2. 域名变更时需手动维护：静态文件需手动同步更新。

建议：

  • 恢复动态生成方案（使用 NEXT_PUBLIC_WEBSITE_URL 环境变量），确保各环境自动适配对应域名，并针对非生产环境注入 Disallow: / 规则；或
  • 若保留静态文件，在 CI/CD 流程中为非生产环境部署显式添加覆盖步骤，确保注入 Disallow: / 规则以防止索引。
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@public/robots.txt` at line 3, robots.txt currently hardcodes "Sitemap:
https://www.furrycons.cn/sitemap/index.xml" and lacks any Disallow rule; update
the implementation so the sitemap URL is generated from NEXT_PUBLIC_WEBSITE_URL
(or equivalent env var) and ensure non-production environments emit a "Disallow:
/" line—specifically modify the robots.txt generation/serving logic (or CI
deployment step that writes public/robots.txt) to build the sitemap entry from
NEXT_PUBLIC_WEBSITE_URL and conditionally prepend "Disallow: /" when NODE_ENV
!== "production" (or when the env indicates a preview), so preview/staging
deployments are not indexed.

12 changes: 0 additions & 12 deletions src/components/announcementSlider/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,6 @@ function AnnouncementSlider() {
</span>
复制群号。
</div>

<div className={clsx(SliderStyle.embla__slide, defaultSliderClass, "ml-4", !IS_CN_REGION && "mr-4")}>
由于不可抗力，兽人控游戏库的网址变更为
<a
target="_blank"
href="https://srk.games/?utm_source=fec"
className="underline cursor-pointer font-bold mx-1"
>
srk.games
</a>
。
</div>
</div>
</div>
);
Expand Down
30 changes: 0 additions & 30 deletions src/pages/robots.txt.ts
View file
Open in desktop

This file was deleted.

Loading