Skip to content

Commit 0f43ed6

Browse files
author
likaia
committed
feat: 新增对IIS日志的解析支持
实时访问页面的卡片面板新增降序排列按钮 概况页卡片面板的预计今日添加算法实现说明 数据日报新增访客TOP10功能 优化内容区域的顶部样式 修复二级路径设置不生效问题
1 parent 9940b2c commit 0f43ed6

26 files changed

Lines changed: 1681 additions & 87 deletions

‎docker-compose-test.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
services:
22
nginxpulse:
3-
image: magiccoders/nginxpulse:v1.6.11-beta.7
3+
image: nginxpulse:test-local
4+
build:
5+
context: .
46
container_name: local_nginxpulse
57
ports:
68
- "9122:8088"
7-
- "8089:8089"
9+
- "19089:8089"
810
volumes:
911
- ./docker_local/pgdata:/app/var/pgdata
1012
- ./docker_local/logs:/share/logs

‎docs/wiki/Configuration-EN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ https://example.com/m/?tabbarBottom=false
9999
- `name` (string, required): site name. ID is derived from this.
100100
- `logPath` (string, required): log path, supports `*` glob.
101101
- `domains` (string[]): domain list.
102-
- `logType` (string): `nginx`, `caddy`, `nginx-proxy-manager` (`npm`), `apache` (`httpd`), `haproxy`, `traefik`, `envoy`, `tengine`, `nginx-ingress` (`ingress-nginx`), `traefik-ingress`, or `haproxy-ingress`, default `nginx`.
102+
- `logType` (string): `nginx`, `caddy`, `nginx-proxy-manager` (`npm`), `apache` (`httpd`), `iis` (`iis-w3c`), `haproxy`, `traefik`, `envoy`, `tengine`, `nginx-ingress` (`ingress-nginx`), `traefik-ingress`, or `haproxy-ingress`, default `nginx`.
103103
- `logFormat` (string): custom format with `$vars`.
104104
- `logRegex` (string): custom regex with named groups.
105105
- `timeLayout` (string): custom time layout.

‎docs/wiki/Configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ https://example.com/m/?tabbarBottom=false
101101
- 示例: `/var/log/nginx/access.log`
102102
- 示例: `/var/log/nginx/access_*.log`
103103
- `domains` (string[]): 站点域名列表。
104-
- `logType` (string): 日志类型,支持 `nginx`、`caddy`、`nginx-proxy-manager`(或 `npm`)、`apache`(或 `httpd`)、`haproxy`、`traefik`、`envoy`、`tengine`、`nginx-ingress`(或 `ingress-nginx`)、`traefik-ingress`、`haproxy-ingress`,默认 `nginx`。
104+
- `logType` (string): 日志类型,支持 `nginx`、`caddy`、`nginx-proxy-manager`(或 `npm`)、`apache`(或 `httpd`)、`iis`(或 `iis-w3c`)、`haproxy`、`traefik`、`envoy`、`tengine`、`nginx-ingress`(或 `ingress-nginx`)、`traefik-ingress`、`haproxy-ingress`,默认 `nginx`。
105105
- `logFormat` (string): 自定义日志格式(带 `$变量`)。
106106
- `logRegex` (string): 自定义正则(需命名分组)。
107107
- `timeLayout` (string): 时间解析格式,留空走默认。

‎docs/wiki/Log-Parsing-EN.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,30 @@ Poll this endpoint to update progress in UI.
2929
- IP geo is resolved in batches after parsing.
3030
- For speed: increase `parseBatchSize`, use faster disk, or split logs by day.
3131

32+
## IIS default rule (W3C Extended)
33+
NginxPulse now supports `logType=iis` (alias: `iis-w3c`). The built-in parser follows the common IIS W3C default field order:
34+
35+
`date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken`
36+
37+
Notes:
38+
- Metadata lines starting with `#` (such as `#Software`, `#Version`, `#Fields`) are skipped automatically.
39+
- URL is built from `cs-uri-stem`; when `cs-uri-query` is not `-`, it is appended as `path?query`.
40+
- IIS W3C timestamps are typically UTC, and the default time layout is `2006-01-02 15:04:05`.
41+
42+
Config example:
43+
```json
44+
{
45+
"name": "iis-site",
46+
"logPath": "/var/log/iis/u_ex*.log",
47+
"logType": "iis"
48+
}
49+
```
50+
51+
Sample line:
52+
```text
53+
2026-02-08 10:05:34 10.0.0.10 GET /index.html a=1&b=2 443 - 203.0.113.8 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64) https://example.com/ 200 0 0 36
54+
```
55+
3256
## Retention
3357
- `system.logRetentionDays` controls cleanup.
3458
- Cleanup runs at 02:00 (system timezone).

‎docs/wiki/Log-Parsing.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,30 @@
2929
- 归属地解析在后台批量回填,不阻塞主解析。
3030
- 如需更快:调大 `parseBatchSize`、提高机器 IO 或将日志按天切分。
3131

32+
## IIS 默认规则(W3C Extended)
33+
NginxPulse 现已支持 `logType=iis`(别名:`iis-w3c`),默认按 IIS W3C 扩展日志的常见默认字段顺序解析:
34+
35+
`date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken`
36+
37+
注意点:
38+
- 日志中以 `#` 开头的元数据行(如 `#Software`、`#Version`、`#Fields`)会自动跳过。
39+
- URL 会优先取 `cs-uri-stem`,当 `cs-uri-query` 不是 `-` 时会自动拼接为 `path?query`。
40+
- IIS W3C 默认时间按 UTC 记录,默认时间格式为 `2006-01-02 15:04:05`。
41+
42+
配置示例:
43+
```json
44+
{
45+
"name": "iis-site",
46+
"logPath": "/var/log/iis/u_ex*.log",
47+
"logType": "iis"
48+
}
49+
```
50+
51+
示例日志行:
52+
```text
53+
2026-02-08 10:05:34 10.0.0.10 GET /index.html a=1&b=2 443 - 203.0.113.8 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64) https://example.com/ 200 0 0 36
54+
```
55+
3256
## 日志清理
3357
- `system.logRetentionDays` 控制保留天数。
3458
- 清理任务在系统时间凌晨 2 点触发(按系统时区)。

‎entrypoint.sh‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -204,9 +204,9 @@ extract_web_base_path() {
204204
if [ -n "${WEB_BASE_PATH:-}" ]; then
205205
raw="$WEB_BASE_PATH"
206206
elif [ -n "${CONFIG_JSON:-}" ]; then
207-
raw="$(printf '%s' "$CONFIG_JSON" | tr '\n' ' ' | sed -n 's/.*\"webBasePath\"[[:space:]]*:[[:space:]]*\"\\([^\\"]*\\)\".*/\\1/p' | head -n 1)"
207+
raw="$(printf '%s' "$CONFIG_JSON" | tr '\n' ' ' | sed -n 's/.*\"webBasePath\"[[:space:]]*:[[:space:]]*\"\([^\"]*\)\".*/\1/p' | head -n 1)"
208208
elif [ -f "$CONFIG_DIR/nginxpulse_config.json" ]; then
209-
raw="$(sed -n 's/.*\"webBasePath\"[[:space:]]*:[[:space:]]*\"\\([^\\"]*\\)\".*/\\1/p' "$CONFIG_DIR/nginxpulse_config.json" | head -n 1)"
209+
raw="$(sed -n 's/.*\"webBasePath\"[[:space:]]*:[[:space:]]*\"\([^\"]*\)\".*/\1/p' "$CONFIG_DIR/nginxpulse_config.json" | head -n 1)"
210210
fi
211211
normalize_web_base_path "$raw"
212212
}
@@ -218,6 +218,7 @@ write_app_config() {
218218
prefix="/$base_path"
219219
fi
220220
printf 'window.__NGINXPULSE_BASE_PATH__ = "%s";\n' "$prefix" > /usr/share/nginx/html/app-config.js
221+
printf 'window.__NGINXPULSE_BASE_PATH__ = "%s";\n' "$prefix" > /usr/share/nginx/html/m/app-config.js
221222
}
222223

223224
write_nginx_conf() {
@@ -279,6 +280,11 @@ server {
279280
try_files \$uri =404;
280281
}
281282
283+
location = /m/app-config.js {
284+
add_header Cache-Control "no-store";
285+
try_files \$uri =404;
286+
}
287+
282288
location = /favicon.svg {
283289
try_files \$uri =404;
284290
}
@@ -287,6 +293,22 @@ server {
287293
try_files \$uri =404;
288294
}
289295
296+
location = /m/favicon.svg {
297+
try_files \$uri =404;
298+
}
299+
300+
location = /m/brand-mark.svg {
301+
try_files \$uri =404;
302+
}
303+
304+
location = /index.html {
305+
try_files \$uri =404;
306+
}
307+
308+
location = /m/index.html {
309+
try_files \$uri =404;
310+
}
311+
290312
location /assets/ {
291313
try_files \$uri =404;
292314
}
@@ -312,14 +334,18 @@ server {
312334
proxy_set_header X-Forwarded-Proto \$scheme;
313335
}
314336
337+
location = /$base_path/api {
338+
return 308 /$base_path/api/;
339+
}
340+
315341
location /$base_path/m/ {
316342
rewrite ^/$base_path/m/(.*)$ /m/\$1 break;
317-
try_files \$uri \$uri/ /m/index.html;
343+
try_files \$uri /m/index.html;
318344
}
319345
320346
location /$base_path/ {
321347
rewrite ^/$base_path/(.*)$ /\$1 break;
322-
try_files \$uri \$uri/ /index.html;
348+
try_files \$uri /index.html;
323349
}
324350
325351
location / {

‎internal/analytics/client_stats.go‎

Lines changed: 65 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,13 @@ func NewrefererStatsManager(userRepoPtr *store.Repository) *ClientStatsManager {
4343
}
4444
}
4545

46+
func NewRefererIPStatsManager(userRepoPtr *store.Repository) *ClientStatsManager {
47+
return &ClientStatsManager{
48+
repo: userRepoPtr,
49+
statsType: "referer_ip",
50+
}
51+
}
52+
4653
func NewBrowserStatsManager(userRepoPtr *store.Repository) *ClientStatsManager {
4754
return &ClientStatsManager{
4855
repo: userRepoPtr,
@@ -141,6 +148,18 @@ func (s *ClientStatsManager) Query(query StatsQuery) (StatsResult, error) {
141148
groupExpr = "u.url"
142149
case "referer":
143150
joinClause = fmt.Sprintf(`JOIN "%s_dim_referer" r ON r.id = l.referer_id`, query.WebsiteID)
151+
case "referer_ip":
152+
joinClause = fmt.Sprintf(
153+
`JOIN "%s_dim_referer" r ON r.id = l.referer_id JOIN "%s_dim_ip" ip ON ip.id = l.ip_id`,
154+
query.WebsiteID,
155+
query.WebsiteID,
156+
)
157+
selectExpr = "ip.ip"
158+
groupExpr = "ip.ip"
159+
sourceKind, _ := query.ExtraParam["sourceKind"].(string)
160+
if sourceCondition := buildRefererSourceCondition(sourceKind, "r.referer"); sourceCondition != "" {
161+
extraCondition += " AND " + sourceCondition
162+
}
144163
case "user_browser":
145164
joinClause = fmt.Sprintf(`JOIN "%s_dim_ua" ua ON ua.id = l.ua_id`, query.WebsiteID)
146165
selectExpr = "ua.browser"
@@ -183,7 +202,7 @@ func (s *ClientStatsManager) Query(query StatsQuery) (StatsResult, error) {
183202

184203
rows, err := s.repo.GetDB().Query(dbQueryStr, startTime.Unix(), endTime.Unix(), limit)
185204
if err != nil {
186-
return result, fmt.Errorf("查询URL统计失败: %v", err)
205+
return result, fmt.Errorf("查询客户端统计失败: %v", err)
187206
}
188207
defer rows.Close()
189208

@@ -194,7 +213,7 @@ func (s *ClientStatsManager) Query(query StatsQuery) (StatsResult, error) {
194213
var url string
195214
var pv, uv int
196215
if err := rows.Scan(&url, &pv, &uv); err != nil {
197-
return result, fmt.Errorf("解析URL统计结果失败: %v", err)
216+
return result, fmt.Errorf("解析客户端统计结果失败: %v", err)
198217
}
199218
result.Key = append(result.Key, url)
200219
result.PV = append(result.PV, pv)
@@ -204,7 +223,7 @@ func (s *ClientStatsManager) Query(query StatsQuery) (StatsResult, error) {
204223
}
205224

206225
if err := rows.Err(); err != nil {
207-
return result, fmt.Errorf("遍历URL统计结果失败: %v", err)
226+
return result, fmt.Errorf("遍历客户端统计结果失败: %v", err)
208227
}
209228

210229
if totalPV > 0 && totalUV > 0 {
@@ -258,3 +277,46 @@ func normalizeDomain(raw string) string {
258277
raw = strings.TrimSuffix(raw, "/")
259278
return raw
260279
}
280+
281+
func buildRefererSourceCondition(sourceKind string, refererColumn string) string {
282+
directCondition := fmt.Sprintf("(%s = '-' OR %s = '')", refererColumn, refererColumn)
283+
searchCondition := buildSearchEngineRefererCondition(refererColumn)
284+
285+
switch sourceKind {
286+
case "search":
287+
return searchCondition
288+
case "direct":
289+
return directCondition
290+
case "external":
291+
if searchCondition == "" {
292+
return fmt.Sprintf("NOT %s", directCondition)
293+
}
294+
return fmt.Sprintf("(NOT %s AND NOT %s)", directCondition, searchCondition)
295+
default:
296+
return ""
297+
}
298+
}
299+
300+
func buildSearchEngineRefererCondition(refererColumn string) string {
301+
patterns := []string{
302+
"baidu.",
303+
"google.",
304+
"bing.",
305+
"sogou.",
306+
"360.",
307+
"so.com",
308+
"yahoo.",
309+
"duckduckgo.",
310+
"yandex.",
311+
}
312+
313+
conditions := make([]string, 0, len(patterns))
314+
for _, pattern := range patterns {
315+
safePattern := strings.ReplaceAll(strings.ToLower(pattern), "'", "''")
316+
conditions = append(conditions, fmt.Sprintf("LOWER(%s) LIKE '%%%s%%'", refererColumn, safePattern))
317+
}
318+
if len(conditions) == 0 {
319+
return ""
320+
}
321+
return fmt.Sprintf("(%s)", strings.Join(conditions, " OR "))
322+
}

0 commit comments

Comments
 (0)