Skip to content

Commit

Permalink
Merge pull request #522 from Gui-Yue/bpf_visual
Browse files Browse the repository at this point in the history 
supply for lock_image
  • Loading branch information
@chenamy2017
chenamy2017 authored Sep 22, 2023
2 parents 9e9e2ac + 3e1426f commit fa2ce91
Show file tree
Hide file tree
Showing 12 changed files with 434 additions and 240 deletions.
3 changes: 3 additions & 0 deletions eBPF_Visualization/eBPF_prometheus/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@ all:

start_service:
./runimages.sh

clean_data:
rm ./dao/data.db
68 changes: 66 additions & 2 deletions eBPF_Visualization/eBPF_prometheus/Readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,29 @@ proc_image:
$ make
$ ./data-visual proc_image
```
打开 localhost:8090/metrics 可查看输出的信息。

打开 localhost:8090/metrics 可查看输出的信息。
启动grafana服务，在grafana中安装JSON API,之后选择使用JSON API连接，使用stateTimeline作为展示图，配置方式如下所示：
![8](https://github.com/Gui-Yue/lmp/assets/78520005/60c4f70b-b51f-409a-9715-4fe3c8a0d87d)
![9](https://github.com/Gui-Yue/lmp/assets/78520005/4bf9a907-1a59-4051-a6e4-133d917f96a7)

效果图如下：
![10](https://github.com/Gui-Yue/lmp/assets/78520005/d053b7ef-82a8-4f61-9a68-fd852c987bea)

lock_image:
先打开/collector/tmux_proc_setting.yaml进行初始化适配，填入需要的信息
```bash
$ make
$ ./data-visual tmux
```
打开 localhost:8090/metrics 可查看输出的信息。
启动grafana服务，在grafana中安装JSON API,之后选择使用JSON API连接，使用stateTimeline作为展示图，配置方式如下所示：
![tmux](https://github.com/Gui-Yue/lmp/assets/78520005/02198183-52b7-49f8-a2bb-43b4458e3552)
![tmuxmap](https://github.com/Gui-Yue/lmp/assets/78520005/262b7b04-9009-48f4-86a9-9bf016458eb3)

效果图如下：
![tmuxexhibition](https://github.com/Gui-Yue/lmp/assets/78520005/1e15f09d-ada4-4742-a3ee-e513ede3bb86)

### 使用prometheus-server的docker镜像监控metrics

编辑prom_core/prometheus.yaml中targets参数，使其符合用户ip地址，默认127.0.0.1。
Expand All @@ -69,4 +83,54 @@ $ ./data-visual proc_image
![捕获2](https://github.com/Gui-Yue/lmp/assets/78520005/b7bb8668-b3cb-496a-bbfc-ba74ea3ef1b7)

### 通过sqlite3查看收集到的数据
进入/dao目录可以看到data.db数据库文件，通过sqlite3访问查看收集到的数据。
进入/dao目录可以看到data.db数据库文件，通过sqlite3访问查看收集到的数据。

# Grafana配置指南

利用eBPF_prometheus工具，可以搭建其metrics页面和Prometheus服务，之后在grafana中进行简单的配置即可获取直观的可视化数据展示。以下是使用`/lmp/eBPF_Supermarket/Network_Subsystem/net_watch`中的`net_watch`工具作为示例展示如何配置多样化的可视化方案。

## Step1 启动相关服务

在启动监控程序之前，先配置`/lmp/eBPF_Visualization/eBPF_prometheus/prom_core/prometheus.yaml`，scrape_interval是采集密度，即两次采集之间的时间差，targets为metrics服务地址，默认127.0.0.1:8090。

```bash
$ make
$ make start_service
$ ./data-visual c path/net_watch -t
```

## Step2 配置grafana服务

利用浏览器打开`127.0.0.1:3000`,使用初始密码登录（user:admin pswd: admin）进入管理界面。

1、建立与Prometheus服务器的连接

![grafana1](https://github.com/Gui-Yue/lmp/assets/78520005/e9b5a550-2f26-4253-a8a4-2fba5e365f9b)


2、选择Prometheus，并按照如图所示进行配置，配置完后点击`Save&test`完成连接的配置

![grafana2](https://github.com/Gui-Yue/lmp/assets/78520005/6a10b977-0aed-4b9b-aac6-305734c6a4a3)


![grafana3](https://github.com/Gui-Yue/lmp/assets/78520005/018aa95a-9ff2-4d6b-9024-495eb906d3b6)

3、可视化配置

按图中所示点击Add，选择Visualization，进入配置界面

![grafana4](https://github.com/Gui-Yue/lmp/assets/78520005/6d182d46-0f1e-47a3-b282-0f9c65a48a57)


在下方query处进行如图所示的配置，点击Run queries即可以可视化的方式监控MAC_TIME字段的输出数据

![grafana5](https://github.com/Gui-Yue/lmp/assets/78520005/91669ba0-b2bf-43e6-9179-1eac68bffa3b)


仪表盘右侧是对图像的一些调整，可以选择想要的可视化效果，也可对图像效果进行调整，例如颜色图形样式等。读者可以自行尝试。

![grafana6](https://github.com/Gui-Yue/lmp/assets/78520005/2d86b7bd-1f86-4e00-97c9-35c5a7ce380d)


配置完成后点击Apply,即可保存可视化方案。

18 changes: 17 additions & 1 deletion eBPF_Visualization/eBPF_prometheus/checker/check.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,13 @@ func IsTCPwatchFirst(string2 string) bool {
}

func IsTcpObjection(string2 string) bool {
pattern := `tcpwatch`
pattern := `netwatch`
match, _ := regexp.MatchString(pattern, string2)
return match
}

func IsProcimage(string2 string) bool {
pattern := `proc`
match, _ := regexp.MatchString(pattern, string2)
return match
}
Expand All @@ -129,3 +135,13 @@ func InvalidTcpData(string2 string) bool {
}
return false
}

func Istmuxlineone(string2 string) bool {
is, _ := regexp.MatchString(`pid`, string2)
return is
}

func Istmuxlinetwo(string2 string) bool {
is, _ := regexp.MatchString(`acq_time`, string2)
return is
}
20 changes: 13 additions & 7 deletions eBPF_Visualization/eBPF_prometheus/collector/collect_output.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,14 @@ func init() {
log.Fatalf("Failed to load ... error:%s\n", err)
return
}
tmuxSvc := Aservice{
Name: "tmuxCollectData",
Desc: "collect data from lock_image",
NewInst: newTmuxCmd}
if err := AddAService(&tmuxSvc); err != nil {
log.Fatalf("Failed to load ... error:%s\n", err)
return
}
}

func newCollectCmd(ctx *cli.Context, opts ...interface{}) (interface{}, error) {
Expand All @@ -110,6 +118,10 @@ func newProcCmd(ctx *cli.Context, opts ...interface{}) (interface{}, error) {
return proc_imageCommand, nil
}

func newTmuxCmd(ctx *cli.Context, opts ...interface{}) (interface{}, error) {
return tmux_command, nil
}

type BPF_name struct {
Name string
}
Expand Down Expand Up @@ -161,13 +173,7 @@ func (b *BPF_name) Run(filePath string) error {

mapchan := make(chan []map[string]interface{}, 2)

if checker.IsTcpObjection(cmdStr) {
log.Println("I am TCPWatch")
go RedirectTcpWatch(stdout, mapchan)
} else {
go redirectStdout(stdout, mapchan)
log.Println("I am normal")
}
go redirectStdout(stdout, mapchan)

metricsobj := &prom_core.MyMetrics{BPFName: b.Name, Sqlinited: false}
sqlobj := &dao.Sqlobj{Tablename: b.Name}
Expand Down
127 changes: 89 additions & 38 deletions eBPF_Visualization/eBPF_prometheus/collector/collect_proc_image.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,18 @@ import (
"log"
"os"
"os/exec"
"regexp"
"strings"
"syscall"
"time"
)

// Pro_Setting 定义了设置项，通过读取同目录下的proc_setting.yaml实现对基本信息的设置。
type Proc_Setting struct {
Name string `yaml:"name"`
Path string `yaml:"path"`
Pid string `yaml:"pid"`
Max_Records int `yaml:"max_records"`
Name string `yaml:"proc_name"`
Path string `yaml:"proc_path"`
Pid string `yaml:"proc_pid"`
Max_Records int `yaml:"proc_max_records"`
}

var proc_imageCommand = cli.Command{
Expand All @@ -50,33 +51,48 @@ var proc_imageCommand = cli.Command{
}

// Get_Setting 函数用于获取设置的信息
func Get_Setting() (error, string, int) {
func Get_Setting(which string) (error, string, int) {
currentDir, _ := os.Getwd()
content, err := os.ReadFile(currentDir + "/collector/proc_setting.yaml")
content, err := os.ReadFile(currentDir + "/collector/tmux_proc_setting.yaml")
if err != nil {
log.Fatalf("Error reading file: %v", err)
return err, "", 0
}
var setting Proc_Setting
err = yaml.Unmarshal(content, &setting)
if err != nil {
log.Fatalf("Error unmarshaling YAML :%v", err)
return err, "", 0
}
command := ""
maxrecords := 0
if which == "proc" {
var setting Proc_Setting
err = yaml.Unmarshal(content, &setting)
if err != nil {
log.Fatalf("Error unmarshaling YAML :%v", err)
return err, "", 0
}

command = setting.Path + " -p " + setting.Pid
maxrecords = setting.Max_Records
} else if which == "tmux" {
var setting Tmux_Setting
err = yaml.Unmarshal(content, &setting)
if err != nil {
log.Fatalf("Error unmarshaling YAML :%v", err)
return err, "", 0
}

command := setting.Path + " -p " + setting.Pid
maxrecords := setting.Max_Records
command = setting.Path + " -p " + setting.Pid
maxrecords = setting.Max_Records
} else {
log.Fatalf("select setting failed.")
}
return nil, command, maxrecords
}

func procCollect(ctx *cli.Context) error {
_, command, _ := Get_Setting()
_, command, _ := Get_Setting("proc")
return ProcRun(command)
}

// ProcRun 是收集器的主函数，通过goroutin的方式实现数据收集，重定向，与prom_core包实现通信。
func ProcRun(command string) error {
_, _, maxrecords := Get_Setting()
cmdStr := CheckFileType(command)
cmd := exec.Command("sh", "-c", cmdStr)

Expand All @@ -93,30 +109,65 @@ func ProcRun(command string) error {

loc, _ := time.LoadLocation("Asia/Shanghai")
currenttime := float64(time.Now().In(loc).UnixNano()) / 1e9
go redirectProc(stdout, mapchan)

procdata := prom_core.ProcMetrics{Max_records: maxrecords, NowTime: currenttime}
sqlobj := &dao.Sqlobj{Tablename: "proc_image_data"}
procdata.Sqlobj = sqlobj
// process chan from redirectProc Stdout
go procdata.BootProcService()

go func() {
for {
select {
case <-mapchan:
procdata.Getorigindata(mapchan)
if procdata.Sqlinted {
procdata.UpdateSql()
} else {
procdata.Initsql()

pathlist := strings.Split(command, "/")
is_proc, _ := regexp.MatchString(`proc`, pathlist[len(pathlist)-1])
is_lifecycle, _ := regexp.MatchString(`lifecycle`, pathlist[len(pathlist)-1])
is_lock, _ := regexp.MatchString(`lock`, pathlist[len(pathlist)-1])

if is_proc || is_lifecycle {
log.Println("This is lifecycle")
_, _, maxrecords := Get_Setting("proc")
go redirectProc(stdout, mapchan)

procdata := prom_core.ProcMetrics{Max_records: maxrecords, NowTime: currenttime}
sqlobj := &dao.Sqlobj{Tablename: "proc_image_data"}
procdata.Sqlobj = sqlobj
// process chan from redirectProc Stdout
go procdata.BootProcService()

go func() {
for {
select {
case <-mapchan:
procdata.Getorigindata(mapchan)
if procdata.Sqlinted {
procdata.UpdateSql()
} else {
procdata.Initsql()
}
procdata.UpdateRecords()
<-mapchan
default:
}
procdata.UpdateRecords()
<-mapchan
default:
}
}
}()
}()

} else if is_lock {
log.Println("This is lock_image.")
_, _, maxrecords := Get_Setting("tmux")
go redirectTmux(stdout, mapchan)
tmuxdata := prom_core.TmuxMetrics{Max_records: maxrecords, NowTime: currenttime}
sqlobj := &dao.Sqlobj{Tablename: "tmux_data"}
tmuxdata.Sqlobj = sqlobj
go tmuxdata.BootProcService()
go func() {
for {
select {
case <-mapchan:
tmuxdata.Getorigindata(mapchan)
if tmuxdata.Sqlinted {
tmuxdata.UpdateSql()
} else {
tmuxdata.Initsql()
}
tmuxdata.UpdateRecords()
<-mapchan
default:
}
}
}()
}

err = cmd.Start()
if err != nil {
Expand Down
Loading

0 comments on commit fa2ce91

Please sign in to comment.