机房网络全国延迟监控smokeping+prometheus+grafana

1. smokeping简介

smokeping是一款监控网络状态和稳定性的开源软件（它是rrdtool的作者开发的），通过它可以监控到公司IDC的网络状况，如延时，丢包率，是否BGP多线等

smokeping会向目标设备和系统发送各种类型的测试数据包，测量、记录，并通过rrdtool制图方式，图形化地展示网络的时延情况，进而能够清楚的判断出网络的即时通信情况；通过smokeping来监控IDC机房网络质量情况，可以从监控图上的延时与丢包情况分辨出机房的网络是否稳定，是否为多线，是否为BGP机房以及到各城市的三个运行商网络各是什么情况。如果出现问题，可以有针对性的去处理；如果选择新机房的时候，还可以根据smokeping的监控结果来判断这个机房是否适合。

需要注意的是：smokeping监控的是网络稳定性，而cacti或zabbix监控的是带宽使用情况（即进出口流量）

smokeping会向目标设备和系统发送各种类型的测试数据包，测量、记录，并通过rrdtool制图方式，图形化地展示网络的时延情况，进而能够清楚的判断出网络的即时通信情况；通过smokeping来监控IDC机房网络质量情况，可以从监控图上的延时与丢包情况分辨出机房的网络是否稳定，是否为多线，是否为BGP机房以及到各城市的三个运行商网络各是什么情况。如果出现问题，可以有针对性的去处理；如果选择新机房的时候，还可以根据smokeping的监控结果来判断这个机房是否适合。

需要注意的是：smokeping监控的是网络稳定性，而cacti或zabbix监控的是带宽使用情况（即进出口流量）

获取相关代码

cd /tmp
git clone https://github.com/wilsonchai8/idc_ping_monitor.git

1 2	cd /tmp git clone https://github.com/wilsonchai8/idc_ping_monitor.git

2. 先部署smokeping

2.1. 环境准备

[root@linux-node1 ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
[root@linux-node1 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
[root@linux-node1 ~]# getenforce 
Disabled

[root@linux-node1 ~]# cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

[root@linux-node1 ~]# systemctl status firewalld.service

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: inactive (dead)

[root@linux-node1 ~]# getenforce

Disabled

2.2. 安装rrdtool与依赖库

[root@linux-node1 ~]# yum install -y fping perl perl-Net-Telnet perl-Net-DNS perl-LDAP perl-libwww-perl perl-RadiusPerl perl-IO-Socket-SSL perl-Socket6 perl-CGI-SpeedyCGI perl-FCGI perl-CGI-SpeedCGI perl-Time-HiRes perl-ExtUtils-MakeMaker perl-RRD-Simple rrdtool rrdtool-perl curl fping echoping httpd httpd-devel gcc make wget libxml2-devel libpng-devel glib pango pango-devel freetype freetype-devel fontconfig cairo cairo-devel libart_lgpl libart_lgpl-devel mod_fastcgi python-rrdtool

[root@linux-node1 ~]# yum install -y fping perl perl-Net-Telnet perl-Net-DNS perl-LDAP perl-libwww-perl perl-RadiusPerl perl-IO-Socket-SSL perl-Socket6 perl-CGI-SpeedyCGI perl-FCGI perl-CGI-SpeedCGI perl-Time-HiRes perl-ExtUtils-MakeMaker perl-RRD-Simple rrdtool rrdtool-perl curl fping echoping httpd httpd-devel gcc make wget libxml2-devel libpng-devel glib pango pango-devel freetype freetype-devel fontconfig cairo cairo-devel libart_lgpl libart_lgpl-devel mod_fastcgi python-rrdtool

2.3. 下载与安装smokeping

[root@linux-node1 ~]# cd /opt/smokeping
[root@linux-node1 src]# wget https://oss.oetiker.ch/smokeping/pub/smokeping-2.6.11.tar.gz
[root@linux-node1 src]# tar xf smokeping-2.6.11.tar.gz
[root@linux-node1 src]# cd smokeping-2.6.11
[root@linux-node1 smokeping-2.6.11]# ./configure --prefix=/usr/local/smokeping
# 如果是第一次安装smokeping程序，一般会出现报错，说没有perl模块支持，需要安装模块后在执行安装程序
** Aborting Configure ******************************
   If you know where perl can find the missing modules, set
   the PERL5LIB environment variable accordingly.
   FIRST though, make sure that 'perl' starts the perl
   binary you want to use for SmokePing.
   Now you can install local copies of the missing modules
   by running
     ./setup/build-perl-modules.sh /usr/local/smokeping/thirdparty
   The RRDs perl module is part of RRDtool. Either use the rrdtool
   package provided by your OS or install rrdtool from source.
   If you install from source, the RRDs module is located
   PREFIX/lib/perl

[root@linux-node1 ~]# cd /opt/smokeping

[root@linux-node1 src]# wget https://oss.oetiker.ch/smokeping/pub/smokeping-2.6.11.tar.gz

[root@linux-node1 src]# tar xf smokeping-2.6.11.tar.gz

[root@linux-node1 src]# cd smokeping-2.6.11

[root@linux-node1 smokeping-2.6.11]# ./configure --prefix=/usr/local/smokeping

# 如果是第一次安装smokeping程序，一般会出现报错，说没有perl模块支持，需要安装模块后在执行安装程序

** Aborting Configure ******************************

If you know where perl can find the missing modules, set

the PERL5LIB environment variable accordingly.

FIRST though, make sure that 'perl' starts the perl

binary you want to use for SmokePing.

Now you can install local copies of the missing modules

by running

./setup/build-perl-modules.sh /usr/local/smokeping/thirdparty

The RRDs perl module is part of RRDtool. Either use the rrdtool

package provided by your OS or install rrdtool from source.

If you install from source, the RRDs module is located

PREFIX/lib/perl

2.4. 安装依赖的Perl模块

[root@linux-node1 smokeping-2.6.11]# cat PERL_MODULES 
FCGI
CGI
CGI::Fast
Config::Grammar
Digest::HMAC_MD5
Net::Telnet
Net::OpenSSH
Net::SNMP
Net::LDAP
Net::DNS
IO::Pty
LWP
# 指定perl源安装perl模块
[root@linux-node1 smokeping-2.6.11]# vim ./setup/sdbs.inc
………………省略内容………………
function perlmodule (){
    path=$1;shift
    pack=${1:-""}
    [ -e $PREFIX/bin/cpanm ] || wget --no-check-certificate -O $PREFIX/bin/cpanm cpanmin.us && chmod 755 $PREFIX/bin/cpanm
    cpanm --mirror http://mirrors.aliyun.com/CPAN/ "$path$pack"
    [ "$KEEP" = "YES" ] || rm -rf $PREFIX/work $PREFIX/latest-build
}
[root@linux-node1 smokeping-2.6.11]# ./setup/build-perl-modules.sh /usr/local/smokeping/thirdparty
[root@linux-node1 smokeping-2.6.11]# ./configure --prefix=/usr/local/smokeping
………………省略内容………………
** Ready to install Smokeping ******************************
   Settings:
     PERL5LIB = not set
     PERL = /usr/bin/perl
   The Smokeping Makefiles use GNU make functionality.
   Continue installation with
     /usr/bin/gmake install
[root@linux-node1 smokeping-2.6.11]# /usr/bin/gmake install

[root@linux-node1 smokeping-2.6.11]# cat PERL_MODULES

FCGI

CGI

CGI::Fast

Config::Grammar

Digest::HMAC_MD5

Net::Telnet

Net::OpenSSH

Net::SNMP

Net::LDAP

Net::DNS

IO::Pty

LWP

# 指定perl源安装perl模块

[root@linux-node1 smokeping-2.6.11]# vim ./setup/sdbs.inc

………………省略内容………………

function perlmodule (){

path=$1;shift

pack=${1:-""}

[ -e $PREFIX/bin/cpanm ] || wget --no-check-certificate -O $PREFIX/bin/cpanm cpanmin.us && chmod 755 $PREFIX/bin/cpanm

cpanm --mirror http://mirrors.aliyun.com/CPAN/ "$path$pack"

[ "$KEEP" = "YES" ] || rm -rf $PREFIX/work $PREFIX/latest-build

}

[root@linux-node1 smokeping-2.6.11]# ./setup/build-perl-modules.sh /usr/local/smokeping/thirdparty

[root@linux-node1 smokeping-2.6.11]# ./configure --prefix=/usr/local/smokeping

………………省略内容………………

** Ready to install Smokeping ******************************

Settings:

PERL5LIB = not set

PERL = /usr/bin/perl

The Smokeping Makefiles use GNU make functionality.

Continue installation with

/usr/bin/gmake install

[root@linux-node1 smokeping-2.6.11]# /usr/bin/gmake install

报错处理：

报错1：
Can't locate Sys/Syslog.pm in @INC (@INC contains: /usr/local/smokeping/thirdparty/lib/perl5/x86_64-linux-thread-multi /usr/local/smokeping/thirdparty/lib/perl5 ../lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ../lib/Smokeping.pm line
解决：yum install -y perl-Sys-Syslog
报错2：smokeping_config.pod around line 81: alternative text 'the master/slave mode' contains non-escaped | or /
POD document had syntax errors at /bin/pod2man line 69.
https://github.com/oetiker/SmokePing/issues/23

报错1：

Can't locate Sys/Syslog.pm in @INC (@INC contains: /usr/local/smokeping/thirdparty/lib/perl5/x86_64-linux-thread-multi /usr/local/smokeping/thirdparty/lib/perl5 ../lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ../lib/Smokeping.pm line

解决：yum install -y perl-Sys-Syslog

报错2：smokeping_config.pod around line 81: alternative text 'the master/slave mode' contains non-escaped | or /

POD document had syntax errors at /bin/pod2man line 69.

https://github.com/oetiker/SmokePing/issues/23

3. 配置smokeping

3.1 创建相关目录和文件

cd /usr/local/smokeping
mkdir cache data var
touch /var/log/smokeping.log
chown apache:apache cache data var
chown apache:apache /var/log/smokeping.log
chmod 600 /usr/local/smokeping/etc/smokeping_secrets.dist

cd /usr/local/smokeping

mkdir cache data var

touch /var/log/smokeping.log

chown apache:apache cache data var

chown apache:apache /var/log/smokeping.log

chmod 600 /usr/local/smokeping/etc/smokeping_secrets.dist

3.2. 复制并修改配置文件

cd /usr/local/smokeping/htdocs
mv smokeping.fcgi.dist smokeping.fcgi

cd /usr/local/smokeping/etc
cp -rf /tmp/idc_ping_monitor/smokeping/config ./

cd /usr/local/smokeping/htdocs

mv smokeping.fcgi.dist smokeping.fcgi

cd /usr/local/smokeping/etc

cp -rf /tmp/idc_ping_monitor/smokeping/config ./

3.3 修改cgiurl，修改为本机的IP地址或者是域名

[root@linux-node1 etc]# sed -i 's#some.url#shooter.cainaio.com#' /usr/local/smokeping/etc/config
# 默认检测时间 300 秒修改 60 秒
[root@linux-node1 etc]#  sed -i 's#300#60#g' /usr/local/smokeping/etc/config
# 默认 60 秒 ping 20 次，修改为 60 秒 ping 60 次
[root@linux-node1 etc]# sed -i 's#pings    = 20#pings    = 60#' /usr/local/smokeping/etc/config

# 图像浏览界面的中文支持(安装字体)
[root@linux-node1 etc]# yum -y install wqy-zenhei-fonts.noarch

[root@linux-node1 etc]# vim /usr/local/smokeping/etc/config
charset = utf-8              //第50行下面添加此行内容(添加这行，解决出图乱码问题)

[root@linux-node1 etc]# sed -i 's#some.url#shooter.cainaio.com#' /usr/local/smokeping/etc/config

# 默认检测时间 300 秒修改 60 秒

[root@linux-node1 etc]# sed -i 's#300#60#g' /usr/local/smokeping/etc/config

# 默认 60 秒 ping 20 次，修改为 60 秒 ping 60 次

[root@linux-node1 etc]# sed -i 's#pings = 20#pings = 60#' /usr/local/smokeping/etc/config

# 图像浏览界面的中文支持(安装字体)

[root@linux-node1 etc]# yum -y install wqy-zenhei-fonts.noarch

[root@linux-node1 etc]# vim /usr/local/smokeping/etc/config

charset = utf-8 //第50行下面添加此行内容(添加这行，解决出图乱码问题)

# 编辑Graphs.pm
[root@linux-node1 etc]# vim /usr/local/smokeping/lib/Smokeping/Graphs.pm
'--font TITLE:20:"WenQuanYi Zen Hei Mono"',                  //第147行下面插入这一行内容

# 编辑Graphs.pm

[root@linux-node1 etc]# vim /usr/local/smokeping/lib/Smokeping/Graphs.pm

'--font TITLE:20:"WenQuanYi Zen Hei Mono"', //第147行下面插入这一行内容

#在第147行，下边插入这一行代码
'--font TITLE:20:"WenQuanYi Zen Hei Mono"',
if ($mode =~ /[anc]/){
        my $val = 0;
        for my $host (@hosts){
            my ($graphret,$xs,$ys) = RRDs::graph
            ("dummy",
            '--start', $tasks[0][1],
            '--end', $tasks[0][2],
            '--font TITLE:20:"WenQuanYi Zen Hei Mono"',#加在这里
            "DEF:maxping=$cfg->{General}{datadir}${host}.rrd:median:AVERAGE",
            'PRINT:maxping:MAX:%le' );
            my $ERROR = RRDs::error();
            return "<div>RRDtool did not understand your input: $ERROR.</div>" if $ERROR;
            $val = $graphret->[0] if $val < $graphret->[0];
        }
        $val = 1e-6 if $val =~ /nan/i;
        $max = { $tasks[0][1] => $val * 1.5 };
    }

#在第147行，下边插入这一行代码

'--font TITLE:20:"WenQuanYi Zen Hei Mono"',

if ($mode =~ /[anc]/){

my $val = 0;

for my $host (@hosts){

my ($graphret,$xs,$ys) = RRDs::graph

("dummy",

'--start', $tasks[0][1],

'--end', $tasks[0][2],

'--font TITLE:20:"WenQuanYi Zen Hei Mono"',#加在这里

"DEF:maxping=$cfg->{General}{datadir}${host}.rrd:median:AVERAGE",

'PRINT:maxping:MAX:%le' );

my $ERROR = RRDs::error();

return "<div>RRDtool did not understand your input: $ERROR.</div>" if $ERROR;

$val = $graphret->[0] if $val < $graphret->[0];

}

$val = 1e-6 if $val =~ /nan/i;

$max = { $tasks[0][1] => $val * 1.5 };

}

# 配置完成之后修改密码文件权限
[root@linux-node1 etc]# chmod 600 /usr/local/smokeping/etc/smokeping_secrets.dist

1 2	# 配置完成之后修改密码文件权限 [root@linux-node1 etc]# chmod 600 /usr/local/smokeping/etc/smokeping_secrets.dist

3.5. 修改apache的配置

[root@linux-node1 etc]# vim /etc/httpd/conf.d/smokeping.conf

ServerName localhost:80
<Directory />
   # AllowOverride none
   # Require all denied
   Options Indexes FollowSymLinks
    AllowOverride  None
    Order allow,deny
    Allow from all
</Directory>
Alias /cache "/usr/local/smokeping/cache/"
#Alias /cropper "/usr/local/smokeping/htdocs/cropper/"
Alias /css "/usr/local/smokeping/htdocs/css"
Alias /smokeping "/usr/local/smokeping/htdocs/smokeping.fcgi"
Alias /js "/usr/local/smokeping/htdocs/js/"
<Directory "/usr/local/smokeping">
        AllowOverride None
        Options All
        AddHandler cgi-script .fcgi .cgi
        Order allow,deny
        Allow from all
        
        AuthName "Smokeping"
        AuthType Basic
        AuthUserFile /usr/local/smokeping/htdocs/htpasswd
        Require valid-user
        DirectoryIndex smokeping.fcgi
</Directory>

[root@linux-node1 etc]# vim /etc/httpd/conf.d/smokeping.conf

ServerName localhost:80

# AllowOverride none

# Require all denied

Options Indexes FollowSymLinks

AllowOverride None

Order allow,deny

Allow from all

</Directory>

Alias /cache "/usr/local/smokeping/cache/"

#Alias /cropper "/usr/local/smokeping/htdocs/cropper/"

Alias /css "/usr/local/smokeping/htdocs/css"

Alias /smokeping "/usr/local/smokeping/htdocs/smokeping.fcgi"

Alias /js "/usr/local/smokeping/htdocs/js/"

AllowOverride None

Options All

AddHandler cgi-script .fcgi .cgi

Order allow,deny

Allow from all

AuthName "Smokeping"

AuthType Basic

AuthUserFile /usr/local/smokeping/htdocs/htpasswd

Require valid-user

DirectoryIndex smokeping.fcgi

</Directory>

3.6 配置全国各区域ip监测点

此处有大佬更新的监测点文件，可下载使用(也可以自行收集运营商ip解决)。

https://github.com/wilsonchai8/idc_ping_monitor

1	https://github.com/wilsonchai8/idc_ping_monitor

cd /usr/local/smokeping/etc
cp -rf /tmp/idc_ping_monitor/smokeping/location/* ./

1 2	cd /usr/local/smokeping/etc cp -rf /tmp/idc_ping_monitor/smokeping/location/* ./

监测点主要由国内的三大运营商的IP站点组成，也可以自定义需要检测的站点，更多的ip地址，可以参考 http://ip.yqie.com/china.aspx

3.7 修改fping位置

修改smokeping的config配置文件中fping路径 (如果和which 相同则不需要修改)

[root@linux-node1 etc]# which fping
/usr/sbin/fping
[root@linux-node1 etc]# vim /usr/local/smokeping/etc/config 
binary = /usr/sbin/fping

[root@linux-node1 etc]# which fping

/usr/sbin/fping

[root@linux-node1 etc]# vim /usr/local/smokeping/etc/config

binary = /usr/sbin/fping

3.8 上述apache配置文件AuthName部分可以先不配置，因为还没配置smokeping用户的密码，先不加，让apache启动成功，然后执行如下命令配置密码

htpasswd -c /usr/local/smokeping/htdocs/htpasswd smokeping

1	htpasswd -c /usr/local/smokeping/htdocs/htpasswd smokeping

编辑/usr/local/smokeping/etc/config

++ James

menu = 内网机房
title = 网络监控
alerts = someloss
slaves = boomer slave2
host = 192.168.0.177

++ James

menu = 内网机房

title = 网络监控

alerts = someloss

slaves = boomer slave2

host = 192.168.0.177

注意：apache访问没有权限问题

# 如果是没有修改过此项配置，那默认的配置如下：
<Directory />
    AllowOverride none
    Require all denied
</Directory>
# 将默认设置改成如下配置：
<Directory />
    Options Indexes FollowSymLinks
    AllowOverride  None
    Order allow,deny
    Allow from all
</Directory>

# 如果是没有修改过此项配置，那默认的配置如下：

AllowOverride none

Require all denied

</Directory>

# 将默认设置改成如下配置：

Options Indexes FollowSymLinks

AllowOverride None

Order allow,deny

Allow from all

</Directory>

4. 启动apache和smokeping

4.1、启动httpd和smokeping服务

systemctl start httpd.service     
   
/usr/local/smokeping/bin/smokeping
Note: logging to syslog as local0/info.
Daemonizing /usr/local/smokeping/bin/smokeping ...

#重启
/usr/local/smokeping/bin/smokeping --restart

systemctl start httpd.service

/usr/local/smokeping/bin/smokeping

Note: logging to syslog as local0/info.

Daemonizing /usr/local/smokeping/bin/smokeping ...

#重启

/usr/local/smokeping/bin/smokeping --restart

4.2、设置smokeping开机启动

echo "/usr/local/smokeping/bin/smokeping" >> /etc/rc.local
或者(推荐)
echo "/usr/local/smokeping/bin/smokeping --logfile=/var/log/smokeping.log 2>&;1 &;" >> /etc/rc.local

#设置smokeping环境变量
echo 'export PATH=/usr/local/smokeping/bin/:$PATH' >> /etc/profile

echo "/usr/local/smokeping/bin/smokeping" >> /etc/rc.local

或者(推荐)

echo "/usr/local/smokeping/bin/smokeping --logfile=/var/log/smokeping.log 2>&;1 &;" >> /etc/rc.local

#设置smokeping环境变量

echo 'export PATH=/usr/local/smokeping/bin/:$PATH' >> /etc/profile

5. 访问smokeping

重新启动apache和smokeping后
浏览器访问http://shooter.cainaio.com/smokeping
输入用户smokeping 你设置的密码登录

重新启动apache和smokeping后

浏览器访问http://shooter.cainaio.com/smokeping

输入用户smokeping 你设置的密码登录

http://shooter.cainaio.com/smokeping （首次访问，采集数据需要一段时间，等一会就会有数据图形展示）
测试的时候，可以一直ping www.baidu.com让产生网络流量。

1 2	http://shooter.cainaio.com/smokeping （首次访问，采集数据需要一段时间，等一会就会有数据图形展示）测试的时候，可以一直ping www.baidu.com让产生网络流量。

6. 把smokeping采集的数据通过rrdtool读取之后，按照一定的格式推送到prometheus的gateway,时间间隔是1分钟

6.1、复制数据推送相关脚本文件

cp /tmp/idc_ping_monitor/prometheus/collection_to_prometheus.py common.py ipgrab.py requirements.txt /usr/local/smokeping/

1	cp /tmp/idc_ping_monitor/prometheus/collection_to_prometheus.py common.py ipgrab.py requirements.txt /usr/local/smokeping/

运行python脚本可能需要安装的依赖：

yum install rrdtool rrdtool-devel
yum install postgresql-devel
pip3 install rrdtool
pip3 install common
pip3 install setuptools_rust
pip3 install LogHandler

yum install rrdtool rrdtool-devel

yum install postgresql-devel

pip3 install rrdtool

pip3 install common

pip3 install setuptools_rust

pip3 install LogHandler

6.2、配置crontab(每1分钟推一次)

crontab -e 

*/1 * * * * python /usr/local/smokeping/collection_to_prometheus.py

crontab -e

*/1 * * * * python /usr/local/smokeping/collection_to_prometheus.py

在/usr/local/smokeping/collection_to_prometheus.py中，按照实际情况配置3个参数

'LOG_FILE' : '/tmp/smoking_pushgateway.log'             # 日志文件  
'prometheus_gateway' : 'http://192.168.0.177:9091'     # Prometheus gateway地址  
'data_dir' : '/usr/local/smokeping/data'                # smokeping rrd文件的存放地址

'LOG_FILE' : '/tmp/smoking_pushgateway.log' # 日志文件

'prometheus_gateway' : 'http://192.168.0.177:9091' # Prometheus gateway地址

'data_dir' : '/usr/local/smokeping/data' # smokeping rrd文件的存放地址

此处用脚本将smokeping数据推送给pushgateway当中(此过程自行修改成自己的pushgateway地址即可)

7、通过grafana展示prometheus收集的数据

将模板grafana/ping_monitor.json 导入grafana即可

1	将模板grafana/ping_monitor.json 导入grafana即可

7.1、最终效果就是这样

参考文章:
https://blog.csdn.net/weixin_33743661/article/details/94336302
https://github.com/wilsonchai8/idc_ping_monitor

参考文章:

https://blog.csdn.net/weixin_33743661/article/details/94336302

https://github.com/wilsonchai8/idc_ping_monitor

本文固定链接: https://www.yoyoask.com/?p=8704
转载请注明: shooter 2022年06月24日于 SHOOTER 发表

最后编辑：2022-07-06

作者：shooter

这个作者貌似有点懒，什么都没有留下。

站内专栏

您可能还会对这些文章感兴趣！