Monitoring with prometheus
==
book
进阶
以后全改成自己 docker-compose build
prometheus python 库编写自定义指标的方法(完整代码)
promethues + python + flask监控后端服务状态
服务发现
基于文件的服务发现 Use file-based service discovery to discover scrape targets
安装 node-exporter by ansible
https://github.com/evan886/my-ansible/tree/main/prome-node-add
配置
#这里以json格式为例子其实 另外一种 也可以用YAML格式的
cd apps/prom/prometheus
sudo mkdir targets
cat targets/nodes.json
[{
"targets": [
"172.16.0.20:9100",
"172.16.0.29:9100"
],
"labels": {
"server": "node_export_live"
}
}]
vi prometheus.yml
# - cadvisor:8080
- job_name: 'cadvisor-nodejs'
static_configs:
- targets: ['121.41.101.200:8080']
- job_name: 'nodes_server_discoverry'
file_sd_configs:
- files:
- targets/*.json
refresh_interval: 1m
sudo curl -X POST http://172.16.0.42:9090/-/reload
解析如下
scrape_configs:
- job_name: 'prometheus'
static_configs: # 这个配置表示 静态发现
- targets: ['localhost:9090']
- job_name: 'node' # 增加一个节点，取名为 node
metrics_path: "/metrics" # 获取数据的路径 http://10.0.20.12:9100/metrics
file_sd_configs: # 这个配置表示通过文件发现
- files:
- targets/nodes/*.json # 读取目录下的所有json结尾的文件
refresh_interval: 1m # 刷新频率，1 分钟
打开
https://prom.com/targets
有 job_name: 'nodes_server_discoverry' 就是成功
Use file-based service discovery to discover scrape targets
ins on docker-composer
git clone https://github.com/evan886/docker-compose-monitor.git
https://gitee.com/atompi/PrometheusStackDockerCompose
注意 grafana update 升级为gra 8.2.7 就行了 不行有 漏洞
docker-compose up -d grafana
安全
node exporter 添加防火墙 curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo # yum install node_exporter
prometheus
Prometheus/prəˈmiː.θi.əs/
Nginx用户密码认证配置 Basic HTTP authentication
不错的书 https://yunlzheng.gitbook.io/prometheus-book/
新型监控告警工具prometheus（普罗米修斯）的入门使用（附视频讲解）
【视频】新型监控告警工具prometheus（普罗米修斯）入门使用（附视频讲解）
为什么说 Prometheus 是足以取代 Zabbix 的监控神器？
usage
Grafana单图表告警邮件&面板图（Dashboard）日报邮件
https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml
reload
检查配置文件 bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml 重新加载配置文件 curl -X POST httP://172.31.248.19:9090/-/reload
grafana
首次用户和密码为 admin admin #防火墙 grafana ng 在 10.3.10.10 firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept " firewall-cmd --reload
grafana.com Run Grafana Docker image
中文 Node Exporte for Prometheus Dashboard
usage creat-->import --> id 8919
Container
https://grafana.com/grafana/dashboards/11174
docker-grafana/config.env https://github.com/stefanwalther/docker-grafana/blob/master/config.env
grafana 添加源和数据
Panel plugin not found: grafana-piechart-panel
升级为7.0.0就好了 不过要看其它模板还能用么 #这个放弃了 是在 6.x系列的 还不好用 grafana-cli plugins install grafana-piechart-panel docker-compose restart grafana 执行过程 grafana-cli plugins install grafana-piechart-panel installing grafana-piechart-panel @ 1.6.1 from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download into: /var/lib/grafana/plugins ✔ Installed grafana-piechart-panel successfully Restart grafana after installing plugins . <service grafana-server restart> [https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]
Panel plugin not found: grafana-piechart-panel
grafana 添加用户
0. 自己的话 在左下角就可以改了 1.Users--> 2.Invite --> 3.进行邀请 Email or Username:输入用户邮箱，建议先跟用户名保持相同，然后在后面在写邮箱，后面会说到，这样不会报错 Name：输入用户名 Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin) Send invite email：勾选时，会将新用户的设置密码的链接发到邮箱 点击创建之后回调到新用户设置密码 --> 4.创建好之后，点击Pending Invites 复制链接到浏览器的地址栏中访问（也可从邮件中收到链接） 进去之后是下面的界面，这里有个坑，复制的IP地址是localhost，我们这里改为自己搭建的IP地址，然后在Email填写邮箱地址，设置新用户的密码 设置完成后，会直接以新用户身份登陆 http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa -->
ng config
cat gf.conf
server
{
listen 80;
server_name gf.linuxsa.org;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;
client_max_body_size 10m;
location / {
proxy_pass http://10.3.10.124:3000;
proxy_http_version 1.1;
#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}
使用教程
grafana使用教程之基本概念（basic concepts）
Grafana快速入门：InfluxDB数据源以及曲线图表仪表盘配置
apached conf
##/etc/httpd/conf.d/gf.conf
<VirtualHost *:80>
ServerName gf.linuxsa.org
ServerAlias gf.linuxsa.org
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
<Location />
ProxyPass http://localhost:3000/
ProxyPassReverse http://localhost:3000/
</Location>
</VirtualHost>
vm监控Node-exporter
可以如上的 node Node-exporter
Blackbox
cadvisor
cadvisor docker container fails to start "mountpoint for cpu not found" 用 Try replacing image by gcr.io/cadvisor/cadvisor registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024 https://github.com/google/cadvisor/issues/1943
pushgateway
https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html
Process-exporter进程监控
应用监控
redis
pika主备部署（redis-sentinel）+监控（prometheus+grafana）
Prometheus监控 Redis , redis-cli_exporter
监控MQ
Monitoring with Prometheus & Grafana prometheus监控RabbitMQ
Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration
监控ES
https://github.com/slok/ecs-exporter
监控docker
像普通export 一样 docker运行后 ,
与 prometheus 集成
- job_name: docker static_configs: - targets: ['192.168.111.65:8088']
cAdvisor+Prometheus+Grafana监控docker
Prometheus+Grafana监控Docker容器和Linux主机
Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读
Prometheus 监控Docker服务器及Granfanna可视化
mysql
prometheus 监控之 mysql 篇（含mysql报警规则）
Prometheus MySQL Exporter源码阅读与分析
Spring boot
https://www.bookstack.cn/read/prometheus_practice/application-spring.md
Prometheus基于java开发exporter样例以及pushgateway的应用
ng conf
cat alert.conf
server
{
listen 80;
server_name alert.com;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;
client_max_body_size 10m;
location / {
# auth_basic "Prometheus";
# auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9093;
proxy_http_version 1.1;
#s#et_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}
cat pro.conf
server
{
listen 80;
server_name pro.linuxsa.org;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;
client_max_body_size 10m;
location / {
auth_basic "Prometheus";
auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9090;
proxy_http_version 1.1;
#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}
cat blackbox.conf
server
{
listen 80;
server_name blackbox.com;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;
client_max_body_size 10m;
location / {
# auth_basic "Prometheus";
# auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9115;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
real_ip_recursive on;
}
}
Alertmanager
教程
Prometheus + Grafana（八）系统监控之Kafka
Prometheus + Grafana（九）系统监控之RabbitMQ
prometheus学习系列五： Prometheus配置文件
see also
Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台（一）
prometheus部署node,server以及域名加告警
linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二)
为什么说 Prometheus 是足以取代 Zabbix 的监控神器？
基于prometheus + grafana + mysql + Telegram 监控告警
容器监控利器 - prometheus在生产落地过程中的思考