Evan：/* Blackbox */

2024-10-06T02:29:46Z

Blackbox

新页面

==

=book=

[https://www.bookstack.cn/read/prometheus-book/README.md Prometheus操作指南]
=进阶=
以后全改成自己 docker-compose build

[https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)有常用监控警告]

[https://blog.51cto.com/wzlinux/2490670 PromQL 内置函数(三)]

[https://www.jb51.net/article/189679.htm prometheus python 库编写自定义指标的方法(完整代码)]

[https://www.jianshu.com/p/19442595e886 promethues + python + flask监控后端服务状态]

[https://blog.csdn.net/weixin_33754913/article/details/91803530 prometheus python client]

[https://blog.csdn.net/JackLiu16/article/details/80381201?utm_medium=distribute.pc_relevant.none-task-blog-title-7&spm=1001.2101.3001.4242 使用Python和Flask编写Prometheus监控]

[https://blog.csdn.net/specter11235/article/details/87927202 python开发prometheus exporter]

==服务发现==
=== 基于文件的服务发现 Use file-based service discovery to discover scrape targets===
==== 安装 node-exporter by ansible====
https://github.com/evan886/my-ansible/tree/main/prome-node-add

==== 配置 ====
<pre>
#这里以json格式为例子其实另外一种也可以用YAML格式的
cd apps/prom/prometheus
sudo mkdir targets

cat targets/nodes.json
[{
"targets": [
"172.16.0.20:9100",
"172.16.0.29:9100"
],
"labels": {
"server": "node_export_live"
}
}]

vi prometheus.yml

# - cadvisor:8080

- job_name: 'cadvisor-nodejs'
static_configs:
- targets: ['121.41.101.200:8080']

- job_name: 'nodes_server_discoverry'
file_sd_configs:
- files:
- targets/*.json
refresh_interval: 1m

sudo curl -X POST http://172.16.0.42:9090/-/reload

解析如下

scrape_configs:
- job_name: 'prometheus'
static_configs: # 这个配置表示静态发现
- targets: ['localhost:9090']

- job_name: 'node' # 增加一个节点，取名为 node
metrics_path: "/metrics" # 获取数据的路径 http://10.0.20.12:9100/metrics
file_sd_configs: # 这个配置表示通过文件发现
- files:
- targets/nodes/*.json # 读取目录下的所有json结尾的文件
refresh_interval: 1m # 刷新频率，1 分钟

打开
https://prom.com/targets

有 job_name: 'nodes_server_discoverry' 就是成功
</pre>
[https://prometheus.io/docs/guides/file-sd/ Use file-based service discovery to discover scrape targets]

=ins on docker-composer=
git clone https://github.com/evan886/docker-compose-monitor.git

https://gitee.com/atompi/PrometheusStackDockerCompose
注意 grafana update 升级为gra 8.2.7 就行了不行有漏洞

docker-compose up -d grafana

=安全=
<pre>
node exporter 添加防火墙

curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo
# yum install node_exporter
</pre>

=prometheus=

Prometheus/prəˈmiː.θi.əs/

[[Nginx用户密码认证配置 Basic HTTP authentication]]

不错的书
https://yunlzheng.gitbook.io/prometheus-book/

[https://blog.csdn.net/lijiaocn/article/details/81865120 新型监控告警工具prometheus（普罗米修斯）的入门使用（附视频讲解）]

[https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/08/03/prometheus-usage.html#prometheus%E7%9A%84%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6 【视频】新型监控告警工具prometheus（普罗米修斯）入门使用（附视频讲解）]

[https://zhuanlan.zhihu.com/p/91215371 为什么说 Prometheus 是足以取代 Zabbix 的监控神器？]

[https://zhuanlan.zhihu.com/p/24811652 基于Prometheus的分布式在线服务监控实践]

[https://juejin.im/post/6844903866064977934 Prometheus学习系列（十三）之配置解析 ]

==usage==
[https://www.jianshu.com/p/03cf0fc9c746 Grafana单图表告警邮件&面板图（Dashboard）日报邮件]

[https://prometheus.io/docs/practices/rules/ Recording rules ]

https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml

=== reload ===
<pre>
检查配置文件
bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml

重新加载配置文件
curl -X POST httP://172.31.248.19:9090/-/reload
</pre>

=grafana=
[https://www.oschina.net/news/213575/grafana-9-2-released Grafana 9.2 发布]
<pre>
首次用户和密码为 admin admin

#防火墙 grafana ng 在 10.3.10.10
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
firewall-cmd --reload

</pre>

[https://grafana.com/docs/grafana/latest/installation/docker/ grafana.com Run Grafana Docker image]

[https://grafana.com/grafana/dashboards/8919 中文 Node Exporte for Prometheus Dashboard]

usage creat-->import --> id 8919

Container

https://grafana.com/grafana/dashboards/11174

docker-grafana/config.env https://github.com/stefanwalther/docker-grafana/blob/master/config.env

==grafana 添加源和数据 ==

[https://www.bookstack.cn/read/prometheus-book/quickstart-use-grafana-create-dashboard.md 使用Grafana创建可视化Dashboard]

===Panel plugin not found: grafana-piechart-panel===
<pre>

升级为7.0.0就好了不过要看其它模板还能用么

#这个放弃了是在 6.x系列的还不好用
grafana-cli plugins install grafana-piechart-panel

docker-compose restart grafana

执行过程
grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.1
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
into: /var/lib/grafana/plugins

✔ Installed grafana-piechart-panel successfully

Restart grafana after installing plugins . <service grafana-server restart>

[https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]

</pre>
[https://www.jianshu.com/p/153d944ab836 Panel plugin not found: grafana-piechart-panel]

==grafana 添加用户 ==
<pre>
0. 自己的话在左下角就可以改了

1.Users--> 2.Invite -->

3.进行邀请
Email or Username:输入用户邮箱，建议先跟用户名保持相同，然后在后面在写邮箱，后面会说到，这样不会报错
Name：输入用户名
Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin)
Send invite email：勾选时，会将新用户的设置密码的链接发到邮箱

点击创建之后回调到新用户设置密码

-->

4.创建好之后，点击Pending Invites
复制链接到浏览器的地址栏中访问（也可从邮件中收到链接）
进去之后是下面的界面，这里有个坑，复制的IP地址是localhost，我们这里改为自己搭建的IP地址，然后在Email填写邮箱地址，设置新用户的密码
设置完成后，会直接以新用户身份登陆

http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

-->

</pre>

[https://blog.csdn.net/liuxiangyang_/article/details/86737103 grafana的用户及权限的配置]

[https://blog.csdn.net/GX_1_11_real/article/details/85119451 grafana的用户及权限的配置]

[https://blog.csdn.net/kozazyh/article/details/93759717?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase grafana-利用Google OAuth2 身份验证]

== ng config==
<pre>
cat gf.conf
server
{
listen 80;
server_name gf.linuxsa.org;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;

client_max_body_size 10m;

location / {
proxy_pass http://10.3.10.124:3000;
proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;

proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;

real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}

</pre>

==使用教程==

[https://mojotv.cn/2019/08/20/dash-graph-of-prometheus 使用Prometheus和Grafana做可视化大屏展示]

[https://www.cnblogs.com/cherylgi/p/13927231.html grafana +Prometheus API 使用 ]

[https://yuerblog.cc/2019/01/04/grafana-usage/ grafana配置与简单使用]

[https://blog.csdn.net/bbwangj/article/details/81109615 Grafana使用教程]

[https://blog.csdn.net/Sagitarioo/article/details/102805852?depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2&utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2 grafana数据可视化-使用教程]

[https://www.cnblogs.com/xkus/p/7462742.html grafana快速入门 ]

[https://blog.csdn.net/u012062455/article/details/79214700 grafana使用教程之基本概念（basic concepts）]

[https://blog.csdn.net/weixin_42674978/article/details/102681969 Grafana从无到有的一个相当粗略的资料总结]

[https://blog.csdn.net/Sagitarioo/article/details/102805852?depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2&utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2 grafana数据可视化-使用教程]

[https://ken.io/note/grafana-quickstart-influxdb-datasource-graph Grafana快速入门：InfluxDB数据源以及曲线图表仪表盘配置]

[https://www.cnblogs.com/fatyao/p/11192234.html Grafana报警]

apached conf
<pre>
##/etc/httpd/conf.d/gf.conf

<VirtualHost *:80>
ServerName gf.linuxsa.org
ServerAlias gf.linuxsa.org

ProxyRequests off

<Proxy *>
Order deny,allow
Allow from all
</Proxy>

<Location />
ProxyPass http://localhost:3000/
ProxyPassReverse http://localhost:3000/
</Location>
</VirtualHost>

</pre>

=vm监控[[Node-exporter]]=
可以如上的 node
[[Node-exporter]]

=Blackbox=

[[Blackbox exporter]]

=cadvisor=
<pre>
cadvisor docker container fails to start "mountpoint for cpu not found"

用 Try replacing image by gcr.io/cadvisor/cadvisor
registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024

https://github.com/google/cadvisor/issues/1943
</pre>

=pushgateway=

[[pushgateway]]

[https://www.cnblogs.com/xiao987334176/p/9933963.html 基于Prometheus的Pushgateway实战 ]

https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html

=[[Process-exporter]]进程监控=

=应用监控=
==redis==
[[prometheus监控Redis集群]]

[https://blog.csdn.net/weixin_43832846/article/details/90024274 pika主备部署（redis-sentinel）+监控（prometheus+grafana）]

[https://blog.csdn.net/weixin_34289744/article/details/92725774?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-16.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-16.nonecase Prometheus监控 Redis , redis-cli_exporter]

==监控MQ==
[https://www.rabbitmq.com/prometheus.html Monitoring with Prometheus & Grafana ]
[https://blog.csdn.net/weixin_38937840/article/details/103144734 prometheus监控RabbitMQ]

[https://chenzhonzhou.github.io/2020/12/03/prometheus-jian-kong-rabbitmq/ Prometheus 监控 Rabbitmq]

[https://grafana.com/blog/2021/06/01/monitor-and-alert-on-essential-rabbitmq-cluster-metrics-with-the-new-grafana-cloud-integration/?src=ggl-s&mdm=cpc&camp=nb-rabbitmq-bmm&cnt=135709386153&trm=rabbitmq%20monitor&device=c&gclid=CjwKCAjwoduRBhA4EiwACL5RPyO-9bp3COrXnxBNs5VYyBYZAxt7p9jUPqueQ8m0Lz_cO1rkVYJ8cRoCs6AQAvD_BwE Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration]

==监控ES==
https://github.com/slok/ecs-exporter
==监控docker==
像普通export 一样 docker运行后 ,

与 prometheus 集成

- job_name: docker
static_configs:
- targets: ['192.168.111.65:8088']

[https://www.cnblogs.com/user-sunli/p/14738679.html Prometheus监控docker容器还带报警呢]

[https://www.cnblogs.com/Dev0ps/p/10546276.html cAdvisor+Prometheus+Grafana监控docker ]

[https://blog.csdn.net/Sebastien23/article/details/113645177#t7 Prometheus+Grafana监控Docker容器和Linux主机]

[https://cloud.tencent.com/developer/article/1584309 docker监控: cAdvisor]

[https://blog.csdn.net/palet/article/details/82889493 Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读]

[https://www.cnblogs.com/edisonchou/p/docker_monitor_introduction_part2.html 你必须知道的容器监控 (2) cAdvisor ]

[https://www.jianshu.com/p/efc7c66638cc prometheus 监控Docker]

[https://www.cnblogs.com/xiangsikai/p/11289518.html Prometheus 监控Docker服务器及Granfanna可视化 ]

[https://www.jianshu.com/p/41831207bbf6 prometheus监控docker]

== mysql==
[https://blog.csdn.net/qq_25934401/article/details/82594478 prometheus 监控之 mysql 篇（含mysql报警规则）]

[https://jiajunhuang.com/articles/2018_12_16-prometheus_mysqld_exporter.md.html Prometheus MySQL Exporter源码阅读与分析]

[https://www.jianshu.com/p/27b979554ef8 python + promethues 监控mysql]

==Spring boot==
https://www.bookstack.cn/read/prometheus_practice/application-spring.md

[https://blog.csdn.net/liangcha007/article/details/86699013#7.%E5%85%B3%E4%BA%8Epushgateway Prometheus基于java开发exporter样例以及pushgateway的应用]

=ng conf=
<pre>
cat alert.conf
server
{
listen 80;
server_name alert.com;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;

client_max_body_size 10m;

location / {
# auth_basic "Prometheus";
# auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9093;
proxy_http_version 1.1;

#s#et_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;

real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}

cat pro.conf
server
{
listen 80;
server_name pro.linuxsa.org;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;

client_max_body_size 10m;

location / {
auth_basic "Prometheus";
auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9090;
proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;

proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;

real_ip_recursive on;
# real_ip_header X-Forwarded-For;
# set_real_ip_from 0.0.0.0/0;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

#proxy_set_header Host $host:$server_port;
#proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
}
}

cat blackbox.conf
server
{
listen 80;
server_name blackbox.com;
# access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
#error_log /var/log/nginx/test-nux-bcb.evan.error.log;
client_max_body_size 10m;
location / {
# auth_basic "Prometheus";
# auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
proxy_pass http://10.3.10.124:9115;
proxy_http_version 1.1;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Remote-Port $remote_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;

proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
real_ip_recursive on;
}
}

</pre>

=[[Alertmanager]]=

=教程=
[https://www.cnblogs.com/caoweixiong/p/12736520.html Prometheus + Grafana（八）系统监控之Kafka ]

[https://www.cnblogs.com/caoweixiong/p/12736573.html Prometheus + Grafana（九）系统监控之RabbitMQ ]

[https://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_liunx_60_prometheus_config.html prometheus学习系列五： Prometheus配置文件 ]

=see also=

[https://www.cnblogs.com/hong-fithing/p/14695803.html Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台（一） ]

[https://www.jianshu.com/p/93412a925da2 规划 Prometheus 的存储用量]

[https://blog.csdn.net/lemon89/article/details/104088184 深入理解监控系统——Prometheus核心特点]

[https://blog.csdn.net/qq_21816375/article/details/80462445?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase prometheus部署node,server以及域名加告警]

[https://zhuanlan.zhihu.com/p/56081874 通过gitlab-ci自动添加prometheus业务监控]

[http://www.linuxea.com/1881.html linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二) ]

[https://www.cnblogs.com/chenqionghe/p/10494868.html 从零搭建Prometheus监控报警系统]

[https://www.jianshu.com/p/1f05476ebcee 使用prometheus自定义监控]

[https://zhuanlan.zhihu.com/p/91215371 为什么说 Prometheus 是足以取代 Zabbix 的监控神器？]

[https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)]

[https://my.oschina.net/54188zz/blog/3030618 基于prometheus + grafana + mysql + Telegram 监控告警]

[https://netsecurity.51cto.com/art/201912/608600.htm 为什么我们放弃Zabbix采用Prometheus]

[https://www.ctolib.com/topics-135208.html 容器监控利器 - prometheus在生产落地过程中的思考]

[https://blog.csdn.net/heian_99/article/details/103952955 Centos7安装Promethus(普罗米修斯）监控系统完整版]

[http://soa.ctocio.com.cn/soa/2020/0821/33627.html Prometheus 的局限这 30 个坑点不要踩]
[[category:ops]] [[category:container]] [[category:prom]]

Monitoring with prometheus - 版本历史

Evan：​/* Blackbox */

Evan：/* Blackbox */