Evan：/* inof */

2020-06-19T12:01:36Z

inof

新页面

[[Redis批量删除key]]

=inof=
<pre>

172.16.200.12:7002> cluster nodes
686cfc039ab7c2c9b55bebff0b97eb9bb4060f78 172.16.20.7:7005@17005 master - 0 1591844242096 9 connected 5461-10922
d73e3338dfebb963967135bd73fe10bd459f9e65 172.16.20.7:7004@17004 master - 0 1591844245101 5 connected 10923-16383

518ca649f51c01498b3ab42b9e62c40f921f6327 172.16.20.12:7003@17003 master - 0 1591844242000 8 connected 0-5460
4c90517a70df13b37752c9199f9ba6da15682ff4 172.16.20.12:7002@17002 myself,slave 686cfc039ab7c2c9b55bebff0b97eb9bb4060f78 0 1591844244000 3 connected

242902c61a434edf2efc15909e7fa33e930351c9 172.16.20.13:7001@17001 slave d73e3338dfebb963967135bd73fe10bd459f9e65 0 1591844243097 5 connected
909a54d51117d2e8bb587656db37c8def525e472 172.16.20.13:7000@17000 slave 518ca649f51c01498b3ab42b9e62c40f921f6327 0 1591844244098 8 connected
172.16.200.12:702>

此机为 172.16.20.12 703 master , 7002 是slave 他的 master 是哪个原来是 mq3 172.16.20.7:7005

redis_cluster]# du -sh data/*
2.7G data/appendonly-7002.aof
2.4G data/appendonly-7003.aof
1.5G data/dump_7002.rdb
144K data/dump_7003.rdb
</pre>

=pre=
有以下几种办法可以知道某个Redis实例是否存在大key：

在redis实例上执行bgsave，然后我们对dump出来的rdb文件进行分析，找到其中的大KEY
有个不太推荐的命令，debug object xxx 可以看到这个key在内存中序列化后的大小，当然我们可以通过SCAN+debug object xxx 得到当前实例所有key的大小。
redis-cli 原生自带 –bigkeys 功能，可以找到某个实例 5种数据类型(String、hash、list、set、zset)的最大key。
最终 redis-cli-new

= redis-cli --bigkeys=
<pre>
debug object key
redis-cli 原生自带 –bigkeys 功能，可以找到某个实例 5种数据类型(String、hash、list、set、zset)的最大key。

edis的--bigkeys参数：对redis整个keyspace进行统计（数据量大时采样，调用scan命令），寻找每种数据类型较大的keys，给出数据统计
redis-cli --bigkeys -i 0.1 -h 127.0.0.1

第一步

第一次执行完会高CPU

172.16.20.12 -p 7003

redis-cli -c -h 172.16.20.12 -p 7003 -a pass --bigkeys
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far 'sns_cache:userFollowLog:1296748078836308812708058410934272' with 1 bytes
[00.00%] Biggest string found so far 'uv:post:12987950080525078' with 59 bytes
[00.00%] Biggest hash found so far 'posts:comment_user:12991077241267048:85412991077241267048' with 3 fields
[00.00%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12966682424339580' with 534 bytes
[00.00%] Biggest hash found so far 'userFollowLog:12926532386375374' with 155 fields
[00.01%] Biggest string found so far 'uv:post:12995339861518175' with 933 bytes
[00.01%] Biggest hash found so far 'postsPraiseLog:12947079027599160' with 12431 fields
[00.01%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12873404201224716' with 1575 bytes
[00.02%] Biggest set found so far 'posts_comment_reply:user_post:12948355526503870' with 421 members
[00.03%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12942125413091883' with 2136 bytes
[00.04%] Biggest hash found so far 'postsPraiseLog:12923869855985512' with 25959 fields
[00.12%] Biggest hash found so far 'postsPraiseLog:12783531839545344' with 74653 fields
[00.13%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12952979705119705' with 2605 bytes
[00.16%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12978438783438619' with 2881 bytes
[00.18%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:13011992277674709' with 3017 bytes
[00.24%] Biggest set found so far 'posts_comment_reply:user_post:12784388711768064' with 731 members
[00.24%] Biggest list found so far 'robot:operation_list:like:oTIpr4SkvLsZ' with 100 items
[00.49%] Biggest string found so far 'sns_cache:five_latest_replies_v5:comment_id:12923545866027011' with 3812 bytes
[00.61%] Biggest string found so far 'uv:post:12984703254167595' with 12304 bytes
[02.06%] Biggest list found so far 'robot:operation_list:follow:ogSLwcZEpvRz' with 102 items
[02.99%] Biggest list found so far 'robot:operation_list:follow:L0rtLUbHQEFC' with 209 items
[04.07%] Biggest set found so far 'posts_comment_reply:user_post:12796982571974656' with 826 members
[04.21%] Biggest hash found so far 'postsPraiseLog:12907301092746796' with 85186 fields
[07.38%] Biggest hash found so far 'postsPraiseLog:12935131931853999' with 103381 fields
[07.97%] Biggest hash found so far 'postsPraiseLog:12817607428722673' with 107824 fields
[08.18%] Sampled 1000000 keys so far
[10.78%] Biggest set found so far 'sns_cache:5dfad08b5f9b1413912353:standard_ref' with 1241 members
[16.37%] Sampled 2000000 keys so far
[24.55%] Sampled 3000000 keys so far
[25.46%] Biggest list found so far 'robot:operation_list:like:n9gok0IcHqgb' with 298 items
[27.36%] Biggest hash found so far 'postsPraiseLog:12786292153868288' with 131584 fields
[27.71%] Biggest string found so far 'area_code:list' with 15464 bytes
[30.35%] Biggest string found so far 'groupMatchRankCache' with 16320 bytes
[32.26%] Biggest hash found so far 'postsPraiseLog:12896706894004677' with 140077 fields
[32.73%] Sampled 4000000 keys so far
[36.49%] Biggest hash found so far 'postsPraiseLog:12786516420624384' with 193502 fields
[39.84%] Biggest string found so far 'stimulateUserStatisticsListKey' with 24063 bytes
[40.92%] Sampled 5000000 keys so far
[43.44%] Biggest list found so far 'robot:operation_list:follow:AsNltfXCtZPR' with 300 items
[49.10%] Sampled 6000000 keys so far
[49.54%] Biggest set found so far 'sns_cache:5dfad630a5251220444421:standard_ref' with 1255 members
[57.29%] Sampled 7000000 keys so far
[65.47%] Sampled 8000000 keys so far
[73.07%] Biggest zset found so far 'groupAuction:socket' with 12 members
[73.65%] Sampled 9000000 keys so far
[81.84%] Sampled 10000000 keys so far
[85.05%] Biggest list found so far 'buyback:buybackShareQueue:20200611' with 600 items
[90.02%] Sampled 11000000 keys so far
[98.20%] Sampled 12000000 keys so far

-------- summary -------

Sampled 12040461 keys in the keyspace!
Total key length in bytes is 743432861 (avg len 61.74)

Biggest list found 'buyback:buybackShareQueue:20200611' has 600 items
Biggest hash found 'postsPraiseLog:12784108337438720' has 219951 fields
Biggest string found 'stimulateUserStatisticsListKey' has 24063 bytes
Biggest set found 'sns_cache:5dfad630a5251220444421:standard_ref' has 1255 members
Biggest zset found 'groupAuction:socket' has 12 members

134 lists with 12778 items (00.00% of keys, avg size 95.36)
137285 hashs with 49491661 fields (01.14% of keys, avg size 360.50)
11899007 strings with 45722370 bytes (98.83% of keys, avg size 3.84)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
4034 sets with 494122 members (00.03% of keys, avg size 122.49)
1 zsets with 12 members (00.00% of keys, avg size 12.00)

输出大概分为两部分，summary之上的部分，只是显示了扫描的过程。summary部分给出了每种数据结构中最大的Key。

第二步用 debug object

debug object postsPraiseLog:12784108337438720
Value at:0x7fbeb046e110 refcount:1 encoding:hashtable serializedlength:4395140 lru:14786916 lru_seconds_idle:25

172.16.20.12:7003> debug object stimulateUserStatisticsListKey
Value at:0x7fbcde2332b0 refcount:1 encoding:raw serializedlength:10675 lru:14786772 lru_seconds_idle:319
172.16.20.12:7003> strlen stimulateUserStatisticsListKey
(integer) 24063

</pre>

=第三方tool=
==redis-cli-new (成功) ==
<pre>

#例子为最大前3个
redis-cli-new -p 7000 --bigkeys --bigkey-numb 3

通过Redis-cli –bigkeys 我们可以很方便的找到某个实例最大的几个KEY，但是只能得到某种类型的最大的一个key，于是思考改改redis-cli findBigKeys 功能，增加查找多个key的代码，用户可以指定大key的数量。

修改后功能预览如下：

VITOXIE-MB1:src xiean$ ./redis-cli-new -p 2837 --bigkeys --bigkey-numb 3

Biggest string Key Top 1 found 'xxxG_NEWMATCH_VOD_DATA_7f7a2a2fb5f780a13fecd9f1e51bdf8a' has 53170 bytes
Biggest string Key Top 2 found 'xxxG_NEWMATCH_VOD_DATA_a9758560d1874493c637dec0753909da' has 53159 bytes
Biggest string Key Top 3 found 'xxxG_NEWMATCH_VOD_DATA_d0971977b0ce028141e53b020b93d822' has 53156 bytes
Biggest list Key Top 1 found 'UserPostInfo122_632789064' has 11028 items
Biggest list Key Top 2 found 'xxxG_FriendCallBack_PushList_23' has 1973 items
Biggest list Key Top 3 found 'xxxG_FriendCallBack_PushList_20' has 1824 items
</pre>
ps，修改的源码放在GitHub上，这里还部分dba日常实用工具：https://github.com/xiepaup/OPS-Tools ，

https://github.com/xiepaup/dbatools
== godis-cli-bigkey ==
<pre>
go run godis-cli-bigkey.go

$GOROOT
$GOPATH

vi /etc/profile
export PATH=/data/apps/go/bin/:$PATH
export GOPATH=/root/go/ #(可选设置) 如果你用root运行
export GOROOT=/data/apps/go/
#export GOARCH=amd64
#export GOOS=linux
export GOTOOLS=$GOROOT/pkg/tool
#export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

#export PATH=$PATH:/usr/local/gobin

source /etc/profile

ERR

# go run godis-cli-bigkey.go
godis-cli-bigkey.go:8:2: cannot find package "github.com/erpeng/godis-cli-bigkey/pool" in any of:
/data/apps/go/src/github.com/erpeng/godis-cli-bigkey/pool (from $GOROOT)
/root/go/src/github.com/erpeng/godis-cli-bigkey/pool (from $GOPATH)
godis-cli-bigkey.go:9:2: cannot find package "github.com/erpeng/godis-cli-bigkey/rdb" in any of:
/data/apps/go/src/github.com/erpeng/godis-cli-bigkey/rdb (from $GOROOT)
/root/go/src/github.com/erpeng/godis-cli-bigkey/rdb (from $GOPATH)

go mod init godis-cli-bigkey#就好了

Get "https://proxy.golang.org/github.com/erpeng/godis-cli-bigkey/pool/@v/list": dial tcp 34.64.4.113:443: i/o timeout
godis-cli-bigkey.go:9:2: module github.com/erpeng/godis-cli-bigkey/rdb: Get "https://proxy.golang.org/github.com/erpeng/godis-cli-bigkey/rdb/@v/list": dial tcp 34.64.4.113:443: i/o timeout
[root@ godis-cli-bigkey]#
[root@ godis-cli-bigkey]# go env -w GO111MODULE=on
[root@ godis-cli-bigkey]# go env -w GOPROXY=https://mirrors.aliyun.com/goproxy/,direct

Usage

mv rdb.rdb rdb.rdbbak
mv dump_7002.rdb rdb.rdb

结果

key:k1,valueSize:9,valueType:0,expireTime:1549533396795,lfu:0,lru:0
key:key,valueSize:9,valueType:0,expireTime:0,lfu:0,lru:0
key:ss1,valueSize:14,valueType:2,expireTime:0,lfu:0,lru:0
key:si1,valueSize:23,valueType:11,expireTime:0,lfu:0,lru:0
key:l1,valueSize:28,valueType:14,expireTime:1549537004535,lfu:0,lru:0
key:h1,valueSize:33,valueType:13,expireTime:0,lfu:0,lru:0
key:z1,valueSize:67,valueType:12,expireTime:0,lfu:0,lru:0
key:testzset,valueSize:1303,valueType:5,expireTime:0,lfu:0,lru:0
key:h3,valueSize:8845,valueType:13,expireTime:0,lfu:0,lru:0
key:h2,valueSize:11680,valueType:4,expireTime:0,lfu:0,lru:0
key:h4,valueSize:11703,valueType:4,expireTime:0,lfu:0,lru:0

go run godis-cli-bigkey.go -topn 40
Rdb Version:0009
key:k1,valueSize:9,valueType:0,expireTime:1549533396795,lfu:0,lru:0
key:key,valueSize:9,valueType:0,expireTime:0,lfu:0,lru:0
key:ss1,valueSize:14,valueType:2,expireTime:0,lfu:0,lru:0
key:si1,valueSize:23,valueType:11,expireTime:0,lfu:0,lru:0
key:l1,valueSize:28,valueType:14,expireTime:1549537004535,lfu:0,lru:0
key:h1,valueSize:33,valueType:13,expireTime:0,lfu:0,lru:0
key:z1,valueSize:67,valueType:12,expireTime:0,lfu:0,lru:0
key:testzset,valueSize:1303,valueType:5,expireTime:0,lfu:0,lru:0
key:h3,valueSize:8845,valueType:13,expireTime:0,lfu:0,lru:0
key:h2,valueSize:11680,valueType:4,expireTime:0,lfu:0,lru:0
key:h4,valueSize:11703,valueType:4,expireTime:0,lfu:0,lru:0

</pre>
https://github.com/erpeng/godis-cli-bigkey

==redis_rdb_tools ==
<pre>

mkdir ~/.pip/
vi ~/.pip/pip.conf

[global]
index-url = http://pypi.douban.com/simple
[install]
trusted-host=pypi.douban.com

pip install rdbtools

pip install python-lzf

生成内存报告

rdb -c memory rdb.rdb --bytes 128 -f dump_memory_mq2_7003.csv

按键值大小排序

awk -F',' '{print $4,$2,$3,$1}' dump_memory.csv | sort > dump_memory_csv.sort

awk -F',' '{print $4,$2,$3,$1}' dump_memory_mq2_7003.csv | sort > dump_memory_mq2_7003_csv.sort

database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry
</pre>
[https://www.cnblogs.com/xingxia/p/redis_rdb_tools.html Redis工具之redis_rdb_tools ]

[https://www.cnblogs.com/cheyunhua/p/10598181.html 使用rdbtools工具来解析redis rdb文件 ]

https://blog.csdn.net/u010522235/article/details/89241765

https://www.jianshu.com/p/c885af575f97

https://www.cnblogs.com/yqzc/p/12425533.html

== rdb_bigkeys工具==
<pre>
git clone https://github.com/weiyanwei412/rdb_bigkeys.git
cd rdb_bigkeys

go mod init rdb_bigkeys
go get
go build

行完成生成可执行文件rdb_bigkeys。
使用方法： ./rdb_bigkeys --bytes 1024 --file bigkeys.csv --sep 0 --sorted --threads 4 /home/redis/dump.rdb
/home/redis/dump.rdb修改为实际的文件路径
</pre>
=see also=

https://redis.io/topics/rediscli

[https://www.jianshu.com/p/c885af575f97 分析redis key大小的几种方法]

[https://segmentfault.com/a/1190000018169117 【Redis源码分析】如何在Redis中查找大key ]

[https://www.cnblogs.com/os-linux/p/11928930.html Redis中如何发现并优化big key？ ]

[https://www.jianshu.com/p/c885af575f97 分析redis key大小的几种方法]

[https://blog.csdn.net/huxianbo0807/article/details/102912172?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase Redis Value过大问题键值过大]

[https://blog.csdn.net/Androilly/article/details/101624196?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase 解决Redis大key问题，看这一篇文章就够了]

[[category:redis]] [[category:ops]]

Redis-bigkeys - 版本历史

Evan：​/* inof */

Evan：/* inof */