在RedisTemplate中使用scan代替keys指令

1、在redistemplate中使用scan代替keys指令在redistemplate中用法scan代替keys命令keys * 這個(gè)指令千萬別在生產(chǎn)環(huán)境亂用。特殊是數(shù)據(jù)浩大的狀況下。由于keys會引發(fā)redis鎖，并且增強(qiáng)redis的cpu占用。無數(shù)公司的運(yùn)維都是禁止了這個(gè)指令的當(dāng)需要掃描key，匹配出自己需要的key時(shí)，可以用法 scan 指令scan操作的helper實(shí)現(xiàn)import java.io.ioexception;import java.nio.charset.standardcharsets;import java.util.arraylist;import java.uti

2、l.list;import java.util.function.consumer;import org.springframework.beans.factory.annotation.autowired;import org.springframework.data.redis.connection.redisconnection;import org.springframework.data.redis.core.cursor;import org.springframework.data.redis.core.scanoptions;import org.springframework

3、.data.redis.core.stringredistemplate;import org.springframework.stereotype.component;componentpublic class redishelper autowired private stringredistemplate stringredistemplate; /* * scan 實(shí)現(xiàn) * param pattern 表達(dá)式 * param consumer 對迭代到的key舉行操作 */ public void scan(string pattern, consumer consumer) this

4、.stringredistemplate.execute(redisconnection connection) -> try (cursor cursor = connection.scan(scanoptions.scanoptions().count(long.max_value).match(pattern).build() cursor.foreachremaining(consumer); return null; catch (ioexception e) e.printstacktrace(); throw new runtimeexception(e); ); /* *

5、 獵取符合條件的key * param pattern 表達(dá)式 * return */ public list keys(string pattern) list keys = new arraylist(); this.scan(pattern, item -> /符合條件的key string key = new string(item,standardcharsets.utf_8); keys.add(key); ); return keys; 但是會有一個(gè)問題：沒法移動cursor，也只能scan一次，并且簡單導(dǎo)致redis鏈接報(bào)錯(cuò)先了解下scan、hscan、sscan、zsc

6、ankeys 為啥擔(dān)心全？keys的操作會導(dǎo)致數(shù)據(jù)庫臨時(shí)被鎖住，其他的哀求都會被阻塞；業(yè)務(wù)量大的時(shí)候會出問題spring redistemplate實(shí)現(xiàn)scan1. hscan sscan zscan例子中的"field"是值redis的key，即從key為"field"中的hash中查找redistemplate的opsforhash，opsforset，opsforzset 可以 分離對應(yīng) sscan、hscan、zscan固然這個(gè)網(wǎng)上的例子其實(shí)也不對，由于沒有拿著cursor遍歷，只scan查了一次可以偷懶用法 .c

7、ount(integer.max_value)，一下子全查回歸；但是這樣子和 keys 有啥區(qū)分呢？搞笑臉 & 疑問臉可以用法 (jediscommands) connection.getnativeconnection()的 hscan、sscan、zscan 辦法實(shí)現(xiàn)cursor遍歷，參照下文2.2章節(jié)try cursor> cursor = redistemplate.opsforhash().scan("field", scanoptions.scanoptions().match("*").

8、count(1000).build(); while (cursor.hasnext() object key = cursor.next().getkey(); object valueset = cursor.next().getvalue(); /關(guān)閉cursor cursor.close(); catch (ioexception e) e.printstacktrace(); cursor.close(); 游標(biāo)一定要關(guān)閉，不然銜接會向來增長；可以用法client listsinfo clientsinfo stats指令查看客戶端銜接狀態(tài)，會發(fā)覺scan操作向來存在我們平常用法的r

9、edistemplate.execute 是會主動釋放銜接的，可以查看源碼確認(rèn)client list.id=1531156 addr=xxx:55845 fd=8 name= age=80 idle=11 flags=n db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=.springframework.data.redis.core.redistemplateexecute(org.springframework.data.redis.core.rediscallback

10、, boolean, boolean)finally redisconnectionutils.releaseconnection(conn, factory); 2. scan2.1 網(wǎng)上給的例子多半是這個(gè)這個(gè) connection.scan 沒法移動cursor，也只能scan一次public set scan(string matchkey) set keys = redistemplate.execute(rediscallback>) connection -> set keystmp = new hashset(); cursor cursor = connection

11、.scan(new scanoptions.scanoptionsbuilder().match("*" + matchkey + "*").count(1000).build(); while (cursor.hasnext() keystmp.add(new string(cursor.next(); return keystmp; ); return keys; 2.2 用法 multikeycommands獵取 connection.getnativeconnection；connection.getnativec

12、onnection()實(shí)際對象是jedis（debug可以看出） ，jedis實(shí)現(xiàn)了無數(shù)接口public class jedis extends binaryjedis implements jediscommands, multikeycommands, advancedjediscommands, scriptingcommands, basiccommands, clustercommands, sentinelcommands當(dāng) scan.getstringcursor() 存在 且不是 0 的時(shí)候，向來移動游標(biāo)獵取public set scan(string key) return

13、redistemplate.execute(rediscallback>) connection -> set keys = sets.newhashset(); jediscommands commands = (jediscommands) connection.getnativeconnection(); multikeycommands multikeycommands = (multikeycommands) commands; scanparams scanparams = new scanparams(); scanparams.match("*&a

14、mp;quot; + key + "*"); scanparams.count(1000); scanresult scan = multikeycommands.scan("0", scanparams); while (null != scan.getstringcursor() keys.addall(scan.getresult(); if (!stringutils.equals("0", scan.getstringcursor() scan = multikeycomman

15、ds.scan(scan.getstringcursor(), scanparams); continue; else break; return keys; ); 發(fā)散思量cursor沒有close，到底誰堵塞了，是 redis 么測試過程中，我基本只要發(fā)起十來個(gè)scan操作，沒有關(guān)閉cursor，接下來的哀求都卡住了redis側(cè)分析client listsinfo clientsinfo stats查看發(fā)覺 銜接數(shù) 惟獨(dú) 十幾個(gè)，也沒有堵塞和被否決的銜接config get maxclients查詢r(jià)edis允許的最大銜接數(shù) 是 100001) "maxclients&a

16、mp;quot;2) "10000"redis-cli在其他機(jī)器上也可以挺直登錄 操作綜上，redis本身沒有卡死應(yīng)用側(cè)分析netstat查看和redis的銜接，6333是redis端口；銜接向來存在? netstat -an | grep 6333netstat -an | grep 6333tcp4 0 0 xx.xx.xx.aa.52981 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52979 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.529

17、76 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52971 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52969 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52967 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52964 xx.xx.xx.bb.6333 establishedtcp4 0 0 xx.xx.xx.aa.52961 xx.xx.xx.bb.6333 estab

18、lishedjstack查看應(yīng)用的堆棧信息發(fā)覺無數(shù) waiting 的 線程，一致是在獵取redis銜接所以基本可以斷定是應(yīng)用的redis線程池滿了"http-nio-7007-exec-2" 139 daemon prio=5 os_prio=31 tid=0x00007fda36c1c000 nid=0xdd03 waiting on condition 0x00007000171ff000 java.lang.thread.state: waiting (parking) at sun.misc.unsafe.park(native method) -

19、 parking to wait for (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) at java.util.concurrent.locks.locksupport.park(locksupport.java:175) at java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(abstractqueuedsynchronizer.java:2039) at mons.pool2.impl.

20、linkedblockingdeque.takefirst(linkedblockingdeque.java:590) at mons.pool2.impl.genericobjectpool.borrowobject(genericobjectpool.java:441) at mons.pool2.impl.genericobjectpool.borrowobject(genericobjectpool.java:362) at redis.clients.util.pool.getresource(pool.java:49) at redis.clients.jedis.jedispoo

21、l.getresource(jedispool.java:226) at redis.clients.jedis.jedispool.getresource(jedispool.java:16) at org.springframework.data.redis.connection.jedis.jedisconnectionfactory.fetchjedisconnector(jedisconnectionfactory.java:276) at org.springframework.data.redis.connection.jedis.jedisconnectionfactory.g

22、etconnection(jedisconnectionfactory.java:469) at org.springframework.data.redis.core.redisconnectionutils.dogetconnection(redisconnectionutils.java:132) at org.springframework.data.redis.core.redistemplate.executewithstickyconnection(redistemplate.java:371) at org.springframework.data.redis.core.def

23、aulthashoperations.scan(defaulthashoperations.java:244) 綜上，是應(yīng)用側(cè)卡死后續(xù)過了一個(gè)中午，redis client lists顯示 scan 銜接還在，沒有釋放；應(yīng)用線程也還是處于卡死狀態(tài)檢查 config get timeout，redis未設(shè)置超時(shí)時(shí)光，可以用 config set timeout xxx設(shè)置，單位秒；但是設(shè)置了redis的超時(shí)，redis釋放了銜接，應(yīng)用還是一樣卡住1) "timeout"2) "0"netstat查看和redis的銜接，633

24、3是redis端口；銜接從established變成了close_wait；jstack和 本來表現(xiàn)一樣，卡在jedisconnectionfactory.getconnection? netstat -an | grep 6333netstat -an | grep 6333tcp4 0 0 xx.xx.xx.aa.52981 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52979 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52976 xx.xx.xx.bb.6333 close_wai

25、ttcp4 0 0 xx.xx.xx.aa.52971 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52969 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52967 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52964 xx.xx.xx.bb.6333 close_waittcp4 0 0 xx.xx.xx.aa.52961 xx.xx.xx.bb.6333 close_wait回顧一下tcp四次揮手established 表示銜接已被

26、建立close_wait 表示遠(yuǎn)程計(jì)算器關(guān)閉銜接，正在等待socket銜接的關(guān)閉和現(xiàn)象符合redis銜接池配置按照上面 netstat -an基本可以確定 redis 銜接池的大小是 8 ；結(jié)合代碼配置，沒有指定的話，默認(rèn)也的確是8redis.clients.jedis.jedispoolconfigprivate int maxtotal = 8;private int maxidle = 8;private int minidle = 0;如何配置更大的銜接池呢？a. 原配置beanpublic redisconnectionfactory redisconnectionfactory()

27、 redisstandaloneconfiguration redisstandaloneconfiguration = new redisstandaloneconfiguration(); redisstandaloneconfiguration.sethostname(redishost); redisstandaloneconfiguration.setport(redisport); redisstandaloneconfiguration.setpassword(redispassword.of(redispasswd); jedisconnectionfactory cf = n

28、ew jedisconnectionfactory(redisstandaloneconfiguration); cf.afterpropertiesset(); return cf; readtimeout，connecttimeout不指定，有默認(rèn)值 2000 msorg.springframework.data.redis.connection.jedis.jedisconnectionfactory.mutablejedisclientconfigurationprivate duration readtimeout = duration.ofmillis(protocol.defau

29、lt_timeout);private duration connecttimeout = duration.ofmillis(protocol.default_timeout);b. 修改后配置配置方式一：部分接口已經(jīng)deprecated了beanpublic redisconnectionfactory redisconnectionfactory() jedispoolconfig jedispoolconfig = new jedispoolconfig(); jedispoolconfig.setmaxtotal(16); / -最多可以建立16個(gè)銜接了 jedispoolconfi

30、g.setmaxwaitmillis(10000); / -10s獵取不到銜接池的銜接， / -挺直報(bào)錯(cuò)could not get a resource from the pool jedispoolconfig.setmaxidle(16); jedispoolconfig.setminidle(0); jedisconnectionfactory cf = new jedisconnectionfactory(jedispoolconfig); cf.sethostname(redishost); / - deprecated cf.setport(redisport); / - deprecated cf.setpassword(redispasswd); / - deprecated cf.settimeout(30000); / - deprecated 貌似沒生效，30s超時(shí)，沒有關(guān)閉銜接池的銜接； / -redis沒有設(shè)置超時(shí)