select * from information_schema.tables get Error 1105 (HY000): context deadline exceeded

[River2000i]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. import data from br
2. inject chaos to shutdown pd, tidb, tikv for 5 minutes in order, then delete chaos (not sure if it is necessary）
3. run query SELECT * FROM `information_schema`.`tables` WHERE `table_schema` = ? AND `table_name` = ? LIMIT 1;

2. What did you expect to see? (Required)

query run success

3. What did you see instead (Required)

<*mysql.MySQLError | 0xc000f1c000>: {
          Number: 1105,
          SQLState: [72, 89, 48, 48, 48],
          Message: "context deadline exceeded",
}
Error 1105 (HY000): context deadline exceeded

some related log

[2023/06/06 09:59:52.387 +00:00] [INFO] [conn.go:1181] ["command dispatched failed"] [conn=7442482058229717603] [connInfo="id:7442482058229717603, addr:172.30.14.77:53262 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select * from `information_schema` . `tables` where `table_schema` = ? and `table_name` = ? limit ? ;"] [txn_mode=PESSIMISTIC] [timestamp=0] [err="context deadline exceeded\ngithub.com/tikv/client-go/v2/internal/client.sendBatchRequest\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:789\ngithub.com/tikv/client-go/v2/internal/client.(*RPCClient).sendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client.go:497\ngithub.com/tikv/client-go/v2/internal/client.(*RPCClient).SendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client.go:540\ngithub.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest.func1\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_interceptor.go:39\ngithub.com/pingcap/tidb/util/topsql/stmtstats.(*KvExecCounter).RPCInterceptor.func1.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/topsql/stmtstats/kv_exec_count.go:57\ngithub.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_interceptor.go:40\ngithub.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_collapse.go:74\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1217\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1046\ngithub.com/tikv/client-go/v2/txnkv/txnsnapshot.(*ClientHelper).SendReqCtx\n\t/go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/txnsnapshot/client_helper.go:146\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1041\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:943\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:653\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1172\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/error/error.go:169\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1042\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:943\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:653\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1172"]
[2023/06/06 09:58:42.239 +00:00] [ERROR] [domain.go:763] ["reload schema in loop failed"] [error="context deadline exceeded"]
[2023/06/06 09:58:42.239 +00:00] [WARN] [backoff.go:158] ["tikvRPC backoffer.maxSleep 40000ms is exceeded, errors:\nsend tikv request error: context deadline exceeded, ctx: region ID: 19073, meta: id:19073 end_key:\"t\\200\\000\\000\\000\\000\\000\\000(_i\\200\\000\\000\\000\\000\\000\\000\\001\\003\\200\\000\\000\\000\\000\\000\\n\\245\\003\\200\\000\\000\\000\\000\\000\\000\\000\\003\\200\\000\\000\\000\\000\\000\\000\\005\\003\\200\\000\\000\\000\\000\\005\\264\\241\" region_epoch:<conf_ver:5 version:52 > peers:<id:19074 store_id:1 > peers:<id:19075 store_id:5 > peers:<id:19076 store_id:2 > , peer: id:19074 store_id:1 , addr: db-tikv-2.db-tikv-peer.tidb1373933076662347175.svc:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv, try next peer later at 2023-06-06T09:58:12.226515675Z\nsend tikv request error: context deadline exceeded, ctx: region ID: 19073, meta: id:19073 end_key:\"t\\200\\000\\000\\000\\000\\000\\000(_i\\200\\000\\000\\000\\000\\000\\000\\001\\003\\200\\000\\000\\000\\000\\000\\n\\245\\003\\200\\000\\000\\000\\000\\000\\000\\000\\003\\200\\000\\000\\000\\000\\000\\000\\005\\003\\200\\000\\000\\000\\000\\005\\264\\241\" region_epoch:<conf_ver:5 version:52 > peers:<id:19074 store_id:1 > peers:<id:19075 store_id:5 > peers:<id:19076 store_id:2 > , peer: id:19074 store_id:1 , addr: db-tikv-2.db-tikv-peer.tidb1373933076662347175.svc:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv, try next peer later at 2023-06-06T09:58:22.230749465Z\nsend tikv request error: context deadline exceeded, ctx: region ID: 19073, meta: id:19073 end_key:\"t\\200\\000\\000\\000\\000\\000\\000(_i\\200\\000\\000\\000\\000\\000\\000\\001\\003\\200\\000\\000\\000\\000\\000\\n\\245\\003\\200\\000\\000\\000\\000\\000\\000\\000\\003\\200\\000\\000\\000\\000\\000\\000\\005\\003\\200\\000\\000\\000\\000\\005\\264\\241\" region_epoch:<conf_ver:5 version:52 > peers:<id:19074 store_id:1 > peers:<id:19075 store_id:5 > peers:<id:19076 store_id:2 > , peer: id:19074 store_id:1 , addr: db-tikv-2.db-tikv-peer.tidb1373933076662347175.svc:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv, try next peer later at 2023-06-06T09:58:32.235082395Z\nlongest sleep type: tikvRPC, time: 41181ms"]

4. What is your TiDB version? (Required)

["Welcome to TiDB."] ["Release Version"=v6.5.3] [Edition=Enterprise] ["Git Commit Hash"=e954f278eba827068e8ec0cfbdcb5299bd0986e0] ["Git Branch"=heads/refs/tags/v6.5.3] ["UTC Build Time"="2023-06-05 10:26:58"] [GoVersion=go1.19.9] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]

[lhy1024]

I also meet similiar problem with different error code.
I inject io-hang to pd-leader between 19:05 and 19:15, which will resign pd-leader and etcd-leader.
Sometimes tidb will report error [2023-06-11 19:11:07] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV.

PTAL @tangenta @crazycs520

[tiancaiamao]

This should have been fixed by #50959
Solved in v8.5