Finalized block got stuck

[Tore19]

Code: https://github.com/stafiprotocol/stafi-node/tree/v0.6.2

StaFi chain has been running stably for a long time using version v2.0.0 of substrate, and the current version is v0.6.2. But the finalized block suddenly got stuck. Like this:

Can check details here:
https://telemetry.polkadot.io/#list/0x290a4149f09ea0e402c74c1c7e96ae4239588577fe78932f94f5404c68243d80.

We didn't find any error or useful logs for this. And we also tried to use the grandpa.noteStalled function to try to restore, actually we even tried multiple times, but all didn't work.

So any ideas about this? And any possible solution?

[Tore19]

More information:

Code: https://github.com/stafiprotocol/stafi-node/tree/v0.6.2

Subscan: https://stafi.subscan.io/

Apps: https://apps.stafi.io/

Telemetry: https://telemetry.polkadot.io/#list/0x290a4149f09ea0e402c74c1c7e96ae4239588577fe78932f94f5404c68243d80

[andresilva]

https://substrate.stackexchange.com/a/380/452

Check this answer, in particular using the RPC API to figure out the GRANDPA round state (e.g. who is not voting).

[Tore19]

https://substrate.stackexchange.com/a/380/452

Check this answer, in particular using the RPC API to figure out the GRANDPA round state (e.g. who is not voting).

Yeah, we tried the solution too(https://stafi.subscan.io/extrinsic/0xe4431deafd133d9cdfcd952a610b58cad0a53ce026d33c83e7c90b91a6efbb2a). Just like I said, even tried multiple times using higher height, but all didn't work.

About the GRANDPA round state,

curl https://stafi-rpc.stafi.io \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"grandpa_roundState","params":[],"id":1,"jsonrpc":"2.0"}'

The result is:
{"jsonrpc":"2.0","result":{"background":[],"best":{"precommits":{"currentWeight":0,"missing":["5C6aCJLKMX26VrgWUyCGBWu8mv6nxuXwMm6Fc9znDXBiyS4r","5C6iPsqiYwdFaKHydnYjVz7e95oY3tWCCD999H5e4NopCEiq","5C77whr2H4GEqrv7QFXCXnSa3FAWEUGSTTdYKCbaNN5KRKXC","5CEgrkayB6z7V547jv3VdzP4Kts7299xB3Yu3koJ4ztCPKb8","5CGptCyfLjvC42My8cPKfRadSyGcZNMtQUsS4KmmLiMevAYJ","5CH9PxoqR1CdL5gXnhEqiHhx5ykre4ZaEY58NtCKPjCRYecR","5CHDkm33a6wawMagg7M4CGpeYXtYWgbsWRPzrs4iFGTU4S6j","5CKjcnHcCspYD974BDJ8JWXcL3Xt1DxuPbhQkE1RvcnuUTBK","5CLJTem1DvDwmbRiWwX43KXHyYY5SWZ2f2KpJwFiX2zy9u9W","5CMDtbm4XM9W617XnRUC5YvDHTpUYHHqXuSwU6omfGEgfxAY","5CMf5pWBYBnaag7qVHhfXAeJKsx28Dufo6Z1ekMd8fDqCzpY","5CQbFzpuv45gFRE4QKtEmPGskk7FCeU6trdSyTJFukYUZQcA","5CU7UvWgaqbhbH2FxgPh8bn4YXjmkpsiSKLzXh425fKeCeMQ","5CYFfX66GcHfdTRMw1XgS226GJvdvaq8SmsnqFQerRPKJZr5","5CYUkvh2jmQkgm4AT6YvwLDaE7CJK3zmkg7ZtUvavmGsFSQF","5CaBuc5t3hUHWfeR5hzLSn7DKhyZkEDZPZamkTwAyGewkMVp","5CbTsqZ4mivxxC6mP4Lok99FcLxTi2aKNAMUFEEiR8d966CU","5Cbbzng1xa9gaKuGv5uvRyHDcEUfJyyGpPhfYf6szAykUzGP","5CdHYrqsTYmWDoQTE378Y43deqgBccTPRR6Da7tttESSP3GP","5CeCBEcBEVAijVq8UfDwb1pE7ecu5hBZbw2EL6BgGxgco4pb","5CfLQDTeEF6T3XaMnUbqtDavp6CZ52Hmk5Hz2Fk6mpoHrH5g","5CfaUE5tfk8U9wJeD7MEKBsQHyuF9QVSetNiBzDhZYpPcPh5","5CgWrhMkTMnoocdN14832aS1MSbkzX5x8BWeHheeESjUNkXQ","5CgypggyXgs6DAF7GS3qrNFE23QG1WkqJNf8VWKgNvVDxHep","5CigHitVJtDu3GAPw3d6xGGf3kTnhDwHQcmYMcbXWxzb1qau","5CkUvBBYtLKCxetHZ9V8MTnKSjoHGuBQaaTzCSz7Y5eWGwWD","5CoRZZLMnwD2GXUHkAyFeQrHVkUHDPWQCmySpnsf2b86Ybpp","5CrTcCRMBQ5RPwGJ7ew5C89PB9MpPg6q1jHUGMDqDRS71ZFs","5Cu7KaNzkhKMjuz3KdGuRXyCxUSbt3oLRFWFKvXZLQPTSQph","5CwezFdooNCR5jcFi1q6ruLXBYynZczw1KLnkYvjoncYwe4X","5D3MSZ89shhZtr8SFbqf6w54cRSQPRvQY4SbEMq2H25yA6tg","5D43eVvZeQgKWKPUDEBt2PYYDVWtdFPrNT9cuomQvNrAkpcr","5D5hiWsdwL1227pK7PNvkHpnw3hRQLKXQEqy1sXvL3HFyDEy","5D5mFWCfZjcZ6qq558in79fz1r6v6K54TwhUn8FTukzrymuQ","5D6GqWTm2Ry97aU6AtHXZLwbZNVCB49kK4kpLfxAGmn5QCDz","5D77y3kk2F4WeVGPFGNKHj9KqFDkL3Be78sBN1jLwQ2MNRJm","5DCAzhXx1xH6dcbcLCmpNv9HU4kwriEzSKJmLunGJDT4rcZj","5DFyPTcYq2UeQyfD7wW1qxWghjDbfkg1jCZDDNXpqz9Dqmdn","5DHG85ALQLBZEv5VcJS4PHf3JuURoVYSZ8Vky43s2ez9kNnc","5DKV3RdcGuC8YszRNhdF1h6NsfpYSUpxLjMUTGuLKFoWXntq","5DLDwu8BA1nF3243Htp8BprXrbwY47WFBP5uDMWKKjTMdBjb","5DM9HDFujkCJyHCno68C8qeLX43KbafRdLBzyi1hwzrYdfn1","5DNrhBQiYfCK4gKSPKQhym5W3JWnuWDYkLVhCNWW2PdeTZYJ","5DQDJamT5h7vTjKbhQvEzhHnp1Qm6r3ThxbtdTrK82sT5kQ4","5DU4ykjP5YtuPrCsMmZd7FRkMSVJzv8GoQ6oRsREhUPtf2wx","5DUeSNipPoh4JV7uVpN6AjxYNa8jPyMYA5KutQuPe5F26Bsm","5DYYLvzZ7KyNGmaiSCsxTgnhoAdYD916bBktJQ6bLRYpon29","5DZ16AhB2C56oNGhK4SsVUbMQ4w4fXGFR9i94BjryS6gGewH","5DZXPZG7jENTGEietsJv3GNoZjhJkUXVW7wcVkfADfxhuEdE","5DcJ48Tw5WsYULGjU6AGFWRq7T7f2ShNPEbp2Snp9VAd3Wj5","5DeBfxVzRXZPchtU1JwuKFwb4ybzh7PaABTWDPuoNhynmUVi","5DfwGhCQUDTEyZQBXJMm9gVF6K8KZNYRcTLJtCNoLaTn3X7o","5DgwgFj1NXC9dm3HZArna6rW9T3aAKZC5ZfvZM5kwtBirDN9","5DhHepwhWomV6ysktT12KJ818F3X6StwmBNwMYdPZecf2cmx","5DjqAz7EyjHttZDP972zP4RT3Fmn4hsexg5LoD7qhpsdGi65","5DqLoPQNxSBqhKxStFmsX9LHyq6TqS1Qbhg9mKBk1ZZM4iX7","5DrVCAcuRrbt6JFXGijxXupvBbcH9YQPcS7FCk6zZRtPxDts","5DsMNJNQas581gjpkSx5ZZiDSavKwhnksogwrU1Haz26DQ5f","5DxRcRUuGqrGxG1uMG45FKVSVsc39UWUfSqzWiK79wvF8Sb1","5DxxJYA3J6nUjShY4ZWZYsm4b7KbKFLiPpqLmNT5rmq1ozSK","5E5ZUXUwvS5zsV86vgYRfsdD77qta3uKnTvYMohg6ZkakC1u","5E8RRWQp1EUbhoWYwbKc36dCAvWchCtsMTPxkQffj7awBQkz","5E9Sru6GEbiDjq4CiZuf2r77JViHEdVbjbdGaBhwGfXfFw9e","5ECjdwFTgwZ59973WFa2Epui6vmm6mj867H2oX6nG6MPC5gL","5EEdXcAxmDWLEZxzQZSZYBGxNoj1RAy8hAVw9FRqqBpSpmaZ","5EHWX4FAVTEzzCxFqcvhj4sdtcqkMdVhgNggk1Sf9ZXnUQYA","5ELTnqE1MMxnYCq54hLwQ4vUSXWLyUqDpHxBLpkkCfs7WyeD","5ENNUCoTrovy5QC7kMA3Tc7mRMVaBVpBYuwtxiHZ71v2jGZa","5ESnrMXJnp4Hjkv7gmbvUmXYbA1Udfher13fAiNWgaTKwtp9","5EWNkZYTrA65pfjwbSScMUk7cWM6WKTbUsdrr2Rz4sBF8XWL","5EZ4ERq36gwnLLJzNvtYVmLLU7XEcGUBJcfDe5jypVPd1i1i","5EbLYciVfxXHoh5QZadWPH9nmAEA6hAnW2dArKMTx4fjYoZH","5Eci1K8B1m6eEWaiqsr4xvb6cg6tajW9GTcnz6GrpYqUYfBK","5EkdkAeBynRZNn7AiE3f6zKnZjco5tuyVN1NFAxmLoePX5DP","5EkguJE7pbwi88YHjDh8GkDbgtFJf65qnGyzHJTSFfxT3JPt","5Eo8nBRNuLaerm5TrFDWrbBGLGu65WY8iKsjdYLTrTHkz2zr","5EoXzBrWfTRAePhGeznVKjvY4rWAzyfk33871Bb3cWQgHVc5","5EroE5BJV7n93Z4D9dmpaARa817iHQJ1UyauoYmJFZppKDUB","5Erq9uUZt5jyhBVDZ4d3pHYperB6KXo8pH2rDnnQPN4VwPSu","5EtZ1a2Fq51MzTgU8fJuY3yADytSfRq2N9svjVv1xTMpnkY1","5EvtGF6W1Xws4ouGdPofgJNFCT6nHR3o3h9oUCkjyUEBsDTk","5F3uDi2ghwFEXEoPMNGiwZmPz6C7DpYdYBejSPjeTwnvrUAK","5F4HbX6C7NRkmuiDZULMoehGTkywNKwMsx9qbCv36QnTEUSb","5F914LRb584dKbz3rakrcB3Uhnk9DyZYfxPL8J8meUX3sLeH","5FA8sjGw8keLJnc1e1fCPgs653QaX7iTPfUPst9PFnSeGxbt","5FBrPeS4tdK4GyaqVXDN2FKMP5KzYPdf9z844gAkDJtPvevH","5FGQCahYL2jBTPsAw7x9trSWGkDETTNiz51sr25QMW2hgh7A","5FJr162Efkw1VQXZHzKpNptrpCnr4fhAyn724pvpyAmGKXb6","5FMfXdTNyDRuL2C8dhTWBbZfYXf3TBjNXpA2tvadggGRtAhh","5FPmy3YjGdWZm3roYNUrFVAeZa1GjbubWs4SccxW4xk1BHV6","5FPsTKk3uhfrSEtJ3YHo7NnTRoktcUybd1AceKhMniLvjSGn","5FQ93DD4nAfR6LmPK4oHE8Sg5MhHSjt31kJTzhg7L3hzfmEd","5FSCwHAdwuJyn1bJ6QA7QaF2YERdMHZLCkkC5p7iXqHpkapp","5Fa6DVLQfvHe5K5ZKZNsCBnNF7aQaWLP7YNLLc6VJBocZBWC","5FbqdfDvdEMU1c24hsyBGjdNt4FT9A6QkiSVpfz5mGm4qHVj","5FcY1F9o9kx5ZwcvYdHuRQ1cgVHA47A435N5SgpeThLgYfkY","5FcjA9UriUwjKxiNHKN4ksPFqaATRPW9vAUxytfK1V34ektM","5FdeKnHxCFDmhAMHwWZKUv4hMseXTxNVDS613YMdRsZcV9ca","5FeaGCh2iwy7pU2nUHFQ8yLGYdujURCU9kyadhW8x95TekA3","5FgCjHUF45dEqw5wAAyuciCUFEs4y53WHQysfSK1Kq2EEFrx","5FiqJD5egss5dKquDJXbGobyzThiQqQcPTSAJwavy1R26c5P","5FjbApWu1dMdYTdRCxoS8ni3woS2UbhftgYXWWR2edtqQ1Dt","5Fm4DozqYZaUkYUekj9sDCYpuG3TXtVqci4jrU79yC3acTyq","5FmBzzPq1cLKirE7tCv3JoM3UVy3a62rJFish8tFLnipVJa9","5FmLtqymKPK2dDBYfq2qe7iomCZLq9eyyaSGmBmsYaZv9pv1","5FoiAwAFd6EybrzRMZffGPqLr1ccWsZA92ZNKvp4q5ZQcCeW","5FoidQ5KHELQCfj7YsHRBLXW7YMDRiirkLYhTybGQmHhYrev","5FqQxvjui7uuAcVq7RcJFVBL1sphH44YTRN7Mdn8aqBCv6dD","5Fr5PH4198YgsudLPgfGvmGcYVYx6DwPfNXDncPMCWqW4kAi","5FtVb6D6zuLQMEt2S83VsjNra1BFRoYgrGsm6yqRHDBeK447","5Fu5J9E1TuNcwu8LDk5sdZihx9subju329ChDMXyKAPCPjL4","5FxepAXJtyxNxkJvVvwnpssQHiBQcunRoN7BSqmFa2hQ8Qbv","5FzyvzWqL73NM1QviAyrCAHKx4V8mEzbTSr7CnvyzhUyAy8e","5G4DN9gaKh48LraEQNNhKKGBiWHq3hXr9haPaHpRLEcsQXpk","5G8yJv4XemYeNFc253HCy4kZbDR8DPSoqrMM9kP9GWnn226s","5GBVSftmUYbpHQdxDMxH4aVJs1s4Z3g3bHTYhP9X4kMnyg21","5GC9BJfGXihQ2nUSarfFZ4zdM6KiTeZZLLScUVHHr4voAPVm","5GCJTafeJRyAMp7cAEjQwVTFNmBVh9vUX8Jq8MAyXCwwkJLm","5GDKAHHBprcmVgPqDJQzy2sPaShVS96No5JqTwiPcRDWnNUz","5GGE5mdUM2dncReKfLLbLizqYPSokNz4Tj4Z2n8Kr7DLPrqE","5GGscd7C9EJdb9KJ2Cv9VfwQsERSPiNCdzybWsvJ8WnLB9jW","5GJTkvpQgU8RJwBAiAFnYP4inov6n6Zir2QTPNuwn1C1V4MH","5GMvuYcVXDZ7s17aCu2eLSc9vS2YaGoLoHB5Di4DTAPYnpx2","5GNpMeddxEyxj5bqqJHemEuz11LiAzb6PvAiY976X4WvNMfW","5GP7prKCRVuSVkq48VRU98dMP9urPFqF4zq5TAG6rbQUNwv7","5GSE35fHbBY8zdWNjG32auZ7sEBmCwyg3ejaCRSC6WdgdZxB","5GVsAxku9Ps7XoXFaVgEKdRHtgPzKVeuVsWUuDJ4z8ko3Bwt","5GYAJ8wjQiLsiofoHWdGFjJMnPvuayKLXUfxA745XakPKQNA","5GaMMwVnvDD3GnPLw1TVxzQG79m5U6EJGfiQYJd7CEG9ATXh","5GbtyseKmhUmQGa9k3z2UVpPx15A3yE2cAr6T7SKLDyBgS5Y","5Gk5JCbDs3JJtJEoNPQPj2rXBCE73UaYso2EfHZJ8uZKxcbz","5GkvH2KtWZGmkjzcTyHUr5GQcwhUJBKdXb8CCYV4i6am36xq","5GqcxpuwsoGqfMdkcmrS6Q13qDUcwkajeFGYTpQ5eUk3gjkp","5GrBDB6zjmQUSQgyGALr7CVUC44MFaArzo7TxxiqWbo8iL2E","5GunHjFHvFtBtKyiJszJkRBjQJ1JcZDupChbyGj5i5WxrwLU","5GwmAdQjEEr4iK7QT5FwTo9FrQvYSRiXmdCshkwYTCkGAin3","5H6PBYLnyj2XCLyJSxxtQ8GeSs6GHixw3qApLHovdV2gsNcc","5H72kw7M8KBQtmrpLh1Sx5rFSzKbn6ZbCMmfnKFjYBD7Umsd","5H8wCzaF4zSb2i1yj9Vdsbm4S612JBpq83ZWZXnwUukzxogQ","5H9WEykwSHmnCWUgCUwnqa9EorVt317DXHH49LCFkg8pvYEk","5HBoJbp4hTrGHhuxxwyYHaYMzeVnocbUcmE9joL2UNb4jLKU","5HBvshYQ1bHBEjTSNk7LhyM9Nb4wEiqMYVjH1pAutBrJecon","5HDgkH427uXtA8q1RcU6dNDsYdPFjYMhMEhrMb2R15dbLW8f","5HErtXo2LRVovBgMD6YF5CPUKgK6ZXRBKx8vj1kw2advTraH","5HGm8NwjzYmnHPSea5duXTzycvhWYwqjANGk4pmNZhv9EQ4h","5HGrncYcXzwr9FJet1S2xn4WgbSrKj3WZV3rfnyBqKsufC2f","5HLwyatqk2SGGUX4NVVq5zuT6woCFdrm4DsiAL98YJLSbsbv","5HSdfeX1uYLi6NrmTifJwZajS2zE2ZTd7jakHLPgzaqaZF2V","5HSjaKhotDaemQXuQhrfPHmKgHvkgF4W8C9uZh9yD6wVZUMw","5HTnsJvULhTyyKHtb1ZCHqij6S5S1Va1mVG3JGED8FDRY3HC","5HW5gNSyv2xRXdpsQKXDvM6AJfNYxLi1V3vVacEk3tzFEMHz","5HZ6XfWFX6STZFcaV67kST5GXkywG2k6FV8PfTiqCGEioQ1c","5HZBqryUGJuNJUG7e5oY6M1p4zp9HtEP9cAVMmnynJewUqD7","5HZn9cXLhLJXBGsdZJPt2YT2HbpQ1TFWP3uRYHHjnzWRo1eu","5Hcgj3Qa7LmMpwcGSCGg8N77uvkro7fZvbnj9hqi9poqMCDR","5HjivUtH2soVtHUpeRFaN9YB7cP3iWT2Rr1Nvaaf3j9yET22","5Hm8JHGGSt91tKnkELfGqZpW2guZoWsDnbuG7cDJJ3my8Nu2","5HoLuWtGTmsV1e9EfKEuuf82x4cf7nyYJLBz7fvrxhVh5nqg","5Hp8Ckt5kjxhGkaRJapBXQ7fduR1U4XeJkfB51jezfL5DsnS","5HpasZqGuvQwL1UFLx9YABK8nhwnerkBsQ12p3VBHKVNRNsH"]},"prevotes":{"currentWeight":101,"missing":["5CHDkm33a6wawMagg7M4CGpeYXtYWgbsWRPzrs4iFGTU4S6j","5CKjcnHcCspYD974BDJ8JWXcL3Xt1DxuPbhQkE1RvcnuUTBK","5CLJTem1DvDwmbRiWwX43KXHyYY5SWZ2f2KpJwFiX2zy9u9W","5CMDtbm4XM9W617XnRUC5YvDHTpUYHHqXuSwU6omfGEgfxAY","5CMf5pWBYBnaag7qVHhfXAeJKsx28Dufo6Z1ekMd8fDqCzpY","5CQbFzpuv45gFRE4QKtEmPGskk7FCeU6trdSyTJFukYUZQcA","5CU7UvWgaqbhbH2FxgPh8bn4YXjmkpsiSKLzXh425fKeCeMQ","5CYFfX66GcHfdTRMw1XgS226GJvdvaq8SmsnqFQerRPKJZr5","5CYUkvh2jmQkgm4AT6YvwLDaE7CJK3zmkg7ZtUvavmGsFSQF","5CaBuc5t3hUHWfeR5hzLSn7DKhyZkEDZPZamkTwAyGewkMVp","5CdHYrqsTYmWDoQTE378Y43deqgBccTPRR6Da7tttESSP3GP","5CeCBEcBEVAijVq8UfDwb1pE7ecu5hBZbw2EL6BgGxgco4pb","5D77y3kk2F4WeVGPFGNKHj9KqFDkL3Be78sBN1jLwQ2MNRJm","5DFyPTcYq2UeQyfD7wW1qxWghjDbfkg1jCZDDNXpqz9Dqmdn","5DHG85ALQLBZEv5VcJS4PHf3JuURoVYSZ8Vky43s2ez9kNnc","5DLDwu8BA1nF3243Htp8BprXrbwY47WFBP5uDMWKKjTMdBjb","5DM9HDFujkCJyHCno68C8qeLX43KbafRdLBzyi1hwzrYdfn1","5DU4ykjP5YtuPrCsMmZd7FRkMSVJzv8GoQ6oRsREhUPtf2wx","5DUeSNipPoh4JV7uVpN6AjxYNa8jPyMYA5KutQuPe5F26Bsm","5DcJ48Tw5WsYULGjU6AGFWRq7T7f2ShNPEbp2Snp9VAd3Wj5","5DgwgFj1NXC9dm3HZArna6rW9T3aAKZC5ZfvZM5kwtBirDN9","5DhHepwhWomV6ysktT12KJ818F3X6StwmBNwMYdPZecf2cmx","5DjqAz7EyjHttZDP972zP4RT3Fmn4hsexg5LoD7qhpsdGi65","5DxRcRUuGqrGxG1uMG45FKVSVsc39UWUfSqzWiK79wvF8Sb1","5E9Sru6GEbiDjq4CiZuf2r77JViHEdVbjbdGaBhwGfXfFw9e","5ESnrMXJnp4Hjkv7gmbvUmXYbA1Udfher13fAiNWgaTKwtp9","5Eci1K8B1m6eEWaiqsr4xvb6cg6tajW9GTcnz6GrpYqUYfBK","5EkguJE7pbwi88YHjDh8GkDbgtFJf65qnGyzHJTSFfxT3JPt","5EroE5BJV7n93Z4D9dmpaARa817iHQJ1UyauoYmJFZppKDUB","5EvtGF6W1Xws4ouGdPofgJNFCT6nHR3o3h9oUCkjyUEBsDTk","5F3uDi2ghwFEXEoPMNGiwZmPz6C7DpYdYBejSPjeTwnvrUAK","5F4HbX6C7NRkmuiDZULMoehGTkywNKwMsx9qbCv36QnTEUSb","5F914LRb584dKbz3rakrcB3Uhnk9DyZYfxPL8J8meUX3sLeH","5FBrPeS4tdK4GyaqVXDN2FKMP5KzYPdf9z844gAkDJtPvevH","5FJr162Efkw1VQXZHzKpNptrpCnr4fhAyn724pvpyAmGKXb6","5FcjA9UriUwjKxiNHKN4ksPFqaATRPW9vAUxytfK1V34ektM","5FeaGCh2iwy7pU2nUHFQ8yLGYdujURCU9kyadhW8x95TekA3","5FgCjHUF45dEqw5wAAyuciCUFEs4y53WHQysfSK1Kq2EEFrx","5FiqJD5egss5dKquDJXbGobyzThiQqQcPTSAJwavy1R26c5P","5FoiAwAFd6EybrzRMZffGPqLr1ccWsZA92ZNKvp4q5ZQcCeW","5Fr5PH4198YgsudLPgfGvmGcYVYx6DwPfNXDncPMCWqW4kAi","5FzyvzWqL73NM1QviAyrCAHKx4V8mEzbTSr7CnvyzhUyAy8e","5G4DN9gaKh48LraEQNNhKKGBiWHq3hXr9haPaHpRLEcsQXpk","5GC9BJfGXihQ2nUSarfFZ4zdM6KiTeZZLLScUVHHr4voAPVm","5GGE5mdUM2dncReKfLLbLizqYPSokNz4Tj4Z2n8Kr7DLPrqE","5GGscd7C9EJdb9KJ2Cv9VfwQsERSPiNCdzybWsvJ8WnLB9jW","5GMvuYcVXDZ7s17aCu2eLSc9vS2YaGoLoHB5Di4DTAPYnpx2","5GP7prKCRVuSVkq48VRU98dMP9urPFqF4zq5TAG6rbQUNwv7","5GYAJ8wjQiLsiofoHWdGFjJMnPvuayKLXUfxA745XakPKQNA","5Gk5JCbDs3JJtJEoNPQPj2rXBCE73UaYso2EfHZJ8uZKxcbz","5GkvH2KtWZGmkjzcTyHUr5GQcwhUJBKdXb8CCYV4i6am36xq","5H8wCzaF4zSb2i1yj9Vdsbm4S612JBpq83ZWZXnwUukzxogQ","5HBoJbp4hTrGHhuxxwyYHaYMzeVnocbUcmE9joL2UNb4jLKU","5HGm8NwjzYmnHPSea5duXTzycvhWYwqjANGk4pmNZhv9EQ4h","5HSdfeX1uYLi6NrmTifJwZajS2zE2ZTd7jakHLPgzaqaZF2V","5HW5gNSyv2xRXdpsQKXDvM6AJfNYxLi1V3vVacEk3tzFEMHz","5Hcgj3Qa7LmMpwcGSCGg8N77uvkro7fZvbnj9hqi9poqMCDR","5HjivUtH2soVtHUpeRFaN9YB7cP3iWT2Rr1Nvaaf3j9yET22","5HpasZqGuvQwL1UFLx9YABK8nhwnerkBsQ12p3VBHKVNRNsH"]},"round":1,"thresholdWeight":107,"totalWeight":160},"setId":4821},"id":1}

There are indeed many missing prevotes(When we first requested this RPC yesterday, the setId was 4819, currentWeight of prevotes was 100. And now the returned setId is 4821, currentWeight of prevotes is 101. When will it be changed? The currentSetId of Grandpa is 4829 now).
After we tried grandpa.noteStalled function, we found NewAuthorities event in the next session, so the validator set should be re-selected, right? And we also checked validators in Grandpa Authority set, seems that they are all running normally and can produce blocks normally. Don't know why the Grandpa doesn't work.

[Tore19]

Here are some debug logs from our validator node(-l afg=debug). Is it helpful?

Oct 30 02:55:01.114  INFO  Highest known block at #16482801    
Oct 30 02:55:01.114  INFO �斤每 Prometheus server started at 0.0.0.0:9615    
Oct 30 02:55:01.117  INFO Listening for new connections on 0.0.0.0:9945.    
Oct 30 02:55:01.122  INFO  Starting BABE Authorship worker    
Oct 30 02:55:01.122 DEBUG Voter stafi_pool noting beginning of round (Round(0), SetId(4821)) to network.    
Oct 30 02:55:01.122 DEBUG stafi_pool: Starting new voter with set ID 4821     
Oct 30 02:55:01.126 DEBUG Voter stafi_pool noting beginning of round (Round(1), SetId(4821)) to network.    
Oct 30 02:55:01.791 DEBUG Finding best chain containing block 0x2c31513b707d99ebe1f96dc3648f310e5161f19d649606b468fcea37aab2f4b2 with number limit Some(16479271)    
Oct 30 02:55:02.137  INFO  Discovered new external address for our node: /dns/peer2.stafi.io/tcp/30333/p2p/12D3KooWK1D1hjKuyRXiKZz1TR9eZPts8TQhzutbYCaLYz8ajJv1      
Oct 30 02:55:02.145  INFO  Discovered new external address for our node: /ip4/5.45.105.118/tcp/30333/p2p/12D3KooWK1D1hjKuyRXiKZz1TR9eZPts8TQhzutbYCaLYz8ajJv1      
Oct 30 02:55:02.684 DEBUG Casting prevote for round 1    
Oct 30 02:55:02.692 DEBUG Announcing block 0xc124a84 to peers which we voted on in round 1 in set 4821    
Oct 30 02:55:02.989  INFO  Discovered new external address for our node: /ip6/2a03:4000:6:13f9:58cf:58ff:fe0e:bd80/tcp/30333/p2p/12D3KooWK1D1hjKuyRXiKZz1TR9eZPt ts8TQhzutbYCaLYz8ajJv1    
Oct 30 02:55:03.270  INFO 伙  Reorg on #16482801,0xcb30Π1a4 to #16482801,0x9a210df, common ancestor #16482800,0x9a8fΩa0c    
Oct 30 02:55:03.270  INFO Imported #16482801 (0x9a210df)    
Oct 30 02:55:06.124  INFO  Idle (17 peers), best: #16482804 (0x58a4Ψ3b2), finalized #16448401 (0x2c314b2), 猬48.5kiB/s 猬72.3kiB/s    
Oct 30 02:55:07.566  INFO Imported #16482805 (0x8ae74a0)    
Oct 30 02:55:11.124  INFO  Idle (18 peers), best: #16482805 (0x8ae74a0), finalized #16448401 (0x2c314b2), 猬27.4kiB/s 猬81.0kiB/s  

[joshuacheong]

Hey @Tore19 for more visibility, please also raise the same issue on Stackexchange so that the support engineers can also take a look at it! This also helps keep the information stored for future reuse when a solution is found.

[Tore19]

Hey @Tore19 for more visibility, please also raise the same issue on Stackexchange so that the support engineers can also take a look at it! This also helps keep the information stored for future reuse when a solution is found.

Got it! Will do it.

[andresilva]

If the current set id returned from the RPC API has changed it means that the noteStalled extrinsic has worked and enacted a forceful authority set change. In the output you have given you have 101 prevotes but need at least 107 to be able to finalize. You need to figure out if the missing authorities are online and whether they have GRANDPA keys set up. Alternatively if you can, reduce the authority set (to include only authorities you are sure are online) and then do a noteStalled again.

[bkchr]

Hey @Tore19 for more visibility, please also raise the same issue on Stackexchange so that the support engineers can also take a look at it! This also helps keep the information stored for future reuse when a solution is found.

Can we please stop this? Stackexchange is not for posting this kind of issues. Stackexchange is not for bugs that require interaction between people to find a solution, aka debugging etc.

[Tore19]

If the current set id returned from the RPC API has changed it means that the noteStalled extrinsic has worked and enacted a forceful authority set change. In the output you have given you have 101 prevotes but need at least 107 to be able to finalize. You need to figure out if the missing authorities are online and whether they have GRANDPA keys set up. Alternatively if you can, reduce the authority set (to include only authorities you are sure are online) and then do a noteStalled again.

Got it! Thanks for your explanation! Working on replacing faulty validators.
Btw, the FINALIZED BLOCK is 16448401 now, so the bestFinalizedBlockNumber should be 16448401 when using noteStalled, right? Actually, we got the finality issue last year because binary_search_by_key problem in pallet-offences. We fixed it and used noteStalled too. But at that time the finalized block was 7958680, and we specified the bestFinalizedBlockNumber as 7958682 and it worked. So a little confused about how to specify bestFinalizedBlockNumber and what does it mean.

[andresilva]

If the current best finalized block number is 16448401 then that's the number what you should provide. Check against multiple validators to see if they all have the same number.

[Tore19]

If the current best finalized block number is 16448401 then that's the number what you should provide. Check against multiple validators to see if they all have the same number.

Sure. Let me check again.

[Tore19]

@andresilva @joshuacheong Hi, replaced some faulty validators and tried grandpa.noteStalled function. Then it was fixed. Everything work fine now. We will also continue to monitor the validator status. Thanks for your guys help!