Inbound federation worker began spinning CPU after a surge in federation traffic

[Half-Shot]

Description

After a restart, it became fine again. I couldn't find a specific issue for "federation stops, cpu spins", so here is one.

Steps to reproduce

• Have a host
• Have lots of incoming traffic because the host runs a major IRC bridge.
• CPU spins itself to death (>100% usage)
• EDU traffic drops to near zero, reports of host feeling "broken".

Version information

• Homeserver: libera.ems.host (Note, this host is only used by the bridge a.k.a all federation traffic, no sync traffic)
• Version: 1.50.0
• Install method: N/A
• Platform: EMS

link to logs

[clokep]

I think this has some overlap with #11818, but I'm unsure it is an identical issue. @Half-Shot any idea if this was from the same event? Doesn't seem like it since they're a few days apart.

[Half-Shot]

It does seem to be very similar, synapse-federation-inbound-0 _process_incoming_pdus_in_room_inner

[erikjohnston]

@Half-Shot if you still have the data can you post screen shots of the per block metrics please? That should narrow down where in the _process_incoming_pdus_in_room_inner code path the CPU and DB is being spent.

[Half-Shot]

Sorry, the data will have evaporated by now :(. I'll catch it next time.