K8s target provider will wait for deployment to be updated/removed since #436. The timeout is 5 minutes for each component.
Assuming we have an instance to deploy 5 components to 2 targets. If all of them timeout, it takes 50 minutes to do one round deployment.
And the retry mechanism in solution Reconcile function makes the case even worse.

In the other side, k8s controller has a fixed timeout to wait for summary in terminated state. If not, it fails with timeout error like below which hides the actual error for each components.

Another side effect is that the update reconcile will block reconcile for further changes. This is because we lack cancellation token to cancel an ongoing job.

We should try to avoid k8s controller timeout by removing unnecessary retry and limiting timeout value.