Let us say that I have services A, B, C, D. For a request, the system needs to do a long call chain to calculate the result, the chain is A -> B -> C -> D. The request is critical and transient faults can happen anywhere. My question is that which nodes I should apply retry mechanism. Should I do it for each call or just at service A?
Currently, my department is working at service C and we do retry when our service calling service D. But now, we realized that the most of the retires are unnecessary because they conduct to fail finally.
My duty for now is to optimize it.
I have to consider the following factors:
- caller's timeout settings
- Does caller retry as well
- P99 of the callee API
- Is callee API idempotent.
I have 3 options
- Do retry with the old code but with different retry strategies, such as exponential back-off
- Switch to Resilience4j because it has a micrometer feature
- Remove all retry logics.
Can you discuss which option is the most suitable one?