Retry Pattern

Transient exceptions are a common occurance in the cloud based applications. Most of these temporary faults are self-healing and could succeed when triggered with a slight delay. Some of the common transient errors include momentary loss of connectivity to services, a busy database server that throttle requests to ease off workloads.

Ideally an application which commnunicate with the cloud services needs to be aware of the possibility of such transient errors and should have a way to handle them. The Retry Pattern is an efficient and simple way to enable applications to be transient fault tolerant, minimizing the effects it has on the business tasks performed by the application.

Retry Pattern

The core concept behind the pattern is to retry after a slight delay, expecting the transient fault to have resolved itself. On detecting a failure, the application can do the following.

• Retry

If the exception is seemingly transient, the application can retry the failing request again, expecting the issue to be resolved immediately.

• Retry after delay

In some cases, the issue might not be resolved immediately. In such circumstances, it would be ideal to retry the failing request again after a slight delay. The duration of delay could be increased exponentially over each retry.

• Cancel

On other occations, some of the exceptions are unlikely to be resolved. This could be due to the nature of exception or due to failure of repeated retries. Under such conditions, the application should cancel the request and report the error.

Sample Implementation

Let us now proceed to write some code to demonstrate the implementation of Retry Pattern. We will later make use of the [Polly] library later in implement the pattern, but for now, let us write some code ourselves.

async Task<T> ExecuteOperation<T>(Func<Task<T>> action,int maxRetries = 5, int delay = 1000)
{
    int currentRetry = 0;
    while (true)
    {
        try
        {
            returnawait action();
        }
        catch (Exception ex)
        {
            currentRetry++;

            if (currentRetry > maxRetries || !ex.IsTransient())
            {
                Debugger.Break();
                throw;
            }

            await Task.Delay(delay);
        }
    }
}

The above ExecuteOperation handler accepts an action to be performed. Optionally, the consumer code could provide the max retries and delay between each retries. After each failure the currentRetry count is incremented and checked if it exceeds the permissible limit.

The client code could look like the following.

app.MapGet("/retrydemo", async (IHttpClientFactory clientFactory) =>
{
    returnawait ExecuteOperation(_);

    async Task<string> _()
    {
        var url = $"https://jsonplaceholder.typicode.com/todos/1";

        var client = clientFactory.CreateClient("Demo");
        var response = await client.GetAsync(url);
        var result = await response.Content.ReadAsStringAsync();
        return result;
    }


}).WithName("RetryDemo");

In this above method, we are using the HttpClient to retrieve data from an external API. In case of any transient errors, the Retry Pattern kicks in and ensure momentary failures do not affect the business tasks.

info: System.Net.Http.HttpClient.Demo.LogicalHandler[100]
     Start processing HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.ClientHandler[100]
     Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.LogicalHandler[100]
     Start processing HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.ClientHandler[100]
     Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.LogicalHandler[100]
     Start processing HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.ClientHandler[100]
     Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.LogicalHandler[100]
     Start processing HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.ClientHandler[100]
     Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.Demo.ClientHandler[101]
     Received HTTP response headers after 468.0136ms - 200
info: System.Net.Http.HttpClient.Demo.LogicalHandler[101]
     End processing HTTP request after 492.6796ms - 200

Implementation using Polly

Polly is a transient fault handling library which makes developers life easy with easy to implement policies like Retry. Let us now attempt to implement the Retry Pattern using the Polly Library.

We will begin by installing the Polly Package.

Install-Package Polly

We will now proceed to register the required policy, which in this case is the Retry Policy.

builder.Services.AddHttpClient("Demo")
        .AddPolicyHandler(GetRetryPolicy()); 

Where the GetRetryPolicy is defined as

IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    return HttpPolicyExtensions
         .HandleTransientHttpError()
         .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
         .WaitAndRetryAsync(6, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2,
                                                                     retryAttempt)));
}

That’s all you would need to do. Now you can easily implement the consuming client code without any decorative/boiler plate code.

app.MapGet("/retrydemo", async (IHttpClientFactory clientFactory) =>
{
    var url = $"https://jsonplaceholder.typicode.com/todos/1";

    var client = clientFactory.CreateClient("Demo");
    var response = await client.GetAsync(url);
    var result = await response.Content.ReadAsStringAsync();
    return result;
}).WithName("RetryDemo");

Conclusion

The retry pattern is an efficient and easy to implement transient fault tolerant applications, which ensure the application would not impacted due to momentary errors. This is done by repeatedly retrying the request which had failed and is likely to be transient. On other hand, if it recognizes that the exception is not likely to be resolved soon, it can raise the fault immediately without any delay.

Access Sample Code illustrated in the post.