Resilience | Foundatio

Resilience policies provide a powerful way to handle transient failures and make your applications more robust. Foundatio's resilience system includes retry logic, circuit breakers, timeouts, and exponential backoff.

The IResiliencePolicy Interface

csharp

public interface IResiliencePolicy
{
    // Synchronous methods
    void Execute(Action<CancellationToken> action, CancellationToken cancellationToken = default);
    TResult Execute<TResult>(Func<CancellationToken, TResult> action, CancellationToken cancellationToken = default);

    // Asynchronous methods
    ValueTask ExecuteAsync(Func<CancellationToken, ValueTask> action, CancellationToken cancellationToken = default);
    ValueTask<TResult> ExecuteAsync<TResult>(Func<CancellationToken, ValueTask<TResult>> action, CancellationToken cancellationToken = default);

    // State-based overloads (zero allocations)
    void Execute<TState>(TState state, Action<TState, CancellationToken> action, CancellationToken cancellationToken = default);
    TResult Execute<TState, TResult>(TState state, Func<TState, CancellationToken, TResult> action, CancellationToken cancellationToken = default);
    ValueTask ExecuteAsync<TState>(TState state, Func<TState, CancellationToken, ValueTask> action, CancellationToken cancellationToken = default);
    ValueTask<TResult> ExecuteAsync<TState, TResult>(TState state, Func<TState, CancellationToken, ValueTask<TResult>> action, CancellationToken cancellationToken = default);
}

Basic Usage

Creating a Policy

csharp

using Foundatio.Resilience;

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))
    .WithJitter()
    .Build();

Executing with Retry (Async)

csharp

await policy.ExecuteAsync(async ct =>
{
    await SomeUnreliableOperationAsync(ct);
});

Executing with Retry (Sync)

csharp

policy.Execute(ct =>
{
    SomeUnreliableOperation();
});

With Return Values

csharp

// Async
var result = await policy.ExecuteAsync(async ct =>
{
    return await GetDataFromApiAsync(ct);
});

// Sync
var result = policy.Execute(ct =>
{
    return GetDataFromDatabase();
});

Zero-Allocation Execution (State-Based)

For performance-critical paths, use state-based overloads to avoid closure allocations:

csharp

// Pass state explicitly instead of capturing in a closure
var userId = 42;
var result = await policy.ExecuteAsync(userId, async (id, ct) =>
{
    return await GetUserAsync(id, ct);
});

// Sync version
policy.Execute(userId, (id, ct) =>
{
    ProcessUser(id);
});

ResiliencePolicyBuilder

Retry Configuration

csharp

var policy = new ResiliencePolicyBuilder()
    // Maximum number of attempts (default: 3)
    .WithMaxAttempts(5)

    // Fixed delay between retries
    .WithDelay(TimeSpan.FromSeconds(2))

    // Or exponential delay (doubles each retry)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))

    // Or linear delay (adds fixed amount each retry)
    .WithLinearDelay(TimeSpan.FromSeconds(1))

    // Maximum delay cap
    .WithMaxDelay(TimeSpan.FromMinutes(1))

    // Add randomness to prevent thundering herd
    .WithJitter()

    .Build();

Custom Delay Function

csharp

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithDelayFunction(attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)))
    .Build();

Timeout Configuration

csharp

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(3)
    .WithTimeout(TimeSpan.FromSeconds(30))  // Overall timeout
    .Build();

Conditional Retry

csharp

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))
    .WithShouldRetry((attempt, exception) =>
    {
        // Only retry on specific exceptions
        return exception is HttpRequestException or TimeoutException;
    })
    .Build();

Unhandled Exceptions

By default, OperationCanceledException and BrokenCircuitException are never retried. You can add additional exception types:

csharp

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    // These exceptions will be thrown immediately without retry
    .WithUnhandledException<ArgumentException>()
    .WithUnhandledException<InvalidOperationException>()
    .Build();

Circuit Breaker

Prevent cascading failures by temporarily stopping calls to failing services:

csharp

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(3)
    .WithCircuitBreaker(cb => cb
        // Open circuit after 50% failure rate
        .WithFailureRatio(0.5)
        // Need at least 10 calls before evaluating
        .WithMinimumCalls(10)
        // Keep circuit open for 1 minute
        .WithBreakDuration(TimeSpan.FromMinutes(1)))
    .Build();

Circuit Breaker States

txt

                    ┌────────────────┐
                    │     Closed     │
                    │ (Normal Ops)   │
                    └───────┬────────┘
                            │
              Failures      │      All calls
              exceed        │      pass through
              threshold     │
                            │
                            ▼
                    ┌────────────────┐
              ┌────▶│      Open      │
              │     │ (Fail Fast)    │
              │     └───────┬────────┘
              │             │
    Test call │             │ Break duration
    fails     │             │ expires
              │             │
              │             ▼
              │     ┌────────────────┐
              └─────│   Half-Open    │
                    │  (Testing)     │
                    └───────┬────────┘
                            │
                            │ Test call
                            │ succeeds
                            │
                            ▼
                    ┌────────────────┐
                    │     Closed     │
                    └────────────────┘

Closed: Normal operation, calls pass through
Open: Calls fail immediately without execution
Half-Open: Single test call allowed to check recovery

Checking Circuit State

csharp

var policy = new ResiliencePolicy();
policy.CircuitBreaker = new CircuitBreaker(new CircuitBreakerBuilder()
    .WithFailureRatio(0.5)
    .WithMinimumCalls(10)
    .WithBreakDuration(TimeSpan.FromMinutes(1)));

// Check state
if (policy.CircuitBreaker.State == CircuitState.Open)
{
    _logger.LogWarning("Circuit is open - skipping operation");
    return;
}

await policy.ExecuteAsync(async ct =>
{
    await CallExternalServiceAsync(ct);
});

ResiliencePolicy Properties

csharp

var policy = new ResiliencePolicy
{
    // Maximum retry attempts
    MaxAttempts = 5,

    // Fixed delay between retries
    Delay = TimeSpan.FromSeconds(2),

    // Custom delay function
    GetDelay = ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1)),

    // Maximum delay cap
    MaxDelay = TimeSpan.FromMinutes(1),

    // Add jitter to delays
    UseJitter = true,

    // Overall timeout
    Timeout = TimeSpan.FromMinutes(5),

    // Circuit breaker
    CircuitBreaker = new CircuitBreaker(...),

    // Exceptions that should not be retried
    UnhandledExceptions = { typeof(OperationCanceledException) },

    // Custom retry logic
    ShouldRetry = (attempt, ex) => ex is TransientException,

    // Logger for retry events
    Logger = logger
};

Static Delay Functions

Foundatio provides built-in delay functions:

csharp

// Exponential: 1s, 2s, 4s, 8s, 16s...
ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1))

// Linear: 1s, 2s, 3s, 4s, 5s...
ResiliencePolicy.LinearDelay(TimeSpan.FromSeconds(1))

// Constant: 2s, 2s, 2s, 2s...
ResiliencePolicy.ConstantDelay(TimeSpan.FromSeconds(2))

IResiliencePolicyProvider

Manage multiple named policies:

csharp

using Foundatio.Resilience;

var provider = new ResiliencePolicyProviderBuilder()
    // Default policy for unspecified operations
    .WithDefaultPolicy(builder => builder
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromSeconds(1)))

    // Named policy for external APIs
    .WithPolicy("external-api", builder => builder
        .WithMaxAttempts(5)
        .WithCircuitBreaker()
        .WithTimeout(TimeSpan.FromSeconds(30)))

    // Named policy for database operations
    .WithPolicy("database", builder => builder
        .WithMaxAttempts(3)
        .WithLinearDelay(TimeSpan.FromMilliseconds(100))
        .WithUnhandledException<ArgumentException>())

    .Build();

// Get and use policies
var apiPolicy = provider.GetPolicy("external-api");
await apiPolicy.ExecuteAsync(async ct =>
{
    await CallExternalApiAsync(ct);
});

var dbPolicy = provider.GetPolicy("database");
await dbPolicy.ExecuteAsync(async ct =>
{
    await SaveToDbAsync(ct);
});

Type-Based Policies

Get policies based on service type:

csharp

var provider = new ResiliencePolicyProviderBuilder()
    .WithPolicy<IExternalApiClient>(builder => builder
        .WithMaxAttempts(5)
        .WithCircuitBreaker())
    .WithPolicy<IDatabaseService>(builder => builder
        .WithMaxAttempts(3)
        .WithLinearDelay())
    .Build();

var policy = provider.GetPolicy<IExternalApiClient>();

Common Patterns

HTTP Client with Resilience

csharp

public class ResilientHttpClient
{
    private readonly HttpClient _client;
    private readonly IResiliencePolicy _policy;

    public ResilientHttpClient(HttpClient client)
    {
        _client = client;
        _policy = new ResiliencePolicyBuilder()
            .WithMaxAttempts(3)
            .WithExponentialDelay(TimeSpan.FromSeconds(1))
            .WithCircuitBreaker(cb => cb
                .WithFailureRatio(0.5)
                .WithMinimumCalls(10)
                .WithBreakDuration(TimeSpan.FromMinutes(1)))
            .WithTimeout(TimeSpan.FromSeconds(30))
            .WithShouldRetry((_, ex) =>
                ex is HttpRequestException or TaskCanceledException)
            .Build();
    }

    public async Task<T> GetAsync<T>(string url)
    {
        return await _policy.ExecuteAsync(async ct =>
        {
            var response = await _client.GetAsync(url, ct);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadFromJsonAsync<T>(ct);
        });
    }
}

Database Operations

csharp

public class ResilientRepository
{
    private readonly IResiliencePolicy _policy;

    public ResilientRepository()
    {
        _policy = new ResiliencePolicyBuilder()
            .WithMaxAttempts(3)
            .WithLinearDelay(TimeSpan.FromMilliseconds(100))
            .WithShouldRetry((_, ex) => IsTransientDbException(ex))
            .Build();
    }

    public async Task<User> GetUserAsync(int id)
    {
        return await _policy.ExecuteAsync(async ct =>
        {
            return await _context.Users.FindAsync(id, ct);
        });
    }

    private bool IsTransientDbException(Exception ex)
    {
        return ex is DbUpdateException or TimeoutException;
    }
}

Graceful Degradation

csharp

public class CatalogService
{
    private readonly IResiliencePolicy _policy;
    private readonly ICacheClient _cache;

    public async Task<Product> GetProductAsync(int id)
    {
        try
        {
            return await _policy.ExecuteAsync(async ct =>
            {
                return await _api.GetProductAsync(id, ct);
            });
        }
        catch (BrokenCircuitException)
        {
            // Fall back to cached data when circuit is open
            var cached = await _cache.GetAsync<Product>($"product:{id}");
            if (cached.HasValue)
            {
                _logger.LogWarning("Returning cached product {Id}", id);
                return cached.Value;
            }
            throw;
        }
    }
}

Retry with Logging

csharp

var policy = new ResiliencePolicy
{
    MaxAttempts = 5,
    GetDelay = ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1)),
    Logger = loggerFactory.CreateLogger("Resilience")
};

// Logs will include attempt number, delay, and exception details
await policy.ExecuteAsync(async ct =>
{
    await UnreliableOperationAsync(ct);
});

Integration with Foundatio

Cache with Resilience

csharp

var resilientCache = new ResilientCacheClient(
    new RedisCacheClient(...),
    new ResiliencePolicyBuilder()
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromMilliseconds(100))
        .Build()
);

Queue with Resilience

csharp

public class ResilientQueueProcessor
{
    private readonly IQueue<WorkItem> _queue;
    private readonly IResiliencePolicy _policy;

    public async Task ProcessAsync(WorkItem item)
    {
        await _policy.ExecuteAsync(async ct =>
        {
            await DoProcessingAsync(item, ct);
        });
    }
}

Dependency Injection

Register Policy Provider

csharp

services.AddSingleton<IResiliencePolicyProvider>(sp =>
{
    var logger = sp.GetRequiredService<ILoggerFactory>().CreateLogger("Resilience");

    return new ResiliencePolicyProviderBuilder()
        .WithDefaultPolicy(b => b
            .WithMaxAttempts(3)
            .WithExponentialDelay(TimeSpan.FromSeconds(1))
            .WithLogger(logger))
        .WithPolicy("http", b => b
            .WithMaxAttempts(5)
            .WithCircuitBreaker()
            .WithTimeout(TimeSpan.FromSeconds(30)))
        .Build();
});

Use in Services

csharp

public class MyService
{
    private readonly IResiliencePolicy _policy;

    public MyService(IResiliencePolicyProvider policyProvider)
    {
        _policy = policyProvider.GetPolicy("http");
    }
}

Best Practices

1. Use Appropriate Timeouts

csharp

// Match timeout to operation type
.WithTimeout(TimeSpan.FromSeconds(5))   // Fast operations
.WithTimeout(TimeSpan.FromSeconds(30))  // API calls
.WithTimeout(TimeSpan.FromMinutes(5))   // Long operations

2. Configure Circuit Breaker Thresholds

csharp

.WithCircuitBreaker(cb => cb
    // High traffic: needs more samples
    .WithMinimumCalls(100)
    .WithFailureRatio(0.5)

    // Low traffic: fewer samples
    .WithMinimumCalls(10)
    .WithFailureRatio(0.3)
)

3. Use Jitter to Prevent Thundering Herd

csharp

.WithExponentialDelay(TimeSpan.FromSeconds(1))
.WithJitter()  // Adds randomness to prevent synchronized retries

4. Handle Specific Exceptions

csharp

.WithShouldRetry((attempt, ex) =>
{
    // Only retry transient failures
    return ex is HttpRequestException
        or TimeoutException
        or SocketException;
})
.WithUnhandledException<OperationCanceledException>()
.WithUnhandledException<ArgumentException>()

Note: By default, OperationCanceledException and BrokenCircuitException are never retried. Additionally, Foundatio's built-in components automatically exclude their feature-specific exceptions from retries:
MessageBusBase excludes MessageBusException
CacheLockProvider excludes CacheException
This ensures that deliberate application-level failures (like a blocked RabbitMQ connection or a cache operation error) are not wastefully retried.

5. Log Retry Attempts

csharp

var policy = new ResiliencePolicy
{
    Logger = loggerFactory.CreateLogger("Resilience"),
    MaxAttempts = 5
};
// Automatically logs each retry attempt

Performance

Foundatio's resilience implementation is optimized for high performance with minimal allocations.

Allocation-Free Execution

Use the state-based overloads to achieve zero heap allocations in hot paths:

csharp

// Instead of capturing variables in a closure (allocates):
var userId = GetUserId();
await policy.ExecuteAsync(async ct => await GetUserAsync(userId, ct));

// Pass state explicitly (zero allocations):
var userId = GetUserId();
await policy.ExecuteAsync(userId, async (id, ct) => await GetUserAsync(id, ct));

Sync vs Async

Choose the appropriate execution method:

csharp

// Use sync for CPU-bound or already-completed operations
var cachedValue = policy.Execute(_ => cache.Get(key));

// Use async for I/O-bound operations
var apiResult = await policy.ExecuteAsync(async ct => await api.CallAsync(ct));

Benchmark Results

Foundatio consistently outperforms alternatives when retry policies are configured:

Scenario	Foundatio	Polly	Foundatio Advantage
Sync with retries	~23 ns	~122 ns	5.3x faster
Async with retries	~37 ns	~141 ns	3.8x faster
State-based (zero-alloc)	~31 ns, 0 B	~131 ns, 88 B	4.2x faster, zero allocations

Benchmarks run on AMD Ryzen 7 9800X3D, .NET 10.0

Next Steps

Caching - Combine with cache fallbacks
Queues - Resilient queue processing
Jobs - Retry job execution

The IResiliencePolicy Interface ​

Basic Usage ​

Creating a Policy ​

Executing with Retry (Async) ​

Executing with Retry (Sync) ​

With Return Values ​

Zero-Allocation Execution (State-Based) ​

ResiliencePolicyBuilder ​

Retry Configuration ​

Custom Delay Function ​

Timeout Configuration ​

Conditional Retry ​

Unhandled Exceptions ​

Circuit Breaker ​

Circuit Breaker States ​

Checking Circuit State ​

ResiliencePolicy Properties ​

Static Delay Functions ​

IResiliencePolicyProvider ​

Type-Based Policies ​

Common Patterns ​

HTTP Client with Resilience ​

Database Operations ​

Graceful Degradation ​

Retry with Logging ​

Integration with Foundatio ​

Cache with Resilience ​

Queue with Resilience ​

Dependency Injection ​

Register Policy Provider ​

Use in Services ​

Best Practices ​

1. Use Appropriate Timeouts ​

2. Configure Circuit Breaker Thresholds ​

3. Use Jitter to Prevent Thundering Herd ​

4. Handle Specific Exceptions ​

5. Log Retry Attempts ​

Performance ​

Allocation-Free Execution ​

Sync vs Async ​

Benchmark Results ​

Next Steps ​

The IResiliencePolicy Interface

Basic Usage

Creating a Policy

Executing with Retry (Async)

Executing with Retry (Sync)

With Return Values

Zero-Allocation Execution (State-Based)

ResiliencePolicyBuilder

Retry Configuration

Custom Delay Function

Timeout Configuration

Conditional Retry

Unhandled Exceptions

Circuit Breaker

Circuit Breaker States

Checking Circuit State

ResiliencePolicy Properties

Static Delay Functions

IResiliencePolicyProvider

Type-Based Policies

Common Patterns

HTTP Client with Resilience

Database Operations

Graceful Degradation

Retry with Logging

Integration with Foundatio

Cache with Resilience

Queue with Resilience

Dependency Injection

Register Policy Provider

Use in Services

Best Practices

1. Use Appropriate Timeouts

2. Configure Circuit Breaker Thresholds

3. Use Jitter to Prevent Thundering Herd

4. Handle Specific Exceptions

5. Log Retry Attempts

Performance

Allocation-Free Execution

Sync vs Async

Benchmark Results

Next Steps