Handle Backlog Processor Startup Failures by nathan-miller23 · Pull Request #3002 · StackExchange/StackExchange.Redis
Behavior
Whenever our client host experiences transient memory contention, we note that a portion of our ConnectionMultiplexer clients get 'stuck' in state where backlog writer is in Activating state. Example error message below:
Further analysis showed that client returned IsConnected=true with PhysicalBridge in ConnectedEstablished state. ProcDump revealed that no StackExchange.Redis Backlog thread was present
Bug
When StartBacklogProcessor fails to start the backlog processor thread (say, when thread.Start fails with System.OutOFMemoryException) system gets stuck in state where _backlogProcessorIsRunning is true but no backlog thread is doing work.
Fix
try-catch-finally pattern to ensure we reset _backlogPRocessorIsRunning state when we fail to start thread
