Home Original page

Prevent too many ProcessBacklog() from being scheduled by devbv · Pull Request #1574 · StackExchange/StackExchange.Redis

@devbv

This PR prevents the situation that a consumer makes the number of backlog 1->0 infinitely and many producers schedule new ProcessBacklog() after they make the number of backlog 0->1.
The situation could lead a starvation of thread pool and a lot of timeout when we use transaction heavily. (#1572)

@devbv devbv changed the title Prevent too many ProcesBacklog() from being scheduled Prevent too many ProcessBacklog() from being scheduled

Oct 12, 2020

@devbv

@arsnyder16

@mgravell I am also seeing these errors in my logs. They are a semi related to the issue solved with #1585 in that in these timeout scenarios seems to cascade and build upon each other, and the library struggles to recover.

I would also be interested in seeing your opinion on getting this change in as well

@devbv

image
Additional info for better understanding. This is stacktrace of threads when the timeout is occurred. You can see 9 of 10 worker threads are stopped at MutexSlim.TakeWithTimeout called by PhysicalBridge.ProcessBacklog()
This is reproduced from the sample code on #1572

@NickCraver

Merging main in here for build check changes

mgravell added a commit that referenced this pull request

Nov 13, 2020 
- make the backlog processor async
- check for work *before* trying to get the lock

@mgravell

This is a great find, and could explain a lot, but I'm not sure that the approach is the best way of solving it - I'm playing with an alternative strategy in #1612 - but: even if I merge that, I'll still give credit for the research here; much appreciated!

@mgravell

mgravell added a commit that referenced this pull request

Nov 13, 2020 
* alternative implementation along the lines of #1574:

- make the backlog processor async
- check for work *before* trying to get the lock

* release notes