"-READONLY You can't write against a read only replica." on ScriptEvaluate(...)
Background:
- AWS MemoryDB (REDIS Engine 6.2)
- Version: 2.5.43.
During some of testing, we've noticed that the error "-READONLY You can't write against a read only replica.", is showing up in our logs when we attempt to resize our shards in MemoryDB (change the size of the nodes, which causes it to fail over to a replica).
Attempting to retry the operation (we do 3 retries with exponential backoff) it fails all 3 times before finally being thrown a final time. Therefore, it looks like it's not reassigning the primary on these errors.
Looking at the exception, this only seems to affect ScriptEvaluate (LUA Scripts). We don't see any of our other various calls throwing this error:
at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1212
Full Stacktrace:
message=ERR Error running script (call to f_f3...0): @user_script:50: @user_script: 50: -READONLY You can't write against a read only replica.
Stack trace
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2836
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 53
at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1212
We don't see this behaviour when we do other cluster operations:
- Primary Node Fail-overs to Replicants
- Adding/removing shards, which cause a hashslot migrations
I will attempt to dig deeper next week, but I wanted to report it.