Backport of #113503 to release/9.0-staging

/cc @BrennanConroy

Customer Impact

• Customer reported
• Found internally

dotnet/aspnetcore#60911
The below code shows the problem, where there is serialized data in the PipeWriter even though there weren't any items returned by the IAsyncEnumerable<>. This differs in behavior from the Stream overload where there isn't any data in the stream.

This is an issue in apps like ASP.NET Core because the data written goes out in the Http response and if the user has error handling (like returning problem details) then the output is invalid due to the data written by the serializer.

Simplified ASP.NET Core app showing issue:

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();
app.UseExceptionHandler(exceptionHandlerApp =>
    exceptionHandlerApp.Run(async context => await Results.Problem().ExecuteAsync(context)));

app.MapGet("/weatherforecast", ReturnAsyncEnumerable);

app.Run();

async IAsyncEnumerable<int> ReturnAsyncEnumerable()
{
    await Task.Delay(100);
    throw new NotImplementedException();

    yield return 1;
}

Output of an http request to /weatherforecast:

[{"type":"https://tools.ietf.org/html/rfc9110#section-15.6.1","title":"An error occurred while processing your request.","status":500}

Note the [ in the beginning.

Another example is a pure console app using the new overload:

Pipe pipe = new Pipe();
try
{
    await JsonSerializer.SerializeAsync(pipe.Writer, AsyncEnumerable());
}
catch { }

Debug.Assert(pipe.Writer.UnflushedBytes == 0); // <-- this is non-zero before the PR fix

async IAsyncEnumerable<int> AsyncEnumerable()
{
    await Task.Delay(100);
    throw new Exception();

    yield return 1;
}

Regression

• Yes
• No

It's technically not a regression for console apps moving from 8.0 -> 9.0, but it is a regression in behavior if you go from ASP.NET Core 8.0 -> 9.0 or use the new PipeWriter overloads in Json in your console apps.

Testing

[How was the fix verified? How was the issue missed previously? What tests were added?]

There was already a regression test for this case for the Stream overload. When adding the PipeWriter overload we missed adding PipeWriter to the suite of tests that the regression test was part of. Fixing that testing oversight showed that the issue would have been caught before and shows that the fix resolves the issue.

Risk

[High/Medium/Low. Justify the indication by mentioning how risks were measured and addressed.]
Low

The testing in Json is extensive, we just missed adding a suite of tests for the new (in 9.0) PipeWriter overload.