…to a `Stream` (#8019)

## Summary of changes

Adds `PostAsJson<T>` method to `IApiWebRequest`

## Reason for change

Currently, if we want to send JSON, we serialize it to a string locally,
convert the string to utf-8 bytes, potentially compress those bytes, and
then copy that to the stream. Doing that as efficiently as we can is
somewhat tricky, and we haven't always got it right. By creating a
central method, and writing directly to the underlying stream where we
can, we can potentially see efficiency gains, and can also potentially
make it easier to move to a more modern serializer later.

## Implementation details

- Added `IApiRequest.PostAsJsonAsync<T>(T payload, MultipartCompression
compression)` (and an overload that accepts json settings)
- Implemented the method as a "push stream" approach in each of the
three implementations we currently have (`ApiRequest`, `HttpClient`,
`HttpStream`)
- Benchmarked the implementation to confirm no regressions (see below)

## Test coverage

Added unit tests by specifically serializing telemetry data, and
confirming we get the correct results when we deserialize the other end
(telemetry is one of the candidates for using this approach).

When running benchmarks, it became apparent that we had a serious
regression in our allocations when we added GZIP-ing of our telemetry 😅
I didn't investigate the root cause, because switching to the new
approach (in #8017) will resolve the issue anyway.

Overall conclusion:
- In general, the new approach allocates slightly less than before
- We have a big allocation and speed regression in GZip (specifically
for telemetry), which the new approach will resolve completely
- In general, the new approach allocates the same whether you use gzip
or not
- Throughput is roughly the same both before and after


**`ApiWebRequest` (.NET FX)**

| Method                    |     Mean | Allocated | Alloc Ratio |
| ------------------------- | -------: | --------: | ----------: |
| ApiWebRequest_Before_Gzip | 6.460 ms | 572.44 KB |        1.00 |
| ApiWebRequest_After_Gzip  | 2.037 ms |  20.75 KB |        0.04 |
|                           |          |           |             |
| ApiWebRequest_Before      | 1.949 ms |  22.34 KB |        1.00 |
| ApiWebRequest_After       | 1.908 ms |  20.75 KB |        0.93 |

**`HttpClientRequest` (.NET Core 3.1, .NET 6)** - had to re-enable
keep-alive to avoid connection exhaustion!

| Method | Runtime | Mean | Allocated | Alloc Ratio |
| ---------------------- | ------------- | ---------: | --------: |
----------: |
| HttpClient_Before_Gzip | .NET 6.0 | 4,980.1 us | 406.27 KB | 0.98 |
| HttpClient_After_Gzip | .NET 6.0 | 161.5 us | 12.04 KB | 0.03 |
| HttpClient_Before_Gzip | .NET Core 3.1 | 4,847.4 us | 414.43 KB | 1.00
|
| HttpClient_After_Gzip | .NET Core 3.1 | 166.3 us | 12.97 KB | 0.03 |
| | | | | |
| HttpClient_Before | .NET 6.0 | 129.2 us | 13.03 KB | 0.91 |
| HttpClient_After | .NET 6.0 | 154.9 us | 12.05 KB | 0.84 |
| HttpClient_Before | .NET Core 3.1 | 162.2 us | 14.27 KB | 1.00 |
| HttpClient_After | .NET Core 3.1 | 189.6 us | 13.2 KB | 0.92 |

**`HttpStreamRequest` over UDS (.NET Core 3.1, .NET 6)**

| Method | Runtime | Mean | Allocated | Alloc Ratio |
| ---------------------- | ------------- | ---------: | --------: |
----------: |
| HttpStream_Before_Gzip | .NET 6.0 | 5,362.8 us | 440.87 KB | 0.99 |
| HttpStream_After_Gzip | .NET 6.0 | 444.8 us | 42.63 KB | 0.10 |
| HttpStream_Before_Gzip | .NET Core 3.1 | 5,421.3 us | 446.78 KB | 1.00
|
| HttpStream_After_Gzip | .NET Core 3.1 | 462.6 us | 42.77 KB | 0.10 |
| | | | | |
| HttpStream_Before | .NET 6.0 | 428.7 us | 43.73 KB | 1.01 |
| HttpStream_After | .NET 6.0 | 433.3 us | 42.3 KB | 0.97 |
| HttpStream_Before | .NET Core 3.1 | 445.3 us | 43.5 KB | 1.00 |
| HttpStream_After | .NET Core 3.1 | 448.5 us | 42.62 KB | 0.98 |

**`SocketHandlerRequest` over UDS (..NET 6)**

| Method                    |        Mean | Allocated | Alloc Ratio |
| ------------------------- | ----------: | --------: | ----------: |
| SocketHandler_Before_Gzip | 5,070.65 us | 406.26 KB |        1.00 |
| SocketHandler_After_Gzip  |    97.66 us |  12.28 KB |        0.03 |
|                           |             |           |             |
| SocketHandler_Before      |    53.95 us |  13.01 KB |        1.00 |
| SocketHandler_After       |    87.64 us |  12.28 KB |        0.94 |



<details><summary>Benchmark used (approx)</summary>


```csharp
using System;
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using Datadog.Trace;
using Datadog.Trace.Agent;
using Datadog.Trace.Agent.StreamFactories;
using Datadog.Trace.Agent.Transports;
using Datadog.Trace.Configuration;
using Datadog.Trace.DogStatsd;
using Datadog.Trace.HttpOverStreams;
using Datadog.Trace.Tagging;
using Datadog.Trace.Telemetry;
using Datadog.Trace.Telemetry.Transports;
using Datadog.Trace.Util;
using Datadog.Trace.Vendors.Newtonsoft.Json;
using Datadog.Trace.Vendors.Newtonsoft.Json.Serialization;

namespace Benchmarks.Trace;

[MemoryDiagnoser]
[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
[CategoriesColumn]
public class TelemetryHttpClientBenchmark
{
    private const string BaseUrl = "http://localhost:5035";
    private const string Socket = @"C:\repos\temp\temp74\bin\Release\net10.0\test.socket";

    private TelemetryData _telemetryData;
    private ApiWebRequestFactory _apiWebRequestFactory;
    private Uri _apiEndpointUri;

#if NETCOREAPP3_1_OR_GREATER
    private HttpClientRequestFactory _httpClientRequestFactory;
    private Uri _httpClientEndpointUri;

    private HttpStreamRequestFactory _httpStreamRequestFactory;
    private Uri _httpStreamEndpointUri;
#endif
#if NET5_0_OR_GREATER
    private SocketHandlerRequestFactory _socketHandlerRequestFactory;
    private Uri _socketHandlerEndpointUri;
#endif

    [GlobalSetup]
    public void GlobalSetup()
    {
        _telemetryData = GetData();

        var config = TracerHelper.DefaultConfig;
        config.Add(ConfigurationKeys.TraceEnabled, false);
        var settings = TracerSettings.Create(config);

        _apiWebRequestFactory = new ApiWebRequestFactory(new Uri(BaseUrl), AgentHttpHeaderNames.MinimalHeaders);
        _apiEndpointUri = _apiWebRequestFactory.GetEndpoint("/");

#if NETCOREAPP3_1_OR_GREATER
        _httpClientRequestFactory = new HttpClientRequestFactory(new Uri(BaseUrl), AgentHttpHeaderNames.MinimalHeaders);
        _httpClientEndpointUri = _httpClientRequestFactory.GetEndpoint("/");

        _httpStreamRequestFactory = new HttpStreamRequestFactory(
            new UnixDomainSocketStreamFactory(Socket),
            new DatadogHttpClient(new MinimalAgentHeaderHelper()),
            new Uri(BaseUrl));
        _httpStreamEndpointUri = _httpStreamRequestFactory.GetEndpoint("/");
#endif

#if NET5_0_OR_GREATER
        _socketHandlerRequestFactory = new SocketHandlerRequestFactory(
            new UnixDomainSocketStreamFactory(Socket),
            AgentHttpHeaderNames.MinimalHeaders,
            new Uri(BaseUrl));
        _socketHandlerEndpointUri = _socketHandlerRequestFactory.GetEndpoint("/");
#endif
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
    }

    [BenchmarkCategory("ApiWebRequest", "Uncompressed"), Benchmark(Baseline = true)]
    public async Task<int> ApiWebRequest_Before()
    {
        var request = _apiWebRequestFactory.Create(_apiEndpointUri);
        var data = SerializeTelemetry(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(Encoding.UTF8.GetBytes(data)), "application/json", contentEncoding: null).ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("ApiWebRequest", "Gzip"), Benchmark(Baseline = true)]
    public async Task<int> ApiWebRequest_Before_Gzip()
    {
        var request = _apiWebRequestFactory.Create(_apiEndpointUri);
        var data = SerializeTelemetryWithGzip(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(data), "application/json", contentEncoding: "gzip").ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("ApiWebRequest", "Uncompressed"), Benchmark]
    public async Task<int> ApiWebRequest_After()
    {
        var request = _apiWebRequestFactory.Create(_apiEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }

    [BenchmarkCategory("ApiWebRequest", "Gzip"), Benchmark]
    public async Task<int> ApiWebRequest_After_Gzip()
    {
        var request = _apiWebRequestFactory.Create(_apiEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }

#if NETCOREAPP3_1_OR_GREATER

    [BenchmarkCategory("HttpClient", "Uncompressed"), Benchmark(Baseline = true)]
    public async Task<int> HttpClient_Before()
    {
        var request = _httpClientRequestFactory.Create(_httpClientEndpointUri);
        var data = SerializeTelemetry(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(Encoding.UTF8.GetBytes(data)), "application/json", contentEncoding: null).ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpClient", "Gzip"), Benchmark(Baseline = true)]
    public async Task<int> HttpClient_Before_Gzip()
    {
        var request = _httpClientRequestFactory.Create(_httpClientEndpointUri);
        var data = SerializeTelemetryWithGzip(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(data), "application/json", contentEncoding: "gzip").ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpClient", "Uncompressed"), Benchmark]
    public async Task<int> HttpClient_After()
    {
        var request = _httpClientRequestFactory.Create(_httpClientEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpClient", "Gzip"), Benchmark]
    public async Task<int> HttpClient_After_Gzip()
    {
        var request = _httpClientRequestFactory.Create(_httpClientEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }
#endif

#if NETCOREAPP3_1_OR_GREATER
    [BenchmarkCategory("HttpStream", "Uncompressed"), Benchmark(Baseline = true)]
    public async Task<int> HttpStream_Before()
    {
        var request = _httpStreamRequestFactory.Create(_httpStreamEndpointUri);
        var data = SerializeTelemetry(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(Encoding.UTF8.GetBytes(data)), "application/json", contentEncoding: null).ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpStream", "Gzip"), Benchmark(Baseline = true)]
    public async Task<int> HttpStream_Before_Gzip()
    {
        var request = _httpStreamRequestFactory.Create(_httpStreamEndpointUri);
        var data = SerializeTelemetryWithGzip(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(data), "application/json", contentEncoding: "gzip").ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpStream", "Uncompressed"), Benchmark]
    public async Task<int> HttpStream_After()
    {
        var request = _httpStreamRequestFactory.Create(_httpStreamEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }

    [BenchmarkCategory("HttpStream", "Gzip"), Benchmark]
    public async Task<int> HttpStream_After_Gzip()
    {
        var request = _httpStreamRequestFactory.Create(_httpStreamEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }
#endif

#if NET5_0_OR_GREATER
    [BenchmarkCategory("SocketHandler", "Uncompressed"), Benchmark(Baseline = true)]
    public async Task<int> SocketHandler_Before()
    {
        var request = _socketHandlerRequestFactory.Create(_socketHandlerEndpointUri);
        var data = SerializeTelemetry(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(Encoding.UTF8.GetBytes(data)), "application/json", contentEncoding: null).ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("SocketHandler", "Gzip"), Benchmark(Baseline = true)]
    public async Task<int> SocketHandler_Before_Gzip()
    {
        var request = _socketHandlerRequestFactory.Create(_socketHandlerEndpointUri);
        var data = SerializeTelemetryWithGzip(_telemetryData);
        using var response = await request.PostAsync(new ArraySegment<byte>(data), "application/json", contentEncoding: "gzip").ConfigureAwait(false);
        return response.StatusCode;
    }

    [BenchmarkCategory("SocketHandler", "Uncompressed"), Benchmark]
    public async Task<int> SocketHandler_After()
    {
        var request = _socketHandlerRequestFactory.Create(_socketHandlerEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }

    [BenchmarkCategory("SocketHandler", "Gzip"), Benchmark]
    public async Task<int> SocketHandler_After_Gzip()
    {
        var request = _socketHandlerRequestFactory.Create(_socketHandlerEndpointUri);
        using var response = await request.PostAsJsonAsync(request, compression: MultipartCompression.None);
        return response.StatusCode;
    }
#endif

    internal static string SerializeTelemetry<T>(T data) => JsonConvert.SerializeObject(data, Formatting.None, JsonTelemetryTransport.SerializerSettings);

    internal static byte[] SerializeTelemetryWithGzip<T>(T data)
    {
        using var memStream = new MemoryStream();
        using (var zipStream = new GZipStream(memStream, CompressionMode.Compress, true))
        {
            using var streamWriter = new StreamWriter(zipStream);
            using var jsonWriter = new JsonTextWriter(streamWriter);
            var serializer = new JsonSerializer { NullValueHandling = NullValueHandling.Ignore, ContractResolver = new DefaultContractResolver { NamingStrategy = new SnakeCaseNamingStrategy(), }, Formatting = Formatting.None };

            serializer.Serialize(jsonWriter, data);
        }

        return memStream.ToArray();
    }

    private TelemetryData GetData() =>
        new TelemetryData(
            requestType: TelemetryRequestTypes.GenerateMetrics,
            runtimeId: "20338dfd-f700-4e5c-b3f6-0d470f054ae8",
            seqId: 5672,
            tracerTime: 1628099086,
            application: new ApplicationTelemetryData(
                serviceName: "myapp",
                env: "prod",
                serviceVersion: "1.2.3",
                tracerVersion: "0.33.1",
                languageName: "node.js",
                languageVersion: "14.16.1",
                runtimeName: "dotnet",
                runtimeVersion: "7.0.3",
                commitSha: "testCommitSha",
                repositoryUrl: "testRepositoryUrl",
                processTags: "entrypoint.basedir:Users,entrypoint.workdir:Downloads"),
            host: new HostTelemetryData(
                hostname: "i-09ecf74c319c49be8",
                os: "GNU/Linux",
                architecture: "x86_64")
            {
                OsVersion = "ubuntu 18.04.5 LTS (Bionic Beaver)",
                KernelName = "Linux",
                KernelRelease = "5.4.0-1037-gcp",
                KernelVersion = "#40~18.04.1-Ubuntu SMP Fri Feb 5 15:41:35 UTC 2021"
            },
            payload: new GenerateMetricsPayload(
                new MetricData[]
                {
                    new(
                        "tracer_init_time",
                        new MetricSeries()
                        {
                            new(1575317847, 2241),
                            new(1575317947, 2352),
                        },
                        common: true,
                        type: MetricTypeConstants.Count)
                    {
                        Tags = new[]
                        {
                            "org_id: 2",
                            "environment:test"
                        }
                    },
                    new(
                        "app_sec_initialization_time",
                        new MetricSeries()
                        {
                            new(1575317447, 254),
                            new(1575317547, 643),
                        },
                        common: false,
                        type: MetricTypeConstants.Gauge)
                    {
                        Namespace = MetricNamespaceConstants.ASM,
                        Interval = 60,
                    },
                }));
}
```

</details> 


## Other details

Part of a small stack

- #8019 👈 
- #8017