Fix benchmark flakiness by NachoEchevarria · Pull Request #7790 · DataDog/dd-trace-dotnet
.NET Framework 4.8 - Baseline
duration192.46 ± (192.43 - 193.26) ms193.93 ± (193.68 - 194.56) ms+0.8%✅⬆️
.NET Framework 4.8 - Bailout
duration196.34 ± (196.07 - 196.67) ms197.33 ± (197.14 - 197.96) ms+0.5%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1104.28 ± (1110.04 - 1119.65) ms1105.45 ± (1111.20 - 1120.81) ms+0.1%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms187.63 ± (187.29 - 187.97) ms188.39 ± (188.06 - 188.72) ms+0.4%✅⬆️
process.time_to_main_ms80.23 ± (80.04 - 80.42) ms81.29 ± (81.07 - 81.51) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.06 ± (16.03 - 16.09) MB16.18 ± (16.15 - 16.21) MB+0.7%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 20)20 ± (20 - 20)+0.6%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms187.08 ± (186.77 - 187.38) ms187.58 ± (187.34 - 187.82) ms+0.3%✅⬆️
process.time_to_main_ms81.72 ± (81.59 - 81.85) ms82.48 ± (82.34 - 82.61) ms+0.9%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.15 ± (16.13 - 16.18) MB16.20 ± (16.17 - 16.23) MB+0.3%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (20 - 21)-0.5%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms401.57 ± (398.56 - 404.58) ms405.80 ± (402.68 - 408.92) ms+1.1%✅⬆️
process.time_to_main_ms469.82 ± (469.27 - 470.37) ms475.89 ± (475.07 - 476.71) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed58.26 ± (58.12 - 58.41) MB58.61 ± (58.49 - 58.74) MB+0.6%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.0%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms191.33 ± (190.98 - 191.68) ms192.57 ± (192.17 - 192.97) ms+0.6%✅⬆️
process.time_to_main_ms69.68 ± (69.51 - 69.86) ms70.62 ± (70.43 - 70.81) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.04 ± (15.88 - 16.20) MB16.23 ± (16.13 - 16.33) MB+1.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 19)+1.7%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms190.15 ± (189.93 - 190.37) ms191.94 ± (191.57 - 192.31) ms+0.9%✅⬆️
process.time_to_main_ms70.37 ± (70.31 - 70.44) ms71.51 ± (71.41 - 71.62) ms+1.6%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.02 ± (15.86 - 16.18) MB16.23 ± (16.10 - 16.35) MB+1.3%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)20 ± (19 - 20)+2.0%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms418.39 ± (415.12 - 421.66) ms414.30 ± (411.36 - 417.23) ms-1.0%✅
process.time_to_main_ms442.71 ± (442.09 - 443.34) ms443.27 ± (442.66 - 443.89) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed58.63 ± (58.49 - 58.76) MB58.79 ± (58.66 - 58.92) MB+0.3%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 30)30 ± (29 - 30)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms190.44 ± (190.10 - 190.78) ms190.34 ± (190.01 - 190.67) ms-0.1%✅
process.time_to_main_ms69.32 ± (69.15 - 69.48) ms69.77 ± (69.61 - 69.92) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.76 ± (11.74 - 11.79) MB11.76 ± (11.72 - 11.81) MB-0.0%✅
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)-0.4%✅
.NET 8 - Bailout
process.internal_duration_ms189.46 ± (189.20 - 189.72) ms190.39 ± (190.10 - 190.67) ms+0.5%✅⬆️
process.time_to_main_ms70.28 ± (70.17 - 70.39) ms70.92 ± (70.79 - 71.05) ms+0.9%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.81 ± (11.76 - 11.87) MB11.83 ± (11.80 - 11.86) MB+0.2%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.8%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms370.02 ± (368.60 - 371.44) ms366.68 ± (365.14 - 368.21) ms-0.9%✅
process.time_to_main_ms433.91 ± (433.08 - 434.74) ms430.24 ± (429.67 - 430.82) ms-0.8%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed47.85 ± (47.82 - 47.88) MB47.84 ± (47.82 - 47.87) MB-0.0%✅
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.6%✅⬆️