Fix #3089 'histograms are dissapearing' by DraggonFantasy · Pull Request #3196 · open-telemetry/opentelemetry-python
This PR fixes #3089 - the problem with histogram metric type when the exporting the metrics with unchanged histogram value crashes the OTEL Collector.
Please delete options that are not relevant.
I've ran the following Python snippet (it's the minimal example I figured out to reproduce the issue):
import time from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter from opentelemetry.metrics import get_meter_provider, set_meter_provider from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader exporter = OTLPMetricExporter(endpoint="http://localhost:7777/v1/metrics") reader = PeriodicExportingMetricReader(exporter, export_interval_millis=1000) set_meter_provider(MeterProvider(metric_readers=[reader])) meter = get_meter_provider().get_meter("myapp", "0.1.2") histogram = meter.create_histogram( "py_histogram", unit="ms", description="Test histogram", ) counter = meter.create_counter( f"py_counter", unit=f"items", description=f"Test counter" ) histogram.record(12) time.sleep(1) counter.add(1)
configuration-otl-collector-1 | 2023-02-22T22:56:21.074Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 1}
configuration-otl-collector-1 | 2023-02-22T22:56:21.074Z info ResourceMetrics #0
configuration-otl-collector-1 | Resource SchemaURL:
configuration-otl-collector-1 | Resource attributes:
configuration-otl-collector-1 | -> telemetry.sdk.language: Str(python)
configuration-otl-collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
configuration-otl-collector-1 | -> telemetry.sdk.version: Str(1.15.0)
configuration-otl-collector-1 | -> service.name: Str(unknown_service)
configuration-otl-collector-1 | ScopeMetrics #0
configuration-otl-collector-1 | ScopeMetrics SchemaURL:
configuration-otl-collector-1 | InstrumentationScope myapp 0.1.2
configuration-otl-collector-1 | Metric #0
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_histogram
configuration-otl-collector-1 | -> Description: Test histogram
configuration-otl-collector-1 | -> Unit: ms
configuration-otl-collector-1 | -> DataType: Histogram
configuration-otl-collector-1 | -> AggregationTemporality: Cumulative
configuration-otl-collector-1 | HistogramDataPoints #0
configuration-otl-collector-1 | StartTimestamp: 2023-02-22 22:56:20.032467796 +0000 UTC
configuration-otl-collector-1 | Timestamp: 2023-02-22 22:56:21.032682472 +0000 UTC
configuration-otl-collector-1 | Count: 1
configuration-otl-collector-1 | Sum: 12.000000
configuration-otl-collector-1 | Min: 12.000000
configuration-otl-collector-1 | Max: 12.000000
configuration-otl-collector-1 | ExplicitBounds #0: 0.000000
configuration-otl-collector-1 | ExplicitBounds #1: 5.000000
configuration-otl-collector-1 | ExplicitBounds #2: 10.000000
configuration-otl-collector-1 | ExplicitBounds #3: 25.000000
configuration-otl-collector-1 | ExplicitBounds #4: 50.000000
configuration-otl-collector-1 | ExplicitBounds #5: 75.000000
configuration-otl-collector-1 | ExplicitBounds #6: 100.000000
configuration-otl-collector-1 | ExplicitBounds #7: 250.000000
configuration-otl-collector-1 | ExplicitBounds #8: 500.000000
configuration-otl-collector-1 | ExplicitBounds #9: 750.000000
configuration-otl-collector-1 | ExplicitBounds #10: 1000.000000
configuration-otl-collector-1 | ExplicitBounds #11: 2500.000000
configuration-otl-collector-1 | ExplicitBounds #12: 5000.000000
configuration-otl-collector-1 | ExplicitBounds #13: 7500.000000
configuration-otl-collector-1 | ExplicitBounds #14: 10000.000000
configuration-otl-collector-1 | Buckets #0, Count: 0
configuration-otl-collector-1 | Buckets #1, Count: 0
configuration-otl-collector-1 | Buckets #2, Count: 0
configuration-otl-collector-1 | Buckets #3, Count: 1
configuration-otl-collector-1 | Buckets #4, Count: 0
configuration-otl-collector-1 | Buckets #5, Count: 0
configuration-otl-collector-1 | Buckets #6, Count: 0
configuration-otl-collector-1 | Buckets #7, Count: 0
configuration-otl-collector-1 | Buckets #8, Count: 0
configuration-otl-collector-1 | Buckets #9, Count: 0
configuration-otl-collector-1 | Buckets #10, Count: 0
configuration-otl-collector-1 | Buckets #11, Count: 0
configuration-otl-collector-1 | Buckets #12, Count: 0
configuration-otl-collector-1 | Buckets #13, Count: 0
configuration-otl-collector-1 | Buckets #14, Count: 0
configuration-otl-collector-1 | Buckets #15, Count: 0
configuration-otl-collector-1 | {"kind": "exporter", "data_type": "metrics", "name": "logging"}
configuration-otl-collector-1 | 2023-02-22T22:56:21.075Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 2}
configuration-otl-collector-1 | 2023-02-22T22:56:21.075Z info ResourceMetrics #0
configuration-otl-collector-1 | Resource SchemaURL:
configuration-otl-collector-1 | Resource attributes:
configuration-otl-collector-1 | -> telemetry.sdk.language: Str(python)
configuration-otl-collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
configuration-otl-collector-1 | -> telemetry.sdk.version: Str(1.15.0)
configuration-otl-collector-1 | -> service.name: Str(unknown_service)
configuration-otl-collector-1 | ScopeMetrics #0
configuration-otl-collector-1 | ScopeMetrics SchemaURL:
configuration-otl-collector-1 | InstrumentationScope myapp 0.1.2
configuration-otl-collector-1 | Metric #0
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_histogram
configuration-otl-collector-1 | -> Description: Test histogram
configuration-otl-collector-1 | -> Unit: ms
configuration-otl-collector-1 | -> DataType: Empty
configuration-otl-collector-1 | Metric #1
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_counter
configuration-otl-collector-1 | -> Description: Test counter
configuration-otl-collector-1 | -> Unit: items
configuration-otl-collector-1 | -> DataType: Sum
configuration-otl-collector-1 | -> IsMonotonic: true
configuration-otl-collector-1 | -> AggregationTemporality: Cumulative
configuration-otl-collector-1 | NumberDataPoints #0
configuration-otl-collector-1 | StartTimestamp: 2023-02-22 22:56:21.033651214 +0000 UTC
configuration-otl-collector-1 | Timestamp: 2023-02-22 22:56:21.074523435 +0000 UTC
configuration-otl-collector-1 | Value: 1
configuration-otl-collector-1 | {"kind": "exporter", "data_type": "metrics", "name": "logging"}
configuration-otl-collector-1 | 2023-02-22T22:56:21.075Z error prometheusexporter@v0.71.0/accumulator.go:105 failed to translate metric {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "py_histogram"}
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.71.0/accumulator.go:105
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.71.0/accumulator.go:82
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.71.0/collector.go:67
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.71.0/prometheus.go:96
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/metrics.go:65
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/common.go:208
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/queued_retry.go:365
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/metrics.go:136
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/queued_retry.go:301
configuration-otl-collector-1 | go.opentelemetry.io/collector/exporter/exporterhelper.NewMetricsExporter.func2
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/exporter/exporterhelper/metrics.go:116
configuration-otl-collector-1 | go.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetrics
configuration-otl-collector-1 | go.opentelemetry.io/collector/consumer@v0.71.0/metrics.go:36
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry.(*wrapperMetricsExporter).ConsumeMetrics
configuration-otl-collector-1 | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry@v0.71.0/resource_to_telemetry.go:43
configuration-otl-collector-1 | go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*metricsConsumer).ConsumeMetrics
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/service/internal/fanoutconsumer/metrics.go:77
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver/internal/metrics.(*Receiver).Export
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver@v0.71.0/internal/metrics/otlp.go:54
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver.handleMetrics
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver@v0.71.0/otlphttp.go:73
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver.(*otlpReceiver).registerMetricsConsumer.func1
configuration-otl-collector-1 | go.opentelemetry.io/collector/receiver/otlpreceiver@v0.71.0/otlp.go:235
configuration-otl-collector-1 | net/http.HandlerFunc.ServeHTTP
configuration-otl-collector-1 | net/http/server.go:2109
configuration-otl-collector-1 | net/http.(*ServeMux).ServeHTTP
configuration-otl-collector-1 | net/http/server.go:2487
configuration-otl-collector-1 | go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/config/confighttp/compression.go:162
configuration-otl-collector-1 | net/http.HandlerFunc.ServeHTTP
configuration-otl-collector-1 | net/http/server.go:2109
configuration-otl-collector-1 | go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
configuration-otl-collector-1 | go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.39.0/handler.go:213
configuration-otl-collector-1 | go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
configuration-otl-collector-1 | go.opentelemetry.io/collector@v0.71.0/config/confighttp/clientinfohandler.go:39
configuration-otl-collector-1 | net/http.serverHandler.ServeHTTP
configuration-otl-collector-1 | net/http/server.go:2947
configuration-otl-collector-1 | net/http.(*conn).serve
configuration-otl-collector-1 | net/http/server.go:1991
configuration-otl-collector-1 | 2023-02-22T22:57:18.991Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 1}
configuration-otl-collector-1 | 2023-02-22T22:57:18.991Z info ResourceMetrics #0
configuration-otl-collector-1 | Resource SchemaURL:
configuration-otl-collector-1 | Resource attributes:
configuration-otl-collector-1 | -> telemetry.sdk.language: Str(python)
configuration-otl-collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
configuration-otl-collector-1 | -> telemetry.sdk.version: Str(1.15.0)
configuration-otl-collector-1 | -> service.name: Str(unknown_service)
configuration-otl-collector-1 | ScopeMetrics #0
configuration-otl-collector-1 | ScopeMetrics SchemaURL:
configuration-otl-collector-1 | InstrumentationScope myapp 0.1.2
configuration-otl-collector-1 | Metric #0
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_histogram
configuration-otl-collector-1 | -> Description: Test histogram
configuration-otl-collector-1 | -> Unit: ms
configuration-otl-collector-1 | -> DataType: Histogram
configuration-otl-collector-1 | -> AggregationTemporality: Cumulative
configuration-otl-collector-1 | HistogramDataPoints #0
configuration-otl-collector-1 | StartTimestamp: 2023-02-22 22:57:17.949824054 +0000 UTC
configuration-otl-collector-1 | Timestamp: 2023-02-22 22:57:18.949918969 +0000 UTC
configuration-otl-collector-1 | Count: 1
configuration-otl-collector-1 | Sum: 12.000000
configuration-otl-collector-1 | Min: 12.000000
configuration-otl-collector-1 | Max: 12.000000
configuration-otl-collector-1 | ExplicitBounds #0: 0.000000
configuration-otl-collector-1 | ExplicitBounds #1: 5.000000
configuration-otl-collector-1 | ExplicitBounds #2: 10.000000
configuration-otl-collector-1 | ExplicitBounds #3: 25.000000
configuration-otl-collector-1 | ExplicitBounds #4: 50.000000
configuration-otl-collector-1 | ExplicitBounds #5: 75.000000
configuration-otl-collector-1 | ExplicitBounds #6: 100.000000
configuration-otl-collector-1 | ExplicitBounds #7: 250.000000
configuration-otl-collector-1 | ExplicitBounds #8: 500.000000
configuration-otl-collector-1 | ExplicitBounds #9: 750.000000
configuration-otl-collector-1 | ExplicitBounds #10: 1000.000000
configuration-otl-collector-1 | ExplicitBounds #11: 2500.000000
configuration-otl-collector-1 | ExplicitBounds #12: 5000.000000
configuration-otl-collector-1 | ExplicitBounds #13: 7500.000000
configuration-otl-collector-1 | ExplicitBounds #14: 10000.000000
configuration-otl-collector-1 | Buckets #0, Count: 0
configuration-otl-collector-1 | Buckets #1, Count: 0
configuration-otl-collector-1 | Buckets #2, Count: 0
configuration-otl-collector-1 | Buckets #3, Count: 1
configuration-otl-collector-1 | Buckets #4, Count: 0
configuration-otl-collector-1 | Buckets #5, Count: 0
configuration-otl-collector-1 | Buckets #6, Count: 0
configuration-otl-collector-1 | Buckets #7, Count: 0
configuration-otl-collector-1 | Buckets #8, Count: 0
configuration-otl-collector-1 | Buckets #9, Count: 0
configuration-otl-collector-1 | Buckets #10, Count: 0
configuration-otl-collector-1 | Buckets #11, Count: 0
configuration-otl-collector-1 | Buckets #12, Count: 0
configuration-otl-collector-1 | Buckets #13, Count: 0
configuration-otl-collector-1 | Buckets #14, Count: 0
configuration-otl-collector-1 | Buckets #15, Count: 0
configuration-otl-collector-1 | {"kind": "exporter", "data_type": "metrics", "name": "logging"}
configuration-otl-collector-1 | 2023-02-22T22:57:18.992Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 2}
configuration-otl-collector-1 | 2023-02-22T22:57:18.992Z info ResourceMetrics #0
configuration-otl-collector-1 | Resource SchemaURL:
configuration-otl-collector-1 | Resource attributes:
configuration-otl-collector-1 | -> telemetry.sdk.language: Str(python)
configuration-otl-collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
configuration-otl-collector-1 | -> telemetry.sdk.version: Str(1.15.0)
configuration-otl-collector-1 | -> service.name: Str(unknown_service)
configuration-otl-collector-1 | ScopeMetrics #0
configuration-otl-collector-1 | ScopeMetrics SchemaURL:
configuration-otl-collector-1 | InstrumentationScope myapp 0.1.2
configuration-otl-collector-1 | Metric #0
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_histogram
configuration-otl-collector-1 | -> Description: Test histogram
configuration-otl-collector-1 | -> Unit: ms
configuration-otl-collector-1 | -> DataType: Histogram
configuration-otl-collector-1 | -> AggregationTemporality: Cumulative
configuration-otl-collector-1 | HistogramDataPoints #0
configuration-otl-collector-1 | StartTimestamp: 2023-02-22 22:57:17.949824054 +0000 UTC
configuration-otl-collector-1 | Timestamp: 2023-02-22 22:57:18.949918969 +0000 UTC
configuration-otl-collector-1 | Count: 1
configuration-otl-collector-1 | Sum: 12.000000
configuration-otl-collector-1 | Min: 12.000000
configuration-otl-collector-1 | Max: 12.000000
configuration-otl-collector-1 | ExplicitBounds #0: 0.000000
configuration-otl-collector-1 | ExplicitBounds #1: 5.000000
configuration-otl-collector-1 | ExplicitBounds #2: 10.000000
configuration-otl-collector-1 | ExplicitBounds #3: 25.000000
configuration-otl-collector-1 | ExplicitBounds #4: 50.000000
configuration-otl-collector-1 | ExplicitBounds #5: 75.000000
configuration-otl-collector-1 | ExplicitBounds #6: 100.000000
configuration-otl-collector-1 | ExplicitBounds #7: 250.000000
configuration-otl-collector-1 | ExplicitBounds #8: 500.000000
configuration-otl-collector-1 | ExplicitBounds #9: 750.000000
configuration-otl-collector-1 | ExplicitBounds #10: 1000.000000
configuration-otl-collector-1 | ExplicitBounds #11: 2500.000000
configuration-otl-collector-1 | ExplicitBounds #12: 5000.000000
configuration-otl-collector-1 | ExplicitBounds #13: 7500.000000
configuration-otl-collector-1 | ExplicitBounds #14: 10000.000000
configuration-otl-collector-1 | Buckets #0, Count: 0
configuration-otl-collector-1 | Buckets #1, Count: 0
configuration-otl-collector-1 | Buckets #2, Count: 0
configuration-otl-collector-1 | Buckets #3, Count: 1
configuration-otl-collector-1 | Buckets #4, Count: 0
configuration-otl-collector-1 | Buckets #5, Count: 0
configuration-otl-collector-1 | Buckets #6, Count: 0
configuration-otl-collector-1 | Buckets #7, Count: 0
configuration-otl-collector-1 | Buckets #8, Count: 0
configuration-otl-collector-1 | Buckets #9, Count: 0
configuration-otl-collector-1 | Buckets #10, Count: 0
configuration-otl-collector-1 | Buckets #11, Count: 0
configuration-otl-collector-1 | Buckets #12, Count: 0
configuration-otl-collector-1 | Buckets #13, Count: 0
configuration-otl-collector-1 | Buckets #14, Count: 0
configuration-otl-collector-1 | Buckets #15, Count: 0
configuration-otl-collector-1 | Metric #1
configuration-otl-collector-1 | Descriptor:
configuration-otl-collector-1 | -> Name: py_counter
configuration-otl-collector-1 | -> Description: Test counter
configuration-otl-collector-1 | -> Unit: items
configuration-otl-collector-1 | -> DataType: Sum
configuration-otl-collector-1 | -> IsMonotonic: true
configuration-otl-collector-1 | -> AggregationTemporality: Cumulative
configuration-otl-collector-1 | NumberDataPoints #0
configuration-otl-collector-1 | StartTimestamp: 2023-02-22 22:57:18.951057692 +0000 UTC
configuration-otl-collector-1 | Timestamp: 2023-02-22 22:57:18.991621073 +0000 UTC
configuration-otl-collector-1 | Value: 1
configuration-otl-collector-1 | {"kind": "exporter", "data_type": "metrics", "name": "logging"}
Also ran unit tests to make sure that the change doesn't break anything.