It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
While it may intuitively may look like there is a large overlap in the three areas there is suprisingly little, and for the few parts there are (e.g. trace <-> log correlation), OTEL does offer a standard.
There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And metrics are something else altogether. Trying to somehow unify them would be a prime example of "wrong abstraction".
> “The next time something like this occurs again, please save me a trace.”
The building blocks for this exist. The observability platform must simply (haha) implement the pattern detectors and use them for sampling decisions.
https://docs.micrometer.io/micrometer/reference/observation....
A metric is a point in time. A metric is very small but you have a lot of them.
A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc.
A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation.
How do you decide when to suddenly do the trace and send it? IF you always do the trace, you have to pay for the overhead of that tracing constantly.
A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.
All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".
The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.
In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.
Historically, logging and metrics have been different problem domains with different implementations for ages.
Now to your point: Note that tracing does get the most of love, and that it does include constructs to add logging and metrics into these traces (spans actually). So you could argue that they are trying to develop a single interface.
> “The next time something like this occurs again, please save me a trace.”
Well, if you want this you either need to propagate this predicate to all points that might be involved, or always emit all traces and have the predicate included in the filter. And then you need to be able to dynamically propagate this predicate from the system/ui where you click to where you filter.
This is one of the reasons why we always propagate and emit traces and just post filter it in processing before it lands in the persistence layer.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Grafana went the other way - your app exports all labels, and then you selectively aggregate on ingest: https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
I wish the observability vendors would move to using it under the covers so it's easier to mix and match.
I wish the otel support wasn't super buggy in most of the frameworks and backends.
If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.
In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog).
We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.
It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.
> In a runtime like Lambda, how would you avoid the need to run an edge collector?
Here's my setup (in Go, very simplified):
> // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed
My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation.
There is no in-between where I can inject the functionality required in a way that is compatible with the application.
If I recall the primary issue was the forced loading of the django settings file by otel.
I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications.
But most production django applications are monoliths and generally larger apps. They have non-trivial configuration processes which are often multi step and source settings from multiple places.
Otel should not assume it can just randomly load a the django settings at an arbitrary time point in the startup process.
In one of our apps the MIDDLEWARE setting specifically is dynamically generated and re-ordered based on enabled features. That application's startup process also has multiple stages and the initialisation of django occurs much later, after dependant config loaders etc have been initialised.
What would allow us to integrate with opentelemetry-instrumentation-django much more easily is a set of smaller primitives that we can configure and call at the appropriate time.
opentelemetry-instrumentation-django has (had?) a lot of logic hidden inside a large "inject" function which could not easily be extracted into the constituent parts and applied in a compatible manner.
https://github.com/open-telemetry/opentelemetry-python-contr...
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.
A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.
I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.
> A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards
By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
Kinda like people hating Obamacare but loving the ACA.
Jaeger does not implement all the OTEL features, though. It's specifically focused on traces rather than metrics.
1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?
2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?
If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.