Jaeger tracing in Ory Keto and other Ory components


We have a microservice that invokes Ory Keto to check if a specific request is allowed. According to the documentation, you can then use the following configuration:

  1. Under CORS configuration, I added

         - x-b3-spanid
         - x-b3-traceid
         - x-b3-sampled
         - x-b3-parentspanid
  2. Under jaeger configuration:

           local_agent_address: <jaeger-agent>l:6831
           propagation: b3
             server_url: http:/<jaeger-agent>:5778/sampling
             type: const
             value: 1
       service_name: ORY Keto
  3. In my client, I add the following http headers, and the below is a dump of the http request:

"POST /engines/acp/ory/glob/allowed HTTP/1.1\r\nHost: <host>:4456\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 84\r\nAccept: application/json\r\nContent-Type: application/json\r\nX-B3-Sampled: 1\r\nX-B3-Spanid: 53a25bd3c253a200\r\nX-B3-Traceid: 61fcfc22185e56ed329a86f6fc233516\r\nx-b3-parentspanid: 53a25bd3c253a200\r\nAccept-Encoding: gzip\r\n\r\n{\"action\":\"read\",\"resource\":\"devicemgmt:instruments\",\"subject\":\"[email protected]\"}

But, looking in the Jaeger UI, I do get a span for /engines/acp/ory/glob/allowed, but it’s not a child span of the span propagated as http headers. What could be the issue here?



Hm it is possible that this is not properly being propagated. Another problem could be that the ory/x dependency in ory keto is not up to date - it might have been fixed there already!

I’m guessing it’s the first option. I tried to do the same thing as our microservice does but with curl and Keto configured to use debug level, and in the logs all I can see is:

time=2020-11-02T21:32:03Z level=info msg=started handling request http_request=map[headers:map[accept:application/json origin:https://localhost user-agent:curl/7.55.1] host:localhost:4456 method:POST path:/engines/acp/ory/glob/allowed query:<nil> remote: scheme:https]

So, none of the headers are present here. Are you supposed to see all the headers in the log, or are they being filtered out intentionally? Where in the code should I look for this? I’m thinking of building a new image and put some trace statements to be able to see all the headers that are coming in the http request.

The second option, regarding the ory/x dependency is most likely not the problem, as far as I can see, as the b3 propagation feature was added July 7th 2019 and the image we are using was built way later than that. I can’t see any other commits having been made to the tracing package that could be relevant after that date, unless I should look in another package?

New update:

So I built a new Keto image, this time with some trace statements in the middleware.go file of the ory/x/tracing package. The headers seem to be propgated to Keto alright. A dump of the received http request looks like this:

"POST /engines/acp/ory/glob/allowed HTTP/1.1\r\nHost:\r\nAccept: application/json\r\nAccept-Encoding: gzip\r\nContent-Length: 84\r\nContent-Type: application/json\r\nUser-Agent: Go-http-client/1.1\r\nX-B3-Sampled: 1\r\nX-B3-Spanid: afe7c6d6e065bed9\r\nX-B3-Traceid: eafa2ceb6b0966c431c349284729268c\r\n\r\n{\"action\":\"read\",\"resource\":\"devicemgmt:instruments\",\"subject\":\"[email protected]\"}\n"

But, the code below returns an error: opentracing: SpanContext not found in Extract carrier

remoteContext, err := opentracing.GlobalTracer().Extract(

Problem solved:

It turned out to be a bug in Ory Keto. The propagation configuration is not being picked up:

The following line should be added:

Propagation: viperx.GetString(v.l, "tracing.providers.jaeger.propagation", "", "TRACING_PROVIDER_JAEGER_PROPAGATION"),

Will provide a PR for this bug.

1 Like

Thank you! PR is merged :slight_smile: