Jaeger tracing in Ory Keto and other Ory components

Other Open Source
#1

Hi,

We have a microservice that invokes Ory Keto to check if a specific request is allowed. According to the documentation, you can then use the following configuration:

  1. Under CORS configuration, I added

     allowed_headers:
     - x-b3-spanid
     - x-b3-traceid
     - x-b3-sampled
     - x-b3-parentspanid

  2. Under jaeger configuration:

       jaeger:
       local_agent_address: <jaeger-agent>l:6831
       propagation: b3
       sampling:
         server_url: http:/<jaeger-agent>:5778/sampling
         type: const
         value: 1
   service_name: ORY Keto

  3. In my client, I add the following http headers, and the below is a dump of the http request:

"POST /engines/acp/ory/glob/allowed HTTP/1.1\r\nHost: <host>:4456\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 84\r\nAccept: application/json\r\nContent-Type: application/json\r\nX-B3-Sampled: 1\r\nX-B3-Spanid: 53a25bd3c253a200\r\nX-B3-Traceid: 61fcfc22185e56ed329a86f6fc233516\r\nx-b3-parentspanid: 53a25bd3c253a200\r\nAccept-Encoding: gzip\r\n\r\n{\"action\":\"read\",\"resource\":\"devicemgmt:instruments\",\"subject\":\"[email protected]\"}

But, looking in the Jaeger UI, I do get a span for /engines/acp/ory/glob/allowed, but it’s not a child span of the span propagated as http headers. What could be the issue here?

Regards

Lars

#2

Hm it is possible that this is not properly being propagated. Another problem could be that the ory/x dependency in ory keto is not up to date - it might have been fixed there already!

#3

I’m guessing it’s the first option. I tried to do the same thing as our microservice does but with curl and Keto configured to use debug level, and in the logs all I can see is:

time=2020-11-02T21:32:03Z level=info msg=started handling request http_request=map[headers:map[accept:application/json origin:https://localhost user-agent:curl/7.55.1] host:localhost:4456 method:POST path:/engines/acp/ory/glob/allowed query:<nil> remote:127.0.0.1:55514 scheme:https]

So, none of the headers are present here. Are you supposed to see all the headers in the log, or are they being filtered out intentionally? Where in the code should I look for this? I’m thinking of building a new image and put some trace statements to be able to see all the headers that are coming in the http request.

The second option, regarding the ory/x dependency is most likely not the problem, as far as I can see, as the b3 propagation feature was added July 7th 2019 and the image we are using was built way later than that. I can’t see any other commits having been made to the tracing package that could be relevant after that date, unless I should look in another package?

#4

New update:

So I built a new Keto image, this time with some trace statements in the middleware.go file of the ory/x/tracing package. The headers seem to be propgated to Keto alright. A dump of the received http request looks like this:

"POST /engines/acp/ory/glob/allowed HTTP/1.1\r\nHost: 10.0.9.133:4456\r\nAccept: application/json\r\nAccept-Encoding: gzip\r\nContent-Length: 84\r\nContent-Type: application/json\r\nUser-Agent: Go-http-client/1.1\r\nX-B3-Sampled: 1\r\nX-B3-Spanid: afe7c6d6e065bed9\r\nX-B3-Traceid: eafa2ceb6b0966c431c349284729268c\r\n\r\n{\"action\":\"read\",\"resource\":\"devicemgmt:instruments\",\"subject\":\"[email protected]\"}\n"

But, the code below returns an error: opentracing: SpanContext not found in Extract carrier

remoteContext, err := opentracing.GlobalTracer().Extract(
  opentracing.HTTPHeaders,
  opentracing.HTTPHeadersCarrier(r.Header),
)
#5

Problem solved:

It turned out to be a bug in Ory Keto. The propagation configuration is not being picked up:

The following line should be added:

Propagation: viperx.GetString(v.l, "tracing.providers.jaeger.propagation", "", "TRACING_PROVIDER_JAEGER_PROPAGATION"),

Will provide a PR for this bug.

1 Like
#6

Thank you! PR is merged :slight_smile: