Oathkeeper (v0.14.2_oryOS.10) returning empty reply on slow/long distance database calls


#1

I have Oathkeeper running in an East US kubernetes cluster, and the DB is currently living in West US. There’s about a 70ms lag between the two sites.

When running the following curl in the same k8s cluster as oathkeeper, I’m seeing the following

curl:
[email protected]:/# curl “http://oathkeeper-api.hydra.svc.cluster.local:4456/rules?limit=50000&offset=0
curl: (52) Empty reply from server

[oathkeeper-api-7f4cc5cb9f-dbmr6] time=“2019-04-24T18:50:58Z” level=info msg=“started handling request” method=GET remote=“10.39.0.47:60820” request="/rules?limit=50000&offset=0"
[oathkeeper-api-7f4cc5cb9f-dbmr6] time=“2019-04-24T18:51:24Z” level=info msg=“completed handling request” measure#oathkeeper-api.latency=26073727174 method=GET remote=“10.39.0.47:60820” request="/rules?limit=50000&offset=0" status=200 text_status=OK took=26.073727174s

Running what I believe to be the same query from the postgres logs and the code (https://github.com/ory/oathkeeper/blob/master/rule/manager_sql.go#L79) from the same pod issuing the curl requests returns within a few ms.

I’m definitely seeing queries from that pod, and connections using the username configured for oathkeeper to the db.

Any pointers?


#2

Hm interesting, how many rules do you have in the DB?


#3

Oh and please move this over to GitHub as it appears to be an issue


#4

Thanks!