Oathkeeper (v0.14.2_oryOS.10) returning empty reply on slow/long distance database calls

I have Oathkeeper running in an East US kubernetes cluster, and the DB is currently living in West US. There’s about a 70ms lag between the two sites.

When running the following curl in the same k8s cluster as oathkeeper, I’m seeing the following

curl:
root@shellpod-bd-55c5b5cc74-nlmrh:/# curl “http://oathkeeper-api.hydra.svc.cluster.local:4456/rules?limit=50000&offset=0
curl: (52) Empty reply from server

[oathkeeper-api-7f4cc5cb9f-dbmr6] time=“2019-04-24T18:50:58Z” level=info msg=“started handling request” method=GET remote=“10.39.0.47:60820” request="/rules?limit=50000&offset=0"
[oathkeeper-api-7f4cc5cb9f-dbmr6] time=“2019-04-24T18:51:24Z” level=info msg=“completed handling request” measure#oathkeeper-api.latency=26073727174 method=GET remote=“10.39.0.47:60820” request="/rules?limit=50000&offset=0" status=200 text_status=OK took=26.073727174s

Running what I believe to be the same query from the postgres logs and the code (https://github.com/ory/oathkeeper/blob/master/rule/manager_sql.go#L79) from the same pod issuing the curl requests returns within a few ms.

I’m definitely seeing queries from that pod, and connections using the username configured for oathkeeper to the db.

Any pointers?

Hm interesting, how many rules do you have in the DB?

Oh and please move this over to GitHub as it appears to be an issue

Thanks!