Ory Kratos + Self-Service UI Node

Hi,

I’ve started to look into using a combo of Ory Kratos and Ory Hydra, and put an API Gateway in front of them, so my first step is to try to get to know Ory Kratos. For that reason I’ve installed Kratos and the Self-Service UI node in my Kubernetes cluster. Having spent some time trying to figure out how to configure Kratos and the Self-service node I think I’ve got the configuration kind of sorted out, but when I try to initiate a browser base login flow, I run into trouble. So far I’ve running everything through port-forwarded traffic as I don’t want to expose anything over the Internet yet. That will explain the use of the IP address 127.0.0.1 in the configurations and the description of the issue.

So, the flow is initiated:

Request URL:http://127.0.0.1:4434/self-service/browser/flows/login
Request Method:GET
Remote Address:127.0.0.1:4434
Status Code: 302

Request Headers:

Host: 127.0.0.1:4434
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Cookie: csrf_token=3ZL9ZXyHlxKkO0H6hx3vdzO/PqVA8Hk/FfL1jfmBtzk=
Upgrade-Insecure-Requests: 1

Response Headers:

HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: http://127.0.0.1:4435/auth/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384
Vary: Cookie
Date: Wed, 25 Mar 2020 19:45:04 GMT
Content-Length: 100

So, for the first request, everything seems to be OK, except maybe for a missing Set-Cookie header in the response?

Now, Kratos redirects to the UI:

Request URL:http://127.0.0.1:4435/auth/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384
Request Method:GET
Remote Address:127.0.0.1:4435
Status Code: 500

Request Headers:

Host: 127.0.0.1:4435
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Response Headers:

HTTP/1.1 500 Internal Server Error
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 10761
ETag: W/"2a09-zz7Y0rMegEgU3Fta4fm+mqMd+WQ"
Date: Wed, 25 Mar 2020 19:45:04 GMT
Connection: keep-alive

And the response is:

An error occurred

{
  "response": {
    "statusCode": 403,
    "body": {
      "error": {
        "code": 403,
        "status": "Forbidden",
        "reason": "A request failed due to a missing or invalid csrf_token value.",
        "debug": "The requested action was forbidden",
        "message": "The requested action was forbidden"
      }
    },
    "headers": {
      "content-type": "application/json",
      "set-cookie": [
        "csrf_token=pNoPNQF60VWUyvylguSn7bdxG3055r3oC28LZEDfS2A=; Domain=127.0.0.1; Max-Age=31536000; HttpOnly"
      ],
      "vary": "Cookie",
      "date": "Wed, 25 Mar 2020 19:45:04 GMT",
      "content-length": "210",
      "connection": "close"
    },
    "request": {
      "uri": {
        "protocol": "http:",
        "slashes": true,
        "auth": null,
        "host": "10.0.27.88:4434",
        "port": "4434",
        "hostname": "10.0.27.88",
        "hash": null,
        "search": "?request=0fb8e59d-1d7d-4f54-b509-101540ac7384",
        "query": "request=0fb8e59d-1d7d-4f54-b509-101540ac7384",
        "pathname": "/self-service/browser/flows/requests/login",
        "path": "/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384",
        "href": "http://10.0.27.88:4434/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384"
      },
      "method": "GET",
      "headers": {
        "Accept": "application/json"
      }
    }
  },
  "body": {},
  "statusCode": 403,
  "name": "HttpError"
}

So, apparently the Self-Service UI expects a csrf_token, which is not present in the request?

The logs for the UI node:

kubectl logs ory-kratos-selfservice-ui-node-57b9dc8789-n9zf6 -n ory

> [email protected] serve /usr/src/app
> node lib/index.js

Listening on http://0.0.0.0:4435
HttpError: HTTP request failed
    at Request._callback (/usr/src/app/node_modules/@oryd/kratos-client/dist/api/adminApi.js:299:40)
    at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
    at Request.emit (events.js:321:20)
    at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:321:20)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:427:28)
    at IncomingMessage.emit (events.js:333:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  response: <ref *1> IncomingMessage {
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: [],
      flowing: true,
      ended: true,
      endEmitted: true,
      reading: false,
      sync: true,
      needReadable: false,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: false,
      destroyed: false,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: true,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: false
    },
    readable: false,
    _events: [Object: null prototype] {
      end: [Array],
      close: [Array],
      data: [Function (anonymous)],
      error: [Function (anonymous)]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: null,
      _readableState: [ReadableState],
      readable: true,
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      writable: false,
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [ClientRequest],
      [Symbol(asyncId)]: 15,
      [Symbol(kHandle)]: [TCP],
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0
    },
    httpVersionMajor: 1,
    httpVersionMinor: 1,
    httpVersion: '1.1',
    complete: true,
    headers: {
      'content-type': 'application/json',
      'set-cookie': [Array],
      vary: 'Cookie',
      date: 'Wed, 25 Mar 2020 19:45:04 GMT',
      'content-length': '210',
      connection: 'close'
    },
    rawHeaders: [
      'Content-Type',
      'application/json',
      'Set-Cookie',
      'csrf_token=pNoPNQF60VWUyvylguSn7bdxG3055r3oC28LZEDfS2A=; Domain=127.0.0.1; Max-Age=31536000; HttpOnly',
      'Vary',
      'Cookie',
      'Date',
      'Wed, 25 Mar 2020 19:45:04 GMT',
      'Content-Length',
      '210',
      'Connection',
      'close'
    ],
    trailers: {},
    rawTrailers: [],
    aborted: false,
    upgrade: false,
    url: '',
    method: null,
    statusCode: 403,
    statusMessage: 'Forbidden',
    client: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: null,
      _readableState: [ReadableState],
      readable: true,
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      writable: false,
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [ClientRequest],
      [Symbol(asyncId)]: 15,
      [Symbol(kHandle)]: [TCP],
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0
    },
    _consuming: false,
    _dumped: false,
    req: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [Socket],
      _header: 'GET /self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384 HTTP/1.1\r\n' +
        'Accept: application/json\r\n' +
        'host: 10.0.27.88:4434\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      path: '/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384',
      _ended: true,
      res: [Circular *1],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    request: Request {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      method: 'GET',
      headers: [Object],
      uri: [Url],
      useQuerystring: false,
      callback: [Function (anonymous)],
      readable: true,
      writable: true,
      explicitMethod: true,
      _qs: [Querystring],
      _auth: [Auth],
      _oauth: [OAuth],
      _multipart: [Multipart],
      _redirect: [Redirect],
      _tunnel: [Tunnel],
      setHeader: [Function (anonymous)],
      hasHeader: [Function (anonymous)],
      getHeader: [Function (anonymous)],
      removeHeader: [Function (anonymous)],
      localAddress: undefined,
      pool: {},
      dests: [],
      __isRequestRequest: true,
      _callback: [Function (anonymous)],
      proxy: null,
      tunnel: false,
      setHost: true,
      originalCookieHeader: undefined,
      _disableCookies: true,
      _jar: undefined,
      port: '4434',
      host: '10.0.27.88',
      url: [Url],
      path: '/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384',
      _json: true,
      httpModule: [Object],
      agentClass: [Function],
      agent: [Agent],
      _started: true,
      href: 'http://10.0.27.88:4434/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384',
      req: [ClientRequest],
      ntick: true,
      response: [Circular *1],
      originalHost: '10.0.27.88:4434',
      originalHostHeaderName: 'host',
      responseContent: [Circular *1],
      _destdata: true,
      _ended: true,
      _callbackCalled: true,
      [Symbol(kCapture)]: false
    },
    toJSON: [Function: responseToJSON],
    caseless: Caseless { dict: [Object] },
    body: { error: [Object] },
    [Symbol(kCapture)]: false
  },
  body: LoginRequest {
    active: undefined,
    expiresAt: undefined,
    id: undefined,
    issuedAt: undefined,
    methods: undefined,
    requestUrl: undefined
  },
  statusCode: 403,
  name: 'HttpError'
}
HttpError: HTTP request failed
    at Request._callback (/usr/src/app/node_modules/@oryd/kratos-client/dist/api/adminApi.js:299:40)
    at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
    at Request.emit (events.js:321:20)
    at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:321:20)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:427:28)
    at IncomingMessage.emit (events.js:333:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
GET /auth/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384 500 10761 - 50.248 ms
GET /index.css 304 - - 1.334 ms
GET /typography.css 304 - - 0.922 ms
GET /auth.css 304 - - 0.991 ms
GET /form.css 304 - - 1.379 ms

Kratos config map (relevant parts):

kubectl get cm/ory-kratos -n ory -o yaml
apiVersion: v1
data:
  kratos.yaml: |
    # serve controls the configuration for the http(s) daemon
    serve:
      admin:
        port: 4433
        host: 0.0.0.0
      public:
        port: 4434
        host: 0.0.0.0
    dsn: postgres://postgres:[email protected]:5432/kratos
    log:
      level: debug
    courier:
      smtp:
        connection_uri: smtp://foo:bar@baz/
    urls:
      default_return_to: http://127.0.0.1:4435/
      mfa_ui: http://127.0.0.1:4435/mfa
      login_ui: http://127.0.0.1:4435/auth/login
      profile_ui: http://127.0.0.1:4435/profile
      verify_ui: http://127.0.0.1:4435/verify
      registration_ui: http://127.0.0.1:4435/auth/registration
      self:
        public: http://127.0.0.1:4433
        admin: http://127.0.0.1:4434
      error_ui: http://127.0.0.1:4435/error
      whitelisted_return_to_domains:
        - http://127.0.0.1

And this is the Kratos Pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: ory-kratos
    app.kubernetes.io/instance: ory-kratos
    app.kubernetes.io/name: ory-krato
  name: ory-kratos-5869584dcf-mq4jx
  namespace: ory
spec:
  containers:
  - args:
    - serve
    - --dev
    - --disable-telemetry
    - --config
    - /etc/config/kratos.yaml
    image: oryd/kratos:v0.1.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /health/alive
        port: http-admin
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
    name: ory-kratos
    ports:
    - containerPort: 4433
      hostPort: 4433
      name: http-public
      protocol: TCP
    - containerPort: 4434
      hostPort: 4434
      name: http-admin
      protocol: TCP
    readinessProbe:
      failureThreshold: 5
      httpGet:
        path: /health/ready
        port: http-admin
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/config
      name: ory-kratos-config-volume
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ory-kratos-token-gjhb8
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - kratos
    - migrate
    - sql
    - -e
    - -y
    - -c
    - /etc/config/kratos.yaml
    image: oryd/kratos:v0.1.1
    imagePullPolicy: IfNotPresent
    name: ory-kratos-init
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/config
      name: ory-kratos-config-volume
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ory-kratos-token-gjhb8
      readOnly: true
  nodeName: xxxx.yyy.compute.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
  serviceAccount: ory-kratos
  serviceAccountName: ory-kratos
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: ory-kratos
    name: ory-kratos-config-volume
  - name: ory-kratos-token-gjhb8
    secret:
      defaultMode: 420
      secretName: ory-kratos-token-gjhb8

And finally the UI node Pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: ory-kratos
    app.kubernetes.io/instance: ory-kratos-selfservice-ui-node
    app.kubernetes.io/name: ory-kratos-selfservice-ui-node
  name: ory-kratos-selfservice-ui-node-57b9dc8789-n9zf6
  namespace: ory
spec:
  containers:
  - env:
    - name: PORT
      value: "4435"
    - name: JWKS_URL
      value: http://ory-hydra.ory.svc.cluster.local:4444/.well-known/jwks.json
    - name: KRATOS_ADMIN_URL
      value: http://10.0.27.88:4434
    - name: KRATOS_PUBLIC_URL
      value: http://10.0.27.88:4433
    - name: KRATOS_BROWSER_URL
      value: http://127.0.0.1:4434
    - name: BASE_URL
      value: /
    image: oryd/kratos-selfservice-ui-node:v0.1.1-alpha.1
    imagePullPolicy: IfNotPresent
    name: ory-ory-kratos-selfservice-ui-node
    ports:
    - containerPort: 3000
      hostPort: 4435
      name: http-public
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ory-kratos-token-gjhb8
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-0-23-162.eu-north-1.compute.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
  serviceAccount: ory-kratos
  serviceAccountName: ory-kratos
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: ory-kratos-token-gjhb8
    secret:
      defaultMode: 420
      secretName: ory-kratos-token-gjhb8

So, why is there no csrf_token in the request to the self-service UI node, if this is the problem? Configuration issues?

Any help is greatly appreciated.

Lars

Try to access your self-service via oathkeeper proxy, I think this issue is related to cookies. In my case, set-cookie header is returned from kratos public api (registration endpont).

I’ve figured out what the problem is. We serve the APIs for the login and registration on two ports:

  • Admin
  • Public

The public API here should only be used for applications that run on a mobile app, in the browser, etc. We protect the public API with CSRF (so you need the CSRF cookie) because otherwise it would be possible to scan the public API and maybe leak some important information

Therefore, all you need to do, I think, is to configure your UI to use the ADMIN API Port to fetch the login request!

Hope this helps

By the way: Thank you for this detailed report, that really helped debug the issue in like 1 minute :slight_smile:

Thanks for the reply!

Unfortunately, that didn’t help. I changed the KRATOS_PUBLIC_URL env variable to port 4434 on the UI node, but looking at the logs for kratos pod, it seems that internally it sends traffic to the public endpoint anyway??

kubectl logs ory-kratos-5869584dcf-mq4jx -n ory

time="2020-03-26T11:02:50Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:36118" request=/self-service/browser/flows/login
time="2020-03-26T11:02:50Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:36118" request=/self-service/browser/flows/login status=302 text_status=Found took=7.545526ms
time="2020-03-26T11:02:50Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.12.223:51254" request="/self-service/browser/flows/requests/login?request=d21d516a-1c80-4947-8eb6-04919a9ef7df"
time="2020-03-26T11:02:50Z" level=error msg="An error occurred while handling a request" code=403 debug="The requested action was forbidden" details="map[]" error="The requested action was forbidden" reason="A request failed due to a missing or invalid csrf_token value." request-id= status=403 trace="Stack trace: \ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).fetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:177\ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).publicFetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:156\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:334\ngithub.com/justinas/nosurf.(*CSRFHandler).handleSuccess\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:187\ngithub.com/justinas/nosurf.(*CSRFHandler).ServeHTTP\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:144\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:140\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/gorilla/context.ClearHandler.func1\n\t/go/pkg/mod/github.com/gorilla/[email protected]/context.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" writer=JSON
time="2020-03-26T11:02:50Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.12.223:51254" request="/self-service/browser/flows/requests/login?request=d21d516a-1c80-4947-8eb6-04919a9ef7df" status=403 text_status=Forbidden took=2.376825ms

I also noticed that wis this configuration change, I received another cookie than before:

The request to http://127.0.0.1:4435/auth/login?request=d21d516a-1c80-4947-8eb6-04919a9ef7df, had a request header Set-Cookie set with the value pga4_session=e12ce1f9-cfaf-4dab-a8c1-2cba3479ba7d!REKWap3SWls0e7F/HVxn9RugsjM=

Lars

Hm yeah, my bad, it seems like you’ve set it up correctly. I also can see in your log that it’s making the call to 4434 which is the admin endpoint.

Can you revert these changes, redo the authentication flow again, and then post the logs of kratos here as well?

What’s confusing me a bit is the following:

- name: KRATOS_ADMIN_URL
  value: http://10.0.27.88:4434
- name: KRATOS_PUBLIC_URL
  value: http://10.0.27.88:4433
- name: KRATOS_BROWSER_URL
  value: http://127.0.0.1:4434

The Browser URL should the the URL that the browser uses to access Kratos (so here it’s correctly 127.0.0.1). However, shouldn’t that be the 4433 port?

I don’t know about the port. I could try changing it to 4433 if you think it makes sense.

Here are the logs, without chaning that port number:

time="2020-03-26T11:23:08Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login
time="2020-03-26T11:23:08Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login status=302 text_status=Found took=6.144167ms
time="2020-03-26T11:23:08Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:59048" request="/self-service/browser/flows/requests/login?request=21b8a110-ba8f-4ee4-bc9e-3f27f7788209"
time="2020-03-26T11:23:08Z" level=error msg="An error occurred while handling a request" code=403 debug="The requested action was forbidden" details="map[]" error="The requested action was forbidden" reason="A request failed due to a missing or invalid csrf_token value." request-id= status=403 trace="Stack trace: \ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).fetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:177\ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).publicFetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:156\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:334\ngithub.com/justinas/nosurf.(*CSRFHandler).handleSuccess\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:187\ngithub.com/justinas/nosurf.(*CSRFHandler).ServeHTTP\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:144\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:140\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/gorilla/context.ClearHandler.func1\n\t/go/pkg/mod/github.com/gorilla/[email protected]/context.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" writer=JSON
time="2020-03-26T11:23:08Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:59048" request="/self-service/browser/flows/requests/login?request=21b8a110-ba8f-4ee4-bc9e-3f27f7788209" status=403 text_status=Forbidden took=1.43911ms
time="2020-03-26T11:23:26Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login
time="2020-03-26T11:23:26Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login status=302 text_status=Found took=5.223461ms
time="2020-03-26T11:23:27Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:36042" request="/self-service/browser/flows/requests/login?request=67cea9a5-739f-48fe-b906-1788fe95d151"
time="2020-03-26T11:23:27Z" level=error msg="An error occurred while handling a request" code=403 debug="The requested action was forbidden" details="map[]" error="The requested action was forbidden" reason="A request failed due to a missing or invalid csrf_token value." request-id= status=403 trace="Stack trace: \ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).fetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:177\ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).publicFetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:156\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:334\ngithub.com/justinas/nosurf.(*CSRFHandler).handleSuccess\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:187\ngithub.com/justinas/nosurf.(*CSRFHandler).ServeHTTP\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:144\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:140\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/gorilla/context.ClearHandler.func1\n\t/go/pkg/mod/github.com/gorilla/[email protected]/context.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" writer=JSON
time="2020-03-26T11:23:27Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:36042" request="/self-service/browser/flows/requests/login?request=67cea9a5-739f-48fe-b906-1788fe95d151" status=403 text_status=Forbidden took=1.403548ms

Ok that’s strange, so the logs say:

time="2020-03-26T11:23:08Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login
time="2020-03-26T11:23:08Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="127.0.0.1:57506" request=/self-service/browser/flows/login status=302 text_status=Found took=6.144167ms
time="2020-03-26T11:23:08Z" level=info msg="started handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:59048" request="/self-service/browser/flows/requests/login?request=21b8a110-ba8f-4ee4-bc9e-3f27f7788209"
time="2020-03-26T11:23:08Z" level=error msg="An error occurred while handling a request" code=403 debug="The requested action was forbidden" details="map[]" error="The requested action was forbidden" reason="A request failed due to a missing or invalid csrf_token value." request-id= status=403 trace="Stack trace: \ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).fetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:177\ngithub.com/ory/kratos/selfservice/flow/login.(*Handler).publicFetchLoginRequest\n\t/home/ory/selfservice/flow/login/handler.go:156\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:334\ngithub.com/justinas/nosurf.(*CSRFHandler).handleSuccess\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:187\ngithub.com/justinas/nosurf.(*CSRFHandler).ServeHTTP\n\t/go/pkg/mod/github.com/justinas/[email protected]/handler.go:144\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:140\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/gorilla/context.ClearHandler.func1\n\t/go/pkg/mod/github.com/gorilla/[email protected]/context.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" writer=JSON
time="2020-03-26T11:23:08Z" level=info msg="completed handling request" method=GET name="public#http://127.0.0.1:4433" remote="10.0.19.115:59048" request="/self-service/browser/flows/requests/login?request=21b8a110-ba8f-4ee4-bc9e-3f27f7788209" status=403 text_status=Forbidden took=1.43911ms

I’m a bit confused, because this call:

public#http://127.0.0.1:4433/self-service/browser/flows/requests/login?request=21b8a110-ba8f-4ee4-bc9e-3f27f7788209

is the one coming from your UI, right? That is the problem though, this should be coming through the admin endpoint.

But your login logs say:

    "href": "http://10.0.27.88:4434/self-service/browser/flows/requests/login?request=0fb8e59d-1d7d-4f54-b509-101540ac7384"

Are you doing some manual things requests here?

Ohhh, I see - you’re using 4433 for admin and 4434 for public! We use 4433 for public and 4434 for admin per default which is why I was confused.

So as you can see, your config is then not correct:

- name: KRATOS_ADMIN_URL
  value: http://10.0.27.88:4434
- name: KRATOS_PUBLIC_URL
  value: http://10.0.27.88:4433
- name: KRATOS_BROWSER_URL
  value: http://127.0.0.1:4434

ADMIN should be on port 4433 and public on 4434 - once you’ve changed that it should work as you expect it to. You can leave the browser URL at 4434.

You should probably also change this:

- containerPort: 4433
  hostPort: 4433
  name: http-public
  protocol: TCP
- containerPort: 4434
  hostPort: 4434
  name: http-admin
  protocol: TCP

Ahh, so the real problem is that I configured Kratos to listen on non-standard ports! Thanks. I will change this and try again.

Or alternatively change

serve:
  admin:
    port: 4433
    host: 0.0.0.0
  public:
    port: 4434
    host: 0.0.0.0

to

serve:
  admin:
    port: 4434
    host: 0.0.0.0
  public:
    port: 4433
    host: 0.0.0.0

Yeah - I mean you can also just leave the port and host definitions empty because it defaults to those values anyway :slight_smile:

Success! Now it’s working and I get the login screen. Let’s hope the rest of the flow works as well.

Thanks a lot!

Lars

Cool, glad to be of help!

I quickly tried the registration flow as well, and faced another problem. Still something wrong in the configuration, perhaps? Anyway, here’s the trace:

curl -v http://127.0.0.1:4433/self-service/browser/flows/registration
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 4433 (#0)
> GET /self-service/browser/flows/registration HTTP/1.1
> Host: 127.0.0.1:4433
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: http://127.0.0.1:4435/error?error=10992351-4540-4e47-8b2c-5bd49eda99a3
< Set-Cookie: csrf_token=xG9cvMBHiEPEj+CkoB2Wd9frpu6Ul9qoN6oqH789Gk0=; Domain=127.0.0.1; Max-Age=31536000; HttpOnly
< Vary: Cookie
< Date: Thu, 26 Mar 2020 12:05:09 GMT
< Content-Length: 93
<
<a href="http://127.0.0.1:4435/error?error=10992351-4540-4e47-8b2c-5bd49eda99a3">Found</a>.


curl -v http://127.0.0.1:4433/self-service/errors?id=10992351-4540-4e47-8b2c-5bd49eda99a3
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 4433 (#0)
> GET /self-service/errors?id=10992351-4540-4e47-8b2c-5bd49eda99a3 HTTP/1.1
> Host: 127.0.0.1:4433
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< Set-Cookie: csrf_token=kLfrAKII/ehUd0O81h9/R1E9fFWt76jOejOxOMu7kWs=; Domain=127.0.0.1; Max-Age=31536000; HttpOnly
< Vary: Cookie
< Date: Thu, 26 Mar 2020 12:06:59 GMT
< Content-Length: 205
<
{"error":{"code":403,"status":"Forbidden","reason":"A request failed due to a missing or invalid csrf_token value.","debug":"Unable to locate the resource","message":"The requested action was forbidden"}}
* Connection #0 to host 127.0.0.1 left intact

According to the documentation, no authentication is required for the registration flow, but it seems as if Kratos is expecting the csrf_token anyway. Why would this be different than the login flow?

urls.registration_ui is set to http://127.0.0.1:4435/auth/registration in Kratos config map.

Same issue as before, error APIs are served both over public and admin endpoints, you’re using the public endpoint I suppose? I believe this is documented, if not we need to fix that

Yes, but the problem here is that the first call to the registration endpoint returns Location: http:///127.0.0.1:4435/error?error=10992351-4540-4e47-8b2c-5bd49eda99a3, which is the error_ui url configured for Kratos. This indicates that Kratos rejected the call to the public registration endpoint (4433). At that point, the error endpoint hasn’t been invoked yet?

what I mean is that

curl -v http://127.0.0.1:4433/self-service/errors?id=10992351-4540-4e47-8b2c-5bd49eda99a3

should be

curl -v http://127.0.0.1:4434/self-service/errors?id=10992351-4540-4e47-8b2c-5bd49eda99a3

then it will work