High Availability and Scaling


#1

Hi,

I’m trying to find info on scaling and High Availability for (atm) Hydra. The main website suggests that Hydra is designed to scale, but this seems more to be focused on the feature/requirements side than on workload side.

Think I remember reading that Hydra scales well with the resources you throw at it, but nowhere did I find any more specific numbers/guidelines.

So as for scaling, am I correct that the way to scale Hydra is by vertical scaling or can Hydra also be scaled horizontally, by clustering multiple Hydra instances?

Also, could not find any info on High Availability strategies for Hydra? Can Hydra instances be clustered? Or can you just run multiple instances against the same database?

Some info/guidelines would be appreciated.

Some background from our side: we’re contemplating using Hydra in a Kubernetes environment to handle the authentication on the api of our saas application


#2

Hydra scales with the datastore. As such, you make it HA if the datastore is HA (eg CloudSpanner, CockroachDB, …). There is no need for an etcd or memcached setup or anything like that. You make it x-region if the datastore is x-region.

Clustering works, and both horizontal and vertical scaling as well.


#3

Tnx for the response.

So, to make sure I 100% understand: Hydra Ory itself is completely stateless, all state is stored in the DB, so you can run as many instances of Hydra you want in parallel, all connecting to the same DB without issues, as long as your DB can handle it?

Might be worthwhile to add a little something about this to the docs…


#4

That’s it! I think we had a section about this way back but it seems to have been lost. A proper place for this would probably be “ORY Hydra in Production” under a new section “Scaling”.

By the way, we might add support for K8S storage at some point, or maybe some other datastore, depending on availability and popularity. But for now, SQL should be enough to handle most workloads!


#5

I saw that hydra has a support of storage plugins. Does it allow to have any storage implementation for customers who want to have different storage (or using multiples storages for different data)?

I saw plugin implementation for oracle database in ory github. Do you have any other examples of storage plugins?


#6

Excellent! Yeah, that’s the place I was looking for it :grinning: Want me to create an issue for adding this?

We happen to be using PostgreSQL already atm, so Hydra fits right in, but being able to plugin in other storage providers would be handy. For example if you’d happen to run on GCP and you operate globally, Google Cloud Spanner might be an interesting datastore for Hydra


#7

No docs on that unfortunately, but it’s possible to switch out the DBAL driver with a plugin without recompiling the whole project. Not sure with SQL dialect Cloud Spanner uses but if it’s PG or MySQL compatible it shouldn’t be a problem.

The OracleDB adapter is really old, for versions 0.9.x. Some things changed since then. My advice (always) is to not implement your own storage adapter. It will just lead to you not upgrading because you have to update the store first and write migrations, and it’s just a ton of work for something that is usually unjustifiable. Performance of PG and MySQL is stellar and you can handle immense traffic with that. If you’re on a scale of Google (3.5 billion searches per day) then yeah, this won’t work. But if you’re on a scale of regular companies then SQL is just enough.


#8

I saw your benchmarks on in-memory storage here: https://www.ory.sh/docs/guides/master/performance/1-hydra

It’s really cool to have performance number but it’s in-memory and show only computation overhead and doesn’t cover communication between sql storage <-> hydra. At least some numbers from your experience or existing success stories.

Just idea that It will be cool to have benchmarks with sql storage as well to show how much we can get without worrying about storage. If we have several million check token requests should we start worrying about custom storage?


#9

I think this is very well covered in the docs:

We do not include benchmarks against databases (e.g. MySQL or PostgreSQL) as the performance greatly differs between deployments (e.g. request latency, database configuration) and tweaking individual things may greatly improve performance. We believe, for that reason, that benchmark results for these database adapters are difficult to generalize and potentially deceiving. They are thus not included.

I still see the same issue, especially when you consider latency between two things (hydra, sql) as performance. Unless you have a way to get unbiased results I don’t think that will change quickly. Since the benchmarks themselves are open source and in the repository, you can quickly replicate them with a SQL store if you want.

I know of some companies that have about 1m requests per day that easily handle this with a small or medium RDS instance.


#10

But in general, we can not and do not replace your internal testing. You yourself have to see if the technology fits for your purposes. If the features are what you need but you worry about performance - do testings that replicate your environment. The whole product suite is already freely available, we can’t do everything for you :slight_smile:


#11

FYI - I wrote a Google Cloud Datastore plugin here: https://github.com/someone1/hydra-gcp - still tinkering with swapping the signing mechanism but you can compile a plugin from the /plugin subpackage

Benchmarks are pretty useless as aforementioned in this thread, but for the sake of relative performance I did run a few for comparison: https://github.com/someone1/hydra-gcp/tree/master/benchmarks

Some notes I did notice: hydra is very much CPU bound vs memory - more cores/speed will dramatically increase performance - especially due to bcrypt hashing.