We (RenkuLab) cannot always own all the resources that are needed for various use-cases. RenkuLab deployments are complex to manage, they require administrators to manage multiple services as Postgres, Keycloak and Gitlab on top of Renkulab itself. At the moment, to give users access to another set of compute resources, we have to deploy an entire new Renku instance with all of these services.
We want to be able to provide additional, elastic resources provisioned from potentially other cloud providers from a central instance, with a minimal set of services required in the “satellite” resource.
We previously described this problem in Exploration: Remote resources and a PoC was made in ‣. The full implementation was blocked by a refactoring in some of the backend components. This is now complete and we can proceed with creating a resource pool that uses remote resources.
6 weeks
Based on ‣.
The configuration should be done by Renku admins, ideally via the existing admin page, but if that is not possible in the first implementation it’s fine to configure these just via the API.
There are two obvious options for extending the current Resource Pool concept to support this functionality. One is to create (reusable) “external compute providers”, similar to the concept of external OAuth services we already have for git repositories. The other option is to simply add this same configuration to each Resource Pool that needs it, which might be simpler to do but more difficult to maintain.
One possibility is to follow the same pattern as we have for OAuth git services, i.e. introduce the notion of an “external compute provider” so that it can be reused for multiple resource pools. The only thing really needed in the configuration should include:
On resource pool creation, the admin can then link the resource pool to this external compute provider.
This option requires the same metadata as above but adds this to each external resource pool instead of having a new entity linked to the configuration.