Up to now, we have taken the position that Renku connects compute and storage infrastructures, but does not offer storage itself. We have not wanted to get into the business of being a storage provider, because it’s a messy business. But, for a lot of people, Renku not coming with storage out of the box is a major annoyance or even an all out blocker. For example, we’ve had the experience of telling course instructors that they can have their students connect their Switchdrive or Polybox accounts for Renku storage, but the instructors say that that’s too big of an ask/too easy for the students to mess up/too much of a hassle. A lot of people want Renku to provide persistent storage so they don’t have to figure it out on their own.
Relatedly, Renku users need performant storage. Renku data connectors mount data into the session, but this is not performant. When people want performance, we tell them to copy their data into the session workspace. This puts the data next to the compute, and persists through pausing and resuming the session. However, its not obvious to the user to do this, and it’s annoying that you have to re-copy the data if your session is shut down.
6 weeks
The idea here is to offer project storage, which is a performant volume dedicated to your project. Project owners can claim 1 project storage per project. Once created, the project storage is available in all sessions (and jobs) launched from the project, by anyone (except anonymous sessions). The project storage can be accessed simultaneously from multiple sessions.
Users go to add Data Connector → new tab for Create project storage → big button to claim project storage
An idea for how creating a project storage could look