🤔 Problem

In Renku 2.0 users can connect to cloud data sources but they cannot easily import and work with a published dataset. It is also not possible to find uses of published datasets in the platform.

Projects requesting this functionality:

🍴 Appetite

6 weeks

🎯 Solution

Renku should allow users to import datasets with DOIs from selected providers. In this feature build, we will only support Zenodo.

<aside> 💡

Note on DOIs:

Not all DOIs mean the same thing. Strictly speaking, a DOI simply has to resolve to a resource on the web, i.e. it is a URI that is guaranteed to be resolvable for some extended period of time. Different services might interpret this resolution differently; in Zenodo, there is a top-level DOI for the resource and additional DOIs for each version. The top-level DOI always resolves to the most recent version. This behavior is not standard and will depend on the service - ❓we need to make some decisions about how to treat these special cases.

Options:

Only allow users to use DOIs that point to specific versions
If DOI is “latest” check whether we need to fetch a new one </aside>

DOI Data Connectors need to:

show up in the data section of the project page
show up in search
show which other projects they are used in
be public
(they may not need a namespace!) </aside>

Data

It is up to the build team to decide whether an RClone extension for Zenodo is doable this build, otherwise a simpler implementation of copying data to a s3 bucket is acceptable.

</aside>

Mount using RClone

→ implement RClone provider
- ask RClone community about this, forum, would they accept it? Has anyone considered this before? → https://forum.rclone.org/t/zenodo-and-invenio-as-a-provider/50507
- can also be used locally with the rclone client
- never copy the entire dataset