We want to enable mounting of SciCat datasets into Renku. Similarly to Envidat. We also want to be able to export and create SciCat datasets from Renku.
6 weeks
Potentially this will expand to 2 builds
SciCat has to push the system that resolves the doi to json-ld to production. This will happen before the build starts.
Once we have a doi we resolve this to the schema.org json-ld that describes the dataset
DOI resolution steps:
curl -I 'Accept: application/ld+json' [<https://doi.org/10.16907%2Fcdcbc412-d7bc-41ea-82f5-32922ddfa0fd>](<https://doi.org/10.16907%2Fcdcbc412-d7bc-41ea-82f5-32922ddfa0fd>)https://doi.psi.ch/detail/10.16907%2Fcdcbc412-d7bc-41ea-82f5-32922ddfa0fdcurl -H 'Accept: application/ld+json' [<https://doi.psi.ch/detail/10.16907%2Fcdcbc412-d7bc-41ea-82f5-32922ddfa0fd>](<https://doi.psi.ch/detail/10.16907%2Fcdcbc412-d7bc-41ea-82f5-32922ddfa0fd>)https://scicat-exporter.development.psi.ch/api/v1/zenodo/10.16907%2Feb75dfba-6c91-4276-8dd6-05f949307213/exportcurl -H "Accept: application/ld+json" [<https://scicat-exporter.development.psi.ch/api/v1/zenodo/10.16907%2Feb75dfba-6c91-4276-8dd6-05f949307213/export>](<https://scicat-exporter.development.psi.ch/api/v1/zenodo/10.16907%2Feb75dfba-6c91-4276-8dd6-05f949307213/export>)Steps 4 and 5 are not pushed to production and the development server is not accessible outside of psi.
Example json-ld payload (at the end of all the redirects):
{
"@id": "<https://doi.org/10.16907/eb75dfba-6c91-4276-8dd6-05f949307213>",
"@type": "Dataset",
"creator": {
"@type": "Person",
"name": "huesser"
},
"dateCreated": "2023-03-10T14:30:38.016Z",
"datePublished": "2023-03-10T14:30:39.900Z",
"description": "First data to public URLs",
"distribution": [
{
"@type": "DataDownload",
"contentUrl": "<https://rgw.cscs.ch/psi_test:psi-public-tar-dev/20.500.11935/eeae04f3-c4f3-4487-bbb3-500acfa989ee_0_2022-10-20-10-25-37.tar>",
"encodingFormat": "application/x-tar",
"expires": "2026-03-31T11:46:31.486Z"
},
{
"@type": "DataDownload",
"contentUrl": "<https://rgw.cscs.ch/firecrest_hpc%3Anoderedd/8414927a-55cb-4b03-8ed5-3af195fe0524/0e54729b-75c5-42fa-a628-aae5dc3f3dae_0_2022-09-08-14-52-32.tar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=S82RBBK66XUCNDL3NGXD%2F20260211%2Fcscs-zonegroup%2Fs3%2Faws4_request&X-Amz-Date=20260211T093601Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=422a4f7e759cf51c99459fa32596baec37c1064fd6c5c9900cc488c80ece097a>",
"encodingFormat": "application/x-tar",
"expires": "2026-02-18T09:36:01Z"
}
],
"expires": "2026-02-18T09:36:01Z",
"identifier": "<https://doi.org/10.16907/eb75dfba-6c91-4276-8dd6-05f949307213>",
"name": "First data to public URLs",
"publisher": {
"@type": "Organization",
"name": "PSI"
},
"@context": {
"@vocab": "<https://schema.org/>"
}
}
In the json-ld there will be a single url in the distribution that points to s3 bucket.
NOTE: The dataset has to be retrieved and will eventually expire. After it expires the user is expected to click a link (provided by Renku in the UI) that leads to SciCat where they can request the dataset to be retrieved again. We have to then handle expiring data connectors in Renku. The user will have to pause/resume the session in order to get the mount to re-appear when the dataset expires.
Risks: How do we spot the s3 url. We discussed mounting each file individually with the pre-signed url but this is not possible with rclone.