🤔 Problem

Many people namely people at the SDSC, use non-interactive jobs for their work, and the absence of this functionality presents a adoption barrier.

Batch job systems are common- what makes Renku different? First, we want Renku to make it super simple to transition from doing interactive work to submitting that same task as a non-interactive job. We’d like tsomeone to be able to do some exploratory work in a regular Renku session, and once they hone in on the task they want to submit as a job, they “launch” the same Renku session as a job with little extra effort. For example, they don’t have to re-set up an environment, or move their data somewhere new. Just enter your job command and hit submit!

Second, we want to offer a unified platform where folks submitting batch jobs can work together on the same project with less technical collaborators. For example, in a single Renku project, an AI Engineer can run model training jobs, and also the domain expert can test out the resulting model in a Jupyter notebook or dashboard. All data and code stays together in one place.

🍴 Appetite

6 weeks

🎯 Solution

We want to support two ways of using jobs on Renku: One is for people who already have a Renku project with an interactive session and they want to run a part of their session as a job. The second is for people who have a docker image they use for a job elsewhere (for example on run.ai), and we want to support them bringing that docker image to Renku and running it as a job. In this second case, the docker image probably does not have a front end, i.e. not be able to run as a “regular” interactive Renku session.

User Flow diagrams are here: ‣ (these diagrams cover both this pitch and Renku Apps)

</aside>

https://www.figma.com/board/SqsBNlhGB0jSXPKUwy0SQd/Shaping-Boards?node-id=155-1218

User Flow: Create a new Job launcher with an External Environment (for example a docker image from run.ai)

Job Story

When I already have a docker image that I use for my work,

I want to set that same image up in Renku

so I can use Renku compute resources for my work.

</aside>

The user creates a new launcher, and they are prompted to choose which kind of launcher to create, and they select Job launcher (as opposed to Session Launcher). They have the option to create the environment for their job launcher through Create from Code or External Environment. They choose External Environment. They enter their container image and all the usual settings, including specifying in the Command and Args the values to launch their job.

User Flow: Add the ability to submit jobs to an existing session launcher

Job Story

When I have a Renku project and I now want to take what I’ve made in an interactive session and run it as a non-interactive job,

I want to make the same launcher I’ve already configured work in a non-interactive mode

so that I don’t have to duplicate work.

</aside>

We assume the user has an interactive session where the environment is a code based environment. If it’s an external environment, then follow the user flow above. We assume it’s unlikely that a user creating a job is using a global environment- they almost surely need custom dependencies.

The user creates a job launcher the same as above, but this time, they choose create from code and choose the same code repository as defines the environment for their interactive session. There is one additional field in the create from code setup compared to when creating a session launcher, which is the user enters a job command.

<aside> 🎀

nice to have: the user can open the drop down on their session launcher and choose an option “duplicate as job launcher”. This creates a job launcher with the same environment as the session launcher, prompting for a job command along the way.

</aside>