🤔 Problem

Sessions, jobs, builds, etc. do not have persisted logs which makes debugging hard for RenkuLab users. This is notably painful for jobs and builds, as we do not have guaranteed log access after the task is completed or failed.

🍴 Appetite

🎯 Solution

Persist logs into the database used by renku-data-services so that logs can be viewed even after the underlying kubernetes resources has been deleted.

Log collection - How to collect logs?

Solution 1: Query loki to copy logs into the database.

Need to update alloy configuration to add more log labels
- renku.io/logs = true for pods belonging to a session, job, build or app
- renku.io/username , workload_kind = { session, job, build, app }, etc.
- Important: may need to introduce the concept of a run: continuous execution of a workload (e.g. from session start to pause/delete, each resume makes a new run)
  - Objective is to assign a TTL to a run so that we can purge old logs (e.g. 7 days TTL)
How to forward logs:
1. Get logs from loki
2. Map labels to a session, job, build or app
3. From the parent workload, determine log line

Solution 2: Query the k8s API (may need to deploy a new service) to forward logs.

Potential issue: how to scale when a lot of workloads are running? (in solution 1, there is one alloy forwarder per node and loki may be scalable)

Log permissions

Workload	Who can view logs?
Session	user ( = session owner)
Job	user ( = job owner)
Build	Project Write + Repo Read
App	??? Should it be same as build?

UI - Access to log history means we need new interfaces

Where to place the log history?
- Session log history
  - LKK: I’ve never heard anyone complaining about not being able to see previous session logs, so I think it’s ok to not support this as part of this scope
- Job log history
  - LKK: What do you think of putting an interaction point in the job launcher side panel at the bottom of the current job list for viewing a history of jobs and their logs?
    - Screenshot
- Build log history
  - LKK: I think the main thing we need is to guarantee that you can see the logs for the current build. I think it’s ok to not support viewing previous build logs as part of this scope
- How to deal with orphaned logs. E.g. logs from a session I ran 2 days ago but then I deleted the project. ⇒ logs still exist and would be accessible but where should we see them?
  - LKK: I think we can not support viewing orphaned logs for the scope of this work
The log history is also a workload history feed
- I ran a session yesterday for 30 minutes
- LKK: I’m picturing that we show the same info we show in the side panel, so start time (nice to have: also duration?), job command, args, and resource class