Introduction

The HPC Job Proxy (formerly known as supercomputing proxy or Unicore job proxy) provides a simplified mean for EBRAINS service providers to launch jobs on Fenix supercomputers on behalf of EBRAINS end-users.

The proxy offers a wrapper over the Unicore service which adds:

• Logging of the jobs run on behalf of the end-user
• Access to the stdout, stderr, and return status of the job for the end-user
• Verification of the end-user’s quotas before submitting jobs
• Update of the end-user’s quotas

In order to use the proxy, an EBRAINS service provider needs to:

• Get a project and service account at one or more Fenix sites
• Get an EBRAINS service account and an EBRAINS IAM OIDC client ID
• Map that Fenix service account to that EBRAINS service account

Why does each service provider need a Fenix service account? The reason is that the Principal Investigator who gets the Fenix service account is legally responsible for the jobs being run on the supercomputers and for not enabling the end-user to run unintended executables.

Use case

1. Bob wants to do use an EBRAINS Application that requires supercomputing.
   1. The Application authenticates Bob as an EBRAINS user.
   2. He provides input which determines the supercomputing job to be run.
2. The Application authenticates itself using its EBRAINS IAM service account with the HPC Job Proxy, and sends it the following information:
   1. a vanilla Unicore job definition,
   2. the EBRAINS IAM username of the end user on whose behalf the job is to be executed,
   3. a maximum amount of resources (managed by the EBRAINS quota manager) to be consumed by the job,
   4. an optional callback URL for notifications regarding the job.
3. The HPC Job Proxy queries the EBRAINS Quota Manager for Bob and the maximum resources indicated.
   1. The EBRAINS Quota Manager queries the EBRAINS IAM to identify the quota for the relevant resources requested.
   2. The EBRAINS Quota Manager checks its database for resources already consumed by Bob and determines whether Bob has enough quota to run the job.
   3. The EBRAINS Quota Manager returns a validation to the Application, possibly indicating Bob’s quota status for those resources.
4. If Bob has enough quota, the HPC Job Proxy accepts the job request.
   1. It logs Bob’s request.
   2. It sends the job to the Unicore API using the same EBRAINS IAM token the Application identified itself with. This will submit the job to Unicore impersonating the Application, and ensuring the job submission is accounted towards the Application’s own Fenix Service Account.
5. When the job finishes (completes or fails), Unicore sends a notification to the HPC Job Proxy’s callback endpoint.
6. The HPC Job Proxy logs the job results, and pushes the actual cost of the job run for Bob to the EBRAINS Quota Manager.
7. The HPC Job Proxy notifies the Application of the results of the job.

Sample transaction diagram

API of the HPC Job Proxy

The API provides 3 endpoints described below. The Swagger documentation for the API is available here.

• Submit a job: POST /api/jobs/

To submit a job to the proxy, you need to provide the following information as part of the POST JSON body:

• job_def - JSON: The Unicore job definition. For more information, please visit the Unicore documentation
• site - string: The Fenix site on which to run the job.
• user_info - string: The Application’s access token user information.

Valid Fenix site values are:

• Fetch a job's details: GET /api/jobs/<job_id>

The proxy will query Unicore on-the-fly for the job’s latest details. Fenix sites retain the information of past jobs for a set amount of days (30 days at CSCS). If the request is made after that delay, the information returned is that which has been stored in the HPC Job Proxy. The full information retrieved and stored is the following:

• Fetch a file present in the job's execution directory: GET /api/jobs/<job_id>/<filename>

​​​​​​​Out of convenience, you can fetch the files that are present in the job's execution directory. The output, error and exit code of a job are placed by Unicore in the job’s working directory. The filenames available are: stdout, stderr, UNICORE_SCRIPT_EXIT_CODE. If the job has pre commands or post commands, the following filenames are also available: stdout and stderr respectively under the folders .UNICORE_POST_0/ or .UNICORE_PRE_0/. If your job is creating files than they are too available to be fetched.

The proxy will respond with the raw content of the requested file. These files are available as long as the site does not delete them (30 days for CSCS PIZ DAINT).

For all 3 endpoints, you need to provide your service account access token in the Authorization header of your request (Bearer <access_token>). Your service account must be linked to a Fenix account, otherwise your job will be rejected (HTTP 502).

Sample usage

A Jupyter Notebook provides sample code for the Application’s access to the HPC Job Proxy.

Source code

The source code of the HPC Job Proxy is available on Gitlab.

Setting up accesses for your application

For your application to work, you need to set up the following accesses.

Your application needs an EBRAINS IAM service account. You should contact EBRAINS support to create it. You will need to provide a username and an email address which is not linked to another EBRAINS account. The mapping between the EBRAINS service account and the Fenix service account is done automatically based on the linked email addresses which need to be identical.

Your application needs an EBRAINS IAM OIDC client. See the instructions.

Your application needs a Fenix project and a Fenix service account. See the instructions to get a project. Then the Principal Investigator in your lab can request a Fenix service account via EBRAINS support.

Then in your app, you should retrieve the access token linked to your service account user. This requires the OIDC client ID, client Secret, EBRAINS service account's username and password. See the instructions.

When requesting a job on behalf of an end-user, your app gives this access token in the field "user_info" of the payload alongside the job request. Note that the access token expires (typically after one week), so your application should generate a new token.

Example:

The EBRAINS Image Service uses the HPC Job Proxy.

• EBRAINS IAM OIDC Client: “img_svc”
• EBRAINS IAM service account: "ich019sa"
• Fenix project: “ich019”
• Fenix service account: “ich019sa”
• Email address of both service accounts: “platform.ich019sa@humanbrainproject.eu”

List of services using the HPC Job Proxy

• EBRAINS image service
• Ilastik (ongoing)
• TVB cloud (ongoing)