doku:projects [VSC Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Creating and managing projects at VSC ======
===== Creating a new project =====
==== Login to service page ====

Go to [[https://service.vsc.ac.at|our service page]] and on the left side (Login for project managers), choose your organization. If it is not there, please contact us for an external access.
Then log in using the credential from your organization. 

**Note:** If your organization is not in our list, go to https://idp.vsc.ac.at/simplesaml/module.php/selfregister/ and choose 'Register new account'.
After you have done that, please tell us the username you created and we will add it to our system. 
Then you can use "IdP VSC" as organization to log in.

==== Request new project ====

Click on 'request project' in the left menu bar.
  * choose the appropriate type: see [[doku:projects#project_types|project types]].
  * give the project a title. This is up to you and need not be identical with any official tilte
  * enter the institute/subunit/department you are doing this project for
  * the item 'Parent project should be left at default, unless you either:
      * want to create a new parent project. This is usually done when you start a new cooporation for an organization, or if you have a large project with several different sub-parts
      * or if you are part of an existing parent project. Please check the dropdown list for such projects
  * choose if you are working with Personal Data according to the GDPO. This refers to the data used in the project, not the data of the users that work on the project.
  * then click on create

=== Project types ===

  * **test:** intended for short test runs, to see if a project is doable on VSC, or if a certain code runs, or to see how to use the cluster. This is limited to 30 days (some extension is usually possible) and to 100000 CPU core hours. Such a project can be created by everyone interested and is not billed or accounted. It cannot be used for any longer running, productive project.
  * **funded:** any project that is from an organization with free access to VSC which is funded by a funding agency, like FWF, FFG, ERC, a federal state or the like. 
  * **internal:** any project that is from an organization with free access to VSC which gets financed in some other way, usually via an internal budget. This needs peer review (see later).
  * **external:** any project from a group without free access to the VSC. This is divided into 2 versions:
     * __external-scientific:__ projects from organizations primarily involved in scientific research. These get a reduced rate to purchase core-hours.
     * __external-commercial:__ projects from organizations primarily involved in applied research or other commercial aims.
  * **application:** this is intended for projects that have applied/are applying for funding but have no grant yet. This way, the VSC resources can be reserved and/or argued towards the funding agency.
  * **private:** these projects give access to privately bought hardware that is integrated into our systems.
==== Add project details ====

After the creation, you get assigned a unique project ID and a corresponding project page. You affiliation (the organization you are from) will be automatically added, and you will be added as the manager of the project. You will also get an email about the project creation that includes further steps to do.

Now you need to choose a resource pool at the item 'Request resources from a resource pool'. This basically means the hardware your project will be running on. The project can run on several of these pools at the same time.
The current options are:
  * VSC-4: Our older cluster (operational since 2019) consisting of Intel nodes with 48 cores and either 96GB, 384GB or 768GB RAM. Recommended of you are reliant on Intel architecture
  * VSC-5: Our newer cluster (operational since 2022) consisting of AMD nodes with 128 cores and either 512GB, 1TB or 2TB RAM, plus some Intel nodes, plus A40 and A100 GPU nodes. Recommended if you need GPUs, AMD architecture or have no special requirements.
  * LEONARDO-B: GPU Booster Module of the European Pre-Exascale cluster LEONARDO. For detail please see [[https://vsc.ac.at//access/aureleo-austrian-users-at-leonardo-supercomputer/|here]].

Click on 'request new project part and enter the requirements:
  * Start date is automatically the current date - can be changed if needed
  * The expected amount of cores used in parallel - this needs not to be exact, as it has no direct influence on the project. It is just used to assess the feasibility.
  * List the software you want to use
  * In the comment section, you can enter any additional information relevant to the project
  * At the lower part, keep all buttons on 'yes'. Usually, the default values are good and can be kept, unless you know you have different requirements. Then change the number in the appropriate field. The comments box can be used to explain, if needed. The resource types are explained [[doku:projects#resource_types|here]].

Then click 'Submit Query'. This will create a project part for the selected resource pool.

=== Resource types ===

  - **cpu contingent:** The CPU core hours that are estimated to be needed for the project.\\ This can be extended at any time during the project runtime. For testprojects, this is fixed to 100000, with extensions only under special circumstances. In private projects, the core-h are not accounted for, so the number is not relevant. In external projects, this must not be higher than the number purchased.
  - **project end:** The runtime of the project in days, counted from the beginning date.\\ This can be extended at any point, even after project expirations (to reactivate a project). The default is 366 days, and projects are only granted 1 year at a time. Private projects are an exception and are usually granted for several years. Testprojects are set to 30 days, with a limited amount of extension possible.
  - **$DATA inodes:** The number of inodes in the DATA storage volume.\\ This correspondents to the number of files. The default is one million, which can be extended at any time.
  - **$DATA size:** The size, in GB, of the DATA storage volume.\\ The default is 10TB, and it can be extended to 100TB without any query. Higher values need to be discussed with us.
  - **$HOME inodes:** The number of inodes in the HOME storage volume.\\ This correspondents to the number of files. The default is one million, which can be extended at any time.
  - **$HOME size:** The size, in GB, of the HOME storage volume.\\ The default is 100GB, which will never be extended except in some special situations.

==== Upload files ====

Next, unless it is a testproject or a private project, you need to upload some files at the section 'Files':

For every project type except the mentioned ones, upload a short project description (2-4 pages).\\ 
For a funded project, upload a proof of funding.\\ 
For an internal project, upload a list of at least 3 possible reviewers. They can be from anywherere except your department.\\ 
For an external project, upload either a quota for core hours or an appropriate contract about calculation times.\\ 
Additionally, if you chose yes for the GDPO data, upload a data handling contract with the TU Vienna.

You get a confirmation mail for every uploaded file.
Once this is done (or directly, if no files are needed), click on the red bar at the top of the new project part to request technical review.

==== Review process ====
=== tech approval ===

All projects are first checked by the VSC team to assess if it can actually be run on the systems, if all project data are set correctly or if any other administrative or technical issue is still open. At that stage, the VSC team also creates the files directories for the project.

=== peer review (optional) ===

Projects of the type internal go through a peer review - the project description is sent out to the mentioned reviewers, ans their answers are then incorporated. If they are positive, the project goes to the next stage.

=== SC (steering committee) review ===

Projects of the type funded, internal (after peer review), external and application are sent to the SC for review. Depending on the requested amount of core-h, this is either just the SC members of the requesting organization (if below 2 Mio core-h) or the entire SC (above 2 Mio core-h). 
Projects of the type test or private only have the tech review.

Once the project is granted, you will get a mail notification

==== Adding users ====

Once a project is granted, user can be created for it. These are the actual cluster suers and independent of any users on the service page.

Click on 'create cluster user account' and fill in //First// and //Last name//, an //email address// and a //phone number//. The phone number can be from any country, but it has to be able to receive text messages. If such a number is not available, please contact us\\ 
Then click 'submit' and enter a //username// and a //password//. If you have project parts on several resource pools, you can activate or deactivate the user there via checkboxes. Then click 'create account'.\\ 
The username must be unique within the entire VSC. The password must be 10 charactes, with at least one lower case letter, one upper case letter, one digit and one special character.

The user gets an email with some informations and instructions. The password is not contained in that mail and has to be conveyed to the user in some other way.

A project can contain any number of users, even multiple one for the same real person, but email and phone number have to be unique within the project.

**As soon as users are created, these can log into the cluster and start working on the project.**

===== Project changes and extensions =====
At any time while the project is running, various resources can be changed or extended
==== Changes in the project ====
The following items can be changed by the project manager via the project page:
  * personal data: If personal data according to the GDPO are used can be toggled between yes or no. Please consult us before doing so
  * backup status: in each resource pool, backup on the volume $DATA can be turned off or on.
  * files: additional files can be uploaded - but not deleted or edited
  * user accounts: see user management
  * resource pools and resource extensions: see below

==== Project extensions ====

At 'Request resources from a resource pool' the project can be extended to another part of our systems. See the drop-down list for the possible options

To extend some of the resources of an existing resource pool, click on 'request more resources' at the top of that part. On the opened page, set those resource types you want to extend to 'yes' (leave all others on 'no') and enter the value you want to have ADDED to the current amount. You can add an explanatory comment as well. Afterwards click on submit.\\   
The following resource types can be extended:
  * **cpu-contingent in core-h:** The amount on additional core hours. Depending on project type and amount, this has to be granted by part or all of the steering committee. For test projects, core-h are only extended for special reasons.
  * **project end in days:** The number of days the project should be extended after the project end date. Depending on the project type, this has to be granted by part of the steering committee. Please note that test projects are only extended for a limited time, and all other projects (except private) are only extended for a year at a time.
  * **\$DATA inodes or \$HOME inodes in # of files:** The additional amount of files that can be stored in the respective volume. 
  * **\$DATA size or \$HOME size in GB:** the maximum size of the respective volume. \$HOME will only be extended for exceptional reasons and is otherwise fixed.

==== Changes done by the VSC team ====

Some changes in the project can only be done by the VSC team. Please contact us if that is needed.\\ 
These changes are for example the project manager, project name, project group memberships (access to certain software or volumes for the entire project)\\ 
The VSC team can also add secretaries to a project - these are persons that have similar rights as the project manager. In contrast to the manager, several secretaries are possible.

===== Project lifecycle =====

During the runtime of a project, you will or might get various reminder or warning mails:

==== core hours ====

There will be a first warning mail when the core hours in your project are **90% used up**. Please note that the mail is sent at a fixed time of the day, which means in the time between crossing the 90% threshold and the mail being sent, the use might already be higher. 

When the core-h are used up to **100%**, there will be another mail. This will also result in your project being set to idle priority. This means only the idle queues on the cluster are available. These queues have a low priority and only allow jobs to run, when no one else needs the respective nodes. Which in turn greatly impacts the chance for running jobs.

It is then highly recommended to request an extension. Funded and internal projects can always extend as long as they are running, external projects need an appropriate quote. In testprojects, core-h are only extended in special circumstances. Note that private projects do not count core-h at all.

==== number of files ====

When the number of files (which equals the number of inodes) in either \$HOME or \$DATA reaches **90% of the allowed quota**, a warning mail is sent. As with core-h, the actual value can be higher as the mail is sent some time after the threshold is reached.

Another mail is sent upon reaching **100%** - at that point, no more files can be created in the affected volume, preventing further work or sometimes even login. In that case, delete any files not needed and if that is not sufficient, request an extension. Any type of project can get an extension in files.

**Note:** the volumes \$HOME and \$FILE are handled independently, so mails can be received for both.

==== volume size ====

When the size of the data in either \$HOME or \$DATA reaches **90% of the allowed quota**, a warning mail is sent. As with core-h, the actual value can be higher as the mail is sent some time after the threshold is reached.

Another mail is sent upon reaching **100%** - at that point, no more files can be created in the affected volume, preventing further work or sometimes even login. In that case, delete any files not needed and if that is not sufficient, request an extension. Any type of project can get an extension in size on \$DATA up to 100TB. Higher values are possible after discussion with us. 
\$HOME cannot be extended, except for very special circumstances, such as more than 20 users in a project. Therefore, put all research data in $DATA.

**Note:** the volumes \$HOME and \$FILE are handled independently, so mails can be received for both.

==== Project expiry ====

**Expiry:** About a month before a project expires, you are sent a notification mail about said expiry. Another mail is sent on the day of expiry. As soon as a project is expired, it is still possible to work normally, including job submission. However, new user can no longer be created.

**Job submission disabled:** One month after project expiry, a mail is sent about job submissions being disabled. From that point on, jobs can no longer be submitted to the clusters. Jobs that are running at that time are not broken off.

**Backup relevant project data:** Two month after project expiry, a mail is sent to warn you to backup any data from your project that you want to keep. The mail says data retention is only guaranteed until three months after project expiry. For details about data see the next paragraph.

**Login disabled:** Three month after job submission, a mail is sent informing about login being disabled. At that point, all user logins to the clusters belonging to that project are disabled. So far, all data belonging to a project was kept indefinitely after the project ran out - but this can not be guaranteed as total storage space might run out. It is therefore highly recommended to back up all relevant data before that point.

If the project shall continue, an extension can be requested. That can even be done after the project has run out. Testprojects can only be extended for short times (30 - 60 days usually), other projects for a maximum of one year at a time.

**Auto-extension:** If a project (that is neither test nor private) has a large amount of core-h left available, it is automatically extended once for 6 month.