Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
doku:projects [2024/02/20 20:26] – [Project expiry] goldenbergdoku:projects [2024/08/01 11:40] (current) – [Project Expiry] goldenberg
Line 1: Line 1:
-====== Creating and managing projects at VSC ====== +====== Creating and Managing Projects at VSC ======
-===== Creating a new project ===== +
-==== Login to service page ====+
  
-Go to [[https://service.vsc.ac.at|our service page]] and on the left side (Login for project managers), choose your organization. If it is not there, please contact us for an external access. +This website guides users through creating and managing projects on the VSC service platform. It starts with logging into the service page, followed by steps to request a new project, detailing the types of projects available, and adding necessary project details. The tutorial includes instructions on selecting appropriate resource pools, uploading required files, and the review process. It also covers adding users, managing project changes and extensions, and outlines the notification system for project lifecycle events, ensuring users are aware of their project's status and necessary actions to maintain smooth operation.
-Then log in using the credential from your organization. +
  
-**Note:** If your organization is not in our list, go to https://idp.vsc.ac.at/simplesaml/module.php/selfregister/ and choose 'Register new account'. +===== Creating a New Project =====
-After you have done that, please tell us the username you created and we will add it to our system.  +
-Then you can use "IdP VSC" as organization to log in.+
  
-==== Request new project ====+==== Logging in to the Service Page ====
  
-Click on 'request project' in the left menu bar. +Navigate to [[https://service.vsc.ac.at|our service page]].
-  * choose the appropriate type: see [[doku:projects#project_types|project types]]. +
-  * give the project a title. This is up to you and need not be identical with any official tilte +
-  * enter the institute/subunit/department you are doing this project for +
-  * the item 'Parent project should be left at default, unless you either: +
-      * want to create a new parent project. This is usually done when you start a new cooporation for an organization, or if you have a large project with several different sub-parts +
-      * or if you are part of an existing parent project. Please check the dropdown list for such projects +
-  * choose if you are working with Personal Data according to the GDPO. This refers to the data used in the project, not the data of the users that work on the project. +
-  * then click on create+
  
-=== Project types ===+Select your organization from the list under 'Login for Project Managers' on the left side.
  
-  * **test:** intended for short test runs, to see if a project is doable on VSC, or if a certain code runs, or to see how to use the cluster. This is limited to 30 days (some extension is usually possible) and to 100000 CPU core hours. Such a project can be created by everyone interested and is not billed or accounted. It cannot be used for any longer running, productive project. +Log in using your organization's credentials. 
-  * **funded:** any project that is from an organization with free access to VSC which is funded by a funding agency, like FWF, FFG, ERC, a federal state or the like.  +   * **Note:** If your organization is not listed, visit https://idp.vsc.ac.at/simplesaml/module.php/selfregister/ to register. After registering, inform us of your username, and we will add it to our system. You may then use "IdP VSC" as your organization for login.
-  * **internal:** any project that is from an organization with free access to VSC which gets financed in some other way, usually via an internal budget. This needs peer review (see later). +
-  * **external:** any project from a group without free access to the VSC. This is divided into 2 versions: +
-     * __external-scientific:__ projects from organizations primarily involved in scientific research. These get a reduced rate to purchase core-hours. +
-     * __external-commercial:__ projects from organizations primarily involved in applied research or other commercial aims. +
-  * **application:** this is intended for projects that have applied/are applying for funding but have no grant yet. This way, the VSC resources can be reserved and/or argued towards the funding agency. +
-  * **private:** these projects give access to privately bought hardware that is integrated into our systems. +
-==== Add project details ====+
  
-After the creation, you get assigned a unique project ID and a corresponding project page. You affiliation (the organization you are from) will be automatically added, and you will be added as the manager of the project. You will also get an email about the project creation that includes further steps to do.+==== Requesting a New Project ====
  
-Now you need to choose a resource pool at the item 'Request resources from a resource pool'. This basically means the hardware your project will be running on. The project can run on several of these pools at the same time. +Click 'Request Project' in the left menu bar.
-The current options are: +
-  * VSC-4: Our older cluster (operational since 2019) consisting of Intel nodes with 48 cores and either 96GB, 384GB or 768GB RAM. Recommended of you are reliant on Intel architecture +
-  * VSC-5: Our newer cluster (operational since 2022) consisting of AMD nodes with 128 cores and either 512GB, 1TB or 2TB RAM, plus some Intel nodes, plus A40 and A100 GPU nodes. Recommended if you need GPUs, AMD architecture or have no special requirements. +
-  * LEONARDO-B: GPU Booster Module of the European Pre-Exascale cluster LEONARDO. For detail please see [[https://vsc.ac.at//access/aureleo-austrian-users-at-leonardo-supercomputer/|here]].+
  
-Click on 'request new project part and enter the requirements: +Select the appropriate project type as listed below.
-  * Start date is automatically the current date - can be changed if needed +
-  * The expected amount of cores used in parallel - this needs not to be exact, as it has no direct influence on the project. It is just used to assess the feasibility. +
-  * List the software you want to use +
-  * In the comment section, you can enter any additional information relevant to the project +
-  * At the lower part, keep all buttons on 'yes'. Usually, the default values are good and can be kept, unless you know you have different requirements. Then change the number in the appropriate field. The comments box can be used to explain, if needed. The resource types are explained [[doku:projects#resource_types|here]].+
  
-Then click 'Submit Query'. This will create a project part for the selected resource pool.+Provide a project title (it need not match any official title).
  
-=== Resource types ===+Specify the institute, subunit, or department for which the project is conducted.
  
-  - **cpu contingent:** The CPU core hours that are estimated to be needed for the project.\\ This can be extended at any time during the project runtime. For testprojects, this is fixed to 100000, with extensions only under special circumstances. In private projects, the core-h are not accounted for, so the number is not relevant. In external projects, this must not be higher than the number purchased. +The 'Parent Project' field should remain at default unless: 
-  - **project end:** The runtime of the project in days, counted from the beginning date.\\ This can be extended at any point, even after project expirations (to reactivate a project). The default is 366 days, and projects are only granted 1 year at a time. Private projects are an exception and are usually granted for several years. Testprojects are set to 30 days, with a limited amount of extension possible. +    * You are initiating a new parent project for a new collaboration or a large project with multiple sub-parts -> select 'create new parent project'. 
-  - **$DATA inodes:** The number of inodes in the DATA storage volume.\\ This correspondents to the number of files. The default is one million, which can be extended at any time. +    * You are part of an existing parent project -> select from the dropdown list.
-  - **$DATA size:** The size, in GB, of the DATA storage volume.\\ The default is 10TB, and it can be extended to 100TB without any query. Higher values need to be discussed with us. +
-  - **$HOME inodes:** The number of inodes in the HOME storage volume.\\ This correspondents to the number of files. The default is one million, which can be extended at any time. +
-  - **$HOME size:** The size, in GB, of the HOME storage volume.\\ The default is 100GB, which will never be extended except in some special situations.+
  
-==== Upload files ====+Indicate whether the project involves Personal Data in accordance with GDPR.
  
-Next, unless it is a testproject or a private project, you need to upload some files at the section 'Files':+Click 'Create'.
  
-For every project type except the mentioned ones, upload a short project description (2-4 pages).\\  +=== Project Types ===
-For a funded project, upload a proof of funding.\\  +
-For an internal project, upload a list of at least 3 possible reviewers. They can be from anywherere except your department.\\  +
-For an external project, upload either a quota for core hours or an appropriate contract about calculation times.\\  +
-Additionally, if you chose yes for the GDPO data, upload a data handling contract with the TU Vienna.+
  
-You get a confirmation mail for every uploaded file. +**Test:** For short-term test runs, with a limit of 90 days and 100,000 CPU core hours. Not billed or accounted.
-Once this is done (or directly, if no files are needed), click on the red bar at the top of the new project part to request technical review.+
  
-==== Review process ==== +**Funded:** Projects from organizations which have free access to VSC resources, if they are funded by agencies such as FWF, FFG, ERC, or federal states.
-=== tech approval ===+
  
-All projects are first checked by the VSC team to assess if it can actually be run on the systems, if all project data are set correctly or if any other administrative or technical issue is still open. At that stage, the VSC team also creates the files directories for the project.+**Internal:** Projects funded through internal budgets, requiring peer review.
  
-=== peer review (optional) ===+**External:** Projects from organizations without free VSC access, categorized as: 
 +  * __External-Scientific:__ For scientific research with reduced core-hour rates. 
 +  * __External-Commercial:__ For applied research or commercial purposes.
  
-Projects of the type internal go through a peer review - the project description is sent out to the mentioned reviewers, ans their answers are then incorporated. If they are positive, the project goes to the next stage.+**Application:** Projects applying for funding, which reserve VSC resources in advance.
  
-=== SC (steering committee) review ===+**Private:** Projects involving privately purchased hardware integrated into VSC systems.
  
-Projects of the type funded, internal (after peer review), external and application are sent to the SC for review. Depending on the requested amount of core-h, this is either just the SC members of the requesting organization (if below 2 Mio core-h) or the entire SC (above 2 Mio core-h).  +==== Adding Project Details ====
-Projects of the type test or private only have the tech review.+
  
-Once the project is granted, you will get a mail notification+Following project creation, you will receive a unique project ID and a dedicated project page.
  
-==== Adding users ====+Your affiliation and role as the project manager will be automatically assigned.
  
-Once a project is granted, user can be created for it. These are the actual cluster suers and independent of any users on the service page.+You will receive an email outlining the next steps.
  
-Click on 'create cluster user account' and fill in //First// and //Last name//, an //email address// and a //phone number//. The phone number can be from any country, but it has to be able to receive text messages. If such a number is not available, please contact us\\  +==== Selecting a Resource Pool ====
-Then click 'submit' and enter a //username// and a //password//. If you have project parts on several resource pools, you can activate or deactivate the user there via checkboxes. Then click 'create account'.\\  +
-The username must be unique within the entire VSC. The password must be 10 charactes, with at least one lower case letter, one upper case letter, one digit and one special character.+
  
-The user gets an email with some informations and instructions. The password is not contained in that mail and has to be conveyed to the user in some other way.+On the project page, select 'Request Resources from a Resource Pool': 
 +  * **VSC-4:** Our older cluster featuring Intel nodes (48 cores, 96GB/384GB/768GB RAM). Recommended if Intel architecture is preferred. 
 +  * **VSC-5:** Our newer cluster with AMD nodes (128 cores, 512GB/1TB/2TB RAM), some Intel nodes(96 cores, 384GB RAM), and A40/A100 GPU nodes. Ideal for GPU use, AMD architecture, or if no specific requirements. 
 +  * **LEONARDO-B:** GPU Booster Module of the European Pre-Exascale cluster [[https://leonardo-supercomputer.cineca.eu/|LEONARDO]]. Can only be requested if there is a specific call. 
 +Click 'Request New Project Part' and input the required details: 
 +  * The start date is pre-set to the current date but may be adjusted if necessary. 
 +  * Specify the anticipated number of cores used in parallel (for feasibility assessment). 
 +  * List the software to be utilized. 
 +  * Use the comment section for any additional pertinent information. 
 +  * At the bottom, retain default settings unless specific adjustments are required. Resource types are explained [[doku:projects#resource_types|here]]. 
 +Click 'Submit Query' to create a project part for the selected resource pool.
  
-A project can contain any number of users, even multiple one for the same real person, but email and phone number have to be unique within the project.+=== Resource Types ===
  
-**As soon as users are created, these can log into the cluster and start working on the project.**+**CPU Contingent:** Estimated CPU core hours required. Extendable during the project. Fixed at 100,000 hours for test projects. Not accounted for private projects.
  
-===== Project changes and extensions ===== +**Project End:** Duration of the project in days from the start date. Extendable even post-expiration. Default is 366 days; test projects are limited to 90 days.
-At any time while the project is running, various resources can be changed or extended +
-==== Changes in the project ==== +
-The following items can be changed by the project manager via the project page: +
-  * personal data: If personal data according to the GDPO are used can be toggled between yes or no. Please consult us before doing so +
-  * backup status: in each resource pool, backup on the volume $DATA can be turned off or on. +
-  * files: additional files can be uploaded - but not deleted or edited +
-  * user accounts: see user management +
-  * resource pools and resource extensions: see below+
  
-==== Project extensions ====+**$DATA Inodes:** Number of inodes in the DATA storage volume. Default is 1 million, extendable.
  
-At 'Request resources from a resource pool' the project can be extended to another part of our systems. See the drop-down list for the possible options+**$DATA Size:** Size of the DATA storage volume in GB. Default is 10TB, extendable up to 100TB. Larger sizes require discussion.
  
-To extend some of the resources of an existing resource pool, click on 'request more resources' at the top of that part. On the opened page, set those resource types you want to extend to 'yes' (leave all others on 'no') and enter the value you want to have ADDED to the current amount. You can add an explanatory comment as well. Afterwards click on submit.\\    +**$HOME Inodes:** Number of inodes in the HOME storage volume. Default is 1 million, extendable.
-The following resource types can be extended: +
-  * **cpu-contingent in core-h:** The amount on additional core hours. Depending on project type and amount, this has to be granted by part or all of the steering committee. For test projects, core-h are only extended for special reasons. +
-  * **project end in days:** The number of days the project should be extended after the project end date. Depending on the project type, this has to be granted by part of the steering committee. Please note that test projects are only extended for a limited time, and all other projects (except private) are only extended for a year at a time. +
-  * **\$DATA inodes or \$HOME inodes in # of files:** The additional amount of files that can be stored in the respective volume.  +
-  * **\$DATA size or \$HOME size in GB:** the maximum size of the respective volume. \$HOME will only be extended for exceptional reasons and is otherwise fixed.+
  
-==== Changes done by the VSC team ====+**$HOME Size:** Size of the HOME storage volume in GB. Default is 100GB, only extended for very specific exceptions.
  
-Some changes in the project can only be done by the VSC team. Please contact us if that is needed.\\  
-These changes are for example the project manager, project name, project group memberships (access to certain software or volumes for the entire project)\\  
-The VSC team can also add secretaries to a project - these are persons that have similar rights as the project manager. In contrast to the manager, several secretaries are possible. 
  
-===== Project lifecycle =====+==== Uploading Required Files ====
  
-During the runtime of a project, you will or might get various reminder or warning mails:+If applicable, upload the necessary documents in the 'Files' section: 
 +  * Project description (2-4 pages). [every project type except test and private] 
 +  * Proof of funding for funded projects. 
 +  * Reviewer list (minimum of 3) for internal projects. 
 +  * Quota or contract for core hours for external projects. 
 +  * GDPR data handling contract if Personal Data is involved. 
 +You will receive a confirmation email for each uploaded file.
  
-==== core hours ====+Once all required files are uploaded (or if no files are needed), click the red bar at the top of the project part to request a technical review.
  
-There will be a first warning mail when the core hours in your project are **90% used up**. Please note that the mail is sent at a fixed time of the day, which means in the time between crossing the 90% threshold and the mail being sent, the use might already be higher. +==== Review Process ====
  
-When the core-h are used up to **100%**, there will be another mail. This will also result in your project being set to idle priority. This means only the idle queues on the cluster are available. These queues have a low priority and only allow jobs to run, when no one else needs the respective nodes. Which in turn greatly impacts the chance for running jobs.+=== Technical Approval ===
  
-It is then highly recommended to request an extension. Funded and internal projects can always extend as long as they are running, external projects need an appropriate quote. In testprojects, core-h are only extended in special circumstances. Note that private projects do not count core-h at all.+The VSC team will review the project to ensure feasibility and resolve any administrative or technical issues.
  
-==== number of files ====+The VSC team will also create the necessary file directories for the project.
  
-When the number of files (which equals the number of inodes) in either \$HOME or \$DATA reaches **90% of the allowed quota**, a warning mail is sent. As with core-h, the actual value can be higher as the mail is sent some time after the threshold is reached.+=== Peer Review (Internal Projects Only) ===
  
-Another mail is sent upon reaching **100%** - at that point, no more files can be created in the affected volume, preventing further work or sometimes even login. In that case, delete any files not needed and if that is not sufficient, request an extension. Any type of project can get an extension in files.+Internal projects will be sent to the listed reviewers.
  
-**Note:** the volumes \$HOME and \$FILE are handled independently, so mails can be received for both.+Feedback from reviewers will be incorporated into the project evaluation.
  
-==== volume size ====+=== Steering Committee (SC) Review ===
  
-When the size of the data in either \$HOME or \$DATA reaches **90% of the allowed quota**, a warning mail is sent. As with core-h, the actual value can be higher as the mail is sent some time after the threshold is reached.+Projects of types funded, internal (post-peer review), external, and application will be reviewed by the SC. 
 +  * For core-hour requests below 5 million, only the SC members of the requesting organization review the project. 
 +  * For requests above 5 million core-hours, the entire SC reviews the project. 
 +Test and private projects are only subject to technical review.
  
-Another mail is sent upon reaching **100%** - at that point, no more files can be created in the affected volume, preventing further work or sometimes even login. In that case, delete any files not needed and if that is not sufficient, request an extension. Any type of project can get an extension in size on \$DATA up to 100TB. Higher values are possible after discussion with us.  +You will receive an email notification once the project is approved.
-\$HOME cannot be extended, except for very special circumstances, such as more than 20 users in a project. Therefore, put all research data in $DATA.+
  
-**Note:** the volumes \$HOME and \$FILE are handled independently, so mails can be received for both.+==== Adding Users ====
  
-==== Project expiry ====+Once the project is granted, create cluster user accounts as follows: 
 +  * Enter the user's first and last name, email address, and phone number (must be capable of receiving text messages). 
 +     * If no SMS capable number can be provided, contact us for an alternative yubikey authorisation.  
 +  * Enter a unique username (only small letters and numbers or underscore are allowed) and a secure password (minimum of 10 characters, including one lowercase letter, one uppercase letter, one digit, and one special character). 
 +Assign the user to the relevant resource pools.
  
-**Expiry:** About a month before a project expires, you are sent a notification mail about said expiry. Another mail is sent on the day of expiry. As soon as a project is expired, it is still possible to work normally, including job submission. However, new user can no longer be created.+The user will receive an email with instructions (password not included).
  
-**Job submission disabled:** One month after project expiry, a mail is sent about job submissions being disabled. From that point on, jobs can no longer be submitted to the clusters. Jobs that are running at that time are not broken off.+Users may now log into the cluster and commence work on the project.
  
-**Backup relevant project data:** Two month after project expiry, a mail is sent to warn you to backup any data from your project that you want to keep. The mail says data retention is only guaranteed until three months after project expiry. For details about data see the next paragraph.+===== Managing Project Changes and Extensions =====
  
-**Login disabled:** Three month after job submission, a mail is sent informing about login being disabled. At that point, all user logins to the clusters belonging to that project are disabled. So far, all data belonging to a project was kept indefinitely after the project ran out - but this can not be guaranteed as total storage space might run out. It is therefore highly recommended to back up all relevant data before that point.+==== Project Changes ====
  
-If the project shall continue, an extension can be requested. That can even be done after the project has run out. Testprojects can only be extended for short times (30 - 60 days usually), other projects for a maximum of one year at a time.+**Personal Data Usage:** Update GDPR status as needed.
  
-**Auto-extension:** If a project (that is neither test nor private) has a large amount of core-h left available, it is automatically extended once for 6 month.+**Backup Status:** Toggle backup settings for the $DATA volume. $HOME is always backed up.
  
 +**File Uploads:** Add additional files (deletion or editing of existing files is not possible).
  
 +**User Accounts:** Manage user accounts as necessary.
  
 +**Resource Pools and Extensions:** Modify resource pools and request additional resources.
  
 +==== Project Extensions ====
  
 +To extend the project to additional parts of the system, select 'Request More Resources' at the top of the project part.
  
 +Specify which resource types you wish to extend and provide additional values if needed. Submit your request for approval.
  
 +==== Changes Performed by the VSC Team ====
  
 +Contact the VSC team for changes such as project manager, project name, and project group memberships.
 +
 +The VSC team can also add secretaries to the project, who have similar rights to the project manager.
 +
 +===== Project Lifecycle Notifications =====
 +
 +==== Core Hours ====
 +
 +**90% Usage:** A warning email will be sent.
 +
 +**100% Usage:** Another email will be sent, and the project will be set to idle priority, affecting job scheduling.
 +
 +**Extension:** Request an extension as necessary. Testprojects only get more core-h in special circumstances.
 +
 +==== Number of Files (independent for DATA and HOME) ====
 +
 +**90% Quota:** A warning email will be sent.
 +
 +**100% Quota:** A notification will be sent, preventing the creation of new files.
 +
 +**Extension:** Request additional file storage if needed.
 +
 +==== Volume Size (independent for DATA and HOME) ====
 +
 +**90% Quota:** A warning email will be sent.
 +
 +**100% Quota:** A notification will be sent, preventing further file creation.
 +
 +**Extension:** Request an extension for the DATA volume size if necessary. Not possible for HOME.
 +
 +==== Project Expiry ====
 +
 +**One Month Before Expiry:** A notification email will be sent.
 +
 +**Day of Expiry:** Another email will be sent. The project remains functional, but new users cannot be added.
 +
 +**One Month After Expiry:** A notification will inform you that job submissions are disabled. Currently running jobs will finish.
 +
 +**Two Months After Expiry:** A reminder email will prompt you to back up any data you wish to retain.
 +
 +**Three Months After Expiry:** User logins will be disabled. It is crucial to back up all relevant data before this deadline.
 +
 +**Extension:** Request an extension if required, even post-expiry. Test projects can only be extended for short periods (30-60 days), while other projects may be extended up to one year.
 +
 +**Auto-Extension:** Projects with significant remaining core hours will be automatically extended for six months to ensure uninterrupted access.
 +
 +For further assistance, please contact us via the [[https://service.vsc.ac.at|service page]].
  
  • doku/projects.txt
  • Last modified: 2024/08/01 11:40
  • by goldenberg