Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:jupyterhub [2021/11/08 19:46] – [Jupyterhub] goldenberg | doku:jupyterhub [2023/11/14 13:51] – [Necessary Packages] katrin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== | + | ====== |
- | The VSC offers a jupyterhub | + | The VSC offers a JupyterHub service |
Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed. | Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed. | ||
- | ===== start a server ===== | + | Also make sure to checkout the [[doku: |
+ | |||
+ | |||
+ | ===== Important Information: | ||
+ | |||
+ | The started Jupyter Server instance runs on a separate hardware partition (' | ||
+ | |||
+ | Since the machines in this partition are shared between all users it is important to be aware of the memory requirements of the processing you are doing. | ||
+ | |||
+ | In order to help the Jupyter users the VSC Team recently added a Memory Resources Monitor plugin to the JupyterLab IDE. It is displayed in the right upper corner of the JupyterLab IDE and shows the current memory consumption of the Jupyter Server instance. | ||
+ | |||
+ | If your job goes out of memory it will affect your job as well as other users running on the same node. This is a hardware/ | ||
+ | |||
+ | <box 80% round orange|Important> | ||
+ | |||
+ | ===== Start a new Jupyter | ||
choose a profile: | choose a profile: | ||
- | * VSC4 Singularity Image - a singularity image running on VSC4 (this is the default) | + | * VSC-4 Singularity Image (this is the default) |
- | * VSC3 Singularity Image - a singularity image running on VSC3+ | + | * VSC-4 python venv |
- | * VSC3 GPU nodes - a singularity image with CUDA support running on GPU nodes of VSC3+ (currently only GTX1080) | + | * VSC-4 conda python env |
- | * VSC4 python venv - a python virtual environment running directly on VSC4 | + | * VSC-3 Singularity Image |
- | * If you are participating in a course, there is most likely only a fixed profile available | + | * VSC-3 GTX1080 |
+ | * VSC-3 A40 GPU Singularity Image | ||
+ | |||
+ | * If you are participating in a training, there is most likely only a single | ||
**Note:** You need a VSC4 user for the VSC4 profiles and a VSC3 user for the VSC3 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. | **Note:** You need a VSC4 user for the VSC4 profiles and a VSC3 user for the VSC3 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. | ||
Line 20: | Line 38: | ||
For all the singularity profiles, you can choose between a predefined image (default) or a custom image: | For all the singularity profiles, you can choose between a predefined image (default) or a custom image: | ||
* Predefined image gives a dropdown list of different singularity images. Each image shows a short description, | * Predefined image gives a dropdown list of different singularity images. Each image shows a short description, | ||
- | * Custom image shows a line where the complete path to the image has to be entered. Note that the image has to reside on the cluster already. Web-upload is not possible. | + | * Custom image shows a line where the complete path to the image has to be entered. Note that the image has to reside on the cluster already. Web-upload is not possible |
- | **Note: | + | **Note: |
- | Independent of the profile you can choose the number of CPUs, size of the RAM, the maximum running time. | + | Independent of the profile you can choose the number of CPUs, size of the RAM and the maximum running time. |
After everything is selected, press start - your server will start and show you the JupyterLab interface or the simpler Notebook interface. In both cases, there is a list of all files in your home directory, which includes previously created notebooks. | After everything is selected, press start - your server will start and show you the JupyterLab interface or the simpler Notebook interface. In both cases, there is a list of all files in your home directory, which includes previously created notebooks. | ||
- | ===== JupyterLab ===== | + | |
+ | ==== Working with JupterLab IDE ==== | ||
Simply click on the desired notebook in your list - it will open in the main panel. | Simply click on the desired notebook in your list - it will open in the main panel. | ||
Line 36: | Line 55: | ||
It is possible to log out of the system and leave the server running with File -> Log Out. Keep in mind that this also keeps the selected resources blocked for other users. | It is possible to log out of the system and leave the server running with File -> Log Out. Keep in mind that this also keeps the selected resources blocked for other users. | ||
- | To stop the server, user File -> Hub Control Panel and choose 'Stop My Server' | + | To stop the server, user //File// -> //Hub Control Panel// |
+ | On the page that opens click the button labelled 'Stop My Server' | ||
- | ===== Jupyter Notebook ===== | ||
- | Simply | + | ==== Working with Jupyter Notebook IDE ==== |
+ | |||
+ | If you have chosen to work with the Jupyter Notebook IDE, simply | ||
On first use, no notebook is present yet. To create one (or to create new ones), click on ' | On first use, no notebook is present yet. To create one (or to create new ones), click on ' | ||
Line 49: | Line 70: | ||
To stop the server, click on ' | To stop the server, click on ' | ||
+ | ===== BYOI: Bring your own (Apptainer/ | ||
+ | |||
+ | It is possible to use a custom apptainer/ | ||
+ | |||
+ | ==== Necessary Packages ==== | ||
+ | |||
+ | A good starting point for creating your own container is the documentation of the official JupyterHub Docker Stacks Images @ [[https:// | ||
+ | |||
+ | Apart from adding your own software the image also needs to have at least the following packages for it to be able to run in our JupyterHub environment: | ||
+ | < | ||
+ | # This package pulls in all the necessary dependencies to start a jupyter server | ||
+ | # Make sure this always matches the current JupyterHub version used by VSC | ||
+ | jupyterhub==3.1.1 | ||
+ | |||
+ | # This package provides functionality needed to run in the slurm environment of VSC (e.g. `batchspawner-singleuser` script). | ||
+ | git+https:// | ||
+ | </ | ||
+ | |||
+ | In addition we usually also install the following jupyterlab extensions in our images but they are not strictly necessary and just provide extended functionality like memory monitoring for the user: | ||
+ | * jupyterlab-system-monitor | ||
+ | * jupyterlab-git | ||
+ | * jupyterlab-widgets (or jupyterlab_widgets in conda) | ||
+ | |||
+ | **An up 2 date list of packages can always be found in our repo: [[https:// | ||
+ | |||
+ | === Run hooks from / | ||
+ | |||
+ | If your image needs to run hooks before startup (e.g. the pyspark image depends on this [[https:// | ||
+ | |||
+ | Unfortunately the batchspawner package does not source them so we are using a custom startscript in our images called `vsc-singleuser.sh`. All it does is to run the hooks from `/ | ||
+ | |||
+ | The script can be found at [[https:// | ||
+ | |||
+ | Note: Contact us to get read rights to the repository. | ||
+ | |||
+ | ==== Build your own custom image from a docker image ==== | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | If you already have a docker image you want to start from or if you are more familiar with docker image creation you can just use that image and create it into a singularity image after you are finished. | ||
+ | |||
+ | Note: Make sure that you use the right versions for the current version of our JupyterHub (the version is displayed at the bottom of the [[http:// | ||
+ | |||
+ | After selecting / building the docker image all that needs to be done is to convert it into a **singularity image**. | ||
+ | |||
+ | You can do this by executing the following lines (assuming your docker image is named " | ||
+ | < | ||
+ | singularity build my_image.sif docker:// | ||
+ | </ | ||
+ | |||
+ | See the documentation for more examples: [[https:// | ||
+ | |||
+ | When the conversion process has finished make sure that the resulting " | ||
+ | |||
+ | Note: the conversion can be done on our cluster since we have singularity installed on our nodes. | ||
+ | |||
+ | ==== Directly build a singularity/ | ||
+ | |||
+ | Instead of starting with docker you can also directly build a singularity image that is ready to use with our JupyterHub instance. | ||
+ | |||
+ | For this you need to create a so called " | ||
+ | |||
+ | Here is a minimal example using the datascience docker stacks image as a basis: | ||
+ | < | ||
+ | BootStrap: docker | ||
+ | From: jupyter/ | ||
+ | |||
+ | %post | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | If we save the file as " | ||
+ | |||
+ | Please not that you have to use apptainer if you want to build on the cluster, since the singularity version that comes installed on all nodes from the OS needs root rights to build an image. | ||
+ | |||
+ | * VSC4: module load --auto apptainer/ | ||
+ | * VSC5: module load --auto apptainer/ | ||
+ | |||
+ | |||
+ | < | ||
+ | # load an apptainer module (see above) | ||
+ | module load --auto apptainer/< | ||
+ | |||
+ | # build the image | ||
+ | apptainer build my_image.sif my_image.def | ||
+ | </ | ||
+ | |||
+ | If you cannot use the versions provided on VSC you can of course also build the image on your own machine and upload it the VSC. | ||
+ | |||
+ | ===== FAQ ===== | ||
+ | |||
+ | * **My Server instance is stuck and I get a timeout when I try to reload the window. Going to the VSC jupyterhub website also results in a timeout.** | ||
+ | In order to to control the running instance, navigate directly to the following URL: [[https:// | ||