Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
doku:jupyterhub [2022/02/11 12:35]
katrin [Important Information: Resource Allocation]
doku:jupyterhub [2023/11/14 13:51] (current)
katrin [Necessary Packages]
Line 1: Line 1:
-====== Jupyterhub ======+====== JupyterHub ======
  
-The VSC offers a JupyterHub service available for all VSC users at [[https://vsc.ac.at/jupyterhub]]+The VSC offers a JupyterHub service available for all VSC users at [[https://jupyterhub.vsc.ac.at]]
  
 Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed. Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed.
Line 24: Line 24:
 choose a profile: choose a profile:
   * VSC-4 Singularity Image (this is the default)   * VSC-4 Singularity Image (this is the default)
 +  * VSC-4 python venv
 +  * VSC-4 conda python env
   * VSC-3 Singularity Image   * VSC-3 Singularity Image
-  * VSC-3 GPU nodes - a singularity image with CUDA support (currently only GTX1080) +  * VSC-3 GTX1080 GPU Singularity Image 
-  * VSC-4 python venv - a python virtual environment+  * VSC-3 A40 GPU Singularity Image 
   * If you are participating in a training, there is most likely only a single fixed profile or special training profiles available   * If you are participating in a training, there is most likely only a single fixed profile or special training profiles available
  
Line 67: Line 70:
 To stop the server, click on 'Control Panel' at the top right and choose 'Stop My Server' To stop the server, click on 'Control Panel' at the top right and choose 'Stop My Server'
  
-===== FAQ =====+===== BYOI: Bring your own (Apptainer/Singularity) Image =====
  
-  * **The OTP SMS says 'login at vsc3'? Is this an issue?**+It is possible to use a custom apptainer/singularity image with our JupyterHub profiles. 
  
-No this is a current limitation of our OTP system and should not cause any problems.+==== Necessary Packages ==== 
 + 
 +A good starting point for creating your own container is the documentation of the official JupyterHub Docker Stacks Images @ [[https://github.com/jupyter/docker-stacks|Docker-Stacks]]  
 + 
 +Apart from adding your own software the image also needs to have at least the following packages for it to be able to run in our JupyterHub environment: 
 +<code> 
 +# This package pulls in all the necessary dependencies to start jupyter server 
 +# Make sure this always matches the current JupyterHub version used by VSC 
 +jupyterhub==3.1.1         
 + 
 +# This package provides functionality needed to run in the slurm environment of VSC (e.g. `batchspawner-singleuser` script). 
 +git+https://github.com/katringoogoo/batchspawner.git@1.2.0+auth_fix         
 +</code> 
 + 
 +In addition we usually also install the following jupyterlab extensions in our images but they are not strictly necessary and just provide extended functionality like memory monitoring for the user: 
 +  * jupyterlab-system-monitor 
 +  * jupyterlab-git  
 +  * jupyterlab-widgets (or jupyterlab_widgets in conda) 
 + 
 +**An up 2 date list of packages can always be found in our repo: [[https://gitlab.tuwien.ac.at/vsc/jupyterhub/jupyterhub-docker-stacks/-/blob/main/tools/requirements.txt|requirements.txt]]** 
 + 
 +=== Run hooks from /usr/local/bin/before-notebook.d === 
 + 
 +If your image needs to run hooks before startup (e.g. the pyspark image depends on this [[https://github.com/jupyter/docker-stacks/blob/master/pyspark-notebook/Dockerfile|PySpark Dockerfile]]), the docker stack images provide a folderfor such startup scripts `/usr/local/bin/before-notebook.d`.  
 + 
 +Unfortunately the batchspawner package does not source them so we are using a custom startscript in our images called `vsc-singleuser.sh`. All it does is to run the hooks from `/usr/local/bin/before-notebook.d` before executing the `batchspawner-singleuser` script. 
 + 
 +The script can be found at [[https://gitlab.tuwien.ac.at/vsc/vsc-jupyterhub-notebooks/-/blob/main/tools/vsc-singleuser.sh|vsc-singleuser.sh]].  
 + 
 +Note: Contact us to get read rights to the repository. 
 + 
 +==== Build your own custom image from a docker image ==== 
 + 
 +[[https://docs.docker.com/engine/reference/builder/|Dockerfile Documentation]] 
 + 
 +If you already have a docker image you want to start from or if you are more familiar with docker image creation you can just use that image and create it into a singularity image after you are finished. 
 + 
 +Note: Make sure that you use the right versions for the current version of our JupyterHub (the version is displayed at the bottom of the [[http://jupyterhub.vsc.ac.at|jupyterhub page]] and was 3.1.1 at the time of writing). 
 + 
 +After selecting / building the docker image all that needs to be done is to convert it into a **singularity image**.  
 + 
 +You can do this by executing the following lines (assuming your docker image is named "my_image"
 +<code> 
 +singularity build my_image.sif docker://my_image 
 +</code> 
 + 
 +See the documentation for more examples: [[https://apptainer.org/user-docs/3.8/build_a_container.html|Build a (singularity) container]] 
 + 
 +When the conversion process has finished make sure that the resulting ".sif" file is placed in a folder that is available from all compute nodes (e.g. home dir; data dir) 
 + 
 +Note: the conversion can be done on our cluster since we have singularity installed on our nodes.  
 + 
 +==== Directly build a singularity/apptainer image ==== 
 + 
 +Instead of starting with docker you can also directly build a singularity image that is ready to use with our JupyterHub instance. 
 + 
 +For this you need to create a so called ".def" file - see the singularity documentation for more information on this format: [[https://apptainer.org/user-docs/3.8/definition_files.html|Definition Files]] 
 + 
 +Here is a minimal example using the datascience docker stacks image as a basis: 
 +<code> 
 +BootStrap: docker 
 +From: jupyter/datascience-notebook:hub-3.1.1 
 + 
 +%post 
 +/opt/conda/bin/pip install jupyterhub==3.1.1 git+https://github.com/katringoogoo/batchspawner.git@1.2.0+auth_fix                
 +</code> 
 + 
 +If we save the file as "my_image.def" we can use the build command to build the image with singularity/apptainer.  
 + 
 +Please not that you have to use apptainer if you want to build on the cluster, since the singularity version that comes installed on all nodes from the OS needs root rights to build an image. 
 + 
 +  * VSC4: module load --auto apptainer/1.1.9-gcc-12.2.0-vflkcfi 
 +  * VSC5: module load --auto apptainer/1.1.6-gcc-12.2.0-xxfuqni 
 + 
 + 
 +<code> 
 +# load an apptainer module (see above) 
 +module load --auto apptainer/<version> 
 + 
 +# build the image 
 +apptainer build my_image.sif my_image.def 
 +</code> 
 + 
 +If you cannot use the versions provided on VSC you can of course also build the image on your own machine and upload it the VSC. 
 + 
 +===== FAQ =====
  
   * **My Server instance is stuck and I get a timeout when I try to reload the window. Going to the VSC jupyterhub website also results in a timeout.**   * **My Server instance is stuck and I get a timeout when I try to reload the window. Going to the VSC jupyterhub website also results in a timeout.**
  
-In order to to control the running instance, navigate directly to the following URL: [[https://vsc.ac.at/jupyterhub/hub/home]]. From there you can click 'Stop My Server' to stop the running instance (if there is any).+In order to to control the running instance, navigate directly to the following URL: [[https://jupyterhub.vsc.ac.at/hub/home]]. From there you can click 'Stop My Server' to stop the running instance (if there is any).
  
  
  • doku/jupyterhub.1644582906.txt.gz
  • Last modified: 2022/02/11 12:35
  • by katrin