Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:jupyterhub [2023/03/22 18:58] – [FAQ] katrin | doku:jupyterhub [2024/09/19 16:52] (current) – katrin | ||
---|---|---|---|
Line 3: | Line 3: | ||
The VSC offers a JupyterHub service available for all VSC users at [[https:// | The VSC offers a JupyterHub service available for all VSC users at [[https:// | ||
- | Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed. | + | Login works with any cluster user and uses the OTP like on the cluster. A VPN connection is not needed. |
Also make sure to checkout the [[doku: | Also make sure to checkout the [[doku: | ||
Line 26: | Line 26: | ||
* VSC-4 python venv | * VSC-4 python venv | ||
* VSC-4 conda python env | * VSC-4 conda python env | ||
- | * VSC-3 Singularity Image | + | * VSC-5 Singularity Image |
- | * VSC-3 GTX1080 | + | * VSC-5 A40 GPU Singularity Image |
- | * VSC-3 A40 GPU Singularity Image | + | * VSC-5 A100 GPU Singularity Image |
+ | * VSC-5 python venv | ||
+ | * VSC-5 conda python env | ||
+ | * VSC-5 A40 GPU conda python env | ||
+ | * VSC-5 A100 GPU Singularity Image | ||
* If you are participating in a training, there is most likely only a single fixed profile or special training profiles available | * If you are participating in a training, there is most likely only a single fixed profile or special training profiles available | ||
- | **Note:** You need a VSC4 user for the VSC4 profiles and a VSC3 user for the VSC3 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. | + | **Note:** You need a VSC4 user for the VSC4 profiles and a VSC5 user for the VSC5 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. |
In all profiles, you can choose the IDE, either Jupyter Notebook or the more modern Jupyter Lab (which is the default) | In all profiles, you can choose the IDE, either Jupyter Notebook or the more modern Jupyter Lab (which is the default) | ||
Line 70: | Line 74: | ||
To stop the server, click on ' | To stop the server, click on ' | ||
- | ===== BYOI: Bring your own (Singularity) Image ===== | + | ===== BYOI: Bring your own (Apptainer/Singularity) Image ===== |
- | It is possible to use a custom singularity image with our JupyterHub profiles. | + | It is possible to use a custom |
- | A good starting point is to have a look at the official JupyterHub Docker Stacks Images @ [[https:// | + | ==== Necessary Packages ==== |
- | In a nutshell it is only necessary | + | A good starting point for creating your own container |
+ | |||
+ | Apart from adding your own software the image also needs to have at least the following packages | ||
< | < | ||
- | jupyterhub==3.0.0 | + | # This package pulls in all the necessary dependencies to start a jupyter server |
- | git+https:// | + | # Make sure this always matches the current JupyterHub version used by VSC |
+ | jupyterhub==3.1.1 | ||
+ | |||
+ | # This package provides functionality needed to run in the slurm environment of VSC (e.g. `batchspawner-singleuser` script). | ||
+ | git+https:// | ||
+ | |||
+ | # June 2024: fix a dependency to an earlier version | ||
+ | nbclassic==0.3.7 | ||
</ | </ | ||
- | In addition we also install the following jupyterlab extensions in our images but they are not strictly necessary and just provide extended functionality like memory monitoring for the user: | + | In addition we usually |
* jupyterlab-system-monitor | * jupyterlab-system-monitor | ||
* jupyterlab-git | * jupyterlab-git | ||
- | * jupyterlab-widgets | + | * jupyterlab-widgets |
- | **An up 2 date list of packages can always be found in our repo: [[https:// | + | **An up 2 date list of packages can always be found in our repo: [[https:// |
- | In order to be able to use the image with JupyterHub you also have to make sure to convert it to a **singularity image** (or build it as a singularity image in the first place) and place it into a folder that is accessible from the VSC compute nodes. | + | === Run hooks from / |
- | + | ||
- | Building a singularity image from e.g. a docker container can easily be done with the `singularity build` command. See the singularity documentation for examples: [[https:// | + | |
- | + | ||
- | ==== Run hooks from / | + | |
If your image needs to run hooks before startup (e.g. the pyspark image depends on this [[https:// | If your image needs to run hooks before startup (e.g. the pyspark image depends on this [[https:// | ||
Line 99: | Line 108: | ||
Unfortunately the batchspawner package does not source them so we are using a custom startscript in our images called `vsc-singleuser.sh`. All it does is to run the hooks from `/ | Unfortunately the batchspawner package does not source them so we are using a custom startscript in our images called `vsc-singleuser.sh`. All it does is to run the hooks from `/ | ||
- | The script can be found at [[https:// | + | The script can be found at [[https:// |
Note: Contact us to get read rights to the repository. | Note: Contact us to get read rights to the repository. | ||
+ | ==== Build your own custom image from a docker image ==== | ||
+ | [[https:// | ||
+ | |||
+ | If you already have a docker image you want to start from or if you are more familiar with docker image creation you can just use that image and create it into a singularity image after you are finished. | ||
+ | |||
+ | Note: Make sure that you use the right versions for the current version of our JupyterHub (the version is displayed at the bottom of the [[http:// | ||
+ | |||
+ | After selecting / building the docker image all that needs to be done is to convert it into a **singularity image**. | ||
+ | |||
+ | You can do this by executing the following lines (assuming your docker image is named " | ||
+ | < | ||
+ | singularity build my_image.sif docker:// | ||
+ | </ | ||
+ | |||
+ | See the documentation for more examples: [[https:// | ||
+ | |||
+ | When the conversion process has finished make sure that the resulting " | ||
+ | |||
+ | Note: the conversion can be done on our cluster since we have singularity installed on our nodes. | ||
+ | |||
+ | ==== Directly build a singularity/ | ||
+ | |||
+ | Instead of starting with docker you can also directly build a singularity image that is ready to use with our JupyterHub instance. | ||
+ | |||
+ | For this you need to create a so called " | ||
+ | |||
+ | Here is a minimal example using the datascience docker stacks image as a basis: | ||
+ | < | ||
+ | BootStrap: docker | ||
+ | From: jupyter/ | ||
+ | |||
+ | %post | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | If we save the file as " | ||
+ | |||
+ | Please not that you have to use apptainer if you want to build on the cluster, since the singularity version that comes installed on all nodes from the OS needs root rights to build an image. | ||
+ | |||
+ | * VSC4: module load --auto apptainer/ | ||
+ | * VSC5: module load --auto apptainer/ | ||
+ | |||
+ | |||
+ | < | ||
+ | # load an apptainer module (see above) | ||
+ | module load --auto apptainer/< | ||
+ | |||
+ | # build the image | ||
+ | apptainer build my_image.sif my_image.def | ||
+ | </ | ||
+ | If you cannot use the versions provided on VSC you can of course also build the image on your own machine and upload it the VSC. | ||
===== FAQ ===== | ===== FAQ ===== |