Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:jupyterhub [2022/07/19 10:44] – [Start a new Jupyter server] katrin | doku:jupyterhub [2024/09/19 16:52] (current) – katrin | ||
---|---|---|---|
Line 3: | Line 3: | ||
The VSC offers a JupyterHub service available for all VSC users at [[https:// | The VSC offers a JupyterHub service available for all VSC users at [[https:// | ||
- | Login works with any cluster user (both VSC3 and VSC4) and uses the OTP like on the cluster. A VPN connection is not needed. | + | Login works with any cluster user and uses the OTP like on the cluster. A VPN connection is not needed. |
Also make sure to checkout the [[doku: | Also make sure to checkout the [[doku: | ||
Line 26: | Line 26: | ||
* VSC-4 python venv | * VSC-4 python venv | ||
* VSC-4 conda python env | * VSC-4 conda python env | ||
- | * VSC-3 Singularity Image | + | * VSC-5 Singularity Image |
- | * VSC-3 GTX1080 | + | * VSC-5 A40 GPU Singularity Image |
- | * VSC-3 A40 GPU Singularity Image | + | * VSC-5 A100 GPU Singularity Image |
+ | * VSC-5 python venv | ||
+ | * VSC-5 conda python env | ||
+ | * VSC-5 A40 GPU conda python env | ||
+ | * VSC-5 A100 GPU Singularity Image | ||
* If you are participating in a training, there is most likely only a single fixed profile or special training profiles available | * If you are participating in a training, there is most likely only a single fixed profile or special training profiles available | ||
- | **Note:** You need a VSC4 user for the VSC4 profiles and a VSC3 user for the VSC3 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. | + | **Note:** You need a VSC4 user for the VSC4 profiles and a VSC5 user for the VSC5 profiles. You also need to have logged into the respective cluster via SSH at least once after getting a username. |
In all profiles, you can choose the IDE, either Jupyter Notebook or the more modern Jupyter Lab (which is the default) | In all profiles, you can choose the IDE, either Jupyter Notebook or the more modern Jupyter Lab (which is the default) | ||
Line 70: | Line 74: | ||
To stop the server, click on ' | To stop the server, click on ' | ||
- | ===== BYOI: Bring your own (Singularity) Image ===== | + | ===== BYOI: Bring your own (Apptainer/Singularity) Image ===== |
- | It is possible to use a custom singularity image with our JupyterHub profiles. | + | It is possible to use a custom |
- | A good starting point is to have a look at the official JupyterHub Docker Stacks Images @ [[https:// | + | ==== Necessary Packages ==== |
- | In a nutshell its necessary to install | + | A good starting point for creating your own container is the documentation of the official JupyterHub Docker Stacks Images @ [[https://github.com/jupyter/docker-stacks|Docker-Stacks]] |
- | Optionally: If your image needs to run hooks before starting | + | Apart from adding |
+ | < | ||
+ | # This package pulls in all the necessary dependencies to start a jupyter server | ||
+ | # Make sure this always matches | ||
+ | jupyterhub==3.1.1 | ||
- | Unfortunately the batchspawner | + | # This package |
+ | git+https://github.com/vsc-ac-at/batchspawner.git@1.2.0+auth_fix | ||
- | The script can be found at [[https://gitlab.tuwien.ac.at/vsc/ | + | # June 2024: fix a dependency to an earlier version |
+ | nbclassic==0.3.7 | ||
+ | </code> | ||
- | In addition we also install the following jupyterlab extensions in our images: | + | In addition we usually |
- | * jupyterlab-system-monitor | + | * jupyterlab-system-monitor |
- | * jupyterlab-git | + | * jupyterlab-git |
- | * jupyterlab-widgets | + | * jupyterlab-widgets |
- | An up 2 date list of packages can always be found in our repo: [[https:// | + | **An up 2 date list of packages can always be found in our repo: [[https:// |
- | In order to then use the image with JupyterHub you have to convert it to a singularity image and place it into a folder that is accessible | + | === Run hooks from / |
- | Building a singularity | + | If your image needs to run hooks before startup (e.g. the pyspark image depends on this [[https://github.com/jupyter/docker-stacks/ |
+ | Unfortunately the batchspawner package does not source them so we are using a custom startscript in our images called `vsc-singleuser.sh`. All it does is to run the hooks from `/ | ||
- | ===== FAQ ===== | + | The script can be found at [[https:// |
- | * **The OTP SMS says 'login at vsc3'? Is this an issue?** | + | Note: Contact us to get read rights to the repository. |
- | No this is a current | + | ==== Build your own custom image from a docker image ==== |
+ | |||
+ | [[https:// | ||
+ | |||
+ | If you already have a docker image you want to start from or if you are more familiar with docker image creation you can just use that image and create it into a singularity image after you are finished. | ||
+ | |||
+ | Note: Make sure that you use the right versions for the current | ||
+ | |||
+ | After selecting / building the docker image all that needs to be done is to convert it into a **singularity image**. | ||
+ | |||
+ | You can do this by executing the following lines (assuming your docker image is named " | ||
+ | < | ||
+ | singularity build my_image.sif docker:// | ||
+ | </ | ||
+ | |||
+ | See the documentation for more examples: [[https:// | ||
+ | |||
+ | When the conversion process has finished make sure that the resulting " | ||
+ | |||
+ | Note: the conversion can be done on our cluster since we have singularity installed on our nodes. | ||
+ | |||
+ | ==== Directly build a singularity/ | ||
+ | |||
+ | Instead of starting with docker you can also directly build a singularity image that is ready to use with our JupyterHub instance. | ||
+ | |||
+ | For this you need to create a so called " | ||
+ | |||
+ | Here is a minimal example using the datascience docker stacks image as a basis: | ||
+ | < | ||
+ | BootStrap: docker | ||
+ | From: jupyter/ | ||
+ | |||
+ | %post | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | If we save the file as " | ||
+ | |||
+ | Please | ||
+ | |||
+ | * VSC4: module load --auto apptainer/ | ||
+ | * VSC5: module load --auto apptainer/ | ||
+ | |||
+ | |||
+ | < | ||
+ | # load an apptainer module (see above) | ||
+ | module load --auto apptainer/< | ||
+ | |||
+ | # build the image | ||
+ | apptainer build my_image.sif my_image.def | ||
+ | </ | ||
+ | |||
+ | If you cannot use the versions provided on VSC you can of course also build the image on your own machine and upload it the VSC. | ||
+ | |||
+ | ===== FAQ ===== | ||
* **My Server instance is stuck and I get a timeout when I try to reload the window. Going to the VSC jupyterhub website also results in a timeout.** | * **My Server instance is stuck and I get a timeout when I try to reload the window. Going to the VSC jupyterhub website also results in a timeout.** |