{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using Climada on the Euler Cluster (ETH internal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Access to Euler\n", "\n", "See https://scicomp.ethz.ch/wiki/Getting_started_with_clusters for details on how to register at and get started with Euler.\n", "\n", "For all steps below, first enter the Cluster via SSH." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "\n", "## Installation directory and working directory\n", "\n", "Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems.
\n", "As a general rule: use `/cluster/project` for installation and `/cluster/work` for data processing.\n", "\n", "For ETH WCR group members, the suggested installation and working directories are `/cluster/project/climate/$USER` and `/cluster/work/climate/$USER` respectively.
\n", "You may have to create the installation directory:\n", "\n", "```bash\n", "mkdir -p /cluster/project/climate/$USER \\\n", " /cluster/work/climate/$USER\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Climada installation in a virtual environment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### 1. Load dependencies\n", "\n", "```bash\n", "module load \\\n", " gcc/12.2.0 \\\n", " stack/2024-06 \\\n", " python/3.11.6 \\\n", " hdf5/1.14.3 \\\n", " geos/3.9.1 \\\n", " sqlite/3.43.2 \\\n", " eccodes/2.25.0 \\\n", " gdal/3.6.3 \\\n", " eth_proxy\n", "\n", "module load proj\n", "module unload proj\n", "```\n", "\n", "(The last two lines may seem odd but they are working around a conficting dependency version situation.)\n", "\n", "You need to execute this every time you login to Euler before Climada can be used.\n", "To safe yourself from doing it manually, append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler." ] }, { "cell_type": "markdown", "id": "4dad7c27", "metadata": {}, "source": [ "\n", "### 2. Create and prepare virtual environment\n", "\n", "```bash\n", "envname=climada_env\n", "\n", "# create environment\n", "python -m venv --system-site-packages /cluster/project/climate/$USER/venv/$envname\n", "\n", "# acitvate it\n", ". /cluster/project/climate/$USER/venv/$envname/bin/activate\n", "\n", "# install python kernel (to be used in JupyterHub, s.b.)\n", "python -m ipykernel install --user --name $envname\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Install dependencies\n", "\n", "```bash\n", "pip install \\\n", " --no-binary fiona \\\n", " --no-binary rasterio \\\n", " pyproj==3.7.1 \\\n", " fiona \\\n", " rasterio\n", "\n", "pip install \\\n", " 'numpy<2' \\\n", " netcdf4 \\\n", " gdal==3.6.3 \\\n", " geopandas \\\n", " xarray \\\n", " cartopy \\\n", " sparse \\\n", " salib \\\n", " pyarrow \\\n", " xlrd\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Install Climada\n", "\n", "There are two options. Either install from the downloaded repository (option A), or use a particular released version (option B).\n", "\n", "#### option A\n", "```bash\n", "cd /cluster/project/climate/$USER # or wherever you plan to download the repository\n", "git clone https://github.com/CLIMADA-project/climada_python.git # unless this has been done before\n", "cd climada_python\n", "pip install -e .\n", "```\n", "If you need to work with a specific branch of Climada, you can do so by checking out to the target branch `your_branch` by running `git checkout your_branch` after having cloned the Climada repository and before running `pip install -e .`.\n", "\n", "#### option B\n", "\n", "```bash\n", "pip install climada=5.0\n", "```\n", "\n", "or whatever version you prefer" ] }, { "cell_type": "markdown", "id": "479e0ac2", "metadata": {}, "source": [ "\n", "### 5. Adjust the Climada configuration\n", "\n", "Edit a configuration file according to your needs (see [Configuration](../development/Guide_Configuration.ipynb)).\n", "Create a climada.conf file e.g., in /cluster/home/$USER/.config with the following content:\n", "\n", "```json\n", "{\n", " \"local_data\": {\n", " \"system\": \"/cluster/work/climate/USERNAME/climada/data\",\n", " \"demo\": \"/cluster/project/climate/USERNAME/climada/data/demo\",\n", " \"save_dir\": \"/cluster/work/climate/USERNAME/climada/results\"\n", " }\n", "}\n", "```\n", "(Replace USERNAME with your nethz-id.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### 6. Test the installation\n", "\n", "Check installation in login node:\n", "\n", "```bash\n", "python -m unittest climada.engine.test.test_impact_calc\n", "```\n", "\n", "This should prompt the usual \"`OK`\" in the end.\n", "Once that succeeded you may want to test the installation also in a compute node, just for the sake of it:\n", "\n", "```bash\n", "sbatch --wrap=\"python -m unittest climada.engine.test.test_impact_calc\"\n", "```\n", "\n", "Look for the \"`OK`\" in the hereby created `slurm-[XXXXXXX].out` file\n", "\n", "Please see the docs at https://slurm.schedmd.com/ on how to use the `slurm` batch system\n", "and the Wiki https://scicomp.ethz.ch/wiki/Transition_from_LSF_to_Slurm for a mapping of `lsf` commands to their `slurm` equivalents." ] }, { "cell_type": "markdown", "id": "e5c90bff", "metadata": {}, "source": [ "### 7. Optional: Install Climada Petals\n", "\n", "To install Climada Petals, repeat the steps described in step 4A, but replacing the climada_python with the climada_petals repository:\n", "```bash\n", "cd /cluster/project/climate/$USER # or wherever you plan to download the repository\n", "git clone https://github.com/CLIMADA-project/climada_petals.git # unless this has been done before\n", "cd climada_petals\n", "pip install -e .\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Run a Jupyter Notebook on Euler" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job.\n", "See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.\n", "\n", "For using climada inside the jupyter notebook, you need to create a customized `jupyterlabrc` file by running the following lines:\n", "\n", "```bash\n", "mkdir -p ~/.config/euler/jupyterhub\n", "cat > ~/.config/euler/jupyterhub/jupyterlabrc <