Clone the repositories

Clone NERSC-HYCOM-CICE into ${HOME} and switch to the develop branch:

cd ${HOME}
git clone https://github.com/nansencenter/NERSC-HYCOM-CICE.git
cd NERSC-HYCOM-CICE
git fetch --all
git checkout develop

Clone the biogeochemical components alongside it:

cd ${HOME}
git clone https://github.com/fabm-model/fabm.git
git clone https://github.com/pmlmodelling/ersem.git
git clone https://github.com/nansencenter/nersc.git

Configure ~/.bashrc

Add the following to your ~/.bashrc on every machine you use:

export WORK=/cluster/projects/nn2993k/$USER

This sets $WORK to the non-purged projects filesystem used throughout the setup. See the directory structure section for details.

Python environment

A dedicated Python environment keeps the model’s Python dependencies isolated from other projects and from the system Python, and makes the setup reproducible across machines.

Set up conda

Betzy requires some one-time setup before conda can be used. On a standard workstation or on Olivia, skip ahead to Create the environment.

Betzy (NRIS/Sigma2)

Betzy only provides Miniforge3 (not Anaconda or Miniconda). Full details are in the Sigma2 conda documentation; the essential steps are:

1. Load and activate Miniforge3

module load Miniforge3/24.1.2-0
source ${EBROOTMINIFORGE3}/bin/activate

After sourcing, your prompt should show (base), confirming you are in the base conda environment. From here you can use conda to create and manage environments and install packages.

2. Configure conda directories

The default locations for package cache and environments are in ${HOME}, which has limited quota. Run the following once to redirect both to your project directory. Both settings are saved to ~/.condarc.

conda config --append pkgs_dirs /cluster/projects/nn2993k/conda/${USER}/package-cache
conda config --append envs_dirs /cluster/projects/nn2993k/conda/${USER}

Create the environment

Betzy (NRIS/Sigma2) and workstation

conda env create -f ${HOME}/NERSC-HYCOM-CICE/environment/python.yaml
conda activate hycom-cice

Then install the NERSC-specific libraries from the cloned repository:

pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/modeltools
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/modelgrid
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/gridxsec
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/abfile

To upgrade the NERSC libraries, add --upgrade to each pip install command.

Olivia (NRIS/Sigma2)

Olivia uses HPC-container-wrapper instead of conda directly. The wrapper encapsulates the environment inside a container, reducing the number of visible files and improving file-system performance.

1. Load the container wrapper

Run the following in your current session. These are only needed during the build, so there is no need to add them to ~/.bashrc.

export http_proxy=http://10.63.2.48:3128/
export https_proxy=http://10.63.2.48:3128/
module load NRIS/CPU
module load hpc-container-wrapper

2. Choose an installation path

Set a variable pointing to where the environment will live permanently. This directory must be kept after installation, as it contains the container and the executables:

export INSTALL_DIR=/cluster/projects/nn2993k/${USER}/hycom-cice-env

3. Build the environment

conda-containerize new --mamba \
    --prefix /cluster/projects/nn2993k/${USER}/hycom-cice-env \
    ${HOME}/NERSC-HYCOM-CICE/environment/python.yaml

4. Install the NERSC-specific libraries

Since the containerised environment cannot be modified directly, install the local libraries via a post-installation script:

cat > /tmp/nersc_libs.sh << 'EOF'
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/modeltools
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/modelgrid
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/gridxsec
pip install ${HOME}/NERSC-HYCOM-CICE/pythonlibs/abfile
EOF
conda-containerize update /cluster/projects/nn2993k/${USER}/hycom-cice-env --post-install /tmp/nersc_libs.sh

5. Activate the environment

Run the following to make the environment available in your current session:

export PATH="/cluster/projects/nn2993k/${USER}/hycom-cice-env/bin:${PATH}"

Run this command at the start of each session or job script where you need the environment. If you only use one Python environment, you can also add it to ~/.bashrc to activate it automatically.

Use the environment

The conda environment must be activated in any job or script that uses the model’s Python tools. On a standard workstation, conda activate hycom-cice is sufficient. On HPC systems the conda installation itself must be loaded first. On Olivia, the containerised environment is activated by prepending its bin directory to PATH.

Betzy (NRIS/Sigma2)

module load Miniforge3/24.1.2-0
source ${EBROOTMINIFORGE3}/bin/activate
conda activate hycom-cice

Olivia (NRIS/Sigma2)

Run the following at the start of each session or job script where you need the environment:

export PATH="/cluster/projects/nn2993k/${USER}/hycom-cice-env/bin:${PATH}"

Update the environment

Betzy (NRIS/Sigma2) and workstation

To sync the environment with changes to python.yaml:

conda activate hycom-cice
conda env update -f ${HOME}/NERSC-HYCOM-CICE/environment/python.yaml --prune

To add a package not in python.yaml:

conda activate hycom-cice
conda install <package>        # or: pip install <package>

Olivia (NRIS/Sigma2)

The containerised environment cannot be modified directly. Use conda-containerize update with a post-installation script listing the changes. First set the path to your existing environment and reload the modules:

export http_proxy=http://10.63.2.48:3128/
export https_proxy=http://10.63.2.48:3128/
module load NRIS/CPU
module load hpc-container-wrapper

Then create a script with the packages to add or remove, and apply it:

cat > /tmp/update.sh << 'EOF'
conda install -y <package>
pip install <package>
EOF
conda-containerize update /cluster/projects/nn2993k/${USER}/hycom-cice-env --post-install /tmp/update.sh

If python.yaml has changed substantially (e.g. a new core dependency was added), it is cleaner to rebuild from scratch by deleting the existing environment and repeating the Create the environment steps:

rm -rf /cluster/projects/nn2993k/${USER}/hycom-cice-env

HPC environment

Each machine has a dedicated HPC environment file under NERSC-HYCOM-CICE/environment/. Source it before compiling and include it in your job submission scripts before launching the model.

Source HPC environment — Betzy (NRIS/Sigma2)

source ${HOME}/NERSC-HYCOM-CICE/environment/betzy_env.sh

Source HPC environment — Olivia (NRIS/Sigma2)

source ${HOME}/NERSC-HYCOM-CICE/environment/olivia_env.sh

The dropdowns below show the full content of each *_env.sh for reference — what you source is exactly what is displayed.

View betzy_env.sh

# Environment for compiling/running HYCOM-CICE on betzy
ml purge
ml load ESMF/8.3.0-iomkl-2022a
ml load UDUNITS/2.2.28-GCCcore-11.3.0
ml load Python/3.10.4-GCCcore-11.3.0
ml load GSL/2.7-intel-compilers-2022.1.0
ml load FFTW/3.3.10-GCC-11.3.0
ml load CMake/3.23.1-GCCcore-11.3.0

ulimit -s 2000000

View olivia_env.sh

#!/usr/bin/bash

# Modules used to compile on Olivia
module reset
module load NRIS/CPU
module load GSL/2.8-intel-compilers-2024.2.0
module load ESMF/8.8.0-intel-2024a-ParallelIO-2.6.5
module load UDUNITS/2.2.28-GCCcore-13.3.0
module load FFTW/3.3.10-GCC-13.3.0
module load CMake/3.29.3-GCCcore-13.3.0