Submit a job

The job script srjob.sh handles three things automatically:

• Atmospheric forcing generation — downloads and prepares ERA5 forcing for the run period

• Stage run files — runs expt_preprocess.sh to stage all files the model needs into the scratch directory (expt_<EXPT_ID>/SCRATCH/). Specifically, it:

  • Reads and validates settings from blkdat.input (time steps, flags, experiment number)

  • Copies the MPI partition file (topo/partit/depth_R_T.NNNN, where NNNN is the MPI task count) to patch.input in the scratch directory

  • Symlinks or copies all forcing files (atmospheric, river, kpar, relaxation climatologies, diffusion fields, nesting files)

  • Verifies that atmospheric forcing covers the full run period

  • Copies restart files (HYCOM and CICE) from the data directory

  • Copies the hycom_cice executable from build/

  • Writes the limits file with start and stop times via hycom_limits.py

  • Moves old output files to KEEP/

  It exits with a non-zero code if any required file is missing, aborting the job before the model starts.

Warning

As all the files needed are staged into the scratch directory (expt_<EXPT_ID>/SCRATCH/), where the model runs, expt_preprocess.sh should be run for every new build or change in the files mentioned above. Otherwise, the previous versions of the files remain staged. By default, this step is handled automatically by the job script srjob.sh.

Run manually

cd $WORK/<CONFIGNAME>/expt_<EXPT_ID>
../bin/expt_preprocess.sh ${START} ${END} $INITFLG

Use the same START, END, and INITFLG values as in srjob.sh (see the table below for formats). A successful run prints No fatal errors. Ok to start model set up in ...

Running expt_preprocess.sh manually beforehand is useful to verify that all required files are in place before submitting the job. If you do, comment out the expt_preprocess.sh call in srjob.sh to avoid staging the files twice.

• Job submission — submits the model to the SLURM queue

• Post-processing — runs expt_postprocess.sh after the model finishes. It moves restart files, daily mean archives (archv.*), and CICE output from the scratch directory to data/ and data/cice/, and writes log/hycom.stop with GOODRUN or BADRUN depending on whether the model reached a normal stop. The data/ directory is then ready for analysis with the MSCPROGS post-processing tools.

Open srjob.sh in a text editor and update:

Variable	Value	Description
NMPI	"<NMPI>" e.g. 504	Update NMPI if needed
START	"YYYY-MM-DDT00:00:00"	Run start time
END	"YYYY-MM-DDT00:00:00"	Run end time
INITFLG	"" or "--init"	"" for a restart run; "--init" to initialize from climatology
#SBATCH --time	"HH:MM:SS"	Wall-clock time limit

INITFLG="--init" (climatological initialization): No restart files are needed. Temperature and salinity (T/S) are set directly from the climatological fields in relax/ (see Climatologies and river forcing); velocities and sea surface height (SSH) start at zero. The model then spins up under realistic atmospheric forcing. The start date must be in September (the month of Arctic sea ice minimum). See Initial conditions for details on both options.

Note

For reference when setting #SBATCH --time: a 1-year TP2 run with BGC on 4 Betzy nodes (504 cores) takes approximately 5–6 hours of wall time.

Then submit:

cd $WORK/<CONFIGNAME>/expt_<EXPT_ID>
sbatch srjob.sh

Check job status with:

squeue -u $USER

A useful tool for generating SLURM job scripts is available at: https://open.pages.sigma2.no/job-script-generator/

Check results

When the job finishes, restart files and daily mean files are moved to the data/ directory. Confirm successful completion by checking the stop file:

cat $WORK/<CONFIGNAME>/expt_<EXPT_ID>/log/hycom.stop

The file should contain GOODRUN. If not, inspect the log files under log/ for error messages.

Restarting after a crash

What happens to output files depends on how the run ended:

• Model exits with an error but the SLURM job completes normally (e.g. a model error or failed assertion): expt_postprocess.sh still runs, so restart and archive files are moved to data/ and log/hycom.stop contains BADRUN. The restart files written before the crash are already in data/ and can be used directly.

• Job is killed by SLURM (wall-time limit, out-of-memory, node failure, or scancel): expt_postprocess.sh never runs. Output files remain in SCRATCH/. Run postprocessing manually before resubmitting:

  cd $WORK/<CONFIGNAME>/expt_<EXPT_ID>
  ../expt_postprocess.sh
  

  This moves the restart and archive files written before the crash to data/, where srjob.sh will pick them up on the next run. If you skip this step and resubmit directly, expt_preprocess.sh will move the files to SCRATCH/KEEP/ instead — they are not lost, but you will need to copy them to data/ manually.

In both cases, update START in srjob.sh to the date of the last restart file written, and set INITFLG="" (restart run).

Visualization

If you prefer to work with xarray for analysis and visualization, check out xhycom.

Otherwise, sample Jupyter notebooks for plotting HYCOM output are also available in the TP2_setup repository.

These notebooks read files from $WORK/<CONFIGNAME>/expt_<EXPT_ID>/data/.