NCAR DTC CCPP-SCM

Downloading CCPP-SCM

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • How do I download CCPP-SCM?

Objectives

We will use GMU Hopper to run this model. We will now download the model and all the setup files needed for it. These are one-time steps.

Go to your /home/username/classes/clim670/ directory

$ cd /home/cstan/classes/clim670

Download the CCPP-SCM from Github

git clone --recursive -b v6.0.0 https://github.com/NCAR/ccpp-scm ccpp-scm-6.0

This will take a while …

Change to the ccpp-scm-6.0 directory

$ cd ccpp-scm-6.0

Let’s take a look in that directory. The recursive option in the git clone command clones the main ccpp-scm repository and all subrepositories (ccpp-physics and ccpp-framework). Using this option, there is no need to execute git submodule init and git submodule update.

Key Points

Building CCPP-SCM

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • How do I setup the SCM?

Objectives

You have now downloaded all the model components. Let’s take a look: From your /home/username/classes/clim670/ccpp-scm-6.0 directory, see what is there.

$ ls

Organization of the CCPP-SCM Directory

Using existing libraries

The Python environment must provide the f90nml module for the SCM scripts to function. Users can test if f90nml is installed using this command in the shell:

$ python -c "import f90nml"

If f90nml is installed, this command will succeed silently, otherwise an ImportError: No module named f90nml will be printed to screen. To install the f90nml (v0.19) Python module, use:

$ module load anaconda3
$ pip install --user f90nml ==0.19

Platform-specific scripts are provided to load modules and set the user environment for preconfigured platforms. These scripts load compiler modules (Fortran 2008-compliant), the NetCDF module, Python environment, etc. and set compiler and environment variables.

$ cd /home/username/classes/clim670/ccpp-scm-6.0/scm/etc/
$ ls

Hopper is not one of the preconfigured platforms. To get the corresponding file for Hopper:

$ cp /home/cstan/classes/clim670/ccpp-scm/scm/etc/Hopper_setup* .

Let’s take a look: From your /home/username/classes/clim670/ccpp-scm-6.0/scm/etc/ directory, see what is there.

$ ls

Now you have two new files that will configure the building environmeent for Hopper. One can be used for the t/csh shell and the other for the bash shell. From the top-level code directory (ccpp-scm-6.0), source the bash script for Hopper:

$ cd ../../
$ source scm/etc/Hopper_setup_gnu.sh

Ignore the warnings.

The first step in compiling the CCPP and SCM is to properly setup your user environment as described in sections above. The second step is to download the lookup tables and other large datasets (large binaries, <1 GB) needed by the physics schemes and place them in the correct directory. I have downloaded these files and you will create symbolic links:

$ cd /home/username/classes/clim670/ccpp-scm-6.0/scm/data/
$ ln -fs /home/cstan/classes/clim670/ccpp-scm/scm/data/comparison_data/
$ ln -fs /home/cstan/classes/clim670/ccpp-scm/scm/data/physics_input_data/
$ ln -fs /home/cstan/classes/clim670/ccpp-scm/scm/data/processed_case_input/
$ ln -fs /home/cstan/classes/clim670/ccpp-scm/scm/data/raw_case_input/

Above were the one-time setup setps, now we move on to the steps you will do everytime to setup a new model experiment and run it.

  1. From the top level code-directory (ccpp-scm-6.0), change directory to the top-level SCM directory:
$ cd scm
  1. Make a build directory and change into it
$ mkdir bin
$ cd bin
  1. Invoke cmake on the source code
$ cmake ../src

This will take a while.

What happened when we ran cmake?

We are now ready to compile. This step will create the executable.

$ make

What happened when we ran make?

Now what?

If your compilation has completed, then you are ready to run the case.

Key Points

Running CCPP-SCM

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • How do I setup a case and run the SCM?

Objectives

You have now build the model. There are several test cases provided with this version of the SCM. For all cases, the SCM will go through the time steps, applying forcing and calling the physics defined in the chosen suite definition file using physics configuration options from an associated namelist. The model is executed through a Python run script that is pre-staged into the bin directory: run_scm.py. It can be used to run one integration or several integrations serially, depending on the command line arguments supplied.

Running a case requires four pieces of information:

  1. The case to run (consisting of initial conditions, geolocation, forcing data, etc.),
  2. The physics suite to use (through a CCPP suite definition file),
  3. A physics namelist (that specifies configurable physics options to use), and
  4. A tracer configuration file.

Cases are set up via their own namelists in ../etc/case_config. A default physics suite is provided as a user-editable variable in the script and default namelists and tracer configurations are associated with each physics suite (through ../src/suite_info.py), so, technically, one must only specify a case to run with the SCM when running just one integration. For running multiple integrations at once, one need only specify one argument (-m) which runs through all permutations of supported suites from ../src/suite_info.py and cases from ../src/supported_cases.py.

The run script’s options are described below where option abbreviations are included in brackets.

--case [-c]

This or the --multirun option are the minimum required arguments. The case should correspond to the name of a case in ../etc/case_config (without the .nml extension).

--suite [-s]

The suite should correspond to the name of a suite in ../ccpp/suites (without the .xml) extension that was supplied in the cmake or ccpp_prebuild step.

--namelist [-n]

The namelist should correspond to the name of a file in ../ccpp/physics_namelists (WITH the .nml extension). If this argument is omitted, the default namelist for the given suite in ../src/suite_info.py will be used.

--tracers [-t]

The tracers file should correspond to the name of a file in ../etc/tracer_config (WITH the .txt extension). If this argument is omitted, the default tracer configuration for the given suite in ../src/suite_info.py will be used.

--multirun [-m]

This or the –case option are the minimum required arguments. When used alone, this option runs through all permutations of supported suites from ../src/suite_info.py and cases from ../src/supported_cases.py. When used in conjunction with the – file option, only the runs configured in the file will be run.

--file [-f]

This option may be used in conjunction with the –multirun argument. It specifies a path and filename to a python file where multiple runs are configured.

--gdb [-g]

Use this to run the executable through the gdb debugger (if it is installed on the system).

--docker [-d]

Use this argument when running in a docker container in order to successfully mount a volume between the host machine and the Docker container instance and to share the output and plots with the host machine.

--runtime

Use this to override the runtime provided in the case configuration namelist.

--runtime_mult

Use this to override the runtime provided in the case configuration namelist by multiplying the runtime by the given value. This is used, for example, in regression testing to reduce total runtimes.

--levels [-l]

Use this to change the number of vertical levels.

--npz_type

Use this to change the type of FV3 vertical grid to produce (see src/scm_vgrid.F90 for valid values).

--vert_coord_file

Use this to specify the path/filename of a file containing the a_k and b_k coefficients for the vertical grid generation code to use.

--bin_dir

Use this to specify the path to the build directory.

--run_dir

Use this to specify the path to the run directory.

--case_data_dir

Use this to specify the path to the directory containing the case data file (useful for using the DEPHY case repository).

--n_itt_out

Use this to specify the period of writing instantaneous output in timesteps (if different than the default specified in the script).

--n_itt_diagt

Use this to specify the period of writing instantaneous and time-averaged diagnostic output in timesteps (if different than the default specified in the script).

--timestep [-dt]

Use this to specify the timestep to use (if different than the default specified in ../src/suite_info.py).

--verbose [-v]

Use this option to see additional debugging output from the run script and screen output from the executable.

When invoking the run script, the only required argument is the name of the case to run. The case name used must match one of the case configuration files located in ../etc/case_config (without the .nml extension!). If specifying a suite other than the default, the suite name used must match the value of the suite name in one of the suite definition files located in ../../ccpp/suites (Note: not the filename of the suite definition file).

As part of the sixth CCPP release, the following suite names are valid:

  1. SCM_GFS_v16
  2. SCM_GFS_v17p8
  3. SCM_RAP
  4. SCM_HRRR
  5. SCM_RRFS_v1beta
  6. SCM_WoFS_v0

Let’s create a case. From the top level code-directory (ccpp-scm-6.0), change directory to the top-level SCM directory and create a directory were to run all cases and then a directory for each case

$ mkdir cases
$ cd cases
$ mkdir gfs_v16_bomex

Create a script (e.g., create_gfs_v16_bomex_case.sh) with the following

#!/bin/bash

export case=bomex
export suite=SCM_GFS_v16
export namelist=input_GFS_v16.nml
export RUN_TIME=86400
export ITT_OUT=1
export BIN_DIR=/home/cstan/scm_sandbox/ccpp-scm-6.0/scm/bin
export RUN_DIR=/scratch/cstan/clim670/ccpp-scm-6.0  # Make sure RUN_DIR exists

python ${BIN_DIR}/run_scm.py -c ${case} -s ${suite} -n ${namelist} --runtime ${RUN_TIME} --n_itt_out ${ITT_OUT} --bin_dir ${BIN_DIR} --run_dir ${RUN_DIR}

Make sure the script is executable. To check

$ ls -l create_gfs_v16_bomex_case.sh

-rwxr--r-- 1 cstan users 456 Feb  5 17:10 create_gfs_v16_bomex_case.sh

If the sequence -rwxr--r-- does not have the execute (x) permission we can add it:

$ chmod u+x create_gfs_v16_bomex_case.sh

Check if your script has the execute permission

Does your file has the execute (x) permsion?

Before execturing the script, we need to set the user environment for Hopper.

From the top-level code directory (ccpp-scm-6.0), source the bash script for Hopper:

$ source scm/etc/Hopper_setup_gnu.sh

Now we are ready to strat running our gfs_v16_bomex_case.

$ cd scm/cases/gfs_v16_bomex/
$ ./create_gfs_v16_bomex_case.sh

A NetCDF output file is generated in a directory located in the RUN_DIR.

$ cd /scratch/cstan/clim670/ccpp-scm-6.0/output_bomex_SCM_GFS_v16
$ ls

bomex_SCM_GFS_v16.nml  logfile  output.nc

The output.nc contains the output written with the frequency set by the --n_itt variable.

What is in this file?

We will look at the file using ncdump -h to understand what is in the file.

What variables are in the file?

How many times are in the output file?

We can read the file using Python xarray. This is a small file.

The namlist file .nml contains the configuration namelist that contains parameters for the SCM infrastructure and the physics configuration namelist.

The case_config namelist expects the following parameters:

The physics_config expects the following parameters:

Key Points