CESM Output Overview
Overview
Teaching: 0 min
Exercises: 0 minQuestions
What kinds of output do I get after the model runs?
Objectives
Last week we setup and ran our first CESM experiment. What did it do? How do I find and interpret the output from it? There are several types of output form our model simulation:
- Information in the
CaseStatusfile - Standard error and standard out
- Log files
- Model output data
In this episode, we will take a look at each of these and understand what they tell us.
Key Points
CaseStatus
Overview
Teaching: 0 min
Exercises: 0 minQuestions
What does the CaseStatus file tell me about my model experiment?
Objectives
The CaseStatus file
Let’s start by going to the CASEROOT directory for your first experiment and looking at the CaseStatus file:
$ cd ~/cases/b.day1.0
$ cat CaseStatus
---------------------------------------------------
2023-02-20 10:25:32: case.submit starting
---------------------------------------------------
2023-02-20 10:25:40: case.submit success case.run:8671655.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:8671656.chadmin1.ib0.cheyenne.ucar.edu
---------------------------------------------------
2023-02-25 19:05:16: case.submit starting
---------------------------------------------------
2023-02-25 19:05:25: case.submit success case.run:8732081.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:8732082.chadmin1.ib0.cheyenne.ucar.edu
---------------------------------------------------
2023-02-25 19:05:29: case.run starting
---------------------------------------------------
2023-02-25 19:05:37: model execution starting
---------------------------------------------------
2023-02-25 19:08:09: model execution success
---------------------------------------------------
2023-02-25 19:08:09: case.run success
---------------------------------------------------
2023-02-25 19:08:16: st_archive starting
---------------------------------------------------
2023-02-25 19:08:24: st_archive success
---------------------------------------------------
From this we can see model execution success indicates the model ran successfully and st_archive_success indicates that the archiving job ran successfully. Remember that when we submitted the model, two jobs were sent to the queue, the model run and the archive job. Both appear to have run successfully.
Troubleshooting
Does anyone have a
CaseStatusfile that indicates the model did not run successfully?
Key Points
Standard error and standard out
Overview
Teaching: 0 min
Exercises: 0 minQuestions
What do the stderr and stdout files tell me?
Objectives
Standard error and standard out files
In your CASEROOT directory, your model produced files like b.day1.0.run.o* and b.day1.0.st_archive.o*. These files produce output related to our submit script. If the model and/or archiving did not run to completion, you will also have a b.day1.0.run.e* and/or b.day1.0.st_archive.e* file. The ones with *.o*' are called standard out (stdout) and the ones with .e` are called standard error (stderr).
The end of the file usually contains the most useful information, so you can use the Unix tail command to look at the end of the file:
$ tail b.day1.0.run.o4141153
- Case input data directory (DIN_LOC_ROOT) is /glade/p/cesmdata/cseg/inputdata
- Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
2023-02-25 19:05:37 MODEL EXECUTION BEGINS HERE
run command is mpiexec_mpt -p "%g:" -np 576 omplace -tm open64 /glade/scratch/cstan/b.day1.0/bld/cesm.exe >> cesm.log.$LID 2>&1
2023-02-25 19:08:09 MODEL EXECUTION HAS FINISHED
check for resubmit
dout_s True
mach cheyenne
resubmit_num 0
Troubleshooting
Does anyone have a
stdoutfile that does not indicateMODEL EXECUTION HAS FINISHED
Key Points
Log files
Overview
Teaching: 0 min
Exercises: 0 minQuestions
What do log files tell me?
Objectives
Log files
Each component of CESM produces a log file that contains output and errors related to the running of that component. If your model experiment ran successfully, then the log files are zipped and stored with the short-term archiving in DOUT_S_ROOT/logs. If you have an error, then the logfiles will be located in RUNDIR/*
Let’s take a look at a log file:
$ ./xmlquery DOUT_S_ROOT
$ cd /glade/scratch/kpegion/archive/b.day1.0
$ ls
$ cd logs
$ gunzip cesm.log.8732081.chadmin1.ib0.cheyenne.ucar.edu.230225-190529.gz
$ more cesm.log.8732081.chadmin1.ib0.cheyenne.ucar.edu.230225-190529
$ tail cesm.log.8732081.chadmin1.ib0.cheyenne.ucar.edu.230225-190529
Log files are very long and contain lots of information which may look like errors, but are usually just diagnostic information unless your run fails and there is a clear error. Useful error information is typically located at the end of the file
$ tail cesm.log.8732081.chadmin1.ib0.cheyenne.ucar.edu.230225-190529
Key Points
History Files
Overview
Teaching: 0 min
Exercises: 0 minQuestions
How can I look at my model output?
Objectives
History Files
This is of course why we run the model experiment in the first place. We want to see the model output from our experiment. Where does it go? If the experiment an the archiving ran successfully, model output is located in DOUT_S_ROOT. Where is that?
$ ./xmlquery DOUT_S_ROOT
DOUT_S_ROOT: /glade/scratch/cstan/archive/b.day1.0
Let’s go to this directory and see what is there:
$ cd /glade/scratch/cstan/archive/b.day1.0
$ ls
atm cpl esp glc ice lnd logs ocn rest rof wav
There are sub-directories for each of the component models and the output, called history files are located in each component sub-directory.
Let’s go look at output files form the ocn model:
$ cd ocn/hist
$ ls
ocn/hist> ls
b.day1.0.pop.dd.0001-01-01-03600 b.day1.0.pop.dv.0001-01-01-03600 b.day1.0.pop.h.once.nc
b.day1.0.pop.do.0001-01-01-03600 b.day1.0.pop.h.ecosys.nday1.0001-01-01.nc b.day1.0.pop.hv.nc
b.day1.0.pop.dt.0001-01-01-03600 b.day1.0.pop.h.nday1.0001-01-01.nc
We can do a quick look at the ocean model output file b.day1.0.pop.h.nday1.0001-01-01.nc
$ module load ncview
$ ncview b.day1.0.pop.h.nday1.0001-01-01.nc
It make take a few seconds, but a viewer will appear on your screen. If you select SST, then a small map of SST will appear. You can advance through all 5-times in this file using the arrows. You can advance through all 5-times in this file using the arrows.
Let’s look at the atmosphere history files:
$ cd ../atm
$ ls
There are no history files for the atmosphere. Why? We ran our first experiment for 5-days. The atmosphere is set to only provide monthly mean history files as its default. Since we did not run a full month, there are no atmosphere history files available. That configuration is set in the user_nl_cam file and we will learn how to change it later in class.
Understanding history files
netcdf format
named based on time frequency of data
h0/hcontain monthly averaged output
h1,h2contain other time frequencies (e.g. daily)contain all variables for a given model component for a given time frequency and produced as the experiment runs
Eventually, we want all timesteps for a specific variable from our model run. These files are called
timeseriesfiles and they are produced using a postprocessing step after the model run is completed. We will learn how to create these in a future lesson.
Key Points