/******************************************************************************
 *
 *
 *
 * Copyright (C) 2009
 *
 * Permission to use, copy, modify, and distribute this software and its
 * documentation under the terms of the GNU General Public License is hereby
 * granted. No representations are made about the suitability of this software
 * for any purpose. It is provided "as is" without express or implied warranty.
 * See the GNU General Public License for more details.
 *
 * Documents produced by Doxygen are derivative works derived from the
 * input used in their production; they are not affected by this license.
 *
 */
/*! \page CAMexample Community Atmosphere Model (CAM)

\section cam Implementation of PIO in CAM

CAM references PIO through both interface routines (which call PIO routines) and
PIO routines proper. The interface routines control much of the setting up of PIO
output and input, whereas the actual output and input is accomplished through
direct PIO calls. The PIO routines are located in models/utils/pio. A
description of the PIO package along with usage information may be found in the
main PIO documentation. We will first describe the interface and then cover the
direct calls.


\section description Description of the CAM Interface

The CAM interface to PIO consists of the pio_utils module, which is the file
models/atm/cam/src/utils/pio_utils.F90.

Init_pio_subsystem is the first routine called. It is called from cam_init and
initializes PIO in CAM. Init_pio_subsystem calls read_namelist_pio, which calls
set_pio_parameters. The main parameters set includes the IO mode (netcdf vs pnetcdf),
number of IO tasks, and IO stride.

Cam_pio_createfile and cam_pio_openfile create and open a PIO file, respectively.
Cam_pio_createfile is called from cam_write_restart, h_define (called from wshist,
which is called from write_restart_history), and atm_write_srfrest_mct.

Cam_pio_openfile is called from setup_initial (called from cam_initial),
cam_read_restart, read_restart_history, atm_read_srfrest_mct, and wrapup (called
from cam_run4).

Clean_iodesc_list, called from startup_initialconds and cam_read_restart, cleans the
list of IO descriptors.

Get_phys_decomp and get_dyn_decomp create decompositions for physics and dynamics
variables, respectively. Get_decomp is an interface to those routines.

Get_phys_decomp is directly called from infld_real_2d, infld_real_3d,
infld_real_3dncol; read_restart_physics, write_restart_physics; and
pbuf_read_restart, pbuf_write_restart.

Get_dyn_decomp is directly called from infld_real_2d and infld_real_3d.

Get_decomp is called from read_restart_history and dump_field (called from wshist,
which is called from write_restart_history). We see that the only time the
get_decomp interface is called is when dealing with history variables (where each
variable contains a description telling whether it relates to dynamics or physics).

Get_phys_decomp and get_dyn_decomp have an optional column (or column_in) argument,
which is used when defining history variables over a range of columns rather than the
whole domain.

Whenever a decomposition is created, a description of that decomposition is stored in
a list. When a new variable is catalogued, that list is searched to see if that needed
decomposition already exists. The routine performing that search is find_iodesc
(called from get_phys_decomp and get_dyn_decomp). This capability is supported only
when the range of history output is the whole domain.

Get_phys_decomp calls get_phys_ldof (or get_column_ldof), and get_dyn_decomp calls
get_dyn_ldof (or get_column_ldof). These routines do the bulk of the work in
constructing the IO decompositions. Get_column_ldof is called when the history output
is restricted to a subset of the domain.


\section direct Direct calls to PIO

Virtually all the calls to PIO routines reference the pio module, located in pio.F90.
Many of the variables contained in the pio module are referenced as well. This
section of the documentation focuses on these calls and surrounding environs
and discusses data types only peripherally.

\section historyWrite Writing a history file

Write_restart_history (in control/cam_history.F90) controls the writing of a history
file. Wshist writes the main variables that are on history tapes. Pio_put_var is used
to write header information, and subroutine dump_field writes the main history
variables, using primarily \ref PIO_write_darray. Additional variables are then written
using \ref PIO_put_var. (Recall that \ref PIO_write_darray isused to write distributed arrays.)

\section historyRead Reading a history file

Read_restart_history (in control/cam_history.F90) controls the reading of a history
file. Individual parameters are first read using pio_get_var. History variables are
then read in using \ref PIO_read_darray (which reads distributed arrays).

\section restartWrite Writing a restart dump

Cam_write_restart (in control/cam_restart.F90) controls the writing of a restart dump.
The calling chain includes write_restart_hycoef (which writes the pressure
coefficients), write_restart_dynamics, write_restart_physics, and write_restart_history.
Write_restart_coef is in hycoef.F90 and uses \ref PIO_put_var calls. Write_restart_dynamics
is in restart_dynamics.F90 (in the dynamics subdirectory relevant to the active
dynamical core) and issues primarily \ref PIO_write_darray calls. Write_restart_physics is
in physics/cam/restart_physics.F90. It calls a number of restart routines relating to
chemistry and aerosols, such as pbuf_write_restart (in physics/cam/phys_buffer.F90),
chem_write_restart (in chemistry/mozart), and write_prescribed_?????_restart (where
????? = ozone, ghg, aero, volcaero), located in chemistry/utils/prescribed_?????.F90.
Pbuf_write_restart calls the relevant decomposition routine followed by
\ref PIO_write_darray. Chem_write_restart calls write_?????_restart (where ????? =
tracer_cnst, tracer_srcs, linoz_data, spedata), located in chemistry/mozart/?????.F90.
These routines typically use \ref PIO_put_var. Write_restart_history is in cam_history.F90
and uses \ref PIO_put_var.

\section restartRead Reading a restart dump

Cam_read_restart (in control/cam_restart.F90) controls the reading of a restart dump.
The calling chain corresponds to that for cam_write_restart, and includes
read_restart_hycoef, read_restart_dynamics, read_restart_physics, and
read_restart_history. Typical routines called are pio_get_var and \ref PIO_read_darray.

\section Init Initialization

Cam_init (in control/cam_comp.F90) calls cam_initial and phys_init. Cam_initial,
which is located in the relevant dynamics subdirectory, calls initial_conds,
located in control/startup_initialconds. Initial_conds calls read_inidat, which is
located in the relevant dynamics subdirectory. Read_inidat calls infld, located in
control/ncdio_atm.F90. Infld calls the relevant decomposition routine followed by
\ref PIO_read_darray.

Phys_init is located in physics/cam/phys_grid.F90 and calls a number of initialization
routines. An important one is phys_inidat, which makes several calls to infld.

*/
