As a point of discussion, please consider the following schematic of our hindcast validation of Hurricane Katrina:
In our hurricane hindcasts, such as the validations of Katrina and Rita described in papers to be published in Monthly Weather Review, the simulations are performed in two stages. First, a spin-up simulation is run for several days before the start of the hurricane winds, so that the tides and rivers can reach a dynamic equilibrium in the resonant basin of the Gulf of Mexico. Then the solution from that spin-up simulation is used as the initial condition for the hurricane hindcast simulation. For example, for Katrina, we employ an 18-day, tides/rivers spin-up simulation that starts on 07 August 2005, and then we run a 7-day, hurricane simulation that starts on 25 August 2005. The solution from the spin-up simulation is used to hot-start the hurricane simulation.
There are two ways in which the coupled SWAN+ADCIRC could be hot-started during this hindcast. First, it is always hot-started at the beginning of its simulation, at 2005/08/25/0000Z. Although SWAN is not run during the tides/rivers spin-up simulation, because its action is forced entirely by the hurricane winds, we do run ADCIRC during that stage. Thus, when the coupled SWAN+ADCIRC model is employed in the second stage, we must hot-start the ADCIRC half of the simulation. SWAN starts from scratch at the beginning of the hurricane simulation.
Second, we may need to hot-start at some time during the SWAN+ADCIRC stage if it ended abruptly, due to machine failure, user interruption, etc. Instead of re-starting that stage at its beginning, we would rather hot-start in the middle. Then both SWAN and ADCIRC would need to be hot-started, using an intermediate solution as an initial condition for the remainder of the simulation.
Thus, there are two distinct methods in which SWAN+ADCIRC might be hot-started. The first method, in which SWAN is cold-started and ADCIRC is hot-started, would occur in the transition between simulations in the Katrina hindcast. The second method would hot-start both SWAN and ADCIRC, such as at some intermediate time during the second stage of the Katrina hindcast. Instructions for both methods are included below.
Hot-Start File Generation
However, first it is important to note how the hot-start files are generated by the two models. In ADCIRC, the hot-start file generation is controlled by parameters near the bottom of the model parameters file (fort.15):
1 86400 ! NHSTAR NHSINC
The first parameter, NHSTAR, is set to either zero or unity, depending on whether the user desires the hot-start files to be generated. If NHSTAR = 1, then ADCIRC will write the hot-start information at an increment of every NHSINC time steps. ADCIRC alternates writing this information to two different files, fort.67 and fort.68, so that the information will not be lost if the simulation ends abruptly during the writing process. For example, if the hot-start file generation is enabled every day in a four-day simulation, then ADCIRC will write the hot-start information twice to the fort.67 file and twice to the fort.68 file. Even if the simulation ends abruptly (due to power outage, user interruption, etc.) during the output to the fort.68 file, the user will still have relatively recent hot-start information from the last output to the fort.67 file.
In SWAN, the hot-start file generation depends on whether SWAN is coupled to ADCIRC. If SWAN is run as a stand-alone model, then a single hot-start file can be written at the end of the computations by including the following line at the end of the SWAN control file:
HOTFILE 'swan.68'
However, this method of hot-start file generation does not make it possible to hot-start in the middle of an existing SWAN simulation. Thus, when SWAN is coupled with ADCIRC, its hot-start file generation is tied to that of ADCIRC, so that SWAN writes intermediate hot-start files during its simulation. These files are called swan.67 and swan.68. They are controlled by the NHSTAR and NHSINC parameters on the ADCIRC side of the coupling, so that both models write the same number of hot-start files and at the same increment. Now it is not necessary to change the fort.26 file for SWAN to generate hot-start files during a SWAN+ADCIRC model simulation.
Thus, if the hot-start file generation is enabled in the ADCIRC fort.15 file, then both SWAN and ADCIRC will write hot-start files. These files can be used to hot-start future SWAN+ADCIRC simulations.
Hot-Starting the Coupled SWAN+ADCIRC
These step-by step instructions describe how to hot-start ADCIRC and SWAN. As noted above, you may not need to hot-start both models. So I have provided the instructions in a unique format: in a table. If you are only hot-starting the ADCIRC half of the coupled model, then you can read the instructions down the ADCIRC column in this table, and ignore the SWAN instructions. However, if you are hot-starting both SWAN and ADCIRC, then you can follow step-by-step in both columns, which will prevent the unnecessary repetition of some actions, such as running adcprep.
| ADCIRC | SWAN | |
|---|---|---|
| A. |
Find the most-recent hot-start files in the PE0000 sub-directory of the previous simulation. As noted above, ADCIRC alternates writing the hot-start information to the
fort.67/swan.67 and fort.68/swan.68 files. The fort.68/swan.68 files usually contains the most recent information, but you can check the time stamps on the files. |
|
| B. |
Update the
fort.15 file from your spin-up simulation for use during your hurricane simulation. The hot-start is enabled in ADCIRC by changing the value of IHOT, which is located near the top of the fort.15 file. The value of IHOT should be equal to the unit number of the hot-start file. So if you are using the fort.68 file to hot-start your run, then you would change that line in the fort.15 file to be:
This will direct ADCIRC to read the local |
Update the
fort.26 file from your previous simulation. The hot-start is enabled in SWAN with the following command:
That command must appear in your |
| C. |
Run
adcprep using options 1 and 2 to decompose the mesh and other input files into the local subdirectories. If adcprep has been compiled with the -DADCSWAN flag (see Step 4 in my instructions on how to compile and run SWAN+ADCIRC), then it will also copy the fort.26 file into the local subdirectories. |
|
| D. |
Use
adcprep to localize the hot-start file into the local subdirectories where you want to run the hurricane simulation. ADCIRC writes the hot-start files as global files into the PE0000 subdirectory, but those global files need to be localized before hot-starting a new simulation. The sixth option in adcprep will take that global fort.68 file and decompose it into the local PE subdirectories. |
Copy the local
swan.68 files from your previous simulation into the directory for your hot-started simulation. There are two ways to do this.
First, if you are running the new simulation in the same set of subdirectories as the previous simulation, then you don’t need to do anythng to the local hot-start files. Just leave them in the same subdirectores, and SWAN will process them without a problem. However, if you want to change directories, then you will need to transfer the local hot-start files to the new set of subdirectories. It is not possible to globalize the local hot-start files for an unstructured SWAN simulation, so you will need to transfer the local files. You can do this by using the
Copy that
to copy the local hot-start files into the local subdirectories. Please note that the number of computational cores cannot change. If the previous simulation was run on 100 compute cores, then the hot-started simulation must also use 100 compute cores. |
UPDATED 2010/09/28: Now it is no longer necessary for the hot-started SWAN simulation to use the same number of compute cores as the original simulation. You can use the HottifySWAN.F90 program to globalize the original set of SWAN hot-start files and then localize them over a different number of compute cores.
This program can be compiled in serial:
pgf90 HottifySWAN.F90
or in parallel:
mpif90 HottifySWAN.F90 -DMPI
The program will ask you for your choice of: (1) globalization, or (2) localization of the hot-start files; and it will ask for the unit number (67 or 68).
This code can be really slow, especially during the localization step. The input/output of the SWAN hot-start files is slow because they are in ASCII format. And the global action density array requires a lot of memory. It is best to run the code in parallel, devoting a lot of memory to each process by using only one core per CPU or node.
That is all it takes. With these steps completed, you should be ready to run a hot-started simulation of SWAN+ADCIRC. In this way, the appropriate hot-start files for both ADCIRC and SWAN will be localized in the PE subdirectories, and the control files (fort.15 and fort.26) will be updated to reflect the hot-start. Both SWAN and ADCIRC will use the previously-computed solution as an initial condition for the new simulation.