Steps to running MESA on TACC Stampede2

Matt A. Wood
matt.wood@tamuc.edu
2019 March 1

Review the Stampede2 User Guide:
https://portal.tacc.utexas.edu/user-guides/stampede2

Stampede2 mounts three file Lustre file systems that are shared across all nodes: the home, work, and scratch file systems. The $HOME file system is limited to 10GB, and will not be sufficient to run MESA. You will probably want to build MESA in your $WORK directory.

Go to your scratch space

cd $WORK

Download MESA and the MESASDK.

Download MESA with svn (you will want to update this to the current release):

svn co -r 10398 https://subversion.assembla.com/svn/mesa^mesa/trunk mesa

Download and untar the SDK (you will want to update this to the current release):

wget --user-agent="" http://www.astro.wisc.edu/~townsend/resource/download/mesasdk/mesasdk-x86_64-linux-20180822.tar.gz
tar xf mesasdk-x86_64-linux-20180822.tar.gz

Edit your .bashrc (bash is the default shell on Stampede2)

  # for MESA
  export MESA_DIR=$WORK/mesa
  export MESASDK_ROOT=$WORK/mesasdk
  source $MESASDK_ROOT/bin/mesasdk_init.sh
  export OMP_NUM_THREADS 68

Or Edit your .cshrc

  # for MESA
  setenv MESA_DIR $WORK/mesa
  setenv MESASDK_ROOT $WORK/mesasdk
  source $MESASDK_ROOT/bin/mesasdk_init.csh
  setenv OMP_NUM_THREADS 48

Compile MESA

    source ~/.bashrc
    cd mesa
    ./install

Make your work directory

You will want to do this in $WORK or $SCRATCH

Run MESA

For small test runs you can get away with running on the login node, but you won't want to do this for production runs.

For large runs you will want to make a batch file. Here is a sample that runs the 'tutorial' example, adapted from a TACC-provided example https://portal.tacc.utexas.edu/user-guides/stampede2#running-jobs-on-the-stampede2-compute-nodes

#!/bin/bash
#----------------------------------------------------
# Sample Slurm job script
#   for TACC Stampede2 KNL nodes
#
#   *** OpenMP Job on Normal Queue ***
# 
# Notes:
#
#   -- Launch this script by executing
#   -- Copy/edit this script as desired.  Launch by executing
#      "sbatch knl.openmp.slurm" on a Stampede2 login node.
#
#   -- OpenMP codes run on a single node (upper case N = 1).
#        OpenMP ignores the value of lower case n,
#        but slurm needs a plausible value to schedule the job.
#
#   -- Default value of OMP_NUM_THREADS is 1; be sure to change it!
#
#   -- Increase thread count gradually while looking for optimal setting.
#        If there is sufficient memory available, the optimal setting
#        is often 68 (1 thread per core) or 136 (2 threads per core).

#----------------------------------------------------

#SBATCH -J MESA           # Job name
#SBATCH -o myjob.o%j       # Name of stdout output file
#SBATCH -e myjob.e%j       # Name of stderr error file
#SBATCH -p skx-normal      # Queue (partition) name
#SBATCH -N 1               # Total # of nodes (must be 1 for OpenMP)
#SBATCH -n 1               # Total # of mpi tasks (should be 1 for OpenMP)
#SBATCH -t 00:10:00        # Run time (hh:mm:ss)
#SBATCH --mail-user=your.email@yourschool.edu
#SBATCH --mail-type=all    # Send email at begin and end of job

# Other commands must follow all #SBATCH directives...

module list
pwd
date

# Set thread count (default value is 1)...

export OMP_NUM_THREADS=48

# Launch OpenMP code...

./rn > output.txt

# --------------------------------------------------

Submit the job with 'sbatch myjobscript'
Check the status of your job with 'squeue -u bjones' (if your username is bjones, for example)
Author note: I'm quite new to this, so if there's errors, please let me know so I can update it.

This writeup was adapted from Josiah Schwab's "Installing and Running MESA on a Cluster"
http://cococubed.asu.edu/mesa_market/cluster01.html)