Windgrams and other plotting scripts  |  Multiple nests  | Post processing fixups  |  Color Maps | my rasp.ncl and rasp.site_load.xxx.ncl script modifications

NEWA full download and installation script! NEW

getneeded_utilities.txt NEW is a script that will download and unpack/install all the needed files and check for the existence and location of needed utilities, including perl, wget, curl, imagemagick etc.

It fetches all the needed files from DrJack as well as things from NCAR and FSL, including GEOG files as well as the rang coastline files. It should be run from $BASEDIR. If it is, the appropriate paths will be preserved.
It checks to see if you have the environment properly set, but it isn't so draconian as to force you to put them in your login script. (I considered that, it would be easy to do.)
After it has run, you should be able to cd $BASEDIR/RASP/RUN and issue run.rasp panoche straight away. (Let me know if that actually works. PLEASE! )

Note that it does NOT fetch new versions of NCL (NCAR Command Language) v5.11 or greater required for windgrams. That must be done manually after free registration at Earth System Grid

Current NCL version as of December 2010 was 5.2.1

As of March 2013 is NCL version 6.1.2 is current. There are some upgrade issues for 6.0 and beyond. See the RASP forum and the NCAR site for details. These are critical issues involving how missing values are treated and the required load order for some of the stock helper files.

Scripts for diagnosis and Installation

The shell scripts have had a extension .txt added so the web server will serve them.
To run them you will have to make them executable.

General diagnostics for when RASP "doesn't" work and you have no clue why.

  • diagnose_rasprun.txt

    I've found that it is very easy to leave errors behind in the rasp.run.parameters.REGIONXYX file. The errors usually have to do with not starting a run at the same time as there is a grib file, or trying to generate forecasts after the last grib file. These leave maddeningly obscure error traces, obviously because they are stupid errors that should not have been made...

    Has anyone worked out a table/chart of what is permissible (and reasonable given forecast spin-up), as far as time of run and time of output? We know that the "standard" ones work, but what else is OK?

    These two scripts are checks to run after you have created a new domain to make sure all the well-known but subtle issues have been take care of.

  • postgui_checks.txt
  • postwindownewchecks.txt

    This is a general "new_domain" script, that runs the gui, creates all needed directories, puts you in an editor to fix the decimal points and finally copies a working rasp.run.parameters.REGION file to a new one and opens it for editing.
  • BETA create_new_domain.txt
  • GOLD startover if something goes wrong ... leaves only the initial non-windowed domain created by wrf_tools and removes all other RASP domain related directories. eg RASP/RUN/OUT/NEWDOMAIN RASP/HTML/NEWDOMAIN etc.

    Multiple Nested Domains under the same Parent -- No rerun of parent.

    This process relies on Alan Crouse's revisions to the rasp.pl script to create a new WINDOW domain that will run of the same PARENT domain as other WINDOW domains without having to re-run the PARENT.
    The new rasp.pl accepts a new -w flag and copies files where appropriate. The one I am using has been uploaded here as rasp2pl.txt. You should use meld or something similar to compare it to your production rasp.pl before implementing it. I know I made some other changes to it, but don't remember what they were. You may or may not want to keep the changes that don't have to do with multiple nests.
    For more on how this all works and problems with it, see the RASP forum discussions this thread and this one


  • (working for me) new_domain.txt Bash script


    Post processing diagnosis, rerunning failed plots, and avoid most race conditions

    These are probably idiosyncratic to my setup where I have a not-so-reliable connection to my web server.

  • runons run from the end of run.rasp to run various post processing options (for me blipspots, windgrams, missing plots)
  • runons2 run from the end of run.rasp when run as nested model with multiple nests. to run various post processing options (for me blipspots, windgrams, missing plots)
  • plotmissing scans for failed ncl based on inabity to find wrfout files and creates a script to plot and upload those missing hours.
  • plotme used by plotmissing or standalone to plot times and domains
  • wronggrids to correct a recurring but not reproducible error where the w2 domain plots d2 data...
  • plot_nowrf to replot without running WRF but including all the ENV_NCL variables that are used to generate plot titles etc.
  • results_output.hook My version using curl of an upload routine called at the end of plotting each time period.
  • getrss Script to post Rasp run status after each major run step (doesn't really provide real rss)

  • Site management scripts for blipspots and windgram sites

    In order to ease the addition of new sites into a running system, I have written some scripts to automate the recreation of the data files to be used by the windgram scripts below as well as by DrJack's extractblipspots.PL script.

    The result is a "poor man's database" for your sounding/windgram sites.
    New site data are simply added to the bottom of an ascii data file with this format
    sitedata.ncl

    DOMAIN,Sitename,grid,lat,lon,
    
    Where grid is the highest resolution grid that contains the site either d2 or w2.
    I have the data read by a script in NCL that, either outputs data directly to my windgram script, or when in debug=1 mode generates a series of different outputs for use by different parts of the system.

    That temporary output is processed by a makenewsitesnew bash script that does all of the following. The bash script contains some hard coding for my site that will need editing for your site and domain(s).
    1. A rewrite of the data for use in bash scripts that invoke extract_blipspots.PL
    2. A rewrite of the data for into a javascript soundings menu. (I have adapted my univiewer so that the menu of soundings and windgrams is generated by a separate javascript.)
    3. A rewrite of the data into two separate javascript menus one for all blipspots and one for all windgrams. These get incorporated into other pages on my site.
    4. The creation of two simple html pages for accessing blipspots and windgrams
    5. An output of a new rasp.ncl.region.data to put sounding labels on plots
    6. Finally a new set of
      push@{$PARAMETER_DOLIST{'FRASER'}},('sounding1','sounding2','sounding3','sounding4','sounding5','sounding6','sounding7' )
      lines for each region.

    Warning Note:

    the input file cannot have any whitespace after the trailing commas!

    rasp.ncl changes

    Warning: all of the scripts below require some modification to work on your site.

    My rasp.site_load.xxxxx.ncl scripts.

    As examples to modify for yourselves, I include my rather messy rasp.site_load.paramxxx.ncl here as tjrasp_site_loads.zip


  • I have also made incremental changes to rasp.ncl and I include my version here as of March 2013rasptj.ncl

  • I have added additional levels to the winds that are calculated by default in rasp.wind_calc.ncl rasp.wind_calctj.ncl now calculates level 0, tenmeter, level 1 and level 2 winds. I have named them sfwind0 for 2 meter, tenmwind, sfwind for the pressure level 1 and sfwind2 for the pressure level 2 that typically varues between 250' and 350' or 100meters.
  • sleep4data.ncl A script to force rasp.ncl to wait until the data file it expected has actually been written to disk -- helps prevent file not found errors /race conditions.
  • timestamp.ncl add a timestamp to your plots.
    Additional color maps I have worked out for use with RASP

    Windgrams and other plotting scripts

    Warning: all of the scripts below require some modification to work on your site.

  • New windgram_tj_rasp_latlon.ncl A cgi-able version to allow people to point and click or enter lat and lon and have your server generate a windgram. You must be able to run cgi scripts on your server! This is not kept in sync with windgramtj.ncl, so may be missing some features but you can add them in from windgramtj.ncl if you want them. It does now have the colormap built in.

  • windgramtj.ncl
    Production version as of March 16, 2013
  • readsitelist.ncl used by windgramtj.ncl
  • Example sitedata.ncl file
    New features
    Continuing features --
  • uses external sitedata.ncl
  • Includes W* as a number at the top of the windbarb column, default units m/s
  • lcl as a cloud marker,
  • "local" level-to-level lapserate as background colors,
  • freezing level as snowflake marker,
  • and a really coarse indicator for cloudiness done by 0/1 for relative humidity greater than a cutoff level (set to 95% default). It also includes a currently unused function to extract parameters from blipspots.
    Notes:
  • This version also still includes the ability to easily change the top altitude of the plot. (ptop)
  • Unlike earlier versions, this one ensures proper placement of markers by using gsn_add_polymarker().
  • The lcl cloud markers don't indicate that there will be clouds, only the lowest level at which clouds are expected to appear if they do and they are created from unheated air rising because it was forced by terrain. The CCL is usually higher and would be the level where clouds would appear if formed by surface level air rising thermically only.
    At present I don't know how to calculate CCL, :-( nor which of DrJack's measure might be the same as CCL.
  •   This version includes the looping logic to do either one site, a whole domain/region, or all the domains and sites in your sitedata.ncl file.
    Example: regiondata.ncl


    Windgram Setup Notes:

    1. Download windgramtj.ncl and sitedata.ncl to your ncl directory
    2. Ensure that you are using NCL v 5.1 or later! (or be prepared to rewrite the data ingestion routines that rely on addfiles.)
      NCL version 6.0 seems to be fully compatible with windgramtj.ncl
    3. Modify sitedata.ncl to match your regionxyz and names of sites and lat and lon of sites within your regions.
    4. Create a line at the end of $BASEDIR/RASP/RUN/run.rasp to run the windgram script after all times for a given regionxyz have run.
    5. Changes you should consider making to windgramtj.ncl
      1. Modify the outdir to suit your setup, or ensure that the outdir exists
      2. Optional: Change units back to metric if you so desire by removing the *3.2808, *196.85, and *1.94386 conversion factors.
        Note that if you do, you will also want to change
            uv_res@vcLevels                 = (/ 3.,6.,9.,12.,15.,18.,21. /)   ;knots
            uv_res@vcLevelColors            = (/2,3,4,5,6,7,8,9,10/)
        
        to whatever units you want for wind speed. And lapse rates from this
        lapse_res@cnLevels =            (/ -3.,-2.5,-2.0,-1.5,-1.2, -.5, 0.,.5 /)
        lapse_res@cnFillColors         = (/ 11,10,7,8,3,-1,-1,13,14 /)         ; -1 is transparent
        
        To your unit-equivalents where -3C°/1000ft is absolutely unstable 0 and greater is inverted.
        Point of interest here is that here, near the Canadian border, most Canadians still think of height in feet, not meters but speed is kph.
      3. Choose your default values for ptop, rhcut, and SITE. ptop is a pressure level index, not an actual pressure.
      4. Comment out, remove, and/or revise as needed the 5 or six-line sections labeled "kludge" that are idiosyncratic to my site.
      5. Choose which "markers" to plot. Default is LCL as calculated by NCL which differs from any of DrJack's various lcl measures and freezing level — which I have found that many long time pilots use as their intuitive surrogate for lapse rate.
      6. Test it from the command line:
        ncl windgramtj.ncl [ptop=xx] [SITE=\"sitename\" | DOMAIN=\"REGIONXYZ\" | \"all\"] [rhcut=xx] [type=\"x11 |ncgm |ps \"]
      7. To make windgrams for your site
        The windgrams made by NCL are now png files but are not particularly suited for putting on the web because of filesize.. You will need to use the same method that the rest of the RASP system uses to convert these to .png or .jpg or .gif files. You can do this automatically through the last line of run.rasp, or using a cron job that runs the windgramtj.ncl script after all the times are in for a region. The same job should convert the *.ncgm files to the desired graphic format, usually .png, move the images to a web space, and do any archiving or house cleaning that you desire.
        See my example script getwindgramtj for the syntax to use with convert.

        PNG notes.

        Since NCL 51.1, png can be output by default. In the first implementations of this, the resulting images were wonderfully crisp, but the file sizes were huge. They also take a long time to render, making them not suitable for cgi windgrams or for doing all the RASP plotting. For a small finite number of windgrams, I use png output. I have used a method to shrink them that involves decreasing the color depth in two stages. for file in $yyyy_mm_dd*.png do /usr/bin/convert $file -depth 4 temp.gif /usr/bin/convert temp.gif $file done
  • my version of sitedata.ncl to serve as a template for yours. NCL does not allow mixed type arrays so I have kept the various site parameters in separate arrays. There two variables that are assigned and will throw errors if your site arrays are off. That is their only purpose.

  • my shell script that runs after all hours for a region have been processed.

    I am currently testing the cloudwater variables, recently discussed on the forum, to plot instead of rh>95. If it works, I'll post when working. October 2009 -- Have studk to rh>95.


    You might find it easier to just use this, where I have stripped out all the site/domain logic and just include sites(),wlats(), and wlons() along with a single region. windgram1domain.ncl

    To Do:

  • DONE get rid of "kludges" by running DOMAIN-WINDOW separate from DOMAIN (get rid of code that adds WINDOW)
  • DONE put plotting lines inside conditionals turned on or off at top of script.
  • figure out what makes a good CCL measure to plot as well.
  • DONE do for any other measures that I might have gotten from blipspots what I did with wstar -- recalculate them.
  • create a form of windgram that is an ncl procedure, like skewt_func and can be called from another NCL script when the last wrfout file is created.
    Earlier versions
  • sites.ncl If using a separate data file for site information this is a template
    Shell scripts for "externally" processed plots
  • getwindgrams A shell script to run process and upload the windgrams (uses the one at a time method)
  • getwindgramspnw A shell script to run process and upload the windgrams for a SINGLE DOMAIN
  • getblipspotsA shell script to run process and upload blipspots for the same sites as windgrams.
    It has been somewhat maddening to make the "all in one -- multiple domains" version, so I am thinking that I will likely give up and use the separate versions depending on my need.

    NCL can conditionally load data using the loadscripts() function. The inefficiencies of the first version don't matter if you are calling for only one site. The one domain version is dead easy to run and modify compared to the all-in-one version. TJ