Windgrams and other plotting scripts | Multiple nests | Post processing fixups | Color Maps
| my rasp.ncl and rasp.site_load.xxx.ncl script modifications
A full download and installation script!
getneeded_utilities.txt
is a script that will download and unpack/install all the needed files and check for the existence and location of needed utilities, including perl, wget, curl, imagemagick etc.
It fetches all the needed files from DrJack as well as things from NCAR and FSL, including
GEOG files as well as the rang coastline files. It should be run from $BASEDIR. If it is, the appropriate paths
will be preserved.
It checks to see if you have the environment properly set, but it isn't so draconian as to force you to put them in your
login script. (I considered that, it would be easy to do.)
After it has run, you should be able to cd $BASEDIR/RASP/RUN and issue run.rasp panoche straight away.
(Let me know if that actually works. PLEASE! )
Note that it does NOT fetch new versions of NCL (NCAR Command Language) v5.11 or greater required for windgrams. That must be done manually after free registration at Earth System Grid
Current NCL version as of December 2010 was 5.2.1
As of March 2013 is NCL version 6.1.2 is current. There are some upgrade issues for 6.0 and beyond. See the RASP forum and the NCAR site for details.
These are critical issues involving how missing values are treated and the required load order for some of the stock helper files.
Scripts for diagnosis and Installation
The shell scripts have had a extension .txt added so the web server will serve them.
To run them you will have to make them executable.
General diagnostics for when RASP "doesn't" work and you have no clue why.
diagnose_rasprun.txt
I've found that it is very easy to leave errors behind in the
rasp.run.parameters.REGIONXYX file. The errors usually have to do with
not starting a run at the same time as there is a grib file, or trying
to generate forecasts after the last grib file. These leave maddeningly
obscure error traces, obviously because they are stupid errors that
should not have been made...
Has anyone worked out a table/chart of what is permissible (and
reasonable given forecast spin-up), as far as time of run and time of
output? We know that the "standard" ones work, but what else is OK?
These two scripts are checks to run after you have created a new domain to make sure all the well-known but subtle issues have been take care of.
postgui_checks.txt
postwindownewchecks.txt
This is a general "new_domain" script, that runs the gui, creates all needed directories, puts you in an editor to fix the decimal points and
finally copies a working rasp.run.parameters.REGION file to a new one and opens it for editing.
BETA create_new_domain.txt
GOLD startover if something goes wrong ... leaves only the initial non-windowed domain created by wrf_tools and removes all other RASP domain related directories. eg RASP/RUN/OUT/NEWDOMAIN RASP/HTML/NEWDOMAIN etc.
Multiple Nested Domains under the same Parent -- No rerun of parent.
This process relies on Alan Crouse's revisions to the rasp.pl script to create a new WINDOW domain that will run of the same PARENT domain as other WINDOW domains without having to re-run the PARENT.
The new rasp.pl accepts a new -w flag and copies files where appropriate. The one I am using has been uploaded here as rasp2pl.txt.
You should use meld or something similar to compare it to your production rasp.pl before implementing it. I know I made some other changes to it, but don't remember what they were. You may or may not want to keep the changes that don't have to do with multiple nests.
For more on how this all works and problems with it, see the RASP forum discussions
this thread
and
this one
(working for me) new_domain.txt Bash script
Post processing diagnosis, rerunning failed plots, and avoid most race conditions
These are probably idiosyncratic to my setup where I have a not-so-reliable connection to my web server.
runons run from the end of run.rasp to run various post processing options (for me blipspots, windgrams, missing plots)
runons2 run from the end of run.rasp when run as nested model with multiple nests. to run various post processing options (for me blipspots, windgrams, missing plots)
plotmissing scans for failed ncl based on inabity to find wrfout files and creates a script to plot and upload those missing hours.
plotme used by plotmissing or standalone to plot times and domains
wronggrids to correct a recurring but not reproducible error where the w2 domain plots d2 data...
plot_nowrf to replot without running WRF but including all the ENV_NCL variables that are used to generate plot titles etc.
results_output.hook My version using curl of an upload routine called at the end of plotting each time period.
getrss Script to post Rasp run status after each major run step (doesn't really provide real rss)
Site management scripts for blipspots and windgram sites
In order to ease the addition of new sites into a running system, I have written some scripts to automate the recreation of the data files to be used by
the windgram scripts below as well as by DrJack's extractblipspots.PL script.
The result is a "poor man's database" for your sounding/windgram sites.
New site data are simply added to the bottom of an ascii data file with this format
sitedata.ncl
DOMAIN,Sitename,grid,lat,lon,
Where grid is the highest resolution grid that contains the site either d2 or w2.
I have the data read by a script in NCL that, either outputs data directly to my windgram script, or when in debug=1 mode generates a series of different outputs for use by different parts of the system.
That temporary output is processed by a makenewsitesnew bash script that does all of the following. The bash script contains some hard coding for my site that will need editing for your site and domain(s).
- A rewrite of the data for use in bash scripts that invoke extract_blipspots.PL
- A rewrite of the data for into a javascript soundings menu. (I have adapted my univiewer so that the menu of soundings and windgrams is generated by a separate javascript.)
- A rewrite of the data into two separate javascript menus one for all blipspots and one for all windgrams. These get incorporated into other pages on my site.
- The creation of two simple html pages for accessing blipspots and windgrams
- An output of a new rasp.ncl.region.data to put sounding labels on plots
- Finally a new set of
push@{$PARAMETER_DOLIST{'FRASER'}},('sounding1','sounding2','sounding3','sounding4','sounding5','sounding6','sounding7' )
lines for each region.
- Most of the above are sorted from North to South by having a temporary lat at the beginning of the output line.
- Lines for different purposes are identified with keywords so that they can be pulled out of the overall output using grep.
- There is an additional script makestaticsoundings that makes static pages for soundings for people with slow connections.
Warning Note:
the input file cannot have any whitespace after the trailing commas!
rasp.ncl changes
Warning: all of the scripts below require some modification to work on your site.
My rasp.site_load.xxxxx.ncl scripts.
As examples to modify for yourselves, I include my rather messy rasp.site_load.paramxxx.ncl here as tjrasp_site_loads.zip
I have also made incremental changes to rasp.ncl and I include my version here as of March 2013rasptj.ncl
I have added additional levels to the winds that are calculated by default in rasp.wind_calc.ncl rasp.wind_calctj.ncl
now calculates level 0, tenmeter, level 1 and level 2 winds. I have named them sfwind0 for 2 meter, tenmwind, sfwind for the pressure level 1 and sfwind2 for the pressure level 2 that typically varues between 250' and 350' or 100meters.
sleep4data.ncl A script to force rasp.ncl to wait until the data file it expected has actually been written to disk -- helps prevent file not found errors /race conditions.
timestamp.ncl add a timestamp to your plots.
Additional color maps I have worked out for use with RASP
Windgrams and other plotting scripts
Warning: all of the scripts below require some modification to work on your site.
New windgram_tj_rasp_latlon.ncl A cgi-able
version to allow people to point and click or enter lat and lon and have your server generate a windgram. You must be able to run cgi scripts on your server!
This is not kept in sync with windgramtj.ncl, so may be missing some features but you can add them in from windgramtj.ncl if you want them. It does now have the colormap built in.
windgramtj.ncl
Production
version as of March 16, 2013
readsitelist.ncl used by windgramtj.ncl
Example sitedata.ncl file
New features
- The output is now png by default and can be controlled by settin the type variable. The output is much crisper, but the file size is very much larger.
- Added an optional pressure contour to show whether heights are rising/falling or staying the same over the course of the day.
- Has flags for turning the available "markers" on or off in the plot.
; options to display or not on the windgram
show_lapse = 1 ;local level to level lapse rate ( ΔC°/Δft expressed as C°/1000ft)
show_wind = 1 ; windbarbs
show_rh = 1 ; relative humidity greater than rh_cut when above your cut level faded white cross hatches will be drawn.
show_temp = 1 ; temperature contours
tmp_units = 1 ; change to 0 for C for Celcius 1 for Farenheight -- only affects the show_temp variable.
show_up_vel= 1 ; show vertical velocity W* at top of time column
; polymarkers all drawn on lapse_plot, so lapse plot must be set to 1.
show_snow = 1 ; lowest freezing level
show_wing = 1 ; hg/pg 225ft/min 1.2m/s soaring level markers
show_lcl = 1 ; little clouds for lifted condensation level
show_p_contour = 1 ; constant pressure lines rise/fall steady during day.
show_z_contour = 1 ; constant altitude to the compare to the pressure contour
Other setable parameters include
debug = 0 ; anything other than zero prints debug info
useblipdata = 0 ; use pre calculated data from the param.curr.hh00lst.grid.data blipdata files.
unit_scheme = 0 ; still waiting for someone to properly implement this unit_scheme
show_condense = 0 ; show level where temp==dewpoint probably useless rh above seems to work reasonably well.
show_rain = 0 ; activate Andrea Barcelona's rain code
show_pbl = 0 ; polymarker for boundary layer +show_pbl
hide_lapse_colors = 0 ; hide_lapse_colors -- used if you want color windbarbs, and lcl, but no lapse colors
type = "ncgm" ;legitimate values are x11, ps, ncgm, png
ptop = 24 ;top of the plot in pressure levels. Change to suit your audience and region.
rhcut = 95 ;plot crosshatching where rh > rhcut -- where there are likely to be clouds
DOMAIN = "DOMAINXYZ"
SITE = "SITE"
Any or all of the above can be passed on the command line if so desired.
But the string variables must be passed enclosed in excaped double quotes.
e.g. ncl windgramtj.ncl type=\"png\"
In addition, there are default parameters within the program that you should change to suit your setup.
outdir="$BASEDIR/RASP/UTIL/windgrams/" ; could be anywhere
operator="TJ Olney" ; your name here :-) credit to me is always nice though
default_domain= "PNW-WINDOW" ;your default domain here
default_grid="w2"
default_lat= 48.61
default_lon= -122.43
default_type = "x11"
default_ptop = 30 ;usually 20 I changed for TS upper instability
default_rhcut = 95
;color choices must be in the palette being used or the closest one in the current pallette will be used.
foreground_color= "white"
background_color= (/.5, .5, .9 /) deep purple-blue
label_color= foreground_color
wstar_color= "yellow"
line_color= 13
templine_color= 14
colorpalette = ""
- Andrea Barcelona's code for rain is now included and working for me. It required some tweaking to run on both d2 and w2 domains.
- A datestamp for when the plot is actually plotted.
- changed absolute paths to $BASEDIR paths -- less work for you.
- The "updraft" colormap is included inline so there is no need to install it. (Some people don't like the colors...)
- Temperature as a potential variable to plot as a contour plot, on and Farenheight by default.
-
Continuing features --
uses external sitedata.ncl —
Includes W* as a number at the top of the windbarb column, default units m/s
lcl as a cloud marker,
"local" level-to-level lapserate as background colors,
freezing level as snowflake marker,
and a really coarse indicator for cloudiness
done by 0/1 for relative humidity greater than a cutoff level (set to
95% default). It also includes a currently unused function to extract
parameters from blipspots.
Notes:
This version also still includes the ability to easily change the top altitude of the plot. (ptop)
Unlike earlier versions, this one ensures proper placement of markers by using gsn_add_polymarker().
The lcl cloud markers don't indicate
that there will be clouds, only the lowest level at which clouds are
expected to appear if they do and they are created from unheated air
rising because it was forced by terrain. The CCL is usually higher and
would be the level where clouds would appear if formed by surface level
air rising thermically only.
At present I don't know how to calculate CCL, :-( nor which of DrJack's measure might be the same as CCL.
This version includes the looping logic to do either one
site, a whole domain/region, or all the domains and sites in your
sitedata.ncl file.
Example: regiondata.ncl
Windgram Setup Notes:
- Download windgramtj.ncl and sitedata.ncl to your ncl directory
- Ensure that you are using NCL v 5.1 or later! (or be prepared to rewrite the data ingestion routines that rely on addfiles.)
NCL version 6.0 seems to be fully compatible with windgramtj.ncl
- Modify sitedata.ncl to match your regionxyz and names of sites and lat and lon of sites within your regions.
- Create a line at the end of $BASEDIR/RASP/RUN/run.rasp to run
the windgram script after all times for a given regionxyz have run.
- Changes you should consider making to windgramtj.ncl
- Modify the outdir to suit your setup, or ensure that the outdir exists
- Optional: Change units back to metric if you so desire by removing the *3.2808, *196.85, and *1.94386 conversion factors.
Note that if you do, you will also want to change
uv_res@vcLevels = (/ 3.,6.,9.,12.,15.,18.,21. /) ;knots
uv_res@vcLevelColors = (/2,3,4,5,6,7,8,9,10/)
to whatever units you want for wind speed.
And lapse rates from this
lapse_res@cnLevels = (/ -3.,-2.5,-2.0,-1.5,-1.2, -.5, 0.,.5 /)
lapse_res@cnFillColors = (/ 11,10,7,8,3,-1,-1,13,14 /) ; -1 is transparent
To your unit-equivalents where -3C°/1000ft is absolutely unstable 0 and greater is inverted.
Point of interest here is that here, near the Canadian border, most Canadians still think of height in feet, not meters but speed is kph.
- Choose your default values for ptop, rhcut, and SITE. ptop is a pressure level index, not an actual pressure.
- Comment out, remove, and/or revise as needed the 5 or six-line sections labeled "kludge" that are idiosyncratic to my site.
- Choose which "markers" to plot. Default is LCL as calculated
by NCL which differs from any of DrJack's various lcl measures and
freezing level — which I have found that many long time pilots use as
their intuitive surrogate for lapse rate.
- Test it from the command line:
ncl windgramtj.ncl [ptop=xx] [SITE=\"sitename\" | DOMAIN=\"REGIONXYZ\" | \"all\"] [rhcut=xx] [type=\"x11 |ncgm |ps \"]
- To make windgrams for your site
The windgrams made by NCL are now png files but are not particularly suited for putting on the web because of filesize..
You will need to use the same method that the rest of the RASP system uses to convert these to .png or .jpg or .gif files. You can do this automatically through the last line of run.rasp, or using a cron job that runs the windgramtj.ncl script after all the times are in for a region. The same job should convert the *.ncgm files to the desired graphic format, usually .png, move the images to a web space, and do any archiving or house cleaning that you desire.
See my example script getwindgramtj for the syntax to use with convert.
PNG notes.
Since NCL 51.1, png can be output by default. In the first implementations of this, the resulting images were wonderfully crisp, but the file sizes were huge. They also take a long time to render, making them not suitable for cgi windgrams or for doing all the RASP plotting.
For a small finite number of windgrams, I use png output.
I have used a method to shrink them that involves decreasing the color depth in two stages.
for file in $yyyy_mm_dd*.png
do
/usr/bin/convert $file -depth 4 temp.gif
/usr/bin/convert temp.gif $file
done
my version of sitedata.ncl to serve as a template for yours. NCL does not allow mixed type arrays so I have kept the various site parameters in separate arrays. There two variables that are assigned and will throw errors if your site arrays are off. That is their only purpose.
my shell script that runs after all hours for a region have been processed.
I am currently testing the cloudwater variables, recently discussed on
the forum, to plot instead of rh>95. If it works, I'll post when
working. October 2009 -- Have studk to rh>95.
You might find it easier to just use this, where I have stripped out all the site/domain logic
and just include sites(),wlats(), and wlons() along with a single region.
windgram1domain.ncl
To Do:
DONE get rid of "kludges" by running DOMAIN-WINDOW separate from DOMAIN (get rid of code that adds WINDOW)
DONE put plotting lines inside conditionals turned on or off at top of script.
figure out what makes a good CCL measure to plot as well.
DONE do for any other measures that I might have gotten from blipspots what I did with wstar -- recalculate them.
create a form of windgram that is an ncl procedure, like skewt_func and can be called from another NCL script when the last wrfout file is created.
Earlier versions
sites.ncl If using a separate data file for site information this is a template
Shell scripts for "externally" processed plots
getwindgrams A shell script to run process and upload the windgrams (uses the one at a time method)
getwindgramspnw A shell script to run process and upload the windgrams for a SINGLE DOMAIN
getblipspotsA shell script to run process and upload blipspots for the same sites as windgrams.
It has been somewhat maddening to make the "all in one -- multiple
domains" version, so I am thinking that I will likely give up and
use the separate versions depending on my need.
NCL can conditionally load data using the loadscripts() function. The
inefficiencies of the first version
don't matter if you are calling for only one site. The one domain version
is dead easy to run and modify compared to the all-in-one version.
TJ