Task #208: Evaluate existing E&O AML scripts - Environment and organisms - NCEAS Projects

Actions

Copy link

Task #208

open

Evaluate existing E&O AML scripts

Added by Jim Regetz about 13 years ago. Updated almost 12 years ago.

Status:

In Progress

Priority:

Normal

Assignee:

Jim Regetz

Category:

Terrain

Start date:

04/16/2011

Due date:

05/04/2011 (over 13 years late)

% Done:

30%

Estimated time:

12.00 h

Activity type:

Coding/analysis

Description

There are 18 *.aml files in various parts of eos:~organisms/topo, although it looks like these are really 9 identical or nearly identical pairs of scripts. Review these scripts, in consultation with Ming as needed, to determine the following. Also confirm with Ming that other relevant scripts (AML or otherwise) aren't hiding elsewhere.

Which are (still) pertinent?
What do they do?
What's the rough cost/benefit tradeoff on continuing to build these out vs switching to something else?

If some task or even key subtask is basically fully implemented (or could be easily completed) in AML, can be run successfully on the full data, and would be more than trivial to convert, then we should probably go with it. But we should not stick with AML if it imposes extra development or runtime overhead, noting that all else being equal:

ArcGIS can't run on our multicore, 64bit, high RAM Linux servers
Even when ArcGIS is the best tool for the job, scripting in Python (AML-to-Python cheatsheet) is preferable

Estimated time is a wild guess -- I don't know how complicated nor how well documented these are -- but given that each script is only 10s to a couple of hundred lines in what is a rather verbose language, and with Ming's input, I don't think this should take long.

Actions

Copy link

Updated by Jim Regetz about 13 years ago

Priority changed from Normal to High

Actions

Copy link

Updated by Rick Reeves about 13 years ago

% Done changed from 0 to 20

Description of ten existing scripts (nine AML, one Python):

Aggregate.aml: Executes the aggregate (resampling) function to generate
standard statistics for elevation, slope, and twi layers.
(Ming's description: (aggregate – 90m to 1km))

Chunkproc.aml: Processes a large(r) Grid by executing a second aml script
over sections of the grid. Generates a second, output grid.
(Ming's description: )

layers.aml: Executes the flowaccumulation() and flowdirection() functions,
which produce output grids of flow direction between raster
cells and their nearest downslope neighbors.
(Ming's description: generating topographic measures)

mrvbf6g-a3.aml: More complicated script that has had an evolution of 6 previous
versions: From input DEM tiles, gnerates smoothed DEM files,
and then calculates slope and statistics.
(Ming's description: mrvbf (i.e., multi-resolution valley bottom
flatness), various AMLs in folder used to generate the layer )

multiscalesmooth9a_clean.aml:
(Ming's description: for smoothing SRTM)

pctl.aml: Executes external 'pctl.exe' command on a DEM image file
(Ming's description: )

pctl-limits.aml: Sets an environment (radius of cells) value.
(Ming's description: )

tilemerge_oregon.aml: merge a collection of tiles in Oregon around Portland for testing method.
(Ming's description: merging Oregon tiles (not working, ‘manual’ merger))

unpacktiles.aml: locate all (5 degree) .asc files in input directory, then
load them and split into 1 degree tiles. Presumably for use
on CGIAR export files.
(Ming's description: AML for unpacking tiles in a file structure
that matches John Gallant’s (see ‘tiles’ folder))

unziptiles.py: PYTHON script unzips a collection of .zip (terrain) files in a folder.
(Ming's description: script for unzipping tiles)

April 18, 2011: I have requested from Ming brief descriptions of the remaining scripts as well as
a summary of the processing workflow/sequence (if any) that he had developed
to produce the data layers required by this project.

Actions

Copy link

Updated by Jim Regetz about 13 years ago

Parent task deleted (~~#207~~)

Actions

Copy link

Updated by Rick Reeves about 13 years ago

Due date changed from 04/16/2011 to 04/26/2011
Status changed from New to In Progress
% Done changed from 20 to 30
Activity type changed from Other to Coding/analysis

I reviewed all of the AML scripts a second time and am ready to test them within ArcMap GIS. Next week I will do so regardless of additional feedback from Ming.

Actions

Copy link

Updated by Rick Reeves about 13 years ago

Thank you Ming, more good information.
I do intend to assemble a workflow document of some type next week.
I will sketch it out and then send you the draft for comment.

Cheers,
RR

On 4/22/2011 11:30 AM, Tien Ming Lee wrote:

Hi Rick and Jim,

Happy Earth Day!

Thanks for your questions. My answers are below.

1-2, 4) Turns out the 'mrvbf6g-a3' AML script invokes the 'pctl' and 'chunkproc' AML scripts. The others are not used as far as I can tell from the script.

For eg. a line of script in 'mrvbf6g-a3': &run chunkproc pctl dem pctl_%l%_1 pctlradius geographic

You would need to run the 'mrvbf6g-a3' from GRID, not ARC, and specify the dem layer ('srtm_or_smth' OR SRTM of Oregon smoothed). So its: &run mrvbf6g-a3 srtm_or_smth

3) The 'aggregate' AML aggregates the 90m data to 1km for measures on elevation, slope and topographic wetness index.

Unfortunately, I didn't develop a workflow for the layer processing although I can see the value in doing so. That is, I would generate the layers (only for Oregon) in separate steps - please excuse my lack-of jargons! But I certainly think this would be an excellent thing to do and I could definitely assist you in assembling it.

Best,
Ming

On 4/21/2011 7:30 PM, Rick Reeves wrote:

Awesome!
RR
On 4/21/2011 4:21 PM, Tien Ming Lee wrote:

Hi Rick,

I am checking on the AMLs and will provide a reply tomorrow.

Best,
Ming

On 4/18/2011 3:01 PM, Rick Reeves wrote:

Hello Ming:

Reviewing the AML scripts, I have a few more questions:

1) A summary of the 'pctl' AML and .exe file (that the AML script runs)?
2) A summary of the 'pctl-limits' AML script?
3) A summary of 'aggregate' script?
4) A summary of Chunkproc script?

More importantly, and perhaps this could be the topic of a phone conversation:
can you summarize the end-to-end processing workflow into which these scripts
fit? In other words, had you developed an ordered set of sequence steps that,
when followed, produce the fused terrain layers (other layers later) required
by the project?

I am working to assemble the existing workflow, as implemented in ArcMap GIS,
determine which 'new' steps need to be added, using ESRI and possibly other
software tools.

Thanks,
Rick

Actions

Copy link

Updated by Rick Reeves about 13 years ago

Due date changed from 04/26/2011 to 05/04/2011
Estimated time changed from 4.00 h to 12.00 h

By May 4, I should have reviewed and tested the AML scripts that produce the terrain-derived data layers. In the absence of further
information on the scripts' function from Ming (or other authors), the best way to learn about them is to run them using the 1KM
fused terrain data layer as a test data set.

I propose that immediately after completing the merged Aster/SRTM boundary validation task (by Friday 29 May), I use the fused terrain layer
of the Western Hemisphere that I have created as a test dataset to execute the AML scripts. When doing this, the highest priority will be to
(re-)construct the work flow that will create the derived terrain products (e.g., slope, aspect) using the existing AML scripts, and if
necessary, new AML scripts.

My Windows/ArcMap GIS workstation is a good candidate for performing these tests.

I would like to review this plan with the IPLANT team on the conference call this Friday.

Actions

Copy link