Task #212
closedValidate fused DEM at SRTM/ASTER boundary
Added by Jim Regetz over 13 years ago. Updated almost 13 years ago.
70%
Description
Particularly for the purposes of examining and validating what happens at the SRTM/ASTER boundary, the fused DEM should be compared to independent high quality DEM data available at an appropriate latitude.
The Canadian Digital Elevation Data may be a good candidate?
http://www.geobase.ca/geobase/en/data/cded/index.html
These data are freely available but require registration.
Files
| SampleDemDiffCols.r (7.91 KB) SampleDemDiffCols.r | Rick Reeves, 05/01/2011 11:46 AM | ||
| ImageDifferenceResultsSundayMay1.txt (7.31 KB) ImageDifferenceResultsSundayMay1.txt | Rick Reeves, 05/01/2011 11:46 AM | ||
| SampleImageMosaicCompHists.zip (72.2 KB) SampleImageMosaicCompHists.zip | Rick Reeves, 05/04/2011 05:08 PM | ||
| AsterSRTMBoundaryMisalign.jpg (112 KB) AsterSRTMBoundaryMisalign.jpg | Rick Reeves, 05/05/2011 02:29 PM | ||
| AsterSRTMBoundaryZoom.jpg (12.4 KB) AsterSRTMBoundaryZoom.jpg | Rick Reeves, 05/05/2011 02:29 PM | ||
| AsterSRTMCompBoundaryEven.jpg (17.2 KB) AsterSRTMCompBoundaryEven.jpg | Rick Reeves, 05/06/2011 09:02 AM |
Related issues
Updated by Rick Reeves over 13 years ago
- Due date changed from 04/16/2011 to 04/21/2011
- Start date changed from 04/16/2011 to 04/19/2011
- % Done changed from 0 to 20
- Estimated time set to 1.00 h
- Activity type set to Coding/analysis
Part of the process of constructing and testing the resampling/mosaicing scripts
for constructing the fused terrain layer, I produced a sample mosaic that prominently
features the boundary between the (30 meter resolution) ASTER GDEM and the (90 meter
resolution) CGIAR SRTM) at a resolution of 90 meters.
Next step: QUANTITATIVELY (e.g., review the pixel value) compare the mosaic boundary with
a second, different DEM image (see Description, above) of the same area; also review the
pixel values along each side of the boundary.
Do this with 4 - 5 other Northern Hemisphere locations containing the ASTER / SRTM interface.
Updated by Rick Reeves over 13 years ago
- Priority changed from Normal to High
- % Done changed from 20 to 80
- Estimated time changed from 1.00 h to 4.00 h
As part of the fused digital terrain layer validation we are looking for 'edge discontinuities' at the boundary between the ASTER and SRTM data types.
One technique that we are using is to compare the fused terrain data layer at the boundary area, which occurs at 58 - 62 degrees North Latitude (upper boundary of SRTM is 60 Deg N), with a second, independantly produced digital terrain data set. We located the Canadian Digital Elevation Data (http://www.geobase.ca/geobase/en/data/cded/index.html) as a candidate.
The data are available from an ftp site via a somewhat cumbersome interactive download process: Ultimately, the data are stored in one-degree square tiles, within a three-level hierarchy: At the top level the ftp site contains folders corresponding to 4 (lat) x 8 (long) degree areas. At the second level, each folder contains 16 separate .zip files (2 degree squares) that must be individually downloaded via a mouse click. At the third level, each zip file contains two .DEM format 'E' and 'W' files, corresponding to the east and west 1 degree segment of the 2 degree sub-tile.
I am documenting this structure because I was unable to find a description of it on the data set's web site.
The GDAL'gdalbuildvrt' feature is very useful in generating 'virtual' mosaics out of these many tiles.
Updated by Rick Reeves over 13 years ago
- % Done changed from 80 to 90
I have assembled a good sample region, in Western British Colombia along the 60 degree latitude line
where SRTM and ASTER data sets meet.
To support the validation, I have extracted a dataset from the Canada DEM data set mentioned earlier, that covers this same region.
These data sets are stored as .tif files on Jupiter:
/data/project/organisms/rcr/AsterCgiarMerge
mergeCgiarAsterBdyFinalBL.tif - Mosaic of ASTER and SRTM components
mergeCgiarAsterBdyASTER_BL.tif - SRTM Component mosaic
mergeCgiarAsterBdySRTM_BL.tif
(BL: bilinear resampling used to create mosaics)
/data/project/organisms/rcr/CanadaDEM
CanadaDemMos.tif - Mosaic covers boundary study region (nearest-neighbor resampling used)
Preliminary visual review of these files, using ArcMap GIS zoom and variable transparency tools,
shows a smooth transition between the SRTM and ASTER components.
To resolve this issue completely, I will perform a second analysis which will check the actual pixel values along the boundary line,within several sub-areas that represent different terrain types (e.g., flat terrain, mountains, areas with stream hydrography that crosses the boundary between the two image types).
I have all three
Updated by Jim Regetz over 13 years ago
Rick -- this is a restatement of what I sketched out on the board earlier as one way of quantifying the boundary behavior:
1. Calculate the difference between the fused ASTER/SRTM layer and the CDEM layer. I'll call this set of differences D, for short. To the extent that we regard the CDEM as "topographic truth", then D basically represents the spatial pattern of fused DEM "errors". I think it's a reasonable null hypothesis that the distribution of D is invariant with respect to the 60N boundary, and we can use this as a starting point for statistically testing for the presence of boundary artifacts.
You said you already did this for a subset of the respective layers using the R raster package, but only after manually forcing the extent of one raster to match the other. As we discussed, verify that the magnitude of this artificial shift was indeed insignificant relative to the cell resolution, as you thought.
2. For exploratory purposes, perhaps compare the distribution of D values south of 60N (SRTM zone) with the distribution of values north of 60N (ASTER zone), taken for some sufficiently large set of randomly selected pixels. This won't tell you about boundary artifacts per se, but I think it would paint a useful picture of how ASTER >60N and SRTM <60N each separately compare to GDEM.
D values in adjacent north-south pixels for each of the following (possibly with randomly selected adjacent pairs, but maybe better to randomly choose longitudinal (N-S) one-pixel "strips" across the 60N boundary, and compare within those strips):
- below 60N
- above 60N
- exactly across the 60N boundary
You can then use these values to test e.g. a null hypothesis that the difference in D between pairs of N-S adjacent pixels that lie across the 60N boundary is the same as the difference between N-S adjacent pairs that do not cross the boundary. I imagine it will be good to test for differences in a few different properties of the distributions (mean? variance? other aspects of shape? something about extremes?). Beyond that, perhaps there is even a specific literature on relevant DEM fusion artifacts that bears on this? This may be exactly the kind of thing worth passing by John Gallant for feedback.
4. As we discussed, go back to look at (and report on in an appropriate Redmine issue) the details of your ASTER-SRTM merge operation. Did you simply splice the data at 60N, such that the fused layer contains unadulterated SRTM values south of the boundary, and unadulterated ASTER values to the north? Or was some sort of resampling done that has already resulting in smoothing or some such at the boundary, which probably means expanding the scope of the analysis outline above?
5. Pending results of the above, and discussion with PIs, you should be prepared to re-run the same boundary assessments on:- DEM aggregated to 1km
- Derived terrain layers (slope, aspect, etc) at 90m
- Derived terrain layers at 1km
Everything you're doing should be fully scripted, so this shouldn't imposed any extra burden, but I just wanted to mention it.
Updated by Rick Reeves over 13 years ago
Overlapping with Jim's notes (aboveon our discussions today:
I have created two images corresponding to southern British Columbia, bounding the 60 degree North Latitude line: the first is extracted from the ASTER/CGIAR SRTM image mosaic, the second is extracted from the Canada CDEM image mosaic.
Next, I created an R script, "filterDiffRasterImage.r", that 1) extracts a narrow (<= 100 pixels) subimage along the 60 degree Latitude line from each of these two images, using the R 'raster' package, and 2) subtracts the CDEM subimage from the ASTER/CGIAR SRTM image, creating a 'difference image'.
The next steps:
1) Resolve a possible mis-alignment between the CDEM image and the mosaic terrain image;
2) is to employ this script and the CDEM terrain map to statistically compare the difference between CDEM and each component terrain mosaic type (ASTER GDEM and CGIAR SRTM), with the difference in the mosaic image region along the boundary between image types. The object of this step is to characterize and compare the distribution of the elevation pixel differences along the image mosaic border region with the distribution of differences in the component images (ASTER, CGIAR/SRTM) in areas away from the border. Ideally, the distributions will be similar.
Jim R and I discussed a procedure for accomplishing step 2). I plan to work on this for the remainder of Tuesday and Wednesday.
Updated by Jim Regetz over 13 years ago
Rick -- I just noticed your r-sig-geo exchange with Hijmans, and I want to expand on the associated issue I raised above about whether your raster 'shifting' approach is okay.
First, I think Robert's estimate of the error is off. For starters, if your artificially imposed shift is 0.00036 degrees (which seems to be the case based on what you reported in that post), and if the cell resolution were indeed 0.0083333 as Robert speculated, then the error would be only ~4% of the cell resolution (not 1/4). But moreover, the 3" data you are working with actually has a resolution of 0.0008333 degrees, meaning the shift is ~40% of a cell width. Note that the worst possible offset would be 50% of a cell width, because the clipped rasters won't be misaligned by more than half a pixel -- otherwise it would've clipped different pixels! So at least relative to cell resolution, this is not what I was hoping for when I said to confirm that the amount of shift is insignificant. Doesn't mean it's a fatal problem, but I don't think it can be dismissed without further probing.
As I described yesterday, and as Robert similarly indicated in his reply, the alternative is to resample the CDEM to match your fused DEM. Actually, I suppose what you did by slightly shifting the raster really equates to nearest-neighbor resampling, but now we're talking bilinear (or cubic, etc) resampling. Unfortunately, this raises its own issues because it actually alters the data values in some way. It's not completely clear to me which is the lesser of two evils, but my suggestion in cases like this is to try it both ways and see if it even matters in the first place (my guess is, probably not). As always, this should be simple enough as long as you are setting everything up in a scripted manner.
Updated by Rick Reeves over 13 years ago
Jim:
I think that Robert pointed me in the correct direction, but I took some steps to quantify the error and test it's effect
on the difference image results by examining the CDEM and merged ASTER/SRTM images at 'close range'.
The 'shift' issue occurs because the pixels in these two terrain images have slightly different 'origins'. The difference
is less than one pixel, but a different amount, in each dimension. By superimposing the images and increasing the magnification
on the lower left corner of the image pair and using the mouse to obtain the corner coords in decimal degrees, I made this estimate:
Origin (LLC) coordinates
CDEM ASTER/CIGAR merge difference/offset
degrees arc-sec meters
X -135.9882 -135.9881 .0001 0.33 ~10 meters @ 58 deg N
Y 58.4653 58.4648 .0005 1.8 ~50 metersAs you say, the offset is a bit larger than Robert estimated, esp in the Y dimension.
So, I created a resampled image in which (using the images from which I obtained the above coordinates),I sampled the
ASTER/CGIAR image into the coordinate space of the (identically-sized) CDEM image, using gdalwarp:
gdalwarp -te -135.9882 58.4653 -r bilinear asterCgiarSub.tif asterCgiarSub.tif (as I remember)
Then, I extracted the same type of 10 row 'boundary image' centered on the 60 degree N lat line, as before, and performed
image difference calculation using this image and the coincident CDEM image. I will review the actual numbers when I get
to work (after the library), but I remember that the mean of the resampled difference image was slightly larger than that
of the 'offset' image. I think that the difference was, -4 vs -3 (the offset image). The Std Dev's were very close, say,
25 vs 24. I want to reconfirm these numbers, and then create separate difference images for CDEM vs Aster and CDEM vs SRTM,
to see how they compare. This should all be done by just after lunch if not sooner. I will check in the (short) R and Shell
scripts that I created to perform this task.
My feeling is that the differences in the boundary regions will not be statistically different from the 'outlying' regions,
but I will wait for the actual numbers to see for sure.
Updated by Rick Reeves over 13 years ago
The book is on order, and should arrive at UCSB Library in 3 to 5 working days.
It will come in handy in several parts of the Fused Terrain Layer work, but especially in constructing the derived terrain layers.
Updated by Mark Schildhauer over 13 years ago
Presumably "the book" refers Terrain Analysis (2000) by Wilson and Gallant? Has the state of terrain analysis not advanced much over the past decade? I think we should be aware that the types of coverages we are using (ASTER GDEM, SRTM) did not exist when the book was written, nor did some of the currently popular analytical libraries (R raster, gdal, etc.). Just a cautionary note that the information in there might be outdated.
Updated by Rick Reeves over 13 years ago
Yes, the book IS the Wilson and Gallant book. This book is a decade old, and new techniques no doubt have been developed
that we need to review and test as we develop workflows. So, more research is in order, even as we wait for this book
to arrive next week.
Good reasons to have a look at this book:
- It is well-reviewed
- One of the authors is a recognized authority on TA, and a member of this WG
- The table of contents contains topics relevant to this project
- The UC Library system is providing it free-of-charge :}
Updated by Jim Regetz over 13 years ago
A few comments:
Rick Reeves wrote:
By superimposing the images and increasing the magnification
on the lower left corner of the image pair and using the mouse to obtain the corner coords in decimal degrees, I made this estimate:
I'm curious, why not obtain these values by querying them from the raster programmatically, rather than doing it manually? E.g., using gdalinfo, or extent() in the raster package, etc. Also, no need to convert the offset to meters (which of course varies by latitude in the X direction anyway). I really do think the more important thing is the offset as a proportion of cell resolution.
So, I created a resampled image in which (using the images from which I obtained the above coordinates),I sampled the
ASTER/CGIAR image into the coordinate space of the (identically-sized) CDEM image, using gdalwarp:
Important: What I had said was to do the opposite, namely resample the CDEM into the space of your fused DEM. Otherwise all of your validation is based on a resampled version on the fused DEM, not the fused DEM itself.
gdalwarp -te -135.9882 58.4653 -r bilinear asterCgiarSub.tif asterCgiarSub.tif (as I remember)
Is the exact command recorded somewhere, whether as part of a script or otherwise?
I will review the actual numbers when I get
to work (after the library), but I remember that the mean of the resampled difference image was slightly larger than that
of the 'offset' image. I think that the difference was, -4 vs -3 (the offset image). The Std Dev's were very close, say,
25 vs 24.
Good to start to see some numbers, though again you'll need to re-do things after obtaining the resampled CDEM. You should then look at the correlation coefficient between the two (i.e., between the shifted CDEM and resampled CDEM). Also, I would calculate the mean/sd/range of the difference between these two CDEM rasters, rather than comparing the stats computed separately on each of them. Lastly, to clarify my earlier comments, I think unless these two different aligned CDEM rasters are so similar that it would be obviously pointless, and assuming it is trivial to just re-run your scripted validation workflow, I think it would be useful to do the same fused layer validation calculations separately using the shifted CDEM and the resampled CDEM to see if it matters.
Updated by Rick Reeves over 13 years ago
Jim:
As I mentioned, after I completed my initial 'screen based' review of the boundary area, I intend to perform the statistical analysis programmatically, as you say.
Important: What I had said was to do the opposite, namely resample the CDEM into the space of your fused DEM. Otherwise all of your validation is based on a resampled version on the fused DEM, not the fused DEM itself.
I initially resampled the fused DEM into the CDEM space in order to see how much the elevation values changed
during the resample operation. There was almost no change in the elevation values. This result should not change if I resampled the CDEM into the DEM space, should it?
And I intend to compute the statistics on the difference between the CDEM rasters, and present the information.
BTW, here is the R script that I used to generate the edge region and difference subimages:
require(raster) require(rgdal) inputFirstRaster <- raster(sFirstImageName) inputSecondRaster <- raster(sSecondImageName) # Create extent objects used to extract raster subimges. # The object will be centered along the 60 degree North latitude line, # and have varying depths (number of rows). # The Western Canada study area runs from -135 (west) to -100 (west) longitude, # and 55.0 to 64.00 degrees (north) latitude. #eTestAreaExtent <- extent(-135.2,-100.2, 59.995,60.005) # Creates a 12 row subimage eTestAreaExtent <- extent(-135.2,-100.2, 59.997,60.001) # Creates a 5 row subimage # Extract a sub image corresponding to the selected extent. # Two different alternatives: # The extract() function returns a vector of cell values, # the crop() function returns a complete raster* object. vEdgeRegionFirst <- extract(inputFirstRaster,eTestAreaExtent) rEdgeRegionFirst <- crop(inputFirstRaster,eTestAreaExtent) vEdgeRegionSecond <- extract(inputSecondRaster,eTestAreaExtent) rEdgeRegionSecond <- crop(inputSecondRaster,eTestAreaExtent) # Important: In order for the image subtraction to work, the extents # of the two images must be IDENTICAL. I used ArcMap GIS Raster Crop By Mask # to create subimages with identical extents. rDelta <- rEdgeRegionFirst - rEdgeRegionSecond # TBD: compute statistics for the input and difference images created here. # Write the extracted subimage and its difference image to disk # For now, use 'gdalinfo' to check image statistics writeRaster(rEdgeRegionFirst,filename="EdgeRegionFirst.tif",format="GTiff",datatype="INT2S",overwrite=TRUE) writeRaster(rEdgeRegionAsterCgiar,filename="EdgeRegionSecond.tif",format="GTiff",datatype="INT2S",overwrite=TRUE) writeRaster(rDelta,filename="EdgeRegionDelta.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
[formatting edit by regetz, 06-May-2011]
Updated by Rick Reeves over 13 years ago
I am (finally) working through the 'image difference' analysis that Jim sketched out earlier this week.
And the first thing that I did was, as he recommends, resample the CDEM image into the 'extent space' of
the ASTER/CGIAR mosaic. As expected, the statistics for the 'before' and 'after' images are very close
(note that as expected) the image sizes and resolutions did not change during the resampling):
gdalinfo -stats output - CDEM before:
reeves@eos:/data/project/organisms/rcr/ValidateBoundary$ gdalinfo -stats CanadaDemMosTuesdayClip.tif | more
Driver: GTiff/GeoTIFF
Files: CanadaDemMosTuesdayClip.tif
CanadaDemMosTuesdayClip.aux
CanadaDemMosTuesdayClip.rrd
Size is 41582, 3738
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-135.987917500000009,61.579680199999999)
Pixel Size = (0.000833300000000,-0.000833300000000)
Metadata:
TIFFTAG_SOFTWARE=IMAGINE TIFF Support
Copyright 1991 - 1999 by ERDAS, Inc. All Rights Reserved
@(#)$RCSfile: etif.c $ $Revision: 1.10.1.9.1.9.2.11 $ $Date: 2004/09/15 18:42:01EDT $
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1
TIFFTAG_RESOLUTIONUNIT=1 (unitless)
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-135.9879175, 61.5796802) (135d59'16.50"W, 61d34'46.85"N)
Lower Left (-135.9879175, 58.4648048) (135d59'16.50"W, 58d27'53.30"N)
Upper Right (-101.3376369, 61.5796802) (101d20'15.49"W, 61d34'46.85"N)
Lower Right (-101.3376369, 58.4648048) (101d20'15.49"W, 58d27'53.30"N)
Center (-118.6627772, 60.0222425) (118d39'46.00"W, 60d 1'20.07"N)
Band 1 Block=64x64 Type=Int16, ColorInterp=Gray
Min=0.000 Max=2799.000
Minimum=0.000, Maximum=2799.000, Mean=613.351, StdDev=420.457
NoData Value=-9999
Overviews: 10396x935, 5198x468, 2599x234, 1300x117, 650x59
Metadata:
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2799
STATISTICS_MEAN=613.35056498368
STATISTICS_MEDIAN=440
STATISTICS_MODE=0
STATISTICS_STDDEV=420.45694241697
STATISTICS_SKIPFACTORX=1
STATISTICS_SKIPFACTORY=1
STATISTICS_EXCLUDEDVALUES=-9999
LAYER_TYPE=athematic
gdalinfo -stats output - Resampled CDEM:
reeves@eos:/data/project/organisms/rcr/ValidateBoundary$ gdalinfo -stats CDemMosTuesdayClipMergeSpace.tif
Driver: GTiff/GeoTIFF
Files: CDemMosTuesdayClipMergeSpace.tif
CDemMosTuesdayClipMergeSpace.aux
CDemMosTuesdayClipMergeSpace.rrd
CDemMosTuesdayClipMergeSpace.tif.aux.xml
Size is 41582, 3738
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-135.988060499999989,61.579322400000002)
Pixel Size = (0.000833300000000,-0.000833300000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N)
Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N)
Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N)
Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N)
Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
Band 1 Block=41582x1 Type=Int16, ColorInterp=Gray
Min=0.000 Max=2796.000
Minimum=0.000, Maximum=2796.000, Mean=613.354, StdDev=420.415
Overviews: 10396x935, 5198x468, 2599x234, 1300x117, 650x59
Metadata:
LAYER_TYPE=athematic
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2796
STATISTICS_MEAN=613.35357465632
STATISTICS_STDDEV=420.41456798272
For reference, extent of merged ASTER/SRTM image:
Corner Coordinates: Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N) Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N) Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N) Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N) Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
This file: CDemMosTuesdayClipMergeSpace.tif - will be the basis for the difference calculations..
[formatting edit by regetz, 06-May-2011]
Updated by Rick Reeves over 13 years ago
Was:
gdalwarp -te -135.9880605 58.4644470 -101.3377799 61.5793224 -r bilinear CanadaDemMosTuesdayClip.tif
CDemMosTuesdayClipMergeSpace.tif
The -te coordinates are the from the gdalinfo command output for the merged Aster/CGIAR mosaic.
A note: the CDEM and merged images that I am using for this were created with a 'clip' operation,
using ArcMap GIS to create a clipping polygon covering the largest area common to both images,
and the Raster Extract By Mask command. Of course, even though clipped to the same polygon, the
origin/extent of the two images differ slightly.
Updated by Rick Reeves over 13 years ago
- File SampleDemDiffCols.r SampleDemDiffCols.r added
- File ImageDifferenceResultsSundayMay1.txt ImageDifferenceResultsSundayMay1.txt added
Here is the output from a set of 'image difference' calculations made using the R script 'SampleDemDiffCols.r",
which implements the procedure that Jim charted out. Both files are attached to this update.
Per Mark's Friday afternoon email question, at this point I am just comparing standard summary statistics
(mean/median/min/max/Var/SD) of the different images. I wanted to consult with Jim and Mark prior to making
other tests.
One of the things that these statistics show is that there IS a distinct discontinuity along the 60 degree border, and that its average magnitude is approximately the same as the DIFFERENCE between the ASTER/CGIAR and CDEM images. This is true for the difference image created for the large study area covered by both CDEM and ASTER/CGIAR and for the narrow subimage that straddles the 60 degree line.
For large study area image:
ASTER/CGIAR: Minimum=-74.000, Maximum=3851.000, Mean=623.511, StdDev=419.871
CDEM : Minimum=0.000, Maximum=2796.000, Mean=613.354, StdDev=420.415
Difference : Minimum=-1603.000, Maximum=2770.000, Mean=9.047, StdDev=132.886
Image
For 'boundary area subimage:
ASTER/CGIAR: Minimum=122.000, Maximum=2030.000, Mean=586.856, StdDev=317.868
CDEM : Minimum=166.000, Maximum=2011.000, Mean=589.811, StdDev=319.20
Difference: Minimum=-310.000, Maximum=310.000, Mean=-2.955, StdDev=25.854
Image
No doubt there are more things to discuss; but these results seem to suggest that
the boundary feature may be dominated by the difference in mean elevation between
the CDEM and ASTER/CGIAR data sets. And that the difference IS small relative to the
average elevation in each image.
Maybe now we test to see whether the different regions? or ?
---Rick
OUTPUT from the SampleDemDiffCols.r (see next update for the code listing)
Two runs in which the same (randomly selected) columns were sampled for each region
statistics from sampling 1000 N/S adjacent pixel pairs from three mosaic image regions:
ASTER sample summary: Min: 64.000000 / Median: 0 / Mean: 0.022000 / Max: 114.000000 / Variance: 112.043560 sDev: 10.585063
Border sample summary: Min: -60.000000 / Median: 11 / Mean: 11.802254 / Max: 97.000000 / Variance: 188.880856 sDev: 13.743393
STRM sample summary: Min: -53.000000 / Median: 0 / Mean: 0.555668 / Max: 81.000000 / Variance: 96.346442 sDev: 9.815622
-----------------
statistics for 1000 N/S adjacent pixel pairs from three mosaic image regions:
ASTER sample summary: Min: -73.000000 / Median: 0 / Mean: -0.108000 / Max: 139.000000 / Variance: 164.106442 sDev: 12.810404
Border sample summary: Min: -49.000000 / Median: 10 / Mean: 11.077236 / Max: 69.000000 / Variance: 172.234110 sDev: 13.123799
STRM sample summary: Min: -57.000000 / Median: 0 / Mean: 0.625253 / Max: 72.000000 / Variance: 98.998957 sDev: 9.949822
Two runs in which different columns were sampled for each region
statistics for 1000 N/S adjacent pixel pairs from three mosaic image regions:
ASTER sample summary: Min: 68.000000 / Median: 0 / Mean: 0.195000 / Max: 169.000000 / Variance: 155.256231 sDev: 12.460186
Border sample summary: Min: -53.000000 / Median: 11 / Mean: 11.782117 / Max: 91.000000 / Variance: 187.481285 sDev: 13.692381
STRM sample summary: Min: -60.000000 / Median: 0 / Mean: 0.464032 / Max: 64.000000 / Variance: 93.520764 sDev: 9.670613
------------------
ASTER sample summary: Min: -95.000000 / Median: 0 / Mean: 0.158000 / Max: 144.000000 / Variance: 134.129165 sDev: 11.581415
Border sample summary: Min: -65.000000 / Median: 11 / Mean: 11.775862 / Max: 79.000000 / Variance: 202.507072 sDev: 14.230498
STRM sample summary: Min: -76.000000 / Median: 0 / Mean: -0.039394 / Max: 58.000000 / Variance: 69.659721 sDev: 8.346240
GDALINFO Statistics:
***************Statistics for LARGE difference image:Files: DeltaEntireImage.tif
Size is 41582, 3738
Corner Coordinates:
Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N)
Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N)
Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N)
Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N)
Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
Band 1 Block=41582x1 Type=Int16, ColorInterp=Gray
Min=-1603.000 Max=2770.000
- Minimum=-1603.000, Maximum=2770.000, Mean=9.047, StdDev=132.886
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-1603
STATISTICS_MAXIMUM=2770
STATISTICS_MEAN=9.0470816079223
STATISTICS_STDDEV=132.8857558074
- Statistics for Input ASTER/CGIAR image:
mergeCgiarAsterBdyTuesdayClip.tif
Size is 41582, 3738
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-135.988060499999989,61.579322388888890)
Pixel Size = (0.000833300000000,-0.000833300000000)
Metadata:
Corner Coordinates:
Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N)
Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N)
Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N)
Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N)
Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
Band 1 Block=64x64 Type=Int16, ColorInterp=Gray
Min=-74.000 Max=3851.000
- Minimum=-74.000, Maximum=3851.000, Mean=623.511, StdDev=419.871
NoData Value=-9999
Overviews: 10396x935, 5198x468, 2599x234, 1300x117, 650x59
Metadata:
STATISTICS_MINIMUM=-74
STATISTICS_MAXIMUM=3851
STATISTICS_MEAN=623.51100299321
STATISTICS_MEDIAN=442
STATISTICS_MODE=207
STATISTICS_STDDEV=419.87071089235
STATISTICS_SKIPFACTORX=1
STATISTICS_SKIPFACTORY=1
STATISTICS_EXCLUDEDVALUES=-9999- Statistics for input CDEM image:
CDemMosTuesdayClipMergeSpace.tif
Size is 41582, 3738
Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N)
Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N)
Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N)
Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N)
Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
Band 1 Block=41582x1 Type=Int16, ColorInterp=Gray
Min=0.000 Max=2796.000
- Statistics for input CDEM image:
- Minimum=0.000, Maximum=2796.000, Mean=613.354, StdDev=420.415
Overviews: 10396x935, 5198x468, 2599x234, 1300x117, 650x59
Metadata:
LAYER_TYPE=athematic
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2796
STATISTICS_MEAN=613.35357465632
STATISTICS_STDDEV=420.41456798272
- Statistics for 'Edge Region' Difference Map subimage (12 rows):
Files: EdgeRegionDelta.tif
Size is 40636, 12
Corner Coordinates:
Upper Left (-135.1997587, 60.0052187) (135d11'59.13"W, 60d 0'18.79"N)
Lower Left (-135.1997587, 59.9952191) (135d11'59.13"W, 59d59'42.79"N)
Upper Right (-101.3377799, 60.0052187) (101d20'16.01"W, 60d 0'18.79"N)
Lower Right (-101.3377799, 59.9952191) (101d20'16.01"W, 59d59'42.79"N)
Center (-118.2687693, 60.0002189) (118d16'7.57"W, 60d 0'0.79"N)
Band 1 Block=40636x1 Type=Int16, ColorInterp=Gray
Min=-310.000 Max=310.000- Minimum=-310.000, Maximum=310.000, Mean=-2.955, StdDev=25.854
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-310
STATISTICS_MAXIMUM=310
STATISTICS_MEAN=-2.9551362502871
STATISTICS_STDDEV=25.854081053692
*********Statstics for ASTER/SRTM raster, 'edge region'
Files: EdgeRegionFirst.tif
Size is 40636, 12
Corner Coordinates:
Upper Left (-135.1997587, 60.0052187) (135d11'59.13"W, 60d 0'18.79"N)
Lower Left (-135.1997587, 59.9952191) (135d11'59.13"W, 59d59'42.79"N)
Upper Right (-101.3377799, 60.0052187) (101d20'16.01"W, 60d 0'18.79"N)
Lower Right (-101.3377799, 59.9952191) (101d20'16.01"W, 59d59'42.79"N)
Center (-118.2687693, 60.0002189) (118d16'7.57"W, 60d 0'0.79"N)
Band 1 Block=40636x1 Type=Int16, ColorInterp=Gray
Min=122.000 Max=2030.000
- Minimum=-310.000, Maximum=310.000, Mean=-2.955, StdDev=25.854
- Minimum=122.000, Maximum=2030.000, Mean=586.856, StdDev=317.868
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=122
STATISTICS_MAXIMUM=2030
STATISTICS_MEAN=586.85635274141
STATISTICS_STDDEV=317.8676281026
******Statistics for CDEM edge region:
Files: EdgeRegionSecond.tif
Size is 40636, 12
Corner Coordinates:
Upper Left (-135.1997587, 60.0052187) (135d11'59.13"W, 60d 0'18.79"N)
Lower Left (-135.1997587, 59.9952191) (135d11'59.13"W, 59d59'42.79"N)
Upper Right (-101.3377799, 60.0052187) (101d20'16.01"W, 60d 0'18.79"N)
Lower Right (-101.3377799, 59.9952191) (101d20'16.01"W, 59d59'42.79"N)
Center (-118.2687693, 60.0002189) (118d16'7.57"W, 60d 0'0.79"N)
Band 1 Block=40636x1 Type=Int16, ColorInterp=Gray
Min=166.000 Max=2011.000 - Minimum=166.000, Maximum=2011.000, Mean=589.811, StdDev=319.203
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=166
STATISTICS_MAXIMUM=2011
STATISTICS_MEAN=589.8114889917
STATISTICS_STDDEV=319.2031048169
Updated by Rick Reeves over 13 years ago
#
# Usage:
# > setwd("/data/project/organisms/rcr/ValidateBoundary/")
# > source("SampleDemDiffCols.r")
# > SampleDemDiffCols() OR simply clip out the script between the outermost { }
##############################################################################################
#
# SampleDemDiffCols
#
# R script generates (or reads in) the CDEM / ASTER-SRTM mosaic 'difference image',
# and for randomly-selected one column / two row subimages, computes and saves tbe
# difference between the pixels in the pair to create a distribution of differences.
# Three distributions created: North (ASTER CDEM), South i(SRTM/CGIAR), and
# Border (boundary between ASTER and SRTM), Summary statistics are created for each
# distribution.
#
# Author: Rick Reeves, NCEAS
# April 29, 2011
##############################################################################################
#
SampleDemDiffCols <- function()
{
require(raster)
require(rgdal)
#inputFirstRaster <- raster(sFirstImageName)
#inputSecondRaster <- raster(sSecondImageName)
inputFirstRaster <- raster("/data/project/organisms/rcr/ValidateBoundary/mergeCgiarAsterBdyTuesdayClip.tif")
inputSecondRaster <- raster("/data/project/organisms/rcr/ValidateBoundary/CDemMosTuesdayClipMergeSpace.tif")
#
# Difference image for entire merged image takes a while to create,
# so we created it once, now read it back in.
#
rDeltaWhole <- raster("/data/project/organisms/rcr/ValidateBoundary/DeltaEntireImage.tif")
rDeltaWhole@data@values <-getValues(rDeltaWhole)
# Create extent objects used to extract raster subimges.
# The object will be centered along the 60 degree North latitude line,
# and have varying depths (number of rows).
# The Western Canada study area runs from -135 (west) to -100 (west) longitude,
# and 55.0 to 64.00 degrees (north) latitude.
# the ASTER and SRTM/CGIAR image components are merged at the 60 Deg N Latitude line.
#eTestAreaExtent <- extent(-135.2,-100.2, 59.997,60.001) # Creates a 5 row subimage
eTestAreaExtent <- extent(-135.2,-100.2, 59.995,60.005) # Creates a 12 row subimage
# Extract a sub image corresponding to the selected extent.
# Two different alternatives:
# The extract() function returns a vector of cell values,
# the crop() function returns a complete raster* object.
vEdgeRegionFirst <- extract(inputFirstRaster,eTestAreaExtent)
rEdgeRegionFirst <- crop(inputFirstRaster,eTestAreaExtent)
vEdgeRegionSecond <- extract(inputSecondRaster,eTestAreaExtent)
rEdgeRegionSecond <- crop(inputSecondRaster,eTestAreaExtent)
# Important: In order for the image subtraction to work, the extents
# of the two images must be IDENTICAL. I used ArcMap GIS Raster Crop By Mask
# to create subimages with identical extents.
# Compute the difference image for the entire study area, and for the region along
# the boundary (narrow, maybe 10 pixels either side)
rDeltaEdge <- rEdgeRegionFirst - rEdgeRegionSecond
# Create this image one time, read it in thereafter.
#rDeltaWhole <- inputFirstRaster - inputSecondRaster
#writeRaster(rDeltaWhole,filename="DeltaEntireImage.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
# Using the large difference image, compute subimagee statistics for areas
# North (ASTER) and South (SRTM) of the boundary. These give us an idea
# re: differences between ASTER and CDEM and CGIAR/SRT and CDEM
# what is raster package way of using subscripts to extract?
# Now, the interesting part: using the boundary difference image, randomly select
# one-degree N-S strips throughout the image, and compare adjacent pixel pairs
# above and below the boundary with pixel pairs straddling the boundary. Subtract
# the pairs, save the collection of (absolute value) of the differences in a vector,
# so that we have a population of differences above, below, and straddling the boundary
# line. Compare the populations.
# get a vector of random column index numbers, constrained by column dimension of image
# Loop three times, sampling pixel pairs from above, below, across the border
nColsToGet <- 1000
iDiffVecNorth <- vector(mode="integer",length=nColsToGet)
iDiffVecBorder <- vector(mode="integer",length=nColsToGet)
iDiffVecSouth <- vector(mode="integer",length=nColsToGet)
#colsToGet <-sample(1:50,nColsToGet)
# Note: initially, sample the same columns in all regions to get a profile.
# other 'sample()' calls can be commented out to sample differenct
# coluns in each 'region'.
# iDiffVecxxxx is a population of differences between adjacent cell pairs.
# Compute iDiffVecNorth/Border/South on either side of border, and across it.
# note that North and South samples taken from larger difference image for
# entire mosaic (sub) image; iDiffBorder taken from the edge region extracted
# from the center of the lerger image.
# Remember, we are sampling a PAIR of pixels (same column from two adjacent rows)
colsToGet <-sample(1:inputFirstRaster@ncols,nColsToGet)
message("North Sample")
#browser()
# debug
#nColsToGet <- 2
#colsToGet <- c(20,100)
iFirstRow <- 300
iCtr = 1
for (iNextCol in colsToGet)
{
rColVec <- cellFromRowCol(rDeltaWhole,iFirstRow:(iFirstRow+1),iNextCol:iNextCol)
neighborCells <- rDeltaWhole@data@values[rColVec]
iDiffVecNorth[iCtr] <- neighborCells[2] - neighborCells[1]
iCtr = iCtr + 1
}
#
message("Border Sample - different columns")
#browser()
colsToGet <-sample(1:inputFirstRaster@ncols,nColsToGet)
iFirstRow <- 6 # straddle the border of 12 row center section
iCtr = 1
for (iNextCol in colsToGet)
{
rColVec <- cellFromRowCol(rDeltaEdge,iFirstRow:(iFirstRow+1),iNextCol:iNextCol)
neighborCells <- rDeltaEdge@data@values[rColVec]
iDiffVecBorder[iCtr] <- neighborCells[2] - neighborCells[1]
iCtr = iCtr + 1
}
#
message("South Sample - different columns")
#browser()
colsToGet <-sample(1:inputFirstRaster@ncols,nColsToGet)
iFirstRow <- 3600
iCtr = 1
for (iNextCol in colsToGet)
{
rColVec <- cellFromRowCol(rDeltaWhole,iFirstRow:(iFirstRow+1),iNextCol:iNextCol)
neighborCells <- rDeltaWhole@data@values[rColVec]
iDiffVecSouth[iCtr] <- neighborCells[2] - neighborCells[1]
iCtr = iCtr + 1
}
# Compute iDiffVecs on either side of border, and across it.
message("Check the cell difference vectors...")
#browser()
# summary stats for each population
sNorthSum <- sprintf("ASTER sample summary: Min: %f / Median: %d / Mean: %f / Max: %f / Variance: %f sDev: %f",
min(iDiffVecNorth,na.rm=TRUE),median(iDiffVecNorth,na.rm=TRUE),mean(iDiffVecNorth,na.rm=TRUE),
max(iDiffVecNorth,na.rm=TRUE),var(iDiffVecNorth,na.rm=TRUE),sd(iDiffVecNorth,na.rm=TRUE))
sBorderSum <- sprintf("Border sample summary: Min: %f / Median: %d / Mean: %f / Max: %f / Variance: %f sDev: %f",
min(iDiffVecBorder,na.rm=TRUE),median(iDiffVecBorder,na.rm=TRUE),mean(iDiffVecBorder,na.rm=TRUE),
max(iDiffVecBorder,na.rm=TRUE),var(iDiffVecBorder,na.rm=TRUE),sd(iDiffVecBorder,na.rm=TRUE))
sSouthSum <- sprintf("STRM sample summary: Min: %f / Median: %d / Mean: %f / Max: %f / Variance: %f sDev: %f",
min(iDiffVecSouth,na.rm=TRUE),median(iDiffVecSouth,na.rm=TRUE),mean(iDiffVecSouth,na.rm=TRUE),
max(iDiffVecSouth,na.rm=TRUE),var(iDiffVecSouth,na.rm=TRUE),sd(iDiffVecSouth,na.rm=TRUE))
#
message(sprintf("statistics for %d N/S adjacent pixel pairs from three mosaic image regions:",nColsToGet))
message(sNorthSum)
message(sBorderSum)
message(sSouthSum)
message("hit key to write output images...")
browser()
# Write the extracted subimage and its difference image to disk
# For now, use 'gdalinfo' to check image statistics
writeRaster(rEdgeRegionFirst,filename="/data/project/organisms/rcr/ValidateBoundary/EdgeRegionFirstDC.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
writeRaster(rEdgeRegionSecond,filename="/data/project/organisms/rcr/ValidateBoundary/EdgeRegionSecondDC.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
writeRaster(rDeltaEdge,filename="/data/project/organisms/rcr/ValidateBoundary/EdgeRegionDeltaDC.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
}
Updated by Rick Reeves over 13 years ago
- Estimated time changed from 4.00 h to 34.00 h
I am reporting report 30 hours spent on this task since April 25.
Email on 5/02 to Rob Guranick:
Re: Border area 'uncertainty': I have performed some analysis to quantify the border area anomalies (edge boundary), and the first results
are posted on our Redmine site. These results need to be consolidated a bit, but you can review them:
https://redmine.nceas.ucsb.edu/nceas/issues/212
..scroll down to entry # 17 to see the output from the R script that I wrote, and 'GDALinfo -stats' output.
I used CDEM digital elevation data, produced independently of the ASTER and CGIAR data (photogrammetrically) and straddling the areas North (ASTER) and South (CGIAR/SRTM) of the boundary, to compute a 'difference image': (ASTER/CGIAR Mosaic - CDEM image). From this, I extracted three 'populations' of differences: Areas north, south, and straddling the 60 degree Latitude (border) area. I am running some statistical tests on these populations; what I initially see is that the means of the North and South area 'difference' populations are both close to zero, while the mean of the border 'difference' population is approx equal to the difference between the means of the North and South ELEVATION DATA populations.
Mean of Difference Image: -2.9
CDEM average elevation in border region: 589
Mosaic average elevation in border region: 586
So the difference is small relative to the elevation values, as well.
More analysis and re-checking needed.
When I further verify these results and report them (very soon) you can evaluate and decide.
Just thought that I would keep you posted...
Rick
Updated by Mark Schildhauer over 13 years ago
Rick Reeves wrote:
Preliminary visual review of these files, using ArcMap GIS zoom and variable transparency tools,
shows a smooth transition between the SRTM and ASTER components.
This is not what I recall when you, Jim, and I met last Wed or so-- rather by zooming closely in to one small area, we observed what appeared to be a very distinct boundary artifact. This clearly recommended that more comprehensive and quantitative treatment is needed to determine the extent to which boundary issues were problematic, and how to develop the best strategy for integrating these two coverages.
mps
Updated by Rick Reeves over 13 years ago
I retrieved the Wilson and Gallant text from UCSB Library, and reviewed several of the chapters.
At the very least, the first five chapters provide very good information re: the nature of the
data source(s) that we are using. Also some insights into solving the '60 degree' boundary issue.
The book is in our office, ready for your perusal.
Looking forward to discussing these with the project team.
Updated by Rick Reeves over 13 years ago
< Rob's request is at the bottom of this posting..>
Rob:
Here is a 'distilled' set of image statistics. I will also post in Redmine, but thought to also send to you.
I will generate image histograms using R and send to you, along with R code. All of these images
are on the /Jupiter server under the subfolders 'rcr/AsterCgiarMerge' and 'rcr/ValidateBoundary'
Three groups of results.....
Rick
*********************************************************************************
1) Statistics for mosaic image components North (ASTER) and South (CGIAR / SRTM) of 60 Deg N Lat:
ASTER image component (mergeCgiarAsterBdyASTER_BL.tif)
Size is 48002, 2400
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=-93.000, Maximum=6366.000, Mean=665.185, StdDev=499.388
Metadata:
STATISTICS_MINIMUM=-93
STATISTICS_MAXIMUM=6366
STATISTICS_MEAN=665.18477873318
STATISTICS_STDDEV=499.38831312001
CGIAR / SRTM image component (mergeCgiarAsterBdySRTM_BL.tif) (different (of course) and larger area than ASTER)
Size is 60002, 6000
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=-329.000, Maximum=4324.000, Mean=750.596, StdDev=450.075
NoData Value=-9999
Metadata:
STATISTICS_MINIMUM=-329
STATISTICS_MAXIMUM=4324
STATISTICS_MEAN=750.59572054153
STATISTICS_STDDEV=450.07531631289
These two files, mosaiced together (bilinear resampling) (mergeCgiarAsterBdyFinalBL.tif)
Size is 60002, 8400
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=-329.000, Maximum=6366.000, Mean=720.989, StdDev=469.431
STATISTICS_MINIMUM=-329
STATISTICS_MAXIMUM=6366
STATISTICS_MEAN=720.98934529784
STATISTICS_STDDEV=469.431391487922
2) Next, for a subiimage focusing on the boundary region, for which CDEM imagery is available:
Merged ASTER/SRTM Image: mergeCgiarAsterBdyTuesdayClip.tif
Size is 41582, 3738
Origin = (-135.988060499999989,61.579322388888890)
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=-74.000, Maximum=3851.000, Mean=623.511, StdDev=419.871
STATISTICS_MINIMUM=-74
STATISTICS_MAXIMUM=3851
STATISTICS_MEAN=623.51100299321
STATISTICS_STDDEV=419.87071089235
CDEM Image: CDemMosTuesdayClipMergeSpace.tif
Size is 41582, 3738
Origin = (-135.988060499999989,61.579322400000002)
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=0.000, Maximum=2796.000, Mean=613.354, StdDev=420.415
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2796
STATISTICS_MEAN=613.35357465632
STATISTICS_STDDEV=420.41456798272
Difference Image: DeltaEntireImage.tif (Merged ASTER/SRTM - CDEM image)
Size is 41582, 3738
Origin = (-135.988060499999989,61.579322388999998)
Pixel Size = (0.000833300000000,-0.000833300000000)
Minimum=-1603.000, Maximum=2770.000, Mean=9.047, StdDev=132.886
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-1603
STATISTICS_MAXIMUM=2770
STATISTICS_MEAN=9.0470816079223
STATISTICS_STDDEV=132.8857558074
3) Statistics for population of randomly sampled pixel pairs (capturing North to South pixel-to-pixel change,
as would occur along the mosaic component boundary) from the DIFFERENCE Image, computed by subtracting
the CDEM image from the ASTER/SRTM mosaic image:
The mean of the difference between adjacent pixels at the boundary (11.8) is close to the
mean of the difference between the larger ASTER/SRTM image and the corresponding CDEM image (9.05).
statistics from sampling 1000 N/S adjacent pixel pairs from three mosaic image regions:
ASTER sample summary: Min: -64.000000 / Median: 0 / Mean: 0.022000 / Max: 114.000000 / Variance: 112.043560 sDev: 10.585063
Border sample summary ***: Min: -60.000000 / Median: 11 / Mean: 11.802254 / Max: 97.000000 / Variance: 188.880856 sDev: 13.743393
STRM sample summary: Min: -53.000000 / Median: 0 / Mean: 0.555668 / Max: 81.000000 / Variance: 96.346442 sDev: 9.815622
statistics from sampling 20000 N/S adjacent pixel pairs from three mosaic image regions:
statistics for 20000 N/S adjacent pixel pairs from three mosaic image regions:
ASTER sample summary: Min: -95.000000 / Median: 0 / Mean: -0.110150 / Max: 169.000000 / Variance: 126.035519 sDev: 11.226554
Border sample summary ***: Min: -92.000000 / Median: 11 / Mean: 11.763342 / Max: 270.000000 / Variance: 214.449517 sDev: 14.644095
STRM sample summary: Min: -83.000000 / Median: 0 / Mean: 0.397250 / Max: 115.000000 / Variance: 74.741780 sDev: 8.645333
*************************************************************************************
On 5/4/2011 12:02 PM, Robert Guralnick wrote:
> Rick --- Just send the most standard measures... not as important as means and SD --- or just show a histogram! Visual is good.
> -r
>
> On Wed, May 4, 2011 at 12:50 PM, Rick Reeves <reeves@nceas.ucsb.edu> wrote:
>
> Hi Rob:
>
> I have already computed the standard summary statistics for these images, can send them now.
>
> Could you (or Jim) be a bit more specific re: which measures of skew/kurtosos you prefer to see?
>
> Thanks,
> Rick
>
>
> On 5/4/2011 11:03 AM, Robert Guralnick wrote:
>
>
> Hey Rick and Jim --- I have gone through the Redmine conversations and generally am happy with progress. I basically follow all the steps you have done, but I would also like a VERY SIMPLE, succinct summary of the amount of error that you are finding between ASTER and SRTM and the fused ASTER/SRTM and CDEM --- I found it hard to pull out those numbers from the outputs provided ... maybe just means, SDs, skew, kurtosis? Or some way to get a handle on the distributions for those differentials between rasters? I might appreciate it, and I am guessing Brian and Walter might as well. Does this make sense to you?
> Best, Rob
>
>
>
Updated by Rick Reeves over 13 years ago
At Rob's request, I used R Raster package hist() function to generate histograms
for the images used in the image boundary analysis.
Here is the R code:
raster::hist(inputCgiarRaster.main="CGIAR/SRTM Image Component") raster::hist(inputCgiarRaster,main="CGIAR/SRTM Image Component") raster::hist(inputAsterRaster,main="ASTER Image Component") raster::hist(inputMosaicRaster,main="Image Mosaic Component") raster::hist(inputMosaicRaster,main="Image Mosaic - BOTH Components") raster::hist(inputCDEMRaster,main="Baseline CDEM Image") rDeltaWhole <- inputMosaicRaster - inputCDEMRaster raster::hist(rDeltaWhole,main="Difference Image: Mosaic - CDEM") writeRaster(rDeltaWhole,filename="DeltaMosaicCDEMSubmage.tif",format="GTiff",datatype="INT2S",overwrite=TRUE)
I extracted some screen shots of each histogram, which are stored in the attached .zip file.
This file also contains the statistics generated with gdalinfo__for the difference image;
they are also presented here:
reeves@eos:/data/project/organisms/rcr/ValidateBoundary$ gdalinfo -stats DeltaMosaicCDEMSubmage.tif
Driver: GTiff/GeoTIFF
Files: DeltaMosaicCDEMSubmage.tif
Size is 41582, 3738
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-135.988060499999989,61.579322388999998)
Pixel Size = (0.000833300000000,-0.000833300000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-135.9880605, 61.5793224) (135d59'17.02"W, 61d34'45.56"N)
Lower Left (-135.9880605, 58.4644470) (135d59'17.02"W, 58d27'52.01"N)
Upper Right (-101.3377799, 61.5793224) (101d20'16.01"W, 61d34'45.56"N)
Lower Right (-101.3377799, 58.4644470) (101d20'16.01"W, 58d27'52.01"N)
Center (-118.6629202, 60.0218847) (118d39'46.51"W, 60d 1'18.78"N)
Band 1 Block=41582x1 Type=Int16, ColorInterp=Gray
Min=-1603.000 Max=2770.000
Minimum=-1603.000, Maximum=2770.000, Mean=9.047, StdDev=132.886
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-1603
STATISTICS_MAXIMUM=2770
STATISTICS_MEAN=9.0470816079223
STATISTICS_STDDEV=132.8857558074
Updated by Rick Reeves over 13 years ago
- File AsterSRTMBoundaryMisalign.jpg AsterSRTMBoundaryMisalign.jpg added
- File AsterSRTMBoundaryZoom.jpg AsterSRTMBoundaryZoom.jpg added
Last week I noted that I had observed a very small (< one image pixel in X and Y for the 3 arcsecond images) misalignment between the pixels in the ASTER and SRTM image MOSAICS that I created to evaluate the merging of image types along the 60 Degree N Latitude boundary. I also noted a very small gap (< 1 pixel @ 3 arcsecond) between the top of the SRTM and bottom of the ASTER images. I have revisited this issue (with the perspective of an additional week working with these data), and I think I have an explanation for the issue and a solution. The misalignment in question is revealed when the components: mergeCgiarAsterBdyASTER_BL.tif mergeCgiarAsterBdySRTM_BL.tif ...are displayed at high magnification in a GIS (I use ArcMap GIS here). See the first attached screenshot: *AsterSRTMBoundaryMisalign.jpg* for he 'synoptic' view. The top horizontal line of the green 'grid' is the 60 degree N Latitude line; the vertical green lines are even 5 degree intervals. The misalignment is barely noticeable at this scale. Now see the second attached screenshot: *AsterSRTMBoundaryZoom.jpg* for the 'extreme mag' view The color map is selected to make the individual 3 arcsecond (~90 meter) pixels visible. Note the relative size of the pixels, the gap, and the image component misalignment in X and Y. Using screen measurements, I estimated the pixel misalignment to be approximately .0005 deg in X, and .0001 deg in Y. The gap appears to be approx .0004 deg. I believe the explanation is that when I created the three mosaics (ASTER, SRTM, and Merged), I did not specify an output image map extent. Instead, 'gdalwarp' generated an extent/origin by evaluating the extents of the incoming images. Notice that the generated extents do not lie exactly on an even boundary, but have a very slight offset: Lower Left (-140.0001389, 60.0002189) (140d 0'0.50"W, 60d 0'0.79"N): mergeCgiarAsterBdyASTER_BL.tif Lower Left (-150.0000000, 55.0002000) (150d 0'0.00"W, 55d 0'0.72"N): mergeCgiarAsterBdySRTM_BL.tif Lower Left (-150.0000000, 55.0004189) (150d 0'0.00"W, 55d 0'1.51"N): mergeCgiarAsterBdyFinalBL.tif My thought is that if I specify an 'even degree' extent using the gdalwarp -te parameter in each batch job that it will resolve this very small misalignment issue. This will not take much time to confirm.
Updated by Rick Reeves over 13 years ago
I have created and checked in a readme file which documents the steps I used to
generate one of the first image mosaics that I am using for exploring the DEM
data set components.
Here is the file: ReadmeCreateBoundaryMosaic.txt, located along with the scripts
that it documents at: /data/project/organisms/rcr/AsterCgiarMerge/.
Note that the specific parameters passed to 'gdalwarp' will probably change
as I adapt the process to the lessons learned working with these data sets
and creating the first image mosaics. But these scripts and this ReadMe file
demonstrate the image mosaic creation process.
May 5, 2011 / Rick Reeves
ReadMe file: Shell scripts to create the mosaic (3 arcsecond resolution)
of CGIAR/SRTM and ASTER GDEM imagery spanning the 60 Degree North Latitude
boundary within western Canada
Three shell scripts used:
1) "mosaicCgiarSrtmBdySRTM.sh" : creates mosaic from CGIAR/SRTM 5 degree tiles
(3 arcsecond resolution) converted to .tif format from original ArcMap ASCII
format.
2) "mosaicCgiarSrtmMBdyAster.sh" : creates mosaic from ASTER GDEM 1 degree tiles
(1 arcsecond resolution) converted to .tif format from original ArcMap ASCII
format. Note: incoming image tiles are resampled to 3 arcsecond resolution
to create outgoing image mosaic,
3) "mosaicSrtmAsterPartsBdy.sh" : creates mosaic from the outputs from scripts
1) and 2) (3 arcsecond resolution):
mergeCgiarAsterBdySRTM_BL.tif
mergeCgiarAsterBdyASTER_BL.tif
To create the ASTER / CGIAR boundary mosaic:
1) make current directory '/data/project/organisms/rcr/AsterCgiarMerge/'
2) run the script "mosaicCgiarSrtmBdySRTM.sh"
3) run the script "mosaicSrtmMBdyAster.sh"
4) run the script "mosaicSrtmAsterPartsBdy.sh"
Note: These initial scripts do NOT specify a new image extent for any of the mosaic components.
(gdalwarp parameter '-te xmin ymin xmax ymax' used to specify the extent)
So for these initial *test* runs, the output image extent is derived from the input files.
In 'production' runs, the extent of output mosaic tile compoents will be specified using -te;
we may also set other 'gdalwarp' parameters to properly establish other output image parameters.
Updated by Jim Regetz over 13 years ago
Two points regarding DEM raster extents and alignments:
- At least for SRTM, the pixel centers are lined up along degree boundaries. This means that the pixel borders (and thus pixel corner coordinates) are offset from the exact degree boundaries by 1/2 cell resolution. This is not a problem per se, but I mention it as a partial explanation of something to be aware of. I didn't check whether this is the case for ASTER because I wasn't sure exactly which ASTER files you are using as inputs.
- When you create an output raster by specifying the x and y cell resolution, I imagine it could cause problems because of rounding error: there is no way to specify 3/3600 (for the 90m) or 30/3600 (for the 1km) exactly in decimal form. Thus, I think it will make more sense to specify the number of pixels in the x and y direction. Specifically, this would mean using the
-tsflag rather than the-trflag in the gdal commands. Of course, you'll first have to calculate the correct number of pixels for a given output extent, but this should be easy enough.
Updated by Rick Reeves over 13 years ago
### Contents of post resolving the 'image offset' error - Adding distinct and accurate output image extents to the 'gdalwarp' commands almost compeletely resolved. HowTo: I used gdalinfo to get the extents for the lower-left and upper-right image tiles from the input tile list for the SRTM and ASTER component mosaics. I used these extent coordinates to set the '-te' parameters in the 'gdalwarp' commands used to create these two mosaics: for- mergeCgiarAsterBdySRTM_BLEven.tif - gdalwarp -of GTiff -ot Int16 -te -140. 55.0 -100.0 60.0 -tr .0008333 .0008333 -r bilinear -srcnodata -9999 -dstnodata -9999 \ for- mergeCgiarAsterBdyASTER_BLEven.tif gdalwarp -of GTiff -ot Int16 -te -140.0 60.0 -100.0 62.0 -tr .0008333 .0008333 -r bilinear -srcnodata -9999 -dstnodata -9999 \ These resulted in mosaic component images with the following coordinates: mergeCgiarAsterBdySRTM_BLEven.tif: Corner Coordinates: Upper Left (-140.0000000, 60.0000000) (140d 0'0.00"W, 60d 0'0.00"N) Lower Left (-140.0000000, 55.0002000) (140d 0'0.00"W, 55d 0'0.72"N) Upper Right ( -99.9999334, 60.0000000) ( 99d59'59.76"W, 60d 0'0.00"N) Lower Right ( -99.9999334, 55.0002000) ( 99d59'59.76"W, 55d 0'0.72"N) Center (-119.9999667, 57.5001000) (119d59'59.88"W, 57d30'0.36"N) mergeCgiarAsterBdyASTER_BLEven.tif: Corner Coordinates: Upper Left (-140.0000000, 62.0000000) (140d 0'0.00"W, 62d 0'0.00"N) Lower Left (-140.0000000, 60.0000800) (140d 0'0.00"W, 60d 0'0.29"N) Upper Right ( -99.9999334, 62.0000000) ( 99d59'59.76"W, 62d 0'0.00"N) Lower Right ( -99.9999334, 60.0000800) ( 99d59'59.76"W, 60d 0'0.29"N) Center (-119.9999667, 61.0000400) (119d59'59.88"W, 61d 0'0.14"N) As this ArcMap GIS screenshot: AsterSRTMCompBoundaryEven.jpg shows, the component images are now aligned in X, although, in Y, a small gap still exists between the two images: ASTER component lower left: Lower Left (-140.0000000, 60.0000800) # an .00008 gap SRTM component upper left : Upper Left (-140.0000000, 60.0000000) This gdalwarp command used to create the final mosaic from these two components: mergeCgiarAsterBdyFinal_Fix.tif: gdalwarp -of GTiff -ot Int16 -te -140. 55.0 -100.0 62.0 -tr .0008333 .0008333 -r bilinear -dstnodata -9999 \ Final image has this extent: Corner Coordinates: Upper Left (-140.0000000, 62.0000000) (140d 0'0.00"W, 62d 0'0.00"N) Lower Left (-140.0000000, 55.0002800) (140d 0'0.00"W, 55d 0'1.01"N) Upper Right ( -99.9999334, 62.0000000) ( 99d59'59.76"W, 62d 0'0.00"N) Lower Right ( -99.9999334, 55.0002800) ( 99d59'59.76"W, 55d 0'1.01"N) Center (-119.9999667, 58.5001400) (119d59'59.88"W, 58d30'0.50"N) The very small gap disappears when the input images are resampled into this extent...
Updated by Rick Reeves over 13 years ago
Jim, I agree. You can in fact see the 1/2 pixel 'offset' in the screenshots that I captured
of the magnified ArcMap GIS screen.
So I will try the -ts flag on the boundary validation test case.
Updated by Rick Reeves over 13 years ago
Workflow description: Performing the Aster/SRTM mosiac 'boundary analysis'
Rick Reeves / May 9, 2011
Objective of this analysis: explore and compare the statistical attributes of the
two Digital Elevation Model products - ASTER GDEM (north) and CGIAR SRTM (south)
with the designated 'baseline' DEM product: Canada DEM (CDEM) (spans border) -
In proximity to the 60 degree North Latitude line, where ASTER and SRTM data sets
are merged to form the Fused Digital Terrain Layer product.
Steps, including scripts used at each stage:
1) Create three image mosaics adjacent to and spanning 60 degree North Latitude region in Western Canada:
CGIAR / SRTM (north of boundary)
ASTER GDEM (south of boundary)
CDEM (spanning boundary)
CGIAR / ASTER mosaic (spanning boundary)
Shell scripts used to create these image mosaics
(in folder "/data/project/organisms/rcr/AsterCgiarMerge")
scripts typically executed in the following order -
1) mosaicCgiarSrtmBdySRTM_Fix.sh - creates SRTM image mosaic
2) mosaicCgiarSrtmBdyAster_Fix.sh - creates Aster image mosaic
3) mosaicSrtmAsterPartsBdyFix.sh - creates merged SRTM/Aster image mosaic
Creates baseline CDEM image for computing 'difference images'
4) mosaicCgiarAsterBdyCDEM_Fix.sh
Note: Input and output .tif image filenames are shown in the respective .sh files...
2) Displayed all of the output files produced by these scripts using ArcMapGIS interactive viewing tools
- Confirm that the boundary area subimages are correctly located, have correct # rows, etc.
3) Developed R scripts to explore the DEM data sources adjacent to 60 degree North latitude boundary
(in folder "/data/project/organisms/rcr/ValidateBoundary")
Scripts typically executed in the following order...
extractDemSubimages.r - Extract subimages from DEM data sources for Boundary Analysis
SampleDemDiffCols.r - Extracts and generates statistics for randomly selected image pixel pairs
makeImagePairPlotTable.r - Randomly samples image columns to generate data for histograms and plots
Note: Input and output image file nasmes are shown in the respective R scripts.
4) Use R and GDAL to generate plots and image statistics for 'images of interest' along boundary area.
Post results to Redmine, Iplant Wiki, etc.
Updated by Rick Reeves over 13 years ago
5/10/2011
Rick R -
To (finally) completely respond to Rob G's request:
(Work was delayed because of unexpected problems with R Raster sub image extraction ('different origin' error)
I had to use gdalwarp to extract the subimages, then read them in to R to create 'delta' images
Added R script to above workflow: 'OnlyImageDifferences.r'
Image statistics for Aster, SRTM, CDEM, and 'Difference Images' (Aster-CDEM and SRTM-CDEM)
for 60 Deg North Latitudeboundary region
First, for comparison, the BIG images -
**************************** mergeCgiarAsterBdyASTER_BLEven.tif ASTER image has higher average value
Minimum=-106.000, Maximum=6384.000, Mean=665.178, StdDev=499.372
Overviews: 12001x600, 6001x300, 3001x150, 1501x75, 751x38
Metadata:
STATISTICS_MINIMUM=-106
STATISTICS_MAXIMUM=6384
STATISTICS_MEAN=665.17825516819
STATISTICS_MEDIAN=9.9582889458621e-113
STATISTICS_MODE=1.7802139099189e-306
STATISTICS_STDDEV=499.37215934449
LAYER_TYPE=athematic
**************************** mergeCgiarAsterBdySRTM_BLEven.tif SRTM image has the smallest average value
Min=-220.000 Max=4326.000
Minimum=-220.000, Maximum=4326.000, Mean=620.707, StdDev=467.307
NoData Value=-9999
Overviews: 12001x1500, 6001x750, 3001x375, 1501x188, 751x94, 376x47
Metadata:
STATISTICS_MINIMUM=-220
STATISTICS_MAXIMUM=4326
STATISTICS_MEAN=620.70715018819
STATISTICS_MEDIAN=4.1631777282017e-199
STATISTICS_MODE=7.5659641201825e-307
STATISTICS_STDDEV=467.30736407449
LAYER_TYPE=athematic
***************************** CDEMMosCgiarAsterBdy_BLEven.tif: CDEM image
Minimum=0.000, Maximum=2661.000, Mean=636.938, StdDev=423.485
NoData Value=-9999
Overviews: 9001x900, 4501x450, 2251x225, 1126x113, 563x57
Metadata:
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2661
STATISTICS_MEAN=636.93846943295
STATISTICS_STDDEV=423.48490257975
Next, the data subimages surrounding the 60 degree North Latitude Boundary
SRTM (SOUTH of boundary) Images:
****** Files: EdgeSRTMOnly24row.tif Along the border, the SRTM image has a higher mean value than ASTER
Size is 42002, 24gdalinfo -sta
Coordinate System is:
Minimum=0.000, Maximum=2039.000, Mean=587.567, StdDev=321.291
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=2039
STATISTICS_MEAN=587.56716743647
STATISTICS_STDDEV=321.29063053814
STATISTICS_MEDIAN=1.1605833708736e-198
STATISTICS_MODE=2.2252055940782e-306
LAYER_TYPE=athematic
********************* Files: EdgeSrtmCDEM24row.tif CEDM image mean higher than SRTM image mean
Size is 2002, 24
Minimum=168.000, Maximum=2026.000, Mean=609.157, StdDev=327.189
NoData Value=-9999
Metadata:
STATISTICS_MINIMUM=168
STATISTICS_MAXIMUM=2026
STATISTICS_MEAN=609.15666462969
STATISTICS_STDDEV=327.18919674128
************* Files: EdgeRegionSRTMOnlyDelta24row.tif SRTM / CDEM Difference image
Size is 40636, 24
Minimum=-1639.000, Maximum=180.000, Mean=4.816, StdDev=21.494
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-1639
STATISTICS_MAXIMUM=180
STATISTICS_MEAN=4.8156474820144
STATISTICS_STDDEV=21.494248970983
ASTER (NORTH of Border) Image
****** Files: EdgeAsterOnly24row.tif
Size is 42002, 24
Minimum=119.000, Maximum=2078.000, Mean=571.326, StdDev=311.359
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=119
STATISTICS_MAXIMUM=2078
STATISTICS_MEAN=571.32635053093
STATISTICS_STDDEV=311.35914650438
STATISTICS_MEDIAN=1.1610171431127e-198
STATISTICS_MODE=9.3460979006185e-307
LAYER_TYPE=athematic
************************ Files: EdgeAsterCDEM24row.tif CDEM image mean higher than ASTER image mean
Size is 42002, 24
Minimum=166.000, Maximum=1988.000, Mean=605.707, StdDev=323.313
NoData Value=-9999
Metadata:
STATISTICS_MINIMUM=166
STATISTICS_MAXIMUM=1988
STATISTICS_MEAN=605.70689703064
STATISTICS_STDDEV=323.3128648326
********************* Files: EdgeRegionAsterOnlyDelta24row.tif
Size is 40636, 24
Minimum=-307.000, Maximum=235.000, Mean=-9.008, StdDev=29.569
NoData Value=-32768
Metadata:
STATISTICS_MINIMUM=-307
STATISTICS_MAXIMUM=235
STATISTICS_MEAN=-9.0081548660685
STATISTICS_STDDEV=29.568978110945
Updated by Rick Reeves over 13 years ago
- Due date changed from 04/21/2011 to 06/15/2011
- % Done changed from 90 to 70
- Estimated time changed from 34.00 h to 48.00 h
Incorporating the suggestions made by Rob Guralnick, Walter Jetz, and Brian McGill during our June 10, 2011 weekly GIS Team conference call, here is my plan for validating and comparing the performance of the four alternative ASTER/SRTM boundary edge mitigation methods
Study Area for validation calculations: covered by 6000 column, 2400 row 3 arc-second cell size image located in Western Canada, Eastern border @ 115 West Longitude, spanning and centered on 60 degree North Latitude line.
There are eight different DEM image 'populations' within the study area to consider:
1) Baseline Canada DEM (CDEM) image
2) Separate ASTER and SRTM images; SRTM covering Southern 1200 rows, ASTER image covering entire (2400 row) study area
3) ASTER / SRTM image mosaic, after assembly, and BEFORE mitigation methods applied
4-8) Five image mosaics each with a different mitigation 'treatment' applied:
a) Column-wise add: SRTM / ASTER 'edge delta' to ASTER (North) mosaic component with
linear decay function
b) Column-wise add: SRTM / ASTER 'edge delta' to ASTER (North) mosaic component:
exponential decay function
c) Row-wise ASTER/SRTM 'image blending' of ASTER/SRTM (South) side: linear decay function
d) Row-wise ASTER/SRTM 'image blending' of ASTER/SRTM (South) side: exponential decay
function
e) Spline Interpolation technique, applied to neighborhood spanning 60 degree border (various
neighborhood dimensions, TBA)
The plan is:
1) Assemble / compute images 3) through 8)
2) Exploration I: Compute 'difference images' for image 'population' members 2) through 8) by subtracting them from the Baseline CDEM image.
3) Exploration II: Compute RMS Error and Cross-correlation images for image 'population' members 2) through 8) using each component vis-a-vis the Baseline CDEM image.
4) Evaluation: Review RMSE and Cross-correlation metrics for each image type, rank the candidate methods. At this point, eliminate one or more of the candidate methods from consideration.
5) Optimization: Fine-tune the 2-3 most promising methods by adjusting the parameters used to compute the components (e.g., by tuning the parameters of the exponential decay function).
I expect to complete plan steps 1) through 4) by Tuesday(June 14), so that these results can be sent to the project PIs by the end of Wednesday (June 15) .
Updated by Rick Reeves over 13 years ago
To compare performance of the candidate solution techniques, I will compute three derived terrain metrics -
Slope, Aspect, and Flow Direction - using the baseline CDEM layer. Then, I will compute them using the other seven mosaic components. Then, I will generate difference images, truth tables, RMSE, and Cross-correlation metrics, using the baseline derived components as 'truth', and the mosaic components as 'estimates' or 'simulated' values.
I will use R and MATLAB (if necessary) to generate the metrics.
Updated by Jim Regetz almost 13 years ago
- Status changed from In Progress to Closed
- Assignee deleted (
Rick Reeves)
The original focus of this task was on using independent DEM data, suggesting CDED, to validate the fused DEM at the 60N boundary. Unfortunately, the signal-to-noise ratio in the subsequent task notes is so low as to obfuscate any useful nuggets of information contained above.
For a more useful and authoritative description of work actually done towards 60N boundary assessment as of summer 2011, see the ASTER-SRTM DEM fusion analysis documents. Specifically regarding CDED validation, note that these assessments revealed 60N artifacts within CDED itself, rendering it fairly useless as a validation data source. In fall 2011, Yuni Nunokawa repeated many of these assessments using ASTER GDEM v2, but dispensed with comparison to CDED and instead focused only on examining internal consistency of SRTM, ASTER GDEM2, and fusion thereof. To date, no concerted effort has been made to obtain an alternative 90m DEM to use for independent validation.
Although we may ultimately want to do more independent validation of the fused DEM product before all is said and done, I'm now closing this monstrosity of a ticket, and we can start fresh with a new one if necessary in the future.