Updating validation to use new cloud data
Finished second complete ee download of mod09 cloud mask using python tiler
Separated ee download from compile script. Switched to use of vrt as intermediate file form
Added multiple tries to python GEE script
Updated EE compiler
Merge branch 'aw/precip' of code.nceas.ucsb.edu:environmental-layers into aw/precip
Updated EE script to run in parallel and minor edits to post-processing
Updating Pleiades script
udpating validation to full globe
updates to MOD09 visualizations
Updating visualization and validation of MOD09 cloud product
addied initial MOD09 visualization script and updates to NDP script to use the new mod09 data
First global mod09 run. Updated cloud flag function
updated EE script to loop over regions (currently global quarters) and check if uncorrupted files exist before downloading
cleaning up comments
Adding ee.MOD09.py script to process and download data from Google Earth Engine
merging
Old code
Changed bias metric to % change rather than p-value
Updated cloud climatology evaluation
Exploring evalution options for mod35c5
Added routine to process identify regions of concern in modis cloud data
nearly finished first global run of MOD35 cloud frequency
Removing random shapefile
Updated C5 MOD35 Processing Path code and C5MOD35_Evaluation in preparation for publication
Added script to extract each separate cloud test from MOD35. This is done to explore which tests are leading to coastal artefacts in the summarized data
Submitted global run for 2009
found and fixed bug in r.out.gdal export of netcdf files with no missing data. The missing data value was set to 0 instead of the 255 that was specified. This resulted in all 0s to be removed from these day-tiles. Fixed by deleting and resetting the attribute _FillValue using ncatted.
Updated to use new swtif and remove additional QC that identified strips introduced by old swtif. Some tiling artifacts still present.
updated to use new swtif
Submitted MOD35-landcover bias paper with code from this commit. Also added short script to test swtif program.
Adding initial version of MOD09 processing script
updating figures
Adding script to upload covariate data to Pleiades
Updating figures for C5MOD35 Manuscript
updated swtif error checking to use r.neighbor instead of r.mapcalc. Submitted job for northern block of tiles
Finished function to make raster extent for any modis tile
Update MOD35 process path extraction to use sensor angle
Add first version of script to develop global modis MODLAND grid
added function to flag bad pixels introduced by bug in HEG software
Added script to test functionality of HEG tool which reveals that v2.12 segfaults when multiple bands are selected for processing. Also updated MOD35_ExtractProcessPath to process one-band-at-a-time instead of in batches.
Adding sensor zenith to quality calculation and improving the filter used to select daily data
Started process of adding sensor zenith filter to MOD35 Processing
Separated daily products into a 'day' and 'night' cloudiness based on day flag
Updated profile plots of transect figure
Added script to make GHCN station shapefile for global station gap analysis
Move file outputs to /nobackupp1 instead of lou due to quota increase. First full 2009 summary
Updated script to process MOD35 Processing Path using HEG to interpolate sensor zenith observations. Also working on figures for MOD35 C5 vs C6 comparison in MOD35C5_Evaluation script
Updated swath-grid section of MOD35 processing to add buffer and (hopefully) reduce edge artifacts in tiles
updated MOD35 processing script to work in separate temporary directories and transfer daily cloud data to lou for archiving in preparation for global processing (which cannot be done in personal directory). Also set up MOD35_Climatology script to be submitted to LDAN queue to run on lou rather than Pleiades. However environment still not set up correctly and script fails when reading netcdf files. Andrew is working on it.
Added script to extract MOD35 processing path from swath level data and another to summarize MOD35 data. Also updated NDP-026D script to use new MOD35/09 summaries from Earth Engine
Added initial code to make KML of cloud climatologies
Fixed the 'missing stripes' problem in the climatology script. Source of problem was related to the _FillValue attribute being incorrectly set to a short integer when the data was saved as byte. Changing both the attribute and the data to type 'byte' appears to have fixed the problem.
Discovered possible cause of weird missing data stripes due to identifying missing data as 225.0 rather than 255s
Explored 30-day moving average
Merging with Pleiades copy
minor clean up to MOD35 Climatology
Removed some unsed code from MOD35_climatology. Still having problem of missing data stripes in February means.
Updated MOD35_Climatology to use percentages instead of classes, but there are too many zeros in result
Simplified MOD35 script and converted output to single band p(Clear) rather than multiple bands
Added initial validation via NDP-026D dataset
Simplifications to Climatology Script
Added new script to process daily MOD35 files
Updates to MOD06 Climatology Script and MDO06 processing to isolate the effects of the MOD35 -landcover bias
Added script to process NDP-026D station cloud climatologies
Adjusted handling of cloud flag to 'hopefully' improve missing data problem in MOD06 climatology
restructured MOD06 processing routine to eliminate functions for more transparent (linear) processing. Also updated to run on either pleiades or litoria
Adding intial code to process the NDP-026D station cloud climatology dataset
Updated submit script to process multiple tiles and days in the same batch. Also separated the daily processing from the climatologies
Merging with Pleiades copies
Adding Climatology script to process daily files to climatologies
Identified problem in handling of missing values of MOD06 product. In some areas limiting quality control to the highest quality results in ~90% NAs.
initial script to handle modified netcdf files from Pleiades.R script
Merging with remote branch"
Updated summary script to work with netcdf output files
MOD06 processing running successfully using Pleiades environment set up by Andrew Michaelis at NASA.
