updated MOD35 processing script to work in separate temporary directories and transfer daily cloud data to lou for archiving in preparation for global processing (which cannot be done in personal directory). Also set up MOD35_Climatology script to be submitted to LDAN queue to run on lou rather than Pleiades. However environment still not set up correctly and script fails when reading netcdf files. Andrew is working on it.
Fixed the 'missing stripes' problem in the climatology script. Source of problem was related to the _FillValue attribute being incorrectly set to a short integer when the data was saved as byte. Changing both the attribute and the data to type 'byte' appears to have fixed the problem.
Discovered possible cause of weird missing data stripes due to identifying missing data as 225.0 rather than 255s
Explored 30-day moving average
Merging with Pleiades copy
minor clean up to MOD35 Climatology
Removed some unsed code from MOD35_climatology. Still having problem of missing data stripes in February means.
Updated MOD35_Climatology to use percentages instead of classes, but there are too many zeros in result
Simplified MOD35 script and converted output to single band p(Clear) rather than multiple bands
Merge branch 'aw/precip' of code.nceas.ucsb.edu:environmental-layers into aw/precip
Added initial validation via NDP-026D dataset
Simplifications to Climatology Script
Added new script to process daily MOD35 files
Updates to MOD06 Climatology Script and MDO06 processing to isolate the effects of the MOD35 -landcover bias
Added script to process NDP-026D station cloud climatologies
Adjusted handling of cloud flag to 'hopefully' improve missing data problem in MOD06 climatology
restructured MOD06 processing routine to eliminate functions for more transparent (linear) processing. Also updated to run on either pleiades or litoria
Adding intial code to process the NDP-026D station cloud climatology dataset
Updated submit script to process multiple tiles and days in the same batch. Also separated the daily processing from the climatologies
Merging with Pleiades copies
Adding Climatology script to process daily files to climatologies
Identified problem in handling of missing values of MOD06 product. In some areas limiting quality control to the highest quality results in ~90% NAs.
initial script to handle modified netcdf files from Pleiades.R script
Merging with remote branch"
Updated summary script to work with netcdf output files
MOD06 processing running successfully using Pleiades environment set up by Andrew Michaelis at NASA.
Script now runs except ncap2 is missing from the nco module
Currently submission hangs using mqueue when loading rgdal library. Will attempt to add a separate submission script that first loads modules before running Rscript on the slaves
MOD06_L2 now running properly but Rmpi is not running due to (probably) incompatability between Rmpi and the version of sgi-mpi available on Pleiades
Updated script to accurately set paths for HEG tool to facilitate parallel processing of gridding procedure using swtif. Multicore package is used to parallelize which limits processing to a single node. Next step is to explore use of foreach package to allow multiple nodes
Updated MOD06 process to output NetCDF files instead of geotif for faster post-processing
Resolved problem with temporary directory for Rscript call.
Script is successfully running and producing the summary files, though the output looks strange. Maybe a problem with sinusoidal output of HEG? Use Pleiades.R to drive the submission and MOD06_L2_process as the processing script
Successfully running MOD06 processing on Pleiades as an array job (though submissions are limited to < 365 jobs so will have to find another way to submit them)
Started working with script to process MOD05 on NASA's Pleiades
Updates to run 12 months of mean daily precipitation using a variety of models. Includes using gamm() to incorporate spatial effects in addition to the smooth parameters. Not all clean and cozy yet, uploading as a work-in-progress...
Further updates to interpolation procedures including:
1) bugfix in MOD06_L2_data_compile. Root access is needed to run swtif in its current directory (bad design!).2) Switched to UTM output from swtif (from sinusoidal) becuase it looked like gdal was struggling with integerized sinusoidal...
Added initial version of LST_Landcover exploration (not finished). Also moved prior version of interpolation procedure (bayesian krig) into repository.
added processing for 'very cloudy days'
Added PRISM comparison to MOD06_summary and added script to calculate 'back-of-the-envelope' estimates of total daily climate layer sizes
Updated MOD06 compile script to process the tiles in parallel using GRASS. This brings processing time for 10 year archive for oregon from 48 hours to ~2 hours on 24 cores. Also added several exploratory analysis to the data_summary script.
Added cloud flag to outputs
First 10-year processing of daily mean tifs
MOD06 processing runs well in sequence, but not parallel
Adding some MOD06 (cloud product) processing routines. Still a work in progress
Initial commit
cleaned up file permissions
added in various previously uncommitted scripts
