Project

General

Profile

Activity

From 05/08/2012 to 06/06/2012

06/06/2012

01:54 PM Revision 44799b5f: Added PRISM comparison to MOD06_summary and added script to calculate 'back-of-the-envelope' estimates of total daily climate layer sizes
Adam M. Wilson
09:35 AM Document: Tasking for climate change
Brian McGill
09:35 AM Climate_Tasking_May1_notes_final.docx
Brian McGill

06/05/2012

11:23 AM Task #363: Assemble all GHCN data into a single database
I did a 30 year query during the night for the 1980-2010 period for the Oregon study area. I just checked the results... Benoit Parmentier
12:35 AM Task #363: Assemble all GHCN data into a single database
A short R script was added to select stations data from the postgres database created by Jim. The user must provide a... Benoit Parmentier
12:27 AM Task #364: Integrate spatial variables and structure in the GAM methodology
Two stage regression: the GAM+Kriging script was updated to integrate specific and general diagnostics:
general diag...
Benoit Parmentier
12:06 AM Revision 95d521f9: GHNCD station data selection using Postgres database, initial commit, task #363
Benoit Parmentier

06/04/2012

11:57 PM Revision 328689cf: GAM LST, added specific and general diagnostic measures (e.g.MAE) GAM+Kriging, task #364 and #406
Benoit Parmentier
01:56 PM Revision c33d3b68: Updated MOD06 compile script to process the tiles in parallel using GRASS. This brings processing time for 10 year archive for oregon from 48 hours to ~2 hours on 24 cores. Also added several exploratory analysis to the data_summary script.
Adam M. Wilson
07:56 AM Revision 0c74b1da: Added cloud flag to outputs
Adam M. Wilson

06/01/2012

12:54 PM Task #423 (New): Develop stratified station sampling procedure
We need to develop a procedure to sample representative stations to use in fitting and validation. In many areas (in... Adam Wilson
12:43 PM Task #415: Process MOD06_L2 Cloud data
I've downloaded and processed the full 10 archive for Oregon and generated monthly climatologies for "Cloud Optical T... Adam Wilson
09:57 AM Task #363: Assemble all GHCN data into a single database
Finished loading the new TMAX, TMIN, and PRCP data (just under 12 hours) and creating the station index (just over 13... Jim Regetz

05/31/2012

06:56 AM Revision 08d8290d: First 10-year processing of daily mean tifs
Adam M. Wilson
06:55 AM Revision e23e9664: First full processing of daily mean tifs
Adam M. Wilson

05/30/2012

04:30 PM Revision 38eccf20: GAM LST, adding lines to summarize samling RMSE using plots, task #409
Benoit Parmentier
04:19 PM Revision 51050314: GAM LST, first code to test effect of sampling on GAM, task #409
Benoit Parmentier
04:10 PM Revision 8d91ca4a: GAM LST, added model and modified summary plots, run on 365 dates GAM+Kriging, task #364 and #406
Benoit Parmentier
04:03 PM Revision ab8957ca: GAM LST, modify code to generalize number of models and assessment,GAM+Kriging task #364
Benoit Parmentier
03:52 PM Revision c9e2af49: GAM LST, major modifications to create GAM+Kriging code, task #364
Benoit Parmentier
03:38 PM Task #363: Assemble all GHCN data into a single database
I wrote and ran a little bash script to download the latest GHCN data and generate a basic log file for future refere... Jim Regetz
03:34 PM Revision 75151566: GAM LST, adding lines to assess RMSE per month over 365 dates, task #406
Benoit Parmentier
02:46 PM Revision f042de0f: GAM LST, added model with forest only term -364 dates assessment, task #406
Benoit Parmentier
02:37 PM Revision 9c848d7a: GAM LST, added section to save training and test residuals in shapefiles, task #361
Benoit Parmentier
02:28 PM Revision f31f59cc: GAM LST, changed triple nesting lat,long,Elev to double, 365 dates run, task #406 and #361
Benoit Parmentier
12:02 PM Revision fb35e17b: added bash script used to download new GHCN data (29-May-2012)
Jim Regetz
11:19 AM Revision 63d8201d: GAM LST, used monthly LST average instead of daily, task #361
Benoit Parmentier
11:11 AM Revision 80402fdf: GAM LST, adding new model with grass land cover, task #361
Benoit Parmentier
10:56 AM Revision 24eedf3a: GAM using LST as input variable task #361
Benoit Parmentier
10:27 AM Revision 90913004: augmented test script to reproduce 8-day LST *counts* from daily LST
Jim Regetz
10:01 AM Revision e84c3d48: tucked misc procedural code into a main function
Jim Regetz

05/29/2012

04:49 PM Revision fbc95c44: GAM, modification of nesting of models and clean up, task #361
Benoit Parmentier
04:16 PM Revision 328528e2: Kriging, adding LST for co-kriging: task #364
Benoit Parmentier
03:53 PM Revision eec8b7e0: Kriging, slight modification and clean up, task #364
Benoit Parmentier
03:37 PM Revision 88248d6c: Kriging, modification to save kriged surfaces in geotiff and png files, task #364
Benoit Parmentier
03:24 PM Revision 3be0f72b: Kriging, modifying code to save training and testing samples in shapefiles: task #364
Benoit Parmentier
03:07 PM Revision 7da7872a: Kriging for OR, this is the first script, task #364
Benoit Parmentier
12:56 PM Revision 15a39f25: GWR modified code to save plots and images, task #364
Benoit Parmentier
12:49 PM Revision defe53e5: GWR assessment by visualizing residuals using kriging
Benoit Parmentier
12:16 PM Revision 5f312940: GWR in adding loop for 10 dates
Benoit Parmentier
11:57 AM Revision 23ed3053: GWR first script for one date in OR, task #364
Benoit Parmentier

05/25/2012

03:21 PM Task #363: Assemble all GHCN data into a single database
The daily data files currently in the database were obtained back in Oct 2010. Benoit and I discussed simply augmenti... Jim Regetz
09:05 AM Task #419 (In Progress): Assessment of results in the context of the literature: Literature review and accuracy
Documenting the new products will necessitate assembling a list of references on climate interpolation to place the r... Benoit Parmentier

05/24/2012

01:59 PM Revision 979e2d4c: MOD06 processing runs well in sequence, but not parallel
Adam M. Wilson

05/23/2012

01:55 PM Revision 2a6b57de: Reorganization: moved mindmap to it's own directory to isolate temporary files created when exporting to other formats
Adam M. Wilson
01:44 PM Revision 5891fba5: Add initial version of the 'mind map' for the interpolation project. This file contains an outline of the various stages of the project and links back to issues and code in the blessed repository.
Created during Benoit's visit to Yale May 21-22. Adam M. Wilson
01:43 PM Revision 147da66d: Adding some MOD06 (cloud product) processing routines. Still a work in progress
Adam M. Wilson
01:35 PM Revision ee6357dc: Initial commit
Adam Wilson

05/22/2012

10:14 AM Task #363: Assemble all GHCN data into a single database
If useful for some specific purpose, we could generate relevant synoptic views or subsetted table dumps pivoted back ... Jim Regetz
08:25 AM Task #363: Assemble all GHCN data into a single database
Perhaps it would make sense to keep the data in the "one-row-per-month" format in the database and transform it on th... Adam Wilson
07:52 AM Task #418 (New): Explore Landcover - LST interactions
Use the full gridded datasets (rather than just station locations) to explore relationships between LST and various (... Adam Wilson

05/17/2012

11:38 PM Task #416: Scope out workflow for calculating monthly LST climatologies
I've now written some Python+GRASS code to do a good bit of what I think we want. For a user-specified _tile_, _year_... Jim Regetz
03:05 PM Task #416 (In Progress): Scope out workflow for calculating monthly LST climatologies
Assess feasibility of running LST climatologies ourselves by implementing a basic scripted workflow and estimating ov... Jim Regetz
03:44 PM Task #375: Assemble monthly mean MODIS LST values for the complete record (2000-2012) for Oregon
See task #416 for progress on developing and evaluating a procedure for doing this ourselves.
But I will add a com...
Jim Regetz
02:16 PM Revision 01b3830e: added function documentation and removed two obsolete functions
Jim Regetz
02:04 PM Revision a05f49eb: automated determination of which days to aggregate for any given month
Jim Regetz
01:53 PM Revision 96a70efc: tweaked climatology calc function to return list of GRASS map names
Jim Regetz
01:43 PM Revision bd9bc508: switched from truncating to rounding mean LST back to integer
Jim Regetz
12:31 PM Revision 5fb982c8: wrote function to download HDF files for given tile/year/day-range
Jim Regetz
10:05 AM Revision b9e1a7f9: wrote function to look up local HDF files for given tile/year/day-range
Jim Regetz
09:12 AM Revision eaf6c10b: wrote function to encapsulate LST loading and QC-adjustment
Jim Regetz
08:45 AM Revision 47cdfebe: added initial Python+GRASS code to download and aggregate daily LST
Jim Regetz
08:45 AM Revision 934d6ab7: added script to reproduce calculation of 8-day LST from daily LST
Jim Regetz

05/16/2012

01:55 PM IPLANT_presentation_meeting_05152012_Benoit.pptx
Benoit Parmentier
01:55 PM Document: Update IPLANT meeting roundup 05152012
This document summarizes ongoing work for the IPLANT meeting on 05152012:
-GAM assessment over year 2010: average pe...
Benoit Parmentier
01:49 PM IPLANT_presentation_meeting_05012012_Benoit.pptx
Benoit Parmentier
01:49 PM Document: Update IPLANT meeting roundup 05012012
This document provides updates presented at the IPLANT meeting 05012012:
- Results from GAM models run for the year ...
Benoit Parmentier
01:41 PM IPLANT_working_roundup_Benoit_04172012.pdf
Benoit Parmentier
01:41 PM Document: Update IPLANT meeting roundup 04172012
This presentation presents mean monthly LST averages derived from daily MOD11A1. Results from Kriging and co-kriging ... Benoit Parmentier
12:01 PM Task #207: Produce global fused DEM layer
A global 90m product is now available, though the individual pieces have not been mosaiced together due to file size.... Natalie Robinson
06:40 AM Task #415: Process MOD06_L2 Cloud data
See presentation on current progress here [[https://projects.nceas.ucsb.edu/nceas/documents/18]] Adam Wilson
06:40 AM Task #415 (New): Process MOD06_L2 Cloud data
Download and process the MODIS cloud product (MOD06_L2) and produce monthly 1km summaries of key variables. Adam Wilson
06:37 AM 20120320_NCEAS_update.pdf
Adam Wilson
06:37 AM Document: Wilson update 5/15/2012
Overview of processing the MOD06 cloud product Adam Wilson

05/15/2012

11:22 AM Task #411 (In Progress): Insert DEM adaptive smoothing procedure into terrain workflow
I developed two alternative implementations of the adaptive DEM smoothing procedure (@multiscalesmooth9a_clean.aml@):... Jim Regetz
09:52 AM Task #411 (In Progress): Insert DEM adaptive smoothing procedure into terrain workflow
The original Oregon terrain processing workflow, implemented as a set of AML scripts, included the application of Joh... Jim Regetz
09:57 AM Task #208: Evaluate existing E&O AML scripts
I recently reorganized all AML scripts into a single directory in the code repo, here:
source:terrain/research/orego...
Jim Regetz

05/14/2012

10:00 AM Task #363: Assemble all GHCN data into a single database
Quick example comparing SQLite vs PostgreSQL timings on a simple aggregate query that doesn't use any indexes. SQLite... Jim Regetz

05/13/2012

11:16 PM Task #363: Assemble all GHCN data into a single database
Vaguely concerned about performance and manageability of such a large SQLite database, I wrote a second script to loa... Jim Regetz
11:14 PM Task #363: Assemble all GHCN data into a single database
Belated update. The TMIN/TMAX data loaded in ~6 hours total, yielding a 25GB SQLite file with nearly 600 million rows. Jim Regetz
10:54 PM Revision c7235eae: added R script to load GHCN data into a PostgreSQL DB
Jim Regetz
10:50 PM Revision a7663038: further optimized functions in R for some minor extra speedup
Jim Regetz

05/11/2012

10:03 AM Revision aac7c36c: Added 3 new models and nesting to GAM, task #361
Benoit Parmentier

05/10/2012

03:03 PM Revision 0d1cde0a: Changes in the diagnostics plots, barplots.
Benoit Parmentier
02:49 PM Revision 25a68ae3: Transform aspect in Eastness and Northness variables in the GAM.
Benoit Parmentier
02:42 PM Revision 4180b123: Transformed the aspect variable and added GAM model closer to PRISM
Benoit Parmentier
02:28 PM Revision 2a89fa5f: GAM prediction added to the loop with RMSE calculation
Benoit Parmentier
02:13 PM Revision ec229367: Streamlining and slight changes to the loop to create subset dataset for different dates
Benoit Parmentier
02:07 PM Revision af42291a: Additional changes to the loop and format
Benoit Parmentier
01:47 PM Revision bc59c6bb: Start of Task #361, modified code to subset for 10 dates
Benoit Parmentier
11:59 AM Revision d7b0ef36: Minor format changes and clean up to code
Benoit Parmentier
11:18 AM Revision e2d673e4: Corrected errors in RMSE calculation and added plots to view results
Benoit Parmentier
08:49 AM Task #363: Assemble all GHCN data into a single database
Data loading slowed to a crawl overnight, thanks to a record count query tucked inside in the bulk insert function th... Jim Regetz
08:34 AM Revision 3b8dd010: dropped unnecessary record-count query from bulk insert function
Jim Regetz

05/09/2012

11:49 PM Task #363: Assemble all GHCN data into a single database
Committed an R script that orchestrates the processing and loading of all 75000+ *.dly files into a SQLite database. ... Jim Regetz
11:10 AM Task #363 (In Progress): Assemble all GHCN data into a single database
We decided to organize and load _all_ available GHCN data (i.e., all dates and stations, globally) into SQLite databa... Jim Regetz
11:06 PM Revision d4e63f5d: superficial code reformatting
Jim Regetz
10:29 PM Revision 38aa5c07: replaced reshape with faster manual split-rbind approach
Jim Regetz
04:11 PM Revision 669c150d: added grep pre-filtering of rows in the *.dly files
Jim Regetz
03:45 PM Revision 2d08ed05: replaced read.fortran with faster system call to awk/tr
Jim Regetz
03:41 PM Revision dd85f6d3: added initial R script to load GHCN data into a SQLite DB
Jim Regetz
03:40 PM Task #408: Capturing LST spatial structure using spatial eigenvectors
I started writing the R code using the two functions from spdep package: ME() and SpatialFiltering()
I am trying to ...
Benoit Parmentier
03:32 PM Task #364: Integrate spatial variables and structure in the GAM methodology
I am currently exploring a two stage regression involving:
1) Step 1: GAM models using Lat, long, ELEV_SRTM, DISTOC,...
Benoit Parmentier

05/08/2012

04:31 PM Task #406: OR-GAM predictions-model assessment over 365 dates, year 2010
RMSE were calculated per month over the year 2010. Results show that contrary to my expectation RMSE are higher in S... Benoit Parmentier
 

Also available in: Atom