Project

General

Profile

Parmentier updates

2013-12-15

What I did the past week
  • Continued the work for the multi-timescale paper, adding analyses on transects (d5829598, ae69b301), difference comparison and land cover (c1e644f6 , 3ca58e9d).
What I'm working on now
  • Adding figures and analyses to the draft as well as writing up some of the results.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Looking into the interpolation review.

2013-12-09

What I did the past week
  • generated gwr daily prediction for mod 4 to mod7 comb5 (c850f186) for multi time scale paper.
  • Continued analyses for multi timescale paper: monthly holdout accuracy figure (b2298f8e) and spatial lag Moran's I (70704c05 , 94e49513)
What I'm working on now
  • Analyses for paper on multi-time scale methods: analyzing results and summarizing for paper.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Looking into the interpolation review.

2013-11-29

What I did the past week
  • generated multiple monthly holdouts (0 to 70%) results for GAM_CAI (12b8c55d) , Kriging CAI (f6e04a80) and GWR_CAI (dae80cc5) for multi time scale paper.
  • Continued analyses for multi timescale paper (043c6c28 , e7295f5b)
What I'm working on now
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Looking into the interpolation review.
  • Run Kriging daily last four models to have all the results ready for the draft (multi-time scale)

2013-11-18

What I did the past week
  • ran Oregon 2010 temperature predictions using multiple monthly holdouts (0 to 70%) for GAM_fss (04ae5759) , Kriging_fss (04ae5759) and GWR_fss (aa674f21) for multi time scale paper.
  • Modified scripts for multi timescale paper (f8ba9fdc) to generate figures for draft (intersects, spatial patterns etc).
What I'm working on now
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Looking into the interpolation review.
  • Run other holdout analyses for CAI method.

2013-11-10

What I did the past week What I'm working on now
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.

2013-11-01

What I did the past week
  • experimented with correlogram for assessment of predictions (80363c49)
  • Modified scripts for single time scale draft (280bb8d2)
  • Submitted single time scale draft.
What I'm working on now
  • Analyses for paper on multi-time scale methods: examining results.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.

2013-10-22

What I did the past week
  • modified code to allow running ever n day over a time period (0a5206c6) , (5fc0c8b6).
  • ran analyses for multiple hold out for every 2 days for year 2010(183days) for kriging daily (0a5206c6), gam daily (5d05bb61) and gwr daily(2bcd5395).
  • Modified scripts and edited single time scale draft following feedback from E&O layer group (3b1a1539)
What I'm working on now
  • Analyses for paper on multi-time scale methods: examining results.
  • Editing single time scale paper following second round of feedback from E&O layer group.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Submit single-time scale paper and continue review of interpolation methods.

2013-10-14

What I did the past week
  • modified daily deviation options for multi-time scale methods (d5d86bbf) and ran one more analysis with DISTOC covariate (7fd742b4).
  • Edited draft following feedback and modified figures for paper on single time scale (145bb8a6)
  • Analyses multi-time scale papers, integrating daily deviation with covariates (2d03921e)
What I'm working on now
  • Analyses for paper on multi-time scale methods: examining results.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Complete edits for single-time scale paper and continue review of interpolation methods.

2013-10-07

What I did the past week
  • Added daily deviation options (multi-time scale methods) in addition to kriging without covariates. There are now three methods available (gam,kriging,gwr) with covariates: (30443fec, 7624aef9, b5ca4884)
  • Running test with gam models for daily deviation surfaces, multi-time scale methods (7624aef9)
  • Edited draft on assessment of methods following feedback from the E&O layer group.
What I'm working on now
  • Analyses for paper on multi-time scale methods: adding interpolation models to daily deviation surfaces.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Complete edits for single-time scale paper and continue review of interpolation methods.

2013-09-30

What I did the past week
  • Ran analyses for 0 to 70% monthly holdout proportions for gwr CAI, and gwr fus (bdd46177, bf9dd6e2, c3f80062)
  • Analyses for the second paper, examining results of monthly hold out proportions from 0 to 70% for all multi time scale methods (992b2eeb)
  • Check on quality flags for paper.
  • Edited draft on assessment of methods (single time scale methods 4823a232).
What I'm working on now
  • Analyses for paper on multi-time scale methods: ongoing.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Pushing on paper 2 (multi-times scale methods) - ongoing work at this stage.
  • Edit review paper on interpolation methods.

2013-09-23

What I did the past week
  • Ran analyses for 0 to 70% monthly holdout proportions for gwr CAI (26f78228, 2cf9c858, 60e1d33c)
  • Examining analyses for the second paper, monthly hold out proportions from 0 to 70% for multi time scale methods (567d1e54)
  • Check on quality flags for paper.
  • Started working again on the review paper (interpolation methods).
What I'm working on now
  • Manuscript draft: improving first paper draft.
  • Analyses for paper on multi-time scale methods: ongoing.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.
  • Edit review paper on interpolation methods.

2013-19-16

What I did the past week
  • Ran analyses for 0 to 70% monthly holdout proportions for gam_CAI,gam_fus, kriging_CAI,kriging_fus (0ec7e980, 34598372, de8a9a29, 98a32cf9, c5d0da95, a6de78b2)
  • Modifications of analyses for paper on contribution of covariates (a6de78b2)
  • Worked on analyses for the second paper, monthly hold out proportions from 0 to 70% for gam_CAI,gam_fus, kriging_CAI,kriging_fus (a8c14dda)
What I'm working on now
  • Manuscript draft: improving first paper draft.
  • Analyses for paper on multi-time scale methods: ongoing.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.

2013-09-09

What I did the past week What I'm working on now
  • Manuscript draft: improving first paper draft.
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.
  • Finish modifiying monthly hold-out: fusion methods.

2013-09-03

What I did the past week What I'm working on now
  • Manuscript draft and analyses: write up/revisions and improvements.
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.
  • Finish modifiying monthly hold-out.

2013-08-26

What I did the past week
  • Focused on analyses and on producing figures and tables for paper on single time scale methods (fff944c5 , 25388c36 )
  • Continued editing the manuscript draft: initial draft revisions.
  • Looked into hold out for gam CAI and gam fusion methods.
What I'm working on now
  • Manuscript draft and analyses: write up/revisions and improvements.
  • Analyses for paper on multi-time scale methods.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.

2013-08-19

What I did the past week
  • Focusing on papers: ran analyses with corrected screening for single (5808debe , a1721e6a) and multiple methods (2168e318 , 863a01f6, b516681e )
  • Analyses paper 1: major modifications towards producing figures and tables for manuscript (1f24e304, d8763ab5, 3ab94e28)
  • Continued editing the manuscript draft: write up ongoing.
What I'm working on now
  • Manuscript draft and analyses: write up.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.

2013-08-12

What I did the past week What I'm working on now
  • Manuscript draft and analyses
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.

2013-08-03

What I did the past week
  • Focusing on papers: ran analyses over four weeks (9287ad3b , 4fbd2f73 ,455b069b), multisampling (aa37eb9e)
  • Read papers and edited the manuscript draft.
  • Added methods to CAI and Fusion: GWR_fusion (a2f26ded), Kriging_fusion (b493e133), GWR_CAI (cbd65694), Kriging_CAI (e11b9a11)
  • Examining and checking results for multi-time scale methods (Fusion and CAI)
What I'm working on now
  • working on papers: analyses (running multisampling, statistical tests) and reading...
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for manuscript drafts: this is currently the priority.
  • Examining possible content for paper 2- ongoing work at this stage.

2013-07-28

What I did the past week
  • Focusing on papers: ran analyses and read papers.
What I'm working on now
  • working on papers: analyses (running multisampling, statistical tests) and reading...
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for paper drafts: this is currently the priority - ongoing work at this stage.

2013-07-21

What I did the past week What I'm working on now
  • working on papers: analyses (running multisampling, statistical tests) and reading...
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for paper drafts: this is currently the priority - ongoing work at this stage.

2013-07-14

What I did the past week
  • Focusing on papers. I'm immersing myself back in the literature and cleaning out the references in the review.
  • Running analyses for paper 1.
  • Documented workflow (workflow and code )
What I'm working on now
  • working on papers: analyses and reading...
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for paper drafts: this is currently the priority - ongoing work at this stage.

2013-07-07

What I did the past week
  • Worked on papers drafts, write up and references.
  • Continued analyses to examine in details fitting of GAM with the te() option compared to s() for OR paper and predictions (899859d9 , b337838a )
  • Ran analyses with screening for gam day and new combinations of variables with nesting (b337838a).
  • Ran analyses with additional screening of data training for model comparison (ca16094e , ca16094e)
  • Fixed some issues in covariate production script (projection assignment: 525fff70).
  • Modifications to handle errors in model fitting GAM fusion (44f43362)
  • Documented workflow and added it on the wiki (workflow and code )
What I'm working on now
  • documenting workflow
  • working on papers
  • working in details on GAM options and analyses for papers.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Moving forward quickly on work for paper drafts: this is currently the priority - ongoing worked at this stage.
  • Finish analyses on issues with "te" tensor option: works at monthly time scale but not at daily, still in progress
  • Solve issues related to calls to python+GRASS script, this will significantly speed up the LST clim production.
  • Assess storage and memory of production for scaling up

2013-06-29

What I did the past week
  • Worked significantly on all three papers and currently preparing the Endnote references library for the review.
  • Started analyses to examine in details fitting of GAM with the te() option compared to s() (for paper and predictions)
  • Started analyses to examine fitting with specified knots and bases in GAM (for paper)
  • Fixed screening function to remove extreme values in covariates (d95d4617).
  • Testing screening in Oregon with GAM (7f46c991)
What I'm working on now
  • documenting workflow
  • working on papers
  • working in details on GAM options and analyses for papers.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Continue work on paper drafts: this is currently the priority - ongoing work at this stage.
  • Continue work on issues with "te" tensor option: works at monthly time scale but not at daily, still in progress
  • Solve issues related to calls to python+GRASS script, this will significantly speed up the LST clim production.
  • Assess storage and memory of production for scaling up
  • workflow document and code for scaling up.

2013-06-22

What I did the past week
  • examined output for Oregon (GWR_daily, GAM_daily, Kriging_daily) and put together a presentation for IPLANT meeting update.
  • examined output for Queensland region, looking into the LST averages and number of valid pixels.
  • added screening function to remove extreme values in covariates (1ee7c0bb).
  • added function to define projection system on the fly based on modis tiles (84774d51, e9283712)
What I'm working on now
  • Examining results for Queensland region
  • documenting workflow
  • checking screening function, it seems to give strange results
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • work on issues with "te" tensor option, did not work
  • Solve issues related to calls to python+GRASS script, this will significantly speed up the LST clim production.
  • Assess storage and memory of production for scaling up
  • Work on paper drafts: priority.

2013-06-15

What I did the past week
  • modifications in results output analyses to evaluate daily predictions (maps creation etc.) (18837c79).
  • added GWR (single time scale) method for daily prediction (3b4bb643,ab884b16).
  • data preparation, slight modifications following debug for Queensland case (4adca9a2)
  • ran GAM fusion for new area for testing: Queensland (007cf8bd, b11dd3ab)
  • worked on documenting workflow and code.
  • adjusted number of cores and models to handle memory usage for Queensland case
What I'm working on now
  • Test full workflow from LST processing to temperature predictions: works in Queensland check on number of models and cores.
  • documenting workflow
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Add screening function in covariate script: next week.
  • work on issues with "te" tensor option, did not work
  • Solve issues related to calls to python+GRASS script, this will significantly speed up the LST clim production.
  • Add on the fly reprojection function with buffer for any study area.
  • Assess storage and memory of production for scaling up
  • Work on paper drafts: priority.

2013-06-08

What I did the past week
  • modifications related to covariates production for automation (3a8a8e72, 195bbbb0).
  • added GAM (single time scale) method for daily prediction (230a3ae4,b2563a44).
  • added Kriging (single time scale) method for daily prediction (40e4d58b, fc7c30c1), runs in 14h compared to 48h previously.
  • running GAM CAI for Venezuela with monthly stations for the 2000-2010 time period (da01a629)
  • worked on the NASA report.
What I'm working on now
  • Test full workflow from LST processing to temperature predictions: works in OR and Venezuela.
  • Adding GWR (single time scale) method for daily prediction (almost done), it is necessary to complete the OR papers.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Add screening function in covariate script: next week.
  • Fix results output analyses to evaluate daily predictions (maps creating etc.)
  • work on issues with "te" tensor option, did not work
  • Solve issues related to calls to python+GRASS script, this will significantly speed up the LST clim production.
  • Add on the fly reprojection function with buffer for any study area.
  • Assess storage and memory of production for scaling up
  • Work on paper drafts!!!

2013-05-25

What I did the past week
  • downloaded tiles and modified LST calculation to attempt GRASS multiple calls, still working on it (dfe13c01, f3e5567c)
  • gam fusion, debugging and testing of additional models with x and y covariates with nesting, as well tensor (b345fd0a, 2d6af3b8,a14b03e1, bd69aaca)
  • Continuing modifications to covariates script production to speed up and to allow new inputs and outputs for any region/tiles (c723ce75)
What I'm working on now
  • Making full workflow work from LST processing to temperature predictions.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Add screening function in covariate script and fully integrate in workflow.
  • Run alternative models for CAI and FSS methods for paper: issues with "te" option
  • Solve issue and debug parallel call to python+GRASS script, this will significantly speed up the LST clim production.
  • Once the backbone of workflow is completed, run all 5 stages from climatology to results.
  • Assess storage and memory of production for scaling up.
  • Work on paper drafts!!!

2013-05-18

What I did the past week
  • Integrate LST processing in the work flow, several scripts modifications (a462ee65, 8f176858, 55408bb4)
  • Automate call to LST downloading and climatology calculation: modifications of python script 2a8747de, 73b1bea9,9bcb2aa4)
  • Modifications of covariates script to handle new LST inputs and outputs for any region/tiles (661e6ef0)
What I'm working on now
  • Making full workflow work from LST processing to temperature predictions.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Add screening function in covariate script and fully integrate in workflow.
  • Run alternative models for CAI and FSS methods for paper.
  • Once the backbone of workflow is completed, run all 5 stages from climatology to results.
  • Assess storage and memory of production for scaling up.
  • Work on paper drafts!!!

2013-05-11

What I did the past week
  • Modifications of workflow script to collect predictions in output folder and in log file (3c422b41, 06aef3e5)
  • Modifications of raster predictions script related to log file and output storage (5ed5c7a3 , 5f91c983)
  • Modifications,clean up of validation and output analyses (f23e6e04 , 5b9ca6d6)
What I'm working on now
  • Working on papers for the Oregon case study.
  • Integrate Climatology python code to allow command line call for workflow.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Add screening function in covariate script and fully integrate in workflow.
  • Modify Climatology python script to allow call from R, this will integrate the step in the workflow.
  • Run alternative models for CAI and FSS methods for paper.
  • Once the backbone of workflow is completed, run all 5 stages from climatology to results.
  • Assess storage and memory of production for scaling up.

2013-05-03

What I did the past week
  • Ran GAM CAI method (67ab82b4; 7cd6d106)
  • Modifying results script for GAM CAI and other methods (b691ffcf)
  • Worked on paper outlines and content for the Oregon case study.
  • Added validation by month (617d83ff)
  • Discussed similarity between CAI and FSS.
What I'm working on now
  • Checking GAM CAI results.
  • Working on papers for the Oregon case study.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Work on log files and estimation of size, time/memory for the production for Venezuela (6 tiles).
  • Workflow considerations: discuss issues related to region of fit and prediction.
  • Add screening function in covariate script.
  • Continue working on paper drafts.

2013-04-26

What I did the past week
  • Worked on adding GAM CAI method (9634f124 ; 65d96c1a ; 079bc1c5)
  • Worked on paper outlines and content for the Oregon case study.
  • Discussed issues related to the fitting and predictions for the workflow
What I'm working on now
  • Adding GAM CAI method: debugging
  • Working on paper outlines from the work in Oregon and review.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Finish debugging GAM CAI
  • Fix log files and assess size and time of production for Venezuela (6 tiles).
  • Continue working on papers.

2013-04-20

What I did the past two weeks
  • Worked on creating figures for the conference and papers (d8f6d848 ; 6494f05a)
  • Worked on paper outlines for the Oregon case study.
What I'm working on now
  • Adding GAM CAI method.
  • Preparing outlines for potential papers.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Get GAM CAI running.
  • Fix log files and assess size and time of production for Venezuela (6 tiles).
  • Continue working on papers.

2013-04-12

I am out of town at the Association of American Geographers conference.

2013-04-06

_*First test run for Oregon and TMIN predictions appear stable.

What I did the past week
  • Ran predictions for Venezuela,TMAX year 2010 using monthly average 2000-2011 (1d3564da).
  • Modified code to examine station and covariate outputs, this will allow to keep track of number of stations and examine input covariates (2ba35352).
  • I worked on TMIN predictions: change in output analyses and various modifications(f9b40867 ; 1883c4c2).
  • Ran TMIN predictions for year 2000, Venezuela with monthly average 1981-2011 (8bb1a487)
  • Worked on AAG figures and transition code to use previous covariate stack for Oregon TMAX interpolation (e3448ab5 ; d0df33ae)
  • General modifications to allow predictions for Oregon test region (1230ccc6 ; e7efff8e ; 68ab6e18)
  • First test run for Oregon TMAX region using continuity data for comparison of results (87cbaa98)
What I'm working on now
  • Examining results of TMIN prediction for Venezuela.
  • Working on Oregon TMAX predictions.
  • Preparing for the AAG conference.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Planning to run analyses in Oregon to test code: started the work.
  • Think about material for Oregon paper (after the AAG conference): will follow test runs.
  • Add and debug methods: GAM CAI (run both for Venezuela and Oregon)
  • Prepare material for the AAG conference next week.
  • Looking into cloud climatology issue??

2013-03-29

What I did the past week
  • Resolved some issues related to sourcing and calls from raster prediction and debugged validation step (7506a985 ; 6cb81608).
    Predictions now running by sourcing directly the master script: stage 3 and 4 are automated ( 2ba31980).
  • I examined the four TMAX predictions (https://projects.nceas.ucsb.edu/nceas/documents/47) made some necessary modifications concerning
    the time period used in the monthly station averages and quality flags (5c5e50b9 ; 0709a332).
  • I worked on TMIN predictions and made a first predictions test (904c6fa3 ; 87a06ee7).
  • Started creating a new script for output output analyses: it summarizes locations, number of stations
    and examines covariates (0847d47c)
What I'm working on now
  • Examining results of the first TMIN prediction for Venezuela.
  • Rerunning predictions for year 2000 and 2010 with two time period for monthly averages (1980-2010 and 2000-2010)
    (a00c326f ; c1bdf4b3)
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Planning to run analyses in Oregon to test code: this will go a lot faster now with the improvement in speed and design (after the AAG)
  • Think about material for Oregon paper (after the AAG conference)
  • Produce minimum temperature predictions for Venezuela: next week
  • Add and debug methods for Venezuela: GAM CAI
  • Prepare material for the AAG conference in April, decide on the presentation.
  • Looking into cloud climatology issue??

2013-03-22

I have now run 4 predictions in Venezuela:
1) tmax for year 2010 with monthly station averages 2000-2010
2) tmax for year 2010 with monthly station averages 1980-2010
3) tmax for year 2000 with monthly station averages 2000-2010
4) tmax for year 2000 with monthly station averages 1980-2010

What I did the past week
  • Turned covariate production script into function and produced LST night climatology (70d0063c ; b8506b64).
  • Modified code for TMIN prediction, still undergoing (62a9286c ; c0c600a6).
  • Ran Venezuela prediction with station climatology for 1980-2010 for year 2000 (83ead2bb).
  • Modified output output analyses into function to examine the four prediction runs (2df5f40f)
What I'm working on now
  • Modifying code to allow TMIN prediction
  • Examining and checking output from multiple tmax predictions
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Planning to run analyses in Oregon to test code: this will go a lot faster now with the improvement in speed and design.
  • Think about material for Oregon paper.
  • Produce minimum temperature predictions for Venezuela.
  • Add and debug methods for Venezuela: GAM CAI.
  • Prepare material for the AAG conference in April, decide on the presentation.
  • Looking into cloud climatology issue??

2013-03-15

The master temperature script seems to work now. I'm still debugging in the validation stage.

What I did the past week
  • Turned raster prediction script into a function (1cd7ecf2).
  • Many modifications and debugging to automate script for any year, date and monthly station climatology (eec6f6d5 ; 5eb62e61).
  • Ran Venezuela prediction with station climatology for 1980-2010, checking results (30c73d6b).
  • Ran first code to produce Venezuela tmax for year 2000: checking results (d7f715b5)
What I'm working on now
  • Modifying code to allow TMin prediction
  • Examining and checking output from multiple tmax predictions
  • Looking into the validation function.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Planning to run analyses in Oregon to test code: this will go a lot faster now with the improvement in speed and design.
  • Think about material for Oregon paper.
  • Start producing TMIN LST climatology for Venezuela: this will necessitate modifications in the python script.
  • Add and debug methods for Venezuela: GAM CAI.
  • Prepare material for the AAG conference in April, decide on the presentation.
  • Looking into cloud climatology issue??

2013-03-09

What I did the past week
  • Put sampling step into a function (e38d3d87).
  • Modified raster prediction to allow more parameters (33095a53).
  • Put together an initial master script for temperature prediction (fff54c78).
  • Working through automating prediction steps to allow testing of different parameters
What I'm working on now
  • Updating workflow document to add in an issue
  • Working on GAM CAI.
  • Transforming raster prediction into function
  • Run Venezuela prediction with different monthly climatology.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study: I plan to get back to the prediction within two weeks.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)--I will produce tmin climatology.
  • Produce GAM CAI predictions for Venezuela.
  • Complete master script stage 4 to do more predictions for Venezuela
  • Looking into cloud climatology issue??

2013-03-04

What I did the past week
  • Modified code for data preparation to make it a function (d8a3533b).
  • Modified validation script to produce boxplots (51833ab5).
  • Modified raster prediction script adding GAM CAI: it still needs some work (57a4fede).
  • Summarized some of the results for Venezuela: Issue #568.
What I'm working on now
  • Reducing the number of parameters and transforming all scripts into functions for easy automation --
    this will also help in testing different settings for predictions.
  • Updating workflow document to add in an issue: I will create an issue this week.
  • Working on GAM CAI.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study: I plan to get back to the prediction within two weeks.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)--I will produce tmin climatology.
  • Produce GAM CAI predictions for Venezuela.

2013-02-23

What I did the past week
  • Modified the prediction script to keep track of processing time using a log file (c1e1cc30).
  • Added script to produce figures of selected for analyses of results(51833ab5), validation.
  • Spent substantial amount of time to check the output to spot errors.
What I'm working on now
  • I'm going through the results for Venezuela to check the dailyTmax predictions.
  • Updating workflow document to add in an issue: I will create an issue next week.
  • Setting up more frequent updates on wiki: I started now see: Issue #568
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study: I hope that we will discuss this point during the meeting.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)--I will produce tmin climatology.
  • Produce GAM CAI predictions for Venezuela using the modified new code.
  • Start adding all the documents (workflow, prediction results etc.) on wiki (ongoing).

2013-02-15

What I did the past week
  • Modified the prediction script to improve the processing time to make raster prediction (0163d0e2) , it now takes about 15-20 minutes for model prediction for 6 tiles (using GAM model).
  • Modified GAM fusion code to separate interpolation into three steps: monthly clim step (3e1b1ed4), daily delta deviation (3e1b1ed4), validation (48e49857).
  • Produced a first prediction for 365 dates in Venezuela (0f602e87). It took 6h25 minutes. I'm checking the results to make sure everything is correct.
What I'm working on now
  • I'm going through the results for Venezuela to check the dailyTmax predictions.
  • Update workflow document to add in an issue.
  • Setting up more frequent updates on wiki.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)
  • Produce GAM CAI predictions for Venezuela using the modified new code.
  • Start adding all the documents (workflow, prediction results etc.) on wiki.

2013-02-09

What I did the past week
  • Modified the script for extraction from the GHCND database-added monthly query (6a5b56da)
  • Worked on the workflow for the general production of layers: will be put on the wiki next week.
  • Modified GAM fusion code to make first tmax predictions (0f4426cb, 760957d7 )
What I'm working on now
  • I'm modifying running GAM fusion code for Venezuela. It will work for any tile/region. Took 2 hours to predict using GAM with one covariate. Very slow.
  • Examining the number of stations available in the Venezuela regions.
  • Examining Venezuela covariates.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)
  • Produce tmax prediction for Venezuela: first production for GAM fusions.
  • Put together the beginning of a workflow for every one to discuss.

2013-02-01

What I did the past two weeks
  • Modified the LST climatology script: LST climatology has now been produced for 6 tiles in the Venezuela region (63f27fec)
  • Produced a covariate script (see 88cf72bb), the aim is to have a general script for any region.
    It is close to be completed. Covariates for Venezuela are ready.
  • Modified the script for extraction from the GHCND database (e6cc535b)
  • Started working on the workflow for the general production of layers.
What I'm working on now
  • I'm modifying the GHCND database script and the LST climatology (add night option for Tmin).
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Summarize and write papers for the Oregon case study.
  • Produce LST climatology for Venezuela case study: night and day (check on the flag option)
  • Produce tmax prediction for Venezuela. I'm hoping to have predictions within a week or two.
  • Put together the beginning of a workflow for every one to discuss.

2013-01-18

What I did the past few weeks
  • Worked on additional analyses for the IBS conference poster.
  • Started working again on the LST climatology script: picking up where I left in November 2012.
What I'm working on now
  • I'm modifying the python+Grass code to produce LST climatology for the Venezuela study region.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Upload the remaining codes in the repository: CAI, GAM, poster script + python codes.
  • Summarize and write papers for the Oregon case study.
  • Produce LST climatology for Venezuela case study.
  • Produce covariates for Venezuela case study and other areas

2012-12-12

What I did this past week
  • Focused on using SNOTEL data to compare CAI and fusion methods.
  • Created plots of MAE and RMSE for two days (20100103 and 20100901) as well as full year for Fusion, CAI using GHCN and SNOTEL datasets.
  • Ran CAI with constant station used as training and Elevation as covariate to compare results using SNOTEL and GHCN datasets.
What I'm working on now
  • Comparing CAI and fusion predictions using SNOTEL data.
  • Modifying code to compare CAI_kr with CAI+Elev and CAI_WorldClim
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Complete the comparison using GHCN and SNOTEL network data.
  • Upload remaining codes in the repository: there are more than 10-15 scripts related to interpolation (GAM, CAI etc.).
  • Summarize and write papers for the Oregon case study.
  • Produce climatology for Venezuela case study.

2012-12-03

What I did this past week
  • Analyzed differences between CAI and fusion in relation to elevation and land cover.
  • Produced spatial transects through CAI and fusion in relation to elevation.
  • Continued analyzing residuals at validation stations for 2 dates (20100103 and 20100901).
  • Downloaded and assembled SNOTEL datasets to compare CAI and fusion performances in higher elevation and sparsely sampled areas.
What I'm working on now
  • Working on residuals analyses: by stations, month, maps and covariates (i.e. elevation, land cover)
  • Comparing CAI and fusion predictions using SNOTEL data.
  • Highlighting differences in predictions related to covariates (elevation, land cover etc.)
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Clean up codes to upload in the repository: there are more than 10-15 scripts to upload at this stage.
  • Summarize and write papers for the Oregon case study.
  • Produce climatology for Venezuela case study.

2012-11-18

What I did this past week
  • Analyzing residuals from models in details: plots for testing stations etc.
  • Analyzing differences between CAI and fusion in relation to elevation and land cover.
What I'm working on now
  • Working on residuals analyses: by stations, month, maps etc.
  • Working on CAI and fusion comparison in details: highlighting differences in prediction related to covariates (elevation, land cover etc.)
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Clean up codes to upload in the repository: there are more than 10 scripts to upload at this stage.
  • Summarize and write papers for the Oregon case study.
  • Produce climatology for Venezuela case study.

2012-11-12

What I did this past week
  • Started analyzing residuals from models in details.
  • Modified a series of codes for residual analysis and methods comparison
What I'm working on now
  • Working on residuals analysis: by stations, month, maps etc.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Perform residuals analyses and examine differences in predictions between CAI and fusion.
  • Clean up codes to upload in the repository: there are more than 10 scripts to upload at this stage.
  • Focus on writing up results in a paper format.
  • Produce climatology for Venezuela case study.

2012-11-05

What I did this past week
  • Ran analyses with all stations at the monthly step for the CAI and Fusion methods.
  • Modified code for screening and ran analyses with screening of LST and ELEV_SRTM.
What I'm working on now
  • Modifying codes to look at specific station residuals.
  • Currently analyzing additional results necessary to end the Oregon Case study.
  • Modifying codes for the method comparison to finish the analysis.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Perform residuals analyses.
  • Clean up codes to upload in the repository: there are about 10 codes to upload.
  • Summarize results in presentation.
  • Write up results in a paper format.
  • Produce climatology for Venezuela.

2012-10-28

What I did this past week
  • Modified the CAI and Fusion codes to use all stations at the monthly time scale.
  • Modified code of method comparison part I, to plot monthly summaries of MAE and RMSE.
What I'm working on now
  • Modifying codes to look at specific station residuals.
  • Currently running codes back to back to get results necessary to end the Oregon Case study.
  • Modifying codes for the method comparison to finish the analysis.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Running remaining codes and analyses to complete the Oregon case study.
  • Modify summary presentation to take into account the new results.
  • Write up results in a paper format.
  • Produce climatology for Venezuela.

2012-10-20

What I did this past week
  • Finished the first version of the method comparison presentation for the meeting.
  • Worked on preparing the data for the next case study (Venezuela).
  • Worked on some more analyses for the CAI and fusion comparison.
What I'm working on now
  • Cleaning out codes for submission to the repository.
  • Completing the run of code for the Oregon Case study.
  • Working with Python and Grass to compute the climatology by modifying Jim's code.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Running remaining codes and analysis and for the Oregon case study.
  • Produce climatology for Venezuela.

2012-10-13

What I did this past week
  • Focused on making a more detailed presentation of the method comparison work for the next meeting.
What I'm working on now
  • Working on the presentation for the method comparison meeting.
  • Cleaning out codes for submission to the repository.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Finishing the presentation. This may serve as a first outline of paper.
  • Running remaining codes.
  • Setting up datasets and codes for the next study area.

2012-10-06

What I did this past week
  • Continued running codes to complete method comparison
  • Recorded results in presentation.
  • Examined accuracy at specific meteorological stations.
What I'm working on now
  • Working on the method comparison and on producing summary figures and presentations.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Examine in more details some meteorological stations to understand differences in the CAI and Fusion.
  • Calculation of the LST average for the next study area (modify GRASS-python code).

2012-09-29

What I did this past week
  • Continuing the work on methods comparison with multisampling, spatial density and accuracy etc.
  • Continuing the compilation of figures and summary numbers.
  • Examining the code for LST average calculation.
What I'm working on now
  • Working on the method comparison and on producing summary figures and presentations.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Continue running all the remaining codes to complete the OR case study.
  • Start report/paper on model comparison.
  • Calculation of the LST average for the next study area (modify GRASS-python code).

2012-09-21

What I did this past week
  • Worked on multisampling and other assessments.
  • Compiling figures and numbers for summary of the results.
  • Started looking at the code to be modified for LST average calculation.
What I'm working on now
  • Producing summary figures and presentations
  • Multi sampling and spatial distance accuracy and other assessments (ongoing)
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Run all the remaining codes to complete the OR case study.
  • Start report on model comparison.
  • Start the process to calculate and produce LST for the next testing region (modify GRASS-python code)

2012-09-13

What I did this past week
  • Task #491, committed initial script for the accuracy assessment based on spatial distance to the closest training station: see 2bdb1ff5
  • Task #491, committed initial script for model comparison: see 170edade
  • Task #491, multisampling and proportion assessment for kriging method and GAM fusion: committed to git (see 4719fdd7 ; e7bf2d1b ; 101f27b0 ; 69864891)
What I'm working on now
  • Model comparison assessment of outputs from scripts (task #491)
  • Producing a summary figures and presentations
  • Multi sampling and spatial distance accuracy assessment (ongoing)
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Run and modify multi sampling spatial accuracy codes for all methods (task #491)
  • Start report on model comparison.
  • Start the process to calculate and produce LST for the next testing region.

2012-09-01

What I did this past week
  • Started working on the accuracy assessment based on spatial distance to fitting station
  • Modified code for model comparison.
  • Started working on Multi sampling and proportion assessment for kriging method.
What I'm working on now
  • Model comparison
  • Examining the results for all four methods (GWR, Kriging, GAM and Fusion): producing a summary power point
  • Multi sampling and spatial distance accuracy assessment
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Run multi sampling on Kriging and spatial accuracy code.
  • Start report on model comparison.
  • Update red mine task to keep track of new and ongoing tasks

2012-08-26

What I did these past week
  • Reran GWR, it took 132 hours.
  • Worked on code for model comparison.
  • Wrote and ran test code for multi sampling and proportion assessment. it is in the repository.
What I'm working on now
  • Model comparison
  • Examining the results for all four methods: GWR, Kriging, GAM and Fusion
  • Multi sampling: waiting for the results for GAM and fusion.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Write a code to assess model in terms of accuracy and distance to station (see review).
  • Modify code for multi sampling in Kriging and GWR.
  • Start report on model comparison.

2012-08-17

What I did these past two weeks
  • Modified the GWR code for model 8, improved the code in general and corrected minor errors.
  • Currently rerunning the prediction for GWR.
  • Started writing code for model comparison.
  • Started writing code for multi sampling and proportion assessment
What I'm working on now
  • Model comparison
  • Multi sampling: I am writing a code to create a curve of accuracy for every method for a given proportion.
What obstacles are blocking progress
  • There are no major obstacles right now.
What's next
  • Write a code to assess model in terms of accuracy and distance to station (see review).
  • Checking and assessing the results from raster prediction for all methods.
  • Start writing the outline of the report for the method selection using the results from GAM, Fusion and Kriging

2012-08-02

What I did this past weeks
  • Worked on the GWR code: I was able to make raster prediction for a date and two models in OR. This required significant work to deal with NA in raster stacks
    since GWR does not work if data is incomplete.
  • Improved the code in general.
  • Worked on the kriging code: I have modified the code and have results for 365 dates.
  • Looked at the land cover and LST interactions.
What I'm working on now
  • GWR: modifying the code for one full year raster prediction.
  • Kriging: checking the results for a full year.
What obstacles are blocking progress
  • Trying to improve visualization of results in R.
  • There is no major obstacle right now.
What's next
  • I plan to finish the GWR and perform one year raster prediction for OR (365 days).
  • Start writing the outline of the report for the method selection using the results from GAM, Fusion and Kriging
  • Once all the results (i.e. time series of predictions and associated models+ accuracy metrics) are ready I will summarize all results.
  • I still need to perform changes in the fusion methods using new accuracy methods.

2012-07-26

What I did this past weeks
  • Literature review: I worked on the editing and references. I posted the initial draft. It is work in progress.
  • Worked on the fusion and GAM code: worked on debugging the code to predict for a full year in OR. It is working now.
  • Worked on the kriging code: used the fusion/GAM code and will be making full prediction soon.
What I'm working on now
  • GWR: started to modify the GAM/fusion code to have one full year raster prediction.
  • Kriging: need to run the code for a full year. I'm waiting for GAM/Fusion to finish since I'm using a lot of resources on ATLAS.
What obstacles are blocking progress
  • I worked with Jim to solve problems related to memory usage. Raster prediction over a full year in OR takes up a lot memory and computing power.
  • Trying to improve visualization of results in R.
  • Big progress this week: solved the problem in memory with Jim's help. Waiting for the code to finish running (it has been running for 28 hours).
What's next
  • I plan to finish the GWR and CAI codes and perform one year raster prediction for OR (365 days).
  • Start writing the outline of the report for the method selection using the results from GAM and fusion.
  • Once all the results (i.e. time series of predictions and associated models+ accuracy metrics) are ready I will summarize all results.
  • I still need to perform changes in the fusion methods using new accuracy methods.

2012-07-18

What I did this past weeks
  • Literature review: still working on the review-writing and references
  • Worked on the fusion code: assessment of the results, I modified the code to save the prediction for a year
  • Worked on the kriging code: Universal Kriging. I have now results. There a better than for Ordinary Kriging but not as good as GAM and fusion.
What I'm working on now
  • Fusion method: I'm modifying the code to assess the model with varying validation proportion.
  • Literature review: working on the references and writing
  • Kriging: analyzing UK results
What obstacles are blocking progress
  • I am trying to learn to use mclapply for use multi cores on Atlas. This is useful when doing full prediction for Oregon over one year.
What's next
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • Finish Kriging assessment: this is well underway.
  • Do GWR assessment
  • Improve and assess the fusion method: this is still in progress.
  • Perform a CAI run using only station data rather than LST with station data.
  • Start writing up the results to assess all the methods

2012-07-11

What I did this past weeks
  • Literature review: finished the structure and content is done.
  • Worked on the fusion code: added code for validation, training and a modeling of LST_BIAS through GAM.
  • Worked on the kriging code: added automatic fit of variograms. Results are still disappointing so I will try Universal Kriging.
What I'm working on now
  • Fusion method: I'm trying to assess the performance of the model and exploring some LST_BIAS models.
  • Literature review: I'm working on the writing and adding references. It is in an unpolished state.
  • Kriging: I'm working on different models using covariates.
What obstacles are blocking progress
  • I need to learn ways to display results using R graphics.
What's next
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • GWR and Kriging: complete the tasks so that results can be compared to Fusion and GAM: I plan to have results for the next IPLANT meeting.
  • Improve and assess the fusion method: this is in progress.

2012-07-03

What I did this past two weeks
  • Worked on the literature review: progress has been made and I hope to be able to complete the first draft soon.
  • Read in details the connection between GAM TPS and Kriging as well as splines in general
  • Worked on the fusion code.
  • Worked on two codes for data preparation: task #363 and task #360
What I'm working on now
  • Fusion method: we are now using data from the new POSTGRES database and the code is being modified to compare results.
  • I am adding references and finishing two sections in the literature review.
What obstacles are blocking progress
  • I need to learn ways to display results using R graphics.
What's next
  • Complete some of the many ongoing issues...
  • Complete the literature review.
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • GWR and Kriging: complete the tasks and use auto map.
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Carry on with the work on the fusion method: this is in progress.

2012-06-20

What I did this past two weeks
  • Working on the literature review: this has taken most of my time. The goal is to have a draft out before next call.
  • Read more details on GAM and other methods.
  • Worked again on the fusion R script from Brian to modify it so that results can be compared to other methods.
What I'm working on now
  • Fusion method: there are many details to make the results fully comparable to other models.
  • Accuracy and validation methods used in interpolation: it is one of the section of the review.
  • Literature review: the paper is organized in different sections and I am adding substantial information with many references.
What obstacles are blocking progress
  • I need to learn ways to display results using R graphics.
What's next
  • Complete some of the many ongoing issues...
  • Complete the literature review.
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • GWR and Kriging: complete the tasks using auto map.
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Carrying on the fusion method.
  • Provide a list of tasks and time frame to complete the Oregon case study.

2012-06-13

What I did this past two weeks
  • Working on the literature review.
  • GAM+Kriging, updated the script using auto map to fit automatically the variogram (task #364): I'm looking at the results now.
  • Worked on the fusion R script from Brian to modify it so that results can be compared to other methods.
  • Worked on the NASA report.
What I'm working on now
  • I'm still looking into the various feature of the automap R package.
  • Literature review was pushed back because I put more effort on the NASA report but it is still on track.
What obstacles are blocking progress
  • I need to learn ways to display results using R graphics.
What's next
  • Complete some of the many ongoing issues...
  • Work on the literature review and on understanding GAM in more depth.
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • GWR and Kriging: complete the tasks using auto map.
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Work on the fusion method.

2012-06-06

What I did this past two weeks
  • Started working on the literature review
  • Worked on two stage regression: GAM+Kriging, updated the script with specific and general diagnostic measures: task #364
  • Worked on a R script to query the GHCND database: this is a general script for Tmax, Tmin and Precip (for any study area)
  • Committed almost the entire sets of R scripts in the repository.
What I'm working on now
  • Looking into the auto map R package to fit automatically variograms for kriging.
  • My main focus is on the literature review which is needed to move to the next stages.
What obstacles are blocking progress
  • I need to learn ways to display results using R graphics.
What's next
  • Complete some of the many ongoing issues...
  • Literature review: highest priority.
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • Include automatic fitting of variogram using the auto map package ( GAM+Kriging): task #364
  • Continue GWR: write code for validation and compare results to GAM and Kriging: task #364
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Continue gathering relevant literature to compare GAM results: task #361.

2012-05-29

What I did this past two weeks
  • Visited Walter and Adam at Yale, CT: coordination of work.
  • Continued gathering literature on climate interpolation: reading through it right now.
  • Worked on spatial filtering in R: task #408
  • Worked on two stage regression: GAM+Kriging, ran the script for 365 dates in 2010: task #364
  • Started committing code in the repository.
What I'm working on now
  • Looking into the auto map R package to fit automatically variograms for kriging.
  • Spatial filtering: writing script.
  • Modifying code to add diagnostic measures for interpolation.
What obstacles are blocking progress
  • Spatial filtering involves calculating eigenvalues and eigenvectors of large matrices. I am trying to see if the method can be tested with R in Oregon.
What's next
  • Complete some of the many ongoing issues...
  • Continue GAM modeling: the plan will be to add MAE and other diagnostic measure to evaluate models : task#361
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • Explore automatic fitting of variogram using the auto map package ( GAM+Kriging): task #364
  • Continue GWR: write code for validation and compare results to GAM and Kriging: task #364
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Continue gathering relevant literature to compare GAM results: task #361

2012-05-16

What I did this past week
  • Worked on GAM assessment for the year 2010 (365 dates) producing average RMSEs per month: task #406.
  • Gathered literature to review and compare the RMSE, MAE and BIAS results from GAM modeling.
  • Worked on spatial filtering in R: task #408
  • Worked on two stage regression: GAM+Kriging, ran the script for 10 dates in 2010: task #364
  • Started using GIT to share code.
What I'm working on now
  • Continuing working on GAM+Kriging: evaluating results over 365 dates
  • Thinking of better ways to include the spatial structure from LST.
  • Continuing the assessment of proportion of hold out and sampling on GAM results.
  • Looking at ways to fit the variogram in kriging automatically.
What obstacles are blocking progress
  • Spatial filtering involves calculating eigenvalues and eigenvectors of large matrices. I am trying to see if the method can be tested with R in Oregon.
  • Need to assess results in the largest context of the literature.
What's next
  • Continue GAM modeling: the plan will be to add MAE and BIAS to RMSE to evaluate models : task#361
  • Assess the contribution of the different smoothing terms in GAM: task #361
  • Explore automatic fitting of variogram using the auto map package ( GAM+Kriging): task #364
  • Continue GWR: write code for validation and compare results to GAM and Kriging: task #364
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Continue gathering relevant literature to compare GAM results: task #361

2012-05-08

What I did this past week
  • Continued working on GAM assessment for the year 2010 (365 dates): task #406.
  • Created Histograms and bar plots to explore the RMSE distribution for the 365 dates: taks #406
  • Worked on spatial filtering in R: task #408
  • Worked on two stage regression: GAM+Kriging: task #364
What I'm working on now
  • Continuing working on GAM+Kriging.
  • Thinking of better ways to include the spatial structure from LST.
  • Continuing the assessment of proportion of hold out and sampling on GAM results.
  • Familiarizing myself with GIT repository: the aim is to start using GIT this week so that codes can be shared.
What obstacles are blocking progress
  • Spatial filtering involves calculating eigenvalues and eigenvectors of large matrices. I am trying to see if the method can be tested with R in Oregon.
What's next
  • Continue GAM modeling using LST monthly averages, Land cover variables and Canopy height: task#361
  • Work on GAM+Kriging (two stage regression): task #364
  • Continue GWR: write code for validation and compare results to GAM and Kriging: task #364
  • Continue the assessment of sampling (training and testing datasets) and proportions on GAM results: task #409.
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset: task #364 and task 408.
  • Read about screening methods used to improve the data in PRISM, WolrdClim and other projects: task #361
  • Use GIT to share code with the team.

2012-05-01

What I did this past week
  • Assess GAM predictions for the year 2010 (365 dates).
  • Working on model diagnostics from GAM predictions:
    - contribution of variables
    - searching for patterns in residuals
    - improving screening of unreliable/bad observations
  • Assess the effect of random samples on the RMSE for the 10 selected dates.
What I'm working on now
  • Continuing assessment of GAM predictions over the year and many random samples
  • Thinking of better ways to include the spatial structure from LST.
  • Documenting codes to start using the GIT repository.
What obstacles are blocking progress
  • There are no major stumbling blocks at this stage.
What's next
  • Continue GAM modeling using LST monthly averages and Land cover variables.
  • Work on GAM+Kriging (two stage regression).
  • Continue GWR: write code for validation and compare results to GAM and Kriging.
  • Continue the assessment of sampling (training and testing datasets) and proportions on the results.
  • Continue the assessment of models over 365 dates (year 2010), in particular seasonal effects.
  • Work on spatial filtering and other methods to better capture the spatial structures from the LST dataset.

2012-04-17

What I did this past week
  • Checking and assessing the monthly mean LST derived from MOD11A1.
  • Kriging and co-kriging in R:
    - Kriging using tmax
    - Co-kriging using tmax, Elev_SRTM, Eastness, Northness
    - Co-kriging using tmax,LST monthly mean
What I'm working on now
  • Continuing GAM prediction adding LST monthly mean as a variable.
  • Experimenting with co-kriging using LST monthly mean as a variable.
  • Documenting codes to start using the GIT repository.
What obstacles are blocking progress
  • No major stumbling blocks at this stage but a learning curve to produce maps and graphics as well as carrying out spatial analyses in R.
  • Documentation: I am behind in terms of documenting.
What's next
  • Continue GAM modeling using LST monthly averages and Land cover variables.
  • Continue to experiment with kriging and co-kriging to produce tmax predictions.
  • GWR: write code for validation and compare results to GAM and Kriging.
  • Assess the impact of the sampled stations (training and testing datasets) on the regression results.
  • Start getting the tmin variable prepared for the regression analysis.

2012-04-10

What I did this past week
  • Worked on getting up to speed on kriging, co-kriging and gstat.
  • Calculated LST monthly mean from MOD11A1.
What I'm working on now
  • Working on GAM models (LST inclusion and sampling).
  • Experimenting with co-kriging in R.
  • Documenting codes to start using the GIT repository.
What obstacles are blocking progress
  • No major stumbling blocks at this stage but a learning curve to produce maps and graphics as well as carrying out spatial analyses in R.
  • Documentation: coding is becoming more complex so I need to learn about git and use it for the project.
What's next
  • Continue GAM modeling using LST monthly mean and Land cover variables.
  • Use co-kriging to produce tmax predictions as well as residuals surfaces.
  • GWR: write code for validation.
  • Assess the impact of the sampled stations (training and testing datasets) on the regression results.
  • Start getting the tmin variable prepared for the regression analysis.
  • Run a GAM test over a whole year i.e. predict tmax for 365 dates?

2012-04-03

What I did this past week
  • Worked on the visualization of RMSE by updating R code to produce Kriged RMSEs and histograms for the GWR regression.
  • Included LST and Land cover in the GAM regression: this is still at an early stage.
What I'm working on now
  • Extending the visualization of RMSE to GAM assessment.
  • Learning about graphical displays and Kriging in R.
What obstacles are blocking progress
  • No major stumbling blocks at this stage but a learning curve to produce maps and graphics as well as carrying out spatial analyses in R.
  • Documentation: I have reached the stage where the number of outputs from regression analyses is increasing rapidly so that I will need to take more time to document steps and outputs.
What's next
  • Calculate monthly LST averages for the Oregon case study.
  • Continue the GAM modeling using LST and Land cover variables.
  • Use Kriging and co-kriging to produce tmax predictions.
  • GWR: write code for validation.
  • Assess the impact of the sampled stations (training and testing datasets) on the regression results.
  • Take some time to summarize and document the various analyses conducted over the past month (since the Tucson meeting).

2012-03-27

What I did this past week
  • Worked on R code to produce graphical display for validation (spatial distribution of RMSE).
  • Downloaded missing tiles and modified python code to use the Quality Check flags (QC) information.
What I'm working on now
  • Task #374 (LST averages): Recalculating LST averages by modifying python code to use the Quality Check flags (QC) to produce screened out daily averages for Oregon.
  • Learning about graphical displays and Kriging in R.
What obstacles are blocking progress
  • No major stumbling blocks but a learning curve to produce maps and graphics in R.
What's next
  • Before the next Webex meeting I intend to:
  • Complete the calculation of daily averages screened out.
  • Include LST-Canopy-Land Cover information in the GAM analysis.
  • Write enough R code to produce RMSE maps.
  • GWR: write code for validation.

2012-03-20

What I did this past week
  • Revised the R code to
    • integrate interaction terms
    • screen out "bad" values
  • Visualized interactions between variables: I produced quite a few figures in GAM.
  • Calculated LST averages: The LST averages have now been extracted and can be used for the interpolation. I am still in the process of downloading missing tiles (NASA Reverb) and working on the quality flags.
  • GWR: I produced several GWR predictions in R.
What I'm working on now
  • Task #374 (LST averages): I have extracted the Quality flags and will be using them to screen out "bad" values.
  • GWR reading about validation procedures
What obstacles are blocking progress
  • I will need some help to discuss the interaction terms in GAM.
What's next
  • Complete the LST processing taking into account QC flags.
  • Include LST-Canopy-Land Cover information in the GAM analysis.
  • Write R code for GWR validation

2012-03-11

What I did this past week
  • Revise the R code to integrate interaction terms and visualizing relationship in GAM.
  • Set up the processing for LST (in r and IDRISI/python). The averages need to be recalculated at the daily and monthly time steps.There are some missing tiles and the series does not extent until the end of the year 2011 so I started some download.
What I'm working on now
  • Task #374 (LST averages).
  • Reading and experimenting with GWR in r.
What obstacles are blocking progress
  • At this point, there are no stumbling blocks, I am getting slowly accustomed to several of the r packages for raster, spatial analyses.
What's next
  • My goal is to try to complete the three tasks (LST, GWR and interaction terms) so that we can have a productive phone call meeting on Tuesday 03/19.

2012-03-05

What I did this past week
  • Revise the R code to output correct RMSE for validation station for date 1507
  • Find out about diagnostic measures to assess the GAM models and integrate them in the r script
  • Organize the workflow in terms of tasks as listed on redmine: I added 6 tasks.
What I'm working on now
  • Task 361: putting together a r script to apply GAM on 10 dates in Oregon: script is running but I'm checking the results.
  • Reading more about GAM to get a deep understanding of the method.
What obstacles are blocking progress
  • At this point, there are no stumbling blocks, it is more an issue of a learning curve...I expect big payback in time soon.
What's next
  • My next task will be to add a spatial structure variable from the LST dataset.
  • I would like to create a map to visualize the RMSE distribution in space as well as improve the visualzation of results in a set of diagnostic chosen plots/graphs.