Activity
From 04/23/2012 to 05/22/2012
05/22/2012
- 10:14 AM Task #363: Assemble all GHCN data into a single database
- If useful for some specific purpose, we could generate relevant synoptic views or subsetted table dumps pivoted back ...
- 08:25 AM Task #363: Assemble all GHCN data into a single database
- Perhaps it would make sense to keep the data in the "one-row-per-month" format in the database and transform it on th...
- 07:52 AM Task #418 (New): Explore Landcover - LST interactions
- Use the full gridded datasets (rather than just station locations) to explore relationships between LST and various (...
05/17/2012
- 11:38 PM Task #416: Scope out workflow for calculating monthly LST climatologies
- I've now written some Python+GRASS code to do a good bit of what I think we want. For a user-specified _tile_, _year_...
- 03:05 PM Task #416 (In Progress): Scope out workflow for calculating monthly LST climatologies
- Assess feasibility of running LST climatologies ourselves by implementing a basic scripted workflow and estimating ov...
- 03:44 PM Task #375: Assemble monthly mean MODIS LST values for the complete record (2000-2012) for Oregon
- See task #416 for progress on developing and evaluating a procedure for doing this ourselves.
But I will add a com... - 02:16 PM Revision 01b3830e: added function documentation and removed two obsolete functions
- 02:04 PM Revision a05f49eb: automated determination of which days to aggregate for any given month
- 01:53 PM Revision 96a70efc: tweaked climatology calc function to return list of GRASS map names
- 01:43 PM Revision bd9bc508: switched from truncating to rounding mean LST back to integer
- 12:31 PM Revision 5fb982c8: wrote function to download HDF files for given tile/year/day-range
- 10:05 AM Revision b9e1a7f9: wrote function to look up local HDF files for given tile/year/day-range
- 09:12 AM Revision eaf6c10b: wrote function to encapsulate LST loading and QC-adjustment
- 08:45 AM Revision 47cdfebe: added initial Python+GRASS code to download and aggregate daily LST
- 08:45 AM Revision 934d6ab7: added script to reproduce calculation of 8-day LST from daily LST
05/16/2012
- 01:55 PM Document: Update IPLANT meeting roundup 05152012
- This document summarizes ongoing work for the IPLANT meeting on 05152012:
-GAM assessment over year 2010: average pe... - 01:49 PM Document: Update IPLANT meeting roundup 05012012
- This document provides updates presented at the IPLANT meeting 05012012:
- Results from GAM models run for the year ... - 01:41 PM Document: Update IPLANT meeting roundup 04172012
- This presentation presents mean monthly LST averages derived from daily MOD11A1. Results from Kriging and co-kriging ...
- 12:01 PM Task #207: Produce global fused DEM layer
- A global 90m product is now available, though the individual pieces have not been mosaiced together due to file size....
- 06:40 AM Task #415: Process MOD06_L2 Cloud data
- See presentation on current progress here [[https://projects.nceas.ucsb.edu/nceas/documents/18]]
- 06:40 AM Task #415 (New): Process MOD06_L2 Cloud data
- Download and process the MODIS cloud product (MOD06_L2) and produce monthly 1km summaries of key variables.
- 06:37 AM Document: Wilson update 5/15/2012
- Overview of processing the MOD06 cloud product
05/15/2012
- 11:22 AM Task #411 (In Progress): Insert DEM adaptive smoothing procedure into terrain workflow
- I developed two alternative implementations of the adaptive DEM smoothing procedure (@multiscalesmooth9a_clean.aml@):...
- 09:52 AM Task #411 (In Progress): Insert DEM adaptive smoothing procedure into terrain workflow
- The original Oregon terrain processing workflow, implemented as a set of AML scripts, included the application of Joh...
- 09:57 AM Task #208: Evaluate existing E&O AML scripts
- I recently reorganized all AML scripts into a single directory in the code repo, here:
source:terrain/research/orego...
05/14/2012
- 10:00 AM Task #363: Assemble all GHCN data into a single database
- Quick example comparing SQLite vs PostgreSQL timings on a simple aggregate query that doesn't use any indexes. SQLite...
05/13/2012
- 11:16 PM Task #363: Assemble all GHCN data into a single database
- Vaguely concerned about performance and manageability of such a large SQLite database, I wrote a second script to loa...
- 11:14 PM Task #363: Assemble all GHCN data into a single database
- Belated update. The TMIN/TMAX data loaded in ~6 hours total, yielding a 25GB SQLite file with nearly 600 million rows.
- 10:54 PM Revision c7235eae: added R script to load GHCN data into a PostgreSQL DB
- 10:50 PM Revision a7663038: further optimized functions in R for some minor extra speedup
05/11/2012
05/10/2012
- 03:03 PM Revision 0d1cde0a: Changes in the diagnostics plots, barplots.
- 02:49 PM Revision 25a68ae3: Transform aspect in Eastness and Northness variables in the GAM.
- 02:42 PM Revision 4180b123: Transformed the aspect variable and added GAM model closer to PRISM
- 02:28 PM Revision 2a89fa5f: GAM prediction added to the loop with RMSE calculation
- 02:13 PM Revision ec229367: Streamlining and slight changes to the loop to create subset dataset for different dates
- 02:07 PM Revision af42291a: Additional changes to the loop and format
- 01:47 PM Revision bc59c6bb: Start of Task #361, modified code to subset for 10 dates
- 11:59 AM Revision d7b0ef36: Minor format changes and clean up to code
- 11:18 AM Revision e2d673e4: Corrected errors in RMSE calculation and added plots to view results
- 08:49 AM Task #363: Assemble all GHCN data into a single database
- Data loading slowed to a crawl overnight, thanks to a record count query tucked inside in the bulk insert function th...
- 08:34 AM Revision 3b8dd010: dropped unnecessary record-count query from bulk insert function
05/09/2012
- 11:49 PM Task #363: Assemble all GHCN data into a single database
- Committed an R script that orchestrates the processing and loading of all 75000+ *.dly files into a SQLite database. ...
- 11:10 AM Task #363 (In Progress): Assemble all GHCN data into a single database
- We decided to organize and load _all_ available GHCN data (i.e., all dates and stations, globally) into SQLite databa...
- 11:06 PM Revision d4e63f5d: superficial code reformatting
- 10:29 PM Revision 38aa5c07: replaced reshape with faster manual split-rbind approach
- 04:11 PM Revision 669c150d: added grep pre-filtering of rows in the *.dly files
- 03:45 PM Revision 2d08ed05: replaced read.fortran with faster system call to awk/tr
- 03:41 PM Revision dd85f6d3: added initial R script to load GHCN data into a SQLite DB
- 03:40 PM Task #408: Capturing LST spatial structure using spatial eigenvectors
- I started writing the R code using the two functions from spdep package: ME() and SpatialFiltering()
I am trying to ... - 03:32 PM Task #364: Integrate spatial variables and structure in the GAM methodology
- I am currently exploring a two stage regression involving:
1) Step 1: GAM models using Lat, long, ELEV_SRTM, DISTOC,...
05/08/2012
- 04:31 PM Task #406: OR-GAM predictions-model assessment over 365 dates, year 2010
- RMSE were calculated per month over the year 2010. Results show that contrary to my expectation RMSE are higher in S...
05/04/2012
- 09:35 AM Revision 6c9ca794: simplified code by using 'a' flag for r.neighbors (improves 3abda63)
- 09:20 AM Revision 5af6ef96: reorganized AML terrain scripts to reduce superficial redundancy
05/02/2012
- 11:57 AM Task #393: Identify Focal Regions
- We are currently proposing the following regions (and MODIS tiles):
Oregon: H08V04, H09V04, H08V05, H09V0...
05/01/2012
- 02:24 PM Task #361: Test and compare the GAM method on several days (10 days for now) for Oregon
- Many models have been explored using the GAM methodology. At this stage, more detailed diagnostics is needed so for t...
- 02:16 PM Task #409 (New): Effect of sampling on GAM: hold out proportions and sampling observations
- This task examines the effect of the sampling and hold out proportions on tmax predictions. This task will help in de...
- 02:08 PM Task #364: Integrate spatial variables and structure in the GAM methodology
- Kriging has been done so far by using variograms fitted from stations' locations. The next step is to examine the fit...
- 02:03 PM Task #408 (New): Capturing LST spatial structure using spatial eigenvectors
- Spatial filtering is explored to assess its usefulness in capturing the LST spatial structure by creating spatial var...
- 01:56 PM Task #406 (New): OR-GAM predictions-model assessment over 365 dates, year 2010
- Models have so far been assessed on 10 dates spread over the year for the Oregon study area. This task provides an as...
- 01:20 PM Task #375: Assemble monthly mean MODIS LST values for the complete record (2000-2012) for Oregon
- The NASA Ames team is willing to generate the monthly LST climatologies, but we'll need to be specific about what we ...
- 01:20 PM Task #393: Identify Focal Regions
- Do we want to look at a mediterranean-type climate? Guess western part of South Africa qualifies. Or interior tempe...
- 01:13 PM Task #393: Identify Focal Regions
- The folks at NASA Ames suggested that scaling up to the globe will much easier to do if we process the data using the...
04/27/2012
- 11:05 AM Task #393: Identify Focal Regions
- Natalie is ready to help MODIS data assembly once we have decided this. Can we plan to talk about this on the Tuesda...
04/24/2012
- 02:00 PM Task #375: Assemble monthly mean MODIS LST values for the complete record (2000-2012) for Oregon
- I just noticed that the IRI data library [[http://iridl.ldeo.columbia.edu/SOURCES/.USGS/.LandDAAC/.MODIS/.1km/.8day/....
- 01:14 PM Task #393 (New): Identify Focal Regions
- We need to identify the small number (3-5?) of focal regions that we will use for further validation after we finish...
Also available in: Atom