Project

General

Profile

Statistics
| Revision:
Name Size Revision Age Author Comment
  _archive 1598 over 12 years Aaron Marcuse-Kubitza Moved _archive/tapir2flatClient/trunk/client/ t...
  analysis 3076 about 12 years Aaron Marcuse-Kubitza Added top-level analysis dir for range modeling
  backups 10267 almost 11 years Aaron Marcuse-Kubitza backups/Makefile: %.backup/restore: documented ...
  bin 10348 almost 11 years Aaron Marcuse-Kubitza bugfix: bin/repl: text mode: repurpose this to ...
  config 7801 over 11 years Aaron Marcuse-Kubitza root Makefile: VegBIEN DB: mk_db: Added command...
  exports 9928 about 11 years Aaron Marcuse-Kubitza added exports/_archive/
  inputs 10380 almost 11 years Aaron Marcuse-Kubitza inputs/BRIT/specimen_flat/: translated single-c...
  lib 10372 almost 11 years Aaron Marcuse-Kubitza lib/runscripts/table.run: postprocess(): added ...
  mappings 10289 almost 11 years Aaron Marcuse-Kubitza mappings/VegCore.htm: regenerated from wiki. Sp...
  planning 10311 almost 11 years Aaron Marcuse-Kubitza planning/timeline/timeline.2013.xls: moved Indi...
  schemas 10378 almost 11 years Aaron Marcuse-Kubitza schemas/util.sql: schema comment: added note th...
  web 10306 almost 11 years Aaron Marcuse-Kubitza web/links/index.htm: updated to Firefox bookmar...
.htaccess 326 Bytes 8771 about 11 years Aaron Marcuse-Kubitza /.htaccess: use canonical URL without symlinks
.rsync_filter.upload 33 Bytes 10042 about 11 years Aaron Marcuse-Kubitza /.rsync_ignore: temp files: hide them on upload...
.rsync_ignore 12 Bytes 10042 about 11 years Aaron Marcuse-Kubitza /.rsync_ignore: temp files: hide them on upload...
Makefile 12.6 KB 10223 almost 11 years Aaron Marcuse-Kubitza /Makefile: mysql-Linux: also install mysql-work...
README.TXT 23.3 KB 10379 almost 11 years Aaron Marcuse-Kubitza /README.TXT: Maintenance: added instructions fo...
fix_perms 97 Bytes 7560 over 11 years Aaron Marcuse-Kubitza Added root fix_perms
map 1001 Bytes 6949 over 11 years Aaron Marcuse-Kubitza vegbien_dest: Changed default $prefix to "", so...
new_terms.csv 38.1 KB 7222 over 11 years Aaron Marcuse-Kubitza new_terms.csv: Regenerated
run 433 Bytes 9916 about 11 years Aaron Marcuse-Kubitza /run: geoscrub_input/make(): documented runtime...
unmapped_terms.csv 13.1 KB 7201 over 11 years Aaron Marcuse-Kubitza **/new_terms.csv, **/unmapped_terms.csv: Regene...

Latest revisions

# Date Author Comment
10380 07/24/2013 08:18 AM Aaron Marcuse-Kubitza

inputs/BRIT/specimen_flat/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "translate single-column filters to postprocessing derived columns"

10379 07/20/2013 05:25 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: added instructions for what to do if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down (sometimes displaying a Not found error)

10378 07/20/2013 05:21 AM Aaron Marcuse-Kubitza

schemas/util.sql: schema comment: added note that IMMUTABLE SQL-language functions should never be declared STRICT, because this prevents them from being inlined. inlining can create a significant speed improvement (7x+), by avoiding function calls and enabling additional constant folding.

10377 07/20/2013 05:09 AM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/postprocess.sql: map_nulls() derived cols: documented total runtime (7.5 min on vegbiendev)

10376 07/20/2013 05:07 AM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/postprocess.sql: map_nulls() derived cols: updated runtimes for map_nulls() inlining, which created a speed improvement of 7x for the numeric columns and 2.5x for the text columns (292563.362->41929.772 ms and 83640.424->35690.797 ms, respectively). note that the map_nulls__coord__*() calls could be optimized further by combining the successive map_nulls() calls into one, with the hstores merged.

10375 07/20/2013 04:37 AM Aaron Marcuse-Kubitza

schemas/util.sql: map_nulls(): documented that inputs/REMIB/Specimen/postprocess.sql > country also shows that inlining is now happening properly. note that the speed improvement due to inlining is not as much, %wise, when the values util._map() is run on are long strings instead of the short strings used in the initial profiling. this is because a greater % of the time is spent in system functions such as hstore>text, which are not affected by the inlining because they are run either way.

10374 07/20/2013 04:18 AM Aaron Marcuse-Kubitza

schemas/util.sql: map_nulls(): use new nulls_map(). proper inlining (i.e. same runtime before and after change) has been verified with the following profiling query:
SELECT util.map_nulls(array[1, 2, 3]::text[], v) FROM unnest(array_fill(1, array100000)) f (v)

10373 07/20/2013 04:05 AM Aaron Marcuse-Kubitza

schemas/util.sql: added nulls_map(), for use with _map()

10372 07/20/2013 03:39 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: postprocess(): added remake action that calls trim_table()

10371 07/20/2013 03:37 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added trim_table(), which calls util.trim(regclass, regclass)

View all revisions | View revisions

Also available in: Atom