Activity
From 06/19/2014 to 07/18/2014
07/18/2014
- 11:40 PM Revision 14125: fix: inputs/.TNRS/schema.sql: taxon_match: cluster the table so the rows are always exported in the same order, even when an UPDATE statement is used to run triggers
- 11:28 PM Revision 14124: bugfix: schemas/util.sql: cluster_once(): need to use just the name of the index, without the schema prefix
- 05:00 PM Revision 14123: schemas/util.sql: cluster_once(): use simpler IS DISTINCT FROM instead of array !=
- 03:18 PM Revision 14122: inputs/.TNRS/schema.sql: added __accepted_{genus,specific_epithet} derived column, using the steps at wiki.vegpath.org/Postgres_queries#maintaining-derived-column-formulas-across-column-renames
- 02:01 AM Revision 14121: web/username_prefix.php: remove padding used to visually separate elements: documented that this also removes a leading __ that indicates an internal field
07/17/2014
- 07:18 PM Revision 14120: web/links/index.htm: updated to Firefox bookmarks: Quick links: added VegBIEN data dictionary spreadsheet, TNRS data dictionary.
- 06:15 PM Revision 14119: fix: web/username_prefix.php: transposes: changed recommended syntax to _y~x@url so that _y~ lines up vertically with [y_] paths
- 06:11 PM Revision 14118: web/username_prefix.php: removed special category comments syntax because this is now handled by transposes and insertion comments
- 06:10 PM Revision 14117: web/username_prefix.php: added support for transposes ( y~x_@url -> x_y@url , Y~x@url -> xY@url ), which can be used to put the category in a column name first without duplicating the category
- 05:35 PM Revision 14116: web/username_prefix.php: removed linewraps code because Google spreadsheets no longer supports wrapping on a - , and because a shorter syntax for category comments is now available
- 05:32 PM Revision 14115: web/username_prefix.php: category comments: moved _-_ outside [] in comparison example since it will be deleted separately
- 05:27 PM Revision 14114: bugfix: web/username_prefix.php: category comments: need /__/ around regexp
- 05:24 PM Revision 14113: web/username_prefix.php: category comments: added support for a shorter syntax than [x_-_]y@url : x+y@url -> y@url
- 04:14 PM Revision 14112: schemas/vegbien.ERD.mwb: updated
- 04:13 PM Revision 14111: fix: inputs/.TNRS/schema.sql: taxon_match: added back * prefixes on TNRS-namespace column names
- 03:34 PM Revision 14110: inputs/.TNRS/schema.sql: MatchedTaxon: taxon_best_match: don't alias to `s` since this is no longer a nested select
- 03:32 PM Revision 14109: inputs/.TNRS/schema.sql: MatchedTaxon: nested select: use just taxon_best_match since this no longer performs renamings
- 03:27 PM Revision 14108: inputs/.TNRS/schema.sql: MatchedTaxon: renamed output columns to match input columns, so that the nested select would not be performing any renamings
- 03:05 PM Revision 14107: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 03:01 PM Revision 14106: fix: inputs/.TNRS/schema.sql: reordered columns to match taxon_match
- 02:38 PM Revision 14105: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 01:02 PM Revision 14104: fix: inputs/.TNRS/schema.sql: added back Family_matched, which had gotten merged with Name_matched_accepted_family when TNRS temporarily stopped populating Name_matched_accepted_family
- 12:32 PM Revision 14103: inputs/.TNRS/schema.sql: MatchedTaxon: taxonomicStatus: moved outside the inner SELECT so that the inner SELECT would consist solely of renamings
- 12:12 PM Revision 14102: inputs/.TNRS/schema.sql: MatchedTaxon: removed unneeded "Name_matched." prefix on source-specific names (only the * is needed because there is only one table)
- 12:10 PM Revision 14101: bugfix: schemas/util.sql: view_is_subset(): auto-updatable views that are *not* plain subsets: include ones with WHERE clauses
- 12:03 PM Revision 14100: bugfix: schemas/util.sql: view_is_subset(): also need to handle auto-updatable views that are *not* plain subsets (there are some of these)
- 09:05 AM Revision 14099: bin/import_all: hidden_srcs(): removed `by_col=1` because these should be done in the same mode as the main datasources
07/16/2014
- 07:55 PM Revision 14098: schemas/public_.sql: removed no longer used threatened_taxonlabel. use iucn_red_list instead.
- 07:39 PM Revision 14097: fix: schemas/public_.sql: analytical_stem_view: don't use threatened_taxonlabel, which was never populated correctly. instead, this can eventually be rewritten to use new iucn_red_list.
- 07:13 PM Revision 14096: bin/make_analytical_db: removed threatened_taxonlabel because this is now handled by iucn_red_list
- 07:12 PM Revision 14095: bin/make_analytical_db: added iucn_red_list_view_modify()
- 06:48 PM Revision 14094: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 06:35 PM Revision 14093: bin/make_analytical_db: merged mk_table and mk_analytical_table since they now do the same thing
- 06:34 PM Revision 14092: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 05:05 PM Revision 14091: /README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it
- 05:04 PM Revision 14090: inputs/SALVIAS/Source/header.csv: updated
- 03:50 PM Revision 14089: bugfix: bin/with_all: isset(): need to use `&>/dev/null` instead of `>&-`, etc because closing an fd causes declare to return false
- 03:31 PM Revision 14088: bugfix: bin/with_all, import_all: don't disown processes because they should be auto-killed if the shell is (disown was only needed before we used screen)
- 03:24 PM Revision 14087: bugfix: Full database import: need nested shell to prevent errexit from closing the window
- 01:24 PM Revision 14086: bugfix: schemas/public_.sql: analytical_stem_view_modify(): added columns: need to include COMMENT statements
- 01:21 PM Revision 14085: bugfix: schemas/util.sql: comment(element oid): need `objsubid = 0` filter so this doesn't use comments for any other objsubids
- 12:53 PM Revision 14084: schemas/public_.sql: added iucn_red_list_view_modify() and use it in iucn_red_list_view's "after updating this" instructions
- 12:47 PM Revision 14083: bugfix: schemas/util.sql: rematerialize_view(): also need to handle the target table's dependent views
- 12:38 PM Revision 14082: bugfix: schemas/util.sql: mk_drop_from_create(): need to use `IF EXISTS` so that the resulting statement also works if the target object does not yet exist
- 12:24 PM Revision 14081: schemas/util.sql: mk_drop_from_create(): also support `SELECT util.drop_*()` statements, in addition to DROP
- 11:37 AM Revision 14080: inputs/.TNRS/Source/test.xml.ref: updated
07/15/2014
- 11:14 PM Revision 14079: bugfix: /README.TXT: logs glob: don't unintentionally match . or ..
- 10:43 PM Revision 14078: inputs/VegBank/stemlocation_/test.xml.ref: updated
- 10:41 PM Revision 14077: /README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)
- 10:07 PM Revision 14076: inputs/TEX/Specimen2/test.xml.ref: updated
- 09:35 PM Revision 14075: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: updated
- 05:50 PM Task #887 (Rejected): fix disk space leak that fills the disk and crashes the import
- 05:48 PM Task #902 (Resolved): fix bug that causes joining on the wrong columns in the import
- 05:47 PM Task #905 (Resolved): narrow down the cause of the import bug (incorrect join columns and disk space leak)
- bug fixed in r14074
- 05:44 PM Revision 14074: bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add *new* entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.
- 04:49 PM Revision 14073: bin/import_all: delete_logs(): documented that `trap EXIT` doesn't run until *shell* exit
- 04:48 PM Revision 14072: bin/import_all: delete_logs(): print when this happens, so it can be verified that it's happening properly
- 04:32 PM Revision 14071: bugfix: bin/import_all: need to run delete_logs manually because `trap EXIT` doesn't run until bg cmds done
- 04:28 PM Revision 14070: bin/import_all: delete_logs: moved testing of whether to delete logs to delete_logs() so that delete_logs() can be run regardless of the $delete_logs setting
- 03:58 PM Revision 14069: bugfix: bin/import_all: delete_logs(): also need to match log filenames when n=""
- 03:11 PM Revision 14068: bugfix: /Makefile: postgres_restart-*: argument to echo always needs quotes
- 03:05 PM Revision 14067: bugfix: /Makefile: $(pg_ctl-*): tell user that if it doesn't work, need to run `rm $(macPostgresDir)/postmaster.pid` and retry
- 02:57 PM Revision 14066: /Makefile: $(pg_ctl-Darwin): documented the pg_ctl version of this command in case `brew services` doesn't work
- 02:39 PM Revision 14065: bugfix: bin/with_all: isset(): need to use `>&- 2>&-` because &> does not work with - as the dest
- 01:19 PM Revision 14064: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: determine if the revision works: use the in# of the query instead of waiting an arbitrary time, because this is more accurate in determining whether the import is successful
- 12:52 PM Revision 14063: config/VirtualBox_VMs/vegbiendev/README.TXT: regressions that occurred during the revision range: named these "concurrent regressions"
- 12:45 PM Revision 14062: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to update system time so wait times can be calculated correctly
- 12:22 PM Revision 14061: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented that the patches are specifically for regressions that occurred during the revision range, not all bugs that were fixed in that range
- 12:14 PM Revision 14060: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: waiting for query: increased time to min. 10 min because some revisions take longer
- 12:09 PM Revision 14059: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: added step to type y and press ENTER
- 12:07 PM Revision 14058: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to determine if the revision works, and then stop the import
- 11:27 AM Revision 14057: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12994
- 11:09 AM Revision 14056: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use for loop to avoid needing to repeat the command
- 11:05 AM Revision 14055: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12760
- 10:54 AM Revision 14054: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: --fuzz apparently needs to be 3 even though we have only 2 lines of context
- 10:52 AM Revision 14053: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use --fuzz to allow patching even when surrounding lines have changed
- 10:44 AM Revision 14052: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: sorted by revision #
- 10:43 AM Revision 14051: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12865
- 10:33 AM Revision 14050: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: patch: need to use --force to avoid prompting user
- 10:16 AM Revision 14049: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn revert`: need to use `--depth infinity` when reverting more than a file
- 10:14 AM Revision 14048: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: use `patch` instead of `svn merge` because `svn merge` sometimes inexplicably doens't apply patches
- 09:43 AM Revision 14047: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: `svn revert` all changes instead of specific files, to support bugfixes to other files
- 09:42 AM Revision 14046: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: use `svn merge` instead of `sed --in-place` because this can be used for more complex changes
- 09:41 AM Revision 14045: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: documented revision
- 09:20 AM Revision 14044: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to apply r12872 bugfix to lib/sh/util.sh because it occurred after r12025
- 09:10 AM Revision 14043: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make` commands: use `yes|` to avoid needing to confirm prompts
- 09:08 AM Revision 14042: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make mk_db ...`: use `yes|` to avoid needing to confirm prompts
- 08:56 AM Revision 14041: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: to test a different revision: added step that if any of the commands after `svn up` are broken, try a different revision
- 08:46 AM Revision 14040: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make db`: need to run `schemas/reinstall` manually because older versions of the codebase just used `schemas/install`
07/14/2014
- 11:30 PM Revision 14039: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented the range from within the revision should be chosen
- 09:19 PM Revision 14038: fix: inputs/.IRMNG/*/: added _no_import because this datasource is only used as a lookup table and does not need to be imported/scrubbed
- 09:18 PM Revision 14037: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn up`: documented runtime (1 min)
- 08:24 PM Revision 14036: bugfix: added nodes/_no_import, because the NCBI import doesn't work due to a Postgres 9.3 bug (#859). _no_import used to not be needed because we didn't import the hidden sources.
- 08:05 PM Revision 14035: fix: bin/with_all: removed debug statements
- 05:19 PM Revision 14034: bugfix: bin/with_all: testing if @inputs is set: `"${inputs+isset}"` syntax doesn't work for empty arrays, so need to use `declare -p` instead
- 04:46 PM Revision 14033: bugfix: to set up the VM: to use a database backup: also need to `make rm_initial_public` because initdb creates a non-empty DB
- 04:42 PM Revision 14032: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to test a different revision
- 04:40 PM Revision 14031: /Makefile: db: use reinstall instead of install to allow running this on an existing DB
- 02:49 PM Revision 14030: config/VirtualBox_VMs/vegbiendev/README.TXT: `sudo apt-get dist-upgrade`: documented runtime (12 min)
- 02:47 PM Revision 14029: config/VirtualBox_VMs/vegbiendev/README.TXT: updated VM snapshot names
- 02:44 PM Revision 14028: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:34 PM Revision 14027: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:29 PM Revision 14026: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to install system updates
- 01:23 PM Revision 14025: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: documented runtime (50 min)
- 01:07 PM Revision 14024: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to run the import
- 12:24 PM Revision 14023: config/VirtualBox_VMs/vegbiendev/README.TXT: to set up the VM: added steps to move vegbiendev.vdi to the internal hard drive
- 12:19 PM Revision 14022: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to start the VM
- 12:16 PM Revision 14021: config/VirtualBox_VMs/vegbiendev/README.TXT: moved "set up the VM" steps to separate section
- 12:06 PM Revision 14020: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added steps to create a booted snapshot
- 12:01 PM Revision 14019: config/VirtualBox_VMs/vegbiendev/README.TXT: moved creation of pg_restore snapshot to be part of just the database backup section
- 11:52 AM Revision 14018: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: documented runtime (15 min)
- 11:51 AM Revision 14017: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: time the operation
- 11:32 AM Revision 14016: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added name for VM snapshot after this step
- 11:31 AM Revision 14015: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to run as DB superuser to restore py_util
- 11:24 AM Revision 14014: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (25 min)
- 11:24 AM Revision 14013: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to create snapshot before running pg_restore
- 10:55 AM Revision 14012: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to configure the VM before booting
- 10:53 AM Revision 14011: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions for selecting the initial snapshot and booting the VM
- 10:36 AM Revision 14010: config/VirtualBox_VMs/vegbiendev/README.TXT: perform long-running operations inside `screen`
- 10:35 AM Revision 14009: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to use a database backup: only run mk_db, not db, to avoid restoring schemas that are part of the backup
- 09:33 AM Revision 14008: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: schemas must be in dependency order
- 09:21 AM Revision 14007: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: don't restore analytical_db schema because we don't use it and it depends on public, which we're not restoring
- 09:16 AM Revision 14006: web/links/index.htm: updated to Firefox bookmarks: WebEx: added steps to join a test meeting. pg_restore: documented that you need to *create* the schema before you restore into it, or else you will get a cryptic error message that it's using pg_catalog instead. added Homebrew links, including instructions for setting psycopg2 up with it. added PostGIS links. username URLs: documented how to enable these in Internet Explorer. TWiki: documented what to do when moving the TWiki install dir.
- 09:06 AM Revision 14005: bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag
- 08:36 AM Revision 14004: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to *create* the schema before you restore into it, to avoid cryptic error messages
- 08:33 AM Revision 14003: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
- 08:18 AM Revision 14002: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
07/13/2014
- 11:50 PM Revision 14001: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: need to run `make db` to restore users, which are not part of the backup
- 11:40 PM Revision 14000: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (35 min)
07/11/2014
- 04:54 PM Revision 13999: config/VirtualBox_VMs/vegbiendev/README.TXT: turning off of non-ARIZ inputs: removed metadata datasources because these have been renamed to start with ".", and so aren't included in */
- 04:48 PM Revision 13998: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: pg_restore: need to use for loop because pg_restore doesn't support multiple --schema options (it just uses the last one)
- 04:35 PM Revision 13997: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to restore the necessary schemas from the DB
- 04:32 PM Revision 13996: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: rsync: documented runtime (13 min)
- 04:11 PM Revision 13995: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to fetch the database backup via rsync
- 04:04 PM Revision 13994: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: moved steps to enable the user to access VirtualBox shared folders to separate section because VirtualBox shared folders currently don't work, likely because of a bug in VirtualBox Guest Additions when installed on something other than a clean Linux install
- 03:54 PM Revision 13993: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added steps to enable the user to access VirtualBox shared folders
- 03:06 PM Revision 13992: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added section for database backup approach, as alternative to clean database approach
- 10:32 AM Revision 13991: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: use ".." instead of - to indicate a range, to avoid this being confused for a minus sign
- 10:26 AM Revision 13990: bugfix: /Makefile: postgres_restart-Darwin: need to have the user run the command outside of make because it doesn't appear to do anything when run inside make (this is the same problem as on Linux)
- 10:25 AM Revision 13989: /Makefile: $(pg_ctl-Darwin): use simpler `brew services` instead of `launchctl __/homebrew.mxcl.postgresql.plist`
- 10:03 AM Revision 13988: bugfix: bin/stop_imports: also need to include `bin/after_import`
- 09:56 AM Revision 13987: inputs/.TNRS/Source/map.csv: datasetURL: updated to indicate that this uses the TNRS dev server (the actual URL is private)
- 09:46 AM Revision 13986: fix: inputs/VASCAN/Source/map.csv: observationGranularity should not be specimen because this datasource contains only taxa
- 09:13 AM Revision 13985: bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off
- 08:45 AM Revision 13984: bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd
- 08:41 AM Revision 13983: bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell
- 08:38 AM Revision 13982: bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window
- 08:32 AM Revision 13981: bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)
- 08:25 AM Revision 13980: bugfix: bin/import_all: need to always use log files for background processes
- 08:12 AM Revision 13979: fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.
- 08:06 AM Revision 13978: fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted
- 07:40 AM Revision 13977: bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources
- 07:39 AM Revision 13976: bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded
- 07:34 AM Revision 13975: fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.
07/10/2014
- 07:25 PM Revision 13974: fix: bin/import_all: updated for new metadata datasource names (see issue #940)
- 07:25 PM Revision 13973: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)
- 06:41 PM Task #940 (Resolved): resequence datasources in dependency order
- 04:22 PM Task #940 (Resolved): resequence datasources in dependency order
- * this enables bootstrapping the DB from a VM backup
steps
# put all [[VegBIEN_contents#datasources|metadata da... - 06:32 PM Revision 13972: fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)
- 06:04 PM Revision 13971: fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)
- 04:59 PM Revision 13970: bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.
- 03:52 PM Revision 13969: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done
- 03:37 PM Revision 13968: config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)
- 03:33 PM Revision 13967: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure
- 03:32 PM Revision 13966: added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import
- 12:17 PM Revision 13965: inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.
- 10:57 AM Revision 13964: fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates
- 09:45 AM Revision 13963: validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column
- 08:41 AM Revision 13962: schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on *each* function call
- 08:39 AM Revision 13961: schemas/util.sql: added map_words()
- 08:16 AM Revision 13960: schemas/util.sql: added date_from_spanish()
- 08:13 AM Revision 13959: schemas/util.sql: added replace_words()
- 08:09 AM Revision 13958: schemas/util.sql: added words(text)
- 08:02 AM Revision 13957: schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these
- 07:20 AM Revision 13956: schemas/util.sql: added spanish_date_words()
- 07:19 AM Revision 13955: schemas/util.sql: added `replacement` type
- 04:09 AM Revision 13954: bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()
- 04:07 AM Revision 13953: schemas/util.sql: added mk_set_comments(regclass)
- 04:06 AM Revision 13952: schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions
- 04:01 AM Revision 13951: schemas/util.sql: added mk_set_col_comments(regclass)
- 03:58 AM Revision 13950: schemas/util.sql: added cols(regclass)
- 03:56 AM Revision 13949: schemas/util.sql: added mk_set_comment(col_ref)
- 03:18 AM Revision 13948: added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets
07/09/2014
- 02:50 PM Revision 13947: fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.
- 02:45 PM Revision 13946: bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import
- 02:34 PM Revision 13945: inputs/FIA/taxon_observation.**/test.xml.ref: updated
- 02:34 PM Revision 13944: inputs/FIA/TREE/test.xml.ref: updated
- 02:29 PM Revision 13943: inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated
- 02:13 PM Revision 13942: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location
- 02:12 PM Revision 13941: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location
- 12:14 PM Revision 13940: fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate
- 12:05 PM Revision 13939: fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)
- 09:40 AM Revision 13938: schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)
- 09:38 AM Revision 13937: schemas/util.sql: added mk_set_comment(col col_ref, comment text)
- 09:37 AM Revision 13936: schemas/util.sql: added sql(col_ref)
- 09:35 AM Revision 13935: schemas/util.sql: added mk_set_comment(on_ text, comment text)
- 07:53 AM Revision 13934: schemas/util.sql: added comment(col_ref)
- 07:50 AM Revision 13933: schemas/util.sql: added col_num(col_ref)
- 07:03 AM Revision 13932: removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run
- 07:03 AM Revision 13931: bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run
- 06:22 AM Revision 13930: schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this
07/08/2014
- 04:17 PM Revision 13929: exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)
- 03:23 PM Revision 13928: bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn
07/07/2014
- 07:33 AM Revision 13927: schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.
- 07:26 AM Revision 13926: fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row
- 07:04 AM Revision 13925: bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled
- 06:55 AM Revision 13924: bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used
- 06:51 AM Revision 13923: bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:48 AM Revision 13922: bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:43 AM Revision 13921: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn
- 06:37 AM Revision 13920: schemas/public_.sql: added in_iucn_red_list()
- 06:36 AM Revision 13919: schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view
- 06:22 AM Revision 13918: bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names
- 06:17 AM Revision 13917: schemas/public_.sql: added iucn_red_list
- 06:14 AM Revision 13916: schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred
- 06:02 AM Revision 13915: schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized
- 06:01 AM Revision 13914: schemas/public_.sql: added iucn_red_list
- 05:44 AM Revision 13913: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added collection_type (using source__observation_type())
- 05:36 AM Revision 13912: schemas/public_.sql: added source__observation_type()
- 05:04 AM Revision 13911: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added accepted_subspecies, and removed accepted_taxon_name_with_author which had been substituting for it
- 04:50 AM Revision 13910: schemas/public_.sql: added subspecies() wrapper
- 04:38 AM Revision 13909: schemas/util.sql: added subspecies()
- 04:10 AM Revision 13908: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only coordinates in South America
- 03:55 AM Revision 13907: schemas/util.sql: added south_america(), in_south_america()
- 03:37 AM Revision 13906: fix: schemas/util.sql: OPERATOR ~@(geocoord, postgis.geometry): renamed to @ because this is not an approximate comparison for geometry
- 03:35 AM Revision 13905: fix: schemas/util.sql: contained_within_approx(geocoord, postgis.geometry): renamed to contained_within__no_dateline(__) because this is not an approximate comparison for geometry
- 03:30 AM Revision 13904: schemas/util.sql: lat_long_in_new_world(): renamed to just in_new_world() because the lat/long is implied by the param type
- 03:08 AM Revision 13903: schemas/util.sql: lat_long_in_new_world(): take a geocoord param instead of separate lat/long params
- 03:00 AM Revision 13902: backups/TNRS.backup.md5: updated
07/06/2014
- 11:08 PM Revision 13901: schemas/util.sql: added contained_within_approx(geocoord, geometry) and corresponding OPERATOR ~@(geocoord, geometry)
- 10:39 PM Revision 13900: schemas/util.sql: added OPERATOR ~@(geocoord, geography)
- 10:23 PM Revision 13899: schemas/util.sql: lat_long_in_new_world(): use new contained_within_approx(geocoord, geography)
- 10:17 PM Revision 13898: schemas/util.sql: added contained_within_approx(geocoord, postgis.geography), which enables specifying just `(lat, long)` without the ::util.geocoord type specifier
- 04:04 PM Revision 13897: schemas/util.sql: OPERATOR @(postgis.geography, postgis.geography): renamed to ~@ because it's approximate
- 03:52 PM Revision 13896: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 03:50 PM Revision 13895: schemas/util.sql: added contained_within__no_dateline(geometry, geometry) and corresponding operator @
- 02:51 PM Revision 13894: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 05:49 AM Revision 13893: schemas/util.sql: geometry(geocoord): documented that it is not possible to create a cast for this, as a bug in pg_dump prevents the cast from being exported, even when no export filters are applied
07/05/2014
- 12:40 PM Revision 13892: schemas/util.sql: point(geocoord): renamed to geometry(geocoord) since this is now a cast
- 12:03 PM Revision 13891: schemas/util.sql: point(): return geometry instead of geography to support using points with geometry arithmetic
- 11:24 AM Revision 13890: schemas/util.sql: point(): take a single util.geocoord param instead of separate lat/long
- 10:45 AM Revision 13889: schemas/util.sql: added geocoord type
07/04/2014
- 08:29 PM Revision 13888: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only *georeferenced* occurrences (lat/long NOT NULL)
- 08:25 PM Revision 13887: schemas/util.sql: bounding_box(): use bounding_box__no_dateline() to construct the postgis.geometry object
- 08:23 PM Revision 13886: schemas/util.sql: added bounding_box__no_dateline(), which is more accurate than util.bounding_box() (latitude lines will be straight), but geocoordinate wraparound is not supported
- 08:14 PM Revision 13885: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added functional traits that we have 1st-class columns for (dbh_cm, height_m)
- 12:29 PM Revision 13884: added exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run
- 11:12 AM Revision 13883: inputs/publishable datasources.xlsx: updated
- 08:45 AM Revision 13882: inputs/publishable datasources.xlsx: updated
07/03/2014
- 08:13 PM Revision 13881: inputs/.TNRS/schema.sql: matchedFamily: just use Name_matched_accepted_family, because TNRS has now been reloaded so that the names that were missing this have it populated
- 09:59 AM Revision 13880: planning/meetings/BIEN conference call availability.xlsx: updated
- 03:47 AM Task #917: TNRS: Instructions for new version with TPL
- a single "-" creates strikethrough formatting, so you have to use "--" instead
- 12:34 AM Revision 13879: inputs/.TNRS/schema.sql: taxon_match: taxon_match__valid_match: replaced with taxon_best_match__valid_match, which also applies taxon_best_match's filters, since taxon_match is now accessed through taxon_best_match
- 12:17 AM Revision 13878: fix: inputs/.TNRS/schema.sql: MatchedTaxon: use taxon_best_match instead of taxon_match because this should provide only one match per taxon
07/02/2014
- 02:17 AM Revision 13877: bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter
- 02:00 AM Revision 13876: /README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi
06/27/2014
- 03:15 PM Task #917: TNRS: Instructions for new version with TPL
- test update now that watcher notifications have been fixed
06/26/2014
- 06:30 PM Revision 13875: inputs/.TNRS/schema.sql: added taxon_best_match view
- 06:12 PM Revision 13874: inputs/.TNRS/schema.sql: taxon_match: added taxon_match__one_selected_match unique index
- 05:06 PM Revision 13873: backups/TNRS.backup.md5: updated
- 09:49 AM Revision 13872: planning/meetings/BIEN conference call availability.xlsx: updated
- 09:25 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications again after e-mail address fix...
- 09:23 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications...
- 05:41 AM Revision 13871: inputs/.TNRS/schema.sql: taxon_match__fill(): split into separate DECLARE blocks for each field for clarity
- 04:58 AM Revision 13870: inputs/.TNRS/data.sql: refreshed
- 04:57 AM Revision 13869: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:33 AM Revision 13868: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:28 AM Revision 13867: inputs/.TNRS/data.sql: refreshed
- 04:11 AM Revision 13866: inputs/.TNRS/schema.sql: taxon_match: insert names via taxon_match_input auto-updatable view instead of directly into taxon_match, to allow the taxon_match columns to be renamed while still supporting inserts using the TNRS column names
- 02:43 AM Revision 13865: fix: schemas/Makefile: vegbien.sql: also need to update inputs/.TNRS/data.sql, since its contents change along with this
- 02:41 AM Revision 13864: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:39 AM Revision 13863: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:38 AM Revision 13862: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:14 AM Revision 13861: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
06/25/2014
- 07:54 PM Revision 13860: lib/tnrs.py: dirty: documented that this actually used to be on in the web app (see r9910, 2013-6-18), but does not appear to be needed (the source_sorting bug alluded to in r9910 is not fixed by enabling the dirty setting)
- 07:48 PM Task #917: TNRS: Instructions for new version with TPL
- multiple @Selected@ names bug fixed in r13855
- 07:46 PM Revision 13859: lib/tnrs.py: requests: also debug-print request URL
- 07:44 PM Revision 13858: lib/tnrs.py: Download: include the same debug info as do_request()
- 07:41 PM Revision 13857: lib/tnrs.py: do_request(): also debug-print request headers
- 07:39 PM Revision 13856: lib/tnrs.py: download_request_template: dirty: documented why this must be off
- 07:36 PM Revision 13855: bugfix: lib/tnrs.py: download_request_template: fixed bug where multiple names were being marked as Selected, because dirty was incorrectly set to true unlike in the web app
- 07:30 PM Revision 13854: added derived/TNRS/web_app/protocol/
- 04:28 PM Revision 13853: inputs/.TNRS/data.sql: refreshed
- 04:25 PM Revision 13852: bugfix: inputs/.TNRS/schema.sql: taxon_name_is_safe(): need to use `NOT (_ = ANY(_)) instead of `_ != ANY(_)`, because the != operator is applied to each element
- 03:47 PM Revision 13851: inputs/.TNRS/schema.sql: tnrs: renamed to tnrs_match to distinguish it from other TNRS-related tables
- 03:33 PM Revision 13850: inputs/.TNRS/schema.sql: tnrs: renamed to tnrs_match to distinguish it from other TNRS-related tables
- 02:38 PM Revision 13849: fix: lib/phpPgAdmin.login.php.diff: use relative file path rather than the path the file was at when the patch was created
- 02:34 PM Revision 13848: /Makefile, lib/phpPgAdmin.login.php.diff: public_ user: added auto-filled password so that users would not be confused as to what to type in the password field
- 02:15 PM Revision 13847: added schemas/VegCore/phpMyAdmin/libraries/plugins/auth/AuthenticationCookie.class.php.diff
- 01:19 PM Task #938 (New): Enquist's taxon validation
- From Brian Enquist:
I can start by checking the accuracy of the fields, and data for a random subset of plots. To ... - 10:39 AM Revision 13846: inputs/.TNRS/schema.sql: `taxon_scrub.scrubbed_unique_taxon_name.*`: added to-modify instructions
- 10:36 AM Revision 13845: inputs/.TNRS/schema.sql: *_modify(): merged these into the "to modify" instructions in the corresponding views, because there is no need to create a separate *_modify() function for every view now that their definitions are all the same
- 10:21 AM Revision 13844: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
- 05:15 AM Revision 13843: schemas/public_.sql: analytical_stem_view and related views: updated COMMENTs from data dictionary spreadsheet, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#copy-data-dictionary-definitions-to-database
- 04:11 AM Revision 13842: schemas/public_.sql: ran analytical_stem_view_modify(), which transfers the column COMMENTs from analytical_stem_view to analytical_stem
- 04:04 AM Revision 13841: bugfix: schemas/util.sql: view_def_to_orig(): need to handle cases when list of cols from the same table is not an expanded * expression
- 03:59 AM Revision 13840: schemas/util.sql: added view_is_subset(view_def text)
- 03:58 AM Revision 13839: schemas/util.sql: added view_is_automatically_updatable(view_def text)
- 03:23 AM Revision 13838: bugfix: schemas/util.sql: show_create_view(): use the overridden version of pg_get_viewdef(), which supports expanded * expressions. this was possibly being used already whenever util happened to be in the search_path.
06/24/2014
- 05:03 PM Revision 13837: bugfix: schemas/public_.sql: analytical_stem_view_modify(): updated to new analytical_stem_view column names
- 05:01 PM Revision 13836: fix: schemas/public_.sql: analytical_stem_view derived and related views: applied data dictionary renamings, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#apply-data-dictionary-renamings-to-database but with the current columns of analytical_stem as the left-hand column
- 04:36 PM Revision 13835: bugfix: schemas/public_.sql: analytical_stem_view_modify(): updated to new analytical_stem_view column names
- 04:34 PM Revision 13834: schemas/public_.sql: analytical_stem_view: applied data dictionary renamings, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#apply-data-dictionary-renamings-to-database
- 03:27 PM Revision 13833: lib/tnrs.py: source_sorting (Constrain by Source): documented the different behavior for this in each match mode (all-matches and best-match)
06/20/2014
06/19/2014
- 04:13 PM Revision 13831: bugfix: web/.phpPgAdmin/.htaccess: : handlers: use `chain` to only run them when the RewriteRule they apply to is run
- 04:11 PM Revision 13830: web/.phpPgAdmin/.htaccess: schema : handler: removed `env=subject:schema` because that has already been set
- 04:08 PM Revision 13829: web/.phpPgAdmin/.htaccess: tables: added : suffix handler
- 04:06 PM Revision 13828: bugfix: web/.phpPgAdmin/.htaccess: view : match: moved to right after `view=` because it only applies to views, not functions
- 03:58 PM Revision 13827: web/.phpPgAdmin/.htaccess: schema: page: indicate by prefixing with : to avoid needing to explicitly list all possible pages
- 03:54 PM Revision 13826: web/.phpPgAdmin/.htaccess: table: action: indicate by prefixing with : to avoid needing to explicitly list all possible actions
- 11:13 AM Revision 13825: exports/2014-6-12.Jeff_Ott.climatic_range_determinants.csv.run: updated export_() runtime (35 min) and rowcount (23 million)
- 11:00 AM Task #916: Taxon name validation: VegBank
- most bugs fixed, just needs TNRS bug workaround
- 02:33 AM Revision 13824: fix: /README.TXT: Full database import: disk space: removed instructions to rerun the import if the disk space gets used up, because this is actually a bug and is *not* generally "fixed just by rerunning the import"
- 02:27 AM Revision 13823: schemas/util.sql: lat_long_in_new_world(): use function rather than operator+search_path to allow inlining, which enables util.new_world() to only be evaluated once
- 02:24 AM Revision 13822: bugfix: schemas/util.sql: operator @(postgis.geography, postgis.geography): must use wrapper function because st_coveredby() needs postgis to be in the search_path
- 01:57 AM Revision 13821: fix: lib/runscripts/extract.run: export_(): explicitly prevent files from becoming web-accessible, to protect against an incorrect umask in the calling process
- 01:45 AM Revision 13820: fix: schemas/util.sql: point(): hide benign "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" notices
Also available in: Atom