schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it

schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name

schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side

schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import

/README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function

/README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this

bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac

fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files

/README.TXT: Full database import: for test import, added step to turn off DB backup

fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell

/README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)

fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell

fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)

bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries

bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists

bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts

bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource

fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run

bugfix: inputs/input.Makefile: install: only run this for datasource dirs

inputs/input.Makefile: install: use ./run's install target for clarity

lib/runscripts/import.run: added install() target

bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system

bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)

inputs/input.Makefile: added $(datasrc_schema_exists)

lib/runscripts/in_datasrc_dir.run: use new local.run

added lib/runscripts/local.run

web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links

web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page

added validation/aggregating/bien3_validation_sequence.xlsx from Brad

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)

bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able

added inputs/VegBank/verify/outputBien.log.url

inputs/input.Makefile: add: verify/: also svn:ignore *.log

added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments

bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable

planning/meetings/BIEN conference call availability.xlsx: updated

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)

schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names

bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan

fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name

schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names

schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource

fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables

schemas/public_.sql: query_view_regexp(): also match auxiliary tables

schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)

schemas/util.sql: added loop_ignore_errors()

schemas/public_.sql: added output_queries(prefix)

schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting

schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param

schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len

schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len

schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()

schemas/public_.sql: added public_validations.query_view_regexp()

bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()

bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir

fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts

bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install

bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install

inputs/input.Makefile: install: also run validate/install

inputs/input.Makefile: added validate/install

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match

bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database

lib/sh/util.sh: import_vars: documented that it's idempotent

/Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`

web/links/index.htm: updated to Firefox bookmarks: fixed broken links

bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)

/Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution

/Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import

/Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)

bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`

schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place

schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet

bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors

fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()

bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead

bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false

bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error

bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->false if error)

lib/sh/util.sh: added is_err()

lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1

fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1

fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)

bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level

fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands

fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error

/README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing

/README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload