bugfix: inputs/input.Makefile: install: only run this for datasource dirs

inputs/input.Makefile: install: use ./run's install target for clarity

lib/runscripts/import.run: added install() target

bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system

bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)

inputs/input.Makefile: added $(datasrc_schema_exists)

lib/runscripts/in_datasrc_dir.run: use new local.run

added lib/runscripts/local.run

web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links

web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page

added validation/aggregating/bien3_validation_sequence.xlsx from Brad

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)

bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able

added inputs/VegBank/verify/outputBien.log.url

inputs/input.Makefile: add: verify/: also svn:ignore *.log

added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments

bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable

planning/meetings/BIEN conference call availability.xlsx: updated

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)

schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names

bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan

fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name

schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names

schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource

fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables

schemas/public_.sql: query_view_regexp(): also match auxiliary tables

schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)

schemas/util.sql: added loop_ignore_errors()

schemas/public_.sql: added output_queries(prefix)

schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting

schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param

schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len

schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len

schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()

schemas/public_.sql: added public_validations.query_view_regexp()

bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()

bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir

fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts

bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install

bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install

inputs/input.Makefile: install: also run validate/install

inputs/input.Makefile: added validate/install

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match

bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database

lib/sh/util.sh: import_vars: documented that it's idempotent

/Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`

web/links/index.htm: updated to Firefox bookmarks: fixed broken links

bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)

/Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution

/Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import

/Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)

bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`

schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place

schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet

bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors

fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()

bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead

bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false

bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error

bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->false if error)

lib/sh/util.sh: added is_err()

lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1

fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1

fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)

bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level

fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands

fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error

/README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing

/README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload

fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files

web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: rsync: documented that this breaks use of ~ to reference the remote user's home dir

/README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape

bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly

/README.TXT: to back up the version history: use $HOME to make paths platform-independent

bugfix: exports/.rsync_filter.upload: *.csv: need to use `protect` rather than `hide` to prevent overwriting

fix: exports/.rsync_filter.upload: *.csv: don't allow test runs' exports to overwrite the backed up ones

fix: lib/sh/db.sh: psql(): removed debugging changes

bugfix: lib/sh/util.sh: highlight_log_msg(): when not can_highlight_log_msg, need to remove any surrounding formatting

fix: lib/sh/util.sh: die_error_hidden(): always log local vars at same log_level as echo_func

fix: *{.sh,run}: always log kw_params at same log_level as echo_func

lib/sh/util.sh: split_lines(): usage: matched up and synced different syntaxes

bugfix: lib/sh/util.sh: log_msg!(): split_lines does not support being invoked by wrapper; need to use `declare lines; wrapper "split_lines" str` instead

fix: lib/sh/util.sh: split_lines(): usage: documented different syntax for when using wrapper

bugfix: lib/sh/util.sh: die_error_hidden(): echo_func to assist debugging

bugfix: lib/sh/util.sh: split(): need to limit the effects of IFS to just the splitting, so it doesn't cause strange errors in other functions

bugfix: stderr2stdout(): fd 2 must be redirected back to fd 2, not log-filtered, in case there are other errors in addition to the benign error. this fixes a bug in pg_schema_exists(), where errors about the DB being down were not displayed because they were log-filtered out.

lib/sh/make.sh: set_make_vars: don't display make vars at verbosity 2 to avoid clutter

bugfix: lib/sh/make.sh: begin_target: don't echo_func twice

inputs/GBIF/_MySQL/.rsync_ignore: added GBIFPortalDB-*.data.sql.gz, because these are intermediate files