added inputs/VegBank/verify/outputBien.log.url
inputs/input.Makefile: add: verify/: also svn:ignore *.log
added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments
bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable
planning/meetings/BIEN conference call availability.xlsx: updated
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)
schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names
bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan
fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).
bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name
schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names
schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource
fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables
schemas/public_.sql: query_view_regexp(): also match auxiliary tables
schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)
schemas/util.sql: added loop_ignore_errors()
schemas/public_.sql: added output_queries(prefix)
schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting
schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param
schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()
schemas/public_.sql: added public_validations.query_view_regexp()
bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()
bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists
lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir
fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts
bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
inputs/input.Makefile: install: also run validate/install
inputs/input.Makefile: added validate/install
bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match
bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database
lib/sh/util.sh: import_vars: documented that it's idempotent
/Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`
web/links/index.htm: updated to Firefox bookmarks: fixed broken links
bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)
/Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution
/Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import
/Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)
bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`
schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place
schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet
bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors
fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()
bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead
bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false
bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error
bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->false if error)
lib/sh/util.sh: added is_err()
lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations
bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed
fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1
fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1
fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)
bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level
fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands
fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error
/README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing
/README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload
fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files
web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: rsync: documented that this breaks use of ~ to reference the remote user's home dir
/README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape
bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly
/README.TXT: to back up the version history: use $HOME to make paths platform-independent
bugfix: exports/.rsync_filter.upload: *.csv: need to use `protect` rather than `hide` to prevent overwriting
fix: exports/.rsync_filter.upload: *.csv: don't allow test runs' exports to overwrite the backed up ones
fix: lib/sh/db.sh: psql(): removed debugging changes
bugfix: lib/sh/util.sh: highlight_log_msg(): when not can_highlight_log_msg, need to remove any surrounding formatting
fix: lib/sh/util.sh: die_error_hidden(): always log local vars at same log_level as echo_func
fix: *{.sh,run}: always log kw_params at same log_level as echo_func
lib/sh/util.sh: split_lines(): usage: matched up and synced different syntaxes
bugfix: lib/sh/util.sh: log_msg!(): split_lines does not support being invoked by wrapper; need to use `declare lines; wrapper "split_lines" str` instead
fix: lib/sh/util.sh: split_lines(): usage: documented different syntax for when using wrapper
bugfix: lib/sh/util.sh: die_error_hidden(): echo_func to assist debugging
bugfix: lib/sh/util.sh: split(): need to limit the effects of IFS to just the splitting, so it doesn't cause strange errors in other functions
bugfix: stderr2stdout(): fd 2 must be redirected back to fd 2, not log-filtered, in case there are other errors in addition to the benign error. this fixes a bug in pg_schema_exists(), where errors about the DB being down were not displayed because they were log-filtered out.
lib/sh/make.sh: set_make_vars: don't display make vars at verbosity 2 to avoid clutter
bugfix: lib/sh/make.sh: begin_target: don't echo_func twice
inputs/GBIF/_MySQL/.rsync_ignore: added GBIFPortalDB-*.data.sql.gz, because these are intermediate files
bugfix: /Makefile: $(pg_ctl-Darwin): need to call the command rather than echoing it, as is needed for the Linux version
bugfix: /Makefile: $(pg_ctl-Darwin): need to `cd /` because due to pg_ctl bug, current directory must be accessible by it
bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.
bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run
lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn
bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)
bin/in_place: diff: use --brief to avoid scanning the entire file for large files
bin/in_place: added $preserve_mtime flag
web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: Apache: documented that MultiViews is actually only broken for redirects with the filename "index"
web/.htaccess: for dirs, redirect to index.*: document it is actually just the filename "index" that MultiViews is broken for, other filenames work fine
bugfix: web/index.php: full directory index: only display if invoked as "vegpath.org/", not "vegpath.org/index.php"
bugfix: web/.htaccess: for dirs, redirect to index.*: added workaround for Ubuntu 14.04, which breaks MultiViews
/Makefile: postgres-Linux: updated to use the official version that comes with Ubuntu 14.04