lib/sh/util.sh: 2nd functions section: moved to 1st functions section

bugfix: lib/sh/util.sh: added workaround for bash bug where exit sometimes inxeplicably ignores $?

fix: lib/sh/util.sh: self_builtin: avoid $() so that $? isn't modified

lib/sh/util.sh: use new self_builtin

lib/sh/util.sh: added self_builtin

lib/sh/util.sh: pv(), pf(): moved to debugging section

bugfix: lib/sh/util.sh: stderr_matches(): also need to handle any filter error, such as caused by Ctrl+C

lib/sh/util.sh: stderr_matches(): echo_vars @PIPESTATUS_ to assist debugging

bugfix: lib/sh/util.sh: stderr2stdout(): use piped_cmd to ignore SIGPIPE since the output of this will piped to another command

lib/sh/util.sh: setup_log_fd(): $log_fd: use 3 (stdlog) since other scripts are likely to use this for logging as well

fix: lib/sh/util.sh: setup_log_fd(): fd_set_default(): use $log_fd instead of repeating the value of it

lib/sh/util.sh: die(): log at higher log_level, since this is logging code

lib/sh/util.sh: highlight_log_msg(): log at higher log_level, since this is logging code

bugfix: lib/runscripts/util.run: $subdirs: adjusted log_level now that echo_vars is one log_level lower

bugfix: lib/sh/util.sh: stderr_matches(): only set benign_error=1 if the matched error occurred

lib/sh/util.sh: ignore_e(): also set benign_error=1

fix: lib/sh/util.sh: prep_try alias: removed inaccurate comment

bugfix: lib/sh/util.sh: stdout2fd(): moved after redir() which it depends on

fix: lib/sh/util.sh: command(): moved `|| die_e` to command__exec so it would be properly indented under the echoed command

lib/sh/util.sh: verbosity_compat(): log at higher log_level because it's logging code

lib/sh/util.sh: $benign_error: log at higher log_level because it's logging code

lib/runscripts/util.run: $wrap_fn: log at higher log_level because it's startup code

lib/sh/util.sh: $top_* vars, $is_outermost: log at higher log_level because it's startup code

lib/sh/util.sh: $top_script: echo_vars this like the other $top_* vars

lib/sh/util.sh: .(): log at higher log_level because it's startup code

lib/sh/util.sh: is_dot_script(): run with higher log_level since this is run at the beginning of the script

lib/sh/util.sh, runscripts/util.run: set_paths(): run with higher log_level to hide all the paths that are set at the beginning of the script

lib/sh/util.sh: added log++ stub

lib/sh/util.sh: added log_local stub

lib/sh/util.sh: added log() stub so internal commands can use it

fix: lib/sh/util.sh: echo_vars(): log at same log_level as echo_func so kw_params are displayed along with positional params

fix: lib/sh/util.sh: rel_path(): log this internal command at a higher log_level so it's normally hidden

fix: lib/sh/util.sh: log_msg!(): log split_lines at a higher log_level so it's normally hidden

bugfix: lib/sh/util.sh: stderr_matches(): `log_local; log++` should apply to just stdout_contains() and part of stderr2stdout() rather than all of stderr_matches()

inputs/Madidi/_src/: set svn:ignore

added backups/vegbien.r13002.backup.md5, vegbien.r13160.backup.md5

backups/TNRS.backup.md5: updated

lib/sh/util.sh: stderr_matches(): run at higher log_level because error-handling internals should not be logged by default

bugfix: lib/sh/db.sh: mysql_ctl(): need to ignore errors if not running

bugfix: lib/sh/util.sh: stderr_matches(): handle any error: only ignore_e if the error exit status was associated with the matched error message

bugfix: lib/sh/util.sh: stderr_matches(): handle any error: need force-exit with rethrow_exit() because caller's test of return status disables errexit

lib/sh/util.sh: added rethrow_exit(), which exits even where errexit is disabled

bugfix: lib/sh/db.sh: mysql_snapshot(): need to run `mysql_ctl start` even if there is an error

lib/sh/db.sh: mysql_snapshot(): $to: default to $from.bak

lib/sh/db.sh: added mysql_snapshot()

lib/sh/db.sh: added mysql_ctl()

lib/sh/db.sh: pg_cmd(): updated to use new sudo()

lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions

fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable

lib/sh/util.sh: sudo alias: use function instead so this can be overridden

lib/sh/util.sh: added is_intern()

lib/sh/util.sh: is_callable(): use just $1 because multiple args are not applicable

lib/sh/util.sh: added is_world_executable()

lib/sh/util.sh: added has_perms()

lib/sh/util.sh: esc_args(): renamed to just esc() because this can also be used on a single value

lib/sh/util.sh: added is_extern()

lib/sh/util.sh: added sudo alias to alias-expand command

lib/sh/db.sh: pg_cmd(): $as_root: use $sudo

lib/sh/util.sh: added $sudo

lib/sh/util.sh: added cp alias

lib/sh/db.sh: removed no longer used pg_as_root(), which was buggy anyway. use `as_root=1 ...` instead.

lib/sh/db.sh: mysql_ANSI: fixed comment

added backups/users.sql.run

lib/sh/db.sh: pg_dump(): support dumping entire cluster, and cluster users

lib/sh/db.sh: pg_cmd(): added $as_root switch

fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: preserve datasets with ipr_specific = '', because they are actually redistributable, according to Brad (http://wiki.vegpath.org/2014-04-17_conference_call#conditions-of-use)

web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: added operator classes. added backups: filesystem-level backup, continuous archiving, WAL logging, etc. virtual collaboration: updated annotations.

lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format=plain: removed comment that this is the plain format, because this is now self-documenting

lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format: use the long form of the formats to make the code self-documenting

validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB

validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql

schemas/vegbien.ERD.mwb: regenerated exports

fix: lib/PostgreSQL-MySQL.csv: preserve schema assignments by translating `SET search_path` to `USE`

schemas/vegbien.ERD.mwb: regenerated exports

schemas/vegbien.ERD.mwb: added geoscrub, TNRS tables, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema)

schemas/Makefile: vegbien.sql: also include geoscrub, TNRS schemas, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema). this involves having a separate public_.sql file for restoring the public schema.

schemas/vegbien.ERD.mwb: regenerated exports

bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: updated filter condition to match output query

inputs/NY/run: `make inputs/NY/validate`: updated runtime (8 min, with added queries)

fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks

fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks

bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()

bugfix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13*: also need to include coordinate pairs which have one of their coordinates NULL, by using OR instead of AND

bugfix: inputs/NY/validations.sql: _specimens_13b_list_of_all_decimal_lat_long: matched column types to output query

bugfix: inputs/NY/validations.sql: _specimens_13a_list_of_all_verbatim_lat_long: matched column types to output query

inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: added breakdowns _specimens_13a_list_of_all_verbatim_lat_long, _specimens_13b_list_of_all_decimal_lat_long to help troubleshoot the diff

fix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: count lat/longs together instead of separately, because the DISTINCT is by coordinate pair, not individual coordinate value (which wouldn't make much sense)

bugfix: schemas/vegbien.sql: rm_output_queries(): need to account for the fact that util.truncated_prefixed_name_regexp() returns a whole-string regexp. this drops support for removing output queries with a particular group prefix, which we no longer use.

bugfix: schemas/vegbien.sql: rm_output_queries(): need to include relations whose names were truncated, as well

fix: schemas/vegbien.sql: public_validations schema comment: to remove a validations query so its columns can be changed: use rm_output_queries() rather than rm_query_view() because that also removes input queries

bugfix: schemas/util.sql: is_castable(): need to pass NULL through, for proper NULL propagation

fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: use new is_castable(), which is much more accurate than Brad's custom regexp for determining if something is numeric

inputs/NY/validations.-.util.sql: added util.is_castable() wrapper

schemas/util.sql: added is_castable()

schemas/util.sql: added try_cast()

schemas/util.sql: added util.cast(), which allows casting to an arbitrary type without eval()

bugfix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: DISTINCT: added coordsaccuracy_m

bugfix: schemas/vegbien.sql: coordinates_unique: added coordsaccuracy_m

fix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because the coordinates_unique unique constraint includes other columns as well, so there may be multiple instances of each lat/long

bugfix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to include both lat and long in the value to DISTINCT on