validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: use our staging tables instead of the BIEN2 MySQL staging tables

validation/aggregating/specimens/**.sql: removed trailing whitespace, using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#translate-to-Postgres

archived validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.sql

added validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql, copied from qualitative_validations_source_db_NYBG.sql

validation/aggregating/specimens/qualitative_validations_specimens.sql: added ; at end of `CREATE OR REPLACE VIEW` statements

inputs/run: postprocess(): documented runtime on vegbiendev (1 h)

validation/aggregating/specimens/qualitative_validations_specimens.sql: removed input-query-specific comments

validation/aggregating/specimens/qualitative_validations_specimens.sql: reworded rowcount check comments to apply to the output queries

validation/aggregating/specimens/qualitative_validations_specimens.sql: shortened view names to fit within the 63-char limit without truncation

/README.TXT: `make inputs/{NVS,SALVIAS,TEAM}/test`: updated runtime (1 min)

schemas/vegbien.sql: specimenreplicate.institution_id: renamed to duplicate_institutions_sourcelist_id, as decided in the conference calls (wiki.vegpath.org/2014-03-13_conference_call#schema-changes-2)

inputs/run: postprocess(): updated runtime (25 min)

fix: validation/aggregating/specimens/qualitative_validations_specimens.sql: changed "Full inner join" to "Full outer join" because a FULL JOIN is a type of outer join, not inner join

/README.TXT: calls to `inputs/run postprocess`: direct user to refer to inputs/run for this, so the runtime doesn't have to be updated in multiple places

inputs/run: postprocess(): updated runtime (20 min)

/README.TXT: Schema changes: added steps to update staging table column names on the local machine and vegbiendev

fix: schemas/VegCore/mk_derived: added `EOF` at end to avoid (benign) "here-document delimited by end-of-file" warnings on Linux

mappings/VegCore.htm: regenerated from wiki: rename specimenHolderInstitutions to specimen_duplicate_institutions, as decided in the 2014-03-13 conference call (wiki.vegpath.org/2014-03-13_conference_call#schema-changes-2). note that most schema changes (such as this one) involve mappings changes, which are handled automatically by `inputs/run postprocess; yes|make inputs/{NVS,SALVIAS,TEAM}/test`.

bugfix: lib/runscripts/table.run: schema/make calls: need to use `make schema` instead because old-style datasources don't have a top-level runscript (the absence of this identifies them as old-style so inputs/input.Makefile works correctly)

/README.TXT: Maintenance: VegCore data dictionary: `make inputs/{NVS,SALVIAS,TEAM}/test`: recorded runtime (30 s)

/README.TXT: Maintenance: VegCore data dictionary: `make inputs/{NVS,SALVIAS,TEAM}/test`: prepended `time` to enable obtaining the runtime

/README.TXT: Maintenance: VegCore data dictionary: `inputs/run postprocess`: updated runtime (20 min)

fix: schemas/util.sql: trim(): by default, cascadingly drop dependent columns so that they don't prevent trim() from succeeding. note that this requires the dependent columns to then be manually re-created.

bugfix: inputs/GBIF/table.run: switched to using lib/runscripts/table.run instead of mysql.table.run because some subdirs (Source/) need the regular table.run to work properly. mysql.table.run should instead be used directly by subdirs that use the MySQL install.

bugfix: lib/sh/util.sh: DON'T do `shopt -s lastpipe` because this causes a segfault on Linux in stderr_matches(). (it also isn't supported on Mac.) use @PIPESTATUS instead. note that we do not currently need lastpipe, since we use @PIPESTATUS (which actually provides more functionality for our purposes).

fix: lib/sh/util.sh: echo_func(): file/line #: display with regular color because the lighter color actually draws attention to rather than away from the faded text

lib/sh/util.sh: added plain()

inputs/XAL/Specimen/test.xml.ref: updated for sample data.csv, which contains the columns as a CSV. this fixes a bug where a map.csv must be used on a table that contains the same set of columns (ie. not one with no columns if there are any mappings).

bugfix: lib/sql_io.py: put_table(): is_literals: `return sql.value(cur): need to use sql.value_or_none() instead to support multi-row functions, such as _split() used in specimens data`

fix: inputs/input.Makefile: don't treat *.xml as data files since these are not currently supported

lib/runscripts/util.run: on_exit(): documented that users can also override gateway()/fallback() to perform other commands (or no commands) after the script is read

bugfix: lib/sh/db.sh: pg_table_exists(): need ! to negate boolean result

fix: lib/runscripts/table.run: table_make_install(): need to inform the user when it skips installing a table, because this is often unexpected

fix: lib/runscripts/util.run: run_args_cmd(): need to indent the output of the target that it's running

lib/runscripts/table.run: removed no longer used datasrc_make_install()

fix: lib/sh/util.sh: fade(): use medium gray instead of light gray because it fades on white and black backgrounds

lib/sh/util.sh: echo_func(): fade the file/line # to avoid distracting from the function call in the default log output

lib/sh/util.sh: added fade()

lib/sh/util.sh: highlight_msg(): renamed to highlight_log_msg() to clarify that this contains log++-specific functionality

lib/sh/util.sh: moved terminal formatting commands to own section

lib/sh/util.sh: highlight_msg(): moved formatting code into separate format() function

lib/sh/util.sh: dp(): renamed to ps() to corresponding with pv/pf

lib/sh/make.sh: echo_target: use `log-- echo_func`, which now puts the target name first but also provides much-needed indentation

lib/sh/util.sh: echo_func(): put file/line # after function call instead of before so the function name is listed first

lib/sh/util.sh: echo_func(): usage: removed no longer used/implemented minor=1 switch. use log++ instead.

lib/runscripts/datasrc_dir.run: import(): use new schema/make, schema/rm

lib/runscripts/table.run: load_data(): use the much simpler `schema/make` run target, rather than outsourcing to the legacy Makefile via the convoluted datasrc_make_install()/table_make_install()

lib/runscripts/datasrc_dir.run: added schema/rm(), schema/make()

lib/sh/util.sh: ignore_err_msg(): usage: added $ignore_e param from stderr_matches()

lib/runscripts/table.run: psql: always include ; at end of statement

fix: lib/sh/db.sh: pg_cmd(): hide PGPASSWORD at the normal verbosity so that the value of it doesn't appear in any log files

lib/sh/util.sh: log_hint(): renamed to log_err_hint() for clarity, because this applies only to hints for errors

bugfix: lib/sh/util.sh: log_hint!(): use log_err instead of log_info because hints as used here are attached to (possibly benign) errors. for other uses, use mk_hint().

fix: lib/sh/util.sh: highlight_msg(): don't ' '-pad already-formatted text

lib/sh/util.sh: manual terminal escape sequences: use highlight_msg() instead

lib/sh/util.sh: highlight_msg(): auto-add padding around text if there is a background

lib/sh/util.sh: highlight_msg(): use $format itself as the $highlight boolean

lib/sh/util.sh: highlight_msg(): split apart the testing of $format and can_highlight_msg

lib/sh/util.sh: added has_bg()

bugfix: lib/sh/util.sh: highlight_msg(): need to reset any existing formatting before applying new formatting

lib/sh/util.sh: added mk_hint() and use it in log_hint!()

lib/sh/util.sh: bg_cmd(): also log the command being run

fix: lib/sh/util.sh: need `function` before functions that have an alias with the same name

lib/sh/util.sh: log!(): use new log:()

lib/sh/util.sh: added log:(), which sets an explicit log_level. this also simplifies log+().

lib/sh/util.sh: log+(): set log_level before PS4 so that the PS4 expr doesn't also need to add to log_level

lib/sh/util.sh: removed no longer needed log+ alias (which had been renamed from clog+)

lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)

: local setting of log_level: use log_local instead of relying on the log aliases, so that these aliases can instead be used for wrapping commands (the more common use case)

bugfix: lib/sh/util.sh: verbosity_compat alias: need to use `declare verbosity="$verbosity"` instead of `declare verbosity`, which would just clear $verbosity

bugfix: lib/sh/util.sh: verbosity_min alias: need to use `declare verbosity="$verbosity"` instead of log_local now that verbosity is not one of the vars changed by log++

lib/sh/util.sh: log+(): use easier-to-understand log_local instead of prefix-assignments to limit assignments to the invoked command

lib/sh/util.sh: log+(): use easier-to-understand log_local instead of prefix-assignments to limit assignments to the invoked command

: use clog instead of "log*"

bugfix: lib/sh/util.sh: log+(): removed spurious ; between setting of PS4 and log_level, which was causing erratic mismatches between PS4 and log_level. (the ; caused $PS4 to be set in the caller when invoked via one of the clog* aliases, rather than being passed as a command-specific env var.)

lib/sh/util.sh: $verbosity: stay constant at what the user set it to instead of changing in tandem with $log_level, to facilitate debugging verbosity/log_level-related issues

lib/sh/util.sh: log+(): usage: use aliases instead of ""-ed function names

added schemas/VegCore.ERD.pdf symlink for easy access

lib/sh/util.sh: log_err(): use red background for better visibility of errors, in the same way that lib/exc.py print_ex() does for column-based import

bugfix: lib/sh/util.sh: removed echo_func in functions used by log++, to avoid spurious highlighted output

lib/sh/util.sh: added missing clog+ alias

bugfix: lib/sh/util.sh: log_hint(): use the standard log_fd and log_info() format, not err_fd and log_err() format, for hint messages

fix: lib/sh/util.sh: log_msg!(): indent each line, not just the first

lib/sh/util.sh: added split_lines()

lib/sh/util.sh: log(): factored out helper function log_msg!()

fix: lib/sh/util.sh: highlight_msg(): bold instead of underlining because the underlining interferes with the readability of the commands

lib/sh/util.sh: highlight_msg(): allow turning off formatting w/ empty $format

fix: lib/sh/util.sh: log_err() calls: removed manual highlighting

lib/sh/util.sh: log_err(): highlight all error messages using highlight_msg()'s new $format

lib/sh/util.sh: highlight_msg(): support custom format

lib/sh/db.sh: pg_*_exists(): log the DB statements to check this at a higher log_level so that they don't clutter up the log output

lib/sh/util.sh: log(): highlight log_level 1 messages to stand out against other output, for easier debugging

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

bugfix: catch(): also need to support $1='' because this is a now a use case of ignore_e()

bugfix: lib/sh/util.sh: ignore_err_msg(): also need to ignore false exit status on no match

lib/sh/util.sh: stderr_matches(): moved prep_try/rethrow into the function itself so that callers don't have to wrap this function in a complex sequence of prep_try/rethrow statements

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

lib/sh/util.sh: stderr_matches(): moved prep_try/rethrow into the function itself so that callers don't have to wrap this function in a complex sequence of prep_try/rethrow statements

lib/sh/util.sh: added rethrow_exit alias

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.