lib/sh/util.sh: highlight_msg(): auto-add padding around text if there is a background

lib/sh/util.sh: highlight_msg(): use $format itself as the $highlight boolean

lib/sh/util.sh: highlight_msg(): split apart the testing of $format and can_highlight_msg

lib/sh/util.sh: added has_bg()

bugfix: lib/sh/util.sh: highlight_msg(): need to reset any existing formatting before applying new formatting

lib/sh/util.sh: added mk_hint() and use it in log_hint!()

lib/sh/util.sh: bg_cmd(): also log the command being run

fix: lib/sh/util.sh: need `function` before functions that have an alias with the same name

lib/sh/util.sh: log!(): use new log:()

lib/sh/util.sh: added log:(), which sets an explicit log_level. this also simplifies log+().

lib/sh/util.sh: log+(): set log_level before PS4 so that the PS4 expr doesn't also need to add to log_level

lib/sh/util.sh: removed no longer needed log+ alias (which had been renamed from clog+)

lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)

: local setting of log_level: use log_local instead of relying on the log aliases, so that these aliases can instead be used for wrapping commands (the more common use case)

bugfix: lib/sh/util.sh: verbosity_compat alias: need to use `declare verbosity="$verbosity"` instead of `declare verbosity`, which would just clear $verbosity

bugfix: lib/sh/util.sh: verbosity_min alias: need to use `declare verbosity="$verbosity"` instead of log_local now that verbosity is not one of the vars changed by log++

lib/sh/util.sh: log+(): use easier-to-understand log_local instead of prefix-assignments to limit assignments to the invoked command

lib/sh/util.sh: log+(): use easier-to-understand log_local instead of prefix-assignments to limit assignments to the invoked command

: use clog instead of "log*"

bugfix: lib/sh/util.sh: log+(): removed spurious ; between setting of PS4 and log_level, which was causing erratic mismatches between PS4 and log_level. (the ; caused $PS4 to be set in the caller when invoked via one of the clog* aliases, rather than being passed as a command-specific env var.)

lib/sh/util.sh: $verbosity: stay constant at what the user set it to instead of changing in tandem with $log_level, to facilitate debugging verbosity/log_level-related issues

lib/sh/util.sh: log+(): usage: use aliases instead of ""-ed function names

added schemas/VegCore.ERD.pdf symlink for easy access

lib/sh/util.sh: log_err(): use red background for better visibility of errors, in the same way that lib/exc.py print_ex() does for column-based import

bugfix: lib/sh/util.sh: removed echo_func in functions used by log++, to avoid spurious highlighted output

lib/sh/util.sh: added missing clog+ alias

bugfix: lib/sh/util.sh: log_hint(): use the standard log_fd and log_info() format, not err_fd and log_err() format, for hint messages

fix: lib/sh/util.sh: log_msg!(): indent each line, not just the first

lib/sh/util.sh: added split_lines()

lib/sh/util.sh: log(): factored out helper function log_msg!()

fix: lib/sh/util.sh: highlight_msg(): bold instead of underlining because the underlining interferes with the readability of the commands

lib/sh/util.sh: highlight_msg(): allow turning off formatting w/ empty $format

fix: lib/sh/util.sh: log_err() calls: removed manual highlighting

lib/sh/util.sh: log_err(): highlight all error messages using highlight_msg()'s new $format

lib/sh/util.sh: highlight_msg(): support custom format

lib/sh/db.sh: pg_*_exists(): log the DB statements to check this at a higher log_level so that they don't clutter up the log output

lib/sh/util.sh: log(): highlight log_level 1 messages to stand out against other output, for easier debugging

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

bugfix: catch(): also need to support $1='' because this is a now a use case of ignore_e()

bugfix: lib/sh/util.sh: ignore_err_msg(): also need to ignore false exit status on no match

lib/sh/util.sh: stderr_matches(): moved prep_try/rethrow into the function itself so that callers don't have to wrap this function in a complex sequence of prep_try/rethrow statements

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

lib/sh/util.sh: stderr_matches(): moved prep_try/rethrow into the function itself so that callers don't have to wrap this function in a complex sequence of prep_try/rethrow statements

lib/sh/util.sh: added rethrow_exit alias

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.

fix: inputs/input.Makefile: removed no longer used special handling of XML inputs, support for which was never added to the Makefile. (bin/map, however, does support importing an XML file into a database.) this fixes a bug in XAL, which used to abort with an error but now just imports an empty table.

fix: inputs/input.Makefile: %/install: don't ignore errors if table does not exist, to ensure a proper errexit. this is now possible because every dir that this target is being run on should be a data dir. (Source/ used to be a metadata-only dir.)

bugfix: inputs/input.Makefile: $(cleanup): need `set -o pipefail`

inputs/VegBank/run: `rm=1 import()`: updated runtime (1 h)

inputs/VegBank/taxon_observation.**/test.xml.ref: updated inserted row count

inputs/VegBank/projectcontributor_/test.xml.ref: updated inserted row count

bugfix: schemas/util.sql: is_constant(util.col_ref): updated to include standard newline at beginning of comment (applies to newly-imported staging tables)

bugfix: inputs/VegBank/import_order.txt: added missing project, needed to trigger the staging table renaming for the project table

inputs/VegBank/run: documented `rm=1 import()` runtime (>1.5 h)

inputs/VegBank/run: documented `datasrc_make sql/install` runtime (25 min)

inputs/MO/Specimen/test.xml.ref: updated, which adds dateCollected mappings

inputs/WIN/Specimen/test.xml.ref: updated to map.csv, which has eventDate->dateCollected

inputs/VegBank/plantconcept_/create.sql: updated runtime (25 min, ~same)

lib/sh/make.sh: begin_target: echo all targets to facilitate debugging without needing the verbose stack trace mode

bugfix: lib/sh/make.sh: echo_target: don't include filename/line #, since this is not for the stack trace mode

lib/sh/make.sh: added echo_target

*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`

lib/runscripts/util.run: runscript template: added sample make target, using new make target template

lib/sh/make.sh: added make target template

inputs/VegBank/plot/postprocess.sql: remove institutions that we have direct data for: CVS: updated runtime (same)

lib/sh/make.sh: added begin_target alias

lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)

bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try

bugfix: lib/sh/util.sh: rethrow*: only `return` if $e is actually nonzero, because rethrow is now being used as a catch-all in situations where there might not be an error

lib/sh/util.sh: prep_try: initialize $e to 0 to simplify error-handling coding

stderr_matches(): wrapper caller usage: added alternative usage when using `||`

lib/sh/util.sh: stderr_matches(): wrapper caller usage: documented usage for a negated condition (ie. prefixed w/ !)

lib/sh/util.sh: stderr_matches(): usage: split into wrapper usage and wrapper caller usage for clarity

fix: *{.sh,run}: stderr_matches() wrappers: usage: added `rethrow`

fix: *{.sh,run}: stderr_matches() wrappers: usage: added `rethrow`

fix: lib/sh/util.sh: stderr_matches(): usage: `rethrow` must be called right after stderr_matches(), to avoid calling running other commands if there is an error

fix: lib/sh/util.sh: stderr_matches(): when using $ignore_e, also set benign_error=1 to suppress the highlighting of the error

bugfix: lib/sh/db.sh: pg_schema_exists(): need to ignore benign error exit status from the "cannot create temporary relation in non-temporary schema" error

lib/sh/util.sh: stderr_matches(): supporting ignoring any benign error exit status associated with the error message being tested for

lib/sh/util.sh: stderr_matches(): usage: documented where any ignore_e statement would go

bugfix: lib/sh/util.sh: stderr_matches(): can't use `try` because this clears the exit status, which is needed for @PIPESTATUS to work. to support this, also need to avoid errexiting since @PIPESTATUS will be used instead.

lib/sh/util.sh: added dp(), which debug-prints a message

bugfix: inputs/VegBank/plot/postprocess.sql: use CVS.plot_ instead because that has the renamed staging table columns, and is compatible with auto-renaming of the SQL script columns

inputs/CVS/plot_/postprocess.sql: add unique constraint on locationName (analogous to the unique constraint in plot), for use by inputs/VegBank/plot/postprocess.sql in removing inter-datasource duplication

fix: schemas/util.sql: explain2notice_msg(): don't include EXPLAIN output for simple, single-value queries, to avoid cluttering up the log output

schemas/util.sql: added fold_explain_msg()

bugfix: bin/repl: only use excluded_prefix_re/excluded_suffix_re in text mode (used in renaming columns in SQL scripts), to prevent the special coding for column renames from also affecting regular regexp/word replacements

inputs/VegBank/taxon_observation.**/test.xml.ref: updated inserted row count

inputs/run: postprocess(): documented runtime (30 min)

bugfix: inputs/input.Makefile: %/postprocess.sql: don't perform replacements using map.csv, because map.csv is not idempotent. this functionality was only there to facilitate switching to new-style import, which is now largely done. (the remaining datasources NVS, SALVIAS, TEAM contain only 1 postprocess.sql: inputs/SALVIAS/projects/postprocess.sql (`st inputs/{NVS,SALVIAS,TEAM}/*/postprocess.sql`).)

bugfix: bin/repl: text mode: also don't match if it's part of a '-'-separated identifier

bugfix: bin/repl: text mode: also don't match if it's a word in a sentence

bugfix: bin/repl: text mode: turned off the suffix matching, because there are cases where a mapping adds a suffix which would cause the same replacement to be performed repeatedly

inputs/input.Makefile: %/postprocess.sql: always run this, not just if the associated map spreadsheets change, to avoid needing to `touch` them to cause %/postprocess.sql to run

bin/repl: text mode: exclude prefixes that should not cause replacement, to avoid doubling leading *

fix: inputs/*/*/postprocess.sql: un-doubled *

bugfix: inputs/input.Makefile: %/postprocess.sql: also need to apply renames from mappings/VegCore.thesaurus.csv, as these have been applied to map.csv

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

bin/repl: text mode: also match w/ suffix (eg. _verbatim)