lib/runscripts/util.run: runscript template: added sample make target, using new make target template

lib/sh/make.sh: added make target template

inputs/VegBank/plot/postprocess.sql: remove institutions that we have direct data for: CVS: updated runtime (same)

lib/sh/make.sh: added begin_target alias

lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)

bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try

bugfix: lib/sh/util.sh: rethrow*: only `return` if $e is actually nonzero, because rethrow is now being used as a catch-all in situations where there might not be an error

lib/sh/util.sh: prep_try: initialize $e to 0 to simplify error-handling coding

stderr_matches(): wrapper caller usage: added alternative usage when using `||`

lib/sh/util.sh: stderr_matches(): wrapper caller usage: documented usage for a negated condition (ie. prefixed w/ !)

lib/sh/util.sh: stderr_matches(): usage: split into wrapper usage and wrapper caller usage for clarity

fix: *{.sh,run}: stderr_matches() wrappers: usage: added `rethrow`

fix: *{.sh,run}: stderr_matches() wrappers: usage: added `rethrow`

fix: lib/sh/util.sh: stderr_matches(): usage: `rethrow` must be called right after stderr_matches(), to avoid calling running other commands if there is an error

fix: lib/sh/util.sh: stderr_matches(): when using $ignore_e, also set benign_error=1 to suppress the highlighting of the error

bugfix: lib/sh/db.sh: pg_schema_exists(): need to ignore benign error exit status from the "cannot create temporary relation in non-temporary schema" error

lib/sh/util.sh: stderr_matches(): supporting ignoring any benign error exit status associated with the error message being tested for

lib/sh/util.sh: stderr_matches(): usage: documented where any ignore_e statement would go

bugfix: lib/sh/util.sh: stderr_matches(): can't use `try` because this clears the exit status, which is needed for @PIPESTATUS to work. to support this, also need to avoid errexiting since @PIPESTATUS will be used instead.

lib/sh/util.sh: added dp(), which debug-prints a message

bugfix: inputs/VegBank/plot/postprocess.sql: use CVS.plot_ instead because that has the renamed staging table columns, and is compatible with auto-renaming of the SQL script columns

inputs/CVS/plot_/postprocess.sql: add unique constraint on locationName (analogous to the unique constraint in plot), for use by inputs/VegBank/plot/postprocess.sql in removing inter-datasource duplication

fix: schemas/util.sql: explain2notice_msg(): don't include EXPLAIN output for simple, single-value queries, to avoid cluttering up the log output

schemas/util.sql: added fold_explain_msg()

bugfix: bin/repl: only use excluded_prefix_re/excluded_suffix_re in text mode (used in renaming columns in SQL scripts), to prevent the special coding for column renames from also affecting regular regexp/word replacements

inputs/VegBank/taxon_observation.**/test.xml.ref: updated inserted row count

inputs/run: postprocess(): documented runtime (30 min)

bugfix: inputs/input.Makefile: %/postprocess.sql: don't perform replacements using map.csv, because map.csv is not idempotent. this functionality was only there to facilitate switching to new-style import, which is now largely done. (the remaining datasources NVS, SALVIAS, TEAM contain only 1 postprocess.sql: inputs/SALVIAS/projects/postprocess.sql (`st inputs/{NVS,SALVIAS,TEAM}/*/postprocess.sql`).)

bugfix: bin/repl: text mode: also don't match if it's part of a '-'-separated identifier

bugfix: bin/repl: text mode: also don't match if it's a word in a sentence

bugfix: bin/repl: text mode: turned off the suffix matching, because there are cases where a mapping adds a suffix which would cause the same replacement to be performed repeatedly

inputs/input.Makefile: %/postprocess.sql: always run this, not just if the associated map spreadsheets change, to avoid needing to `touch` them to cause %/postprocess.sql to run

bin/repl: text mode: exclude prefixes that should not cause replacement, to avoid doubling leading *

fix: inputs/*/*/postprocess.sql: un-doubled *

bugfix: inputs/input.Makefile: %/postprocess.sql: also need to apply renames from mappings/VegCore.thesaurus.csv, as these have been applied to map.csv

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

bin/repl: text mode: also match w/ suffix (eg. _verbatim)

bugfix: /README.TXT: Maintenance: VegCore data dictionary: apply new data dict mappings: need to use postprocess rather than import runscript target, so that the command also works on an svn checkout without the flat files (the flat files are not needed for the staging table renaming)

lib/sh/db.sh: psql(): $verbose_ok: renamed to $bypass_ok for clarity, because this applies only to the `--output /dev/fd/41` bypass (which when not possible, requires turning off verbose output

fix: lib/sh/db.sh: psql(): added $output_data switch analogous to what mysql() has. this causes query results of eg. void-returning functions to be correctly filtered by the logging mechanism, rather than output to stdout.

fix: lib/sh/db.sh: psql(): verbosity=0 (errors only) mode: use `SET client_min_messages = WARNING;` instead of NOTICE to hide verbose messages within psql as well

lib/sh/db.sh: psql(): replaced `test "$verbose_ok" && can_log` with bool var $verbose_

fix: lib/sh/db.sh: psql(): $verbose_: renamed to $verbose_ok for clarity

fix: lib/sh/util.sh: stdout_contains(): add another pipe_delay because the `grep` statement was sometimes getting printed before its filtered output

bugfix: schemas/util.sql: set_col_types(): need to COALESCE the executed SQL to '' because util.eval() does not support NULL (and shouldn't, because this indicates a missing COALESCE in constructing the statement)

schemas/util.sql: set_col_types(): use simpler util.eval() instead of manual EXECUTE/util.debug_print_sql()

schemas/util.sql: set_col_types(): use string_agg() instead of array_to_string(ARRAY) for clarity

bugfix: lib/sh/util.sh: die_error_hidden(): min verbosity to display error should not be hardcoded

lib/sh/db.sh: psql(): "to see error details" msg: use new die_error_hidden()

lib/sh/util.sh: added die_error_hidden()

lib/sh/db.sh: psql(): "to see error details" msg: use new log_hint()

lib/sh/util.sh: added log_hint(), whose msg is only displayed if not a benign error

bugfix: lib/sh/db.sh: psql(): "to see error details" msg: also don't print it for benign errors ($benign_error)

schemas/util.sql: added mk_not_null()

lib/sh/db.sh: psql(): on error, display message describing how to see error details (prepend `vb=2` to the command)

bugfix: lib/sh/util.sh: log_err(): don't override verbosity manually, as this will not set log_level or PS4. instead, use new log! , which sets these correctly.

lib/sh/util.sh: added log! , which force-displays next log message

lib/sh/util.sh: save_e: made it idempotent so that it also works if save_e was already called

lib/sh/util.sh: save_e: made it idempotent so that it also works if save_e was already called

lib/sh/util.sh: rethrow: documented why can't use `(exit "$e")` (bash bug that prevents errexit)

bugfix: /README.TXT: Maintenance: VegCore data dictionary: apply new data dict mappings: need to use import rather than mappings runscript target, to rename the staging tables

bugfix: /README.TXT: Maintenance: VegCore data dictionary: also need to apply new data dict mappings on vegbiendev

fix: /README.TXT: Maintenance: VegCore data dictionary: added steps to apply the new data dictionary mappings to the datasource mappings and staging tables

bugfix: lib/runscripts/util.run: $auto_ignore: need to unexport it so don't pass this to invoked scripts except through fwd()

added inputs/run, which runs all the inputs' runscripts using the new auto-forwarding

bugfix: lib/runscripts/util.run: auto_fwd's fallback() must be set after auto_ignore's fallback() to overwrite it (auto_ignore should only apply if an error would otherwise have been generated by the fallback)

lib/runscripts/util.run: fwd(): support subdirs that don't contain a runscript, so that the default value of @subdirs will work in most cases

lib/runscripts/util.run: fwd(): set default @subdirs (`{.,}*/`)

lib/sh/util.sh: added enter_top_dir and use it in in_top_dir

fix: lib/sh/util.sh: commands run inside $(...): need to run with log++ so that these aren't normally debug-printed

lib/sh/util.sh: added pv(), which debug-prints var(s)

lib/sh/util.sh: added wildcard.()

lib/sh/util.sh: added wildcard/()

lib/sh/util.sh: added esc_args()

web/links/index.htm: updated to Firefox bookmarks: Google Drive: listed bugs that make it very difficult to use (the need to re-download all files when reconnecting a client to an account). added recommendation not to use it (unstable).

removed unused inputs/table.run. inputs/*/table.run include lib/runscripts/table.run directly.

lib/runscripts/datasrc_dir.run: removed postprocess(), which now does the same thing its auto-forwarded equivalent would

lib/runscripts/datasrc_dir.run: removed separate @table_subdirs, because the table-only targets can now safely be invoked on all subdirs, being auto-ignored in subdirs that don't support them

lib/runscripts/util.run: fwd(): enable $auto_ignore so that each subdir doesn't have to have a definition for the forwarded target

lib/runscripts/util.run: added $auto_ignore switch, which causes fallback() not to generate an error that a non-existant target doesn't exist

lib/runscripts/datasrc_dir.run: use new fwd_self alias

lib/runscripts/util.run: added fwd_self alias

lib/runscripts/datasrc_dir.run: enable $auto_fwd, to create the functionality of lib/forwarding.Makefile's `%` target

lib/runscripts/util.run: added $auto_fwd switch

bugfix: lib/runscripts/util.run: gateway(): need to use is_callable() rather than func_exists() to check whether the target exists, because external commands (eg. echo) are supported as targets, too

lib/sh/util.sh: added is_callable()

lib/runscripts/util.run: support custom handlers for all targets (gateway()) as well as targets w/o function (fallback())

lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names

lib/sh/util.sh: stderr_matches(): inline the stderr_matches alias to avoid needing to quote stderr_matches as "stderr_matches" in the most common use case (with pattern as a prefix env var)

bugfix: lib/sh/util.sh: stderr_matches(): when passing `pattern=...` as a prefix env var, must be invoked as `"stderr_matches"` to avoid the env var applying to the prep_try portion of the stderr_matches alias

added schemas/VegCore/Brad_Boyle/bien3_data_provenance_use_cases.docx* from e-mail from Brad

schemas/vegbien.sql: _plots_08_list_of_plots_which_use_percent_cover, _plots_15_pct_cover_of_each_verb_taxon_in_each_plot_in_each_pro: reran with fixes, which removes the incorrectly auto-added copies columns. (they were only able to be auto-added because the tables had no rows.)

bugfix: drop_column(regclass[]): need to run `SELECT NULL::void;` at end of function to avoid folding away functions called in previous query

fix: schemas/util.sql: diff(regclass, regclass): moved try_create() of copies column in parent table to auto_rm_freq() so that it would only happen if both tables actually contain a copies column (otherwise, the try_create() will create an empty copies column if both tables are empty)

schemas/util.sql: try_create(): also handle "child table is missing column" errors

schemas/util.sql: added coalesce(anyarray), which can be used to force evaluation of all values of a COALESCE

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

fix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: removed `public.` qualifier

schemas/vegbien.sql: implemented _plots_19_count_of_censuses_per_plot_in_each_project

inputs/SALVIAS/validations.sql: implemented _plots_19_count_of_censuses_per_plot_in_each_project