lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)
: local setting of log_level: use log_local instead of relying on the log aliases, so that these aliases can instead be used for wrapping commands (the more common use case)
bugfix: lib/sh/util.sh: verbosity_compat alias: need to use `declare verbosity="$verbosity"` instead of `declare verbosity`, which would just clear $verbosity
bugfix: lib/sh/util.sh: verbosity_min alias: need to use `declare verbosity="$verbosity"` instead of log_local now that verbosity is not one of the vars changed by log++
lib/sh/util.sh: log+(): use easier-to-understand log_local instead of prefix-assignments to limit assignments to the invoked command
: use clog instead of "log*"
bugfix: lib/sh/util.sh: log+(): removed spurious ; between setting of PS4 and log_level, which was causing erratic mismatches between PS4 and log_level. (the ; caused $PS4 to be set in the caller when invoked via one of the clog* aliases, rather than being passed as a command-specific env var.)
lib/sh/util.sh: $verbosity: stay constant at what the user set it to instead of changing in tandem with $log_level, to facilitate debugging verbosity/log_level-related issues
lib/sh/util.sh: log+(): usage: use aliases instead of ""-ed function names
added schemas/VegCore.ERD.pdf symlink for easy access
lib/sh/util.sh: log_err(): use red background for better visibility of errors, in the same way that lib/exc.py print_ex() does for column-based import
bugfix: lib/sh/util.sh: removed echo_func in functions used by log++, to avoid spurious highlighted output
lib/sh/util.sh: added missing clog+ alias
bugfix: lib/sh/util.sh: log_hint(): use the standard log_fd and log_info() format, not err_fd and log_err() format, for hint messages
fix: lib/sh/util.sh: log_msg!(): indent each line, not just the first
lib/sh/util.sh: added split_lines()
lib/sh/util.sh: log(): factored out helper function log_msg!()
fix: lib/sh/util.sh: highlight_msg(): bold instead of underlining because the underlining interferes with the readability of the commands
lib/sh/util.sh: highlight_msg(): allow turning off formatting w/ empty $format
fix: lib/sh/util.sh: log_err() calls: removed manual highlighting
lib/sh/util.sh: log_err(): highlight all error messages using highlight_msg()'s new $format
lib/sh/util.sh: highlight_msg(): support custom format
lib/sh/db.sh: pg_*_exists(): log the DB statements to check this at a higher log_level so that they don't clutter up the log output
lib/sh/util.sh: log(): highlight log_level 1 messages to stand out against other output, for easier debugging
*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow
bugfix: catch(): also need to support $1='' because this is a now a use case of ignore_e()
bugfix: lib/sh/util.sh: ignore_err_msg(): also need to ignore false exit status on no match
lib/sh/util.sh: stderr_matches(): moved prep_try/rethrow into the function itself so that callers don't have to wrap this function in a complex sequence of prep_try/rethrow statements
lib/sh/util.sh: added rethrow_exit alias
fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.
fix: inputs/input.Makefile: removed no longer used special handling of XML inputs, support for which was never added to the Makefile. (bin/map, however, does support importing an XML file into a database.) this fixes a bug in XAL, which used to abort with an error but now just imports an empty table.
fix: inputs/input.Makefile: %/install: don't ignore errors if table does not exist, to ensure a proper errexit. this is now possible because every dir that this target is being run on should be a data dir. (Source/ used to be a metadata-only dir.)
bugfix: inputs/input.Makefile: $(cleanup): need `set -o pipefail`
inputs/VegBank/run: `rm=1 import()`: updated runtime (1 h)
inputs/VegBank/taxon_observation.**/test.xml.ref: updated inserted row count
inputs/VegBank/projectcontributor_/test.xml.ref: updated inserted row count
bugfix: schemas/util.sql: is_constant(util.col_ref): updated to include standard newline at beginning of comment (applies to newly-imported staging tables)
bugfix: inputs/VegBank/import_order.txt: added missing project, needed to trigger the staging table renaming for the project table
inputs/VegBank/run: documented `rm=1 import()` runtime (>1.5 h)
inputs/VegBank/run: documented `datasrc_make sql/install` runtime (25 min)
inputs/MO/Specimen/test.xml.ref: updated, which adds dateCollected mappings
inputs/WIN/Specimen/test.xml.ref: updated to map.csv, which has eventDate->dateCollected
inputs/VegBank/plantconcept_/create.sql: updated runtime (25 min, ~same)
lib/sh/make.sh: begin_target: echo all targets to facilitate debugging without needing the verbose stack trace mode
bugfix: lib/sh/make.sh: echo_target: don't include filename/line #, since this is not for the stack trace mode
lib/sh/make.sh: added echo_target
*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`
lib/runscripts/util.run: runscript template: added sample make target, using new make target template
lib/sh/make.sh: added make target template
inputs/VegBank/plot/postprocess.sql: remove institutions that we have direct data for: CVS: updated runtime (same)
lib/sh/make.sh: added begin_target alias
lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)
bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try
bugfix: lib/sh/util.sh: rethrow*: only `return` if $e is actually nonzero, because rethrow is now being used as a catch-all in situations where there might not be an error
lib/sh/util.sh: prep_try: initialize $e to 0 to simplify error-handling coding
stderr_matches(): wrapper caller usage: added alternative usage when using `||`
lib/sh/util.sh: stderr_matches(): wrapper caller usage: documented usage for a negated condition (ie. prefixed w/ !)
lib/sh/util.sh: stderr_matches(): usage: split into wrapper usage and wrapper caller usage for clarity
fix: *{.sh,run}: stderr_matches() wrappers: usage: added `rethrow`
fix: lib/sh/util.sh: stderr_matches(): usage: `rethrow` must be called right after stderr_matches(), to avoid calling running other commands if there is an error
fix: lib/sh/util.sh: stderr_matches(): when using $ignore_e, also set benign_error=1 to suppress the highlighting of the error
bugfix: lib/sh/db.sh: pg_schema_exists(): need to ignore benign error exit status from the "cannot create temporary relation in non-temporary schema" error
lib/sh/util.sh: stderr_matches(): supporting ignoring any benign error exit status associated with the error message being tested for
lib/sh/util.sh: stderr_matches(): usage: documented where any ignore_e statement would go
bugfix: lib/sh/util.sh: stderr_matches(): can't use `try` because this clears the exit status, which is needed for @PIPESTATUS to work. to support this, also need to avoid errexiting since @PIPESTATUS will be used instead.
lib/sh/util.sh: added dp(), which debug-prints a message
bugfix: inputs/VegBank/plot/postprocess.sql: use CVS.plot_ instead because that has the renamed staging table columns, and is compatible with auto-renaming of the SQL script columns
inputs/CVS/plot_/postprocess.sql: add unique constraint on locationName (analogous to the unique constraint in plot), for use by inputs/VegBank/plot/postprocess.sql in removing inter-datasource duplication
fix: schemas/util.sql: explain2notice_msg(): don't include EXPLAIN output for simple, single-value queries, to avoid cluttering up the log output
schemas/util.sql: added fold_explain_msg()
bugfix: bin/repl: only use excluded_prefix_re/excluded_suffix_re in text mode (used in renaming columns in SQL scripts), to prevent the special coding for column renames from also affecting regular regexp/word replacements
inputs/run: postprocess(): documented runtime (30 min)
bugfix: inputs/input.Makefile: %/postprocess.sql: don't perform replacements using map.csv, because map.csv is not idempotent. this functionality was only there to facilitate switching to new-style import, which is now largely done. (the remaining datasources NVS, SALVIAS, TEAM contain only 1 postprocess.sql: inputs/SALVIAS/projects/postprocess.sql (`st inputs/{NVS,SALVIAS,TEAM}/*/postprocess.sql`).)
bugfix: bin/repl: text mode: also don't match if it's part of a '-'-separated identifier
bugfix: bin/repl: text mode: also don't match if it's a word in a sentence
bugfix: bin/repl: text mode: turned off the suffix matching, because there are cases where a mapping adds a suffix which would cause the same replacement to be performed repeatedly
inputs/input.Makefile: %/postprocess.sql: always run this, not just if the associated map spreadsheets change, to avoid needing to `touch` them to cause %/postprocess.sql to run
bin/repl: text mode: exclude prefixes that should not cause replacement, to avoid doubling leading *
fix: inputs/*/*/postprocess.sql: un-doubled *
bugfix: inputs/input.Makefile: %/postprocess.sql: also need to apply renames from mappings/VegCore.thesaurus.csv, as these have been applied to map.csv
bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run
bin/repl: text mode: also match w/ suffix (eg. _verbatim)
bugfix: /README.TXT: Maintenance: VegCore data dictionary: apply new data dict mappings: need to use postprocess rather than import runscript target, so that the command also works on an svn checkout without the flat files (the flat files are not needed for the staging table renaming)
lib/sh/db.sh: psql(): $verbose_ok: renamed to $bypass_ok for clarity, because this applies only to the `--output /dev/fd/41` bypass (which when not possible, requires turning off verbose output
fix: lib/sh/db.sh: psql(): added $output_data switch analogous to what mysql() has. this causes query results of eg. void-returning functions to be correctly filtered by the logging mechanism, rather than output to stdout.
fix: lib/sh/db.sh: psql(): verbosity=0 (errors only) mode: use `SET client_min_messages = WARNING;` instead of NOTICE to hide verbose messages within psql as well
lib/sh/db.sh: psql(): replaced `test "$verbose_ok" && can_log` with bool var $verbose_
fix: lib/sh/db.sh: psql(): $verbose_: renamed to $verbose_ok for clarity
fix: lib/sh/util.sh: stdout_contains(): add another pipe_delay because the `grep` statement was sometimes getting printed before its filtered output
bugfix: schemas/util.sql: set_col_types(): need to COALESCE the executed SQL to '' because util.eval() does not support NULL (and shouldn't, because this indicates a missing COALESCE in constructing the statement)
schemas/util.sql: set_col_types(): use simpler util.eval() instead of manual EXECUTE/util.debug_print_sql()
schemas/util.sql: set_col_types(): use string_agg() instead of array_to_string(ARRAY) for clarity
bugfix: lib/sh/util.sh: die_error_hidden(): min verbosity to display error should not be hardcoded
lib/sh/db.sh: psql(): "to see error details" msg: use new die_error_hidden()