lib/sh/util.sh: $explicit_errors_only: noted that verbosity=0 displays everything that isn't explicitly hidden

lib/sh/local.sh: psql(): use new stderr_is_errors=1 since psql's logging output comes on stdout, so stderr contains only errors. (the caveat of this is that the query output is normally mixed with logging info on stdout, and needs to be separated out to a different file descriptor using --output.)

lib/sh/util.sh: added $explicit_errors_only env var which shows only explicitly-displayed errors (those which have been redirected to fd 22). note that most of the time, this has the same effect as `verbosity=0 script...`.

lib/sh/util.sh: command(): support ensuring errors are visible by redirecting them to fd 22 (global stderr) when $stderr_is_errors is set

lib/sh/util.sh: `command`: always prepend -- to the args list (to ensure that the command name is not interpreted as a `command` option), not just when the command alias is used

lib/sh/util.sh: set_global_fds(): increase the log_level so the shadow_fd()/set_fd() function names aren't output at the default verbosity (3), and the redirection commands themselves aren't output at verbosity <= 2. set_global_fds() happens at the beginning of every script that uses util.sh, and is fairly predictable, so it isn't necessary to always echo these commands.

lib/sh/local.sh: psql(): use new convention for command-specific alternate stdout

lib/sh/make.sh: inline_make: use new convention for command-specific alternate stdin

*{.sh,run}: removed extra space between function name and ()

lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function

lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.

lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr

lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers

lib/sh/util.sh: added echo_minor_func alias

lib/sh/util.sh: added echo_eval()

lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file doesn't exist

lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places

lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting

lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.

lib/sh/util.sh: propagate $log_level_indent to invoked commands

lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent

bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().

bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)

bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded

bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)

lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`

lib/sh/util.sh: verbose output: put commands into subsections

lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()

bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run

bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME¹} in the function, so that callers can also call mysql_cmd via a nested function, etc.

lib/sh/util.sh: added ensure_nested_func()

lib/sh/util.sh: added contains()

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.

bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call

lib/sh/util.sh: import_vars alias: use new get_prefix_vars()

lib/sh/util.sh: add get_prefix_vars()

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

lib/sh/db.sh: added mk_select alias and use it in pg_export()

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

lib/sh/db.sh: pg_export(): rtrim $query

lib/sh/util.sh: added rtrim()

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

lib/sh/util.sh: added repeat()

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

lib/sh/util.sh: log_e (): use new log_err

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

lib/sh/util.sh: echo_cmd (): use new log ()

lib/sh/util.sh: added log ()

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

lib/sh/make.sh: added remaking alias

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

/run: moved geoscrub_input export into separate geoscrub_input/make () target

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

lib/sh/util.sh: moved canon_rel_path () into separate paths section

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

lib/sh/util.sh: added func_override (), for use in runscript inheritance