lib/sh/util.sh: merged limit_stderr_cmd/limit_stdout_cmd into command(), using flag vars to control what limiting actions it needs to perform. in command invocations, this involves setting the appropriate flag vars instead of using a limit_std*_cmd alias.

lib/sh/util.sh: added run_cmd

lib/sh/util.sh: limit_stderr(): moved before command echoing because it does not depend on it and is a separate feature

lib/sh/util.sh: run_args_cmd(): use $top_script instead of `"$(canon_rel_path "$0")"`

lib/sh/util.sh: $top_script: run canon_rel_path on $0

lib/sh/util.sh: use `minor=1 echo_func` instead of echo_minor_func because the syntax is simple enough that you don't need a separate alias for it

lib/sh/util.sh: echo_func: support using minor=1 to increase the log_level that the function call is echoed at. use this in echo_minor_func.

lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.

lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.

lib/sh/util.sh: echo_func: only indent if the function call was displayed. this avoids confusing extra whitespace when a function call causes an indent but the function call itself isn't printed.

bugfix: lib/sh/util.sh: echo_minor_func: need to include "$@" in echo_func() call

lib/sh/util.sh: echo_func(): added usage

lib/sh/util.sh: echo_func: added usage

lib/sh/util.sh: $log_level_indent: changed to a single space rather than two, to avoid very deep indents as the call tree becomes more deeply nested. due to the + at the beginning of every line, a single space is enough to visualize the structure of the call tree (unlike source code, which requires more whitespace because it doesn't have a call tree structure).

lib/sh/util.sh: echo_*func aliases: increase the indent for the duration of the function call. this causes function calls to be displayed in an indented call-tree structure, which makes the verbose output much easier to understand.

bugfix: lib/sh/util.sh: recursive aliases (i.e. aliases that call a command of the same name): use "" around the command name so it isn't reexpanded if the function using the alias is copied using copy_func(). (aliases will be reexpanded every time a function is redeclared since they appear unquoted in the function definition.) this is a bug in `declare -f` for which there is no fix, necessitating the "" alias workaround instead.

lib/sh/util.sh: moved log-indent-related vars/aliases before they are first used (by log())

lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--

lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--

lib/sh/util.sh: renamed cmd_indent to just indent now that indent is not being used for the PS4 level

*{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||

lib/sh/util.sh: renamed indent->PS4++ , outdent->PS4-- because these relate just to the PS4 length, not to the (whitespace-based) log_indent

lib/sh/util.sh: removed no longer used echo_run(). use command() instead.

*{.sh,run}: use command instead of deprecated echo_run (don't prepend anything when the command is already aliased with `command`)

lib/sh/util.sh: command(): perform echo_run_prep itself instead of requiring echo_run to be added in an alias. echo_run_prep will not be performed if limit_stderr_cmd() has performed it already outside the limit_stderr section. this requires using `builtin command` instead of "command" when you don't want the echoing (such as in include_guard_var()).

*{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||

bugfix: lib/sh/util.sh: .(): also need to echo the . before the included file

lib/sh/util.sh: added echo_params alias and use it instead of `log "$*"`

lib/sh/util.sh: .(): echo the include at log_level 2 because it's an internal command

lib/sh/util.sh: echo_cmd(): also remove `builtin` before the command name

lib/sh/util.sh: echo_cmd(): don't remove -- after command, because it is added by the command() function instead and does not appear in the command sent to echo_run()

lib/sh/util.sh: command alias: don't alias-expand next word, because the next word should only be interpreted as a command (part of the effect of the `command` builtin)

lib/sh/util.sh: external commands: always use command rather than echo_run to indicate that a command is external, because command() ensures that what it invokes is not a shell function, and sets 2>&22 where needed

lib/sh/util.sh: .(): use echo_run so that included files are echoed as they are included. note that echo_run uses cmd_indent, so the included files' load-time logging output will be indented according to the nesting level of the include.

lib/sh/util.sh: moved .() command echoing > internal commands section

lib/sh/util.sh: moved echo_run() to general command echoing section because it can also be used for internal commands

removed no longer used inputs/GBIF/MySQL_export. use lib/sh/local.sh mysql_export_local() instead.

inputs/GBIF/raw_occurrence_record/run: override table.tsv/make() instead of export_mysql()

inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export

lib/sh/local.sh: added mysql_export_local()

lib/sh/db.sh: added mysql_export(). documented that mysql_export_outfile() supports CSV, but requires the FILE privilege.

lib/sh/db.sh: mysql(): use --column-names to ensure the output is formatted appropriately. note that --column-names is the default, but better to specify it to be sure.

lib/sh/db.sh: mysql(): output_data mode: use --batch to ensure the output is formatted appropriately. note that --batch is the default when stdin is from a pipe, but better to specify it to be sure.

*{.sh,run}: use new limit_stdout_cmd instead of `stdout2stderr=1 limit_stderr_cmd`

lib/sh/util.sh: added limit_stdout_cmd alias

lib/sh/make.sh: added $rm short var for $remake

lib/sh/util.sh: verbosity: removed `declare -i verbosity` because this does not actually ensure that $verbosity is an integer; it only has "arithmetic evaluation [...] done when the variable is assigned to" (`help declare`)

lib/sh/util.sh: verbosity: support setting this in alternate short-name var $vb

lib/sh/util.sh: override the `.` builtin (a.k.a. `source`) and run canon_rel_path on the included script. this removes .. in the path when it's displayed in bash error messages.

lib/sh/util.sh: log_err(): ensure errors are visible by using 2>&22. since this is uses log(), verbosity=1 also needs to be set.

lib/sh/util.sh: $explicit_errors_only: explicitly set verbosity=0 to hide startup logging

lib/sh/util.sh: limit_stderr(): if stdout goes to stderr (i.e. is logging info), assume that stderr_is_errors

lib/sh/util.sh: $explicit_errors_only: documented that this option should only be used for testing the explicit error displaying, as otherwise, important output may be missed

lib/sh/util.sh: $explicit_errors_only: added usage

bugfix: lib/sh/util.sh: command(): updated $errors_only var name to $explicit_errors_only

lib/sh/util.sh: $explicit_errors_only: noted that verbosity=0 displays everything that isn't explicitly hidden

lib/sh/local.sh: psql(): use new stderr_is_errors=1 since psql's logging output comes on stdout, so stderr contains only errors. (the caveat of this is that the query output is normally mixed with logging info on stdout, and needs to be separated out to a different file descriptor using --output.)

lib/sh/util.sh: added $explicit_errors_only env var which shows only explicitly-displayed errors (those which have been redirected to fd 22). note that most of the time, this has the same effect as `verbosity=0 script...`.

lib/sh/util.sh: command(): support ensuring errors are visible by redirecting them to fd 22 (global stderr) when $stderr_is_errors is set

lib/sh/util.sh: `command`: always prepend -- to the args list (to ensure that the command name is not interpreted as a `command` option), not just when the command alias is used

lib/sh/util.sh: set_global_fds(): increase the log_level so the shadow_fd()/set_fd() function names aren't output at the default verbosity (3), and the redirection commands themselves aren't output at verbosity <= 2. set_global_fds() happens at the beginning of every script that uses util.sh, and is fairly predictable, so it isn't necessary to always echo these commands.

lib/sh/local.sh: psql(): use new convention for command-specific alternate stdout

lib/sh/make.sh: inline_make: use new convention for command-specific alternate stdin

*{.sh,run}: removed extra space between function name and ()

lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function

lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.

lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr

lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers

lib/sh/util.sh: added echo_minor_func alias

lib/sh/util.sh: added echo_eval()

lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file doesn't exist

lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places

lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting

lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.

lib/sh/util.sh: propagate $log_level_indent to invoked commands

lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent

bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().

bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)

bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded

bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)

lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`

lib/sh/util.sh: verbose output: put commands into subsections

lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()

bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run

bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME¹} in the function, so that callers can also call mysql_cmd via a nested function, etc.

lib/sh/util.sh: added ensure_nested_func()

lib/sh/util.sh: added contains()

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.