inputs/GBIF/raw_occurrence_record/run: override table.tsv/make() instead of export_mysql()

inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export

lib/sh/local.sh: added mysql_export_local()

lib/sh/db.sh: added mysql_export(). documented that mysql_export_outfile() supports CSV, but requires the FILE privilege.

lib/sh/db.sh: mysql(): use --column-names to ensure the output is formatted appropriately. note that --column-names is the default, but better to specify it to be sure.

lib/sh/db.sh: mysql(): output_data mode: use --batch to ensure the output is formatted appropriately. note that --batch is the default when stdin is from a pipe, but better to specify it to be sure.

*{.sh,run}: use new limit_stdout_cmd instead of `stdout2stderr=1 limit_stderr_cmd`

lib/sh/util.sh: added limit_stdout_cmd alias

lib/sh/make.sh: added $rm short var for $remake

lib/sh/util.sh: verbosity: removed `declare -i verbosity` because this does not actually ensure that $verbosity is an integer; it only has "arithmetic evaluation [...] done when the variable is assigned to" (`help declare`)

lib/sh/util.sh: verbosity: support setting this in alternate short-name var $vb

lib/sh/util.sh: override the `.` builtin (a.k.a. `source`) and run canon_rel_path on the included script. this removes .. in the path when it's displayed in bash error messages.

lib/sh/util.sh: log_err(): ensure errors are visible by using 2>&22. since this is uses log(), verbosity=1 also needs to be set.

lib/sh/util.sh: $explicit_errors_only: explicitly set verbosity=0 to hide startup logging

lib/sh/util.sh: limit_stderr(): if stdout goes to stderr (i.e. is logging info), assume that stderr_is_errors

lib/sh/util.sh: $explicit_errors_only: documented that this option should only be used for testing the explicit error displaying, as otherwise, important output may be missed

lib/sh/util.sh: $explicit_errors_only: added usage

bugfix: lib/sh/util.sh: command(): updated $errors_only var name to $explicit_errors_only

lib/sh/util.sh: $explicit_errors_only: noted that verbosity=0 displays everything that isn't explicitly hidden

lib/sh/local.sh: psql(): use new stderr_is_errors=1 since psql's logging output comes on stdout, so stderr contains only errors. (the caveat of this is that the query output is normally mixed with logging info on stdout, and needs to be separated out to a different file descriptor using --output.)

lib/sh/util.sh: added $explicit_errors_only env var which shows only explicitly-displayed errors (those which have been redirected to fd 22). note that most of the time, this has the same effect as `verbosity=0 script...`.

lib/sh/util.sh: command(): support ensuring errors are visible by redirecting them to fd 22 (global stderr) when $stderr_is_errors is set

lib/sh/util.sh: `command`: always prepend -- to the args list (to ensure that the command name is not interpreted as a `command` option), not just when the command alias is used

lib/sh/util.sh: set_global_fds(): increase the log_level so the shadow_fd()/set_fd() function names aren't output at the default verbosity (3), and the redirection commands themselves aren't output at verbosity <= 2. set_global_fds() happens at the beginning of every script that uses util.sh, and is fairly predictable, so it isn't necessary to always echo these commands.

lib/sh/local.sh: psql(): use new convention for command-specific alternate stdout

lib/sh/make.sh: inline_make: use new convention for command-specific alternate stdin

*{.sh,run}: removed extra space between function name and ()

lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function

lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.

lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr

lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers

lib/sh/util.sh: added echo_minor_func alias

lib/sh/util.sh: added echo_eval()

lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file doesn't exist

lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places

lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting

lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.

lib/sh/util.sh: propagate $log_level_indent to invoked commands

lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent

bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().

bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)

bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded

bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)

lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`

lib/sh/util.sh: verbose output: put commands into subsections

lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()

bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run

bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME¹} in the function, so that callers can also call mysql_cmd via a nested function, etc.

lib/sh/util.sh: added ensure_nested_func()

lib/sh/util.sh: added contains()

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.

bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call

lib/sh/util.sh: import_vars alias: use new get_prefix_vars()

lib/sh/util.sh: add get_prefix_vars()

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

lib/sh/db.sh: added mk_select alias and use it in pg_export()

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

lib/sh/db.sh: pg_export(): rtrim $query

lib/sh/util.sh: added rtrim()

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

lib/sh/util.sh: added repeat()

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

lib/sh/util.sh: log_e (): use new log_err

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

lib/sh/util.sh: echo_cmd (): use new log ()

lib/sh/util.sh: added log ()

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them