lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function

lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.

lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr

lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers

lib/sh/util.sh: added echo_minor_func alias

lib/sh/util.sh: added echo_eval()

lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file doesn't exist

lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places

lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting

lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.

lib/sh/util.sh: propagate $log_level_indent to invoked commands

lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent

bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().

bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)

bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded

bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)

lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`

lib/sh/util.sh: verbose output: put commands into subsections

lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()

bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run

bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME¹} in the function, so that callers can also call mysql_cmd via a nested function, etc.

lib/sh/util.sh: added ensure_nested_func()

lib/sh/util.sh: added contains()

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.

bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call

lib/sh/util.sh: import_vars alias: use new get_prefix_vars()

lib/sh/util.sh: add get_prefix_vars()

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

lib/sh/db.sh: added mk_select alias and use it in pg_export()

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

lib/sh/db.sh: pg_export(): rtrim $query

lib/sh/util.sh: added rtrim()

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

lib/sh/util.sh: added repeat()

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

lib/sh/util.sh: log_e (): use new log_err

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

lib/sh/util.sh: echo_cmd (): use new log ()

lib/sh/util.sh: added log ()

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

lib/sh/make.sh: added remaking alias

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

/run: moved geoscrub_input export into separate geoscrub_input/make () target

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

lib/sh/util.sh: moved canon_rel_path () into separate paths section

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

lib/sh/util.sh: added func_override (), for use in runscript inheritance

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

lib/sh/util.sh: exceptions: added die ()

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

lib/sh/util.sh: added save_e, which now sets $e just locally

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

lib/sh/util.sh: removed no longer used save_e_cmd

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd