bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME¹} in the function, so that callers can also call mysql_cmd via a nested function, etc.

lib/sh/util.sh: added ensure_nested_func()

lib/sh/util.sh: added contains()

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.

bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call

lib/sh/util.sh: import_vars alias: use new get_prefix_vars()

lib/sh/util.sh: add get_prefix_vars()

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

lib/sh/db.sh: added mk_select alias and use it in pg_export()

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

lib/sh/db.sh: pg_export(): rtrim $query

lib/sh/util.sh: added rtrim()

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

lib/sh/util.sh: added repeat()

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

lib/sh/util.sh: log_e (): use new log_err

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

lib/sh/util.sh: echo_cmd (): use new log ()

lib/sh/util.sh: added log ()

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

lib/sh/make.sh: added remaking alias

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

/run: moved geoscrub_input export into separate geoscrub_input/make () target

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

lib/sh/util.sh: moved canon_rel_path () into separate paths section

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

lib/sh/util.sh: added func_override (), for use in runscript inheritance

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

lib/sh/util.sh: exceptions: added die ()

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

lib/sh/util.sh: added save_e, which now sets $e just locally

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

lib/sh/util.sh: removed no longer used save_e_cmd

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd

bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.

lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.

lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?

lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed

lib/sh/util.sh: added func_exists ()

lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow

lib/sh/util.sh: added copy_func ()

lib/sh/util.sh: save_e (): usage: added rethrow example

bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh

*{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.

lib/sh/util.sh: split make utils out into separate make.sh

lib/sh/util.sh: split archive (zip) utils out into separate archives.sh

lib/sh/util.sh: split databases utils out into separate db.sh

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.

lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)

lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.

lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice

lib/util.sh: limit_stderr_cmd: alias-expand command after it