lib/sh/util.sh: add get_prefix_vars()

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

lib/sh/db.sh: added mk_select alias and use it in pg_export()

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

lib/sh/db.sh: pg_export(): rtrim $query

lib/sh/util.sh: added rtrim()

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

lib/sh/util.sh: added repeat()

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

lib/sh/util.sh: log_e (): use new log_err

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

lib/sh/util.sh: echo_cmd (): use new log ()

lib/sh/util.sh: added log ()

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

lib/sh/make.sh: added remaking alias

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

/run: moved geoscrub_input export into separate geoscrub_input/make () target

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

lib/sh/util.sh: moved canon_rel_path () into separate paths section

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

lib/sh/util.sh: added func_override (), for use in runscript inheritance

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

lib/sh/util.sh: exceptions: added die ()

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

lib/sh/util.sh: added save_e, which now sets $e just locally

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

lib/sh/util.sh: removed no longer used save_e_cmd

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd

bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.

lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.

lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?

lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed

lib/sh/util.sh: added func_exists ()

lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow

lib/sh/util.sh: added copy_func ()

lib/sh/util.sh: save_e (): usage: added rethrow example

bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh

*{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.

lib/sh/util.sh: split make utils out into separate make.sh

lib/sh/util.sh: split archive (zip) utils out into separate archives.sh

lib/sh/util.sh: split databases utils out into separate db.sh

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.

lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)

lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.

lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice

lib/util.sh: limit_stderr_cmd: alias-expand command after it

lib/util.sh: $verbosity: ensure it's an integer using `declare -i`

lib/util.sh: grouped set verbosity statements together and commented them

bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.

lib/util.sh: added bool2int ()

*{.sh,run}: use new isset

lib/util.sh: added isset ()

*{.sh,run}: use ${var+isset} instead of ${var+t} for clarity

lib/util.sh: propagate the verbosity to invoked commands by exporting it

lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.

lib/util.sh: echo_stdin (): use new pipe_delay

lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)

lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed outside of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.

lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).

lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged

lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function

inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.

inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error