lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

lib/sh/make.sh: added remaking alias

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

/run: moved geoscrub_input export into separate geoscrub_input/make () target

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

lib/sh/util.sh: moved canon_rel_path () into separate paths section

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

lib/sh/util.sh: added func_override (), for use in runscript inheritance

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

lib/sh/util.sh: exceptions: added die ()

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

lib/sh/util.sh: added save_e, which now sets $e just locally

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

lib/sh/util.sh: removed no longer used save_e_cmd

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd

bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.

lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.

lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?

lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed

lib/sh/util.sh: added func_exists ()

lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow

lib/sh/util.sh: added copy_func ()

lib/sh/util.sh: save_e (): usage: added rethrow example

bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh

*{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.

lib/sh/util.sh: split make utils out into separate make.sh

lib/sh/util.sh: split archive (zip) utils out into separate archives.sh

lib/sh/util.sh: split databases utils out into separate db.sh

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.

lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)

lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.

lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice

lib/util.sh: limit_stderr_cmd: alias-expand command after it

lib/util.sh: $verbosity: ensure it's an integer using `declare -i`

lib/util.sh: grouped set verbosity statements together and commented them

bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.

lib/util.sh: added bool2int ()

*{.sh,run}: use new isset

lib/util.sh: added isset ()

*{.sh,run}: use ${var+isset} instead of ${var+t} for clarity

lib/util.sh: propagate the verbosity to invoked commands by exporting it

lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.

lib/util.sh: echo_stdin (): use new pipe_delay

lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)

lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed outside of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.

lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).

lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged

lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function

inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.

inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error

inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists

lib/util.sh: make: added check_target_exists

lib/util.sh: moved $top_file* to runscripts/util.run because they only apply to runscripts

lib/util.sh: make: added to_target, which uses to_file on $target

lib/util.sh: added to_file (), which auto-removes a command's output file on error (like make's .DELETE_ON_ERROR)

lib/util.sh: echo_run_extern, extern: fixed/added comments to indicate that echo_run_extern echoes and controls stderr of an external command, while the extern alias does this for all external commands

lib/util.sh: auto-echo common external commands, such as rm

lib/util.sh: echo_run_extern, extern aliases: alias-expand next word (the command) by adding trailing space to alias def

bugfix: lib/util.sh: exceptions: log_e: must include `declare e` in the alias and not when save_e is called, so that $e is a local var of the caller. this bug did not appear in testing because the save_e alias, which re-scopes $e within log_e (), was not expanded inside log_e () (since new aliases were not loaded between save_e and log_e ()).

lib/util.sh: exceptions: added log_e (), which prints a "command exited with error" message (like make) when applicable

lib/util.sh: exceptions: try_ (): renamed to try () and use the function keyword to distinguish it from the alias

lib/util.sh: exceptions: try_ (): use new save_e

lib/util.sh: exceptions: added save_e

lib/util.sh: exceptions: end_try* aliases: use new rethrow*

lib/util.sh: exceptions: added rethrow, rethrow_subshell aliases

lib/util.sh: always use '' rather than "" around alias definitions, to ensure that variables are evaluated at expand time rather than compile time

lib/util.sh: zip_newer/unzip_newer: evaluate $no_force at alias expansion time rather than at alias compile time, so that the $no_force can be overridden in the context the alias is used in

lib/util.sh: use `|| return` instead of `|| exit` because `|| exit` doesn't seem to work inside functions (it does not have the errexit effect). also, `|| return` has the advantage of not exiting the program if the caller used || after the command (i.e. as an error handler) to temporarily disable errexit.

lib/runscripts/util.run: tell users to override run_args_cmd rather than this function to perform other commands, so that on_exit () can contain other exit-related processing that should not be overriden by the user

lib/util.sh: removed "" around $?, $# because they are guaranteed to always be non-empty and contain no special chars

lib/util.sh: say "last space" instead of "trailing space" so the comment will be more likely to fit at the end of the line

lib/local.sh: removed mysql () since its functionality is now provided by mysql () in util.sh

lib/util.sh: mysql_cmd (): $server: like $ssh_server, clear the value when it's equal to this machine's hostname

lib/util.sh: mysql_cmd (): $ssh_server: use new localize_url to clear the value when it's equal to this machine's hostname

lib/util.sh: added localize_url ()

inputs/GBIF/table.run: table.tsv/make (): use extern so the MySQL_export command is echoed

inputs/GBIF/table.run: table.tsv/make (): don't make the target if it already exists

inputs/GBIF/table.run: table.tsv/make (): create the file directly rather than using inline_make, so that the command to create the file can use shell functions such as mysql