lib/sh/util.sh: split make utils out into separate make.sh

lib/sh/util.sh: split archive (zip) utils out into separate archives.sh

lib/sh/util.sh: split databases utils out into separate db.sh

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.

lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)

lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.

lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice

lib/util.sh: limit_stderr_cmd: alias-expand command after it

lib/util.sh: $verbosity: ensure it's an integer using `declare -i`

lib/util.sh: grouped set verbosity statements together and commented them

bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.

lib/util.sh: added bool2int ()

*{.sh,run}: use new isset

lib/util.sh: added isset ()

*{.sh,run}: use ${var+isset} instead of ${var+t} for clarity

lib/util.sh: propagate the verbosity to invoked commands by exporting it

lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.

lib/util.sh: echo_stdin (): use new pipe_delay

lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)

lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed outside of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.

lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).

lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged

lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function

inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.

inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error

inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists

lib/util.sh: make: added check_target_exists

lib/util.sh: moved $top_file* to runscripts/util.run because they only apply to runscripts

lib/util.sh: make: added to_target, which uses to_file on $target

lib/util.sh: added to_file (), which auto-removes a command's output file on error (like make's .DELETE_ON_ERROR)

lib/util.sh: echo_run_extern, extern: fixed/added comments to indicate that echo_run_extern echoes and controls stderr of an external command, while the extern alias does this for all external commands

lib/util.sh: auto-echo common external commands, such as rm

lib/util.sh: echo_run_extern, extern aliases: alias-expand next word (the command) by adding trailing space to alias def

bugfix: lib/util.sh: exceptions: log_e: must include `declare e` in the alias and not when save_e is called, so that $e is a local var of the caller. this bug did not appear in testing because the save_e alias, which re-scopes $e within log_e (), was not expanded inside log_e () (since new aliases were not loaded between save_e and log_e ()).

lib/util.sh: exceptions: added log_e (), which prints a "command exited with error" message (like make) when applicable

lib/util.sh: exceptions: try_ (): renamed to try () and use the function keyword to distinguish it from the alias

lib/util.sh: exceptions: try_ (): use new save_e

lib/util.sh: exceptions: added save_e

lib/util.sh: exceptions: end_try* aliases: use new rethrow*

lib/util.sh: exceptions: added rethrow, rethrow_subshell aliases

lib/util.sh: always use '' rather than "" around alias definitions, to ensure that variables are evaluated at expand time rather than compile time

lib/util.sh: zip_newer/unzip_newer: evaluate $no_force at alias expansion time rather than at alias compile time, so that the $no_force can be overridden in the context the alias is used in

lib/util.sh: use `|| return` instead of `|| exit` because `|| exit` doesn't seem to work inside functions (it does not have the errexit effect). also, `|| return` has the advantage of not exiting the program if the caller used || after the command (i.e. as an error handler) to temporarily disable errexit.

lib/runscripts/util.run: tell users to override run_args_cmd rather than this function to perform other commands, so that on_exit () can contain other exit-related processing that should not be overriden by the user

lib/util.sh: removed "" around $?, $# because they are guaranteed to always be non-empty and contain no special chars

lib/util.sh: say "last space" instead of "trailing space" so the comment will be more likely to fit at the end of the line

lib/local.sh: removed mysql () since its functionality is now provided by mysql () in util.sh

lib/util.sh: mysql_cmd (): $server: like $ssh_server, clear the value when it's equal to this machine's hostname

lib/util.sh: mysql_cmd (): $ssh_server: use new localize_url to clear the value when it's equal to this machine's hostname

lib/util.sh: added localize_url ()

inputs/GBIF/table.run: table.tsv/make (): use extern so the MySQL_export command is echoed

inputs/GBIF/table.run: table.tsv/make (): don't make the target if it already exists

inputs/GBIF/table.run: table.tsv/make (): create the file directly rather than using inline_make, so that the command to create the file can use shell functions such as mysql

bugfix: inputs/GBIF/table.run: table.tsv/make (): need to use $(target) (target's path relative to curdir) instead of just the filename, now that make does not change the directory

inputs/GBIF/table.run: moved table.tsv creation to separate table.tsv/make () function

lib/util.sh: echo_stdin (): display the echoed input after leading output of the echoed-to command, by using sleep with a small fractional # of seconds

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

lib/local.sh: root_make (): use make () rather than extern make now that make () does not set its own current dir. using make () also automatically enables \$stdout2stderr, which is needed to redirect the echoed make commands.

lib/util.sh: extern (): use exec to ensure that the executed command is in fact external, and not a shell builtin or other type of non-function command. this also simplifies the code because any shell function for the command does not need to be unset.

lib/util.sh: make (): don't change the directory to $top_dir, because paths used by the caller will instead be relative to the current dir (note that in runscripts, paths are generally not absolute because of canon_rel_path ()). this requires manually specifying $top_dir when running make on a physical (rather than inlined) makefile.

lib/util.sh: log_stderr_cmd (): replaced the `|| exit` after the () with `|| return`, for the same reason that this was a problem for extern () (see r8954)

lib/util.sh: extern (): replaced the `|| exit` after the () with `|| return`, because `|| exit` strangely does not use $? as its exit status (even though `|| exit $?` works fine). in a shell function, it is actually sufficient, for errexit purposes, to use `|| return`, because the exit status is checked upon return from the function. it is in fact preferable to use return rather than exit, because then the caller can catch the exception using `|| true`.

lib/util.sh: extern (): removed the `|| exit` inside the (), because this strangely prevents the exit status from being propagated to the exit status of the ()

lib/util.sh: added explicit `set -o errexit` in case caller did not have -e in #! line

inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run

inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run

inputs/FIA/_archive/2011-10-17/: set svn:ignore to *

web/links/index.htm: updated to Firefox bookmarks. added Mac upgrade and recovery-related bookmarks.

bugfix: lib/import.sh: remake_VegBIEN_mappings (): don't run if there is no public schema to perform the test on (e.g. if the staging tables have just been reinstalled)

lib/import.sh: added public_schema_exists (), which just calls psql_script_vegbien (i.e. with the default search_path temp, public, etc.) and sees if returns an error

/README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: added step to download TWiki separately to your machine, to ensure that it gets backed up even if you don't sync the rest of the files. also, TWiki needs to be run with delete enabled to remove old session files, while the rest of the files should be run without delete.

schemas/VegCore/VegCore.ERD.mwb.run: use my2pg's new sql_ascii mode

bin/my2pg: support $sql_ascii mode, where varbinary is translated to text instead of bytea

bin/my2pg: remove any () after text

schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make (): leave out the DROP TABLE statements for now using --skip-add-drop-table

added schemas/VegCore/VegCore.pg.sql.run with install () command (also part of all ())

lib/local.sh: added root_make ()

bin/my2pg: unpack custom types set('...')

lib/local.sh: psql (): use fd 4 for the logging output instead of 3 because 3 is generally used for passing inline input commands (it is used this way by inline_make)

schemas/VegCore/VegCore.ERD.mwb.run: sync (): don't pass "$@" to export_/import because they do not accept args, and the same set of args would not be meaningful for both

schemas/VegCore/VegCore.ERD.mwb.run: use new $top_filename instead of $archive_filename

lib/util.sh: added $top_script (alias for $0), $top_file, $top_filename. use new $top_file in $top_dir.

schemas/VegCore/VegCore.ERD.mwb.run: put zip_newer/unzip_newer on the same line as the `cd "$top_dir"`

bin/my2pg: translate varbinary to bytea

schemas/VegCore/VegCore.ERD.mwb.run: added export_postgres (), which calls VegCore.pg.sql/make ()

lib/util.sh: wrapper functions of external commands: use new self alias

lib/util.sh: added self alias, for use with wrapper functions of external commands

schemas/VegCore/VegCore.my.sql: regenerated using `VegCore.ERD.mwb.run export_mysql` (also part of the all target). note that this uses mysqldump, so the format and table order is different than it was for MySQL Workbench's custom export script.

schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()

schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()

bugfix: lib/local.sh: $PATH: use $bin_dir_abs so the path will remain valid if the current directory is changed

lib/local.sh: added \$bin_dir_abs

bugfix: lib/util.sh: extern (): added `|| exit` after "$@" in addition to after the (), because this additional `|| exit` is apparently needed in order to propagate the exit status of the command to the exit status of the () and cause an error to abort the script properly

lib/util.sh: echo_vars (): output vars with log_level 3 so they can be excluded separately from function calls to reduce clutter in the logging output. increase the default util.run verbosity so that vars are still printed by default.

bugfix: lib/util.sh: echo_vars (): output each var separately so that each is prefixed by $PS4

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

bugfix: lib/util.sh: make (): use stdout2stderr=1 so that commands make runs are properly output to stderr

bugfix: lib/util.sh: include_guard_var (): use extern () when invoking sed so that this internal use of it isn't verbosely logged along with other external commands. note the "" around extern to refer to the function, not the alias.