lib/local.sh: added \$bin_dir_abs

bugfix: lib/util.sh: extern (): added `|| exit` after "$@" in addition to after the (), because this additional `|| exit` is apparently needed in order to propagate the exit status of the command to the exit status of the () and cause an error to abort the script properly

lib/util.sh: echo_vars (): output vars with log_level 3 so they can be excluded separately from function calls to reduce clutter in the logging output. increase the default util.run verbosity so that vars are still printed by default.

bugfix: lib/util.sh: echo_vars (): output each var separately so that each is prefixed by $PS4

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

bugfix: lib/util.sh: make (): use stdout2stderr=1 so that commands make runs are properly output to stderr

bugfix: lib/util.sh: include_guard_var (): use extern () when invoking sed so that this internal use of it isn't verbosely logged along with other external commands. note the "" around extern to refer to the function, not the alias.

bugfix: lib/util.sh: zip/unzip (): use new $stdout2stderr instead of >&2 so that the redirect happens in the right order

lib/util.sh: log_stderr_cmd (): support redirecting stdout to stderr *after*setting up the stderr redirect, so that stdout is also subject to that redirect

lib/util.sh: added stdout2stderr ()

lib/util.sh: changed echo_run_extern () to an alias

lib/util.sh: echo_run_extern (): use new log_stderr_cmd

lib/util.sh: added log_stderr_cmd ()

lib/util.sh: use extern instead of env to run external commands, because it's simpler and auto-captures the logging output

lib/util.sh: echo_func (): use the much simpler function keyword to avoid expanding the echo_func alias when defining the function, rather than going through the complex process of unaliasing and realiasing the echo_func alias

lib/util.sh: mark env as always being an external command. it is no longer necessary to automatically echo it using echo_run because the extern alias does this now.

lib/util.sh: automatically echo external commands. this requires using the function keyword when declaring extern () so that the new extern alias is not expanded in the function name.

lib/util.sh: added echo_run_extern (), which automatically runs log_stderr to capture the external command's stderr

lib/util.sh: added extern ()

lib/util.sh: added log_stderr ()

lib/util.sh: added end_try_subshell alias

lib/util.sh: zip/unzip: redirect logging output to stderr

lib/util.sh: use inc_log_level where logging info should only be output for verbosity >= 2: echo_func, echo_vars, echo_stdin

lib/util.sh: verbose output: added inc_log_level, dec_log_level aliases which manipulate $verbosity. note that changes in $log_level are now instead indicated by moving $verbosity in the opposite direction.

lib/util.sh: added let () wrapper that prevents it from triggering error-exit

lib/util.sh: verbose output: use just one var ($verbosity) instead of two ($verbosity and $log_level), to track log level, for simplicity. $verbosity would now be subtracted from to alter the log level instead of $log_level being added to.

lib/util.sh: use new can_log wherever logging info is output to stderr

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

lib/util.sh: verbose output: added $log_level (must always be > 0 so verbosity=0 turns off all logging) and can_log (), which compares $log_level to $verbosity

bugfix: lib/local.sh: updated $root_dir

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

lib/runscripts/local.run: added mysqldump_local, which uses vegbiendev

lib/runscripts/local.run: use postgres_compat mode because all our MySQL schemas should be usable as inputs to my2pg

lib/runscripts/local.run: added connection vars for local (bien) user

lib/util.sh: moved unalias inside include guard since it is not needed by anything before the include guard (even though it is used by commands outside the include guard, i.e. when the include guard is temporarily closed and reopened to load/unload aliases)

lib/util.sh: use declare instead of local in aliases so that the aliases can also be used outside a function (declare will create a local var when used inside a function, and a global var otherwise)

lib/util.sh: zip: ignore "zip has nothing to do" exit status as this is not an error

lib/util.sh: added exception-handling (i.e. error-suppressing) functions

bugfix: lib/util.sh: echo_func alias: moved it outside the include guard because the unalias statement is needs to be outside the include guard, and therefore so does the alias statement in order to always restore the alias

lib/util.sh: renamed echo_func_ () back to echo_func () and instead add an unalias statement before the if statement containing the function definition. this is necessary because if statements are "compound commands", which have aliases expanded when their definition is read (i.e. at the beginning of the if statement, for the entire if statement), not when they are executed, and any aliases for function names in them need to be unaliased before the if statement containing the function definition. this is necessary even if the if statement is not executed, because the expansion of such aliases will cause syntax errors in the parsing of the if statement itself....

lib/util.sh: added unalias () override, which doesn't produce an error if the alias is undefined

bugfix: lib/util.sh: renamed echo_func () to echo_func_ (), because you apparently can't have a function with the same name as an alias, even if the function is defined before the alias and an include guard prevents the function from being defined again after the alias is set. there may be some kind of alias-expanding preprocessing pass that bash only does in script mode, or maybe there is some other problem related to our use of aliases inside if statements.

*{.sh,run}: changed echo_func to an alias that includes the "$", so that callers don't need to specify the "$" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.

lib/util.sh: added set_make_vars alias, for use with make-target-style shell functions

lib/util.sh: mysql_cmd (): interpret an empty string for $user, $database as if they were unset

bugfix: lib/util.sh: mysql_cmd (): add --tables after specifying --databases so that future unnamed arguments are correctly interpreted as tables

bugfix: lib/util.sh: mysql_cmd (): need to use `--databases ...` rather than `--database=...` to specify the database as a named argument

bugfix: lib/util.sh: mysql_cmd (): need space before $database in command

lib/util.sh: mysql_cmd (): use $schema as $database when provided

lib/util.sh: mysql_cmd (): accept $database connection var

*{.sh,run}: use just env instead of echo_run env now that env is an auto-echoing alias

lib/util.sh: automatically echo commands that use env

lib/util.sh: moved strings after verbose output so strings commands can use verbose output aliases

lib/util.sh: mysql_cmd (): don't ssh if the ssh_server is this machine

lib/util.sh: mysql_cmd (): allow specifying the ssh_dest as a separate ssh_server and ssh_user

lib/util.sh: using prefixed connection vars: added use_local, use_local_remote aliases

lib/util.sh: added mysqldump_diffable ()

lib/util.sh: added mysqldump () wrapper, which does what my2pg_export does, but integrates with mysql_cmd and allows turning off postgres-compatible mode

lib/util.sh: mysql_cmd (): use ${FUNCNAME¹} to get the caller's command name instead of requiring the caller to pass the command name explicitly

lib/util.sh: databases: added use_remote alias to use the remote_* connection vars (which uses new import_vars)

lib/util.sh: added import_vars alias

lib/util.sh: mysql_cmd (): added doc comment

lib/util.sh: mysql (): use new mysql_cmd to auto-add connection/login opts when specified

lib/util.sh: added mysql_cmd ()

lib/util.sh: added local_inv alias

lib/util.sh: zip/unzip_newer: support extracting/updating the file even if it has an older mtime by setting $force, using new set_inv to get a var to use to determine whether to include the -u flag

lib/util.sh: added set_inv ()

lib/util.sh: echo_vars (): prefix the declare output with "+ " because it's actually a command, not just an indication of the var's value

lib/util.sh: added sed ()

lib/util.sh: added section labels

web/links/index.htm: removed broken favicons

web/links/index.htm: removed broken favicons. this requires using the SQLite Manager Firefox add-on <https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search> to manually remove the favicon entries.

lib/Firefox_bookmarks.reformat.csv: formatting fixes: don't put the src attribute of the favicon image on its own line, because it no longer contains long inline image data

lib/Firefox_bookmarks.reformat.csv: favicon image: remove ICON attributes with inline image data, since these are no longer needed for the favicon images and this avoids cluttering up the svn diff

lib/Firefox_bookmarks.reformat.csv: favicon image: use ICON_URI instead of ICON to fetch the favicon directly from the original site. this allows the favicons to be cached separately from the bookmarks page, so that the inline favicon contents don't need to be re-retrieved every time the bookmarks page is updated. this also avoids cluttering up the svn diff.

web/links/index.htm: updated to Firefox bookmarks. added description of the formats supported by PostgreSQL's COPY command and their escape sequences.

bugfix: schemas/vegbien.sql: geoscrub_input_view: only include places whose country is not NULL, because at least the country is required for geoscrubbing

/README.TXT: Schema changes: Reinstall staging tables: added step to run `make schemas/public/install` after the datasources are installed. note that this must be run after the datasources are installed, because views in public depend on some of the datasources.

/README.TXT: Schema changes: Reinstall staging tables: use `make db` instead of mk_db, which includes the required schemas/install command. some datasources use the util schema in their postprocess scripts, so this must be installed before the staging tables.

bugfix: /Makefile: moved schemas/install from install to db so that it is also run whenever the DB is reinstalled. schemas/install is needed by the staging tables reinstallation performed by reinstall_all.

bugfix: /README.TXT: Schema changes: Reinstall staging tables: also need to run `make mk_db` before running reinstall_all

/README.TXT: Schema changes: Reinstall staging tables: drop the saved previous DB at the end of the operation

/README.TXT: Schema changes: Reinstall staging tables: run all of the local machine steps on vegbiendev, including the vegbien DB rename, to ensure that the existing data is saved in case the reinstall has errors

/README.TXT: Schema changes: Reinstall staging tables: added step to first rename existing vegbien database, so it is not modified in the reinstallation

inputs/FIA/: archived no longer used BIEN2 FIA data from the nimoy geoscrub DB

inputs/FIA/_archive/: moved FIA_COND_unique, Organism into 2011-10-17/ subdir

/README.TXT: Schema changes: Reinstall staging tables: added step to first run this on the local machine to check for any errors, before running it on the live DB, to prevent the live DB from becoming unrestorable in the case of bugs

bugfix: bin/with_all: pipe yes to the command in case it makes any interactive prompts, as in e.g. the TNRS staging tables reinstall

bugfix: inputs/input.Makefile: sql/install: manually specify $no_search_path option to psql_script_vegbien, which is added automatically in $(psqlNoSearchPath) but that uses psql_verbose_vegbien

bugfix: *{.sh,run}: use `local` when setting default values for params to prevent these values from leaking into the calling context, where it may have unexpected effects (e.g. pg_copy_to would permanently set $source, which would then be used on the next invocation of pg_copy_to instead of regenerating $source from the next $table value). this involves using ${var-default} or ${var:-default} to return the value to the local statement instead of setting it directly with ${var=default} or ${var:=default}.

*{.sh,run}: commands that take named arguments: usage messages: changed to use simpler bash syntax for specifying command-specific env vars, which does not require a subshell ("The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments" <http://www.gnu.org/software/bash/manual/bash.html#Environment&gt;)

schemas/pg_hba*.conf: removed trailing whitespace

schemas/pg_hba*.conf: allow the bien group to access all databases, including test (on the Mac). note that this does not grant access to the vegbank DB on vegbiendev, because that is owned by postgres.

web/links/index.htm: explicitly separate page's description from author comments where needed

web/links/index.htm: use - instead of --- to separate the page's description from the author comments because it's faster to type and one hyphen is sufficient

lib/Firefox_bookmarks.reformat.csv: labeling page's description: don't match descriptions with a leading blank line as an author comment, because sometimes page descriptions themselves start with a blank line. instead, author comments at the beginning of bookmark descriptions should always start with a " or be preceded by a line containing only hyphens (-). note that it was never reliable to use a blank line for this purpose, because it would be trimmed out upon reimport of the HTML file into Firefox.

web/links/index.htm: updated to Firefox bookmarks

inputs/NVS/*/map.csv: Taxon Growth Form: use new _lowercase() to avoid needing to manually map each value that is already valid but just needs to be lowercased

schemas/util.sql: added _lowercase()