web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: added fixes for Apache ~/Sites dirs, Apache PHP

web/links/index.htm: updated to Firefox bookmarks. added bookmarks for upgrading Mac OS X to 10.8 Mountain Lion. (WARNING: DO NOT upgrade unless you are prepared to fix several programs broken by the upgrade: svn, PostgreSQL. instructions are in the corresponding bookmark subdirs. these programs will be COMPLETELY UNAVAILABLE until they are manually fixed!)

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include scientific_name, which is preferable as a TNRS input because it also contains lower ranks

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include id, institution_code, collection_code, catalogue_number

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added filter for institution_codes in herbaria.ih (in PostgreSQL)

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added column subset (from http://vegpath.org/twiki/bin/view/Main/ConfCall20130509#subsetting_strategy > include)

lib/sh/db.sh: mk_select: constructed queries: in ${var:+if_true} syntax, put the newline at the end of the if_true value instead of the beginning, so that each ${var:+if_true} expression starts at the beginning of a line

lib/sh/db.sh: mk_select: constructed queries: support custom columns list using $cols

lib/sh/db.sh: mk_select: constructed queries: support WHERE clause using $filter

lib/sh/util.sh: echo_stdin(): don' increase the log_level, because the input being sent to the command (which is usually a set of interpreted commands itself) is necessary to fully know what action is being performed

bugfix: lib/sh/local.sh: add a manual errexit for $() exprs by embedding them in just a var assignment (without local or declare), whose exit status will then equal the of the $(). a `|| return` also needs to be added because errexit does not work on assignment statements. this commit adds them for func_loc(), echo_func(), canon_rel_path(), set_paths(), save_cache, cached realpath(), local.sh global vars

lib/sh/util.sh: shell-variable-based caching: usage: updated alias names

web/links/index.htm: updated to Firefox bookmarks. regenerating also removed extra <a name=...> tags that were added when running index.htm.run on an already-processed index.htm.

bugfix: lib/sh/util.sh: `set -o`: need a -o before every option to set

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names: hyperlink the added anchors as clickable paragraph marks (like Redmine), which take you to the HTML anchor. this is analogous to the clickable folder names which take you to their anchors.

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names, in addition to their URLs

bugfix: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' URLs: use .*? instead of .* to match the contents of the <A> tag before the HREF

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

web/links/index.htm: updated to Firefox bookmarks

lib/sh/util.sh: `set -o`: added pipefail option, to ensure that exit statuses (esp. for errexit) also work with pipelines (a|b)

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++. use the command form of log-- (`log-- echo_func`) to counter the normal log++ performed by echo_func, for cases when the function name is descriptive and should be output at the same log_level as the commands it runs.

*{.sh,run}: removed extra space between function name and ()

lib/sh/util.sh: echo_func(): take the FUNCNAME as an argument (auto-added in the echo_func alias) instead of getting it from the FUNCNAME array (which would have produced an inaccurate value if another function call (such as log++) intervened between the caller and echo_func())

lib/sh/util.sh: echo_func(): display where the function was declared (using new func_loc()) instead of where echo_func() was called from. this is more intuitive when debugging, becaues the line # is where the function starts. it also helps remove the dependency on the FUNCNAME/BASH_* arrays, which would produce an inaccurate value if another function call (such as log++) intervened between the caller and echo_func().

lib/sh/util.sh: added func_loc(), which gets where a function was declared in the format file:line, and helper alias set_func_loc

lib/sh/util.sh: str2varname(): use bash's internal ${var//glob/repl} syntax, which supports character classes ([a-z], etc.), instead of sed (which is slower because it's an external command and uses regexps)

lib/sh/util.sh: added support for caching realpath() using the new shell-variable-based caching. this can be enabled via $realpath_cache and defaults to off because it's currently slower than without. note that the cache needs to be cleared in cd() because relative paths will become invalid.

lib/sh/util.sh: added str2varname() and use it in include_guard_var()

lib/sh/util.sh: added shell-variable-based caching functions

lib/sh/util.sh: canon_rel_path(): use $PWD instead of the slower $(pwd -P) now that $PWD has symlinks expanded

lib/sh/util.sh: expand symlinks in $PWD: moved it before canon_rel_path(), which will require it

lib/sh/util.sh: expand symlinks in $PWD using `cd -P .` so it matches the output of realpath

lib/sh/util.sh: cd(): documented what -P option does and why it's used

lib/sh/util.sh: cd(): always run cd with -P (expand symlinks) so that $PWD will contain a canonical path which matches the output of `readlink -f`. however, don't display the added -P in the logging output because it distracts from the directory being changed to.

lib/sh/util.sh: logging: added log+/-, which take a variable log_level step, and use them in log++/--

lib/sh/util.sh: repeat(): for simplicity and speed, just append to a local string var (and echo the result at the end) instead of using printf

bugfix: lib/sh/util.sh: repeat(): need to use %s instead of %q (escaped string) in printf

lib/sh/util.sh: logging: log++/--: comment about the different uses can be performed with one function: clarified that without a cmd, "$@" expands to nothing and assignments are applied to the caller

lib/sh/util.sh: log++/--: support running a command with the given log++/-- setting instead of applying it in the calling context. note that with no args, "$@" will expand to nothing and the var assignments will be applied in the calling context instead of to an executed command. this requires using $(()) syntax instead of let! to set the verbosity.

lib/sh/util.sh: logging: inlined PS4 functions into log++/-- because that's the only place they're used and it greatly simplifies the code

lib/sh/util.sh: log++/-- (and helpers PS4++/--): added functions for these so they can also be used in other contexts that don't support aliases. note that direct callers of the functions would still need to localize the PS4/$verbosity values using log_local.

lib/sh/util.sh: run_args_cmd(): moved test of $? to lib/runscripts/util.run on_exit() since it needs to be performed for all run_args_cmd() functions, not just the default implementation. (this test ensures that no commands are executed if the EXIT trap was encountered due to a parsing/loading error).

bugfix: lib/sh/util.sh: fwd(): need to use $top_dir (the runscript's file) instead of "${BASH_SOURCE¹}" (the caller's file)

lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd and can_log: don't call set_fds because no redirection is needed. this also avoids the need to increase the log_level of the set_fds call that used to set cmd_log_fd to itself.

bugfix: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd: need to increase the log_level after can_log is checked, to avoid limiting the command output itself

bugfix: lib/sh/util.sh: command__set_fds(): log the command output at the same log_level as the command name (i.e. leave the log_level as-is), to allow nesting verbose commands within one another without the inner commands always getting suppressed

lib/sh/util.sh: command__set_fds(): set the src fd outside of the set_fds call, for clarity

lib/sh/util.sh: command(): if cmd_log_fd = log_fd, log the filtering of cmd_log_fd at 2 log_levels higher (verbosity >= 3)

lib/sh/util.sh: command(): documented that $cmd_log_fd's default (no cmd_log_fd) limits log_fd in case command uses util.sh

bugfix: lib/sh/util.sh: command(): need to make $cmd_log_fd a local var when setting its default value

lib/sh/util.sh: logging: moved indent for call tree to its own section, so it isn't confused with the log_level

lib/sh/util.sh: logging: log_level definition: added instructions to view a message's log_level (count the # of + signs before it in the output)

lib/sh/util.sh: logging: log_level definition: replaced command with message because not just commands can be logged

lib/sh/util.sh: logging: log_level definition: emphasized that the log_level starts with 1, not 0 (0 has a special meaning, which refers to unfiltered messages)

lib/sh/util.sh: logging: log_level definition: documented that for commands that use can_log(), the log_level starts with 1. for unfiltered commands, the log_level is 0 (i.e. still output at verbosity=0).

lib/sh/util.sh: logging: added definition that the log_level is the minimum verbosity needed to display a command

lib/sh/util.sh: set_global_fds(): documented that the logging output for this is visible at verbosity >= 3

lib/sh/util.sh: set_global_fds(): moved stdlog fd setup to separate setup_log_fd() function, which runs at a much higher log_level (5 instead of 3). because the stdlog fd setup statements themselves are logged to fd 2 (because fd 30 is not yet open), you normally don't want to display these, if you are using fd 2 for errors only and fd 30 for logging.

lib/sh/util.sh: command(): $cmd_log_fd: if not set, assume it's a script using util.sh (i.e. with log fd=$log_fd)

lib/sh/util.sh: set_global_fds(): also open stdlog logging stream on fd 30, and use this in the logging functions by setting $log_fd

lib/sh/util.sh: moved streams section before external commands echoing because command() uses set_fds

lib/sh/util.sh: moved echo_eval() to general command echoing section so it can be used by other echoing commands

lib/sh/util.sh: don't load new aliases after defining echo_stdout because it's no longer an alias

lib/sh/util.sh: load new aliases after defining echo_func

lib/sh/util.sh: .(): use "$FUNCNAME" like cd()

lib/sh/util.sh: .(): wrapped lines in the same way as cd()

lib/sh/util.sh: moved functions echoing section before other types of echoing so they can use echo_func()

lib/sh/util.sh: $verbosity: decreased default value to 1 so that normally, only external commands are echoed. this avoids cluttering up the output with the function call tree, which is really only needed for debugging. it also mirrors make's default verbosity (i.e. print external commands and subdir recursions, but not make's internal actions).

*{.sh,run}: put doc comment on next line when possible, so that the function name always comes before it

lib/sh/util.sh: put usage message at end of line when possible, so that the function name always comes before it

bugfix: lib/sh/util.sh: $top_script: need to realpath this before doing any cd so this points to the right place. set and use $top_script_abs for this purpose.

lib/runscripts/util.run: added setting of $top_file, $top_filename to set_paths()

lib/runscripts/util.run: set $top_file, $top_filename in a function (set_paths__util_run())

lib/sh/util.sh: cd(): re-run set_paths() to update $top_script, $top_dir for the new current dir

lib/sh/util.sh: added set_paths(), which sets $top_script, $top_dir

lib/sh/util.sh: moved $top_script, $top_dir before internal commands echoing so they can be reset in cd()

lib/sh/util.sh: moved vars echoing before internal commands echoing so that echo_vars can be used in it

lib/sh/util.sh: use set_fds() instead of `echo_eval exec` for clarity

lib/sh/util.sh: set_fd(): allow setting multiple fds at once, and changed the function name accordingly

lib/sh/util.sh: set_fd(): moved noclobber functionality to fd_set_default() since that is only used by fd_set_default() (whose purpose has become just to run set_fd without clobbering an existing dest fd)

lib/sh/util.sh: shadow_fd(): renamed to fd_set_default() for clarity (since this can't just be used for shadowing stdin/stdout/stderr)

lib/sh/util.sh: set_fd()/shadow_fd(): take the redirect expr as an arg instead of a kw_param because it's the only param

lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect expr containing the entire dest[<>]src expr, and parse the dest out from it

lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params

lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params

lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect source expr containing the <> and the fd, to allow the caller to specify these in the shell redirection syntax. this also allows using a file (such as /dev/null) as the redirect source.

lib/sh/util.sh: added cd() wrapper that logs itself and indents (indent is permanent within subshell cd was used in)

lib/sh/util.sh: added caller_indent alias

lib/sh/util.sh: log_err(): changed symbol to # instead of * so that it stands out better at the outmost indent level and doesn't blend in with the + that will be prefixing other messages

lib/sh/util.sh: log_info(): changed symbol to : so that it blends in better with the | lines (looking like a dotted segment of the line), but still stands out (because of the dots)

*{.sh,run}: don't use aliases unless it's actually necessary (e.g. to define a var in the calling function, or to alias-expand the next word)

lib/runscripts/util.run: $top_file: also match dir-level runscripts by removing /run suffix

lib/sh/util.sh: shadow_fd(): take a dest fd instead of a prefix to prepend to the src fd, so that the full name of the dest fd is in the function call. this makes it much clearer what fd will be created from a shadow_fd() call.

lib/sh/util.sh: rel_path(): also remove base_dir when path exactly = to it. this involves adding and removing an extra trailing /.

bugfix: lib/sh/util.sh: rel_path(): ensure non-empty by using . if empty

lib/sh/util.sh: split canon_rel_path() into itself and rel_path() helper for clarity

lib/sh/local.sh: $root_dir: run canon_rel_path on it

lib/sh/util.sh: log_custom(): use keyword-args syntax instead of local vars