Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9371 05/15/2013 01:42 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9370 05/15/2013 01:39 PM Aaron Marcuse-Kubitza

web/links/index.htm: Mac 10.8 Mountain Lion > PostgreSQL: appended tab to name to disambiguate it from the general PostgreSQL section

9369 05/15/2013 01:33 PM Aaron Marcuse-Kubitza

bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: also renamed the corresponding self-anchor hyperlinks

9368 05/15/2013 01:32 PM Aaron Marcuse-Kubitza

bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: need to replace all occurrences (using /g option to sed) to include both HTML anchors on the line

9367 05/15/2013 01:26 PM Aaron Marcuse-Kubitza

web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name

9366 05/15/2013 01:23 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: sed: changed it to an alias so it will also be expanded when passed to an external command (like in_place) that can only run an executable, not a shell function (this occurs as long as the external command is defined as an alias which ends in space, to alias-expand the next word). added associated $sed_cmd var for cases when there is no alias wrapper around the external command, and the literal alias body must be used instead.

9365 05/15/2013 11:02 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+/-(): setting verbosity: added space around operators to support negative numbers

9364 05/15/2013 10:55 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added float functions (esp. float+int()) for dealing with decimal verbosities used by sql.py and column-based import

9363 05/15/2013 10:53 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: inputs/GBIF/raw_occurrence_record/run: added check_target_exists so you know why make skipped the file (for other, non-silent targets, it would also avoid make's verbose output when the file exists)

9362 05/15/2013 10:38 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): moved making of table.tsv.md5 to separate function

9361 05/15/2013 10:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: $log_fd: indicated that this is only initially stderr (however, the new port will just use stderr if it's not redirected separately)

9360 05/15/2013 10:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbosities: 3: added that this includes values of kw params

9359 05/15/2013 10:25 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added description of the verbosities available, including what each one does and what it's useful for:

  1. verbosities:
  2. 0: just print errors. useful for cron jobs.
  3. 1: also external commands run. useful for running at the command line....
9358 05/15/2013 10:11 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: command__set_fds(): echo the >& line at a higher log_level, because this information (i.e. which fd is used by the command for logging) is primarily for debugging and should not normally be printed

9357 05/15/2013 10:00 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): also add md5 sum for table.tsv

9356 05/15/2013 09:50 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added back filter kw args, which had gotten deleted in a commit without update (although actually, svn should not allow a commit without update, so the working copy may have gotten corrupted)

9355 05/15/2013 09:27 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added $silent flag which turns on make's --silent option

9354 05/15/2013 09:25 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: set_make_vars: usage: added "use $target" to indicate that vars are made available by this alias

9353 05/15/2013 09:24 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: inline_make(): take the script stdin from caller-provided stdin and get make's stdin from global stdin, so that the caller can just use <<'EOF' rather than having to include a specific fd before the <<

9352 05/15/2013 08:46 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): documented that it requires a Makefile in the datasrc dir containing `include ../input.Makefile

9351 05/15/2013 08:41 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added comment explaining that "${@/#/$table/}" replaces the empty str at the beginning of str (/#) with $table/

9350 05/15/2013 08:38 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: put functions on one line where possible

9349 05/15/2013 08:35 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): use the local make() function instead of external make directly, because it sets $cmd_log_fd appropriately to ensure that all the echoed make commands get properly logged to stdlog/stderr (stdlog is fd 30 when it's redirected to a file)

9348 05/15/2013 08:22 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run table.tsv/make() and functions used by it: added usage comments for cmd line usage, caller usage, and declaring function usage

9347 05/15/2013 07:29 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9346 05/15/2013 07:27 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9345 05/15/2013 07:00 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed no longer needed - at the beginning of every folder's description

9344 05/15/2013 07:00 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: label page's description: don't do this for folders (i.e. descriptions preceded by an <H3> tag) because their descriptions are always author-added rather than from a web page. this avoids needing to add a - at the beginning of every folder's description.

9343 05/14/2013 09:43 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.

9342 05/14/2013 09:42 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.

9341 05/14/2013 08:56 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: added fixes for Apache ~/Sites dirs, Apache PHP

9340 05/14/2013 05:55 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added bookmarks for upgrading Mac OS X to 10.8 Mountain Lion. (WARNING: DO NOT upgrade unless you are prepared to fix several programs broken by the upgrade: svn, PostgreSQL. instructions are in the corresponding bookmark subdirs. these programs will be COMPLETELY UNAVAILABLE until they are manually fixed!)

9339 05/13/2013 07:43 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include scientific_name, which is preferable as a TNRS input because it also contains lower ranks

9338 05/13/2013 07:40 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include id, institution_code, collection_code, catalogue_number

9337 05/13/2013 07:38 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added filter for institution_codes in herbaria.ih (in PostgreSQL)

9336 05/13/2013 07:19 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added column subset (from http://vegpath.org/twiki/bin/view/Main/ConfCall20130509#subsetting_strategy > include)

9335 05/13/2013 07:14 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: in ${var:+if_true} syntax, put the newline at the end of the if_true value instead of the beginning, so that each ${var:+if_true} expression starts at the beginning of a line

9334 05/13/2013 07:10 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: support custom columns list using $cols

9333 05/13/2013 07:07 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: support WHERE clause using $filter

9332 05/13/2013 06:52 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_stdin(): don' increase the log_level, because the input being sent to the command (which is usually a set of interpreted commands itself) is necessary to fully know what action is being performed

9331 05/13/2013 06:30 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: add a manual errexit for $() exprs by embedding them in just a var assignment (without local or declare), whose exit status will then equal the of the $(). a `|| return` also needs to be added because errexit does not work on assignment statements. this commit adds them for func_loc(), echo_func(), canon_rel_path(), set_paths(), save_cache, cached realpath(), local.sh global vars

9330 05/13/2013 06:14 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: shell-variable-based caching: usage: updated alias names

9329 05/13/2013 05:33 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. regenerating also removed extra <a name=...> tags that were added when running index.htm.run on an already-processed index.htm.

9328 05/13/2013 05:19 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: `set -o`: need a -o before every option to set

9327 05/13/2013 05:10 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names: hyperlink the added anchors as clickable paragraph marks (like Redmine), which take you to the HTML anchor. this is analogous to the clickable folder names which take you to their anchors.

9326 05/13/2013 04:52 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names, in addition to their URLs

9325 05/13/2013 04:47 AM Aaron Marcuse-Kubitza

bugfix: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' URLs: use .*? instead of .* to match the contents of the <A> tag before the HREF

9324 05/13/2013 04:45 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

9323 05/13/2013 04:43 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

9322 05/13/2013 04:35 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9321 05/13/2013 04:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: `set -o`: added pipefail option, to ensure that exit statuses (esp. for errexit) also work with pipelines (a|b)

9320 05/13/2013 04:13 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++

9319 05/13/2013 04:06 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++. use the command form of log-- (`log-- echo_func`) to counter the normal log++ performed by echo_func, for cases when the function name is descriptive and should be output at the same log_level as the commands it runs.

9318 05/12/2013 08:33 PM Aaron Marcuse-Kubitza

*{.sh,run}: removed extra space between function name and ()

9317 05/12/2013 08:13 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func(): take the FUNCNAME as an argument (auto-added in the echo_func alias) instead of getting it from the FUNCNAME array (which would have produced an inaccurate value if another function call (such as log++) intervened between the caller and echo_func())

9316 05/12/2013 08:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func(): display where the function was declared (using new func_loc()) instead of where echo_func() was called from. this is more intuitive when debugging, becaues the line # is where the function starts. it also helps remove the dependency on the FUNCNAME/BASH_* arrays, which would produce an inaccurate value if another function call (such as log++) intervened between the caller and echo_func().

9315 05/12/2013 07:57 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added func_loc(), which gets where a function was declared in the format file:line, and helper alias set_func_loc

9314 05/12/2013 07:09 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: str2varname(): use bash's internal ${var//glob/repl} syntax, which supports character classes ([a-z], etc.), instead of sed (which is slower because it's an external command and uses regexps)

9313 05/12/2013 07:00 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added support for caching realpath() using the new shell-variable-based caching. this can be enabled via $realpath_cache and defaults to off because it's currently slower than without. note that the cache needs to be cleared in cd() because relative paths will become invalid.

9312 05/12/2013 06:48 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added str2varname() and use it in include_guard_var()

9311 05/12/2013 06:47 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added shell-variable-based caching functions

9310 05/12/2013 05:08 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: canon_rel_path(): use $PWD instead of the slower $(pwd -P) now that $PWD has symlinks expanded

9309 05/12/2013 05:06 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: expand symlinks in $PWD: moved it before canon_rel_path(), which will require it

9308 05/12/2013 05:03 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: expand symlinks in $PWD using `cd -P .` so it matches the output of realpath

9307 05/12/2013 05:01 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: cd(): documented what -P option does and why it's used

9306 05/12/2013 04:55 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: cd(): always run cd with -P (expand symlinks) so that $PWD will contain a canonical path which matches the output of `readlink -f`. however, don't display the added -P in the logging output because it distracts from the directory being changed to.

9305 05/12/2013 04:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added log+/-, which take a variable log_level step, and use them in log++/--

9304 05/12/2013 04:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: repeat(): for simplicity and speed, just append to a local string var (and echo the result at the end) instead of using printf

9303 05/12/2013 04:12 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: repeat(): need to use %s instead of %q (escaped string) in printf

9302 05/12/2013 03:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log++/--: comment about the different uses can be performed with one function: clarified that without a cmd, "$@" expands to nothing and assignments are applied to the caller

9301 05/12/2013 03:00 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log++/--: support running a command with the given log++/-- setting instead of applying it in the calling context. note that with no args, "$@" will expand to nothing and the var assignments will be applied in the calling context instead of to an executed command. this requires using $(()) syntax instead of let! to set the verbosity.

9300 05/12/2013 02:48 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: inlined PS4 functions into log++/-- because that's the only place they're used and it greatly simplifies the code

9299 05/12/2013 02:34 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log++/-- (and helpers PS4++/--): added functions for these so they can also be used in other contexts that don't support aliases. note that direct callers of the functions would still need to localize the PS4/$verbosity values using log_local.

9298 05/12/2013 02:08 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: run_args_cmd(): moved test of $? to lib/runscripts/util.run on_exit() since it needs to be performed for all run_args_cmd() functions, not just the default implementation. (this test ensures that no commands are executed if the EXIT trap was encountered due to a parsing/loading error).

9297 05/12/2013 01:54 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: fwd(): need to use $top_dir (the runscript's file) instead of "${BASH_SOURCE1}" (the caller's file)

9296 05/12/2013 01:38 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd and can_log: don't call set_fds because no redirection is needed. this also avoids the need to increase the log_level of the set_fds call that used to set cmd_log_fd to itself.

9295 05/12/2013 01:30 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd: need to increase the log_level after can_log is checked, to avoid limiting the command output itself

9294 05/12/2013 01:23 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command__set_fds(): log the command output at the same log_level as the command name (i.e. leave the log_level as-is), to allow nesting verbose commands within one another without the inner commands always getting suppressed

9293 05/12/2013 01:08 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command__set_fds(): set the src fd outside of the set_fds call, for clarity

9292 05/12/2013 01:02 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): if cmd_log_fd = log_fd, log the filtering of cmd_log_fd at 2 log_levels higher (verbosity >= 3)

9291 05/12/2013 12:57 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): documented that $cmd_log_fd's default (no cmd_log_fd) limits log_fd in case command uses util.sh

9290 05/12/2013 12:54 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command(): need to make $cmd_log_fd a local var when setting its default value

9289 05/12/2013 12:49 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: moved indent for call tree to its own section, so it isn't confused with the log_level

9288 05/12/2013 12:40 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log_level definition: added instructions to view a message's log_level (count the # of + signs before it in the output)

9287 05/12/2013 12:39 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log_level definition: replaced command with message because not just commands can be logged

9286 05/12/2013 12:38 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log_level definition: emphasized that the log_level starts with 1, not 0 (0 has a special meaning, which refers to unfiltered messages)

9285 05/12/2013 12:34 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log_level definition: documented that for commands that use can_log(), the log_level starts with 1. for unfiltered commands, the log_level is 0 (i.e. still output at verbosity=0).

9284 05/12/2013 12:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added definition that the log_level is the minimum verbosity needed to display a command

9283 05/12/2013 12:22 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set_global_fds(): documented that the logging output for this is visible at verbosity >= 3

9282 05/12/2013 12:21 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set_global_fds(): moved stdlog fd setup to separate setup_log_fd() function, which runs at a much higher log_level (5 instead of 3). because the stdlog fd setup statements themselves are logged to fd 2 (because fd 30 is not yet open), you normally don't want to display these, if you are using fd 2 for errors only and fd 30 for logging.

9281 05/12/2013 12:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): $cmd_log_fd: if not set, assume it's a script using util.sh (i.e. with log fd=$log_fd)

9280 05/12/2013 12:00 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set_global_fds(): also open stdlog logging stream on fd 30, and use this in the logging functions by setting $log_fd

9279 05/12/2013 11:51 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved streams section before external commands echoing because command() uses set_fds

9278 05/12/2013 11:50 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved echo_eval() to general command echoing section so it can be used by other echoing commands

9277 05/12/2013 11:45 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: don't load new aliases after defining echo_stdout because it's no longer an alias

9276 05/12/2013 11:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: load new aliases after defining echo_func

9275 05/12/2013 11:36 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: .(): use "$FUNCNAME" like cd()

9274 05/12/2013 11:36 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: .(): wrapped lines in the same way as cd()

9273 05/12/2013 11:34 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved functions echoing section before other types of echoing so they can use echo_func()

9272 05/12/2013 11:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: $verbosity: decreased default value to 1 so that normally, only external commands are echoed. this avoids cluttering up the output with the function call tree, which is really only needed for debugging. it also mirrors make's default verbosity (i.e. print external commands and subdir recursions, but not make's internal actions).