Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9240 05/12/2013 07:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved paths section after logging so it can use logging functions

9239 05/12/2013 06:10 AM Aaron Marcuse-Kubitza

*{.sh,run}: echo all global vars (except those defined before echo_vars is available)

9238 05/12/2013 05:58 AM Aaron Marcuse-Kubitza

web/links/index.htm.run: use the runscript-common $top_file, which is equivalent to $linked_file

9237 05/12/2013 05:55 AM Aaron Marcuse-Kubitza

*{.sh,run}: use new local_export instead of local...export. this still requires calling echo_vars manually, because local_export is an alias (and must be, to define the var in the caller's context).

9236 05/12/2013 05:54 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added local_export alias, which combines the effects of local and export

9235 05/12/2013 05:36 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed params() to kw_params() to clarify the type of params it handles

9234 05/12/2013 05:33 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: echo all keyword params by overriding params(). this requires removing manual echo_vars calls.

9233 05/12/2013 05:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved functions section right after exceptions, so that other sections can use it. in particular, func_override is needed by verbose output to override non-verbose versions of functions.

9232 05/12/2013 05:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: only try to echo vars that are set, to avoid `declare -p` "not found" errors

9231 05/12/2013 05:09 AM Aaron Marcuse-Kubitza

: use new `params` in all functions that have keyword params, in order to remove their vars from the environment. note that functions that use $? (such as die()) must save it *before running params, because params will overwrite $?.

9230 05/12/2013 04:28 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added bash and MySQL-related bookmarks, including MySQL doc subfolder.

9229 05/12/2013 04:11 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: params(): added doc comment

9228 05/12/2013 04:10 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added params(), which removes keyword-param-only vars from the environment

9227 05/12/2013 03:50 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added unexport() (note that `declare +x` won't work because it defines the var if it isn't set)

9226 05/12/2013 03:39 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved vars section to beginning so other sections can use it (it does not contain logging statements, so it doesn't need to be after verbose output)

9225 05/12/2013 03:34 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command(): limit cmd_log_fd directly instead of limiting log_fd (usually fd 2) and then redirecting cmd_log_fd to it. this ensures that stderr is not limited unless it's actually used as the cmd_log_fd, so that a non-logging stderr will still reach the user properly regardless of whether err_fd is set to something other than 2. (previously, without an err_fd at 22, closing fd 2 at the beginning of the set of redirections would leave no fd pointing to a place to echo errors to.)

9224 05/11/2013 11:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: disable_logging(): use echo_eval instead of plain eval so the disabling of logging is nevertheless itself logged

9223 05/11/2013 11:11 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): made usage message an end-of-line comment

9222 05/11/2013 11:10 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: inlined limit_log_fd into command() since it's only used once

9221 05/11/2013 10:22 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_err(), log_info(): use bolder symbols (*, #) to make the symbol stand out visually from the line of | above and below it

9220 05/11/2013 10:09 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $verbosity: lowered the default from 3 to 2 because level 3 output (vars, etc.) is too verbose for the default output and is only needed for debugging

9219 05/11/2013 10:06 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $log_level_indent: added space after | to visually separate the nesting levels and make the output look like an expandable tree (similar to a filesystem browser tree)

9218 05/11/2013 10:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_custom(): replace every non-whitespace character of $log_indent, and the first character of PS4, with the symbol, to make the symbol stand out on the line instead of the |||... before it

9217 05/11/2013 09:19 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer used stdout2log_fd(). set $cmd_log_fd instead.

9216 05/11/2013 09:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set $log_level_indent to | because this visually connects commands at the same indent by effectively connecting their + prefixes with a line of |

9215 05/11/2013 09:14 PM Aaron Marcuse-Kubitza

lib/sh/archives.sh: zip(), unzip(): added missing echo_func

9214 05/11/2013 09:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): use just one control var $cmd_log_fd instead of three flags (limit_log_fd, limit_stdout, stderr_is_errors) that indicated various common fd configurations. this is much clearer (you state which fd the common uses as its logging fd), more configurable (the logging fd can be any fd, not just 1 or 2), and more automatic (redirecting fd 2 to err_fd happens automatically if it isn't used for logging).

9213 05/11/2013 08:58 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): need to use limit_stdout=1 instead of stderr_is_errors=1 because logging info is output to stdout, not stderr, in spite of the >&2 redirection, since the >&2 is applied after command()'s logging redirects. note that the >&2 can now be removed, because it will be performed by command().

9212 05/11/2013 08:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $log_fd and use it instead of 2 where the log_fd is intended

9211 05/11/2013 08:11 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved stdout2log_fd() to command verbose output subsection of verbose output since it is primarily intended for use with external commands

9210 05/11/2013 08:05 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed limit_stderr to limit_log_fd, stdout2stderr to stdout2log_fd to clarify their purpose. log_fd is currently assumed to be 2.

9209 05/11/2013 08:00 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added disable_logging() and use it instead of `exec 2>/dev/null`

9208 05/11/2013 07:54 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $err_fd global var and use it instead of the 22 magic value. it starts out at 2 until fd 22 has been set up, so commands that required 22 will now work before set_global_fds() is called.

9207 05/11/2013 07:11 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: shadow_fd(): don't require it to be run with log++ and instead let callers apply log++ if needed. in set_global_fds(), log++ can just be applied once for all the shadow_fd() calls.

9206 05/11/2013 07:09 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: shadow_fd(): instead of applying minor=1 (i.e. an additional log++) to echo_func and then log++ again separately to set_fd, just apply it to the entire function

9205 05/11/2013 07:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set_fd(): don't increase the log_level for this function, and instead let callers increase it if needed

9204 05/11/2013 06:48 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): inlined log++ with the can_log call because it should only apply to it. this also makes it obvious that log++ should not apply outside of this function. this also enables the contents of this function to be used in an alias.

9203 05/11/2013 06:01 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): use exec instead of `builtin command` to avoid spawning an extra process (since a subshell is already open)

9202 05/11/2013 05:43 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command(): only indent if command was logged

9201 05/11/2013 05:39 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: .(): only indent if include statement was logged

9200 05/11/2013 05:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): moved $stdout2stderr to command() so that limit_stderr()'s only purpose is to log-limit stderr

9199 05/11/2013 05:23 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr(): moved $stderr_is_errors default to command() where it's used

9198 05/11/2013 05:20 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): do stdout2stderr after limit_stderr to avoid having two similar `can_log ... >/dev/null` commands

9197 05/11/2013 05:09 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_cmd(): removed no longer used removal of first builtin or command arg

9196 05/11/2013 05:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: inlined echo_run_prep into command() since it's now only used once

9195 05/11/2013 05:06 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): removed no longer used $cmd_echoed flag

9194 05/11/2013 04:57 PM Aaron Marcuse-Kubitza

*{.sh,run}: use new limit_stdout=1 instead of stdout2stderr=1 limit_stderr=1

9193 05/11/2013 04:56 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): added limit_stdout=1 flag which implies stdout2stderr=1 limit_stderr=1

9192 05/11/2013 04:41 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: merged limit_stderr_cmd/limit_stdout_cmd into command(), using flag vars to control what limiting actions it needs to perform. in command invocations, this involves setting the appropriate flag vars instead of using a limit_std*_cmd alias.

9191 05/11/2013 04:37 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added run_cmd

9190 05/11/2013 04:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): moved before command echoing because it does not depend on it and is a separate feature

9189 05/11/2013 03:47 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: run_args_cmd(): use $top_script instead of `"$(canon_rel_path "$0")"`

9188 05/11/2013 03:42 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $top_script: run canon_rel_path on $0

9187 05/11/2013 03:35 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: use `minor=1 echo_func` instead of echo_minor_func because the syntax is simple enough that you don't need a separate alias for it

9186 05/11/2013 03:32 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func: support using minor=1 to increase the log_level that the function call is echoed at. use this in echo_minor_func.

9185 05/11/2013 03:28 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.

9184 05/11/2013 03:27 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.

9183 05/11/2013 03:18 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func: only indent if the function call was displayed. this avoids confusing extra whitespace when a function call causes an indent but the function call itself isn't printed.

9182 05/11/2013 03:10 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: echo_minor_func: need to include "$@" in echo_func() call

9181 05/11/2013 03:08 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func(): added usage

9180 05/11/2013 03:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_func: added usage

9179 05/11/2013 02:50 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $log_level_indent: changed to a single space rather than two, to avoid very deep indents as the call tree becomes more deeply nested. due to the + at the beginning of every line, a single space is enough to visualize the structure of the call tree (unlike source code, which requires more whitespace because it doesn't have a call tree structure).

9178 05/11/2013 02:46 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_*func aliases: increase the indent for the duration of the function call. this causes function calls to be displayed in an indented call-tree structure, which makes the verbose output much easier to understand.

9177 05/11/2013 02:43 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: recursive aliases (i.e. aliases that call a command of the same name): use "" around the command name so it isn't reexpanded if the function using the alias is copied using copy_func(). (aliases will be reexpanded every time a function is redeclared since they appear unquoted in the function definition.) this is a bug in `declare -f` for which there is no fix, necessitating the "" alias workaround instead.

9176 05/11/2013 02:01 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved log-indent-related vars/aliases before they are first used (by log())

9175 05/11/2013 01:54 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--

9174 05/11/2013 01:54 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--

9173 05/11/2013 01:51 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed cmd_indent to just indent now that indent is not being used for the PS4 level

9172 05/11/2013 01:47 PM Aaron Marcuse-Kubitza

*{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||

9171 05/11/2013 01:44 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed indent->PS4++ , outdent->PS4-- because these relate just to the PS4 length, not to the (whitespace-based) log_indent

9170 05/11/2013 01:38 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer used echo_run(). use command() instead.

9169 05/11/2013 01:36 PM Aaron Marcuse-Kubitza

*{.sh,run}: use command instead of deprecated echo_run (don't prepend anything when the command is already aliased with `command`)

9168 05/11/2013 01:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): perform echo_run_prep itself instead of requiring echo_run to be added in an alias. echo_run_prep will not be performed if limit_stderr_cmd() has performed it already outside the limit_stderr section. this requires using `builtin command` instead of "command" when you don't want the echoing (such as in include_guard_var()).

9167 05/11/2013 01:07 PM Aaron Marcuse-Kubitza

*{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||

9166 05/11/2013 12:56 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: .(): also need to echo the . before the included file

9165 05/11/2013 12:48 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_params alias and use it instead of `log "$*"`

9164 05/11/2013 12:43 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: .(): echo the include at log_level 2 because it's an internal command

9163 05/11/2013 12:34 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_cmd(): also remove `builtin` before the command name

9162 05/11/2013 12:32 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_cmd(): don't remove -- after command, because it is added by the command() function instead and does not appear in the command sent to echo_run()

9161 05/11/2013 12:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command alias: don't alias-expand next word, because the next word should only be interpreted as a command (part of the effect of the `command` builtin)

9160 05/11/2013 12:27 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: external commands: always use command rather than echo_run to indicate that a command is external, because command() ensures that what it invokes is not a shell function, and sets 2>&22 where needed

9159 05/11/2013 12:23 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: .(): use echo_run so that included files are echoed as they are included. note that echo_run uses cmd_indent, so the included files' load-time logging output will be indented according to the nesting level of the include.

9158 05/11/2013 12:20 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved .() command echoing > internal commands section

9157 05/11/2013 12:19 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved echo_run() to general command echoing section because it can also be used for internal commands

9156 05/11/2013 12:09 PM Aaron Marcuse-Kubitza

removed no longer used inputs/GBIF/MySQL_export. use lib/sh/local.sh mysql_export_local() instead.

9155 05/11/2013 12:07 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: override table.tsv/make() instead of export_mysql()

9154 05/11/2013 12:06 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export

9153 05/11/2013 12:05 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: added mysql_export_local()

9152 05/11/2013 12:05 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added mysql_export(). documented that mysql_export_outfile() supports CSV, but requires the FILE privilege.

9151 05/11/2013 12:02 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql(): use --column-names to ensure the output is formatted appropriately. note that --column-names is the default, but better to specify it to be sure.

9150 05/11/2013 12:01 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql(): output_data mode: use --batch to ensure the output is formatted appropriately. note that --batch is the default when stdin is from a pipe, but better to specify it to be sure.

9149 05/11/2013 11:49 AM Aaron Marcuse-Kubitza

*{.sh,run}: use new limit_stdout_cmd instead of `stdout2stderr=1 limit_stderr_cmd`

9148 05/11/2013 11:40 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added limit_stdout_cmd alias

9147 05/11/2013 07:25 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: added $rm short var for $remake

9146 05/11/2013 07:22 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbosity: removed `declare -i verbosity` because this does not actually ensure that $verbosity is an integer; it only has "arithmetic evaluation [...] done when the variable is assigned to" (`help declare`)

9145 05/11/2013 07:20 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbosity: support setting this in alternate short-name var $vb

9144 05/11/2013 07:15 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: override the `.` builtin (a.k.a. `source`) and run canon_rel_path on the included script. this removes .. in the path when it's displayed in bash error messages.

9143 05/11/2013 06:58 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_err(): ensure errors are visible by using 2>&22. since this is uses log(), verbosity=1 also needs to be set.

9142 05/11/2013 06:50 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: $explicit_errors_only: explicitly set verbosity=0 to hide startup logging

9141 05/11/2013 06:46 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): if stdout goes to stderr (i.e. is logging info), assume that stderr_is_errors