Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9128 05/11/2013 05:12 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function

9127 05/11/2013 05:05 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.

9126 05/11/2013 05:05 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr

9125 05/11/2013 05:04 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers

9124 05/11/2013 05:01 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_minor_func alias

9123 05/11/2013 05:00 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_eval()

9122 05/11/2013 04:32 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file doesn't exist

9121 05/11/2013 02:26 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places

9120 05/11/2013 02:23 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting

9119 05/11/2013 02:19 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.

9118 05/11/2013 01:55 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: propagate $log_level_indent to invoked commands

9117 05/11/2013 01:54 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent

9116 05/11/2013 01:49 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().

9115 05/11/2013 01:44 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)

9114 05/11/2013 01:32 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded

9113 05/11/2013 01:31 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)

9112 05/11/2013 01:25 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace

9111 05/11/2013 01:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`

9110 05/11/2013 01:23 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbose output: put commands into subsections

9109 05/11/2013 01:10 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()

9108 05/11/2013 01:06 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run

9107 05/11/2013 01:00 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function

9106 05/11/2013 12:51 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query

9105 05/11/2013 12:49 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE

9104 05/11/2013 12:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands

9103 05/11/2013 12:33 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace

9102 05/11/2013 12:29 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE

9101 05/11/2013 12:22 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.

9100 05/11/2013 12:17 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database

9099 05/11/2013 12:15 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME1} in the function, so that callers can also call mysql_cmd via a nested function, etc.

9098 05/11/2013 12:00 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added ensure_nested_func()

9097 05/10/2013 11:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added contains()

9096 05/10/2013 11:57 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin

9095 05/10/2013 10:16 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)

9094 05/10/2013 10:13 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()

9093 05/10/2013 10:11 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it must be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.

9092 05/10/2013 10:07 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call

9091 05/10/2013 10:05 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: import_vars alias: use new get_prefix_vars()

9090 05/10/2013 10:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: add get_prefix_vars()

9089 05/10/2013 10:03 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function

9088 05/10/2013 02:47 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added mk_select alias and use it in pg_export()

9087 05/10/2013 02:36 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query

9086 05/10/2013 02:27 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_export(): rtrim $query

9085 05/10/2013 02:22 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added rtrim()

9084 05/10/2013 01:55 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.

9083 05/10/2013 01:34 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL

9082 05/10/2013 01:23 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it

9081 05/10/2013 12:55 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added repeat()

9080 05/10/2013 12:47 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops

9079 05/10/2013 12:43 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions

9078 05/10/2013 12:32 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented

9077 05/10/2013 12:27 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: propagate indent to invoked commands by exporting PS4

9076 05/10/2013 12:24 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level

9075 05/10/2013 12:14 PM Aaron Marcuse-Kubitza

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

9074 05/10/2013 12:08 PM Aaron Marcuse-Kubitza

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

9073 05/10/2013 12:03 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type

9072 05/10/2013 12:00 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())

9071 05/10/2013 11:58 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): use new log_err

9070 05/10/2013 11:57 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols

9069 05/10/2013 11:55 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved die () into verbose output so it can use logging functions

9068 05/10/2013 11:53 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks

9067 05/10/2013 11:41 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions

9066 05/10/2013 11:29 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump

9065 05/10/2013 11:23 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_cmd (): use new log ()

9064 05/10/2013 11:21 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added log ()

9063 05/10/2013 10:53 AM Aaron Marcuse-Kubitza

lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.

9062 05/10/2013 10:51 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added require_exists (), used to skip make commands for existing files

9061 05/10/2013 10:33 AM Aaron Marcuse-Kubitza

*{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating

9060 05/10/2013 10:31 AM Aaron Marcuse-Kubitza

/run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet

9059 05/10/2013 10:31 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)

9058 05/10/2013 10:29 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists

9057 05/10/2013 10:08 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()

9056 05/10/2013 10:06 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them

9055 05/10/2013 10:02 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist

9054 05/10/2013 10:01 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: added remaking alias

9053 05/10/2013 09:45 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

9052 05/10/2013 09:45 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)

9051 05/10/2013 09:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.

9050 05/10/2013 09:06 AM Aaron Marcuse-Kubitza

*{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)

9049 05/10/2013 08:52 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level

9048 05/10/2013 08:45 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself

9047 05/10/2013 08:41 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

9046 05/10/2013 08:40 AM Aaron Marcuse-Kubitza

lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh

9045 05/10/2013 08:32 AM Aaron Marcuse-Kubitza

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

9044 05/10/2013 08:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

9043 05/10/2013 08:27 AM Aaron Marcuse-Kubitza

/run: moved geoscrub_input export into separate geoscrub_input/make () target

9042 05/10/2013 08:24 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

9041 05/10/2013 07:51 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

9040 05/10/2013 07:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved canon_rel_path () into separate paths section

9039 05/10/2013 07:39 AM Aaron Marcuse-Kubitza

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

9038 05/10/2013 07:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added func_override (), for use in runscript inheritance

9037 05/10/2013 07:03 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

9036 05/10/2013 07:00 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: exceptions: added die ()

9035 05/10/2013 06:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

9034 05/10/2013 06:35 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

9033 05/10/2013 06:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

9032 05/10/2013 06:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added save_e, which now sets $e just locally

9031 05/10/2013 06:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

9030 05/10/2013 06:26 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer used save_e_cmd

9029 05/10/2013 06:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd