Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9045 05/10/2013 08:32 AM Aaron Marcuse-Kubitza

*{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity

9044 05/10/2013 08:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)

9043 05/10/2013 08:27 AM Aaron Marcuse-Kubitza

/run: moved geoscrub_input export into separate geoscrub_input/make () target

9042 05/10/2013 08:24 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error

9041 05/10/2013 07:51 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)

9040 05/10/2013 07:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved canon_rel_path () into separate paths section

9039 05/10/2013 07:39 AM Aaron Marcuse-Kubitza

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

9038 05/10/2013 07:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added func_override (), for use in runscript inheritance

9037 05/10/2013 07:03 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.

9036 05/10/2013 07:00 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: exceptions: added die ()

9035 05/10/2013 06:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.

9034 05/10/2013 06:35 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e

9033 05/10/2013 06:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e

9032 05/10/2013 06:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added save_e, which now sets $e just locally

9031 05/10/2013 06:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context

9030 05/10/2013 06:26 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer used save_e_cmd

9029 05/10/2013 06:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd

9028 05/10/2013 06:19 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.

9027 05/10/2013 06:15 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.

9026 05/10/2013 06:11 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?

9025 05/10/2013 06:08 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed

9024 05/10/2013 05:50 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added func_exists ()

9023 05/10/2013 05:50 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow

9022 05/10/2013 05:44 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added copy_func ()

9021 05/10/2013 05:39 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: save_e (): usage: added rethrow example

9020 05/10/2013 05:26 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded

9019 05/10/2013 05:24 AM Aaron Marcuse-Kubitza

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

9018 05/10/2013 05:19 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh

9017 05/10/2013 05:06 AM Aaron Marcuse-Kubitza

*{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.

9016 05/10/2013 05:01 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: split make utils out into separate make.sh

9015 05/10/2013 04:38 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: split archive (zip) utils out into separate archives.sh

9014 05/10/2013 04:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: split databases utils out into separate db.sh

9013 05/10/2013 04:05 AM Aaron Marcuse-Kubitza

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

9012 05/10/2013 03:32 AM Aaron Marcuse-Kubitza

lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.

9011 05/10/2013 03:29 AM Aaron Marcuse-Kubitza

lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)

9010 05/10/2013 03:13 AM Aaron Marcuse-Kubitza

lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.

9009 05/10/2013 03:07 AM Aaron Marcuse-Kubitza

lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice

9008 05/10/2013 02:56 AM Aaron Marcuse-Kubitza

lib/util.sh: limit_stderr_cmd: alias-expand command after it

9007 05/10/2013 02:53 AM Aaron Marcuse-Kubitza

lib/util.sh: $verbosity: ensure it's an integer using `declare -i`

9006 05/10/2013 02:45 AM Aaron Marcuse-Kubitza

lib/util.sh: grouped set verbosity statements together and commented them

9005 05/10/2013 02:43 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.

9004 05/10/2013 02:40 AM Aaron Marcuse-Kubitza

lib/util.sh: added bool2int ()

9003 05/10/2013 02:32 AM Aaron Marcuse-Kubitza

*{.sh,run}: use new isset

9002 05/10/2013 02:31 AM Aaron Marcuse-Kubitza

lib/util.sh: added isset ()

9001 05/10/2013 02:28 AM Aaron Marcuse-Kubitza

*{.sh,run}: use ${var+isset} instead of ${var+t} for clarity

9000 05/10/2013 02:22 AM Aaron Marcuse-Kubitza

lib/util.sh: propagate the verbosity to invoked commands by exporting it

8999 05/10/2013 02:20 AM Aaron Marcuse-Kubitza

lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.

8998 05/10/2013 02:15 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_stdin (): use new pipe_delay

8997 05/10/2013 02:12 AM Aaron Marcuse-Kubitza

lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)

8996 05/10/2013 02:06 AM Aaron Marcuse-Kubitza

lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed outside of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.

8995 05/10/2013 01:47 AM Aaron Marcuse-Kubitza

lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).

8994 05/10/2013 01:39 AM Aaron Marcuse-Kubitza

lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged

8993 05/10/2013 01:36 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function

8992 05/10/2013 01:25 AM Aaron Marcuse-Kubitza

inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.

8991 05/10/2013 01:00 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error

8990 05/10/2013 12:59 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists

8989 05/10/2013 12:58 AM Aaron Marcuse-Kubitza

lib/util.sh: make: added check_target_exists

8988 05/10/2013 12:42 AM Aaron Marcuse-Kubitza

lib/util.sh: moved $top_file* to runscripts/util.run because they only apply to runscripts

8987 05/10/2013 12:36 AM Aaron Marcuse-Kubitza

lib/util.sh: make: added to_target, which uses to_file on $target

8986 05/10/2013 12:35 AM Aaron Marcuse-Kubitza

lib/util.sh: added to_file (), which auto-removes a command's output file on error (like make's .DELETE_ON_ERROR)

8985 05/10/2013 12:31 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_run_extern, extern: fixed/added comments to indicate that echo_run_extern echoes and controls stderr of an external command, while the extern alias does this for all external commands

8984 05/10/2013 12:28 AM Aaron Marcuse-Kubitza

lib/util.sh: auto-echo common external commands, such as rm

8983 05/10/2013 12:24 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_run_extern, extern aliases: alias-expand next word (the command) by adding trailing space to alias def

8982 05/10/2013 12:21 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: exceptions: log_e: must include `declare e` in the alias and not when save_e is called, so that $e is a local var of the caller. this bug did not appear in testing because the save_e alias, which re-scopes $e within log_e (), was not expanded inside log_e () (since new aliases were not loaded between save_e and log_e ()).

8981 05/10/2013 12:12 AM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: added log_e (), which prints a "command exited with error" message (like make) when applicable

8980 05/10/2013 12:09 AM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: try_ (): renamed to try () and use the function keyword to distinguish it from the alias

8979 05/10/2013 12:06 AM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: try_ (): use new save_e

8978 05/10/2013 12:05 AM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: added save_e

8977 05/10/2013 12:01 AM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: end_try* aliases: use new rethrow*

8976 05/09/2013 11:59 PM Aaron Marcuse-Kubitza

lib/util.sh: exceptions: added rethrow, rethrow_subshell aliases

8975 05/09/2013 11:47 PM Aaron Marcuse-Kubitza

lib/util.sh: always use '' rather than "" around alias definitions, to ensure that variables are evaluated at expand time rather than compile time

8974 05/09/2013 11:44 PM Aaron Marcuse-Kubitza

lib/util.sh: zip_newer/unzip_newer: evaluate $no_force at alias expansion time rather than at alias compile time, so that the $no_force can be overridden in the context the alias is used in

8973 05/09/2013 11:18 PM Aaron Marcuse-Kubitza

lib/util.sh: use `|| return` instead of `|| exit` because `|| exit` doesn't seem to work inside functions (it does not have the errexit effect). also, `|| return` has the advantage of not exiting the program if the caller used || after the command (i.e. as an error handler) to temporarily disable errexit.

8972 05/09/2013 11:11 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: tell users to override run_args_cmd rather than this function to perform other commands, so that on_exit () can contain other exit-related processing that should not be overriden by the user

8971 05/09/2013 11:05 PM Aaron Marcuse-Kubitza

lib/util.sh: removed "" around $?, $# because they are guaranteed to always be non-empty and contain no special chars

8970 05/09/2013 10:17 PM Aaron Marcuse-Kubitza

lib/util.sh: say "last space" instead of "trailing space" so the comment will be more likely to fit at the end of the line

8969 05/09/2013 06:11 PM Aaron Marcuse-Kubitza

lib/local.sh: removed mysql () since its functionality is now provided by mysql () in util.sh

8968 05/09/2013 05:59 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): $server: like $ssh_server, clear the value when it's equal to this machine's hostname

8967 05/09/2013 05:58 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): $ssh_server: use new localize_url to clear the value when it's equal to this machine's hostname

8966 05/09/2013 05:37 PM Aaron Marcuse-Kubitza

lib/util.sh: added localize_url ()

8965 05/09/2013 05:36 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use extern so the MySQL_export command is echoed

8964 05/09/2013 05:24 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): don't make the target if it already exists

8963 05/09/2013 05:16 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): create the file directly rather than using inline_make, so that the command to create the file can use shell functions such as mysql

8962 05/09/2013 05:07 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv/make (): need to use $(target) (target's path relative to curdir) instead of just the filename, now that make does not change the directory

8961 05/09/2013 05:01 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: moved table.tsv creation to separate table.tsv/make () function

8960 05/09/2013 04:59 PM Aaron Marcuse-Kubitza

lib/util.sh: echo_stdin (): display the echoed input after leading output of the echoed-to command, by using sleep with a small fractional # of seconds

8959 05/09/2013 04:38 PM Aaron Marcuse-Kubitza

*{.sh,run}: put functions on one line where possible (and where they are not expected to expand)

8958 05/09/2013 04:32 PM Aaron Marcuse-Kubitza

lib/local.sh: root_make (): use make () rather than extern make now that make () does not set its own current dir. using make () also automatically enables \$stdout2stderr, which is needed to redirect the echoed make commands.

8957 05/09/2013 04:26 PM Aaron Marcuse-Kubitza

lib/util.sh: extern (): use exec to ensure that the executed command is in fact external, and not a shell builtin or other type of non-function command. this also simplifies the code because any shell function for the command does not need to be unset.

8956 05/09/2013 04:20 PM Aaron Marcuse-Kubitza

lib/util.sh: make (): don't change the directory to $top_dir, because paths used by the caller will instead be relative to the current dir (note that in runscripts, paths are generally not absolute because of canon_rel_path ()). this requires manually specifying $top_dir when running make on a physical (rather than inlined) makefile.

8955 05/09/2013 04:01 PM Aaron Marcuse-Kubitza

lib/util.sh: log_stderr_cmd (): replaced the `|| exit` after the () with `|| return`, for the same reason that this was a problem for extern () (see r8954)

8954 05/09/2013 03:55 PM Aaron Marcuse-Kubitza

lib/util.sh: extern (): replaced the `|| exit` after the () with `|| return`, because `|| exit` strangely does not use $? as its exit status (even though `|| exit $?` works fine). in a shell function, it is actually sufficient, for errexit purposes, to use `|| return`, because the exit status is checked upon return from the function. it is in fact preferable to use return rather than exit, because then the caller can catch the exception using `|| true`.

8953 05/09/2013 03:51 PM Aaron Marcuse-Kubitza

lib/util.sh: extern (): removed the `|| exit` inside the (), because this strangely prevents the exit status from being propagated to the exit status of the ()

8952 05/09/2013 03:44 PM Aaron Marcuse-Kubitza

lib/util.sh: added explicit `set -o errexit` in case caller did not have -e in #! line

8951 05/09/2013 02:47 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run

8950 05/09/2013 02:21 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run

8949 05/09/2013 02:06 PM Aaron Marcuse-Kubitza

inputs/FIA/_archive/2011-10-17/: set svn:ignore to *

8948 05/09/2013 01:52 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added Mac upgrade and recovery-related bookmarks.

8947 05/09/2013 12:07 PM Aaron Marcuse-Kubitza

bugfix: lib/import.sh: remake_VegBIEN_mappings (): don't run if there is no public schema to perform the test on (e.g. if the staging tables have just been reinstalled)

8946 05/09/2013 12:06 PM Aaron Marcuse-Kubitza

lib/import.sh: added public_schema_exists (), which just calls psql_script_vegbien (i.e. with the default search_path temp, public, etc.) and sees if returns an error