Project

General

Profile

Statistics
| Revision:

# Date Author Comment
8934 05/09/2013 08:00 AM Aaron Marcuse-Kubitza

lib/util.sh: added $top_script (alias for $0), $top_file, $top_filename. use new $top_file in $top_dir.

8933 05/09/2013 07:54 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb.run: put zip_newer/unzip_newer on the same line as the `cd "$top_dir"`

8932 05/09/2013 07:45 AM Aaron Marcuse-Kubitza

bin/my2pg: translate varbinary to bytea

8931 05/09/2013 07:42 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb.run: added export_postgres (), which calls VegCore.pg.sql/make ()

8930 05/09/2013 07:29 AM Aaron Marcuse-Kubitza

lib/util.sh: wrapper functions of external commands: use new self alias

8929 05/09/2013 07:28 AM Aaron Marcuse-Kubitza

lib/util.sh: added self alias, for use with wrapper functions of external commands

8928 05/09/2013 07:18 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.my.sql: regenerated using `VegCore.ERD.mwb.run export_mysql` (also part of the all target). note that this uses mysqldump, so the format and table order is different than it was for MySQL Workbench's custom export script.

8927 05/09/2013 07:05 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()

8926 05/09/2013 07:05 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()

8925 05/09/2013 07:03 AM Aaron Marcuse-Kubitza

bugfix: lib/local.sh: $PATH: use $bin_dir_abs so the path will remain valid if the current directory is changed

8924 05/09/2013 07:02 AM Aaron Marcuse-Kubitza

lib/local.sh: added \$bin_dir_abs

8923 05/09/2013 06:58 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: extern (): added `|| exit` after "$@" in addition to after the (), because this additional `|| exit` is apparently needed in order to propagate the exit status of the command to the exit status of the () and cause an error to abort the script properly

8922 05/09/2013 06:51 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_vars (): output vars with log_level 3 so they can be excluded separately from function calls to reduce clutter in the logging output. increase the default util.run verbosity so that vars are still printed by default.

8921 05/09/2013 06:47 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: echo_vars (): output each var separately so that each is prefixed by $PS4

8920 05/09/2013 06:39 AM Aaron Marcuse-Kubitza

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

8919 05/09/2013 06:39 AM Aaron Marcuse-Kubitza

lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent

8918 05/09/2013 06:31 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: make (): use stdout2stderr=1 so that commands make runs are properly output to stderr

8917 05/09/2013 06:21 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: include_guard_var (): use extern () when invoking sed so that this internal use of it isn't verbosely logged along with other external commands. note the "" around extern to refer to the function, not the alias.

8916 05/09/2013 06:16 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: zip/unzip (): use new $stdout2stderr instead of >&2 so that the redirect happens in the right order

8915 05/09/2013 06:15 AM Aaron Marcuse-Kubitza

lib/util.sh: log_stderr_cmd (): support redirecting stdout to stderr *after*setting up the stderr redirect, so that stdout is also subject to that redirect

8914 05/09/2013 06:11 AM Aaron Marcuse-Kubitza

lib/util.sh: added stdout2stderr ()

8913 05/09/2013 06:08 AM Aaron Marcuse-Kubitza

lib/util.sh: changed echo_run_extern () to an alias

8912 05/09/2013 06:04 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_run_extern (): use new log_stderr_cmd

8911 05/09/2013 06:03 AM Aaron Marcuse-Kubitza

lib/util.sh: added log_stderr_cmd ()

8910 05/09/2013 05:57 AM Aaron Marcuse-Kubitza

lib/util.sh: use extern instead of env to run external commands, because it's simpler and auto-captures the logging output

8909 05/09/2013 05:44 AM Aaron Marcuse-Kubitza

lib/util.sh: echo_func (): use the much simpler function keyword to avoid expanding the echo_func alias when defining the function, rather than going through the complex process of unaliasing and realiasing the echo_func alias

8908 05/09/2013 05:38 AM Aaron Marcuse-Kubitza

lib/util.sh: mark env as always being an external command. it is no longer necessary to automatically echo it using echo_run because the extern alias does this now.

8907 05/09/2013 05:31 AM Aaron Marcuse-Kubitza

lib/util.sh: automatically echo external commands. this requires using the function keyword when declaring extern () so that the new extern alias is not expanded in the function name.

8906 05/09/2013 05:28 AM Aaron Marcuse-Kubitza

lib/util.sh: added echo_run_extern (), which automatically runs log_stderr to capture the external command's stderr

8905 05/09/2013 05:27 AM Aaron Marcuse-Kubitza

lib/util.sh: added extern ()

8904 05/09/2013 05:05 AM Aaron Marcuse-Kubitza

lib/util.sh: added log_stderr ()

8903 05/09/2013 04:54 AM Aaron Marcuse-Kubitza

lib/util.sh: added end_try_subshell alias

8902 05/09/2013 04:29 AM Aaron Marcuse-Kubitza

lib/util.sh: zip/unzip: redirect logging output to stderr

8901 05/09/2013 04:23 AM Aaron Marcuse-Kubitza

lib/util.sh: use inc_log_level where logging info should only be output for verbosity >= 2: echo_func, echo_vars, echo_stdin

8900 05/09/2013 04:03 AM Aaron Marcuse-Kubitza

lib/util.sh: verbose output: added inc_log_level, dec_log_level aliases which manipulate $verbosity. note that changes in $log_level are now instead indicated by moving $verbosity in the opposite direction.

8899 05/09/2013 04:01 AM Aaron Marcuse-Kubitza

lib/util.sh: added let () wrapper that prevents it from triggering error-exit

8898 05/09/2013 03:16 AM Aaron Marcuse-Kubitza

lib/util.sh: verbose output: use just one var ($verbosity) instead of two ($verbosity and $log_level), to track log level, for simplicity. $verbosity would now be subtracted from to alter the log level instead of $log_level being added to.

8897 05/09/2013 03:09 AM Aaron Marcuse-Kubitza

lib/util.sh: use new can_log wherever logging info is output to stderr

8896 05/09/2013 03:05 AM Aaron Marcuse-Kubitza

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

8895 05/09/2013 02:53 AM Aaron Marcuse-Kubitza

lib/util.sh: verbose output: added $log_level (must always be > 0 so verbosity=0 turns off all logging) and can_log (), which compares $log_level to $verbosity

8894 05/09/2013 02:32 AM Aaron Marcuse-Kubitza

bugfix: lib/local.sh: updated $root_dir

8893 05/09/2013 02:31 AM Aaron Marcuse-Kubitza

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

8892 05/09/2013 02:22 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added mysqldump_local, which uses vegbiendev

8891 05/09/2013 02:21 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: use postgres_compat mode because all our MySQL schemas should be usable as inputs to my2pg

8890 05/09/2013 02:20 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added connection vars for local (bien) user

8889 05/09/2013 02:18 AM Aaron Marcuse-Kubitza

lib/util.sh: moved unalias inside include guard since it is not needed by anything before the include guard (even though it is used by commands outside the include guard, i.e. when the include guard is temporarily closed and reopened to load/unload aliases)

8888 05/09/2013 02:06 AM Aaron Marcuse-Kubitza

lib/util.sh: use declare instead of local in aliases so that the aliases can also be used outside a function (declare will create a local var when used inside a function, and a global var otherwise)

8887 05/09/2013 02:00 AM Aaron Marcuse-Kubitza

lib/util.sh: zip: ignore "zip has nothing to do" exit status as this is not an error

8886 05/09/2013 01:58 AM Aaron Marcuse-Kubitza

lib/util.sh: added exception-handling (i.e. error-suppressing) functions

8885 05/09/2013 01:03 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: echo_func alias: moved it outside the include guard because the unalias statement is needs to be outside the include guard, and therefore so does the alias statement in order to always restore the alias

8884 05/09/2013 12:59 AM Aaron Marcuse-Kubitza

lib/util.sh: renamed echo_func_ () back to echo_func () and instead add an unalias statement before the if statement containing the function definition. this is necessary because if statements are "compound commands", which have aliases expanded when their definition is read (i.e. at the beginning of the if statement, for the entire if statement), not when they are executed, and any aliases for function names in them need to be unaliased before the if statement containing the function definition. this is necessary even if the if statement is not executed, because the expansion of such aliases will cause syntax errors in the parsing of the if statement itself....

8883 05/09/2013 12:45 AM Aaron Marcuse-Kubitza

lib/util.sh: added unalias () override, which doesn't produce an error if the alias is undefined

8882 05/09/2013 12:04 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: renamed echo_func () to echo_func_ (), because you apparently can't have a function with the same name as an alias, even if the function is defined before the alias and an include guard prevents the function from being defined again after the alias is set. there may be some kind of alias-expanding preprocessing pass that bash only does in script mode, or maybe there is some other problem related to our use of aliases inside if statements.

8881 05/08/2013 08:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: changed echo_func to an alias that includes the "$", so that callers don't need to specify the "$" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.

8880 05/08/2013 07:21 PM Aaron Marcuse-Kubitza

lib/util.sh: added set_make_vars alias, for use with make-target-style shell functions

8879 05/08/2013 06:37 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): interpret an empty string for $user, $database as if they were unset

8878 05/08/2013 06:34 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): add --tables after specifying --databases so that future unnamed arguments are correctly interpreted as tables

8877 05/08/2013 06:33 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): need to use `--databases ...` rather than `--database=...` to specify the database as a named argument

8876 05/08/2013 06:26 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): need space before $database in command

8875 05/08/2013 06:23 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): use $schema as $database when provided

8874 05/08/2013 06:14 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): accept $database connection var

8873 05/08/2013 06:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: use just env instead of echo_run env now that env is an auto-echoing alias

8872 05/08/2013 05:56 PM Aaron Marcuse-Kubitza

lib/util.sh: automatically echo commands that use env

8871 05/08/2013 05:55 PM Aaron Marcuse-Kubitza

lib/util.sh: moved strings after verbose output so strings commands can use verbose output aliases

8870 05/08/2013 05:28 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): don't ssh if the ssh_server is this machine

8869 05/08/2013 05:27 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): allow specifying the ssh_dest as a separate ssh_server and ssh_user

8868 05/08/2013 05:08 PM Aaron Marcuse-Kubitza

lib/util.sh: using prefixed connection vars: added use_local, use_local_remote aliases

8867 05/08/2013 04:42 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysqldump_diffable ()

8866 05/08/2013 04:41 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysqldump () wrapper, which does what my2pg_export does, but integrates with mysql_cmd and allows turning off postgres-compatible mode

8865 05/08/2013 04:15 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): use ${FUNCNAME1} to get the caller's command name instead of requiring the caller to pass the command name explicitly

8864 05/08/2013 03:56 PM Aaron Marcuse-Kubitza

lib/util.sh: databases: added use_remote alias to use the remote_* connection vars (which uses new import_vars)

8863 05/08/2013 03:54 PM Aaron Marcuse-Kubitza

lib/util.sh: added import_vars alias

8862 05/08/2013 03:53 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): added doc comment

8861 05/08/2013 03:52 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql (): use new mysql_cmd to auto-add connection/login opts when specified

8860 05/08/2013 03:50 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysql_cmd ()

8859 05/08/2013 03:48 PM Aaron Marcuse-Kubitza

lib/util.sh: added local_inv alias

8858 05/08/2013 03:34 PM Aaron Marcuse-Kubitza

lib/util.sh: zip/unzip_newer: support extracting/updating the file even if it has an older mtime by setting $force, using new set_inv to get a var to use to determine whether to include the -u flag

8857 05/08/2013 03:32 PM Aaron Marcuse-Kubitza

lib/util.sh: added set_inv ()

8856 05/08/2013 03:30 PM Aaron Marcuse-Kubitza

lib/util.sh: echo_vars (): prefix the declare output with "+ " because it's actually a command, not just an indication of the var's value

8855 05/08/2013 03:28 PM Aaron Marcuse-Kubitza

lib/util.sh: added sed ()

8854 05/08/2013 03:23 PM Aaron Marcuse-Kubitza

lib/util.sh: added section labels

8853 05/08/2013 10:57 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed broken favicons

8852 05/08/2013 10:55 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed broken favicons. this requires using the SQLite Manager Firefox add-on <https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search> to manually remove the favicon entries.

8851 05/08/2013 10:18 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: formatting fixes: don't put the src attribute of the favicon image on its own line, because it no longer contains long inline image data

8850 05/08/2013 10:15 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: favicon image: remove ICON attributes with inline image data, since these are no longer needed for the favicon images and this avoids cluttering up the svn diff

8849 05/08/2013 10:11 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: favicon image: use ICON_URI instead of ICON to fetch the favicon directly from the original site. this allows the favicons to be cached separately from the bookmarks page, so that the inline favicon contents don't need to be re-retrieved every time the bookmarks page is updated. this also avoids cluttering up the svn diff.

8848 05/08/2013 10:01 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added description of the formats supported by PostgreSQL's COPY command and their escape sequences.

8847 05/08/2013 08:44 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: geoscrub_input_view: only include places whose country is not NULL, because at least the country is required for geoscrubbing

8846 05/08/2013 08:34 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to run `make schemas/public/install` after the datasources are installed. note that this must be run after the datasources are installed, because views in public depend on some of the datasources.

8845 05/06/2013 07:10 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: use `make db` instead of mk_db, which includes the required schemas/install command. some datasources use the util schema in their postprocess scripts, so this must be installed before the staging tables.

8844 05/06/2013 07:05 AM Aaron Marcuse-Kubitza

bugfix: /Makefile: moved schemas/install from install to db so that it is also run whenever the DB is reinstalled. schemas/install is needed by the staging tables reinstallation performed by reinstall_all.

8843 05/06/2013 06:33 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Schema changes: Reinstall staging tables: also need to run `make mk_db` before running reinstall_all

8842 05/06/2013 06:30 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: drop the saved previous DB at the end of the operation

8841 05/06/2013 06:18 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: run all of the local machine steps on vegbiendev, including the vegbien DB rename, to ensure that the existing data is saved in case the reinstall has errors

8840 05/06/2013 05:57 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to first rename existing vegbien database, so it is not modified in the reinstallation

8839 05/06/2013 05:27 AM Aaron Marcuse-Kubitza

inputs/FIA/: archived no longer used BIEN2 FIA data from the nimoy geoscrub DB

8838 05/06/2013 05:26 AM Aaron Marcuse-Kubitza

inputs/FIA/_archive/: moved FIA_COND_unique, Organism into 2011-10-17/ subdir

8837 05/06/2013 04:44 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to first run this on the local machine to check for any errors, before running it on the live DB, to prevent the live DB from becoming unrestorable in the case of bugs

8836 05/06/2013 04:27 AM Aaron Marcuse-Kubitza

bugfix: bin/with_all: pipe yes to the command in case it makes any interactive prompts, as in e.g. the TNRS staging tables reinstall

8835 05/06/2013 04:25 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: sql/install: manually specify $no_search_path option to psql_script_vegbien, which is added automatically in $(psqlNoSearchPath) but that uses psql_verbose_vegbien