Project

General

Profile

Statistics
| Revision:

# Date Author Comment
13473 05/17/2014 06:22 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts

13465 05/17/2014 02:15 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled

13413 05/07/2014 05:18 PM Aaron Marcuse-Kubitza

lib/runscripts/import.run: added install() target

13409 05/07/2014 03:29 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: use new local.run

13408 05/07/2014 03:25 PM Aaron Marcuse-Kubitza

added lib/runscripts/local.run

13373 05/01/2014 01:37 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir

13372 05/01/2014 01:29 PM Aaron Marcuse-Kubitza

fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts

13367 05/01/2014 04:09 AM Aaron Marcuse-Kubitza

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

13344 04/29/2014 09:38 PM Aaron Marcuse-Kubitza

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

13313 04/24/2014 03:49 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.

13312 04/24/2014 03:43 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run

13311 04/24/2014 03:40 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn

13310 04/24/2014 03:36 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)

13268 04/21/2014 08:01 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: sudo(): avoid slow $wrap_fn when using `command` (ie. always executable)

13216 04/20/2014 05:54 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $subdirs: adjusted log_level now that echo_vars is one log_level lower

13208 04/20/2014 05:06 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: $wrap_fn: log at higher log_level because it's startup code

13203 04/20/2014 04:44 PM Aaron Marcuse-Kubitza

lib/sh/util.sh, runscripts/util.run: set_paths(): run with higher log_level to hide all the paths that are set at the beginning of the script

13182 04/19/2014 07:14 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions

13181 04/19/2014 07:13 PM Aaron Marcuse-Kubitza

fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable

13148 04/16/2014 06:35 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()

13058 04/07/2014 01:43 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: table_make_install(): need to ignore skip_table() errexit

12984 03/30/2014 05:11 PM Aaron Marcuse-Kubitza

added lib/runscripts/extract_header.run

12982 03/30/2014 05:06 PM Aaron Marcuse-Kubitza

fix: lib/runscripts/util.run: to_top_file(): handle $_remake properly, without requiring deferred_check_target_exists to set to_file()'s flags

12968 03/29/2014 04:06 AM Aaron Marcuse-Kubitza

*{.sh,run}: runscript targets: use begin_target instead of echo_func so the target name is properly echoed. note that this requires using with_rm so that $rm is properly progagated to applicable invoked targets. (previously, $rm was progagated to all invoked targets. note that with_rm only works inside a runscript target that starts with begin_target.)

12967 03/29/2014 03:58 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: self_make(): renamed to with_rm() for clarity, since this is used only to progagate $rm, and does not also invoke a command with the same name as the current function, as the name might suggest

12964 03/28/2014 07:03 AM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support conventional main() method as well as `all` target

12878 03/24/2014 01:43 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: schema/make calls: need to use `make schema` instead because old-style datasources don't have a top-level runscript (the absence of this identifies them as old-style so inputs/input.Makefile works correctly)

12866 03/22/2014 04:55 AM Aaron Marcuse-Kubitza

lib/runscripts/util.run: on_exit(): documented that users can also override gateway()/fallback() to perform other commands (or no commands) after the script is read

12864 03/22/2014 04:44 AM Aaron Marcuse-Kubitza

fix: lib/runscripts/table.run: table_make_install(): need to inform the user when it skips installing a table, because this is often unexpected

12863 03/22/2014 04:43 AM Aaron Marcuse-Kubitza

fix: lib/runscripts/util.run: run_args_cmd(): need to indent the output of the target that it's running

12862 03/22/2014 04:15 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: removed no longer used datasrc_make_install()

12851 03/22/2014 03:07 AM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: import(): use new schema/make, schema/rm

12850 03/22/2014 02:59 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): use the much simpler `schema/make` run target, rather than outsourcing to the legacy Makefile via the convoluted datasrc_make_install()/table_make_install()

12849 03/22/2014 02:26 AM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: added schema/rm(), schema/make()

12847 03/22/2014 02:14 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: psql: always include ; at end of statement

12829 03/21/2014 11:48 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)

12804 03/21/2014 04:31 PM Aaron Marcuse-Kubitza

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

12800 03/21/2014 03:42 PM Aaron Marcuse-Kubitza

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

12797 03/21/2014 03:10 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.

12779 03/20/2014 07:58 PM Aaron Marcuse-Kubitza

*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`

12778 03/20/2014 07:51 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: runscript template: added sample make target, using new make target template

12774 03/20/2014 07:17 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)

12773 03/20/2014 07:13 PM Aaron Marcuse-Kubitza

bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try

12743 03/18/2014 04:04 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

12715 03/14/2014 07:53 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $auto_ignore: need to unexport it so don't pass this to invoked scripts except through fwd()

12713 03/14/2014 07:34 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: auto_fwd's fallback() must be set after auto_ignore's fallback() to overwrite it (auto_ignore should only apply if an error would otherwise have been generated by the fallback)

12712 03/14/2014 07:30 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: fwd(): support subdirs that don't contain a runscript, so that the default value of @subdirs will work in most cases

12711 03/14/2014 07:29 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: fwd(): set default @subdirs (`{.,}*/`)

12702 03/14/2014 05:02 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: removed postprocess(), which now does the same thing its auto-forwarded equivalent would

12701 03/14/2014 05:01 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: removed separate @table_subdirs, because the table-only targets can now safely be invoked on all subdirs, being auto-ignored in subdirs that don't support them

12700 03/14/2014 04:53 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: fwd(): enable $auto_ignore so that each subdir doesn't have to have a definition for the forwarded target

12699 03/14/2014 04:52 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $auto_ignore switch, which causes fallback() not to generate an error that a non-existant target doesn't exist

12698 03/14/2014 03:55 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: use new fwd_self alias

12697 03/14/2014 03:55 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added fwd_self alias

12696 03/14/2014 03:49 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: enable $auto_fwd, to create the functionality of lib/forwarding.Makefile's `%` target

12695 03/14/2014 03:47 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $auto_fwd switch

12694 03/14/2014 03:36 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: gateway(): need to use is_callable() rather than func_exists() to check whether the target exists, because external commands (eg. echo) are supported as targets, too

12692 03/14/2014 03:23 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support custom handlers for all targets (gateway()) as well as targets w/o function (fallback())

12691 03/14/2014 03:03 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names

12224 02/14/2014 03:09 PM Aaron Marcuse-Kubitza

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

12215 02/14/2014 11:28 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/validations.pg.sql.run: updated table match pattern to include the type prefix that validations queries now contain

12214 02/14/2014 11:16 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/validations.pg.sql.run: --table: need to include explicit schema so that matching tables from other schemas are not included

12212 02/14/2014 11:00 AM Aaron Marcuse-Kubitza

added lib/runscripts/validations.pg.sql.run

12189 02/13/2014 05:01 PM Aaron Marcuse-Kubitza

lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@

11978 01/20/2014 07:22 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: datasrc_make(): use `if remaking ...` instead of accessing $_remake manually, for clarity

11977 01/20/2014 07:17 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/in_datasrc_dir.run: datasrc_make(): use set_make_vars and $_remake as required in lib/sh/make.sh `remaking`

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11913 12/16/2013 04:45 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)

11875 12/09/2013 05:44 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).

11872 12/09/2013 03:54 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/datasrc_dir.run: import(): can't run `datasrc_make reinstall` anymore because this now defers to the runscript for new-style import datasources (which was done so that `make .../install` properly reinstalls all the datasources). instead, call the applicable make targets manually (there are just 2 of them).

11770 11/26/2013 12:47 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make_install(): simplified the setting of $noclobber since there no longer needs to be a different command for when the log exists

11769 11/26/2013 12:08 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: need to errexit the make target, so that errors in the SQL install scripts are not suppressed. this requires pre-checking if the table exists (using new pg_table_exists), so that the install target's errexit does not then need to be suppressed for cases when the table already exists.

11624 11/10/2013 12:04 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/file.pg.sql.run: export_(): exclude Source and related tables so that these will be re-created by the staging tables installation instead, ensuring that they are always in sync with the Source/ subdir

11603 11/08/2013 10:57 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)

11602 11/08/2013 10:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: reordered functions in the order they are called by import()

11574 11/05/2013 10:47 PM Aaron Marcuse-Kubitza

lib/runscripts/extract.run: added export_sample()

11462 10/29/2013 07:46 AM Aaron Marcuse-Kubitza

lib/runscripts/import_subset.run: $version: use new $extract_view, which is set to the same value that this was

11461 10/29/2013 07:45 AM Aaron Marcuse-Kubitza

lib/runscripts/extract.run: use the extract-specific view instead of all of analytical_stem

11436 10/24/2013 07:08 PM Aaron Marcuse-Kubitza

added lib/runscripts/import_subset.run, extract.run

11428 10/24/2013 03:26 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.

11427 10/24/2013 03:21 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: if is_dot_script, fix $ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).

11418 10/23/2013 11:02 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args

11417 10/23/2013 11:00 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)

11414 10/23/2013 10:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV

11359 10/18/2013 09:52 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file

11358 10/18/2013 08:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command

11348 10/18/2013 05:28 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.

11252 10/12/2013 12:12 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable

11247 10/10/2013 05:46 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed

11170 10/08/2013 05:31 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions

11169 10/08/2013 05:29 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir

11000 09/17/2013 10:05 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/*: calls to rm: use `rm -f` instead to avoid an error (which aborts the program) if the file does not yet exist

10946 09/14/2013 04:20 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: added postprocess target to run postprocess in just the table subdirs, skipping any additional subdirs that don't have this target

10945 09/14/2013 04:19 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: @subdirs: moved import_order.txt subdirs into separate @table_subdirs, which provides access to just the table subdirs when the user adds other dirs to @subdirs

10929 09/12/2013 03:34 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/view.run: remake_VegBIEN_mappings(): also need to remake header.csv, not just map.csv as for tables, because view columns may change when the view is regenerated

10835 08/30/2013 06:06 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: usage: documented that this usage also applies to all files that include this file

10834 08/30/2013 06:06 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: usage: clarified that the cmd to run is a function

10832 08/30/2013 06:02 PM Aaron Marcuse-Kubitza

added lib/runscripts/pg.conf.run, which installs PostgreSQL config files

10831 08/30/2013 06:01 PM Aaron Marcuse-Kubitza

added lib/runscripts/install.run, analogous to import.run

10781 08/27/2013 09:35 PM Aaron Marcuse-Kubitza

added lib/runscripts/data.pg.sql.run (analogous to schema.pg.sql.run for data-only SQL scripts)