lib/runscripts/table.run: added 1st_src()

lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir

lib/runscripts/subdir.run: added $datasrc_dir

fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting

fix: lib/runscripts/util.run: runscript template: all(): moved example commands to target(), where they would more likely be located

lib/runscripts/extract.run: export_(): also compress created file

fix: lib/runscripts/extract.run: export_(): explicitly prevent files from becoming web-accessible, to protect against an incorrect umask in the calling process

bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts

bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled

lib/runscripts/import.run: added install() target

lib/runscripts/in_datasrc_dir.run: use new local.run

added lib/runscripts/local.run

bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir

fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.

bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run

lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn

bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)

lib/runscripts/util.run: sudo(): avoid slow $wrap_fn when using `command` (ie. always executable)

bugfix: lib/runscripts/util.run: $subdirs: adjusted log_level now that echo_vars is one log_level lower

lib/runscripts/util.run: $wrap_fn: log at higher log_level because it's startup code

lib/sh/util.sh, runscripts/util.run: set_paths(): run with higher log_level to hide all the paths that are set at the beginning of the script

lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions

fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable

bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()

bugfix: lib/runscripts/table.run: table_make_install(): need to ignore skip_table() errexit

added lib/runscripts/extract_header.run

fix: lib/runscripts/util.run: to_top_file(): handle $_remake properly, without requiring deferred_check_target_exists to set to_file()'s flags

*{.sh,run}: runscript targets: use begin_target instead of echo_func so the target name is properly echoed. note that this requires using with_rm so that $rm is properly progagated to applicable invoked targets. (previously, $rm was progagated to all invoked targets. note that with_rm only works inside a runscript target that starts with begin_target.)

lib/sh/make.sh: self_make(): renamed to with_rm() for clarity, since this is used only to progagate $rm, and does not also invoke a command with the same name as the current function, as the name might suggest

lib/runscripts/util.run: support conventional main() method as well as `all` target

bugfix: lib/runscripts/table.run: schema/make calls: need to use `make schema` instead because old-style datasources don't have a top-level runscript (the absence of this identifies them as old-style so inputs/input.Makefile works correctly)

lib/runscripts/util.run: on_exit(): documented that users can also override gateway()/fallback() to perform other commands (or no commands) after the script is read

fix: lib/runscripts/table.run: table_make_install(): need to inform the user when it skips installing a table, because this is often unexpected

fix: lib/runscripts/util.run: run_args_cmd(): need to indent the output of the target that it's running

lib/runscripts/table.run: removed no longer used datasrc_make_install()

lib/runscripts/datasrc_dir.run: import(): use new schema/make, schema/rm

lib/runscripts/table.run: load_data(): use the much simpler `schema/make` run target, rather than outsourcing to the legacy Makefile via the convoluted datasrc_make_install()/table_make_install()

lib/runscripts/datasrc_dir.run: added schema/rm(), schema/make()

lib/runscripts/table.run: psql: always include ; at end of statement

lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.

*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`

lib/runscripts/util.run: runscript template: added sample make target, using new make target template

lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)

bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

bugfix: lib/runscripts/util.run: $auto_ignore: need to unexport it so don't pass this to invoked scripts except through fwd()

bugfix: lib/runscripts/util.run: auto_fwd's fallback() must be set after auto_ignore's fallback() to overwrite it (auto_ignore should only apply if an error would otherwise have been generated by the fallback)

lib/runscripts/util.run: fwd(): support subdirs that don't contain a runscript, so that the default value of @subdirs will work in most cases

lib/runscripts/util.run: fwd(): set default @subdirs (`{.,}*/`)

lib/runscripts/datasrc_dir.run: removed postprocess(), which now does the same thing its auto-forwarded equivalent would

lib/runscripts/datasrc_dir.run: removed separate @table_subdirs, because the table-only targets can now safely be invoked on all subdirs, being auto-ignored in subdirs that don't support them

lib/runscripts/util.run: fwd(): enable $auto_ignore so that each subdir doesn't have to have a definition for the forwarded target

lib/runscripts/util.run: added $auto_ignore switch, which causes fallback() not to generate an error that a non-existant target doesn't exist

lib/runscripts/datasrc_dir.run: use new fwd_self alias

lib/runscripts/util.run: added fwd_self alias

lib/runscripts/datasrc_dir.run: enable $auto_fwd, to create the functionality of lib/forwarding.Makefile's `%` target

lib/runscripts/util.run: added $auto_fwd switch

bugfix: lib/runscripts/util.run: gateway(): need to use is_callable() rather than func_exists() to check whether the target exists, because external commands (eg. echo) are supported as targets, too

lib/runscripts/util.run: support custom handlers for all targets (gateway()) as well as targets w/o function (fallback())

lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

bugfix: lib/runscripts/validations.pg.sql.run: updated table match pattern to include the type prefix that validations queries now contain

bugfix: lib/runscripts/validations.pg.sql.run: --table: need to include explicit schema so that matching tables from other schemas are not included

added lib/runscripts/validations.pg.sql.run

lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@

lib/runscripts/in_datasrc_dir.run: datasrc_make(): use `if remaking ...` instead of accessing $_remake manually, for clarity

bugfix: lib/runscripts/in_datasrc_dir.run: datasrc_make(): use set_make_vars and $_remake as required in lib/sh/make.sh `remaking`

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)

bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).

bugfix: lib/runscripts/datasrc_dir.run: import(): can't run `datasrc_make reinstall` anymore because this now defers to the runscript for new-style import datasources (which was done so that `make .../install` properly reinstalls all the datasources). instead, call the applicable make targets manually (there are just 2 of them).

lib/runscripts/table.run: table_make_install(): simplified the setting of $noclobber since there no longer needs to be a different command for when the log exists

bugfix: lib/runscripts/table.run: need to errexit the make target, so that errors in the SQL install scripts are not suppressed. this requires pre-checking if the table exists (using new pg_table_exists), so that the install target's errexit does not then need to be suppressed for cases when the table already exists.

bugfix: lib/runscripts/file.pg.sql.run: export_(): exclude Source and related tables so that these will be re-created by the staging tables installation instead, ensuring that they are always in sync with the Source/ subdir

bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)

lib/runscripts/table.run: reordered functions in the order they are called by import()

lib/runscripts/extract.run: added export_sample()

lib/runscripts/import_subset.run: $version: use new $extract_view, which is set to the same value that this was

lib/runscripts/extract.run: use the extract-specific view instead of all of analytical_stem

added lib/runscripts/import_subset.run, extract.run

bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.

bugfix: lib/runscripts/util.run: if is_dot_script, fix $ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).

lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV

bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file

lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command

lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.

bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable

bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed

lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions

lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir

bugfix: lib/runscripts/*: calls to rm: use `rm -f` instead to avoid an error (which aborts the program) if the file does not yet exist

lib/runscripts/datasrc_dir.run: added postprocess target to run postprocess in just the table subdirs, skipping any additional subdirs that don't have this target