bugfix: lib/runscripts/util.run: $wrap_fn: this always needs to contain `is_runscript=1` because $is_runscript is not (and cannot be) exported
lib/runscripts/table.run: added check_headers()
bugfix: lib/runscripts/table.run: srcs: missing []
srcs: missing [
lib/runscripts/table.run: added header() and use it in header.txt()
lib/runscripts/table.run: 1st_src(): use a variable for this instead, to avoid needing to run this function each time it's used, and to make @srcs available
lib/runscripts/table.run: added header.txt()
lib/runscripts/table.run: added 1st_src()
lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir
lib/runscripts/subdir.run: added $datasrc_dir
fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting
fix: lib/runscripts/util.run: runscript template: all(): moved example commands to target(), where they would more likely be located
lib/runscripts/extract.run: export_(): also compress created file
fix: lib/runscripts/extract.run: export_(): explicitly prevent files from becoming web-accessible, to protect against an incorrect umask in the calling process
bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts
bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled
lib/runscripts/import.run: added install() target
lib/runscripts/in_datasrc_dir.run: use new local.run
added lib/runscripts/local.run
bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir
fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts
lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed
bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.
bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run
lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn
bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)
lib/runscripts/util.run: sudo(): avoid slow $wrap_fn when using `command` (ie. always executable)
bugfix: lib/runscripts/util.run: $subdirs: adjusted log_level now that echo_vars is one log_level lower
lib/runscripts/util.run: $wrap_fn: log at higher log_level because it's startup code
lib/sh/util.sh, runscripts/util.run: set_paths(): run with higher log_level to hide all the paths that are set at the beginning of the script
lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions
fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable
bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()
bugfix: lib/runscripts/table.run: table_make_install(): need to ignore skip_table() errexit
added lib/runscripts/extract_header.run
fix: lib/runscripts/util.run: to_top_file(): handle $_remake properly, without requiring deferred_check_target_exists to set to_file()'s flags
*{.sh,run}: runscript targets: use begin_target instead of echo_func so the target name is properly echoed. note that this requires using with_rm so that $rm is properly progagated to applicable invoked targets. (previously, $rm was progagated to all invoked targets. note that with_rm only works inside a runscript target that starts with begin_target.)
lib/sh/make.sh: self_make(): renamed to with_rm() for clarity, since this is used only to progagate $rm, and does not also invoke a command with the same name as the current function, as the name might suggest
lib/runscripts/util.run: support conventional main() method as well as `all` target
bugfix: lib/runscripts/table.run: schema/make calls: need to use `make schema` instead because old-style datasources don't have a top-level runscript (the absence of this identifies them as old-style so inputs/input.Makefile works correctly)
lib/runscripts/util.run: on_exit(): documented that users can also override gateway()/fallback() to perform other commands (or no commands) after the script is read
fix: lib/runscripts/table.run: table_make_install(): need to inform the user when it skips installing a table, because this is often unexpected
fix: lib/runscripts/util.run: run_args_cmd(): need to indent the output of the target that it's running
lib/runscripts/table.run: removed no longer used datasrc_make_install()
lib/runscripts/datasrc_dir.run: import(): use new schema/make, schema/rm
lib/runscripts/table.run: load_data(): use the much simpler `schema/make` run target, rather than outsourcing to the legacy Makefile via the convoluted datasrc_make_install()/table_make_install()
lib/runscripts/datasrc_dir.run: added schema/rm(), schema/make()
lib/runscripts/table.run: psql: always include ; at end of statement
lib/sh/util.sh: clog*: renamed to log* for clarity (possible now that log* is no longer used for function-local log_level setting)
*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow
fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.
*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`
lib/runscripts/util.run: runscript template: added sample make target, using new make target template
lib/runscripts/datasrc_dir.run: documented how to reinstall staging tables (`rm=1 .../run import`)
bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try
bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run
bugfix: lib/runscripts/util.run: $auto_ignore: need to unexport it so don't pass this to invoked scripts except through fwd()
bugfix: lib/runscripts/util.run: auto_fwd's fallback() must be set after auto_ignore's fallback() to overwrite it (auto_ignore should only apply if an error would otherwise have been generated by the fallback)
lib/runscripts/util.run: fwd(): support subdirs that don't contain a runscript, so that the default value of @subdirs will work in most cases
lib/runscripts/util.run: fwd(): set default @subdirs (`{.,}*/`)
lib/runscripts/datasrc_dir.run: removed postprocess(), which now does the same thing its auto-forwarded equivalent would
lib/runscripts/datasrc_dir.run: removed separate @table_subdirs, because the table-only targets can now safely be invoked on all subdirs, being auto-ignored in subdirs that don't support them
lib/runscripts/util.run: fwd(): enable $auto_ignore so that each subdir doesn't have to have a definition for the forwarded target
lib/runscripts/util.run: added $auto_ignore switch, which causes fallback() not to generate an error that a non-existant target doesn't exist
lib/runscripts/datasrc_dir.run: use new fwd_self alias
lib/runscripts/util.run: added fwd_self alias
lib/runscripts/datasrc_dir.run: enable $auto_fwd, to create the functionality of lib/forwarding.Makefile's `%` target
lib/runscripts/util.run: added $auto_fwd switch
bugfix: lib/runscripts/util.run: gateway(): need to use is_callable() rather than func_exists() to check whether the target exists, because external commands (eg. echo) are supported as targets, too
lib/runscripts/util.run: support custom handlers for all targets (gateway()) as well as targets w/o function (fallback())
lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names
validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.
bugfix: lib/runscripts/validations.pg.sql.run: updated table match pattern to include the type prefix that validations queries now contain
bugfix: lib/runscripts/validations.pg.sql.run: --table: need to include explicit schema so that matching tables from other schemas are not included
added lib/runscripts/validations.pg.sql.run
lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@
lib/runscripts/in_datasrc_dir.run: datasrc_make(): use `if remaking ...` instead of accessing $_remake manually, for clarity
bugfix: lib/runscripts/in_datasrc_dir.run: datasrc_make(): use set_make_vars and $_remake as required in lib/sh/make.sh `remaking`
moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).
lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)
bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).
bugfix: lib/runscripts/datasrc_dir.run: import(): can't run `datasrc_make reinstall` anymore because this now defers to the runscript for new-style import datasources (which was done so that `make .../install` properly reinstalls all the datasources). instead, call the applicable make targets manually (there are just 2 of them).
lib/runscripts/table.run: table_make_install(): simplified the setting of $noclobber since there no longer needs to be a different command for when the log exists
bugfix: lib/runscripts/table.run: need to errexit the make target, so that errors in the SQL install scripts are not suppressed. this requires pre-checking if the table exists (using new pg_table_exists), so that the install target's errexit does not then need to be suppressed for cases when the table already exists.
bugfix: lib/runscripts/file.pg.sql.run: export_(): exclude Source and related tables so that these will be re-created by the staging tables installation instead, ensuring that they are always in sync with the Source/ subdir
bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)
lib/runscripts/table.run: reordered functions in the order they are called by import()
lib/runscripts/extract.run: added export_sample()
lib/runscripts/import_subset.run: $version: use new $extract_view, which is set to the same value that this was
lib/runscripts/extract.run: use the extract-specific view instead of all of analytical_stem
added lib/runscripts/import_subset.run, extract.run
bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.
bugfix: lib/runscripts/util.run: if is_dot_script, fix $ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).
when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $
lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args
lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)
lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV
bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file
lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command
lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.