lib/sh/util.sh: stderr_matches(): supporting ignoring any benign error exit status associated with the error message being tested for

lib/sh/util.sh: stderr_matches(): usage: documented where any ignore_e statement would go

bugfix: lib/sh/util.sh: stderr_matches(): can't use `try` because this clears the exit status, which is needed for @PIPESTATUS to work. to support this, also need to avoid errexiting since @PIPESTATUS will be used instead.

lib/sh/util.sh: added dp(), which debug-prints a message

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

lib/sh/db.sh: psql(): $verbose_ok: renamed to $bypass_ok for clarity, because this applies only to the `--output /dev/fd/41` bypass (which when not possible, requires turning off verbose output

fix: lib/sh/db.sh: psql(): added $output_data switch analogous to what mysql() has. this causes query results of eg. void-returning functions to be correctly filtered by the logging mechanism, rather than output to stdout.

fix: lib/sh/db.sh: psql(): verbosity=0 (errors only) mode: use `SET client_min_messages = WARNING;` instead of NOTICE to hide verbose messages within psql as well

lib/sh/db.sh: psql(): replaced `test "$verbose_ok" && can_log` with bool var $verbose_

fix: lib/sh/db.sh: psql(): $verbose_: renamed to $verbose_ok for clarity

fix: lib/sh/util.sh: stdout_contains(): add another pipe_delay because the `grep` statement was sometimes getting printed before its filtered output

bugfix: lib/sh/util.sh: die_error_hidden(): min verbosity to display error should not be hardcoded

lib/sh/db.sh: psql(): "to see error details" msg: use new die_error_hidden()

lib/sh/util.sh: added die_error_hidden()

lib/sh/db.sh: psql(): "to see error details" msg: use new log_hint()

lib/sh/util.sh: added log_hint(), whose msg is only displayed if not a benign error

bugfix: lib/sh/db.sh: psql(): "to see error details" msg: also don't print it for benign errors ($benign_error)

lib/sh/db.sh: psql(): on error, display message describing how to see error details (prepend `vb=2` to the command)

bugfix: lib/sh/util.sh: log_err(): don't override verbosity manually, as this will not set log_level or PS4. instead, use new log! , which sets these correctly.

lib/sh/util.sh: added log! , which force-displays next log message

lib/sh/util.sh: save_e: made it idempotent so that it also works if save_e was already called

lib/sh/util.sh: save_e: made it idempotent so that it also works if save_e was already called

lib/sh/util.sh: rethrow: documented why can't use `(exit "$e")` (bash bug that prevents errexit)

bugfix: lib/runscripts/util.run: $auto_ignore: need to unexport it so don't pass this to invoked scripts except through fwd()

bugfix: lib/runscripts/util.run: auto_fwd's fallback() must be set after auto_ignore's fallback() to overwrite it (auto_ignore should only apply if an error would otherwise have been generated by the fallback)

lib/runscripts/util.run: fwd(): support subdirs that don't contain a runscript, so that the default value of @subdirs will work in most cases

lib/runscripts/util.run: fwd(): set default @subdirs (`{.,}*/`)

lib/sh/util.sh: added enter_top_dir and use it in in_top_dir

fix: lib/sh/util.sh: commands run inside $(...): need to run with log++ so that these aren't normally debug-printed

lib/sh/util.sh: added pv(), which debug-prints var(s)

lib/sh/util.sh: added wildcard.()

lib/sh/util.sh: added wildcard/()

lib/sh/util.sh: added esc_args()

lib/runscripts/datasrc_dir.run: removed postprocess(), which now does the same thing its auto-forwarded equivalent would

lib/runscripts/datasrc_dir.run: removed separate @table_subdirs, because the table-only targets can now safely be invoked on all subdirs, being auto-ignored in subdirs that don't support them

lib/runscripts/util.run: fwd(): enable $auto_ignore so that each subdir doesn't have to have a definition for the forwarded target

lib/runscripts/util.run: added $auto_ignore switch, which causes fallback() not to generate an error that a non-existant target doesn't exist

lib/runscripts/datasrc_dir.run: use new fwd_self alias

lib/runscripts/util.run: added fwd_self alias

lib/runscripts/datasrc_dir.run: enable $auto_fwd, to create the functionality of lib/forwarding.Makefile's `%` target

lib/runscripts/util.run: added $auto_fwd switch

bugfix: lib/runscripts/util.run: gateway(): need to use is_callable() rather than func_exists() to check whether the target exists, because external commands (eg. echo) are supported as targets, too

lib/sh/util.sh: added is_callable()

lib/runscripts/util.run: support custom handlers for all targets (gateway()) as well as targets w/o function (fallback())

lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names

lib/sh/util.sh: stderr_matches(): inline the stderr_matches alias to avoid needing to quote stderr_matches as "stderr_matches" in the most common use case (with pattern as a prefix env var)

bugfix: lib/sh/util.sh: stderr_matches(): when passing `pattern=...` as a prefix env var, must be invoked as `"stderr_matches"` to avoid the env var applying to the prep_try portion of the stderr_matches alias

fix: lib/phpPgAdmin.login.php.diff: "For user public_, leave password blank" instruction: moved this to right under the Password field and increased the font size so people would be more likely to see it

lib/Firefox_bookmarks.reformat.csv: "page's description": changed to "page's own description" to clarify that this is a description provided by the page itself

bugfix: lib/common.Makefile: `svn add`: need to run with --force because the new version of svn in the latest Mac upgrade errors if the file is already under version control

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

lib/sh/util.sh: stderr_matches(): documented that the purpose of not redirecting fd 2 back to fd 2 is to allow log-filtering out an otherwise-confusing benign error

lib/sh/util.sh: ignore_err_msg(): documented that unlike `|| true`, this suppresses only errors caused by a particular error message, rather than all error exit statuses

bugfix: lib/runscripts/validations.pg.sql.run: updated table match pattern to include the type prefix that validations queries now contain

bugfix: lib/runscripts/validations.pg.sql.run: --table: need to include explicit schema so that matching tables from other schemas are not included

added lib/runscripts/validations.pg.sql.run

lib/sh/util.sh: stdout_contains(): usage: documented that this requires a `{ ... } 41>&1` wrapper

lib/sh/util.sh: stderr2stdout(): clarified that fd 2 is not redirected back to fd 2

bugfix: lib/sh/util.sh: stderr_matches(): need to avoid redirecting stderr and stdout to the same place, because this prevents redirecting stdout back to the original stdout after stderr has been filtered using |

lib/sh/util.sh: local_export_array: renamed to just export_array because this was a replacement for export, not local_export

lib/sh/util.sh: local_*array: don't need -a because that it's an array is autodetected by the ()

lib/sh/util.sh: stderr_matches(): actually don't need to declare PIPESTATUS_ in a separate command, because local does support arrays

lib/sh/util.sh: added local_export_array

fix: lib/sh/util.sh: local_array: clarified that this is only needed for older versions of bash (the lack of support for arrays has apparently been fixed)

lib/sh/util.sh: added local_array

fix: lib/sh/util.sh: stderr_matches(): usage: documented that now need to manually rethrow any command error, if applicable

*{.sh,run}: stderr_matches calls: don't need to wrap the command in `"try"` because stderr_matches now does this

fix: lib/sh/util.sh: `shopt -s lastpipe`: suppress error message if not supported

lib/sh/util.sh: ignore_err_msg(): use new stderr_matches alias, which includes prep_try (requires loading new aliases)

bugfix: lib/sh/util.sh: stderr_matches(): need to save PIPESTATUS and then use the saved var because it's reset after each cmd

lib/sh/util.sh: stderr_matches(): place exit status of cmd in $e for use with exception handling

lib/sh/util.sh: set `shopt -s lastpipe`, to allow setting vars in the last command of a pipeline

bugfix: lib/sh/util.sh: ignore_err_msg(): use `try` properly with prep_try and `"try"`

lib/sh/util.sh: added prep_try and use it in try alias. try: documented how to run it with a wrapper command.

lib/sh/db.sh: pg_dump(): ignore "No matching tables were found" error, using new ignore_err_msg()

lib/sh/util.sh: added ignore_err_msg()

lib/sh/db.sh: pg_schema_exists(): documented that `try` is used to suppress the error exit status

lib/sh/util.sh: stderr_matches(): usage: surrounded command in if statement to indicate what context it would usually be used in

lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@

lib/sql_io.py: automatic handling of input/output column type mismatches: also do this for identifying columns, which first cause an error in a join in sql.distinct_table() rather than in the main insert (and thus were not handled by the existing error handling). previously, the user would have had to manually cast the input column in postprocess.sql. this involves getting handle_MissingCastException() to update join_cols as well as mapping.

lib/sql.py: distinct_table(): don't cache the table creation, because this prevents different distinct_tables from being created for the same input table

lib/sql.py: create_table(), copy_table_struct(): support custom query options, such as cacheable

lib/sql_io.py: put_table(): main loop MissingCastException handler: factored out into nested function so that it can also be used elsewhere

lib/sql.py: parse_exception(): parse "operator does not exist" errors as MissingCastExceptions (these appear when a staging table column is of the wrong type)

fix: lib/Parser.py: syntax_err(): error message: lightened the shade of gray so the boundary between the strings before and after the error would be clearer

lib/runscripts/in_datasrc_dir.run: datasrc_make(): use `if remaking ...` instead of accessing $_remake manually, for clarity

bugfix: lib/runscripts/in_datasrc_dir.run: datasrc_make(): use set_make_vars and $_remake as required in lib/sh/make.sh `remaking`

lib/sh/make.sh: remaking alias: documented that you MUST use set_make_vars at the beginning of any function that uses this, so that $_remake is properly set to $remake and not left at its previous value

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

fix: lib/sh/util.sh: verbosity_min(): usage: clarified that '' is a special value that causes $verbosity to be overwritten to ''

lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)

lib/common.Makefile: added %/live, for use with `make inputs/download`

bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).

bugfix: lib/runscripts/datasrc_dir.run: import(): can't run `datasrc_make reinstall` anymore because this now defers to the runscript for new-style import datasources (which was done so that `make .../install` properly reinstalls all the datasources). instead, call the applicable make targets manually (there are just 2 of them).

lib/sh/local.sh: public_schema_exists(): use a higher log_level for pg_schema_exists, to avoid all the verbose output involved in running the query

bugfix: lib/sh/local.sh: public_schema_exists(): can no longer use psql_script_vegbien for this, because using `SET search_path` (called by psql_script_vegbien) with a schema that does not exist no longer produces an error. instead, use new pg_schema_exists(), which uses a different command that does produce an error if the schema does not exist.

lib/sh/db.sh: added pg_require_schema()

lib/sh/util.sh: stderr2stdout(): documented that this redirects fd 2->1 and log_fd (but not back to 2)

bugfix: lib/sh/util.sh: stderr2stdout() use `command` before tee, which re-filters log_fd so that stderr itself is also filtered. this allows log-filtering out an otherwise-confusing benign error when using e.g. stderr_matches().

lib/sh/util.sh: added not(), for use in prefixing wrapped commands