added inputs/bien2_traits/validations.sql.run

added lib/runscripts/validations.pg.sql.run

lib/sh/util.sh: stdout_contains(): usage: documented that this requires a `{ ... } 41>&1` wrapper

lib/sh/util.sh: stderr2stdout(): clarified that fd 2 is not redirected back to fd 2

bugfix: lib/sh/util.sh: stderr_matches(): need to avoid redirecting stderr and stdout to the same place, because this prevents redirecting stdout back to the original stdout after stderr has been filtered using |

lib/sh/util.sh: local_export_array: renamed to just export_array because this was a replacement for export, not local_export

lib/sh/util.sh: local_*array: don't need -a because that it's an array is autodetected by the ()

lib/sh/util.sh: stderr_matches(): actually don't need to declare PIPESTATUS_ in a separate command, because local does support arrays

lib/sh/util.sh: added local_export_array

fix: lib/sh/util.sh: local_array: clarified that this is only needed for older versions of bash (the lack of support for arrays has apparently been fixed)

lib/sh/util.sh: added local_array

fix: lib/sh/util.sh: stderr_matches(): usage: documented that now need to manually rethrow any command error, if applicable

*{.sh,run}: stderr_matches calls: don't need to wrap the command in `"try"` because stderr_matches now does this

fix: lib/sh/util.sh: `shopt -s lastpipe`: suppress error message if not supported

lib/sh/util.sh: ignore_err_msg(): use new stderr_matches alias, which includes prep_try (requires loading new aliases)

bugfix: lib/sh/util.sh: stderr_matches(): need to save PIPESTATUS and then use the saved var because it's reset after each cmd

lib/sh/util.sh: stderr_matches(): place exit status of cmd in $e for use with exception handling

lib/sh/util.sh: set `shopt -s lastpipe`, to allow setting vars in the last command of a pipeline

bugfix: lib/sh/util.sh: ignore_err_msg(): use `try` properly with prep_try and `"try"`

lib/sh/util.sh: added prep_try and use it in try alias. try: documented how to run it with a wrapper command.

lib/sh/db.sh: pg_dump(): ignore "No matching tables were found" error, using new ignore_err_msg()

lib/sh/util.sh: added ignore_err_msg()

lib/sh/db.sh: pg_schema_exists(): documented that `try` is used to suppress the error exit status

lib/sh/util.sh: stderr_matches(): usage: surrounded command in if statement to indicate what context it would usually be used in

lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@

validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql: added view names using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#prepend-CREATE-VIEW

validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql: removed trailing whitespace

schemas/vegbien.sql: public_validations_*: merged into single public_validations schema, with type-specific prefixes for each query, so that datasources can use validations queries from multiple type categories, and so that each datasource doesn't have to indicate which validations output schema it's using

schemas/util.sql: diff_cols(): documented how to run EXPLAIN on the FULL JOIN query

schemas/util.sql: diff_cols(): embed left_query/right_query directly in the main SQL statement, so that the query planner can take the sort order of the queries into account in planning the FULL JOIN (and where possible, use a fast merge join)

schemas/vegbien.sql: regenerated from DB, which adds an autogenerated column alias

schemas/vegbien.sql, validation/aggregating/*/*.sql: renamed public_validation_* to plural public_validations_* since we refer to them as aggregating validation*s*

schemas/util.sql: added eval2col_pair()

web/.phpPgAdmin/.htaccess: support linking to a function

web/.phpPgAdmin/.htaccess: support linking to the functions list

web/.phpPgAdmin/.htaccess: support linking to a view

web/.phpPgAdmin/.htaccess: allow linking directly to the views list of a schema

added backups/vegbien.r12025.backup.md5

web/.phpPgAdmin/.htaccess: refactored to build the URL incrementally, extracting path components 1st->last, instead of handling each # of path components as a separate case (which was more rigid and created unnecessary duplication)

backups/TNRS.backup.md5: updated

bugfix: web/.phpPgAdmin/.htaccess: path RewriteRules: need to go directly to the applicable tab, rather than going via redirect.php, because redirect.php will use the last selected tab, rather than the first tab for that section, leading to unexpected results when a non-default tab was previously selected

schemas/vegbien.sql, validation/aggregating/*/*.sql: renamed public_validation_* to plural public_validations_* since we refer to them as aggregating validation*s*

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: made "diff tables" the same size as the other major labels

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: added CSVs export arrows and import process arrows, as decided in conference call

schemas/vegbien.sql: public_validation_plots: added Brad's notes as comments

schemas/vegbien.sql: public_validation_plots: populated queries from validation/aggregating/plots/bien3_validations_salvias_vegbien.sql

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _12_count_of_verbatim_taxa_per_plot_in_each_project: removed typo (in the original queries, not added by the refactoring)

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: shortened view names to fit in the 63-char limit

schemas/vegbien.sql: public_validation_traits: added Brad's notes as comments

schemas/vegbien.sql: public_validation_traits: populated queries from validation/aggregating/traits/bien3_validations_traits_bien3.sql

validation/aggregating/*/*.sql: use "" to preserve case in column aliases, so that they match the input queries

validation/aggregating/*/*.sql: use current_schema instead of a hardcoded datasource name or psql variable

validation/aggregating/*/*.sql: search_path: added public

validation/aggregating/*/*.sql: use the new type-specific validation schema names

validation/aggregating/*/*.sql: removed type-specific prefix from the view name, which is now part of the validation schema

inputs/import.stats.xls: updated import times

inputs/import.stats.xls: updated import times

schemas/vegbien.sql: analytical_stem.datasource: removed fkey to source.shortname because this prevents individually-reloaded datasources from being published

schemas/vegbien.sql: publish(), rm(): documented runtime (1 min)

schemas/vegbien.sql: added datasource_unpublish()

lib/sql_io.py: automatic handling of input/output column type mismatches: also do this for identifying columns, which first cause an error in a join in sql.distinct_table() rather than in the main insert (and thus were not handled by the existing error handling). previously, the user would have had to manually cast the input column in postprocess.sql. this involves getting handle_MissingCastException() to update join_cols as well as mapping.

lib/sql.py: distinct_table(): don't cache the table creation, because this prevents different distinct_tables from being created for the same input table

lib/sql.py: create_table(), copy_table_struct(): support custom query options, such as cacheable

lib/sql_io.py: put_table(): main loop MissingCastException handler: factored out into nested function so that it can also be used elsewhere

lib/sql.py: parse_exception(): parse "operator does not exist" errors as MissingCastExceptions (these appear when a staging table column is of the wrong type)

bugfix: /README.TXT: Full database import: Check that source contains [# datasources] rows up through XAL: added alternative verification method when this is not the case (some datasources may be near the end depending on import order)

schemas/vegbien.sql: public_validation: split into separate subschemas for plots, specimens, traits

schemas/Makefile: vegbien.sql: support auxiliary schemas besides *_validation

fix: schemas/Makefile: %/publish: removed no longer applicable instructions about running `make schemas/rotate`. you should not publish a schema until you are satisfied that it can replace the previous public schema, so that the previous public schema doesn't need to be saved.

fix: schemas/vegbien.sql: public_validation comment: undid incorrect replacement of "validation" with "public_validation"

bugfix: schemas/Makefile: "public": dont

schemas/Makefile: %/publish: don't ignore errors if schema doesn't exist, because it should always exist if you are publishing it

schemas/Makefile: `%/uninstall: vegbien.sql`: use util.schema_bundle_rm() instead of public.rm(), because when the public schema is incompletely imported, public.rm() will not yet exist

schemas/Makefile: `%/install: vegbien.sql`: support auxiliary schemas besides *_validation

schemas/Makefile: %/uninstall, %/publish: use new public.rm(), publish(), which encapsulate the process of renaming multiple related schemas and also allow additional auxiliary schemas beyond just *_validation

schemas/vegbien.sql: added rm(), which deletes this schema

schemas/vegbien.sql: added publish(), which publishes this schema

schemas/vegbien.sql: added public_schema_publish()

schemas/util.sql: added functions for maintaining schema bundles, groups of schemas with a common prefix

schemas/util.sql: added schema_bundle_get_schemas()

schemas/util.sql: added schema_replace()

schemas/util.sql: added schema_rm(), schema_rename()

schemas/vegbien.sql: datasource_publish(): removed nested transaction, which is not needed because Postgres runs each top-level command in a transaction (including in autocommit mode)

bugfix: schemas/Makefile,vegbien.sql: renamed validation to public_validation to match the suffixed name applied by `make schemas/public/install`

schemas/Makefile: %/install, %/publish for public schema: added usage

/README.TXT: to back up the version history: added back `git svn fetch` so we keep the git export up-to-date, too

/README.TXT: to back up the version history: added runtimes (1.5 h for the initial svnsync)

/README.TXT: to back up the version history: added trailing /s to dirs

bugfix: /README.TXT: to back up the version history: fixed svn_repo/ path

**/LICENSE.TXT: corrected "it recommended" to "it is recommended"

_license/non-open-source/applies_to.txt: deleted items: tip revision needs to refer to the revision at which the problem was discovered, not the revision before that

_license/non-open-source/applies_to.txt: deleted items: clarified that some things deleted after r11705 (when we started copyright scrubbing) might also be non-open-source, because they might have been deleted for reasons other than copyright scrubbing and thus would be missing the "copyright scrub" tag

_license/non-open-source/applies_to.txt: deleted items: clarified that some things deleted after r11705 (when we started copyright scrubbing) might also be non-open-source, because they might have been deleted for reasons other than copyright scrubbing and thus would be missing the "copyright scrub" tag

/LICENSE.TXT: removed separate non-open-source disclaimer about parts of the version history before r11705, because this is now listed in _license/non-open-source/applies_to.txt

_license/non-open-source/applies_to.txt: included deleted items that are non-open-source, so that the list correctly applies to the entire repository (which includes the version history)

**/LICENSE.TXT: disclaimer about things that are not open-source: removed "these are copyright their respective owners", because this is understood

_license/README.TXT: changed "parts" to "components" for clarity

_license/non-open-source/applies_to.txt: UArizona?: added BIEN2/traits_observation_counts.xls

/README.TXT: to back up the version history: use svnsync instead of `git svn fetch`, so that the backup is in a format that can be directly reimported into an svn repo

/LICENSE.TXT: parts of the repository that are not open-source: added ,s to clarify the meaning