schemas/*.sql: make sure every COMMENT starts and ends on its own line, so that it appears correctly in the formats it's most likely to be read in (ie. in the DDL export, not the COMMENT edit box in pgAdmin)

schemas/util.sql: added materialize_view()

schemas/util.sql: added table_name()

schemas/util.sql: table_schema(): removed unneeded STRICT, which prevents inlining

fix: schemas/util.sql: fix_array(): removed no longer accurate comment about effects of STRICT, which is no longer used

fix: schemas/util.sql: materialize_query(): 1st param should be named table_, not view_

schemas/util.sql: added drop_view()

schemas/util.sql: added materialize_query()

/README.TXT: Schema changes: clarified that the staging tables should only be reinstalled if needed

/README.TXT: put ... around all uppercased text, for consistency

_license/non-open-source/applies_to.txt: clarified that this is an exclusions list, not an inclusions list, in terms of what's open-source

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

validation/aggregating/*/*.sql: renamed views to match the current naming scheme in the DB. these files still need to be kept up-to-date because there are some queries that haven't been implemented yet, and therefore exist only in these files, not the DB.

schemas/util.sql: ifnull(): use COALESCE instead, because it turns out that MySQL's IFNULL is just a special case of this

added inputs/bien2_traits/validations.sql, from validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql

inputs/input.Makefile: $(svnFilesGlob): added validations.sql

bugfix: validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql: added util to search_path so that IFNULL would link to util.ifnull()

schemas/util.sql: added ifnull(), equivalent to MySQL's IFNULL (Postgres auto-lowercases the name)

lib/sh/util.sh: stderr_matches(): documented that the purpose of not redirecting fd 2 back to fd 2 is to allow log-filtering out an otherwise-confusing benign error

lib/sh/util.sh: ignore_err_msg(): documented that unlike `|| true`, this suppresses only errors caused by a particular error message, rather than all error exit statuses

bugfix: lib/runscripts/validations.pg.sql.run: updated table match pattern to include the type prefix that validations queries now contain

bugfix: lib/runscripts/validations.pg.sql.run: --table: need to include explicit schema so that matching tables from other schemas are not included

added inputs/bien2_traits/validations.sql.run

added lib/runscripts/validations.pg.sql.run

lib/sh/util.sh: stdout_contains(): usage: documented that this requires a `{ ... } 41>&1` wrapper

lib/sh/util.sh: stderr2stdout(): clarified that fd 2 is not redirected back to fd 2

bugfix: lib/sh/util.sh: stderr_matches(): need to avoid redirecting stderr and stdout to the same place, because this prevents redirecting stdout back to the original stdout after stderr has been filtered using |

lib/sh/util.sh: local_export_array: renamed to just export_array because this was a replacement for export, not local_export

lib/sh/util.sh: local_*array: don't need -a because that it's an array is autodetected by the ()

lib/sh/util.sh: stderr_matches(): actually don't need to declare PIPESTATUS_ in a separate command, because local does support arrays

lib/sh/util.sh: added local_export_array

fix: lib/sh/util.sh: local_array: clarified that this is only needed for older versions of bash (the lack of support for arrays has apparently been fixed)

lib/sh/util.sh: added local_array

fix: lib/sh/util.sh: stderr_matches(): usage: documented that now need to manually rethrow any command error, if applicable

*{.sh,run}: stderr_matches calls: don't need to wrap the command in `"try"` because stderr_matches now does this

fix: lib/sh/util.sh: `shopt -s lastpipe`: suppress error message if not supported

lib/sh/util.sh: ignore_err_msg(): use new stderr_matches alias, which includes prep_try (requires loading new aliases)

bugfix: lib/sh/util.sh: stderr_matches(): need to save PIPESTATUS and then use the saved var because it's reset after each cmd

lib/sh/util.sh: stderr_matches(): place exit status of cmd in $e for use with exception handling

lib/sh/util.sh: set `shopt -s lastpipe`, to allow setting vars in the last command of a pipeline

bugfix: lib/sh/util.sh: ignore_err_msg(): use `try` properly with prep_try and `"try"`

lib/sh/util.sh: added prep_try and use it in try alias. try: documented how to run it with a wrapper command.

lib/sh/db.sh: pg_dump(): ignore "No matching tables were found" error, using new ignore_err_msg()

lib/sh/util.sh: added ignore_err_msg()

lib/sh/db.sh: pg_schema_exists(): documented that `try` is used to suppress the error exit status

lib/sh/util.sh: stderr_matches(): usage: surrounded command in if statement to indicate what context it would usually be used in

lib/runscripts/file.pg.sql.run, schema.pg.sql.run: support custom options to pg_dump in $@

validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql: added view names using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#prepend-CREATE-VIEW

validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql: removed trailing whitespace

schemas/vegbien.sql: public_validations_*: merged into single public_validations schema, with type-specific prefixes for each query, so that datasources can use validations queries from multiple type categories, and so that each datasource doesn't have to indicate which validations output schema it's using

schemas/util.sql: diff_cols(): documented how to run EXPLAIN on the FULL JOIN query

schemas/util.sql: diff_cols(): embed left_query/right_query directly in the main SQL statement, so that the query planner can take the sort order of the queries into account in planning the FULL JOIN (and where possible, use a fast merge join)

schemas/vegbien.sql: regenerated from DB, which adds an autogenerated column alias

schemas/vegbien.sql, validation/aggregating/*/*.sql: renamed public_validation_* to plural public_validations_* since we refer to them as aggregating validation*s*

schemas/util.sql: added eval2col_pair()

web/.phpPgAdmin/.htaccess: support linking to a function

web/.phpPgAdmin/.htaccess: support linking to the functions list

web/.phpPgAdmin/.htaccess: support linking to a view

web/.phpPgAdmin/.htaccess: allow linking directly to the views list of a schema

added backups/vegbien.r12025.backup.md5

web/.phpPgAdmin/.htaccess: refactored to build the URL incrementally, extracting path components 1st->last, instead of handling each # of path components as a separate case (which was more rigid and created unnecessary duplication)

backups/TNRS.backup.md5: updated

bugfix: web/.phpPgAdmin/.htaccess: path RewriteRules: need to go directly to the applicable tab, rather than going via redirect.php, because redirect.php will use the last selected tab, rather than the first tab for that section, leading to unexpected results when a non-default tab was previously selected

schemas/vegbien.sql, validation/aggregating/*/*.sql: renamed public_validation_* to plural public_validations_* since we refer to them as aggregating validation*s*

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: made "diff tables" the same size as the other major labels

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: added CSVs export arrows and import process arrows, as decided in conference call

schemas/vegbien.sql: public_validation_plots: added Brad's notes as comments

schemas/vegbien.sql: public_validation_plots: populated queries from validation/aggregating/plots/bien3_validations_salvias_vegbien.sql

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _12_count_of_verbatim_taxa_per_plot_in_each_project: removed typo (in the original queries, not added by the refactoring)

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: shortened view names to fit in the 63-char limit

schemas/vegbien.sql: public_validation_traits: added Brad's notes as comments

schemas/vegbien.sql: public_validation_traits: populated queries from validation/aggregating/traits/bien3_validations_traits_bien3.sql

validation/aggregating/*/*.sql: use "" to preserve case in column aliases, so that they match the input queries

validation/aggregating/*/*.sql: use current_schema instead of a hardcoded datasource name or psql variable

validation/aggregating/*/*.sql: search_path: added public

validation/aggregating/*/*.sql: use the new type-specific validation schema names

validation/aggregating/*/*.sql: removed type-specific prefix from the view name, which is now part of the validation schema

inputs/import.stats.xls: updated import times

inputs/import.stats.xls: updated import times

schemas/vegbien.sql: analytical_stem.datasource: removed fkey to source.shortname because this prevents individually-reloaded datasources from being published

schemas/vegbien.sql: publish(), rm(): documented runtime (1 min)

schemas/vegbien.sql: added datasource_unpublish()

lib/sql_io.py: automatic handling of input/output column type mismatches: also do this for identifying columns, which first cause an error in a join in sql.distinct_table() rather than in the main insert (and thus were not handled by the existing error handling). previously, the user would have had to manually cast the input column in postprocess.sql. this involves getting handle_MissingCastException() to update join_cols as well as mapping.

lib/sql.py: distinct_table(): don't cache the table creation, because this prevents different distinct_tables from being created for the same input table

lib/sql.py: create_table(), copy_table_struct(): support custom query options, such as cacheable

lib/sql_io.py: put_table(): main loop MissingCastException handler: factored out into nested function so that it can also be used elsewhere

lib/sql.py: parse_exception(): parse "operator does not exist" errors as MissingCastExceptions (these appear when a staging table column is of the wrong type)

bugfix: /README.TXT: Full database import: Check that source contains [# datasources] rows up through XAL: added alternative verification method when this is not the case (some datasources may be near the end depending on import order)

schemas/vegbien.sql: public_validation: split into separate subschemas for plots, specimens, traits

schemas/Makefile: vegbien.sql: support auxiliary schemas besides *_validation

fix: schemas/Makefile: %/publish: removed no longer applicable instructions about running `make schemas/rotate`. you should not publish a schema until you are satisfied that it can replace the previous public schema, so that the previous public schema doesn't need to be saved.

fix: schemas/vegbien.sql: public_validation comment: undid incorrect replacement of "validation" with "public_validation"

bugfix: schemas/Makefile: "public": dont

schemas/Makefile: %/publish: don't ignore errors if schema doesn't exist, because it should always exist if you are publishing it

schemas/Makefile: `%/uninstall: vegbien.sql`: use util.schema_bundle_rm() instead of public.rm(), because when the public schema is incompletely imported, public.rm() will not yet exist

schemas/Makefile: `%/install: vegbien.sql`: support auxiliary schemas besides *_validation

schemas/Makefile: %/uninstall, %/publish: use new public.rm(), publish(), which encapsulate the process of renaming multiple related schemas and also allow additional auxiliary schemas beyond just *_validation

schemas/vegbien.sql: added rm(), which deletes this schema

schemas/vegbien.sql: added publish(), which publishes this schema

schemas/vegbien.sql: added public_schema_publish()