schemas/public_.sql: validation views: use analytical_stem_view instead of analytical_stem because analytical_stem is no longer materialized (viewFullOccurrence_individual_view is materialized instead)

schemas/public_.sql: derived views: use analytical_stem_view instead of analytical_stem because analytical_stem is no longer materialized (viewFullOccurrence_individual_view is materialized instead)

bugfix: schemas/util.sql: expansion_min_cols(): increased to properly handle all public-schema views (the largest is currently 2014-6-12.Jeff_Ott.climatic_range_determinants)

schemas/util.sql: view_def_to_orig(): use new expansion_min_cols() for easier configuration

schemas/util.sql: added expansion_min_cols() (the minimum # of cols from the same table to be treated as a * expression)

fix: schemas/public_.sql: analytical_stem_view: regenerated columns list for viewFullOccurrence_individual (instead of viewFullOccurrence_individual_view)

fix: schemas/util.sql: col_re(): support column names with " in them

schemas/util.sql: added _concat_nullify(), which uses || instead of concat()

schemas/public_.sql: removed no longer used threatened_taxonlabel_view. use iucn_red_list_view instead.

schemas/public_.sql: viewFullOccurrence_individual_view: documented materialize time (22 h)

inputs/.TNRS/schema.sql: taxon_match: removed no longer used scrubbed_unique_taxon_name. the scrubbed name ranks are now generated from the other TNRS columns instead.

schemas/public_.sql: viewFullOccurrence_*: documented materialize time

schemas/public_.sql: analytical_stem_view: use new materialized viewFullOccurrence_individual

schemas/public_.sql: added viewFullOccurrence_individual by running viewFullOccurrence_individual_view_modify()

schemas/public_.sql: added viewFullOccurrence_individual_view_modify(), analogous to analytical_stem_view_modify()

inputs/.TNRS/schema.sql: removed no longer used view ValidMatchedTaxon. use taxon_scrub instead.

schemas/public_.sql: iucn_red_list_view: use taxon_scrub instead of ValidMatchedTaxon since they are equivalent

inputs/.TNRS/schema.sql: taxon_scrub: use taxon_best_match directly, to avoid the need for a separate ValidMatchedTaxon view

fix: inputs/.TNRS/schema.sql: taxon_scrub: merged synonymous columns

schemas/vegbien.sql: taxon_scrub: documented steps to merge synonymous columns

inputs/.TNRS/schema.sql: removed no longer used view MatchedTaxon. use taxon_best_match instead.

inputs/.TNRS/schema.sql: ValidMatchedTaxon: use taxon_best_match now that it's equivalent to MatchedTaxon

bugfix: schemas/public_.sql: tnrs_input_name: use taxon_match instead of taxon_best_match because there is no index on taxon_match that includes just the filters used by taxon_best_match

schemas/public_.sql: tnrs_input_name: use taxon_best_match now that it's equivalent to MatchedTaxon

fix: inputs/.TNRS/schema.sql: MatchedTaxon: merged synonymous columns

fix: schemas/util.sql: view_is_subset_or_renaming(): views with CASE statements are also not subsets/renamings

inputs/.TNRS/schema.sql: removed no longer used taxon_scrub.scrubbed_unique_taxon_name.* . use taxon_scrub instead.

inputs/.TNRS/schema.sql: taxon_scrub: use taxon_match derived columns instead of the incorrect values in taxon_scrub.scrubbed_unique_taxon_name.* (which does not work with the multi-match strategy)

inputs/.TNRS/schema.sql: MatchedTaxon: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.

inputs/.TNRS/schema.sql: taxon_scrub: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.

schemas/public_.sql: added viewFullOccurrence_{CVS,VegBank,NCU} for Bob

bugfix: schemas/util.sql: added GRANT USAGE for bien_read, public_ so that util schema functions can be called by other users

inputs/.TNRS/schema.sql: taxon_match: to port derived column changes to vegbiendev: derived_cols_export() code: documented runtime (6 h)

bugfix: inputs/.TNRS/schema.sql: removed no longer used derived column __accepted_infraspecific_label, which had a buggy formula that broke derived_cols_populate()

bugfix: schemas/util.sql: drop_derived_col(): also need to run util.derived_cols_trigger_update()

bugfix: inputs/.TNRS/schema.sql: removed no longer used derived column __accepted_infraspecific_label, which had a buggy formula that broke derived_cols_populate()

fix: inputs/.TNRS/schema.sql: taxon_match: to remove a column: updated instructions

inputs/.TNRS/schema.sql: added new derived columns to derived views

fix: schemas/util.sql: view_def_to_orig(): need to support aliased columns (produced when a column is renamed)

schemas/util.sql: added aliased_col_re()

schemas/util.sql: added alias_re()

fix: schemas/util.sql: view_is_subset(): renamed to view_is_subset_or_renaming() because this also supports views that just rename columns, which should not be .*-ed by view_def_to_orig()

fix: schemas/util.sql: view_def_to_orig(): can't use FROM to alias util.col_re() because that prevents inlining the function

fix: schemas/util.sql: view_def_to_orig(): can't use FROM to alias util.col_re() because that prevents inlining the function

schemas/util.sql: view_def_to_orig(): use util.col_re() for clarity

schemas/util.sql: added col_re()

fix: schemas/util.sql: derived_col_update(): also need steps to drop column, because DROP __ CASCADE doesn't work when there are dependent views

inputs/.TNRS/schema.sql: _accepted_infraspecific{rank,epithet}: use array slice of new _accepted{genus,specific_epithet,infra_{rank,epithet}}, which is simpler than using remove_prefix() in __accepted_infraspecific_label

inputs/.TNRS/schema.sql: "[accepted_]genus__@DwC__@vegpath.org": don't need to use *Accepted_name anymore because _accepted{genus,specific_epithet,infra_{rank,epithet}} is now generated from *Accepted_name

inputs/.TNRS/schema.sql: taxon_match."__accepted_{genus,specific_epithet}": renamed to "__accepted_{genus,specific_epithet,infra_{rank,epithet}}" since this now includes these other ranks as well

bugfix: inputs/.TNRS/schema.sql: taxon_match."__accepted_{genus,specific_epithet}": use "*Accepted_name" instead of "Accepted_species[_binomial]__@TNRS__@vegpath.org" (from "*Accepted_name_species") because Accepted_name_species apparently sometimes does not match the Accepted_name and uses malformed Unicode characters

bugfix: schemas/util.sql: derived_cols_export(): also need to first run util.derived_cols_remove() if changing the derived column order

schemas/util.sql: added derived_cols_remove(), which allows derived columns to be re-created in a different order

schemas/util.sql: added drop_derived_col(), which ensures that drop_column() only cascades to views

fix: schemas/util.sql: drop_column(): also need to recreate dependent views

schemas/util.sql: added drop_constraint()

inputs/.TNRS/schema.sql: taxon_match: `inputs/.TNRS/data.sql.run refresh`: documented runtime (1 min)

bugfix: inputs/.TNRS/schema.sql: taxon_match: use "Accepted_species[_binomial]__@TNRS__@vegpath.org" instead of "*Accepted_name_species". this fixes a bug in __accepted_infraspecific_label where Accepted_name_species with trailing whitespace could not be prefix-removed from names that contained just a species binomial.

fix: inputs/.TNRS/schema.sql: taxon_match: added derived column "Accepted_species[_binomial]__@TNRS__@vegpath.org", which removes trailing whitespace

inputs/.TNRS/schema.sql: added steps to remove a column and to add a non-derived column

inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: merged into "to add a new derived column"

inputs/.TNRS/schema.sql: to add columns in the middle: renamed to "to move a column to the middle" for clarity

inputs/.TNRS/schema.sql: to populate a new column: updated to use util.derived_col_update()

fix: inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns: also need to run util.recreate_view()

inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: don't need to run `rm=1 inputs/.TNRS/data.sql.run` because this is now run by `make schemas/remake`

inputs/.TNRS/schema.sql: added new derived columns to derived views

schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching

bugfix: inputs/.TNRS/schema.sql: taxon_match.__accepted_infraspecific_label: need to use case-insensitive matching of the removed prefix because TNRS lowercases part of the Accepted_name

schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching

schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching

schemas/util.sql: added case-sensitive-optional versions of starts_with(), ends_with()

schemas/util.sql: added canon_case(str text, case_sensitive boolean)

schemas/util.sql: check_constraint_disable(): documented that we can't use ADD CONSTRAINT NOT VALID for this because it does not disable the constraint completely, but instead causes new rows to be validated against it. the ADD CONSTRAINT NOT VALID behavior does not work for the derived columns use case of this function.

schemas/public_.sql: *_view_modify(): added repopulate param, which is passed to rematerialize_view()

schemas/util.sql: rematerialize_view(): support updating just the table's schema, without populating it

schemas/util.sql: rematerialize_view(): don't need extra eval() around table_mod_sql because recreate() performs eval()

bugfix: inputs/.TNRS/schema.sql: taxon_match: use wrapper for util.remove_prefix() so CHECK constraints that use it don't get dropped when the util schema is reinstalled

schemas/public_.sql: *_view_modify(): use util.rematerialize_view(), which now supports table mods

schemas/util.sql: rematerialize_view(): added support for table mods to run after creating table

bugfix: schemas/util.sql: rematerialize_view(): don't populate unless recreate() can successfully recreate the dependent views, to avoid populating the entire table (which for some views can take hours) only to have the changes rolled back by a failing dependent view

schemas/util.sql: removed no longer used materialize_view(). use util.copy() instead.

schemas/util.sql: rematerialize_view(): call util.copy() directly instead of via util.materialize_view()

schemas/util.sql: removed no longer used copy_types_and_data(). use util.materialize_view() instead.

schemas/util.sql: to_freq(): use util.copy() instead of util.copy_types_and_data() to avoid a dependency on util.copy_types_and_data()

fix: schemas/util.sql: copy_data(): also need to ANALYZE table after loading data into it

schemas/public_.sql: analytical_stem_view: use viewFullOccurrence_individual_view to avoid duplicated code between the views. this moves the stem-related columns to the end of the view.

schemas/public_.sql: added viewFullOccurrence_individual_view

schemas/util.sql: derived_col_constraint_initially_enabled(): clarified that this is always based on whether the existing constraint is enabled

inputs/.TNRS/schema.sql: taxon_match: COMMENT: added steps to port derived column changes to vegbiendev

schemas/util.sql: added derived_cols_export(), which creates SQL to port the derived cols to another DB

bugfix: inputs/.TNRS/schema.sql: taxon_match: derived columns: use new "matched~Name[_no_author]___@TNRS__@vegpath.org" instead of "*Name_matched" so that "No suitable matches found." is removed before concatenating with other fields

inputs/.TNRS/schema.sql: taxon_match: added derived column "matched~Name[_no_author]___@TNRS__@vegpath.org", which removes the "No suitable matches found." string

inputs/.TNRS/schema.sql: reordered derived columns in dependency order

bugfix: inputs/.TNRS/schema.sql: "[accepted_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org": need to use "[accepted_]genus__@DwC__@vegpath.org" rather than "*Accepted_name" for this for rank = genus

inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org"

bugfix: inputs/.TNRS/schema.sql: "[accepted_]genus__@DwC__@vegpath.org": need to populate this for rank = genus

inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]taxonomicStatus__@DwC__@vegpath.org"

bugfix: inputs/.TNRS/schema.sql: derived columns: use "Accepted_family__@TNRS__@vegpath.org" instead of "*Accepted_name_family" because "*Accepted_name_family" is sometimes missing

bugfix: derived_col_constraints_enable(): also need to update col comments, in addition to table__fill_derived()

schemas/util.sql: check_constraint_enabled(): made it IMMUTABLE