schemas/vegbien.sql: source_by_shortname(): documented that it must be run as a nested SELECT, because otherwise Postgres will not inline it. not inlining causes the query to be run for each row in a table of potentially millions, and creates a significant slowdown (eg. >1.5 hours for SALVIAS._plots_06_list_of_plots_with_stem_measurements)

schemas/util.sql: eval2val(): made it STABLE so that functions that use it can themselves be inlined. (Postgres apparently ignores the STABLE qualifier, without warning, if the function contains any VOLATILE statements.)

fix: schemas/vegbien.sql: source_by_shortname(): needs to be declared STABLE so it can be constant-folded and only run once per query. (it had previously been VOLATILE because of constraints imposed on functions that run `SET LOCAL search_path`.)

fix: schemas/vegbien.sql: location: added index on top_plot

fix: schemas/vegbien.sql: location: added index on top_plot

fix: schemas/vegbien.sql: _plots_06_list_of_plots_with_stem_measurements: changed columns to match input query

schemas/vegbien.sql: added more ~type tables for plots queries

schemas/vegbien.sql: added public_validations.rm_all_queries(schema), which removes all validations queries in a schema, or in all schemas (useful when recreating validations queries from the DDL export file)

schemas/util.sql: validation_views(): moved filtering out of _validations schemas to validatable_datasources() so that calling this function with no args gives the validation views in *all schemas as one would expect

schemas/util.sql: query_relations(): support passing in an input query instead of an output query, which will remove only in the datasource's schema. (note that it was not clear that passing in an input-side query was not previously supported.)

schemas/util.sql: added schema_regexp(regclass)

schemas/vegbien.sql: schema_anchor: clarified that this identifies this function's schema (for use in locating helper functions), but is not necessarily the schema operated on, as the comment implied

schemas/util.sql: drop_relations_like(): use util.schema_regexp()

schemas/util.sql: added schema_regexp(schema_anchor)

fix: schemas/vegbien.sql: _plots_04_count_of_plots_in_each_project_in_this_source: count top-level plots to match the input query

schemas/vegbien.sql: remake_diff_table(): instructions for how to regenerate the diff table: put these at the beginning of the table comment instead of the end, because they are the most important info about the table that a user needs to know

schemas/util.sql: diff(text, text): documented how to write a custom keys() function to match up rows using a subset of the columns (including a sample keys() function template)

schemas/vegbien.sql: replaced custom %== operators with much simpler custom keys() functions. this avoids both the need to write out an = comparison for each field, and especially, also the need to write both an operator and a function implementing that operator.

bugfix: schemas/util.sql: diff(text, text): in the %== comparison, it turns out you do need to cast the values to the same base type, even though this is optional when using a custom %==

schemas/util.sql: remake_diff_table(): prepend the "view this table in human-readable form" comment instead of appending because it is more important than the "contents generated from" comment previously added

schemas/util.sql: added prepend_comment()

schemas/util.sql: removed no longer used mk_set_search_path(VARIADIC schemas text[]). use mk_search_path() instead.

schemas/util.sql, vegbien.sql: comment about "function option search_path": don't include `mk_set_search_path()` since that is no longer used

schemas/util.sql: diff(text, text): always use the schema of col_type_null (the common base type) as the search_path, since any custom %== operator for it will always be in the same schema as it

schemas/util.sql: use_own_schema(): renamed to use_schema() because this can be used for any type-linked schema

bugfix: rematerialize_out_view(): set_search_path() does not take the same arguments as mk_set_search_path() did, so need to use mk_search_path() instead

schemas/util.sql: use_own_schema(): auto-append util to the search_path to enable use of util operators

schemas/util.sql: mk_set_search_path(): no need to debug_print_return_value() anymore because functions now use set_search_path() (or something that calls it), which debug-prints the statement (`EXECUTE util.mk_set_search_path()` did not)

schemas/util.sql: mk_search_path(), mk_set_search_path(text...): auto-append util to the search_path to enable use of util operators

schemas/util.sql: removed no longer used mk_use_own_schema(). use util.use_own_schema() instead.

schemas/util.sql, vegbien.sql: use util.use_own_schema()/util.set_search_path() instead of EXECUTE util.mk_use_own_schema()/util.mk_set_search_path()

schemas/util.sql: added use_own_schema()

schemas/util.sql: added set_search_path()

schemas/util.sql: runnable_sql(): don't output search_path before a SET statement

schemas/util.sql: added is_set_stmt()

schemas/util.sql: diff(text, text): use mk_diff_query()'s new cols param to avoid a nested SELECT

schemas/util.sql: mk_diff_query(): parameterized the customizable parts of the query, to allow them to be replaced with their EXPLAIN expansion (which is what we ultimately want, so that the query does not refer to any internal tables or views)

schemas/util.sql: diff(text, text): use util.mk_diff_query(). this splits the complex query-generation code apart from the query-execution code.

bugfix: schemas/util.sql: undid r12472 because the command needed to limit the effects of any `SET LOCAL search_path` to the current function will unfortunately also clear any existing search_path, which may be needed for the eval query to execute

schemas/util.sql: added mk_diff_query()

schemas/util.sql: debug_print_sql(): use runnable_sql()

schemas/util.sql: added runnable_sql()

fix: schemas/util.sql: eval2*(): added search_path function option in order to limit the effects of any `SET LOCAL search_path` in the invoked query to the current function. however, plain eval() is not changed because it is often used to execute a `SET LOCAL search_path` in the calling function.

schemas/util.sql: diff(text, text): removed unnecessary casts to the base type in the join condition and the WHERE filter. these had been presumed necessary due to errors, but the errors turned out to be caused by the operator not being in the search_path. note that the casts in the columns list are still needed, as described in the associated comment.

schemas/util.sql: materialize_query(): add a comment on the table with the query it was generated from

schemas/util.sql: debug_print_sql(): util.mk_set_search_path(): use for_printing := true to comment out LOCAL

schemas/util.sql: mk_set_search_path(text): include LOCAL commented out, because the user might want to run it with another statement as a single command, in which case it would be useful because they will be in the same transaction (http://www.postgresql.org/docs/9.3/static/sql-set.html#AEN81154)

schemas/util.sql: mk_set_search_path(): added for_printing option like for mk_set_search_path(text)

schemas/util.sql: mk_set_search_path(): support creating a SET search_path statement for display as well, which would exclude LOCAL because it doesn't work as a standalone command (http://www.postgresql.org/docs/9.3/static/sql-set.html#AEN81154)

schemas/util.sql: debug_print_sql(): include the search_path in case the query contains search_path-dependent elements (such as operators)

schemas/util.sql: util.explain2notice_msg(): add newline before and after to visually separate it from other debug info

schemas/util.sql: added mk_set_search_path() (which uses the current search_path)

schemas/util.sql: diff(text, text): in the outputted FULL JOIN query, documented that you should refer to the EXPLAIN output for the expansion of %==

bugfix: schemas/util.sql: first_word(): ignore leading whitespace: need to use util.ltrim_nl(), as ltrim() only removes spaces

bugfix: schemas/util.sql: first_word(): need to ignore leading whitespace. this applies to many of our queries, which have a leading newline.

schemas/util.sql: explain(): don't debug-print the EXPLAIN, to avoid cluttering up the debug output

schemas/util.sql: eval2set(): make debug-printing optional, for use with internal statements

schemas/util.sql: added is_explain()

fix: schemas/util.sql: diff(text, text): FULL JOIN SELECT statement: don't put a comment at the very beginning, because this prevents it from being autoexplained (this is the query we particularly want EXPLAIN output for)

fix: schemas/util.sql: debug_print_sql(): only surround EXPLAIN output with newlines if there is any

fix: schemas/util.sql: debug_print_sql(): need newline before and after EXPLAIN output to visually separate it from other debug info

schemas/util.sql: util.debug_print_sql(): include the EXPLAIN output of applicable queries, for easier query plan debugging (just like lib/sql.py's db.autoexplain)

schemas/util.sql: added explain2notice_msg_if_can()

schemas/util.sql: added is_explainable(sql)

schemas/util.sql: added first_word()

schemas/util.sql: explain2notice(): use util.raise_notice()

schemas/util.sql: explain2notice(): added helper function explain2notice_msg() which can also be used individually

fix: schemas/util.sql: schema comment: noted that avoiding use of STRICT also makes functions much easier to troubleshoot, because they won't mysteriously do nothing if called with only NULL arguments, even when you have added debug-print statements

schemas/util.sql, vegbien.sql: do not declare any SQL-language functions as STRICT, because this prevents them from being inlined (as documented in the util schema comment). the STRICT qualifier has been preserved where its behavior is necessary, rather than just used for optimization.

fix: schemas/util.sql: schema comment: indicate that all SQL-language functions should never be declared STRICT, not just IMMUTABLE ones, since inlining is not limited to IMMUTABLE (or STABLE) functions (although constant folding is)

bugfix: schemas/util.sql: join_strs_transform(): added back STRICT qualifier because it must be declared STRICT to use the special handling of STRICT aggregating functions

schemas/util.sql: %==(anyelement, anyelement): needs to be declared STABLE instead of IMMUTABLE because it depends on the search_path (as described at http://vegpath.org/links/#PostgreSQL:%20Documentation:%209.3:%20Function%20Volatility%20Categories%20**)

schemas/util.sql, vegbien.sql: do not declare IMMUTABLE SQL-language functions as STRICT, because this prevents them from being inlined (as documented in the util schema comment)

fix: schemas/util.sql: diff(text, text): use util.typeof() instead of pg_typeof() so the outputted query is runnable in any search_path

schemas/util.sql: added typeof(), which unlike pg_typeof() uses util.qual_name() to create a search_path-independent name

fix: schemas/util.sql: quote_typed(): schema-qualify the type so the expression does not depend on the search_path

schemas/util.sql: %== : use custom keys() function for the type, so that the user only has to get the keys from their value, not also write the comparison of those keys

schemas/vegbien.sql: synced from DB, which changes the order of the ~type tables

schemas/util.sql: diff(regclass, regclass): support custom %== by including the col_type_null's schema in the search_path

schemas/util.sql: debug_print_return_value(): don't '-encode the value by default, because text values usually look better without the '-escaping

bugfix: schemas/util.sql: mk_set_search_path(): debug_print_return_value() needs to be on mk_set_search_path(text), not mk_search_path(VARIADIC text[])

schemas/util.sql: mk_search_path(): use debug_print_return_value() because this function is used with EXECUTE rather than util.eval() (in order to affect the calling function), so the search_path would not otherwise be printed

schemas/util.sql: added debug_print_return_value()

schemas/util.sql: diff(text, text): support custom search_path because custom %== operators are usually not in the util schema

schemas/util.sql: diff(text, text): renamed IN params to be different from OUT params to support LANGUAGE plpgsql

bugfix: schemas/util.sql: diff(*): changed from STABLE to VOLATILE because these functions create tables

schemas/vegbien.sql: _plots_04_count_of_plots_in_each_project_in_this_source: added operator %== to match the rows by project_name

bugfix: schemas/util.sql: diff(text, text) and types that use its custom FULL JOIN matching: need to use separate operator %== for determining which records to put on the same row, so that the = filter for identical rows only excludes rows that are actually the same, not all rows with the same keys (which would usually remove all rows)

schemas/util.sql: added operator %== , which returns whether the map-keys of the compared values are the same

schemas/vegbien.sql: synced from DB, which changes the order of the ~type tables

schemas/vegbien.sql: _plots_02_list_of_project_names: added = operator that causes FULL JOIN to only use some columns, so that rows for the same label column are put on the same row in the diff table

schemas/util.sql: diff(): support custom = operator for the table type (to FULL JOIN on only some columns). this requires casting to the base type, to avoid an ambiguity with the default = operator for record.

schemas/util.sql: remake_diff_table(): use util.quote_typed() instead of hardcoding the type

fix: inputs/SALVIAS/validations.sql: _plots_02_list_of_project_names: altered column aliases to match output query

fix: schemas/vegbien.sql: rm_query_group(): schemas/vegbien.sql: util.drop_relations_like(): also need to drop ~type tables, so that they will be re-created for the new queries of that group

schemas/vegbien.sql: added rm_query_group(), which removes all validations queries with a particular group prefix

schemas/vegbien.sql: generated ~type for _plots_04_count_of_plots_in_each_project_in_this_source

schemas/util.sql: added drop_relations_like()

fix: schemas/vegbien.sql: public_validations._plots_* that use projectname: re-alias to project_name to match input queries. (reexporting apparently changes the relative order of the ~type tables.)

fix: schemas/vegbien.sql: _plots_03_count_of_all_plots_in_this_source: use top_plot instead of location because the corresponding input query includes just the top-level plots