bugfix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: undid rename of accepted name columns to scrubbed_* (r13435), because these are actually not the same (scrubbed_* is the combination of accepted and no-opinion names). the accepted name columns will now be named accepted_*, following the standard naming scheme.
bugfix: schemas/public_.sql: _plots_20_tnrs_names: morphospeciesSuffix must still be called morphospecies_suffix, not taxon_morphospecies, because that column has not been updated to the taxon_morphospecies yet (fixes r13435)
fix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: scrubbed_*: use columns from MatchedTaxon whenever possible, to as much as possible avoid the need to join to taxon_scrub.scrubbed_unique_taxon_name.*
schemas/Makefile: vegbien.sql: exclude the same set of Source tables excluded by inputs/.TNRS/schema.sql, so that reinstalling TNRS doesn't change the contents of this file
bugfix: inputs/.TNRS/grants.sql: added GRANT statements from schema.sql because these aren't run by `make inputs/.TNRS/reinstall`
lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off
schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it
schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name
schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side
schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import
/README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function
/README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this
bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac
fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files
/README.TXT: Full database import: for test import, added step to turn off DB backup
fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell
/README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)
fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell
fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)
bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries
bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists
bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts
bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource
fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run
bugfix: inputs/input.Makefile: install: only run this for datasource dirs
inputs/input.Makefile: install: use ./run's install target for clarity
lib/runscripts/import.run: added install() target
bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system
bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)
inputs/input.Makefile: added $(datasrc_schema_exists)
lib/runscripts/in_datasrc_dir.run: use new local.run
added lib/runscripts/local.run
web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links
web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page
added validation/aggregating/bien3_validation_sequence.xlsx from Brad
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)
bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able
added inputs/VegBank/verify/outputBien.log.url
inputs/input.Makefile: add: verify/: also svn:ignore *.log
added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee
schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments
bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable
planning/meetings/BIEN conference call availability.xlsx: updated
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)
schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names
bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan
fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).
bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name
schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names
schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource
fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables
schemas/public_.sql: query_view_regexp(): also match auxiliary tables
schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)
schemas/util.sql: added loop_ignore_errors()
schemas/public_.sql: added output_queries(prefix)
schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting
schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param
schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()
schemas/public_.sql: added public_validations.query_view_regexp()
bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()
bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists
lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir
fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts
bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
inputs/input.Makefile: install: also run validate/install
inputs/input.Makefile: added validate/install
bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match
bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database
lib/sh/util.sh: import_vars: documented that it's idempotent
/Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`
web/links/index.htm: updated to Firefox bookmarks: fixed broken links
bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)
/Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution
/Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import
/Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)
bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`
schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place
schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet
bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors
fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()
bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead
bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false
bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error
bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->false if error)
lib/sh/util.sh: added is_err()
lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations
bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed
fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1
fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1