schemas/util.sql: added restore_views() and use it in force_recreate()

bugfix: schemas/util.sql: save_drop_views(): views must be dropped in reverse dependency order, but returned in forward dependency order

bugfix: schemas/util.sql: force_recreate(): dependent views need to be re-created in forward dependency order. this is done by reversing the dependencies only for save_drop_views().

bugfix: schemas/util.sql: mk_set_comment(): need to use VIEW instead of TABLE for views

bugfix: schemas/util.sql: show_create_view(): also need to include view comment

fix: schemas/util.sql: mk_set_comment(): added ;

fix: schemas/util.sql: show_set_comment(): don't display a COMMENT ON statement if no comment, because this will be appended to table defs, etc. and would create clutter

bugfix: schemas/util.sql: mk_set_comment(): need to handle NULL comment properly

schemas/util.sql: added show_set_comment(regclass)

schemas/util.sql: set_comment(): use util.mk_set_comment()

schemas/Makefile: added vegbien/install, vegbien/uninstall which also handle TNRS

fix: schemas/Makefile: vegbien.sql: also remake inputs/.TNRS/schema.sql since its contents change along with vegbien.sql

schemas/util.sql: added mk_set_comment()

bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts

backups/pg_snapshot: documented runtime for initial backup (4 h)

bugfix: schemas/util.sql: force_recreate(): recreate_users_cmds: use create_if_not_exists() rather than eval(), because cmd might manually re-create a deleted dependent view, causing it to already exist

bugfix: schemas/util.sql: save_drop_view(): util.show_create_view(): don't include `OR REPLACE` because that causes nonuniform errors (eg. invalid_table_definition), instead of the standard duplicate_table exception caught by util.create_if_not_exists()

fix: schemas/util.sql: show_create_view(): removed extra trailing ; because pg_get_viewdef() now includes one

schemas/util.sql: show_create_view(): made inclusion of `OR REPLACE` configurable

schemas/util.sql: save_drop_view(): use util.drop_view()

/README.TXT: Maintenance: added steps to back up the vegbiendev databases

bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled

lib/tnrs.py: use the TNRS dev server (with private URL in tnrs.url) instead of the live server, since that contains datasources that we need

lib/streams.py: added file_get_contents()

lib/tnrs.py: configure the server separately from the base URL

lib/: svn:ignore tnrs.url so the TNRS dev server URL does not become public

bugfix: web/.phpPgAdmin/.htaccess: only append subject if non-empty, to avoid confusing redirect.php

bugfix: web/.phpPgAdmin/.htaccess: custom pages: need to handle in query string because .php filename would be treated as path element

web/.phpPgAdmin/.htaccess: support custom .php pages

planning/meetings/BIEN conference call availability.xlsx: updated

schemas/util.sql: save_drop_views(): return text[] instead of concatenated text so that the queries will be run individually and be easier to debug

schemas/util.sql: added eval(queries text[])

bugfix: schemas/util.sql: force_recreate(): need to drop users in reverse to avoid dependencies that block the drop

schemas/util.sql: added array_reverse()

schemas/util.sql: added in_reverse()

bugfix: schemas/util.sql: force_recreate(): need to use :lower:, not [:lower:]

schemas/util.sql: force_recreate(): debug-print PG_EXCEPTION_DETAIL, users to assist in debugging

fix: schemas/util.sql: debug_print_value(): added trailing newline to visually distinguish the printed value

schemas/util.sql: added debug_print_var()

schemas/util.sql: debug_print_return_value(): use new util.debug_print_value()

schemas/util.sql: added debug_print_value()

fix: schemas/util.sql: util.force_recreate(): also support dropping things besides tables

inputs/.TNRS/schema.sql: MatchedTaxon.Accepted_name_species: mapped to accepted_species_binomial

fix: inputs/.TNRS/schema.sql: COMMENTs: always include newline before and after

bugfix: schemas/public_.sql: plots_20_tnrs_names: tnrs_accepted*: use new accepted_*, not scrubbed_*, as scrubbed_* does not contain only the accepted name (as implied by the tnrs_accepted_* column name)

bugfix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: undid rename of accepted name columns to scrubbed_* (r13435), because these are actually not the same (scrubbed_* is the combination of accepted and no-opinion names). the accepted name columns will now be named accepted_*, following the standard naming scheme.

bugfix: schemas/public_.sql: _plots_20_tnrs_names: morphospeciesSuffix must still be called morphospecies_suffix, not taxon_morphospecies, because that column has not been updated to the taxon_morphospecies yet (fixes r13435)

fix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: scrubbed_*: use columns from MatchedTaxon whenever possible, to as much as possible avoid the need to join to taxon_scrub.scrubbed_unique_taxon_name.*

schemas/Makefile: vegbien.sql: exclude the same set of Source tables excluded by inputs/.TNRS/schema.sql, so that reinstalling TNRS doesn't change the contents of this file

bugfix: inputs/.TNRS/grants.sql: added GRANT statements from schema.sql because these aren't run by `make inputs/.TNRS/reinstall`

lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off

schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it

schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name

schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side

schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import

/README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function

/README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this

bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac

fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files

/README.TXT: Full database import: for test import, added step to turn off DB backup

fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell

/README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)

fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell

fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)

bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries

bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists

bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts

bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource

fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run

bugfix: inputs/input.Makefile: install: only run this for datasource dirs

inputs/input.Makefile: install: use ./run's install target for clarity

lib/runscripts/import.run: added install() target

bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system

bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)

inputs/input.Makefile: added $(datasrc_schema_exists)

lib/runscripts/in_datasrc_dir.run: use new local.run

added lib/runscripts/local.run

web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links

web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page

added validation/aggregating/bien3_validation_sequence.xlsx from Brad

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)

bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able

added inputs/VegBank/verify/outputBien.log.url

inputs/input.Makefile: add: verify/: also svn:ignore *.log

added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments

bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable

planning/meetings/BIEN conference call availability.xlsx: updated

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)

schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names

bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan

fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).

validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB

bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name

schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names

schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource

fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables