schemas/util.sql: added cluster(), which reclusters each time

schemas/util.sql: cluster_once(): use simpler `LANGUAGE sql` now that we're using util.eval() instead of EXECUTE

schemas/util.sql: cluster_once(): use util.eval() instead of EXECUTE to print the CLUSTER statement (if any) it's using

fix: inputs/.TNRS/schema.sql: taxon_match: cluster the table so the rows are always exported in the same order, even when an UPDATE statement is used to run triggers

bugfix: schemas/util.sql: cluster_once(): need to use just the name of the index, without the schema prefix

schemas/util.sql: cluster_once(): use simpler IS DISTINCT FROM instead of array !=

inputs/.TNRS/schema.sql: added _accepted{genus,specific_epithet} derived column, using the steps at wiki.vegpath.org/Postgres_queries#maintaining-derived-column-formulas-across-column-renames

web/username_prefix.php: remove padding used to visually separate elements: documented that this also removes a leading __ that indicates an internal field

web/links/index.htm: updated to Firefox bookmarks: Quick links: added VegBIEN data dictionary spreadsheet, TNRS data dictionary.

fix: web/username_prefix.php: transposes: changed recommended syntax to y~x@url so that _y~ lines up vertically with [y] paths

web/username_prefix.php: removed special category comments syntax because this is now handled by transposes and insertion comments

web/username_prefix.php: added support for transposes ( y~x_@url -> x_y@url , Y~x@url -> xY@url ), which can be used to put the category in a column name first without duplicating the category

web/username_prefix.php: removed linewraps code because Google spreadsheets no longer supports wrapping on a - , and because a shorter syntax for category comments is now available

web/username_prefix.php: category comments: moved _-_ outside [] in comparison example since it will be deleted separately

bugfix: web/username_prefix.php: category comments: need /__/ around regexp

web/username_prefix.php: category comments: added support for a shorter syntax than [x_-_]y@url : x+y@url -> y@url

schemas/vegbien.ERD.mwb: updated

fix: inputs/.TNRS/schema.sql: taxon_match: added back * prefixes on TNRS-namespace column names

inputs/.TNRS/schema.sql: MatchedTaxon: taxon_best_match: don't alias to `s` since this is no longer a nested select

inputs/.TNRS/schema.sql: MatchedTaxon: nested select: use just taxon_best_match since this no longer performs renamings

inputs/.TNRS/schema.sql: MatchedTaxon: renamed output columns to match input columns, so that the nested select would not be performing any renamings

fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)

fix: inputs/.TNRS/schema.sql: reordered columns to match taxon_match

fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)

fix: inputs/.TNRS/schema.sql: added back Family_matched, which had gotten merged with Name_matched_accepted_family when TNRS temporarily stopped populating Name_matched_accepted_family

inputs/.TNRS/schema.sql: MatchedTaxon: taxonomicStatus: moved outside the inner SELECT so that the inner SELECT would consist solely of renamings

inputs/.TNRS/schema.sql: MatchedTaxon: removed unneeded "Name_matched." prefix on source-specific names (only the * is needed because there is only one table)

bugfix: schemas/util.sql: view_is_subset(): auto-updatable views that are not plain subsets: include ones with WHERE clauses

bugfix: schemas/util.sql: view_is_subset(): also need to handle auto-updatable views that are not plain subsets (there are some of these)

bin/import_all: hidden_srcs(): removed `by_col=1` because these should be done in the same mode as the main datasources

schemas/public_.sql: removed no longer used threatened_taxonlabel. use iucn_red_list instead.

fix: schemas/public_.sql: analytical_stem_view: don't use threatened_taxonlabel, which was never populated correctly. instead, this can eventually be rewritten to use new iucn_red_list.

bin/make_analytical_db: removed threatened_taxonlabel because this is now handled by iucn_red_list

bin/make_analytical_db: added iucn_red_list_view_modify()

bin/make_analytical_db: removed unused code to create views in the analytical_db schema

bin/make_analytical_db: merged mk_table and mk_analytical_table since they now do the same thing

bin/make_analytical_db: removed unused code to create views in the analytical_db schema

/README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it

inputs/SALVIAS/Source/header.csv: updated

bugfix: bin/with_all: isset(): need to use `&>/dev/null` instead of `>&-`, etc because closing an fd causes declare to return false

bugfix: bin/with_all, import_all: don't disown processes because they should be auto-killed if the shell is (disown was only needed before we used screen)

bugfix: Full database import: need nested shell to prevent errexit from closing the window

bugfix: schemas/public_.sql: analytical_stem_view_modify(): added columns: need to include COMMENT statements

bugfix: schemas/util.sql: comment(element oid): need `objsubid = 0` filter so this doesn't use comments for any other objsubids

schemas/public_.sql: added iucn_red_list_view_modify() and use it in iucn_red_list_view's "after updating this" instructions

bugfix: schemas/util.sql: rematerialize_view(): also need to handle the target table's dependent views

bugfix: schemas/util.sql: mk_drop_from_create(): need to use `IF EXISTS` so that the resulting statement also works if the target object does not yet exist

schemas/util.sql: mk_drop_from_create(): also support `SELECT util.drop_*()` statements, in addition to DROP

inputs/.TNRS/Source/test.xml.ref: updated

bugfix: /README.TXT: logs glob: don't unintentionally match . or ..

inputs/VegBank/stemlocation_/test.xml.ref: updated

/README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)

inputs/TEX/Specimen2/test.xml.ref: updated

inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: updated

bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add new entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.

bin/import_all: delete_logs(): documented that `trap EXIT` doesn't run until shell exit

bin/import_all: delete_logs(): print when this happens, so it can be verified that it's happening properly

bugfix: bin/import_all: need to run delete_logs manually because `trap EXIT` doesn't run until bg cmds done

bin/import_all: delete_logs: moved testing of whether to delete logs to delete_logs() so that delete_logs() can be run regardless of the $delete_logs setting

bugfix: bin/import_all: delete_logs(): also need to match log filenames when n=""

bugfix: /Makefile: postgres_restart-*: argument to echo always needs quotes

bugfix: /Makefile: $(pg_ctl-*): tell user that if it doesn't work, need to run `rm $(macPostgresDir)/postmaster.pid` and retry

/Makefile: $(pg_ctl-Darwin): documented the pg_ctl version of this command in case `brew services` doesn't work

bugfix: bin/with_all: isset(): need to use `>&- 2>&-` because &> does not work with - as the dest

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: determine if the revision works: use the in# of the query instead of waiting an arbitrary time, because this is more accurate in determining whether the import is successful

config/VirtualBox_VMs/vegbiendev/README.TXT: regressions that occurred during the revision range: named these "concurrent regressions"

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to update system time so wait times can be calculated correctly

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented that the patches are specifically for regressions that occurred during the revision range, not all bugs that were fixed in that range

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: waiting for query: increased time to min. 10 min because some revisions take longer

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: added step to type y and press ENTER

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to determine if the revision works, and then stop the import

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12994

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use for loop to avoid needing to repeat the command

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12760

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: --fuzz apparently needs to be 3 even though we have only 2 lines of context

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use --fuzz to allow patching even when surrounding lines have changed

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: sorted by revision #

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12865

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: patch: need to use --force to avoid prompting user

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn revert`: need to use `--depth infinity` when reverting more than a file

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: use `patch` instead of `svn merge` because `svn merge` sometimes inexplicably doens't apply patches

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: `svn revert` all changes instead of specific files, to support bugfixes to other files

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: use `svn merge` instead of `sed --in-place` because this can be used for more complex changes

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: documented revision

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to apply r12872 bugfix to lib/sh/util.sh because it occurred after r12025

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make` commands: use `yes|` to avoid needing to confirm prompts

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make mk_db ...`: use `yes|` to avoid needing to confirm prompts

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: to test a different revision: added step that if any of the commands after `svn up` are broken, try a different revision

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make db`: need to run `schemas/reinstall` manually because older versions of the codebase just used `schemas/install`

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented the range from within the revision should be chosen

fix: inputs/.IRMNG/*/: added _no_import because this datasource is only used as a lookup table and does not need to be imported/scrubbed

config/VirtualBox_VMs/vegbiendev/README.TXT: `svn up`: documented runtime (1 min)

bugfix: added nodes/_no_import, because the NCBI import doesn't work due to a Postgres 9.3 bug (#859). _no_import used to not be needed because we didn't import the hidden sources.

fix: bin/with_all: removed debug statements

bugfix: bin/with_all: testing if @inputs is set: `"${inputs+isset}"` syntax doesn't work for empty arrays, so need to use `declare -p` instead

bugfix: to set up the VM: to use a database backup: also need to `make rm_initial_public` because initdb creates a non-empty DB

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to test a different revision

/Makefile: db: use reinstall instead of install to allow running this on an existing DB

config/VirtualBox_VMs/vegbiendev/README.TXT: `sudo apt-get dist-upgrade`: documented runtime (12 min)

config/VirtualBox_VMs/vegbiendev/README.TXT: updated VM snapshot names