bugfix: lib/sh/util.sh: stderr_matches(): handle any error: need force-exit with rethrow_exit() because caller's test of return status disables errexit

lib/sh/util.sh: added rethrow_exit(), which exits even where errexit is disabled

bugfix: lib/sh/db.sh: mysql_snapshot(): need to run `mysql_ctl start` even if there is an error

lib/sh/db.sh: mysql_snapshot(): $to: default to $from.bak

lib/sh/db.sh: added mysql_snapshot()

lib/sh/db.sh: added mysql_ctl()

lib/sh/db.sh: pg_cmd(): updated to use new sudo()

lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions

fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable

lib/sh/util.sh: sudo alias: use function instead so this can be overridden

lib/sh/util.sh: added is_intern()

lib/sh/util.sh: is_callable(): use just $1 because multiple args are not applicable

lib/sh/util.sh: added is_world_executable()

lib/sh/util.sh: added has_perms()

lib/sh/util.sh: esc_args(): renamed to just esc() because this can also be used on a single value

lib/sh/util.sh: added is_extern()

lib/sh/util.sh: added sudo alias to alias-expand command

lib/sh/db.sh: pg_cmd(): $as_root: use $sudo

lib/sh/util.sh: added $sudo

lib/sh/util.sh: added cp alias

lib/sh/db.sh: removed no longer used pg_as_root(), which was buggy anyway. use `as_root=1 ...` instead.

lib/sh/db.sh: mysql_ANSI: fixed comment

added backups/users.sql.run

lib/sh/db.sh: pg_dump(): support dumping entire cluster, and cluster users

lib/sh/db.sh: pg_cmd(): added $as_root switch

fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: preserve datasets with ipr_specific = '', because they are actually redistributable, according to Brad (http://wiki.vegpath.org/2014-04-17_conference_call#conditions-of-use)

web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: added operator classes. added backups: filesystem-level backup, continuous archiving, WAL logging, etc. virtual collaboration: updated annotations.

lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format=plain: removed comment that this is the plain format, because this is now self-documenting

lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format: use the long form of the formats to make the code self-documenting

validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB

validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql

schemas/vegbien.ERD.mwb: regenerated exports

fix: lib/PostgreSQL-MySQL.csv: preserve schema assignments by translating `SET search_path` to `USE`

schemas/vegbien.ERD.mwb: regenerated exports

schemas/vegbien.ERD.mwb: added geoscrub, TNRS tables, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema)

schemas/Makefile: vegbien.sql: also include geoscrub, TNRS schemas, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema). this involves having a separate public_.sql file for restoring the public schema.

schemas/vegbien.ERD.mwb: regenerated exports

bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: updated filter condition to match output query

inputs/NY/run: `make inputs/NY/validate`: updated runtime (8 min, with added queries)

fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks

fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks

bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()

bugfix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13*: also need to include coordinate pairs which have one of their coordinates NULL, by using OR instead of AND

bugfix: inputs/NY/validations.sql: _specimens_13b_list_of_all_decimal_lat_long: matched column types to output query

bugfix: inputs/NY/validations.sql: _specimens_13a_list_of_all_verbatim_lat_long: matched column types to output query

inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: added breakdowns _specimens_13a_list_of_all_verbatim_lat_long, _specimens_13b_list_of_all_decimal_lat_long to help troubleshoot the diff

fix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: count lat/longs together instead of separately, because the DISTINCT is by coordinate pair, not individual coordinate value (which wouldn't make much sense)

bugfix: schemas/vegbien.sql: rm_output_queries(): need to account for the fact that util.truncated_prefixed_name_regexp() returns a whole-string regexp. this drops support for removing output queries with a particular group prefix, which we no longer use.

bugfix: schemas/vegbien.sql: rm_output_queries(): need to include relations whose names were truncated, as well

fix: schemas/vegbien.sql: public_validations schema comment: to remove a validations query so its columns can be changed: use rm_output_queries() rather than rm_query_view() because that also removes input queries

bugfix: schemas/util.sql: is_castable(): need to pass NULL through, for proper NULL propagation

fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: use new is_castable(), which is much more accurate than Brad's custom regexp for determining if something is numeric

inputs/NY/validations.-.util.sql: added util.is_castable() wrapper

schemas/util.sql: added is_castable()

schemas/util.sql: added try_cast()

schemas/util.sql: added util.cast(), which allows casting to an arbitrary type without eval()

bugfix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: DISTINCT: added coordsaccuracy_m

bugfix: schemas/vegbien.sql: coordinates_unique: added coordsaccuracy_m

fix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because the coordinates_unique unique constraint includes other columns as well, so there may be multiple instances of each lat/long

bugfix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to include both lat and long in the value to DISTINCT on

fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because they are merged by the coordinates_unique unique constraint in the import

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: integrated row labels into table

validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: added line for different rows (vs. missing/extra)

inputs/NY/run: `make inputs/NY/validate`: documented slow queries: _specimens_12_distinct_collector_name_collect_num_date_w_count

inputs/SALVIAS/run_: `make inputs/SALVIAS/validate`: documented slow queries (_plots_06a_list_of_stems). these may need to have their query plans rechecked.

inputs/NY/run, inputs/SALVIAS/run_: `make inputs/.../validate`: updated runtime (+2 min)

fix: inputs/NY/validations.sql: specimens*_of_unique_verbatim_author_taxa_with_genus: use scientificName rather than the concatenated ranks, because that is what is imported to taxonlabel.taxonomicname

validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql

validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB

fix: schemas/vegbien.sql: specimens*_of_unique_verb_subsp_taxa_with_author: include only names with subspecies (filtering by taxonverbatim.subspecies rather than taxonlabel.taxonomicname)

bugfix: /README.TXT: Full database import: to import just a subset of the datasources: array env var needs to be set after opening the `screen` shell because array vars are apparently not inherited by the `screen` shell

/README.TXT: Full database import: to import just a subset of the datasources: added step to set custom import name

/README.TXT: Full database import: added instructions for importing just a subset of the datasources

bugfix: lib/sh/util.sh: local_array/export_array: do need -a because that it's an array is apparently not autodetected by the () on Mac

mappings/VegCore-VegBIEN.csv: mapped subspecies to new taxonverbatim.subspecies for easier access by validations queries

bugfix: web/.phpPgAdmin/.htaccess: work around phpPgAdmin bug that causes page to be ignored when not logged in

fix: inputs/test_taxonomic_names/Taxon/map.csv: scientificName: remapped to scientificName instead of taxonName as this does include the author for some names

fix: inputs/NY/Ecatalog_all/map.csv: ScientificName: remapped to scientificName instead of taxonName as this does include the author

fix: inputs/NY/validations.sql: specimens*_of_unique_verb_subsp_taxa_with_author: use taxonName instead of concatenating the ranks, as that corresponds to what we use as the concatenated taxonomic name

bugfix: inputs/NY/validations.sql: specimens*_of_verbatim_subspecific_taxa_with_author: need `subspecies IS NOT NULL` filter

bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: need to include subspecies (as _specimens_06_count_of_unique_verb_subsp_taxa_with_author does)

web/.phpPgAdmin/.htaccess: extract path components 1st->last: documented that can't use subject param for this because that goes to the last selected tab, not the default (leftmost) tab

bugfix: inputs/NY/validations.sql: specimens*_of_species_binomials: removed incorrect `subspecies IS NOT NULL` filter (this should be on *_of_unique_verb_subsp_taxa_with_author instead)

schemas/vegbien.sql: taxonverbatim: added subspecies, as decided in the conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema-2)

fix: schemas/vegbien.sql: plots* with duplicated rows: removed duplicated rows

schemas/vegbien.sql: specimens*: ran through pipeline

removed old version validation/aggregating/plots/SALVIAS/bien3_validations_salvias_db_original.sql. use validation/aggregating/plots/SALVIAS/_archive/bien3_validations_salvias_db_original.sql instead.

validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql

validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB

schemas/vegbien.sql: _specimens_16_list_distinct_specimen_descriptions: re-ran through pipeline after removing duplicated rows

schemas/vegbien.sql: rm_output_queries(): also support removing just a particular output query

bugfix: schemas/util.sql: remake_diff_table(): need to rm_freq() type_table, because left/right_table don't have freq yet

schemas/util.sql: auto_rm_freq(): use new rm_freq()

schemas/util.sql: added rm_freq(regclass[])

fix: inputs/NY/validations.sql: _specimens_16_list_distinct_specimen_descriptions: removed duplicated rows using DISTINCT

schemas/vegbien.sql: _specimens_11_list_of_three_standard_political_divisions: ran through pipeline

fix: schemas/vegbien.sql: _specimens_11_list_of_three_standard_political_divisions: use same column names as input query

schemas/util.sql: remake_diff_table(): result table comment: documented how to display NULL values that are extra or missing

schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: ran through pipeline

fix: schemas/vegbien.sql: _specimens_12_distinct_collector_name_collect_num_date_w_count: dateCollected: also need to convert to text in GROUP BY/ORDER BY