fix: lib/sh/util.sh: rel_path(): log this internal command at a higher log_level so it's normally hidden
fix: lib/sh/util.sh: log_msg!(): log split_lines at a higher log_level so it's normally hidden
bugfix: lib/sh/util.sh: stderr_matches(): `log_local; log++` should apply to just stdout_contains() and part of stderr2stdout() rather than all of stderr_matches()
inputs/Madidi/_src/: set svn:ignore
added backups/vegbien.r13002.backup.md5, vegbien.r13160.backup.md5
backups/TNRS.backup.md5: updated
lib/sh/util.sh: stderr_matches(): run at higher log_level because error-handling internals should not be logged by default
bugfix: lib/sh/db.sh: mysql_ctl(): need to ignore errors if not running
bugfix: lib/sh/util.sh: stderr_matches(): handle any error: only ignore_e if the error exit status was associated with the matched error message
bugfix: lib/sh/util.sh: stderr_matches(): handle any error: need force-exit with rethrow_exit() because caller's test of return status disables errexit
lib/sh/util.sh: added rethrow_exit(), which exits even where errexit is disabled
bugfix: lib/sh/db.sh: mysql_snapshot(): need to run `mysql_ctl start` even if there is an error
lib/sh/db.sh: mysql_snapshot(): $to: default to $from.bak
lib/sh/db.sh: added mysql_snapshot()
lib/sh/db.sh: added mysql_ctl()
lib/sh/db.sh: pg_cmd(): updated to use new sudo()
lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions
fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable
lib/sh/util.sh: sudo alias: use function instead so this can be overridden
lib/sh/util.sh: added is_intern()
lib/sh/util.sh: is_callable(): use just $1 because multiple args are not applicable
lib/sh/util.sh: added is_world_executable()
lib/sh/util.sh: added has_perms()
lib/sh/util.sh: esc_args(): renamed to just esc() because this can also be used on a single value
lib/sh/util.sh: added is_extern()
lib/sh/util.sh: added sudo alias to alias-expand command
lib/sh/db.sh: pg_cmd(): $as_root: use $sudo
lib/sh/util.sh: added $sudo
lib/sh/util.sh: added cp alias
lib/sh/db.sh: removed no longer used pg_as_root(), which was buggy anyway. use `as_root=1 ...` instead.
lib/sh/db.sh: mysql_ANSI: fixed comment
added backups/users.sql.run
lib/sh/db.sh: pg_dump(): support dumping entire cluster, and cluster users
lib/sh/db.sh: pg_cmd(): added $as_root switch
fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: preserve datasets with ipr_specific = '', because they are actually redistributable, according to Brad (http://wiki.vegpath.org/2014-04-17_conference_call#conditions-of-use)
web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: added operator classes. added backups: filesystem-level backup, continuous archiving, WAL logging, etc. virtual collaboration: updated annotations.
lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format=plain: removed comment that this is the plain format, because this is now self-documenting
lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format: use the long form of the formats to make the code self-documenting
validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB
validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql
schemas/vegbien.ERD.mwb: regenerated exports
fix: lib/PostgreSQL-MySQL.csv: preserve schema assignments by translating `SET search_path` to `USE`
schemas/vegbien.ERD.mwb: added geoscrub, TNRS tables, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema)
schemas/Makefile: vegbien.sql: also include geoscrub, TNRS schemas, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema). this involves having a separate public_.sql file for restoring the public schema.
bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: updated filter condition to match output query
inputs/NY/run: `make inputs/NY/validate`: updated runtime (8 min, with added queries)
fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks
bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()
bugfix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13*: also need to include coordinate pairs which have one of their coordinates NULL, by using OR instead of AND
bugfix: inputs/NY/validations.sql: _specimens_13b_list_of_all_decimal_lat_long: matched column types to output query
bugfix: inputs/NY/validations.sql: _specimens_13a_list_of_all_verbatim_lat_long: matched column types to output query
inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: added breakdowns _specimens_13a_list_of_all_verbatim_lat_long, _specimens_13b_list_of_all_decimal_lat_long to help troubleshoot the diff
fix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: count lat/longs together instead of separately, because the DISTINCT is by coordinate pair, not individual coordinate value (which wouldn't make much sense)
bugfix: schemas/vegbien.sql: rm_output_queries(): need to account for the fact that util.truncated_prefixed_name_regexp() returns a whole-string regexp. this drops support for removing output queries with a particular group prefix, which we no longer use.
bugfix: schemas/vegbien.sql: rm_output_queries(): need to include relations whose names were truncated, as well
fix: schemas/vegbien.sql: public_validations schema comment: to remove a validations query so its columns can be changed: use rm_output_queries() rather than rm_query_view() because that also removes input queries
bugfix: schemas/util.sql: is_castable(): need to pass NULL through, for proper NULL propagation
fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: use new is_castable(), which is much more accurate than Brad's custom regexp for determining if something is numeric
inputs/NY/validations.-.util.sql: added util.is_castable() wrapper
schemas/util.sql: added is_castable()
schemas/util.sql: added try_cast()
schemas/util.sql: added util.cast(), which allows casting to an arbitrary type without eval()
bugfix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: DISTINCT: added coordsaccuracy_m
bugfix: schemas/vegbien.sql: coordinates_unique: added coordsaccuracy_m
fix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because the coordinates_unique unique constraint includes other columns as well, so there may be multiple instances of each lat/long
bugfix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to include both lat and long in the value to DISTINCT on
fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because they are merged by the coordinates_unique unique constraint in the import
validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: integrated row labels into table
validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: added line for different rows (vs. missing/extra)
inputs/NY/run: `make inputs/NY/validate`: documented slow queries: _specimens_12_distinct_collector_name_collect_num_date_w_count
inputs/SALVIAS/run_: `make inputs/SALVIAS/validate`: documented slow queries (_plots_06a_list_of_stems). these may need to have their query plans rechecked.
inputs/NY/run, inputs/SALVIAS/run_: `make inputs/.../validate`: updated runtime (+2 min)
fix: inputs/NY/validations.sql: specimens*_of_unique_verbatim_author_taxa_with_genus: use scientificName rather than the concatenated ranks, because that is what is imported to taxonlabel.taxonomicname
fix: schemas/vegbien.sql: specimens*_of_unique_verb_subsp_taxa_with_author: include only names with subspecies (filtering by taxonverbatim.subspecies rather than taxonlabel.taxonomicname)
bugfix: /README.TXT: Full database import: to import just a subset of the datasources: array env var needs to be set after opening the `screen` shell because array vars are apparently not inherited by the `screen` shell
/README.TXT: Full database import: to import just a subset of the datasources: added step to set custom import name
/README.TXT: Full database import: added instructions for importing just a subset of the datasources
bugfix: lib/sh/util.sh: local_array/export_array: do need -a because that it's an array is apparently not autodetected by the () on Mac
mappings/VegCore-VegBIEN.csv: mapped subspecies to new taxonverbatim.subspecies for easier access by validations queries
bugfix: web/.phpPgAdmin/.htaccess: work around phpPgAdmin bug that causes page to be ignored when not logged in
fix: inputs/test_taxonomic_names/Taxon/map.csv: scientificName: remapped to scientificName instead of taxonName as this does include the author for some names
fix: inputs/NY/Ecatalog_all/map.csv: ScientificName: remapped to scientificName instead of taxonName as this does include the author
fix: inputs/NY/validations.sql: specimens*_of_unique_verb_subsp_taxa_with_author: use taxonName instead of concatenating the ranks, as that corresponds to what we use as the concatenated taxonomic name
bugfix: inputs/NY/validations.sql: specimens*_of_verbatim_subspecific_taxa_with_author: need `subspecies IS NOT NULL` filter
bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: need to include subspecies (as _specimens_06_count_of_unique_verb_subsp_taxa_with_author does)
web/.phpPgAdmin/.htaccess: extract path components 1st->last: documented that can't use subject param for this because that goes to the last selected tab, not the default (leftmost) tab
bugfix: inputs/NY/validations.sql: specimens*_of_species_binomials: removed incorrect `subspecies IS NOT NULL` filter (this should be on *_of_unique_verb_subsp_taxa_with_author instead)
schemas/vegbien.sql: taxonverbatim: added subspecies, as decided in the conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema-2)
fix: schemas/vegbien.sql: plots* with duplicated rows: removed duplicated rows
schemas/vegbien.sql: specimens*: ran through pipeline
removed old version validation/aggregating/plots/SALVIAS/bien3_validations_salvias_db_original.sql. use validation/aggregating/plots/SALVIAS/_archive/bien3_validations_salvias_db_original.sql instead.
schemas/vegbien.sql: _specimens_16_list_distinct_specimen_descriptions: re-ran through pipeline after removing duplicated rows
schemas/vegbien.sql: rm_output_queries(): also support removing just a particular output query
