schemas/public_.sql: removed no longer used threatened_taxonlabel_view. use iucn_red_list_view instead.
schemas/public_.sql: viewFullOccurrence_individual_view: documented materialize time (22 h)
inputs/.TNRS/schema.sql: taxon_match: removed no longer used scrubbed_unique_taxon_name. the scrubbed name ranks are now generated from the other TNRS columns instead.
/README.TXT: Full database import: added steps to re-run geoscrubbing
exports/geoscrub_input.csv.run: export_(): updated runtime (30 s)
fix: exports/viewFullOccurrence_*.csv.run: need to limit # rows to 5000 as requested by Bob
schemas/public_.sql: viewFullOccurrence_*: documented materialize time
bin/make_analytical_db: materialize viewFullOccurrence_individual_view instead of analytical_stem_view because analytical_stem_view is now generatable via a simple join onto viewFullOccurrence_individual_view. this avoids running into potential disk space constraints when materializing and backing up both tables (~50 GB/table * 2 tables * 2 copies (incl. the backup) = 200 GB, which is very close to the available disk space).
schemas/public_.sql: analytical_stem_view: use new materialized viewFullOccurrence_individual
schemas/public_.sql: added viewFullOccurrence_individual by running viewFullOccurrence_individual_view_modify()
schemas/public_.sql: added viewFullOccurrence_individual_view_modify(), analogous to analytical_stem_view_modify()
bin/make_analytical_db: removed extra () around psql_verbose_vegbien
bin/make_analytical_db: removed no longer used mk_table()
bin/make_analytical_db: use more up-to-date *_view_modify() functions instead of mk_table()
inputs/.TNRS/schema.sql: removed no longer used view ValidMatchedTaxon. use taxon_scrub instead.
schemas/public_.sql: iucn_red_list_view: use taxon_scrub instead of ValidMatchedTaxon since they are equivalent
inputs/.TNRS/schema.sql: taxon_scrub: use taxon_best_match directly, to avoid the need for a separate ValidMatchedTaxon view
fix: inputs/.TNRS/schema.sql: taxon_scrub: merged synonymous columns
schemas/vegbien.sql: taxon_scrub: documented steps to merge synonymous columns
inputs/.TNRS/schema.sql: removed no longer used view MatchedTaxon. use taxon_best_match instead.
inputs/.TNRS/schema.sql: ValidMatchedTaxon: use taxon_best_match now that it's equivalent to MatchedTaxon
bugfix: schemas/public_.sql: tnrs_input_name: use taxon_match instead of taxon_best_match because there is no index on taxon_match that includes just the filters used by taxon_best_match
schemas/public_.sql: tnrs_input_name: use taxon_best_match now that it's equivalent to MatchedTaxon
backups/TNRS.backup.md5: updated
fix: inputs/.TNRS/schema.sql: MatchedTaxon: merged synonymous columns
fix: schemas/util.sql: view_is_subset_or_renaming(): views with CASE statements are also not subsets/renamings
inputs/.TNRS/schema.sql: removed no longer used taxon_scrub.scrubbed_unique_taxon_name.* . use taxon_scrub instead.
inputs/.TNRS/schema.sql: taxon_scrub: use taxon_match derived columns instead of the incorrect values in taxon_scrub.scrubbed_unique_taxon_name.* (which does not work with the multi-match strategy)
inputs/.TNRS/schema.sql: MatchedTaxon: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.
inputs/.TNRS/schema.sql: taxon_scrub: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.
added exports/viewFullOccurrence_{CVS,VegBank,NCU}.csv.run
schemas/public_.sql: added viewFullOccurrence_{CVS,VegBank,NCU} for Bob
bugfix: schemas/util.sql: added GRANT USAGE for bien_read, public_ so that util schema functions can be called by other users
inputs/.TNRS/schema.sql: taxon_match: to port derived column changes to vegbiendev: derived_cols_export() code: documented runtime (6 h)
bugfix: inputs/.TNRS/schema.sql: removed no longer used derived column __accepted_infraspecific_label, which had a buggy formula that broke derived_cols_populate()
bugfix: schemas/util.sql: drop_derived_col(): also need to run util.derived_cols_trigger_update()
fix: inputs/.TNRS/schema.sql: taxon_match: to remove a column: updated instructions
bugfix: lib/sh/util.sh: str2varname(): need to lowercase str because on case-insensitive filesystems, paths sometimes canonicalize to a different capitalization than the original
lib/sh/util.sh: added lowercase()
bugfix: lib/sh/util.sh: die(): need stub since this is invoked before it's defined
bugfix: lib/sh/util.sh: setup_log_fd(): don't change $log_fd to stdlog until stdlog is set up, to avoid "Bad file descriptor" errors
**: updated to use the local machine's new hostname, frenzy
lib/sh/util.sh: func_override(), copy_func(): added echo_func to facilitate debugging
bugfix: lib/sh/util.sh: stubs: log++ alias also needs to be moved to stub section
bugfix: lib/Firefox_bookmarks.reformat.csv: URLs: match only the uppercase tags used by Firefox, not any lowercase tags added by the user
fix: lib/Firefox_bookmarks.reformat.csv: page's self-description: updated comment to match regexp
bugfix: lib/Firefox_bookmarks.reformat.csv: page's self-description: updated "page's self-description: " prefix to remove
web/links/index.htm: updated to Firefox bookmarks: added links for SSD vs. HDD reliability
/README.TXT: added steps to restore from Time Machine
/README.TXT: added steps to back up the local machine's hard drive
/README.TXT: to synchronize a Mac's settings with my testing machine's: renamed to "to back up the local machine's settings"
/README.TXT: to synchronize a Mac's settings with my testing machine's: removed download section because we don't use this
added backups/vegbien.r14089.backup.md5
bin/after_import: use new bin/make_backups
bugfix: bin/make_backups: need to `popd` when done
bugfix: bin/make_backups: need to `set +x` when done
bin/make_backups: run with initial "." so background processes will be owned by the invoking shell
added bin/make_backups
web/links/index.htm: updated to Firefox bookmarks: VirtualBox: performance: added that context-switching also imposes an overhead, in addition to sandboxing.
web/links/index.htm: updated to Firefox bookmarks: BIEN: added list of machines this has been tested on. VirtualBox: added links to performance considerations of running in a VM.
inputs/.TNRS/schema.sql: added new derived columns to derived views
fix: schemas/util.sql: view_def_to_orig(): need to support aliased columns (produced when a column is renamed)
schemas/util.sql: added aliased_col_re()
schemas/util.sql: added alias_re()
fix: schemas/util.sql: view_is_subset(): renamed to view_is_subset_or_renaming() because this also supports views that just rename columns, which should not be .*-ed by view_def_to_orig()
fix: schemas/util.sql: view_def_to_orig(): can't use FROM to alias util.col_re() because that prevents inlining the function
schemas/util.sql: view_def_to_orig(): use util.col_re() for clarity
schemas/util.sql: added col_re()
fix: schemas/util.sql: derived_col_update(): also need steps to drop column, because DROP __ CASCADE doesn't work when there are dependent views
inputs/.TNRS/schema.sql: _accepted_infraspecific{rank,epithet}: use array slice of new _accepted{genus,specific_epithet,infra_{rank,epithet}}, which is simpler than using remove_prefix() in __accepted_infraspecific_label
inputs/.TNRS/schema.sql: "[accepted_]genus__@DwC__@vegpath.org": don't need to use *Accepted_name anymore because _accepted{genus,specific_epithet,infra_{rank,epithet}} is now generated from *Accepted_name
inputs/.TNRS/schema.sql: taxon_match."__accepted_{genus,specific_epithet}": renamed to "__accepted_{genus,specific_epithet,infra_{rank,epithet}}" since this now includes these other ranks as well
bugfix: inputs/.TNRS/schema.sql: taxon_match."__accepted_{genus,specific_epithet}": use "*Accepted_name" instead of "Accepted_species[_binomial]__@TNRS__@vegpath.org" (from "*Accepted_name_species") because Accepted_name_species apparently sometimes does not match the Accepted_name and uses malformed Unicode characters
backups/README.TXT: added backup retention policy
bugfix: schemas/util.sql: derived_cols_export(): also need to first run util.derived_cols_remove() if changing the derived column order
schemas/util.sql: added derived_cols_remove(), which allows derived columns to be re-created in a different order
schemas/util.sql: added drop_derived_col(), which ensures that drop_column() only cascades to views
fix: schemas/util.sql: drop_column(): also need to recreate dependent views
schemas/util.sql: added drop_constraint()
inputs/.TNRS/schema.sql: taxon_match: `inputs/.TNRS/data.sql.run refresh`: documented runtime (1 min)
bugfix: inputs/.TNRS/schema.sql: taxon_match: use "Accepted_species[_binomial]__@TNRS__@vegpath.org" instead of "*Accepted_name_species". this fixes a bug in __accepted_infraspecific_label where Accepted_name_species with trailing whitespace could not be prefix-removed from names that contained just a species binomial.
fix: inputs/.TNRS/schema.sql: taxon_match: added derived column "Accepted_species[_binomial]__@TNRS__@vegpath.org", which removes trailing whitespace
inputs/.TNRS/schema.sql: added steps to remove a column and to add a non-derived column
inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: merged into "to add a new derived column"
inputs/.TNRS/schema.sql: to add columns in the middle: renamed to "to move a column to the middle" for clarity
inputs/.TNRS/schema.sql: to populate a new column: updated to use util.derived_col_update()
fix: inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns: also need to run util.recreate_view()
inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: don't need to run `rm=1 inputs/.TNRS/data.sql.run` because this is now run by `make schemas/remake`
schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching
bugfix: inputs/.TNRS/schema.sql: taxon_match.__accepted_infraspecific_label: need to use case-insensitive matching of the removed prefix because TNRS lowercases part of the Accepted_name
schemas/util.sql: added case-sensitive-optional versions of starts_with(), ends_with()
schemas/util.sql: added canon_case(str text, case_sensitive boolean)
schemas/util.sql: check_constraint_disable(): documented that we can't use ADD CONSTRAINT NOT VALID for this because it does not disable the constraint completely, but instead causes new rows to be validated against it. the ADD CONSTRAINT NOT VALID behavior does not work for the derived columns use case of this function.
schemas/public_.sql: *_view_modify(): added repopulate param, which is passed to rematerialize_view()