schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
schemas/vegbien.sql: fkeys to source: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
schemas/vegbien.sql: commconcept: Renamed source_id back to reference_id (it was previously renamed to source_id in a bulk rename)
schemas/vegbien.sql: taxondetermination: Added back reference_id, which is different than the scoping source_id (reference_id was previously renamed to source_id in a bulk rename)
schemas/vegbien.sql: Renamed taxonconcept_concept_source_id_fkey back to taxonconcept_concept_reference_id_fkey
schemas/vegbien.sql: Renamed *_reference_id_fkey fkeys to *_source_id_fkey
inputs/CTFS/_no_import: Temporarily remove CTFS from the public DB per Rick Condit's request (due to validation issues)
import_all: Run import with $public_import set in order to exclude excluded datasources
input.Makefile: Import to VegBIEN: %/import: Don't run the import if $public_import flag is set and the datasource contains a _no_import file. This allows just excluding a datasource from the public DB, without also removing it from automated testing.
lib/common.Makefile: Added $(and), $(or), $(not)
mappings/VegCore.csv: Regenerated from wiki
schemas/vegbien.sql: taxondetermination: Added scoping source_id field to allow other datasources (e.g. TNRS) to make taxondeterminations. (Repurposed existing non-scoping source_id.)
make_analytical_db: Fixed bug where can't give public_ select access to all analytical_db views because this apparently adds access rather than passing through the underlying table's permissions
make_analytical_db: Give public_ select access to analytical_db views. This causes the actual access to depend on the underlying table's permissions.
mappings/Makefile: VegCore.csv: Include only terms that start with a lowercase letter or are all caps. This also avoids the need to filter out VegCore.tables.csv.
mappings/VegCore.csv: Changed line endings to \n in preparation for not running filter_out_cs on the file (which changes line endings to \r\n)
import_all: `make backups/vegbien.$version.backup/test`: Documented that this uses $dump_opts. $dump_opts must be manually set when running this command outside of import_all.
backups/Makefile: Synchronization: %/download: Download the .md5 file first, so that the user is prompted right away for their password rather than after the main file has finished downloading, by which time the password prompt has timed out
mappings/Makefile: VegCore.csv: Fixed bug where need to filter out VegCore.tables.csv case-sensitively so that field names which are the same as a table name don't get filtered out
Added filter_out_cs
README.TXT: Data import: Added step to ensure there are no local modifications using `svn st`
make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_
README.TXT: Data import: After import: Also check that the provider_count table contains entries for all inputs
Added inputs/.geoscrub/_src/geovalidity-table.txt, which was attached to Jim's geovalidation e-mail (provided in README.TXT)
inputs/import.stats.xls: Updated import times
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated column group header to "By column"
backups/Makefile: Removed no longer used $(psqlVerbose)
backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB
backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien
import_all: Allow caller to override $dump_opts
pg_dump_vegbien: Renamed $opts env var to $dump_opts to avoid conflicting with other commands' vars of the same name
schemas/vegbien.sql: location: Removed location_unique_within_parent_by_sourceaccessioncode, which duplicates location_unique_within_creator_by_sourceaccessioncode because the sourceaccessioncode is globally unique
schemas/vegbien.sql: analytical_stem_view: projectID: Use project.projectname if project.sourceaccessioncode isn't provided
schemas/vegbien.sql: location: location_unique_within_parent: Split into *_by_sourceaccessioncode and *_by_authorlocationcode_position, with each ID being matched separately. This way, if the initial import of a subplot's location provides both fields, but fkey references use only one field, the fkey references will still match the existing location because only one of the fields needs to match.
schemas/vegbien.sql: analytical_stem_view: elevationInMeters: Use parent_location.elevation_m when location.elevation_m not provided
schemas/vegbien.sql: analytical_stem_view: scientificName: Fixed bug where need to use accepted_taxon*label*.taxonomicname instead of accepted_taxonverbatim.taxonomicname, because taxonverbatim's name component fields aren't populated if the name doesn't match a scrubbed name. The datasource's own taxonverbatim can't be used for this because the canon_label_id refers to the concatenated taxonomic name owned by the TNRS datasource.
inputs/NVS/Plot/map.csv: Corrected Plot ID mapping to go to subplotID instead of locationID, because each subplot gets its own ID in this field
schemas/vegbien.sql: location: location_unique_within_parent: Also apply this constraint when sourceaccessioncode is provided, because it may be a concatenated value populated for use by the analytical DB but which is not used as an fkey by the datasource itself
schemas/vegbien.sql: analytical_stem_view: locationID: Concatenate parent location's and subplot's IDs using '; ' instead of ' '
schemas/vegbien.sql: analytical_*: Renamed locationName to locationID because it's now globally unique (within the datasource) and can be used as a sourceaccessioncode
schemas/vegbien.sql: analytical_stem_view: locationName: For subplots without their own sourceaccessioncode (globally unique ID), prepend the parent location's unique ID so that locationName is globally unique
mappings/VegCore-VegBIEN.csv: locationID/locationName + subplot -> location.sourceaccessioncode mapping: Fixed bug where subplot was incorrectly being mapped to this field even when there was no location*. (This field can only be populated if both location* and subplot are specified.) Also only map locationID for this, to avoid inconsistencies where one table supplies locationID+subplot, while another table supplies locationName+subplot, but they both get mapped to the same field, preventing plots from being matched up with their observations when creating the analytical_stem.
xml_func.py: Simplifying functions: Logic: _and(), _or(): Evaluate an expression of only constant values
lists.py: Added and_(), or_()
xml_func.py: is_scalar(): Fixed bug where need to check if value is a string before calling is_var_name()
inputs/NVS/StemObservation/map.csv: Remapped Verbatim Code to authorTaxonCode, because as it's used this is actually an identifier for the taxon, not the stem, despite Nick Spencer's revised mapping
schemas/vegbien.ERD.mwb: Regenerated exports
README.TXT: Schema changes: Update graphical ERD exports: Added step to commit changes
inputs/NVS/*/map.csv: Remapped with Nick Spencer's suggested changes
xml_func.py: _first(): Fixed bug where need to choose the first non-empty param, by first pruning empty child nodes
mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode
schemas/vegbien.sql: analytical_*: Renamed individualID to individualObservationID because this actually corresponds to plantobservation.sourceaccessioncode, which is an observation of an individual
README.TXT: Data import: Recording the import times: Changed <version> back to $version because these commands are actually run on vegbiendev, where $version is set. (Modifications to import.stats.xls would be made on your local machine.)
README.TXT: Data import: Added step to unset $version before starting the import, to avoid importing on top of the last import's data
README.TXT: Data import: Replaced $version with <version> where it needs to be manually filled in
README.TXT: Data import: On nimoy: Added command to set $version
mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only use authorTaxonCode if there is no plant ID, because an individual plant gets its own taxonoccurrence and thus needs the taxonoccurrence's IDs to be unique to the plant, regardless of what the author designates as the taxonoccurrence code
Generated inputs/NVS/new_terms.csv
input.Makefile: SVN: $(svnFilesGlob): Also match *terms.csv in top-level dir
mappings/VegCore-VegBIEN.csv: Mapped authorTaxonCode
README.TXT: Maintenance: VegCore data dictionary: Added step to commit updated mappings/VegCore.csv
schemas/Makefile: %/publish: Fixed bug where commands were not being run transactionally, because --single-transaction requires `--file -` to work properly
input.Makefile: Editing import: Removed rotate because appending the current svn revision doesn't make sense, since this is not related to the revision used to import the datasource
input.Makefile: Editing import: Added rename/% and use it in rotate
schemas/Makefile: Use $* instead of $(@D) for clarity. $(@D) is only needed when the dir part of the target includes a prefix in addition to the % stem.
make_analytical_db: Automatically call export_analytical_db when finished
schemas/vegbien.sql: make_family_higher_plant_group(): Added `taxonepithet IS NOT NULL` filter, to allow make_analytical_db to proceed even when the NCBI import fails (leaving some nodes with rank = 'family' but no associated taxonepithet). The most recent NCBI import failed due to the search_path/DuplicateException bug resulting from the import schema and public being in the search_path together.
schemas/Makefile: Fixed bug where need `SHELL := /bin/bash` for \$(confirmRmPublicSchema) to work correctly
lib/common.Makefile: $(confirm): Added comment that this requires `SHELL := /bin/bash` to work correctly
import_all: after_import(): Added `make backups/vegbien.$version.backup/test`
sql.py: DbConn._db(): search_path: Don't append the existing search_path, because it usually includes the public schema, which is now different from the schema being imported into. This fixes a bug where sql.function_exists() would find public-schema functions in both the public schema and the import's schema because both were in the search_path, causing a DuplicateException "more than one function named ...". Note that the elements of the existing search_path are no longer needed now that vegbien_dest's $schemas includes $public. Also note that if an instance of DbConn does not specify the schemas param, the existing search_path will be left as-is rather than overwritten with an empty list.
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Added step to determine the import date using import_date
import_date: Added note that Mac and Linux differ in the order they sort the logs in
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated pattern for new log filename format
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Removed extra ./ before bin/import_times
import_date: Added note that the time this outputs is the time the first special input finished importing. The import itself generally starts a few minutes before that, and the exact time is in that import's public schema comment.
import_date: Removed duplicate Usage message at top of file, which is repeated in the Usage message provided when the program is run with no arguments
Added import_date
Added mtime
lib/common.Makefile: System: Added $(mtime)
lib/common.Makefile: $(date): Factored date format out into $(dateFmt)
backups/Makefile: Factored $(isMac) out into lib/common.Makefile
README.TXT: Data import: tailing logs: Updated pattern for new log filename format
schemas/Makefile: Installation: %/publish: Fixed bug where need quotes around source schema name
README.TXT: Data import: Moved deletion previous imports before the import, so that full DB backup can be automated
README.TXT: Data import: `make backups/vegbien.$version.backup/test`: Added --exclude-schema=public to leave out the previous (now published) import so it doesn't bloat the backup. Note that public is included in the vegbien.$version.backup for the previous import, named according to its version.
import_all: after_import(): Added `make backups/TNRS.backup-remake`
README.TXT: Data import: Added step to publish the import to the public schema
import_all: after_import(): Added export_analytical_db
README.TXT: Data import: bin/export_analytical_db: Removed `env public=$version` because export_analytical_db now uses $version as $public when provided
README.TXT: Data import: To remake analytical DB: Removed `env public=...` because $version (which replaces $public) is now set automatically by import_all
