Added filter_out_cs
README.TXT: Data import: Added step to ensure there are no local modifications using `svn st`
make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_
README.TXT: Data import: After import: Also check that the provider_count table contains entries for all inputs
Added inputs/.geoscrub/_src/geovalidity-table.txt, which was attached to Jim's geovalidation e-mail (provided in README.TXT)
inputs/import.stats.xls: Updated import times
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated column group header to "By column"
backups/Makefile: Removed no longer used $(psqlVerbose)
backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB
backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien
import_all: Allow caller to override $dump_opts
pg_dump_vegbien: Renamed $opts env var to $dump_opts to avoid conflicting with other commands' vars of the same name
mappings/VegCore.csv: Regenerated from wiki
schemas/vegbien.sql: location: Removed location_unique_within_parent_by_sourceaccessioncode, which duplicates location_unique_within_creator_by_sourceaccessioncode because the sourceaccessioncode is globally unique
schemas/vegbien.sql: analytical_stem_view: projectID: Use project.projectname if project.sourceaccessioncode isn't provided
schemas/vegbien.sql: location: location_unique_within_parent: Split into *_by_sourceaccessioncode and *_by_authorlocationcode_position, with each ID being matched separately. This way, if the initial import of a subplot's location provides both fields, but fkey references use only one field, the fkey references will still match the existing location because only one of the fields needs to match.
schemas/vegbien.sql: analytical_stem_view: elevationInMeters: Use parent_location.elevation_m when location.elevation_m not provided
schemas/vegbien.sql: analytical_stem_view: scientificName: Fixed bug where need to use accepted_taxon*label*.taxonomicname instead of accepted_taxonverbatim.taxonomicname, because taxonverbatim's name component fields aren't populated if the name doesn't match a scrubbed name. The datasource's own taxonverbatim can't be used for this because the canon_label_id refers to the concatenated taxonomic name owned by the TNRS datasource.
inputs/NVS/Plot/map.csv: Corrected Plot ID mapping to go to subplotID instead of locationID, because each subplot gets its own ID in this field
schemas/vegbien.sql: location: location_unique_within_parent: Also apply this constraint when sourceaccessioncode is provided, because it may be a concatenated value populated for use by the analytical DB but which is not used as an fkey by the datasource itself
schemas/vegbien.sql: analytical_stem_view: locationID: Concatenate parent location's and subplot's IDs using '; ' instead of ' '
schemas/vegbien.sql: analytical_*: Renamed locationName to locationID because it's now globally unique (within the datasource) and can be used as a sourceaccessioncode
schemas/vegbien.sql: analytical_stem_view: locationName: For subplots without their own sourceaccessioncode (globally unique ID), prepend the parent location's unique ID so that locationName is globally unique
mappings/VegCore-VegBIEN.csv: locationID/locationName + subplot -> location.sourceaccessioncode mapping: Fixed bug where subplot was incorrectly being mapped to this field even when there was no location*. (This field can only be populated if both location* and subplot are specified.) Also only map locationID for this, to avoid inconsistencies where one table supplies locationID+subplot, while another table supplies locationName+subplot, but they both get mapped to the same field, preventing plots from being matched up with their observations when creating the analytical_stem.
xml_func.py: Simplifying functions: Logic: _and(), _or(): Evaluate an expression of only constant values
lists.py: Added and_(), or_()
xml_func.py: is_scalar(): Fixed bug where need to check if value is a string before calling is_var_name()
inputs/NVS/StemObservation/map.csv: Remapped Verbatim Code to authorTaxonCode, because as it's used this is actually an identifier for the taxon, not the stem, despite Nick Spencer's revised mapping
schemas/vegbien.ERD.mwb: Regenerated exports
README.TXT: Schema changes: Update graphical ERD exports: Added step to commit changes
inputs/NVS/*/map.csv: Remapped with Nick Spencer's suggested changes
xml_func.py: _first(): Fixed bug where need to choose the first non-empty param, by first pruning empty child nodes
mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode
schemas/vegbien.sql: analytical_*: Renamed individualID to individualObservationID because this actually corresponds to plantobservation.sourceaccessioncode, which is an observation of an individual
README.TXT: Data import: Recording the import times: Changed <version> back to $version because these commands are actually run on vegbiendev, where $version is set. (Modifications to import.stats.xls would be made on your local machine.)
README.TXT: Data import: Added step to unset $version before starting the import, to avoid importing on top of the last import's data
README.TXT: Data import: Replaced $version with <version> where it needs to be manually filled in
README.TXT: Data import: On nimoy: Added command to set $version
mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only use authorTaxonCode if there is no plant ID, because an individual plant gets its own taxonoccurrence and thus needs the taxonoccurrence's IDs to be unique to the plant, regardless of what the author designates as the taxonoccurrence code
Generated inputs/NVS/new_terms.csv
input.Makefile: SVN: $(svnFilesGlob): Also match *terms.csv in top-level dir
mappings/VegCore-VegBIEN.csv: Mapped authorTaxonCode
README.TXT: Maintenance: VegCore data dictionary: Added step to commit updated mappings/VegCore.csv
schemas/Makefile: %/publish: Fixed bug where commands were not being run transactionally, because --single-transaction requires `--file -` to work properly
input.Makefile: Editing import: Removed rotate because appending the current svn revision doesn't make sense, since this is not related to the revision used to import the datasource
input.Makefile: Editing import: Added rename/% and use it in rotate
schemas/Makefile: Use $* instead of $(@D) for clarity. $(@D) is only needed when the dir part of the target includes a prefix in addition to the % stem.
make_analytical_db: Automatically call export_analytical_db when finished
schemas/vegbien.sql: make_family_higher_plant_group(): Added `taxonepithet IS NOT NULL` filter, to allow make_analytical_db to proceed even when the NCBI import fails (leaving some nodes with rank = 'family' but no associated taxonepithet). The most recent NCBI import failed due to the search_path/DuplicateException bug resulting from the import schema and public being in the search_path together.
schemas/Makefile: Fixed bug where need `SHELL := /bin/bash` for \$(confirmRmPublicSchema) to work correctly
lib/common.Makefile: $(confirm): Added comment that this requires `SHELL := /bin/bash` to work correctly
import_all: after_import(): Added `make backups/vegbien.$version.backup/test`
sql.py: DbConn._db(): search_path: Don't append the existing search_path, because it usually includes the public schema, which is now different from the schema being imported into. This fixes a bug where sql.function_exists() would find public-schema functions in both the public schema and the import's schema because both were in the search_path, causing a DuplicateException "more than one function named ...". Note that the elements of the existing search_path are no longer needed now that vegbien_dest's $schemas includes $public. Also note that if an instance of DbConn does not specify the schemas param, the existing search_path will be left as-is rather than overwritten with an empty list.
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Added step to determine the import date using import_date
import_date: Added note that Mac and Linux differ in the order they sort the logs in
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated pattern for new log filename format
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Removed extra ./ before bin/import_times
import_date: Added note that the time this outputs is the time the first special input finished importing. The import itself generally starts a few minutes before that, and the exact time is in that import's public schema comment.
import_date: Removed duplicate Usage message at top of file, which is repeated in the Usage message provided when the program is run with no arguments
Added import_date
Added mtime
lib/common.Makefile: System: Added $(mtime)
lib/common.Makefile: $(date): Factored date format out into $(dateFmt)
backups/Makefile: Factored $(isMac) out into lib/common.Makefile
README.TXT: Data import: tailing logs: Updated pattern for new log filename format
schemas/Makefile: Installation: %/publish: Fixed bug where need quotes around source schema name
README.TXT: Data import: Moved deletion previous imports before the import, so that full DB backup can be automated
README.TXT: Data import: `make backups/vegbien.$version.backup/test`: Added --exclude-schema=public to leave out the previous (now published) import so it doesn't bloat the backup. Note that public is included in the vegbien.$version.backup for the previous import, named according to its version.
import_all: after_import(): Added `make backups/TNRS.backup-remake`
README.TXT: Data import: Added step to publish the import to the public schema
import_all: after_import(): Added export_analytical_db
README.TXT: Data import: bin/export_analytical_db: Removed `env public=$version` because export_analytical_db now uses $version as $public when provided
README.TXT: Data import: To remake analytical DB: Removed `env public=...` because $version (which replaces $public) is now set automatically by import_all
schemas/Makefile: Installation: py_functions/install: Removed `env public=`, which is not needed since $(psqlAsAdminVegbien) does not use psql_script_vegbien (which uses $public)
export_analytical_db: Use vegbien_dest to set the default value for $public
README.TXT: Data import: If many inputs have errors: Updated command to `make schemas/$version/uninstall` because the current import's schema is now named $version
schemas/Makefile: Installation: $(schemas), $(schemasReversed) (used e.g. by `make schemas/reinstall`): Removed public so that when `make schemas/reinstall` is run before an import, it will not remove any active (published) import which resides in the public schema
README.TXT: Schema changes: Reinstall public separately from the other schemas so that it will still be reinstalled when schemas/reinstall excludes the public schema to avoid removing any active (published) import
vegbien_dest callers: Removed no longer needed explicit setting $prefix to "", because this is now the default value
vegbien_dest: Changed default $prefix to "", so that the majority of callers don't need to manually set $prefix to "" to avoid it defaulting to out_
README.TXT: Data import: Use env var $version, which is now set by import_all, instead of manually inserting the version for <version>
vegbien_dest: Also export $version
import_all: Run the import directly into a new, already-versioned public schema. This removes the need to manually rename the schema after import, and allows the backup commands to use the stored $version shell variable to refer to the last import.
schemas/Makefile: %/publish: Added instruction to run `unset version` after the command, to clear the $version shell variable which will be set by import_all
README.TXT: Data import: Replaced <import_name> with <version> because the import name is now just the version
README.TXT: Data import: Replaced r<revision> with <version> because the version string is now equal to r<revision>
README.TXT: Backups: Replaced <date> with <version> because the date is no longer included in the version string
README.TXT: Name archived imports without the "public." prefix so that their backups will work with the new `make backups/%.backup/remove` command, which does not add back the prefix
backups/Makefile; $(public*): Don't add a "public." prefix to get the name of the public schema
backups/Makefile: Removed no longer used $(rmSchema)
backups/Makefile: Use \$(rmSchemaCmd) from lib/common.Makefile instead of \$(rmSchema)
vegbien_dest: Use $version as $public when $public not provided. When neither is provided, continue to use "public" and also set $version to that.
schemas/Makefile: Installation: rotate: Use just the version, without the "public." prefix
schemas/Makefile: Installation: `public/install public%/install`: Generalized to /install to allow public schema versions with any name. This requires moving `/install: %.sql` before it to override it.
schemas/Makefile: Installation: Merged public/install and public%/install
