import_all: Make $import_source a local var, so it will be automatically unset if the script is aborted
vegbien_dest: Schema override for referring to a table in the $public schema: Only process the override when $!schemaVar and $!tableVar are non-*empty*, to allow setting $schema=""
schemas/Makefile: DDL generation: vegbien.sql: Unset $dump_opts so that pg_dump does not use env vars left after running import_all
schemas/Makefile: DDL generation: vegbien.sql: Unset $version so that pg_dump always uses the public schema, even after running import_all
mappings/VegCore.csv: Regenerated from wiki
README.TXT: Testing: Added commands to put in .profile on a development machine
import_all: Added command to add scrubbed taxondeterminations
import_all: Start tnrs-remake after starting the inputs, so that for subset imports (e.g. n=2), there will already be names to scrub when tnrs-remake starts up and it won't enter pause mode to wait for new rows (the pause is calibrated for full imports, and is too long for subset imports)
with_all: Also exclude .archive/ from the subdirs to forward commands to
inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Added option to wait for new rows, in the same way tnrs_db does
inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Factored new rows added test out into rowsAdded() function
Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make, which adds scrubbed taxondeterminations to VegBIEN
root Makefile: Removed $(subMake), which is now defined properly by lib/common.Makefile
lib/common.Makefile: $(subMake): Removed `--makefile=../input.Makefile`, which is specific just to inputs/Makefile
input.Makefile: Import to VegBIEN: $(import): Print the date at the beginning of the import, so successive imports to the same version can be distinguished
input.Makefile: Import to VegBIEN: \$(import): Fixed bug where 2>&1 needs to come after >>$(log_) rather than before
inputs/.TNRS/tnrs/tnrs.make: Usage: Added tnrs_db's $wait flag
inputs/.TNRS/tnrs/tnrs.make: Fixed Usage message to use make, which is needed to set the PATH correctly
Makefiles: Changed "Usage: `make -s ...`" to "Run with `make -s` to avoid echoing make commands"
input.Makefile: Import to VegBIEN: Added %/log_file to view the import log file path
input.Makefile: Import to VegBIEN: $(import): Append to the log file instead of replacing it, to avoid overwriting the log for a previous import to the same versioned schema. This allows a datasource to be (re-)imported multiple times, and is needed by the new method for linking taxonoccurrences to scrubbed taxonomic names.
input.Makefile: Import to VegBIEN: $(import): Always output just to log file if $(log) is on, rather than also copying output to the terminal when $(n) is set. When $(log) is on, the output can still be viewed by tailing the log.
input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(n),...)
input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(log),...)
Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/
mappings/VegCore-VegBIEN.csv: Mapped taxonOccurrencePkey
input.Makefile: Staging tables installation: Added %_view/install, to prevent trying to edit a view during installation
vegbien_dest: Added schema override support for referring to a table in the $public schema
input.Makefile: Staging tables installation: $(cleanup): Moved setting of $schema, $table before vegbien_dest is run, so it can modify them if needed
inputs/.TNRS/tnrs/tnrs.make: Removed unnecessary setting of $prefix, which now defaults to ""
schemas/vegbien.sql: Added unscrubbed_taxondetermination_view
inputs/import.stats.xls: Moved CTFS to Deleted section
make_analytical_db: ANALYZE each table after its created so that queries use index scans instead of seq scans
schemas/vegbien.sql: sync_analytical_*_to_view(): Added datasource fkey to source.shortname so removing a datasource will also remove the corresponding rows in the analytical views
schemas/vegbien.sql: Regenerated analytical_stem using sync_analytical_stem_to_view()
input.Makefile: Editing import: rm: Time the command
schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
schemas/vegbien.sql: Fixed fkey constraint names
schemas/vegbien.sql: fkeys to source: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
schemas/vegbien.sql: commconcept: Renamed source_id back to reference_id (it was previously renamed to source_id in a bulk rename)
schemas/vegbien.sql: taxondetermination: Added back reference_id, which is different than the scoping source_id (reference_id was previously renamed to source_id in a bulk rename)
schemas/vegbien.sql: Renamed taxonconcept_concept_source_id_fkey back to taxonconcept_concept_reference_id_fkey
schemas/vegbien.sql: Renamed *_reference_id_fkey fkeys to *_source_id_fkey
inputs/CTFS/_no_import: Temporarily remove CTFS from the public DB per Rick Condit's request (due to validation issues)
import_all: Run import with $public_import set in order to exclude excluded datasources
input.Makefile: Import to VegBIEN: %/import: Don't run the import if $public_import flag is set and the datasource contains a _no_import file. This allows just excluding a datasource from the public DB, without also removing it from automated testing.
lib/common.Makefile: Added $(and), $(or), $(not)
schemas/vegbien.sql: taxondetermination: Added scoping source_id field to allow other datasources (e.g. TNRS) to make taxondeterminations. (Repurposed existing non-scoping source_id.)
make_analytical_db: Fixed bug where can't give public_ select access to all analytical_db views because this apparently adds access rather than passing through the underlying table's permissions
make_analytical_db: Give public_ select access to analytical_db views. This causes the actual access to depend on the underlying table's permissions.
mappings/Makefile: VegCore.csv: Include only terms that start with a lowercase letter or are all caps. This also avoids the need to filter out VegCore.tables.csv.
mappings/VegCore.csv: Changed line endings to \n in preparation for not running filter_out_cs on the file (which changes line endings to \r\n)
import_all: `make backups/vegbien.$version.backup/test`: Documented that this uses $dump_opts. $dump_opts must be manually set when running this command outside of import_all.
backups/Makefile: Synchronization: %/download: Download the .md5 file first, so that the user is prompted right away for their password rather than after the main file has finished downloading, by which time the password prompt has timed out
mappings/Makefile: VegCore.csv: Fixed bug where need to filter out VegCore.tables.csv case-sensitively so that field names which are the same as a table name don't get filtered out
Added filter_out_cs
README.TXT: Data import: Added step to ensure there are no local modifications using `svn st`
make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_
README.TXT: Data import: After import: Also check that the provider_count table contains entries for all inputs
Added inputs/.geoscrub/_src/geovalidity-table.txt, which was attached to Jim's geovalidation e-mail (provided in README.TXT)
inputs/import.stats.xls: Updated import times
README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated column group header to "By column"
backups/Makefile: Removed no longer used $(psqlVerbose)
backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB
backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien
import_all: Allow caller to override $dump_opts
pg_dump_vegbien: Renamed $opts env var to $dump_opts to avoid conflicting with other commands' vars of the same name
schemas/vegbien.sql: location: Removed location_unique_within_parent_by_sourceaccessioncode, which duplicates location_unique_within_creator_by_sourceaccessioncode because the sourceaccessioncode is globally unique
schemas/vegbien.sql: analytical_stem_view: projectID: Use project.projectname if project.sourceaccessioncode isn't provided
schemas/vegbien.sql: location: location_unique_within_parent: Split into *_by_sourceaccessioncode and *_by_authorlocationcode_position, with each ID being matched separately. This way, if the initial import of a subplot's location provides both fields, but fkey references use only one field, the fkey references will still match the existing location because only one of the fields needs to match.
schemas/vegbien.sql: analytical_stem_view: elevationInMeters: Use parent_location.elevation_m when location.elevation_m not provided
schemas/vegbien.sql: analytical_stem_view: scientificName: Fixed bug where need to use accepted_taxon*label*.taxonomicname instead of accepted_taxonverbatim.taxonomicname, because taxonverbatim's name component fields aren't populated if the name doesn't match a scrubbed name. The datasource's own taxonverbatim can't be used for this because the canon_label_id refers to the concatenated taxonomic name owned by the TNRS datasource.
inputs/NVS/Plot/map.csv: Corrected Plot ID mapping to go to subplotID instead of locationID, because each subplot gets its own ID in this field
schemas/vegbien.sql: location: location_unique_within_parent: Also apply this constraint when sourceaccessioncode is provided, because it may be a concatenated value populated for use by the analytical DB but which is not used as an fkey by the datasource itself
schemas/vegbien.sql: analytical_stem_view: locationID: Concatenate parent location's and subplot's IDs using '; ' instead of ' '
schemas/vegbien.sql: analytical_*: Renamed locationName to locationID because it's now globally unique (within the datasource) and can be used as a sourceaccessioncode
schemas/vegbien.sql: analytical_stem_view: locationName: For subplots without their own sourceaccessioncode (globally unique ID), prepend the parent location's unique ID so that locationName is globally unique
mappings/VegCore-VegBIEN.csv: locationID/locationName + subplot -> location.sourceaccessioncode mapping: Fixed bug where subplot was incorrectly being mapped to this field even when there was no location*. (This field can only be populated if both location* and subplot are specified.) Also only map locationID for this, to avoid inconsistencies where one table supplies locationID+subplot, while another table supplies locationName+subplot, but they both get mapped to the same field, preventing plots from being matched up with their observations when creating the analytical_stem.
xml_func.py: Simplifying functions: Logic: _and(), _or(): Evaluate an expression of only constant values
lists.py: Added and_(), or_()
xml_func.py: is_scalar(): Fixed bug where need to check if value is a string before calling is_var_name()
inputs/NVS/StemObservation/map.csv: Remapped Verbatim Code to authorTaxonCode, because as it's used this is actually an identifier for the taxon, not the stem, despite Nick Spencer's revised mapping
schemas/vegbien.ERD.mwb: Regenerated exports
README.TXT: Schema changes: Update graphical ERD exports: Added step to commit changes
inputs/NVS/*/map.csv: Remapped with Nick Spencer's suggested changes
xml_func.py: _first(): Fixed bug where need to choose the first non-empty param, by first pruning empty child nodes
mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode
schemas/vegbien.sql: analytical_*: Renamed individualID to individualObservationID because this actually corresponds to plantobservation.sourceaccessioncode, which is an observation of an individual
README.TXT: Data import: Recording the import times: Changed <version> back to $version because these commands are actually run on vegbiendev, where $version is set. (Modifications to import.stats.xls would be made on your local machine.)
