schemas/vegbien.sql: taxondetermination: Added is_datasource_current, which is autopopulated to the most recent datasource-provided taxondetermination
schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_accepted_determination unique index to facilitate joining on the accepted determination
schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_matched_determination unique index to facilitate joining on the matched determination
schemas/vegbien.sql: taxondetermination: Removed notespublic, notesmgt, which are not used by VegBIEN
schemas/vegbien.sql: taxon_trait_view: scientificName: Use taxonverbatim.taxonname when taxonlabel/taxonverbatim.taxonomicname are not provided, to accommodate TNRS names. This is part of the workaround for the bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated.
schemas/vegbien.sql: taxon_trait_view: Added workaround for bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated due to a taxonlabel constraint violation, by using the taxonverbatim's taxonomicname instead in these cases. This bug, which appeared in the r7317 import, is so far not reproducible (tested on Mac OS X), so its cause is unknown, but may be caused by a bug in functions._merge_prefix(), which is run on the taxonlabel's taxonomicname but not the taxonverbatim's taxonomicname.
schemas/vegbien.sql: analytical_stem_view: Added dateIdentified, identificationRemarks per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#E-mail-on-2013-1-16)
inputs/FIA/_src/Makefile: Added extraction targets to extract zip archives
inputs/FIA/_src/download: Use new Makefile, which uses make logic to determine if a file needs to be downloaded
Added inputs/FIA/_src/Makefile, with targets to download each zip archive
schemas/vegbien.sql: analytical_stem_view: derived terms: Added _bien suffix per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#Brad-Boyles-comments)
Added inputs/FIA/_src/FIADB_version4.accdb.url
inputs/FIA/_src/download: Only run wget on files that don't yet exist
inputs/FIA/_src/download: Run wget in same directory as script to ensure files get downloaded there
inputs/FIA/_src/download: Set svn:executable
Added inputs/FIA/_src/download to download archives of CSVs for each state
to_do/timeline.2013.xls: Updated with changes during conference call
schemas/vegbien.sql: taxon_trait_view: Renamed datasource_taxonverbatim to taxonverbatim because there is now only one taxonverbatim
schemas/vegbien.sql: taxon_trait_view: Moved the taxondetermination.iscurrent filter to the join condition to allow using the taxondetermination_single_current_determination index
schemas/vegbien.sql: taxon_trait_view: Join only on the primary taxonlabel, not the accepted taxonlabel, because the scrubbed name is now available directly via the taxonlabel attached to the scrubbed taxondetermination
schemas/vegbien.sql: analytical_stem_view: Added locality
inputs/UNCC/Specimen/map.csv: accession: Remapped to catalogNumber per Bob's corrections
schemas/vegbien.ERD.mwb: Regenerated exports
mappings/VegCore.csv: Regenerated from wiki
README.TXT: Schema changes: Added instructions to run the appropriate sync function when changing the analytical views
schemas/vegbien.sql: analytical_stem_view: Added georeferenceProtocol, which is set to 'county centroid' when county centroid coordinates are used
make_analytical_db: Don't run export_analytical_db if the SQL script exits with an error
README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added `export version=<version>` because import_times may be run in a shell different from the one that the import was run in
inputs/import.stats.xls: Updated import times
schemas/vegbien.sql: taxonverbatim: taxonverbatim_unique: Added morphoname for cases when there is just a morphoname, and to distinguish taxonverbatims with the same taxonlabel but different morphonames
schemas/vegbien.sql: stemobservation: Added stemobservation_non_empty CHECK constraint to prevent creating an empty stemobservation for plantobservation rows without stem data but with stem mappings
schemas/vegbien.sql: taxonverbatim: Allow taxonlabel_id to be NULL when morphoname is provided
schemas/vegbien.sql: taxonverbatim: Added source_id to allow creating taxonverbatims without a (scoping) taxonlabel
schemas/vegbien.sql: analytical_stem_view: Removed speciesBinomialWithMorphospecies now that it's duplicated by scientificNameWithMorphospecies
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Create it using the speciesBinomialWithMorphospecies formula, per Brad's request at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#2013-1-18>
schemas/vegbien.sql: analytical_stem_view: Added coordinateSource to indicate whether coordinates are from county_centroids (georeferencing) or the source data
schemas/vegbien.sql: Added coordinatesource enum
schemas/vegbien.sql: analytical_stem_view: coordinates: Also use the county_centroids coordinates when the datasource coordinates are not geovalid. (Note that canon_place.geovalid will be NULL, i.e. not true, when the datasource coordinates are NULL.)
schemas/vegbien.sql: scientificName: Set to taxonverbatim.taxonname instead per Brad's changes at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#2013-1-18>. Renamed to taxonName since this now doesn't include the author, which is part of DwC's scientificName field.
schemas/vegbien.sql: sync_analytical_stem_to_view(): Support running the function when dependent views do not exist. This allows using the sync function when changing column names of the analytical_stem_view, which sometimes requires manually dropping and re-creating the analytical_aggregate_view.
backups/Makefile: %.md5/test: Added comment to run with `make -s` to avoid echoing make commands
README.TXT: Full database import: Added steps to scrub unscrubbed taxondeterminations (if they are not scrubbed automatically)
inputs/.geoscrub/_src/README.TXT: Added e-mails from Jim about how the county_centroids data was generated
schemas/vegbien.sql: analytical_stem_view: coordinates: Use new county_centroids coordinates and uncertainty when the datasource's coordinates are not available
Added inputs/.geoscrub/county_centroids/ from Jim
inputs/.geoscrub/import_order.txt: Added geoscrub_output
README.TXT: Full database import: In PostgreSQL: Added step to check that there are TNRS taxondeterminations
README.TXT: Full database import: In PostgreSQL: Added step to check that unscrubbed_taxondetermination_view returns no rows
Added inputs/newWorld/newWorldCountries/_no_import
to_do/timeline.2013.xls: Updated with Brad's modifications
Added inputs/FIA/_src/FIA_summary.b-e.00079.pdf from Bob
Added inputs/.herbaria/_archive/
inputs/.herbaria/: Removed no longer needed geoscrub.*.sql, which has been replaced with bien3_adb.*.sql
inputs/.herbaria/: Removed no longer needed herbaria/. Use ih/ instead.
Added inputs/.herbaria/ih/ and corresponding bien3_adb MySQL export
mappings/VegCore-VegBIEN.csv: Don't create NCBI crosslinks for the matched taxonomic name. These crosslinks are no longer needed now that TNRS provides a separate accepted name on which crosslinks can be made.
schemas/vegbien.sql: unscrubbed_taxondetermination_view: Include the accepted name's row next to the matched name's row instead of merging the two together into one TNRS row, to allow including separate taxondeterminations for the matched and accepted names. Added Max_score from TNRS.tnrs.
schemas/vegbien.sql: taxondetermination_set_iscurrent(): Added new determinationtype accepted to sort order
mappings/VegCore-VegBIEN.csv: Mapped accepted* taxonomic name, now to separate accepted taxondetermination
schemas/vegbien.sql: taxondetermination_set_iscurrent(): Changed TNRS determinationtype from computer to matched, to allow for a separate accepted determinationtype
schemas/vegbien.sql: taxonlabel: Removed creationdate, which duplicates taxondetermination.determinationdate
schemas/vegbien.sql: analytical_stem_view: isNewWorld: Removed no longer needed COALESCE to false, because newWorldCountries now uses false where applicable instead of NULL. This also ensures that isNewWorld will be NULL if there is no country name to test, which was not the case in the previous workaround.
Added inputs/newWorld/newWorldCountries/ with postprocess.sql that sets isNewWorld to false wherever it's NULL. (The input table only marks New World countries as true, but doesn't mark non-New World countries as false.)
schemas/vegbien.sql: analytical_stem_view: isNewWorld: Fixed bug where need to COALESCE "newWorldCountries"."isNewWorld" to false, because it is only set to a boolean for countries that are New World
README.TXT: Full database import: freeing disk space: Updated import schema size, which is smaller due to the removed CTFS staging tables, removed duplicate rows, and possibly fewer index holes
README.TXT: Full database import: After running `make schemas/$version/publish`, added `unset version` to make sure future version-dependent commands use the public schema
schemas/vegbien.sql: taxon_trait_view: Fixed bug where measurementUnit needed to be set to trait.units, not name
schemas/vegbien.sql: provider_count_view: Don't set default values for sourcetype/observationtype, because the appropriate values are now set for all top-level inputs and these defaults are not applicable for data owners not in geoscrub.herbaria
inputs/bien2_traits/Source/map.csv: Mapped observationType
schemas/vegbien.sql: taxondetermination: Removed taxondetermination_computer_min_fit CHECK constraint, whose functionality is now duplicated by unscrubbed_taxondetermination_view's Max_score filter condition. The score threshold value should only be maintained in one place, namely unscrubbed_taxondetermination_view.
schemas/vegbien.sql: unscrubbed_taxondetermination_view: Fixed bug where need to filter out any names that will be rejected by taxondetermination's constraints, because otherwise, these names will stay in unscrubbed_taxondetermination_view and be repeatedly reimported
inputs/.TNRS/schema.sql: tnrs: Added Max_score column for use in filtering out names that will be rejected by taxondetermination's constraints
inputs/.TNRS/schema.sql: Renamed tnrs_populate_accepted_scientific_name() trigger to tnrs_populate_derived_fields() to accommodate additional derived fields
tnrs_db: Support multiple appended columns in the tnrs table
csvs.py: ColInsertFilter: Support adding multiple, consecutive columns
schemas/functions.sql: _max(), _min(): Put $n params all on one line to match other aggregating functions
schemas/functions.sql: _max(), _min(): Use PostgreSQL built-in functions GREATEST, LEAST instead of a query with aggregating functions
README.TXT: Added Single datasource import section with commands to import/reimport/scrub just a datasource rather than the full DB
schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent_on_delete() trigger: Fixed bug where need to suppress any foreign key exception, which occurs during a cascading delete because the associated taxonoccurrence has already been deleted, preventing any other taxondeterminations of that taxonoccurrence from being updated
input.Makefile: Taxonomic scrubbing: Added reimport_scrub
input.Makefile: Import to VegBIEN: Added reimport
input.Makefile: Taxonomic scrubbing: Added rescrub
input.Makefile: Taxonomic scrubbing: Added scrub target and use it in import_scrub
input.Makefile: Import to VegBIEN: Moved import, rm to top of section since they are top-level targets and don't depend on the variables defined for %/import
input.Makefile: Moved rm to Import to VegBIEN section
input.Makefile: Moved taxonomic scrubbing targets to separate Taxonomic scrubbing section
schemas/vegbien.sql: provider_count_view: Include only sources with at least one row. Currently (as of r7023), all entries in BIEN2's geoscrub.herbaria are also in VegBIEN, so the filter is not yet necessary, but switching to bien3_adb.ih could create source entries without data rows which should be excluded from the providers list.
import_all: Output the PIDs of the import_scrub and after_import processes, so those processes can be managed without shell job control. This is useful if the connection is lost to the remote shell running the import, which prevents using job control on the import processes.
input.Makefile: Import to VegBIEN: import_scrub: Run `make scrub` in the background, to allow the import to continue with the next table rather than having to wait for the current table to be scrubbed
inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Moved waitself call to top of script
inputs/import.stats.xls: Added Postprocessing section for use with the next import
