fix: mappings/VegCore-VegBIEN.csv: taxonoccurrence.sourceaccessioncode: need to populate from aggregateOrganismObservationID when only that is available
schemas/vegbien.sql: specimenreplicate.institution_id: renamed to duplicate_institutions_sourcelist_id, as decided in the conference calls (wiki.vegpath.org/2014-03-13_conference_call#schema-changes-2)
mappings/VegCore.htm: regenerated from wiki: rename specimenHolderInstitutions to specimen_duplicate_institutions, as decided in the 2014-03-13 conference call (wiki.vegpath.org/2014-03-13_conference_call#schema-changes-2). note that most schema changes (such as this one) involve mappings changes, which are handled automatically by `inputs/run postprocess; yes|make inputs/{NVS,SALVIAS,TEAM}/test`.
inputs/input.Makefile: verify/%.out: use a *.sql file in the verify/ directory itself to generate *.out, so that each datasource can have its own set of output queries. for datasources that should share the same set of queries, they can instead be symlinked to the same file.
mappings/VegCore-VegBIEN.csv: event__participant: allow multiple event__participant columns
mappings/VegCore-VegBIEN.csv: project_participant: use [!...] negative lookahead assertion so that multiple project_participant columns will properly map to separate projectcontributor rows
moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).
mappings/VegCore-VegBIEN.csv: mapped project_participant
mappings/VegCore.htm: regenerated from wiki. merged confusingly-named TaxonObservation into TaxonOccurrence. added project_participant.
mappings/VegCore-VegBIEN.csv: mapped taxon_determination__is_current, taxon_determination__is_original
bugfix: mappings/VegCore-VegBIEN.csv: main taxondetermination: use [!isoriginal=true] instead of [!isoriginal] so that adding a manual isoriginal field does not prevent this selector from matching
mappings/VegCore.htm: regenerated from wiki. added taxon_determination__is_current, taxon_determination__is_original.
bugfix: mappings/VegCore-VegBIEN.csv: nest all taxonoccurrences inside a stratum event, so that the parent locationevent is always fully populated before child locationevents point to it. (previously, a stub parent event was created when the child event was imported first, which blocked the fully-populated parent event from being inserted later on.) this uses auto-folding (for VegBank/CVS) and auto-forwarding (for other datasources) to prune empty stratum events for taxonoccurrences that don't have strata. (see wiki.vegpath.org/Auto-folding, wiki.vegpath.org/Auto-forwarding for more info about these normalization techniques.) note that the inserted row counts stay exactly the same for all datasources except VegBank (which was being fixed), indicating that this signficant change to the mappings did not change the semantics of the import of taxonoccurrences.
bugfix: mappings/VegCore-VegBIEN.csv: stratum's locationevent: link this to the parent locationevent, so that the parent locationevent's information (such as locationeventcontributors) is accessible to the stratum's locationevent
mappings/VegCore-VegBIEN.csv: mapped event__participant
mappings/VegCore.htm: regenerated from wiki. added EventParticipant table containing event__participant.
mappings/VegCore-VegBIEN.csv: mapped stratum__name
schemas/vegbien.sql: added source_id to allow different datasources to have their own strata
mappings/VegCore.htm: regenerated from wiki. added Stratum table.
bugfix: mappings/VegCore-VegBIEN.csv: don't map datasetURL to source.url for taxa-only data (this mapping should only occur for Source tables)
mappings/VegCore-VegBIEN.csv: mapped datasetURL
fix: mappings/VegCore-VegBIEN.csv: source__modified_date: remapped to pubdate instead of datelastmodified because this is actually metadata for the source itself, rather than for the VegBIEN record of the source
mappings/VegCore-VegBIEN.csv: mapped source__modified_date (different from vegcore.vegpath.org?modified, which is for the data record)
mappings/VegCore.htm: regenerated from wiki. added source__version (= edition), source__modified_date.
mappings/VegCore-VegBIEN.csv: mapped edition
mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).
mappings/VegCore-VegBIEN.csv: mapped municipality
mappings/VegCore.htm: regenerated from wiki. added dayOfYear (=julianDay), which is different from startDayOfYear/endDayOfYear.
mappings/VegCore.htm: regenerated from wiki. Special terms: added instructions for adding a distinguishing suffix to each special term in the format special_term#suffix. this is needed for new-style import to make the resulting column name unique within the staging table.
mappings/VegCore-VegBIEN.csv: mapped DUPLICATE to nothing so that it would not be treated as an unmapped term
mappings/VegCore.htm: regenerated from wiki. Special terms: added DUPLICATE.
mappings/VegCore.htm: regenerated from wiki. moved UNUSED, PRIVATE underneath OMIT as subterms.
mappings/VegCore.htm: Regenerated from wiki
mappings/VegCore-VegBIEN.csv: cultivated, oldGrowth: use just cultivated if it's provided, rather than /_alt-ing it back with oldGrowth (which it was generated from)
bugfix: mappings/VegCore-VegBIEN.csv: fixed priority of cultivated and oldGrowth so cultivated is used first if it's available
mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.
bugfix: mappings/VegCore-VegBIEN.csv: place.geovalid: added missing /1 after _alt
schemas/vegbien.sql: place.geovalid: added latLongDomainValid to the values to _and together
mappings/VegCore-VegBIEN.csv: place.geovalid: use false instead of NULL
mappings/VegCore.htm: Renamed salvias_plots table plotMetadata to PlotMetadata because of SALVIAS refresh on nimoy
mappings/VegCore.htm: Regenerated from wiki. Added flower, fruit, commonName.
mappings/Makefile: $(vocab); bin/redmine_synonyms: Support crossed out (deprecated) terms
mappings/VegCore-VegBIEN.csv: subplotID,subplot -> location.sourceaccessioncode: Fixed bug where need /_first to handle the case where both subplotID and subplot are provided
mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field
mappings/VegCore.htm: Regenerated from wiki. Renamed specimenDonor to specimenDonorInstitution to clarify that this is an institution, not a collector.
mappings/VegCore.htm: Regenerated from wiki. Renamed specimenHolders to specimenHolderInstitutions to make it obvious that this is a list of institutions, such as would be in institutionCode in a DwC export.
mappings/VegCore.htm: Regenerated from wiki. Taxon: Moved scrubbing-related terms to MatchedTaxon.
mappings/VegCore.htm: Regenerated from wiki. Renamed MatchedTaxonDetermination/AcceptedTaxonDetermination to MatchedTaxon/AcceptedTaxon because they relate to a taxon name rather than to an application of that name to a TaxonOccurrence
mappings/VegCore.htm: Regenerated from wiki. Split specimenIndexer into specimenStampingInstitution and specimenDonor (which are often the same).
mappings/VegCore-VegBIEN.csv: Removed specimenIndexer->institutionCode mappings because the institutionCode should refer only to the specimenHolder
mappings/VegCore.htm: Regenerated from wiki. Editing: Changed ambiguous terms templates to list the name of the alternative itself in the alternatives list, instead of the ambiguous term + a source suffix. This avoids the need to choose a source suffix for each alternative of an ambiguous term, as the ambiguous term itself (without the suffix) can be listed as the synonym. Coordinates: Disambiguated verbatimCoordinateSystem.
mappings/Makefile: VegCore.thesaurus.csv: `uniq`ify the map after applying initial translations, to support mappings that appear twice (once as the result of an alternative's synonym entry and again as the result of an ambiguous term's alternatives listing)
mappings/VegCore.htm: Regenerated from wiki. Added specimenDescription.
mappings/VegCore.htm: Regenerated from wiki. Split apart identificationQualifier and taxonFit. Split apart taxonConfidence and identificationVerificationStatus. Moved identificationQualifier to new QualifiedTaxonName table.
mappings/VegCore.htm: Regenerated from wiki. Remapped organismNotes to be a synonym of occurrenceRemarks, since notes on an organism are more generally notes on an occurrence.
mappings/VegCore-VegBIEN.csv: Mapped occurrenceRemarks
mappings/VegCore-VegBIEN.csv: Only use one of collection and collectionURL instead of concatenating them when both are provided (as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>)
mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_DMS to coordinates.latitude_deg using new _dms_to_dd(text)
mappings/VegCore-VegBIEN.csv: latitude/longitude_deg,min,sec: Also mapped to the geoscrub coordinates entry
mappings/VegCore-VegBIEN.csv: latitude/longitude_sec: Fixed name, which had been incorrectly automapped to verbatim*
mappings/VegCore.htm: Regenerated from wiki. latitude/longitude: Moved _deg,min,sec components after unit alternatives so that unit alternatives following them wouldn't incorrectly be interpreted as alternatives for the *components instead of for the overall, unitless term.
mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_deg,min,sec
mappings/VegCore.htm: Regenerated from wiki. Merged coordinateUncertainty and coordinatePrecision, since datasources often use these interchangeably (the only difference is the DwC-recommended units).
mappings/VegCore.htm: Regenerated from wiki. Source: Added authors. Renamed creator to firstPublisher. Moved dataOwner to Source as list. Coordinates: latitude/longitude, coordinatePrecision: Added units disambiguation. latitude/longitude_DMS: Added components. Global IDs: Changed "globally unique" tag to "global ID" and hyperlinked it to <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs> (which now includes detailed examples of the various types of global IDs).
mappings/VegCore.htm: Regenerated from wiki. Moved Forming IDs section to separate Global IDs wiki page (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs).
mappings/VegCore.htm: Regenerated from wiki. Moved preferred ID order to Forming IDs section as examples list. Made examples globally unique by prefixing the outer ID that scopes the value.
mappings/VegCore.htm: Regenerated from wiki. Forming IDs: Added instructions for making a local ID global. Added note that these globally unique IDs are not the same as UUIDs. Forming preferred IDs: Reworded as a set of instructions rather than a definition. Clarified that this becomes the database primary key.
mappings/VegCore.htm: Regenerated from wiki. Added section on forming preferred IDs, which are the ID which is most directly associated with the identified object.
mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
mappings/VegCore.htm: Regenerated from wiki. verbatimScientificName, verbatimTaxonName have been merged into concatenatedScientificName, and ID has been disambiguated.
mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition
mappings/VegCore.htm: Regenerated from wiki. matched*Fit_fraction has been renamed to matched*Confidence_fraction.
mappings/VegCore.htm: Regenerated from wiki. Data owner terms and taxon synonyms have been added, and morphospecies has been disambiguated.
mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.
mappings/Makefile: VegCore.vocab.csv: Ignore leading ? when sorting so that ambiguous terms sort alphabetically with other terms. This prevents terms from moving from their previous location when they become ambiguous.
mappings/VegCore.vocab.csv: Changed line endings to \r\n in preparation for having a Python script run on it (which changes the line endings)
mappings/Makefile: VegCore.vocab.csv: Added back ambiguous terms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are ambiguous or unambiguous terms
mappings/Makefile: VegCore.vocab.csv: Added back synonyms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are synonyms or primary terms. This also prevents VegCore.vocab.csv from losing entries when terms are renamed, which made it difficult to verify that no terms were lost when refactoring.
mappings/VegCore.htm: Regenerated from wiki. The original*, accepted*, and verbatim* Taxon fields have now been moved to separate OriginalTaxonDetermination, AcceptedTaxonDetermination, and TaxonVerbatim tables.
mappings/VegCore.*.csv: Regenerated from wiki. A plain text label is now used for Replace with, which fixes a bug where the PRIVATE permalink pointed to its Replace with in realLatitude instead of its definition.
mappings/VegCore.*.csv: Regenerated from wiki. Alternatives now contain the "Alternative" label as plain text rather than as an image title, thus avoiding an HTML anchor conflict with the definition and allowing ambiguous terms to be placed before their alternatives as well as after.
mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch
mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID
mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.
mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).
mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.
mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match
Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity
mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary
mappings/Makefile: Replaced occurrences of VegCore.csv with $(vocab) and Veg+-VegCore.csv with $(dict)
mappings/Makefile: Veg+-VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause Veg+-VegCore.csv to change. This makes it much easier to identify synonyms and ambiguous terms that were accidentally deleted during a data dictionary refactoring. (Note that these are no longer included in VegCore.csv, so this is required in addition to sorting VegCore.csv by name.)
mappings/Makefile: VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause VegCore.csv to change. This makes it much easier to identify terms that were accidentally deleted during a data dictionary refactoring.
mappings/VegCore.csv: Regenerated from wiki. This adds cf_aff.
