moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).
inputs/VegBank/taxonobservation_/map.csv, postprocess.sql: mapped identifiedBy (the join_words() of identifiedBy_first, etc.)
inputs/VegBank/taxonobservation_/create.sql: also join party_id to get the identifiedBy (not mapped yet). note that the inserted row count changes, because taxonobservation_ does not yet have a pkey to do a stable ordering with.
inputs/VegBank/vegbank.~.clean_up.sql: taxoninterpretation.party_id: don't rename to taxoninterpretation_party_id, so that this can be used directly in taxonobservation_/create.sql with a USING join
inputs/VegBank/taxonobservation_/create.sql: join taxonobservation to taxoninterpretation (as in CVS) instead of vice versa, since taxonobservation is the primary, operative table. having VegBank and CVS do things the same way helps ensure that fixes in one can transfer easily to the other.
inputs/VegBank/taxonobservation_/map.csv: originalinterpretation, currentinterpretation: removed table name prefix so these would automap
fix: inputs/VegBank/taxonobservation_/map.csv: remapped authorplantname to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
fix: inputs/VegBank/taxonobservation_/map.csv: remapped int_* to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
inputs/VegBank/plantconcept_/: mapped columns, since this is now included in import_order.txt and therefore gets processed by the column-renaming runscripts. note that this means that in taxonobservation_/map.csv, the plantconcept_ input column names need to be changed to what they are mapped to.
bugfix: inputs/VegBank/: taxonOccurrenceID: include the aggregateOrganismObservationID in this so that there is one taxonoccurrence for each stratum's taxonImportance. this allows the different strata to have separate taxonoccurrences that are associated with the stratum-specific locationevents, rather than all being lumped into one taxonoccurrence, with only one locationevent.
inputs/VegBank/taxonobservation_/map.csv: taxonomic ranks not in VegCore: removed table prefix so they will be automapped (they are globally unique)
inputs/VegBank/taxonobservation_/create.sql: join starting with taxoninterpretation so that we can use the taxoninterpretation_id as the row_num (text strings, formed from concatenated #s cannot be used as a row_num). there is only 1 taxonobservation without a taxoninterpretation, so we can just include one row for each taxoninterpretation.
inputs/VegBank/taxonobservation_/postprocess.sql: added primary key. note that the inserted row count changes, most likely because the rows are now in sorted order.
inputs/VegBank/: prepended the table name to each column name to prevent column collisions, using the steps at http://wiki.vegpath.org/Left-joining_a_datasource
inputs/VegBank/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
bugfix: inputs/VegBank/taxonobservation_/map.csv: put columns in table order, which is needed by new-style import
inputs/VegBank/taxonobservation_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/VegBank/taxonobservation_/map.csv: plantname: remapped to DUPLICATE#of:plantconcept_plantname because this is an exact duplicate
bugfix: inputs/VegBank/taxonobservation_/map.csv: updated input column names for renamings in inputs/VegBank/vegbank.~.clean_up.sql
inputs/VegBank/taxonobservation_/map.csv: Species and lower ranks: remapped to EQUIV#to:plantname because these contain the taxonomic name at specific ranks, but plantname contains the taxonomic name of the plant itself, which is longer and populated more often
inputs/VegBank/taxonobservation_/create.sql: also join to plantname, since plantconcept.plantname may not always be populated when plantname.plantname is
fix: inputs/VegBank/taxonobservation_/map.csv: Species and below: remapped to _alts of scientificName, because these are actually the full taxonomic name at that rank, not just the epithet. Genus: documented that it includes the genus author.
fix: inputs/VegBank/taxonobservation_/map.csv: also mapped plantname to scientificName, since int_currplantscifull is not always provided when this is. (it cannot replace int_currplantscifull, because when int_currplantscifull also provided, this often leaves out lower ranks.) this should fill in taxonomic information for taxonobservations that are currently missing it.
inputs/VegBank/taxonobservation_/map.csv: collector_id: remapped to UNUSED. removed LEFT JOIN collector_id->party since this field is never populated.
inputs/VegBank/taxonobservation_/map.csv: int_origplantscifull: remapped to EQUIV (to authorplantname). this is the scrubbed originalScientificName, but we do our own scrubbing.
fix: inputs/VegBank/taxonobservation_/map.csv: authorplantname: remapped to originalScientificName because it includes the name author
inputs/VegBank/taxonobservation_/map.csv: mapped int_origplant*, int_currplant* to *scientificName/*taxonName/etc.
inputs/VegBank/taxonobservation_/map.csv: Mapped int_currplantcommon to vernacularName
inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)
mappings/VegCore.htm: Regenerated from wiki. Remapped organismNotes to be a synonym of occurrenceRemarks, since notes on an organism are more generally notes on an occurrence.
mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
inputs/*/*/map.csv: Automapped taxonConfidence
mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.
mappings/VegCore-VegBIEN.csv, inputs/*/*/map.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv, which reflects the current state of the data dictionary. (Permanently switching to the new Veg+-VegCore.csv will be a separate change.) Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping and inferring units on several fields.
mappings/VegCore.csv: Regenerated from wiki
mappings/VegCore.csv: Term names: Changed special characters to _ because Redmine doesn't support special characters in HTML anchors (it removes everything except letters, numbers, _, and -)
mappings/VegCore.csv: Renamed sampleType to observationType to match the SALVIAS term it's derived from
inputs/VegBank/taxonobservation_/map.csv: Fixed bug where need to indicate that data is plots data to prevent the specimenreplicate ID from being forwarded to the location ID
inputs/VegBank/taxonobservation_/map.csv: Mapped new givenname, surname (from collector_id's party) to recordedBy
inputs/VegBank/taxonobservation_/create.sql: Also join to collector_id's party to include collector name
mappings/VegCore.csv: Renamed *Binomial to *TaxonName because this field can store more ranks than just the genus+specificEpithet binomial (that goes in speciesBinomial)
mappings: Renamed *scientificName to *binomial because DwC defines the scientificName as "The full scientific name, with authorship and date information if known", but many datasources do not include the author in their scientific name, and the fields scientificName is mapped to in VegBIEN assume it does not include the author
mappings/VegCore.csv: Renamed verbatim* taxonomic terms to original* because in most datasources, they are in fact for the original taxon determination of the organism (which can be a completely different name than the primary determination), rather than merely unscrubbed versions of the primary taxonomic name elements. Note that SALVIAS's orig_* terms do appear to be merely unscrubbed versions, but it's not a problem to add an additional taxon determination for them.
input.Makefile: Staging tables installation: `%/install: %/create.sql`: Don't add a row number column to the created table because it is now added automatically to the temp table by column-based import (row-based import now also does not require a pkey for DB inputs)
inputs/*/*/map.csv: Prefix a * to every term that's not in Veg+ for easy identification of unmapped terms when editing map.csv. Note that canon will remove the * when it finds a matching Veg+ term.
inputs/*/*/map.csv: Mapped variants of subspecies directly to new subspecies term
mappings/VegCore-VegBIEN.csv: Mapped subspecies, variety, forma, cultivar
inputs/*/*/map.csv: Changed output column header from Veg+ to VegCore because the names will be VegCore names after automapping. This is possible now that we're using new automapping scripts that do not require a particular column header.
inputs/*/*/map.csv: Added Filter column to contain any suffix added after the term, so that the automapping mechanism does not have to deal with the filter expressions
inputs/*/*/map.csv: Removed no longer needed [Veg+] suffix in root, because the input column is no longer used by old-style map utilities such as union that needed this
inputs/*/*/map.csv: Changed empty mappings to self mappings, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Map_refactoring#Change-empty-mappings-to-self-mappings>. Note that in map.full.csv and VegBIEN.csv, lines that have changed are always the result of the input field's case being changed to match the case of the datasource's actual column name.
inputs/*/*/map.csv: Added back automapped mappings to map.csv, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Map_refactoring#Add-back-automapped-mappings-to-mapcsv>
inputs/VegBank/taxonobservation_/map.csv: Updated with new renamings of colliding join columns
input.Makefile: Maps building: %/.map.csv.last_cleanup: Translate map.csv using $(mappings)/$(via)-VegCore.csv
inputs/VegBank/taxonobservation_/map.csv: Mapped observation_id to eventID
inputs/VegBank/: Added taxonobservation_/
