mappings/VegCore-VegBIEN.csv: infraspecific taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts, or in the lowest-level taxonconcept as the taxonname and rank
mappings/VegCore-VegBIEN.csv: higher-level taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts
schemas/vegbien.sql: taxonconcept: Merged taxonconcept_unique_within_creator_by_name unique index into taxonconcept_unique_within_parent, placed parent_id first, and removed index condition, so that this index can be used as a lookup index by taxonconcept_update_ancestors() (which requires no index condition in order to apply to all taxonconcepts) in addition to as a unique index. Note that an index condition should not be necessary for the index's uniquifying task, because if a set of taxonconcepts provides only the identifyingtaxonomicname, that should collide in the taxonconcept_unique_within_creator_by_identifying_name unique index before this index collides. This assumes that the collision order when multiple indexes collide is alphabetical by the index name.
schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Also allow a taxonconcept to have just an author when it has a parent_id, so that an author can uniquely identify a taxon within a more general taxon, such as a species name, that has no author
strings.py: concat(): Fixed bug where end index of returned str0 portion would wrap around to a negative number if str1 itself was too long, causing incorrect truncation
schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_parent to taxonconcept_unique because the index does not apply only to taxonconcepts with a parent, and because it's the primary unique index for taxonconcept
schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_creator_by_identifying_name to taxonconcept_0_unique_identifying_name to ensure that it is always applied before taxonconcept_unique_within_parent if both collide
mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree for taxonconcepts created from original*, verbatim*, and accepted* taxonomic terms
mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree if datasource provided separated components of the taxonomic name and/or its own tree of life with higher classifications. This enables storing the datasource's own tree of life to supplement any official tree (TROPICOS, USDA, etc.).
mappings/VegCore-VegBIEN.csv: taxonconcept tree: Don't map infraspecificEpithet+taxonRank to a taxonconcept in the tree of parent concepts because it has already been mapped to the primary, lowest-level taxonconcept
schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Fixed bug where index filter overlapped with taxonconcept_unique_within_parent's index filter, causing these unique indexes to sometimes both apply at the same time and prevent column-based import from correctly choosing which index to use for each taxonconcept import
schemas/vegbien.ERD.mwb: Fixed lines
schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id comment: Changed comment to use "concept" rather than "name" where applicable. Documented that a synonym between taxonconcepts of different sources is indicated by choosing one taxonconcept to be authoritative and pointing the other taxonconcept to it using this field.
sql_io.py: put_table(): Resolving default value column: Fixed bug where the default value col needed to have its table changed from in_table to full_in_table if it's a table column, and needed to have any column rename removed if it's a literal value
Regenerated vegbien.ERD exports
schemas/vegbien.sql: Renamed plant* taxonomic tables -> taxon*, as part of the taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
schemas/vegbien.ERD.mwb: Rearranged to fit more of location table on the diagram, using the newly available space from taxon
schemas/tree_cross-links.sql: Synced with schema, updating with new table names
schemas/vegbien.sql: Removed no longer used taxon table. Use taxonconcept instead.
schemas/vegbien.sql: taxonconcept.taxonname: comment: Stated that this is the name of the taxon within its parent taxon
schemas/vegbien.sql: taxonconcept: comment: Removed no longer accurate comment that an accepted taxonconcept points to the identified taxon in the tree of life, because it is the identified taxon in the tree of life
schemas/filter_ERD.csv: Changed the table with the visible fkey from plant* to taxon* to be plantstatus rather than plantusage, since it contains more core fields
schemas/vegbien.sql: taxonconcept: Removed taxon_id, since taxonconcept now contains all the information needed to represent a taxonomic hierarchy, including both conceptual and nomenclature information
schemas/vegbien.sql: plantusage: Point just to taxonconcept instead of both to taxonconcept and taxon
schemas/vegbien.sql: taxonconcept: rank, verbatimrank comments: Added info from corresponding fields in taxon that also applies to taxonconcept
schemas/vegbien.sql: taxonconcept: comment: Added info from taxon that also applies to taxonconcept
schemas/vegbien.sql: Added taxonconcept_ancestor cross-link table
schemas/vegbien.sql: taxonconcept: Added description field
mappings/VegCore-VegBIEN.csv: Remapped taxon hierarchy for accepted taxonconcepts to taxonconcept parent_id hierarchy
schemas/vegbien.sql: Fixed bug where taxonconcept.parent_id was missing a foreign key constraint
schemas/vegbien.sql: taxonconcept: Changed instructions for including a taxon name at a rank with no explicit column to create a parent taxonconcept for it and point to it using parent_id instead of using otherranks. Removed no longer used otherranks field.
schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Added taxonname
schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Removed duplicate entry for creator_id
schemas/vegbien.sql: taxonconcept: Added parent_id to point to the parent taxonconcept
sql_gen.py: null_sentinels: Added 'unknown' for taxonrank
schemas/vegbien.sql: taxonrank: Added 'unknown'
mappings/VegCore-VegBIEN.csv: Also map *taxonRank to taxonconcept.rank, so that if it's in the taxonrank enum, it will automatically populate this field
mappings/VegCore-VegBIEN.csv: Remapped *infraspecificEpithet to new taxonconcept.taxonname rather than placing it in subspecies prefixed with the taxonRank, because it isn't necessarily the subspecies and because taxonname is defined to contain the lowest-rank portion of the taxonomic name. Note that when both morphospecies and infraspecificEpithet are provided, infraspecificEpithet takes priority for the taxonname field, because if TNRS leaves unmatched terms (which are tentatively mapped to morphospecies) but also matches an infraspecificEpithet, then the unmatched terms can't be for a morphospecies (because an infraspecificEpithet and therefore also a specificEpithet was matched, so the species is definite and formally named).
schemas/vegbien.sql: taxonconcept: Renamed morphospecies to taxonname since it's used in the same way as taxon.taxonname: to store the lowest-rank portion of the taxonomic name, such as the morphospecies suffix
inputs/.TNRS/tnrs/map.csv: Mapped *_matched terms that are both matched in the input name and which correspond to the matched taxonconcept (Genus_matched, Specific_epithet_matched, etc.) to both the input and matched taxonconcepts
inputs/.TNRS/tnrs/map.csv: Mapped terms matched in the original string (rather than deduced from the matched taxonconcept) to new verbatim* taxonomic terms
mappings/VegCore-VegBIEN.csv: Mapped verbatim* taxonomic terms to the TNRS input taxonconcept
mappings/VegCore-VegBIEN.csv: TNRS input taxonconcept: Split single _if statement controlling where morphospecies goes into two _if statements for each case, so that other verbatim* terms don't need to have an _if statement in their mapping to the input taxonconcept
mappings/VegCore.csv: Added back verbatim* taxonomic terms, which will now be used for the TNRS input taxonconcept. Note that they will have a different meaning than the original* taxonomic terms that they were renamed to in r5062.
mappings/VegCore-VegBIEN.csv: In TNRS mode, remapped morphospecies (Unmatched_terms) to the input name's taxonconcept, because this does not relate to the matched taxon concept
mappings/VegCore-VegBIEN.csv: TNRS-only mappings: Switch them on when verbatimScientificNameWithAuthorship is provided rather than when acceptedScientificNameWithAuthorship is provided, because it's the presence of a separate TNRS input name that really determines when TNRS is being mapped
Makefiles: .last_cleanup targets: Also make the file that's being cleaned up .PRECIOUS so it doesn't get deleted if the .last_cleanup target has an error
Makefiles: .last_cleanup targets: Make each individual target .PRECIOUS (don't delete on error) because just making %.last_cleanup precious doesn't seem to prevent deletion
mappings/VegCore-VegBIEN.csv: Mapped *taxonRank to new taxonconcept.verbatimrank
schemas/vegbien.sql: taxonconcept: Added rank, verbatimrank analogous to those fields in taxon
Makefiles: Don't delete %.last_cleanup on error because it's a mod time record rather than a generated file, and so that it's left at the last successful cleanup time when a cleanup operation is cancelled
input.Makefile: Maps building: %/.map.csv.last_cleanup: Removed no longer accurate comment about mappings being autoremoved
inputs/.TNRS/tnrs/map.csv: Remapped Name_submitted to new verbatimScientificNameWithAuthorship to create an additional level of taxonconcept for the concatenated (TNRS input) name separate from the parsed (TNRS output) name
mappings/VegCore-VegBIEN.csv: Mapped verbatimScientificNameWithAuthorship as an additional level of taxonconcept for the concatenated (TNRS input) name separate from the parsed (TNRS output) name
schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id: comment: Changed three-level hierarchy to four-level hierarchy which separates the concatenated (TNRS input) name from the parsed (TNRS output) name
mappings/VegCore.csv: Added back verbatimScientificNameWithAuthorship, which will now be used to store the TNRS input name
schemas/filter_ERD.csv: Removed no longer used table taxonscope
schemas/vegbien.sql: voucher: Removed accessioncode because this table has no sourceaccessioncode which it would be generated from (it just links a taxonoccurrence to a vouchering specimenreplicate)
schemas/vegbien.sql: Renamed datasource_id to creator_id so it can apply generally to any entity (such as a person), not just an aggregated datasource. This also enables taxonconcept.datasource_id to merge with creator_id, which now serves the same purpose.
schemas/vegbien.sql: taxonconcept: Renamed definer_id to creator_id to allow merging with datasource_id when datasource_id is renamed to creator_id
mappings/VegCore-VegBIEN.csv: Populated new taxonconcept.definer_id from identifiedBy, or when no identifiedBy is specified, from the datasource itself (using _simplifyPath:[next=datasource_id])
sql_io.py: put_table(): Resolve default value column after the main loop (inserts and selects), so that the default value column can refer to an output column that is not in the original mapping but is added to the mapping from a col_defaults entry. This requires deferring the "Missing mapping for NOT NULL column" warning until the default value column is resolved, and including all columns in the full_in_table since the default value input column is not yet known.
schemas/vegbien.sql: taxonconcept: comment: Changed definition to "A taxon concept defined by an entity" to correspond with the table's new name and usage
mappings/VegCore-VegBIEN.csv: Fixed bug where needed to set datasource_id=0 on the TNRS party (which concatenated names/TNRS inputs are owned by) in order to make it a datasource (a root party)
schemas/vegbien.sql: party: Fixed bug where needed separate unique index for roots (datasources), whose organizationnames must be globally unique rather than unique within a datasource
schemas/vegbien.sql: taxonconcept: Renamed concept_reference_id to definer_id because this is a clearer name and because this will allow merging with datasource_id, which serves the same purpose
schemas/vegbien.sql: party: Made it datasource-scoped. Since this creates a recursive fkey, a datasource (a root party) should point to itself in this field, which will happen automatically by setting it to the special value 0.
lib/PostgreSQL-MySQL.csv: Changed translation of fulltext to quote the identifier instead of appending characters to make it not a reserved word
schemas/vegbien.sql: taxonconcept: Moved concept_reference_id to the top of the table because it is now a key scoping field
schemas/vegbien.sql: concept_reference_id: Made it an fkey to party instead of taxonscope, because this is now the entity that defined the taxon concept, and is no longer specific to morphospecies. Removed no longer used table taxonscope.
schemas/vegbien.sql: taxonconcept: Documented that it's equivalent to VegBank's plantConcept table
schemas/filter_ERD.csv: taxonconcept inward fkeys: Removed not applicable taxon filtered table, since the fkey points in the opposite direction and thus is not part of this filter
schemas/vegbien.sql: taxonconcept: Renamed scope_id -> concept_reference_id as part of taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
README.TXT: Schema changes: Moved "update the following files with any renamings" out of "Sync ERD with vegbien.sql schema" because this is needed for any schema changes, not just as part of syncing the ERD
README.TXT: Schema changes: Added Refactoring tips section with steps to rename a table and a column
schemas/vegbien.sql: Renamed taxonpath -> taxonconcept as part of taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
README.TXT: Schema changes: Syncing ERD with vegbien.sql schema: Added step to update mappings/VegCore-VegBIEN.csv with any renamings
README.TXT: Schema changes: Syncing ERD with vegbien.sql schema: Added step to update schemas/filter_ERD.csv with any table renamings
inputs/import.stats.xls: Updated import times. This now includes the half-hour-long pre-import of the TNRS taxonomic names (which the datasources then match up with), as well as the concatenation of the datasource's taxonomic name components to create or match up with the TNRS input name.
README.TXT: Data import: make backups/TNRS.backup/restore: Run it in the background because it takes awhile
README.TXT: Data import: Added steps to sync the TNRS schema to the latest version on vegbiendev
README.TXT: Data import: make inputs/download-logs: Added tnrs_log=1 so the TNRS daemon log is downloaded as well
Added inputs/test_taxonomic_names/Taxon/testNames.txt since this is test data, and thus can be under version control
Added inputs/test_taxonomic_names/README.TXT with Bob's comments
schemas/vegbien.sql: taxonpath.taxon_id: Changed comment to indicate that this used for parsed, not just accepted names. Parsed names have been standardized by TNRS but may be synonyms.
README.TXT: Documentation: To import and scrub just the test taxonomic names: Added `yes|` before make schemas/public/reinstall so the user isn't prompted to confirm the reinstallation a second time, and can just copy and paste the set of 5 commands directly into the terminal
tnrs_db: Made wait option default to off to facilitate running tnrs_db by itself, rather than as part of an import
tnrs_db: Added wait option to have tnrs_db exit as soon as no more names are available. This is useful for running tnrs_db when there is no concurrent import running, and therefore no need to wait for new data.
tnrs_db: Fixed the time of the "Waited" message so it that the total_pause (containing the next wait) would be incremented after the message was displayed. Split the "Waited" and "Waiting" messages into two separate messages.
README.TXT: Data import: Added steps to back up the TNRS cache, since it takes a long time to recreate. This also enables syncing it with a local machine when `make backups/download` is run.
README.TXT: Documentation: Added instructions to import and scrub just the test taxonomic names
input.Makefile: Staging tables installation: uninstall: For the TNRS datasource, prompt the user before deleting the schema, since the data in it is not easily reconstructible from a flat file
sql.py: map_expr(): When matching without quotes, support names containing spaces by not matching words when preceded or followed by quotes
sql.py: Expressions: bool_re: Also match parentheses surrounding the boolean value
README.TXT: Data import: import_all: Don't run with & because this prevents the created jobs from being owned by the calling shell. Instead, import the TNRS names as a separate backgrounded step and wait for it to finish before starting import_all. Removed TNRS import steps from import_all since these are now invoked separately.
README.TXT: Data import: Run import_all in the background, because it needs to import all the taxonomic names synchronously before it can start the datasource import in the background
