import_all: Moved `waitpid $jobs` into after_import()
schemas/vegbien.ERD.mwb: Fixed table sizes
schemas/vegbien.ERD.mwb: Regenerated exports
schemas/vegbien.sql: removed all accessioncode fields, as VegBIEN does not use them
Added inputs/FIA/_src/FIADB_version4.accdb and FIADB_version4.sql (created from it using Access To PostgreSQL and the additional transformations at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Tools#MS-Access-database-MDB>)
Added inputs/FIA/COND_unique/, generated from new FIA data
inputs/FIA/FIA_COND_unique/create.sql: Fixed bug where need to remove `CREATE TABLE :table AS` at beginning because that is added by the make target
inputs/FIA/geoscrub.~.clean_up.sql: Moved creation of FIA_COND_unique to FIA_COND_unique/create.sql
README.TXT: Full database import: Updated time until import_all returns control to the shell to account for the TNRS names now being imported concurrently with the inputs rather than before them
mappings/VegCore-VegBIEN.csv: Also include morphospecies in the accepted taxondetermination's taxonverbatim, so that it can easily be retrieved by the analytical DB views
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use the taxonName or scientificName when the name components are not provided, as is the case when there is no scrubbed taxondetermination (because TNRS returns no match)
mappings/VegCore.csv: Regenerated from wiki. This adds Brad's DwC ID terms and their definitions in <https://projects.nceas.ucsb.edu/nceas/attachments/download/621/vegbien_identifier_examples.xlsx>.
join: Added support for direct mappings to VegBIEN by passing through outputs that start with / (indicating an XPath rather than a term)
mappings/VegCore.csv: Regenerated from wiki
schemas/vegbien.sql: analytical_stem_view: Added family_matched, taxonName_matched, scientificNameAuthorship_matched
schemas/vegbien.sql: analytical_stem_view: Added family_verbatim, scientificName_verbatim, scientificNameAuthorship_verbatim from datasource taxondetermination
schemas/vegbien.sql: analytical_stem_view: Fixed bug where need to use identifiedBy and dateIdentified from the datasource taxondetermination rather than the canonical taxondetermination (whichever taxondetermination is most scrubbed)
schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent(): is_datasource_current: Fixed bug where need to filter out determinationtypes for matched/accepted determinations, which are not datasource determinations
schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent(): Fixed bug where need to also set existing datasource_current taxondetermination's is_datasource_current to false
xml_dom.py: replace_with_text(): Added support for all scalar (non-Node) types, which will be stringified using strings.ustr()
schemas/functions.sql: Added _fix_date()
sql_io.py: put_table(): Documented that much of the complexity of the normalizing algorithm is due to PostgreSQL not having a native command for insert/on duplicate select
sql_io.py: put_table(): Corrected "insert/if not exists get" to "insert/on duplicate select"
sql_io.py: put_table(): Removed no longer applicable requirement that it be run at the beginning of a transaction, which was only required when the output table was locked during the function call
sql_io.py: put_table(): Documented that the function's insert/if not exists get algorithm does not support database triggers that populate fields covered by a unique constraint
inputs/FIA/_src/_README.TXT: Documented that FIA does not provide data for some states, e.g. HI
config/: Set svn:ignore to exclude *password files
Removing config/bien_read_password from version control
Removing config/bien_password from version control
inputs/FIA/: Added refreshed data (not yet mapped)
input.Makefile: Existing maps discovery: $(exts): Also match uppercase versions of extensions
lib/common.Makefile: Added $(ucase) and $(ci)
inputs/FIA/_src/Makefile: Table bundling: $(tableCsvs): Fixed bug where need to replace % with $* in $(csvPattern)
inputs/FIA/_src/Makefile: Table bundling: Fixed bug where need to remove trailing slashes from dirs that will match a target pattern
inputs/FIA/_src/Makefile: Added Table bundling targets to regroup CSVs by tables
lib/common.Makefile: Added $(mkdir)
Added inputs/FIA/_src/_README.TXT with Bob's comments
input.Makefile: SVN: $(_svnFilesGlob): Added README.TXT
mappings/VegCore.csv: Regenerated from wiki. Synonym lists have now been translated to sections to create a web page anchor for each synonym, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore_refactoring#Index-synonyms-as-web-page-anchors>. This enables searching for VegCore synonyms in the data dictionary as well as terms, and makes it possible to swap a term and a synonym while still keeping both as indexed anchors.
mappings/VegCore.csv: Regenerated from wiki. All uncategorized terms have now been moved to tables.
README.TXT: Maintenance: VegCore data dictionary: Added steps to check that no terms were lost when moving terms
inputs/import.stats.xls: Updated import times
schemas/vegbien.sql: analytical_stem_view: coordinates: Only use county_centroids coordinates when datasource coordinates are not provided, not also when datasource coordinates aren't geovalid. This also fixes a bug where (NULL) county_centroids coordinates were used for non-geovalid coordinates even when there was no county_centroids match, rather than including the non-geovalid coordinates.
schemas/vegbien.sql: taxondetermination: Added is_datasource_current, which is autopopulated to the most recent datasource-provided taxondetermination
schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_accepted_determination unique index to facilitate joining on the accepted determination
schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_matched_determination unique index to facilitate joining on the matched determination
schemas/vegbien.sql: taxondetermination: Removed notespublic, notesmgt, which are not used by VegBIEN
schemas/vegbien.sql: taxon_trait_view: scientificName: Use taxonverbatim.taxonname when taxonlabel/taxonverbatim.taxonomicname are not provided, to accommodate TNRS names. This is part of the workaround for the bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated.
schemas/vegbien.sql: taxon_trait_view: Added workaround for bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated due to a taxonlabel constraint violation, by using the taxonverbatim's taxonomicname instead in these cases. This bug, which appeared in the r7317 import, is so far not reproducible (tested on Mac OS X), so its cause is unknown, but may be caused by a bug in functions._merge_prefix(), which is run on the taxonlabel's taxonomicname but not the taxonverbatim's taxonomicname.
schemas/vegbien.sql: analytical_stem_view: Added dateIdentified, identificationRemarks per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#E-mail-on-2013-1-16)
inputs/FIA/_src/Makefile: Added extraction targets to extract zip archives
inputs/FIA/_src/download: Use new Makefile, which uses make logic to determine if a file needs to be downloaded
Added inputs/FIA/_src/Makefile, with targets to download each zip archive
schemas/vegbien.sql: analytical_stem_view: derived terms: Added _bien suffix per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#Brad-Boyles-comments)
Added inputs/FIA/_src/FIADB_version4.accdb.url
inputs/FIA/_src/download: Only run wget on files that don't yet exist
inputs/FIA/_src/download: Run wget in same directory as script to ensure files get downloaded there
inputs/FIA/_src/download: Set svn:executable
Added inputs/FIA/_src/download to download archives of CSVs for each state
to_do/timeline.2013.xls: Updated with changes during conference call
schemas/vegbien.sql: taxon_trait_view: Renamed datasource_taxonverbatim to taxonverbatim because there is now only one taxonverbatim
schemas/vegbien.sql: taxon_trait_view: Moved the taxondetermination.iscurrent filter to the join condition to allow using the taxondetermination_single_current_determination index
schemas/vegbien.sql: taxon_trait_view: Join only on the primary taxonlabel, not the accepted taxonlabel, because the scrubbed name is now available directly via the taxonlabel attached to the scrubbed taxondetermination
schemas/vegbien.sql: analytical_stem_view: Added locality
inputs/UNCC/Specimen/map.csv: accession: Remapped to catalogNumber per Bob's corrections
README.TXT: Schema changes: Added instructions to run the appropriate sync function when changing the analytical views
schemas/vegbien.sql: analytical_stem_view: Added georeferenceProtocol, which is set to 'county centroid' when county centroid coordinates are used
make_analytical_db: Don't run export_analytical_db if the SQL script exits with an error
README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added `export version=<version>` because import_times may be run in a shell different from the one that the import was run in
schemas/vegbien.sql: taxonverbatim: taxonverbatim_unique: Added morphoname for cases when there is just a morphoname, and to distinguish taxonverbatims with the same taxonlabel but different morphonames
schemas/vegbien.sql: stemobservation: Added stemobservation_non_empty CHECK constraint to prevent creating an empty stemobservation for plantobservation rows without stem data but with stem mappings
schemas/vegbien.sql: taxonverbatim: Allow taxonlabel_id to be NULL when morphoname is provided
schemas/vegbien.sql: taxonverbatim: Added source_id to allow creating taxonverbatims without a (scoping) taxonlabel
schemas/vegbien.sql: analytical_stem_view: Removed speciesBinomialWithMorphospecies now that it's duplicated by scientificNameWithMorphospecies
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Create it using the speciesBinomialWithMorphospecies formula, per Brad's request at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#2013-1-18>
schemas/vegbien.sql: analytical_stem_view: Added coordinateSource to indicate whether coordinates are from county_centroids (georeferencing) or the source data
schemas/vegbien.sql: Added coordinatesource enum
schemas/vegbien.sql: analytical_stem_view: coordinates: Also use the county_centroids coordinates when the datasource coordinates are not geovalid. (Note that canon_place.geovalid will be NULL, i.e. not true, when the datasource coordinates are NULL.)
schemas/vegbien.sql: scientificName: Set to taxonverbatim.taxonname instead per Brad's changes at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#2013-1-18>. Renamed to taxonName since this now doesn't include the author, which is part of DwC's scientificName field.
schemas/vegbien.sql: sync_analytical_stem_to_view(): Support running the function when dependent views do not exist. This allows using the sync function when changing column names of the analytical_stem_view, which sometimes requires manually dropping and re-creating the analytical_aggregate_view.
backups/Makefile: %.md5/test: Added comment to run with `make -s` to avoid echoing make commands
README.TXT: Full database import: Added steps to scrub unscrubbed taxondeterminations (if they are not scrubbed automatically)
inputs/.geoscrub/_src/README.TXT: Added e-mails from Jim about how the county_centroids data was generated
schemas/vegbien.sql: analytical_stem_view: coordinates: Use new county_centroids coordinates and uncertainty when the datasource's coordinates are not available
Added inputs/.geoscrub/county_centroids/ from Jim
inputs/.geoscrub/import_order.txt: Added geoscrub_output
README.TXT: Full database import: In PostgreSQL: Added step to check that there are TNRS taxondeterminations
