units.py: convert(): Support ranges
Moved parse_range() from xml_func.py to units.py
units.py: conversions: Added ft->m conversion
src_map: Reopen stdin (/dev/stdin) in universal newlines mode to support files with \r as the line ending
csv2db: Open input stream in universal newlines mode to support files with \r as the line ending
mappings/VegCore.htm: Regenerated from wiki. Moved Forming IDs section to separate Global IDs wiki page (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs).
input.Makefile: $(exts): Added .tab
mappings/VegCore.htm: Regenerated from wiki. Moved preferred ID order to Forming IDs section as examples list. Made examples globally unique by prefixing the outer ID that scopes the value.
mappings/VegCore.htm: Regenerated from wiki. Forming IDs: Added instructions for making a local ID global. Added note that these globally unique IDs are not the same as UUIDs. Forming preferred IDs: Reworded as a set of instructions rather than a definition. Clarified that this becomes the database primary key.
mappings/VegCore.htm: Regenerated from wiki. Added section on forming preferred IDs, which are the ID which is most directly associated with the identified object.
mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
inputs/NY/Ecatalog_all/test.xml.ref: Updated inserted row count
schemas/vegbien.sql: analytical_stem_view: scientificName_verbatim: Use the taxonname (which omits the family and author) instead when available, at Brad's request
schemas/vegbien.sql: Disambiguated taxonlabel.taxonomicname, taxonverbatim.taxonomicname, and taxonverbatim.taxonname
schemas/vegbien.sql: taxon_trait_view: Use taxonverbatim.taxonname (the binomial) in preference to taxonverbatim.taxonomicname (the binomial+author) or taxonlabel.taxonomicname (the family+binomial+author), at Mark's request
to_do/timeline.2013.xls: Obtain and load any additional new data: Extended beyond April because it's ongoing
README.TXT: Full database import: On nimoy: Moved analytical_aggregate table creation right after analytical_aggregate..csv download because it can happen while the file is downloaded
to_do/timeline.2013.xls: Reload existing data in need of refresh: Added additional cell because this is still in progress
to_do/timeline.2013.xls: Placed check mark in completed cells. Multi-cell tasks that are partially complete have check marks in part of their cells to indicate this.
Added inputs/FIA/REF_*/
inputs/FIA/_src/Makefile: Table bundling: Also unpack FIADB_REFERENCE/
Added inputs/FIA/_src/FIADB_REFERENCE.ZIP.md5, .url
schemas/vegbien.sql: provider_count_view: Group data by dataprovider and dataset, rather than by each separately, to satisfy the GBIF attribution requirements as described by Brad
schemas/vegbien.sql: analytical_plot/specimen: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
schemas/vegbien.sql: analytical_stem_view: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
schemas/vegbien.sql: sync_analytical_stem_to_view(): Also drop and re-create new dependent views analytical_plot, analytical_specimen
inputs/TEAM/V*/map.csv: Mapped plotArea_ha based on "1ha Plot Number" field name
inputs/TEAM/V*/map.csv: Id: Remapped to individualObservationID
mappings/VegCore.htm: Regenerated from wiki. verbatimScientificName, verbatimTaxonName have been merged into concatenatedScientificName, and ID has been disambiguated.
inputs/*/*/map.csv: Automapped taxonConfidence
schemas/vegbien.sql: analytical_stem_view: plotArea_ha: Fixed bug where need to use parent plot's area when subplot's area is not provided
inputs/SALVIAS/plotMetadata_/create.sql: Added primary key
inputs/NY/NY_DATA_20130212.~.utils.sql: pkey: Made table an enum instead of a text field so that Ecatalog_LatLong could be sorted before Ecatalog
inputs/NY/Ecatalog_all/create.sql: primary key: Combine table, key into one value so that the primary key has a single column. This is necessary because multi-column primary keys are not supported by the import algorithm.
inputs/NY/NY_DATA_20130212.~.utils.sql: Added pkey record type for use by Ecatalog_all's generated pkey
inputs/NY/Ecatalog_all/create.sql: Add table column which scopes the key field. Add primary key to allow sorting by it. inputs/NY/Ecatalog_all/map.csv: Documented that UniqueNYInternalRecordNumber is sometimes duplicated.
make_analytical_db: Wait for tnrs.make to acquire the tnrs.make lock if being run right after import
schemas/vegbien.sql: analytical_stem_view: taxonName_matched: Fixed bug where need to use the matched_taxonverbatim instead of the datasource_taxonverbatim
Moved wait on tnrs.make lock from import_all to make_analytical_db, so that running make_analytical_db for a one-time import also waits on the lock
schemas/vegbien.sql: taxondetermination: taxondetermination_unique: Added determinationtype so that when the matched and accepted determinations are the same, they still both get created rather than the second one being removed due to the unique constraint
schemas/vegbien.sql: analytical_specimen: Removed speciesBinomialWithMorphospecies because it doesn't apply to specimens
schemas/vegbien.sql: Added analytical_plot view
schemas/vegbien.sql: Added analytical_specimen view
schemas/vegbien.sql: analytical_stem_view: Moved recordedBy, recordNumber before dateCollected as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
schemas/vegbien.ERD.mwb: Synced with schema
schemas/vegbien.sql: Added reproductiveCondition
mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition
schemas/vegbien.sql: plantobservation: Added reproductivecondition
mappings/VegCore.htm: Regenerated from wiki. matched*Fit_fraction has been renamed to matched*Confidence_fraction.
inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Updated for new mappings/VegCore.htm
inputs/bien_web/observation/map.csv: Re-automapped taxonMorphospecies
mappings/VegCore.htm: Regenerated from wiki. Data owner terms and taxon synonyms have been added, and morphospecies has been disambiguated.
schemas/vegbien.sql: analytical_stem_view: Moved identifiedBy, dateIdentified, identificationRemarks right after the *_verbatim terms that they relate to, as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
schemas/vegbien.sql: analytical_stem_view: Use new concat_delim() instead of array_to_string() surrounded by NULLIF
schemas/vegbien.sql: Added concat_delim()
schemas/vegbien.sql: analytical_stem_view: Removed deprecated taxonNameWithMorphospecies now that we have speciesBinomialWithMorphospecies
schemas/vegbien.sql: analytical_stem_view: speciesBinomial: Added morphospecies suffix to create speciesBinomialWithMorphospecies
inputs/import.stats.xls: Updated import times
README.TXT: Full database import: Check that unscrubbed_taxondetermination_view returns no rows: Documented that this takes 90 s with LIMIT 1
schemas/vegbien.sql: _taxon_family_require_std(): Also allow non-aceae families accepted by TNRS
Added inputs/SALVIAS/_archive/salvias_plots.*.sql.zip.md5
Added inputs/VegBank/_archive/vegbank_for_bien.tar.gz.url
Added inputs/U/UtrechtHerbarium.csv.tar.gz.url
Added inputs/TEAM/_archive/ci-team_extract.tar.gz.url
Added inputs/SpeciesLink/_archive/specieslink*.txt.gz.url
Added inputs/REMIB/_archive/remib_raw.csv.tar.gz.url
Added inputs/NY/NYSpecimenDataAmericas.csv.tar.gz.url
Added inputs/NCU/_archive/NCU-NCSC_2010-02-12.csv.tar.gz.url
Added inputs/MO/mo_digirexport.tar.gz.url
Added inputs/Madidi/_archive/2010-1-2/madidi_plots_original_12jan2010.zip.url
Added inputs/GBIF/gbif_extract.tar.gz.url
Added inputs/FIA/fia_extract.tar.gz.url
Added inputs/CVS/_archive/CVS-allTaxonOccurrences_2010-01-12.txt.tar.gz.url
Added inputs/ARIZ/ARIZ_DiGIR_21012010.csv.tar.gz.url
Added inputs/UNCC/Specimen/UNCC.csv.url, UNCC.csv.md5
Added inputs/XAL/_src/digir.xml.gz.md5
Added inputs/UNCC/_src/ with UNCC.csv.zip.md5
Added inputs/SpeciesLink/_src
README.TXT: Datasource setup: MySQL inputs: .sql exports: Use new mysql_bien to connect to the MySQL DB created for the datasource
Added mysql_bien, which runs a MySQL command on the local MySQL server
Added inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump.md5 (md5sum of the expanded file)
root Makefile: MySQL: mysql-Linux: Also install phpMyAdmin
root Makefile: MySQL: mysql-Linux: Split apt-get dependencies into separate commands, like for other apt-get commands, to avoid having one failed dependency prevent the following dependencies from being installed
root Makefile: MySQL: *mysql_users: Also add bien_read user
root Makefile: MySQL: Renamed *mysql_user to *mysql_users because there can be multiple users
inputs/: Added .md5 files for all .zip, .gz
Added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.url
lib/common.Makefile: rsync: $(rsync*): Use --no-group because the file group is different depending on the machine
input.Makefile: SVN: $(_svnFilesGlob): Also add .md5 files. This allows svn to track where unversioned files should be in the directory tree.
input.Makefile: SVN: $(_svnFilesGlob): .url, .pdf, and README.TXT in the top-level dir: Fixed bug where had extra / after brace expr
input.Makefile: SVN: $(_svnFilesGlob): Also add .url, .pdf, and README.TXT in the top-level dir
input.Makefile: SVN: $(_svnFilesGlob): Add .url, .pdf, and README.TXT files in all subdirs, not just _src
lib/common.Makefile: remote server: Use jupiter instead of vegbiendev, to ensure that all files get uploaded there rather than only to vegbiendev. This involves adding an extra database import step to download the uploaded files from jupiter onto vegbiendev.
inputs/FIA/_src/Makefile: all: Extract zip files before running tables target, because it requires the created dirs
schemas/vegbien.ERD.mwb: Fixed table sizes
Removed no longer used fix_permissions. Use root fix_perms instead.
Added root fix_perms
Moved Checksums from backups/Makefile to lib/common.Makefile so all dirs (including inputs/) can use md5sum testing
lib/common.Makefile: $(remote): Made remote basepath configurable in $(remote_basepath)
lib/common.Makefile: Renamed $(src_server) to $(remote_host) and $(src_user) to $(remote_user) for clarity
