inputs/VegBank/vegbank.~.clean_up.sql: disambiguated plantconcept.plantname, plantname.reference_id to enable joining plantconcept_->plantname
fix: inputs/VegBank/taxonobservation_/map.csv: also mapped plantname to scientificName, since int_currplantscifull is not always provided when this is. (it cannot replace int_currplantscifull, because when int_currplantscifull also provided, this often leaves out lower ranks.) this should fill in taxonomic information for taxonobservations that are currently missing it.
bugfix: schemas/vegbien.sql: analytical_stem_view: coordinates: use the coordinates from datasource_place instead of canon_place, because canon_place's coordinates are only what the geoscrubbing output and do not contain datasource-specific information such as coordsaccuracy_m
inputs/VegBank/taxonobservation_/map.csv: collector_id: remapped to UNUSED. removed LEFT JOIN collector_id->party since this field is never populated.
inputs/VegBank/plot_/map.csv: area|country|territory, region|state|province (from place table): remapped to DUPLICATE, since these have the same data as, and are populated less often than, their country/stateprovince couterparts
bugfix: inputs/VegBank/plot_/create.sql: need to join place.*plot_id* to plot.plot_id instead of plotplace_id. this is the cause of the "State is wrong, not Wyoming, but Tennessee" and "County is incorrect (not Powell, but Orange)" bugs in the VegBank spot-checking (wiki.vegpath.org/Spot-checking#Great-Smoky-Mountains-National-Park).
inputs/VegBank/taxonobservation_/map.csv: int_origplantscifull: remapped to EQUIV (to authorplantname). this is the scrubbed originalScientificName, but we do our own scrubbing.
fix: inputs/VegBank/taxonobservation_/map.csv: authorplantname: remapped to originalScientificName because it includes the name author
inputs/VegBank/taxonobservation_/map.csv: mapped int_origplant*, int_currplant* to *scientificName/*taxonName/etc.
inputs/VegBank/plot_/map.csv: elevation: documented that it has only 5 decimal places of precision, with only 9s and random #s after that
inputs/VegBank/plot_/test.xml.ref: update rowcount
inputs/VegBank/stemlocation_/map.csv: stemcode, stemxposition, stemyposition: remapped to UNUSED. stemhealth is the only data field in this table that is populated, which means that VegBank does not provide data on reobservable stems even though the schema supports it.
inputs/VegBank/observation_/postprocess.sql: added pkey
schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
schemas/VegCore/ERD/index.htm: regenerated
schemas/VegCore/VegCore.ERD.mwb: taxon_observation: added observation_in_parent_place, which points to the observation of the same taxon/individual in the parent place. this accounts for VegBank allowing multiple taxonImportances per taxonObservation, which only makes sense when each taxonImportance is from a different stratum and they point to a common taxonObservation for the parent plot.
bugfix: web/.htaccess: auto-detect dotpath in query string: need to include the path ($0) in the replacement, to avoid reverting to the root dir. (mod_rewrite replacements are not like relative URLs, which would interpret ?... as being relative to the current path, not the root.)
bugfix: web/.htaccess: auto-detect dotpath in query string: added missing $ at end of regexp
inputs/VegBIEN/db/.htaccess: allow access to schemas other than the public schema by requiring a schema before the table. currently, this does not break any existing URLs.
planning/timeline/timeline.2013.xls: rebalanced between the months. Geoscrubbing re-run: extended to September since it's not clear when Jim will have time for this.
planning/timeline/timeline.2013.xls: split Data provider validations (spot-checking) into separate tasks for VegBIEN and VegCore
planning/timeline/timeline.2013.xls: updated for July progress
schemas/VegCore/VegCore.ERD.mwb: geovalidation: added step at end to georeference all places to corrected_geoplace
schemas/VegCore/VegCore.ERD.mwb: moved all required fields before optional fields
schemas/VegCore/VegCore.ERD.mwb: individual: identifying_place: documented that for specimens, this is the place the specimen was collected at. within_place: documented that this is not specified for specimens since their coordinates are usually not precise enough to identify an individual. specimen: documented that when there are multiple specimen replicates (copies) of a specimen, each gets its own specimen_observation pointing to the same specimen.
schemas/VegCore/VegCore.ERD.mwb: individual: added identifying_place since some individuals may be identified by their coordinates rather than a tag
schemas/VegCore/VegCore.ERD.mwb: added back aggregate_observation as a superclass of taxon_absence, taxon_presence. have only aggregate_observation, not taxon_observation, inherit from taxa_sampling_event because taxa_sampling_event is designed for multiple or aggregate samplings, but taxon_observation can be for just one occurrence. taxa_sampling_event: added required within_place since taxa must be sampled within a particular place. individual.place: renamed to within_place to clarify that each individual does not necessarily need its own place instance. individual_observation: added place_observed_at to serve the role that individual.place previously did (the place of an individual may change between observations).
schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: renamed to individual_count for clarity. added count field. renamed traits to aggregating_traits since these are used in uniquely identifying the individual_count. taxon_presence: added cover_percent.
schemas/VegCore/VegCore.ERD.mwb: taxon_presence: documented that this is actually equivalent to VegX's aggregateOrganismObservation ("An observation applying to all occurrences of an organism"), in spite of VegCore having its own aggregate_observation table with a different purpose
schemas/VegX/.htaccess, TCS/.htaccess: prepend / to subpath so that it becomes an absolute XPath. the subpath is now usually provided as a dotpath translated to a dirpath, which would not have a leading / .
fix: schemas/VegCore/VegCore.ERD.mwb.run: comment about why the Mac version does not export the PDF correctly: clarified that it is the single-page PDF where the lines take very long to load, and the printed PDF where the colors are wrong in Adobe Reader
schemas/VegCore/VegCore.ERD.mwb.run: documented that to edit VegCore.ERD.mwb, you must use the Linux version of MySQL Workbench. this is because the Mac version does not export the PDF correctly: the colors are wrong in Adobe Reader and the lines take very long to load. unfortunately, you also cannot switch back and forth between the two versions, because they render the lines with a different spacing, which makes the tables different sizes and misaligns the connector lines. it is especially unfortunate that the Mac version messes up the PDFs, because it uses a smaller line spacing and therefore smaller tables which allows the tables to be spaced farther apart for better readability.
schemas/VegCore/VegCore.ERD.mwb: aggregate_observation- and stratum-related tables: documented VegBank equivalents
schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements
bugfix: schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make(): use new create_db=1 option to mysqldump_local so that the dumpfile contains the necessary schema-qualifiers to be syncable with the ERD (i.e. to allow editing the ERD by changing the dumpfile)
lib/sh/db.sh: mysqldump(): added create_db=1 flag to print the CREATE DATABASE statement
bugfix: lib/sh/util.sh: pf(): echo func decl to stderr instead of stdout
lib/sh/util.sh: echo_vars(): documented that it only prints vars that are defined
lib/sh/util.sh: local_inv: also echo_vars the new var
schemas/VegCore/VegCore.ERD.mwb: fixed lines
bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: updated name of sampling_event to taxa_sampling_event
schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved place_observation-related tables before sampling_event because plots-related tables should come before occurrence-related tables
schemas/VegCore/VegCore.ERD.mwb: soil_observation: allow multiple soil_observations per place_observation (e.g. at different spots), as VegBank does
schemas/VegCore/VegCore.ERD.mwb: method: spaced more evenly from other tables
schemas/VegCore/VegCore.ERD.mwb: *.footprint_geom_WKT: renamed to boundary_WKT for clarity
schemas/VegCore/VegCore.ERD.mwb: added reobservable_presence, which allows a vouchered taxon_presence (e.g. a set of aggregate_observations) to have taxon redeterminations
schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: changed relationship to taxon_presence to many:1 instead of 1:1 inheritance, because there can be multiple aggregate_observations (e.g. size classes) for each aggregated taxon. added optional size_class field to indicate the aggregating group.
bugfix: schemas/VegCore/VegCore.ERD.mwb: method: made parent optional because top-level methods will not have a parent. rearranged to make more room for place_observation, soil_observation.
schemas/VegCore/VegCore.ERD.mwb: sampling_event: renamed to taxa_sampling_event for clarity. (fkey order has reversed.)
schemas/VegCore/VegCore.ERD.mwb: place_observation: don't inherit from sampling_event because sampling_event refers specifically to sampling taxa, and does not have anything to do with environmental measurements on the place itself. size_class: moved to occurrence half since it relates specifically to aggregate_observations, rather than plot strata.
schemas/VegCore/VegCore.ERD.mwb: stratum: added layer, size_class subclasses (with name now only required for layers)
schemas/VegCore/VegCore.ERD.mwb: stratum_event: added fkey to parent sampling_event. stratum: made space for subclasses.
schemas/VegCore/VegCore.ERD.mwb: added stratum_event. (the fkey order also got reversed.)
bugfix: inputs/VegBank/observation_/create.sql: ensure only one row per observation by selecting the first soilobs for each observation
bugfix: inputs/VegBank/stemlocation_/create.sql: include 1 row for each stemlocation rather than for each stemcount, because stemcounts are instead imported as part of stemcount_ (which also includes taxonimportance)
inputs/TEX/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/TEX/Specimen2/: translated new multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
bugfix: inputs/TEX/Specimen2/map.csv: DETERMINER FIRST NAME/LAST NAME: combined using _join_words() instead of the default _alt()
inputs/TEX/Specimen2/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/TEX/Specimen2/map.csv: MISC NOTES: remapped to locationRemarks since these primarily relate to the coords. PLANT DATA: remapped to occurrenceRemarks since this is the specimenDescription.
inputs/TEX/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/UNCC/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/UNCC/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns. reproductiveCondition: use the suggested values at DwC.vegpath.org?reproductiveCondition instead of just joining together the values in flower and fruit.
inputs/UNCC/Specimen/map.csv: usdaRank, infrarank: documented that they should actually be mapped to infraspecificRank*. it is better to map them to taxonRank for now because infraspecificRank is now mapped to VegBIEN.
mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).
inputs/UNCC/Specimen/map.csv: infrarank: remapped to EQUIV (of usdaRank). usdaRank, infrarank: documented the VegCore term that each would map to.
inputs/UNCC/Specimen/postprocess.sql: infrarank: remove mismapped values "L." (belong in authors, and are already there)
schemas/util.sql: added ?*>= operator (is populated more often than) from wiki.vegpath.org/Postgres_queries#is-populated-more-often-than
bugfix: web/.htaccess: parse dotpath in query string: make dotpath replacement start with ./ instead of / (which would incorrectly create an absolute URL path). this bug only appears when there is a trailing /, because this causes the matched text to be empty instead of a filesystem path with leading /.
bugfix: web/.htaccess: parse dotpath in query string: redirect so REQUEST_URI is populated from REQUEST_FILENAME: need to use - instead of $0 as the redirect destination so that REQUEST_FILENAME is used instead of the empty REQUEST_URI (i.e. the redirect vars are left untouched)
bugfix: web/.htaccess: parse dotpath in query string: need to add trailing / if dotpath so that dest path is URL path, not filesystem path (when trailing / missing, matched path is filesystem path)
backups/TNRS.backup.md5: updated for last backup
schemas/util.sql: hstore(text[], anyelement): replaced with just hstore(text[], text) so that bare literals (of type unknown) are auto-cast to text instead of causing a "could not determine polymorphic type because input has type "unknown"" error
inputs/UNCC/Specimen/map.csv: loanto, inorout: removed mapping to occurrenceRemarks because this information does not relate to the occurrence. it is also rarely populated.
inputs/import.stats.xls: added backup MD5 test time for last import
inputs/import.stats.xls: added backup upload time for last import
inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
/README.TXT: Full database import: min disk space: updated import schema size for last import
inputs/import.stats.xls: added backup times from last import
inputs/import.stats.xls: Updated import times
/README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run
bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the beginning of the line only
bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list
inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)
bugfix: schemas/util.sql: trim(): need to remove all added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state
schemas/util.sql: added added_cols()
bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name
inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.
inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count
bin/after_import: usage: documented that it requires $version
bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background
/README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import
