schemas/VegCore/VegCore.ERD.mwb: fixed lines
bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: updated name of sampling_event to taxa_sampling_event
schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved place_observation-related tables before sampling_event because plots-related tables should come before occurrence-related tables
schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
schemas/VegCore/VegCore.ERD.mwb: soil_observation: allow multiple soil_observations per place_observation (e.g. at different spots), as VegBank does
schemas/VegCore/VegCore.ERD.mwb: method: spaced more evenly from other tables
schemas/VegCore/VegCore.ERD.mwb: *.footprint_geom_WKT: renamed to boundary_WKT for clarity
schemas/VegCore/VegCore.ERD.mwb: added reobservable_presence, which allows a vouchered taxon_presence (e.g. a set of aggregate_observations) to have taxon redeterminations
schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: changed relationship to taxon_presence to many:1 instead of 1:1 inheritance, because there can be multiple aggregate_observations (e.g. size classes) for each aggregated taxon. added optional size_class field to indicate the aggregating group.
bugfix: schemas/VegCore/VegCore.ERD.mwb: method: made parent optional because top-level methods will not have a parent. rearranged to make more room for place_observation, soil_observation.
schemas/VegCore/VegCore.ERD.mwb: sampling_event: renamed to taxa_sampling_event for clarity. (fkey order has reversed.)
schemas/VegCore/VegCore.ERD.mwb: place_observation: don't inherit from sampling_event because sampling_event refers specifically to sampling taxa, and does not have anything to do with environmental measurements on the place itself. size_class: moved to occurrence half since it relates specifically to aggregate_observations, rather than plot strata.
schemas/VegCore/VegCore.ERD.mwb: stratum: added layer, size_class subclasses (with name now only required for layers)
schemas/VegCore/VegCore.ERD.mwb: stratum_event: added fkey to parent sampling_event. stratum: made space for subclasses.
schemas/VegCore/VegCore.ERD.mwb: added stratum_event. (the fkey order also got reversed.)
bugfix: inputs/VegBank/observation_/create.sql: ensure only one row per observation by selecting the first soilobs for each observation
bugfix: inputs/VegBank/stemlocation_/create.sql: include 1 row for each stemlocation rather than for each stemcount, because stemcounts are instead imported as part of stemcount_ (which also includes taxonimportance)
inputs/TEX/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/TEX/Specimen2/: translated new multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
bugfix: inputs/TEX/Specimen2/map.csv: DETERMINER FIRST NAME/LAST NAME: combined using _join_words() instead of the default _alt()
inputs/TEX/Specimen2/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/TEX/Specimen2/map.csv: MISC NOTES: remapped to locationRemarks since these primarily relate to the coords. PLANT DATA: remapped to occurrenceRemarks since this is the specimenDescription.
inputs/TEX/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/UNCC/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/UNCC/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns. reproductiveCondition: use the suggested values at DwC.vegpath.org?reproductiveCondition instead of just joining together the values in flower and fruit.
inputs/UNCC/Specimen/map.csv: usdaRank, infrarank: documented that they should actually be mapped to infraspecificRank*. it is better to map them to taxonRank for now because infraspecificRank is now mapped to VegBIEN.
mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).
inputs/UNCC/Specimen/map.csv: infrarank: remapped to EQUIV (of usdaRank). usdaRank, infrarank: documented the VegCore term that each would map to.
inputs/UNCC/Specimen/postprocess.sql: infrarank: remove mismapped values "L." (belong in authors, and are already there)
schemas/util.sql: added ?*>= operator (is populated more often than) from wiki.vegpath.org/Postgres_queries#is-populated-more-often-than
bugfix: web/.htaccess: parse dotpath in query string: make dotpath replacement start with ./ instead of / (which would incorrectly create an absolute URL path). this bug only appears when there is a trailing /, because this causes the matched text to be empty instead of a filesystem path with leading /.
bugfix: web/.htaccess: parse dotpath in query string: redirect so REQUEST_URI is populated from REQUEST_FILENAME: need to use - instead of $0 as the redirect destination so that REQUEST_FILENAME is used instead of the empty REQUEST_URI (i.e. the redirect vars are left untouched)
bugfix: web/.htaccess: parse dotpath in query string: need to add trailing / if dotpath so that dest path is URL path, not filesystem path (when trailing / missing, matched path is filesystem path)
backups/TNRS.backup.md5: updated for last backup
schemas/util.sql: hstore(text[], anyelement): replaced with just hstore(text[], text) so that bare literals (of type unknown) are auto-cast to text instead of causing a "could not determine polymorphic type because input has type "unknown"" error
inputs/UNCC/Specimen/map.csv: loanto, inorout: removed mapping to occurrenceRemarks because this information does not relate to the occurrence. it is also rarely populated.
inputs/import.stats.xls: added backup MD5 test time for last import
inputs/import.stats.xls: added backup upload time for last import
inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
/README.TXT: Full database import: min disk space: updated import schema size for last import
inputs/import.stats.xls: added backup times from last import
inputs/import.stats.xls: Updated import times
/README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run
bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the beginning of the line only
bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list
inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)
bugfix: schemas/util.sql: trim(): need to remove all added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state
schemas/util.sql: added added_cols()
bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name
inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.
inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count
bin/after_import: usage: documented that it requires $version
bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background
/README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import
bin/import_all: use new bin/after_import
added bin/after_import, which performs post-normalized-import actions separately from bin/import_all
bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1
/README.TXT: Full database import: added steps to restart an aborted import for a specific table
inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.
inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table
bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately
bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.
bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)
/README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all
bin/import_all: don't set $dump_opts until running the backup command that uses it, so that the user can run this backup command separately just by copying the line out of the script (without worrying about env vars that need to be set, other than $version which is visible outside the script)
inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/U/Specimen/map.csv: Gazetteer/Newgazett, Majorarea: documented that these are the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Majorarea: mapped to stateProvince, which is the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Gazetteer/Newgazett: remapped to county, which is the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Gazetteer, Newgazett: combine them with _alt() instead of _join() because only one of Gazetteer, Newgazett is ever populated
inputs/NCU/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
bugfix: placed inputs/NCU/Specimen/postprocess.sql under version control
bugfix: inputs/NCU/Specimen/map.csv: CatalogSeriesPrefix: enclosed comment in "
inputs/NCU/Specimen/map.csv: OwnerInstitution: remapped to specimenOwner rather than specimenHolderInstitutions. OwnerInstitution, CatalogSeriesPrefix: documented the VegCore SQL dotpath (http://wiki.vegpath.org/SQL_dotpaths) that would be used to refer to the field. this specifies the destination field at a much finer level of detail than the one-size-fits-all denormalized name.
mappings/VegCore-VegBIEN.csv: mapped municipality
inputs/NCU/Specimen/map.csv: CityLocality: remapped to municipality because this is a placename, not a verbatim locality description
inputs/NCU/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/NY/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/$dest/$subdir/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/NY/Ecatalog_all/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/bien_web/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/UBC/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
/README.TXT: Full database import: don't exit the screen until after getting $version, which is defined within it
planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Attribution and conditions of use up because it's high priority. deferred Importing to normalized VegCore until after October due to decision to use VegBIEN for the October DB.
schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: refactored to separate geoscrubbing (matching a named place) from prepping for geovalidation (uniquifying the lat/long and parent place). note that only bare lat/longs without official placenames will be geovalidated.
schemas/VegCore/VegCore.ERD.mwb: geovalidation: inherit from georeferencing, since applying a corrected (or confirmed) lat/long is a form of georeferencing
bugfix: schemas/VegCore/VegCore.ERD.mwb: scrubbed_geoplace: made parent_geoplace nullable to allow a scrubbed root node which has no parent. added scrubbed_name to emphasize that any place.name applied in this table is scrubbed.
inputs/CTFS/StemObservation/unmapped_terms.csv: regenerated
inputs/UNCC/Specimen/new_terms.csv: regenerated
/README.TXT: Full database import: make test by_col=1: documented that if you encounter errors, they are most likely related to the PostgreSQL error parsing in /lib/sql.py parse_exception()
bugfix: lib/sql.py: parse_exception(): MissingCastException from DoesNotExistException for function: handle overloaded functions where none of the overloads supports the given arg types (so assume text). this may have become a bug from system upgrades?
added inputs/newWorld/iso_code_gadm/VegBIEN.csv, etc., generated by running ./run
added inputs/newWorld/newWorldCountries/map.csv, etc., generated by running ./run
inputs/NY/Ecatalog_all/map.csv: PlantFungDescription: documented that PlantFung confusingly refers to the plant/fungus the specimen came from, rather than to a fungus growing on the plant
schemas/VegCore/VegCore.ERD.mwb: fixed lines, including changing inheritance connectors to 1:1/optional on child table
