added bin/after_import, which performs post-normalized-import actions separately from bin/import_all
bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1
/README.TXT: Full database import: added steps to restart an aborted import for a specific table
inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.
inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table
bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately
bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.
bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)
/README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all
bin/import_all: don't set $dump_opts until running the backup command that uses it, so that the user can run this backup command separately just by copying the line out of the script (without worrying about env vars that need to be set, other than $version which is visible outside the script)
inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/U/Specimen/map.csv: Gazetteer/Newgazett, Majorarea: documented that these are the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Majorarea: mapped to stateProvince, which is the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Gazetteer/Newgazett: remapped to county, which is the closest equivalent for Guyana/Suriname
inputs/U/Specimen/map.csv: Gazetteer, Newgazett: combine them with _alt() instead of _join() because only one of Gazetteer, Newgazett is ever populated
inputs/NCU/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
bugfix: placed inputs/NCU/Specimen/postprocess.sql under version control
bugfix: inputs/NCU/Specimen/map.csv: CatalogSeriesPrefix: enclosed comment in "
inputs/NCU/Specimen/map.csv: OwnerInstitution: remapped to specimenOwner rather than specimenHolderInstitutions. OwnerInstitution, CatalogSeriesPrefix: documented the VegCore SQL dotpath (http://wiki.vegpath.org/SQL_dotpaths) that would be used to refer to the field. this specifies the destination field at a much finer level of detail than the one-size-fits-all denormalized name.
mappings/VegCore-VegBIEN.csv: mapped municipality
inputs/NCU/Specimen/map.csv: CityLocality: remapped to municipality because this is a placename, not a verbatim locality description
inputs/NCU/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/NY/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/$dest/$subdir/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/NY/Ecatalog_all/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/bien_web/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/UBC/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
/README.TXT: Full database import: don't exit the screen until after getting $version, which is defined within it
planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Attribution and conditions of use up because it's high priority. deferred Importing to normalized VegCore until after October due to decision to use VegBIEN for the October DB.
schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: refactored to separate geoscrubbing (matching a named place) from prepping for geovalidation (uniquifying the lat/long and parent place). note that only bare lat/longs without official placenames will be geovalidated.
schemas/VegCore/VegCore.ERD.mwb: geovalidation: inherit from georeferencing, since applying a corrected (or confirmed) lat/long is a form of georeferencing
bugfix: schemas/VegCore/VegCore.ERD.mwb: scrubbed_geoplace: made parent_geoplace nullable to allow a scrubbed root node which has no parent. added scrubbed_name to emphasize that any place.name applied in this table is scrubbed.
inputs/CTFS/StemObservation/unmapped_terms.csv: regenerated
inputs/UNCC/Specimen/new_terms.csv: regenerated
/README.TXT: Full database import: make test by_col=1: documented that if you encounter errors, they are most likely related to the PostgreSQL error parsing in /lib/sql.py parse_exception()
bugfix: lib/sql.py: parse_exception(): MissingCastException from DoesNotExistException for function: handle overloaded functions where none of the overloads supports the given arg types (so assume text). this may have become a bug from system upgrades?
added inputs/newWorld/iso_code_gadm/VegBIEN.csv, etc., generated by running ./run
added inputs/newWorld/newWorldCountries/map.csv, etc., generated by running ./run
inputs/NY/Ecatalog_all/map.csv: PlantFungDescription: documented that PlantFung confusingly refers to the plant/fungus the specimen came from, rather than to a fungus growing on the plant
schemas/VegCore/VegCore.ERD.mwb: fixed lines, including changing inheritance connectors to 1:1/optional on child table
schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that bin/redmine_synonyms can be used as a template
schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that the dimensions precede the matched name
schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: added reset() stub, with instructions of how to autogenerate this using dimensions in ../document.mwb.xml
schemas/VegCore/VegCore.ERD.mwb: fixed lines
bugfix: /Makefile: postgres-Linux: phppgadmin.conf: updated `ln -s` to /etc/apache2/conf-available/ for current name of /etc/apache2/conf.d/phppgadmin.conf, which is now just phppgadmin
added planning/workflow/derived_columns/range_measurements/low-high_vs_midpoint-uncertainty.VegBank.png diagram, generated from a screenshot of the VegBank Data Dictionary (VegBank.vegpath.org/plot#Elevation)
schemas/VegCore/VegCore.ERD.mwb: party category: made it pink to match VegBank (http://vegpath.org/exchange_schemas/VegBank/vegbank.ERD.pdf), swapping colors with source. (note that source isn't red like in VegBank because black text doesn't show up well against it.)
bugfix: schemas/VegCore/VegCore.ERD.pdf: regenerated from schemas/VegCore/VegCore.ERD.mwb, without any tables moused-over. apparently, mousing over a table while saving the PDF also saves the highlighting on the fields, not just when saving the PNG.
inputs/HVAA/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/HVAA/Specimen/map.csv: fieldNumber: remapped to UNUSED
inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
inputs/SpeciesLink/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
inputs/SpeciesLink/Specimen/map.csv: renamed DUPLICATE#of:... output columns to be <= 63 chars long, in order to be valid PostgreSQL columns without collisions
inputs/SpeciesLink/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
schemas/VegCore/VegCore.ERD.mwb: rel_place: renamed to subplace to clarify that this is any place contained within another place, not just places with a position relative to their parent place
schemas/VegCore/VegCore.ERD.mwb: individual.place: don't require this to be a rel_place, because some individuals will be associated with standalone specimens, which don't have a parent place
schemas/VegCore/VegCore.ERD.mwb: require rel_place to have a parent place. merge plot_element into subplot because subplot is the only place the parent_plot constraint is imposed (individuals can now be located in any place, not just a plot).
schemas/VegCore/VegCore.ERD.mwb: individual: require it to have a place (which may be a rel_place within another place). do not require the place to be a plot_element, because individuals can be located in places other than uniformly-shaped plots.
schemas/VegCore/VegCore.ERD.mwb: regenerated exports. this now includes updating the image map for moved and new tables.
schemas/VegCore/VegCore.ERD.mwb: added plot_element to require that the associated parent place of a rel_place be a plot. added subplot to require that a subplot be a plot_element (i.e. have a parent_plot) and to show where to put subplot data.
schemas/VegCore/ERD/VegCore.ERD.png.map.htm: ERD.pdf: fixed URL since the map is now in a subdir
schemas/VegCore/ERD/VegCore.ERD.png.map.htm: added URLs, which now autoredirect to the appropriate DB table on vegbiendev
schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: cleanup(): trim growing whitespace: remove multiple spaces at a time, in case the user saved and reopened VegCore.ERD.png.map.htm but didn't run this script in between
added schemas/VegCore/ERD/.htaccess to forward unknown subdirs to vegbiendev MySQL as tables
added web/servers/vegbiendev/db/my/, which forwards to MySQL instead of PostgreSQL
web/servers/vegbiendev/db/: moved PostgreSQL engine to separate pg/ subdir to allow for other engines
bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: #$AUTHOR: trim growing whitespace which Gimp repeatedly adds on each save
bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL
schemas/VegCore/ERD/VegCore.ERD.png.map.htm, index.htm: re-ran VegCore.ERD.png.map.htm.run (it needs to be run every time VegCore.ERD.png.map.htm is edited)
added schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run, which cleans up and formats Gimp's image map for publishing; along with derived file index.htm
added schemas/VegCore/ERD/VegCore.ERD.png.map.htm image map for VegCore.ERD.png. note that the tables are sorted, and this sort order supersedes the data dictionary sort order (which is somewhat similar). the table URLs have not been added yet.
lib/sh/util.sh: $sed_cmd: added usage
schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: 2. GNRS: split into substeps 2a. unique lat/long, 2b. names, 2c. geovalidatable place for clarity. don't refer to the scrubbed_geoplace as the GADM shape, because only the parent_geoplace has the shape (the scrubbed_geoplace just has the scrubbed names).
added schemas/VegCore/VegCore.ERD.letter_size.pdf. this must be generated in Linux rather than Mac, because the Mac PDF printer messes up the colors in the PDF (missing color profile?).
schemas/VegCore/VegCore.ERD.mwb: added labels for GNRS and geovalidation steps, analogous to the TNRS taxonomic scrubbing steps labels
schemas/VegCore/VegCore.ERD.mwb: regenerated exports. VegCore.ERD.png now gets the sRGB color profile attached in Gimp so that the colors don't look washed out on some LCD screens.
schemas/VegCore/VegCore.ERD.mwb: georeferencing: added hstore extender
schemas/VegCore/VegCore.ERD.mwb: individual: made it a plot element by optionally attaching a plot position (a rel_place whose parent is the containing plot)
schemas/VegCore/VegCore.ERD.mwb: plot: replaced bounding_box_rect with length_m, width_m, since the bounding box was intended to store plot dimensions (along the plot azimuth) rather than an actual bounding box aligned to the compass directions. added azimuth_deg_N, which is used to resolve plot element x/y coordinates to absolute geocoordinates while taking into account the rotation of the plot.
schemas/VegCore/VegCore.ERD.mwb: tables with parent hierarchies: made parent optional, since the root(s) of the hierarchy will not have an entry for this, and any unique constraints that include this column should be ignored (which they will be if the value is NULL instead of a self-pointer)
schemas/VegCore/VegCore.ERD.mwb: subplot: renamed to rel_place and inherit from place directly, in order to store other plot elements that are relative to their containing plot
schemas/VegCore/VegCore.ERD.mwb: taxon_path: converted to an auxiliary table of taxon_name instead of a subclass of it (like geopath for the place table). this causes distinct taxon_paths to be stored only once, instead of repeatedly for each taxon_name.
schemas/VegCore/VegCore.ERD.mwb: place hierarchy: reorganized to store scrubbed geoplaces in a containment hierarchy instead of a denormalized geopath. this allows each source-specific place to be GNRS-scrubbed to a GADM place, and then have its coordinates geovalidated to see if it is within the matched GADM place. this uses the georeferencing table to store the matched GADM place (scrubbed_geoplace) for each input place, instead of geopath_scrub to store the matched GADM geo*path* for each input geo*path*. (this avoids the need to scrub every combination of place ranks, because just the name of each place is scrubbed relative to its parent place.) geopath instead becomes an auxiliary table to store the place table's verbatim ranks, for easy access and storage.
inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_BoundingBox: remapped to UNUSED
schemas/VegCore/VegCore.ERD.mwb: place: renamed to local_place to distinguish it from geoplace, which is not a subclass of place (it is a separate, global table, while local_place is source-specific). note that renames sometimes need to be done manually on vegbiendev, to avoid triggering a MySQL bug that blocks the new table from being created and requires the entire database to be recreated to clear the error.
schemas/VegCore/VegCore.ERD.mwb: stem, stem_observation: made associated individual/individual_observation optional, because some stems (e.g. in VegBank) are not grouped together into individuals. note that a stem is still considered to BE-AN individual, but it is a type of individual which may be grouped under another, plant-level individual.
schemas/VegCore/VegCore.ERD.mwb: specimen_observation: added description (vegcore.vegpath.org?specimenDescription). taxon_presence: added occurrence_status (vegcore.vegpath.org?occurrenceStatus). stem_observation, aggregate_observation: made room for them to expand with additional first-class fields.
schemas/VegCore/VegCore.ERD.mwb: taxon_presence, taxon_absence: inherit from taxon_determination rather than taxon_observation, so that the taxon_determination's taxon can be used as the identifying taxon (i.e. the authorPlantName, VegCore.vegpath.org?authorPlantName)
schemas/VegCore/VegCore.ERD.mwb: taxon_determination: inherit from taxon_observation again because now that redeterminations can only occur on reobservable things, it makes sense to only allow one taxon_determination per observation event. this means that each redetermination on a specimen would get its own taxon_observation (where any additional attributes noted in the reobservation could also be included).
schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: renamed to reobservable to emphasize that this is only for things on which taxon redeterminations can be made, such as individuals and specimens (including voucher specimens). a redetermination on an aggregate_observation would instead be made on its voucher specimen, which is the only reobservable part of it.
schemas/VegCore/VegCore.ERD.mwb: moved taxon_observation subclasses closer to taxon_observation so that it would be clear they were observation-related rather than occurrence-related (e.g. there is no concept of "repeat-sampling" of an aggregate_observation, because each sampling it is the collector's opinion that the plants correspond to a particular taxon)
