Project

General

Profile

Statistics
| Revision:

# Date Author Comment
10524 07/31/2013 02:55 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10523 07/31/2013 02:49 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

10522 07/31/2013 02:45 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: require rel_place to have a parent place. merge plot_element into subplot because subplot is the only place the parent_plot constraint is imposed (individuals can now be located in any place, not just a plot).

10521 07/31/2013 02:38 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: individual: require it to have a place (which may be a rel_place within another place). do not require the place to be a plot_element, because individuals can be located in places other than uniformly-shaped plots.

10520 07/31/2013 02:23 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports. this now includes updating the image map for moved and new tables.

10519 07/31/2013 02:11 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added plot_element to require that the associated parent place of a rel_place be a plot. added subplot to require that a subplot be a plot_element (i.e. have a parent_plot) and to show where to put subplot data.

10518 07/31/2013 01:28 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.png.map.htm: ERD.pdf: fixed URL since the map is now in a subdir

10517 07/31/2013 01:22 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.png.map.htm: added URLs, which now autoredirect to the appropriate DB table on vegbiendev

10516 07/31/2013 01:21 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: cleanup(): trim growing whitespace: remove multiple spaces at a time, in case the user saved and reopened VegCore.ERD.png.map.htm but didn't run this script in between

10515 07/31/2013 01:01 AM Aaron Marcuse-Kubitza

added schemas/VegCore/ERD/.htaccess to forward unknown subdirs to vegbiendev MySQL as tables

10514 07/31/2013 01:00 AM Aaron Marcuse-Kubitza

added web/servers/vegbiendev/db/my/, which forwards to MySQL instead of PostgreSQL

10513 07/31/2013 01:00 AM Aaron Marcuse-Kubitza

web/servers/vegbiendev/db/: moved PostgreSQL engine to separate pg/ subdir to allow for other engines

10512 07/31/2013 12:23 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: #$AUTHOR: trim growing whitespace which Gimp repeatedly adds on each save

10511 07/31/2013 12:21 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL

10510 07/31/2013 12:19 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL

10509 07/31/2013 12:10 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.png.map.htm, index.htm: re-ran VegCore.ERD.png.map.htm.run (it needs to be run every time VegCore.ERD.png.map.htm is edited)

10508 07/31/2013 12:02 AM Aaron Marcuse-Kubitza

added schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run, which cleans up and formats Gimp's image map for publishing; along with derived file index.htm

10507 07/30/2013 11:59 PM Aaron Marcuse-Kubitza

added schemas/VegCore/ERD/VegCore.ERD.png.map.htm image map for VegCore.ERD.png. note that the tables are sorted, and this sort order supersedes the data dictionary sort order (which is somewhat similar). the table URLs have not been added yet.

10506 07/30/2013 10:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $sed_cmd: added usage

10505 07/30/2013 09:46 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: 2. GNRS: split into substeps 2a. unique lat/long, 2b. names, 2c. geovalidatable place for clarity. don't refer to the scrubbed_geoplace as the GADM shape, because only the parent_geoplace has the shape (the scrubbed_geoplace just has the scrubbed names).

10504 07/30/2013 09:10 PM Aaron Marcuse-Kubitza

added schemas/VegCore/VegCore.ERD.letter_size.pdf. this must be generated in Linux rather than Mac, because the Mac PDF printer messes up the colors in the PDF (missing color profile?).

10503 07/30/2013 08:34 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added labels for GNRS and geovalidation steps, analogous to the TNRS taxonomic scrubbing steps labels

10502 07/30/2013 07:25 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports. VegCore.ERD.png now gets the sRGB color profile attached in Gimp so that the colors don't look washed out on some LCD screens.

10501 07/30/2013 07:13 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: georeferencing: added hstore extender

10500 07/30/2013 07:08 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: individual: made it a plot element by optionally attaching a plot position (a rel_place whose parent is the containing plot)

10499 07/30/2013 06:54 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: plot: replaced bounding_box_rect with length_m, width_m, since the bounding box was intended to store plot dimensions (along the plot azimuth) rather than an actual bounding box aligned to the compass directions. added azimuth_deg_N, which is used to resolve plot element x/y coordinates to absolute geocoordinates while taking into account the rotation of the plot.

10498 07/30/2013 06:42 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: tables with parent hierarchies: made parent optional, since the root(s) of the hierarchy will not have an entry for this, and any unique constraints that include this column should be ignored (which they will be if the value is NULL instead of a self-pointer)

10497 07/30/2013 06:37 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: subplot: renamed to rel_place and inherit from place directly, in order to store other plot elements that are relative to their containing plot

10496 07/30/2013 06:26 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_path: converted to an auxiliary table of taxon_name instead of a subclass of it (like geopath for the place table). this causes distinct taxon_paths to be stored only once, instead of repeatedly for each taxon_name.

10495 07/30/2013 06:16 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place hierarchy: reorganized to store scrubbed geoplaces in a containment hierarchy instead of a denormalized geopath. this allows each source-specific place to be GNRS-scrubbed to a GADM place, and then have its coordinates geovalidated to see if it is within the matched GADM place. this uses the georeferencing table to store the matched GADM place (scrubbed_geoplace) for each input place, instead of geopath_scrub to store the matched GADM geo*path* for each input geo*path*. (this avoids the need to scrub every combination of place ranks, because just the name of each place is scrubbed relative to its parent place.) geopath instead becomes an auxiliary table to store the place table's verbatim ranks, for easy access and storage.

10494 07/30/2013 04:51 PM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_BoundingBox: remapped to UNUSED

10493 07/30/2013 03:22 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place: renamed to local_place to distinguish it from geoplace, which is not a subclass of place (it is a separate, global table, while local_place is source-specific). note that renames sometimes need to be done manually on vegbiendev, to avoid triggering a MySQL bug that blocks the new table from being created and requires the entire database to be recreated to clear the error.

10492 07/30/2013 03:02 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: stem, stem_observation: made associated individual/individual_observation optional, because some stems (e.g. in VegBank) are not grouped together into individuals. note that a stem is still considered to BE-AN individual, but it is a type of individual which may be grouped under another, plant-level individual.

10491 07/30/2013 02:47 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10490 07/30/2013 02:45 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: specimen_observation: added description (vegcore.vegpath.org?specimenDescription). taxon_presence: added occurrence_status (vegcore.vegpath.org?occurrenceStatus). stem_observation, aggregate_observation: made room for them to expand with additional first-class fields.

10489 07/30/2013 02:22 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_presence, taxon_absence: inherit from taxon_determination rather than taxon_observation, so that the taxon_determination's taxon can be used as the identifying taxon (i.e. the authorPlantName, VegCore.vegpath.org?authorPlantName)

10488 07/30/2013 01:41 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_determination: inherit from taxon_observation again because now that redeterminations can only occur on reobservable things, it makes sense to only allow one taxon_determination per observation event. this means that each redetermination on a specimen would get its own taxon_observation (where any additional attributes noted in the reobservation could also be included).

10487 07/30/2013 01:31 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: renamed to reobservable to emphasize that this is only for things on which taxon redeterminations can be made, such as individuals and specimens (including voucher specimens). a redetermination on an aggregate_observation would instead be made on its voucher specimen, which is the only reobservable part of it.

10486 07/30/2013 01:07 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: moved taxon_observation subclasses closer to taxon_observation so that it would be clear they were observation-related rather than occurrence-related (e.g. there is no concept of "repeat-sampling" of an aggregate_observation, because each sampling it is the collector's opinion that the plants correspond to a particular taxon)

10485 07/30/2013 02:00 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.png: switched back to attaching the sRGB color profile directly, because actually, the native->sRGB translation happens in the monitor driver itself (and can be adjusted in System Preferences > Displays > Color), rather than in the specific application. this means that the hex color values color-matched in MySQL Workbench were actually sRGB (translated by the OS to monitor-native for display), and that the sRGB profile merely needed to be explicitly indicated for other monitors that are not close to sRGB (and thus need the translation). the closeness of the 27-inch iMac screen to sRGB can be verified by selecting sRGB in System Preferences > Displays > Color, and noting that the desktop background does not change from when the default "iMac" setting is selected.

10484 07/30/2013 01:38 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.png: convert to sRGB color profile after attaching the native monitor profile instead of attaching it directly. this allows the hex colors that were color-matched in MySQL Workbench (which presumably uses raw monitor RGB) to be translated to the universal sRGB space, where they can then be localized to a different monitor's local color space. note that this does not visibly change the image on the 27-inch iMac screen from what was produced via the previous, incorrect method (attaching the sRGB profile without conversion from native), which would imply that the iMac's screen is very close to the sRGB color space already. if this is the case, it is instead older LCDs that have off-white color spaces that need translation from sRGB.

10483 07/29/2013 11:39 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.png: attached sRGB color profile using Gimp (gimp.org), so that the colors don't look completely washed out and off-hue on older LCDs (i.e. other than the 27-inch iMac screen)

10482 07/27/2013 12:13 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports

10481 07/27/2013 12:09 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added separate geo category (turquoise) to visually distinguish the broader geoplace tables from the more specific plot tables. (note that georeferencing is actually a plot table despite geo- in its name, because it assigns a geoplace to a plot.)

10480 07/27/2013 11:51 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: georeferencing: added georeferenced_by

10479 07/27/2013 11:46 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added georeferencing table for georeference* DwC fields. this can be used to link a place to a georeferenced geoplace other than (or in addition to) the original geoplace.

10478 07/27/2013 11:14 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added geopath_scrub for GNRS results (separate from point-in-polygon validation)

10477 07/27/2013 10:51 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place: factored optional geocoords, geopath out into separate geoplace table (with both nullable), which validatable_geoplace (renamed from geoplace, with both NOT NULL) extends

10476 07/27/2013 10:25 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: made scrubbed_geoplace optional because not all geoplaces will scrub to a valid geoplace

10475 07/27/2013 10:22 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: need to inherit from record now that this is source-specific

10474 07/27/2013 10:17 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: HAVE-AN input geoplace rather than BEING-ONE, to allow multiple geovalidations for a geoplace by different sources

10473 07/27/2013 10:05 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_determination: changed IS-A relationship with taxon_observation to HAS-A so that a separate taxon_observation doesn't need to be created for each taxon_determination (even though each taxon_determination event is theoretically a reobservation of the specimen, etc.). instead, inherit from sampling_event to include the necessary event-related fields.

10472 07/27/2013 09:45 AM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb: geopath: made country NOT NULL so that every geoplace (for input to geovalidation) has something on the geopath side. geocoords: made latitude_deg/longitude_deg NOT NULL so that every geoplace (for input to geovalidation) has something on the geocoords side. added geocoords_unique constraint since this is a global table with one entry for each lat/long.

10471 07/27/2013 09:30 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place: added coords hstore extender, for verbatim coordinates, etc.

10470 07/27/2013 09:14 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: coordinates: abbreviated to coords (unambiguous abbreviation)

10469 07/27/2013 08:59 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: replaced parsed_taxon_assertion with taxon_scrub, which HAS-A parsed taxon_assertion rather than BEING-A parsed_taxon_assertion. (multiple TNRS results may parse to the same thing.)

10468 07/27/2013 08:08 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: geovalidatable_place: renamed to geoplace, since this uniquification is useful independently of geovalidation. note that the MySQL upgrade on vegbiendev has now reordered the fkeys again, this time in forwards order.

10467 07/27/2013 08:02 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for July progress

10466 07/27/2013 07:43 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place tables that are absolute within Earth rather than relative to a parent place: prefixed geo- to table name for clarity

10465 07/27/2013 07:23 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: plot, subplot: added hstore extenders (dimensions, coordinates)

10464 07/27/2013 07:17 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed inheritance connectors to be 1:1, optional on subclass

10463 07/27/2013 07:11 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: plot: added shape. bounding_box: changed units to rect, since this just needs a width/height (the x/y coord is the lat/long).

10462 07/27/2013 07:05 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: plot: added footprint_geom_WKT. bounding_box: added units (WKT).

10461 07/27/2013 06:51 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: back-synced from staging copy on vegbiendev to flush out sync changes that it kept trying to re-make

10460 07/27/2013 06:47 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: event: moved method to separate sampling_event subclass

10459 07/27/2013 06:28 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10458 07/27/2013 06:25 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: inherit from taxon_presence, since this is a type of taxon_presence and it avoids duplicating the taxon_concept field

10457 07/27/2013 06:11 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added taxon_absence, to avoid including absence observations in the same table as presence observations (which needlessly complicates queries). note that the fkey order now gets set back to forwards whenever a table is changed.

10456 07/27/2013 06:07 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: re-saved. the fkey order is now apparently reversed for recently-changed tables.

10455 07/26/2013 11:07 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: collector, identified_by: allow multiple parties for these fields, using the new party_list array table

10454 07/26/2013 10:44 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: party arrays: use new party_list array table instead of adding a separate many:many table for each table that uses a party array. this also allows using the party_list ID in a unique constraint, because it is now a first-class field.

10453 07/26/2013 10:06 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: party: added party_list array table

10452 07/26/2013 09:45 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: party: added optional fkey to organization

10451 07/26/2013 09:32 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: geovalidation: renamed lat_long_in_ranks to lat_long_in_place_ranks for clarity

10450 07/26/2013 09:12 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: individual: added tag_history hstore to store custom identity attributes

10449 07/26/2013 08:39 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_string: documented that to get the parsed_taxon_assertion (TNRS result) for a taxon_string, you would join using the SQL dotpath taxon_string.string<-taxon_assertion(string)::parsed_taxon_assertion[source='TNRS.version'] (see wiki.vegpath.org/SQL_dotpaths). important how-to comments such as this one are now included in the version-controlled MySQL schema file itself, not just the .mwb file and the staging copy on vegbiendev.

10448 07/26/2013 08:16 PM Aaron Marcuse-Kubitza

bin/my2pg: use s!...!...! when either the regexp or the replacement contains / , to avoid unnecessary \-s

10447 07/26/2013 08:09 PM Aaron Marcuse-Kubitza

bin/my2pg: commenting out table options: added explanatory comment, because it is not obvious from the regexp what this does

10446 07/26/2013 08:06 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysqldump(): don't use --compatible=postgresql when the table structure is being exported, because this removes the table options (which include the COMMENT attribute). --compatible=postgresql remains on in data-only mode because embedded ` in data cannot easily be distinguished from ` around column names, so ANSI_QUOTES is needed to do the translation to " (and data sections do not contain table options). note that all --compatible modes that offer ANSI_QUOTES unfortunately exclude the table options, and there is no way to run a SQL query to set the SQL mode before beginning the dump, so ANSI_QUOTES translation must be handled by my2pg instead.

10445 07/26/2013 06:35 PM Aaron Marcuse-Kubitza

bin/my2pg: comment out table options (http://dev.mysql.com/doc/refman/5.5/en/server-sql-mode.html#sqlmode_no_table_options) instead of removing them, because they include table COMMENTs, which contain important metadata such as table definitions. (note that table COMMENTs use a slightly different syntax than column COMMENTs, so the table COMMENTs will not be commented out twice.)

10444 07/26/2013 06:19 PM Aaron Marcuse-Kubitza

bin/my2pg: comment out COMMENTs instead of removing them so that they will be included in the PostgreSQL translation. COMMENTs contain important metadata about columns, such as definitions and the meanings of integer flag values.

10443 07/26/2013 05:58 PM Aaron Marcuse-Kubitza

inputs/{.,}*/*.schema.sql: regenerated using the instructions in bin/my2pg. this primarily replaces timestamp with text/*timestamp*/ (to preserve indefinite dates).

10442 07/26/2013 05:56 PM Aaron Marcuse-Kubitza

bin/my2pg: added instructions for regenerating *.schema.sql whenever this script is changed

10441 07/26/2013 05:22 PM Aaron Marcuse-Kubitza

bin/my2pg: COMMENT: also match COMMENTs with embedded ', because there will only be one COMMENT per line, so the contents of the COMMENT can just extend to the last ' on the line

10440 07/26/2013 05:16 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: $sed_cmd: make output unbuffered, so that running e.g. bin/my2pg at the command line produces output as each line is read

10439 07/26/2013 04:29 PM Aaron Marcuse-Kubitza

bin/my2pg: replace MySQL ` quotes with " quotes to support exports that were generated without ANSI_QUOTES mode. (this replacement only applies to schema exports, not data.) ANSI_QUOTES is only available with mysqldump --compatible modes that also include NO_TABLE_OPTIONS, which omits important table options such as comments. in particular, these comments are part of schemas/VegCore/VegCore.ERD.mwb but were not being included in VegCore.my.sql.

10438 07/26/2013 01:41 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_string: removed parsed_taxon_assertion field, since there may be more than one parsing (TNRS result) for a given taxon_string. the parsing relationship can better be represented by adding a parsed_taxon_assertion whose taxon_assertion.string points to the parsed taxon_string. getting the parsed_taxon_assertion for a taxon_string now requires joining on parsed_taxon_assertion using a backwards instead of forwards fkey, and filtering the corresponding assertions to include only the ones for TNRS (of the desired TNRS version). documented that taxon_assertion.string was previously the concatenated matched name, but is now the TNRS input name. the concatenated matched name is still in parsed_taxon_assertion.matched_taxon_concept->:taxon_name.unique_name.

10437 07/26/2013 01:22 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.my.sql: regenerated from .mwb schema, which apparently reverses the order of the fkeys (possibly a Linux MySQL bug?)

10436 07/26/2013 12:26 PM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: remapped Darwin Core synonyms to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic alts to perform to resolve filter-less collisions. the svn diff was verified by replacing DUPLICATE#of:dwc_terms<term>#... with <term>, removing the comment, and checking that this removes the diff (except where VegCore has renamed a DwC term).

10435 07/26/2013 12:17 PM Aaron Marcuse-Kubitza

bugfix: inputs/SpeciesLink/Specimen/map.csv: *scientificName: remapped to scientificName instead of taxonName to match the DwC term's name (this is the same dwc_terms_scientificName mismapping that was fixed in r10434)

10434 07/26/2013 11:56 AM Aaron Marcuse-Kubitza

bugfix: inputs/SpeciesLink/Specimen/map.csv: dwc_terms_scientificName: remapped to scientificName instead of taxonName to match that DwC term name, as well as the mappings of other *scientificName terms

10433 07/26/2013 11:06 AM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: marked dwc_geospatial_VerbatimLatitude,Longitude as exact duplicates of dwc_terms_*

10432 07/26/2013 10:52 AM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: remapped identical _alt-ed fields to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic _alts to perform to resolve filter-less collisions.

10431 07/26/2013 10:06 AM Aaron Marcuse-Kubitza

bugfix: inputs/SpeciesLink/Specimen/map.csv: *CollectorNumber: moved these to the same _alt group as recordNumber, because they are actually duplicates

10430 07/26/2013 09:43 AM Aaron Marcuse-Kubitza

correction: inputs/SpeciesLink/Specimen/map.csv: FieldNumber: fixed incorrect comment that these fields are identical to recordNumber, when instead they have the same *meaning but not the same values. instead, values are stored under either of the two terms. the previous conclusion had been based on an incorrect query, which used != instead of the NULL-sensitive IS NOT DISTINCT FROM.

10429 07/25/2013 08:14 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: Adding derived columns: extended to overlap with all subtasks

10428 07/25/2013 08:12 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: Geoscrubbing: split into separate re-run and automated pipeline tasks

10427 07/25/2013 08:09 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved Data provider validations before Adding derived columns because ensuring that the source data is in the database is more important than the derived data, which can always be added later

10426 07/25/2013 08:00 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: Data provider validations: added dot in July because some amount of datasource-level validation happens when mappings issues are discovered during the refactoring

10425 07/25/2013 07:34 PM Aaron Marcuse-Kubitza

bugfix: inputs/*/*/map.csv for specimen tables: remapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself rather than to any parent event (specimens don't have a parent event)