Activity
From 07/25/2013 to 08/23/2013
08/23/2013
- 11:43 PM Revision 10723: inputs/.geoscrub/Source/map.csv: mapped datasetURL
- 11:41 PM Revision 10722: mappings/VegCore-VegBIEN.csv: mapped datasetURL
- 11:38 PM Revision 10721: mappings/VegCore-VegBIEN.csv: mapped datasetURL
08/22/2013
- 06:12 PM Revision 10720: fix: mappings/VegCore-VegBIEN.csv: source__modified_date: remapped to pubdate instead of datelastmodified because this is actually metadata for the source itself, rather than for the VegBIEN record *of* the source
- 05:56 PM Revision 10719: fix: inputs/.geoscrub/Source/map.csv: source__modified_date: use the mtime of the CSV file instead, since this is closer to the actual version of the biengeo code at the time it was run
- 05:41 PM Revision 10718: inputs/.geoscrub/Source/map.csv: mapped source__modified_date. note that the test must be run with inputs/.geoscrub/Source/run instead of `make inputs/.geoscrub/Source/test` to add these metadata columns to the staging table.
- 05:38 PM Revision 10717: mappings/VegCore-VegBIEN.csv: mapped source__modified_date (different from vegcore.vegpath.org?modified, which is for the data record)
- 05:36 PM Revision 10716: mappings/VegCore.htm: regenerated from wiki. added source__version (= edition), source__modified_date.
- 05:33 PM Revision 10715: bugfix: schemas/util.sql: set_col_names_with_metadata(): *rename* any metadata cols rather than re-adding them with new names
- 04:38 PM Revision 10714: mappings/VegCore-VegBIEN.csv: mapped edition
- 04:36 PM Revision 10713: bugfix: inputs/.geoscrub/{Source,geoscrub_output}/VegBIEN.csv: switched to the version needed for new-style datasources
- 04:12 PM Revision 10712: inputs/.geoscrub/Source/map.csv: mapped edition (the version), using `svn info derived/biengeo/`
- 03:53 PM Revision 10711: schemas/vegbien.sql: source.revision: renamed to import_revision for clarity
- 03:52 PM Revision 10710: schemas/vegbien.my.sql: updated with `make schemas/remake`
- 03:44 PM Revision 10709: schemas/vegbien.sql: source: datecreated, datelastmodified: default to now() like in VegBank (schemas/VegBank/vegbank.sql)
- 03:29 PM Revision 10708: schemas/vegbien.sql: source: added datecreated, datelastmodified, etc. for source-level tracking of import and revision (wiki.vegpath.org/2013-08-22_conference_call#source-level-tracking-of-import-and-revision)
- 02:54 PM Revision 10707: added derived/biengeo/ from https://projects.nceas.ucsb.edu/nceas/projects/biengeo/repository/
- 02:50 PM Revision 10706: added /derived
- 01:04 PM Revision 10705: web/links/index.htm: updated to Firefox bookmarks. Gmvault: added steps to do full backup and to backup only new e-mails
- 12:48 PM Revision 10704: planning/timeline/timeline.2013.xls: flagged timeline issues that can be done by iPlant personnel: Attribution and conditions of use, Geoscrubbing re-run, Geoscrubbing automated pipeline, Improve and complete data provider metadata, Obtain any additional new data
- 11:12 AM Revision 10703: web/links/index.htm: updated to Firefox bookmarks. Gmvault: added run instructions for Mac.
- 10:50 AM Revision 10702: web/links/index.htm: updated to Firefox bookmarks. added link to Gmvault (Gmail backup), which wouldn't install for me on Mac (but that may be because I'm using 10.8, and Gmvault is for 10.7/10.6)
- 10:26 AM Revision 10701: added planning/meetings/BIEN conference call availability.xlsx (backup of Google spreadsheet)
- 10:10 AM Revision 10700: planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Data provider validations (spot-checking) to beginning since that seems to have been decided to be a higher priority than architectural changes
08/21/2013
- 06:50 PM Revision 10699: schemas/util.sql: combining functions taking anyelement params which could be text: take text param instead, so that other argument types (e.g. integer) will first be implicitly cast to text instead of trying to concatenate integers directly. this fixes a bug in the VegBank.stemcount_,stemlocation_ _join() of two integer pkeys, which first needed to be cast to text. anyelement was previously used so that other text-like types such as varchar could also be used, but varchar is implicitly castable to text so keeping anyelement should not be necessary.
- 06:07 PM Revision 10698: planning/timeline/timeline.2013.xls: added tasks to Avoid DB restructuring when ingesting a new datasource and Streamline process of mapping and adding a new datasource (not yet put in priority order)
08/20/2013
- 08:49 PM Revision 10697: inputs/VegBank/observation_/test.xml.ref: updated inserted row count
- 02:40 PM Revision 10696: bugfix: inputs/VegBank/stemlocation_/map.csv: also _join together taxonimportance_id, stemcount_id for aggregateOrganismObservationID so that the aggregateoccurrence pkeys match up with those imported from stemcount_
- 01:40 PM Revision 10695: bugfix: inputs/VegBank/stemcount_/map.csv: aggregateOrganismObservationID: prepend taxonimportance_id so that rows with only a taxonimportance entry (no stemcounts) will also have the required sourceaccessioncode
- 12:59 PM Revision 10694: schemas/vegbien.sql: analytical_stem: synced with analytical_stem_view using sync_analytical_stem_to_view()
- 12:58 PM Revision 10693: bugfix: schemas/vegbien.sql: sync_analytical_stem_to_view(): added re-creation of range_modeling_input view
- 11:40 AM Revision 10692: schemas/vegbien.sql: analytical_stem_view: added aggregateOrganismObservationID (aggregateoccurrence.sourceaccessioncode) so aggregateoccurrences can be matched back up to their input rows (e.g. VegBank.stemcount)
- 10:31 AM Revision 10691: inputs/VegBank/taxonobservation_/map.csv: plantname: remapped to DUPLICATE#of:plantconcept_plantname because this is an exact duplicate
- 10:24 AM Revision 10690: bugfix: inputs/VegBank/taxonobservation_/map.csv: updated input column names for renamings in inputs/VegBank/vegbank.~.clean_up.sql
- 10:21 AM Revision 10689: inputs/VegBank/taxonobservation_/map.csv: Species and lower ranks: remapped to EQUIV#to:plantname because these contain the taxonomic name at specific ranks, but plantname contains the taxonomic name of the plant itself, which is longer and populated more often
08/19/2013
- 05:48 AM Revision 10688: inputs/VegBank/taxonobservation_/create.sql: also join to plantname, since plantconcept.plantname may not always be populated when plantname.plantname is
- 05:36 AM Revision 10687: fix: inputs/VegBank/taxonobservation_/map.csv: Species and below: remapped to _alts of scientificName, because these are actually the *full* taxonomic name at that rank, not just the epithet. Genus: documented that it includes the genus author.
- 05:28 AM Revision 10686: inputs/VegBank/vegbank.~.clean_up.sql: disambiguated plantconcept.plantname, plantname.reference_id to enable joining plantconcept_->plantname
- 05:13 AM Revision 10685: fix: inputs/VegBank/taxonobservation_/map.csv: also mapped plantname to scientificName, since int_currplantscifull is not always provided when this is. (it cannot replace int_currplantscifull, because when int_currplantscifull also provided, this often leaves out lower ranks.) this should fill in taxonomic information for taxonobservations that are currently missing it.
- 03:27 AM Revision 10684: bugfix: schemas/vegbien.sql: analytical_stem_view: coordinates: use the coordinates from datasource_place instead of canon_place, because canon_place's coordinates are only what the geoscrubbing output and do not contain datasource-specific information such as coordsaccuracy_m
- 01:53 AM Revision 10683: inputs/VegBank/taxonobservation_/map.csv: collector_id: remapped to UNUSED. removed LEFT JOIN collector_id->party since this field is never populated.
- 01:16 AM Revision 10682: inputs/VegBank/plot_/map.csv: area|country|territory, region|state|province (from place table): remapped to DUPLICATE, since these have the same data as, and are populated less often than, their country/stateprovince couterparts
08/18/2013
- 11:39 PM Revision 10681: bugfix: inputs/VegBank/plot_/create.sql: need to join place.*plot_id* to plot.plot_id instead of plotplace_id. this is the cause of the "State is wrong, not Wyoming, but Tennessee" and "County is incorrect (not Powell, but Orange)" bugs in the VegBank spot-checking (wiki.vegpath.org/Spot-checking#Great-Smoky-Mountains-National-Park).
- 10:42 PM Revision 10680: inputs/VegBank/taxonobservation_/map.csv: int_origplantscifull: remapped to EQUIV (to authorplantname). this is the scrubbed originalScientificName, but we do our own scrubbing.
- 10:23 PM Revision 10679: fix: inputs/VegBank/taxonobservation_/map.csv: authorplantname: remapped to originalScientificName because it includes the name author
- 10:21 PM Revision 10678: inputs/VegBank/taxonobservation_/map.csv: mapped int_origplant*, int_currplant* to *scientificName/*taxonName/etc.
- 09:48 PM Revision 10677: inputs/VegBank/plot_/map.csv: elevation: documented that it has only 5 decimal places of precision, with only 9s and random #s after that
- 09:47 PM Revision 10676: inputs/VegBank/plot_/test.xml.ref: update rowcount
- 09:15 PM Revision 10675: inputs/VegBank/stemlocation_/map.csv: stemcode, stemxposition, stemyposition: remapped to UNUSED. stemhealth is the only data field in this table that is populated, which means that VegBank does not provide data on reobservable stems even though the schema supports it.
- 09:12 PM Revision 10674: inputs/VegBank/observation_/postprocess.sql: added pkey
- 08:19 PM Revision 10673: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 08:18 PM Revision 10672: schemas/VegCore/ERD/index.htm: regenerated
- 08:05 PM Revision 10671: schemas/VegCore/VegCore.ERD.mwb: taxon_observation: added observation_in_parent_place, which points to the observation of the same taxon/individual in the parent place. this accounts for VegBank allowing multiple taxonImportances per taxonObservation, which only makes sense when each taxonImportance is from a different stratum and they point to a common taxonObservation for the parent plot.
- 07:58 PM Revision 10670: bugfix: web/.htaccess: auto-detect dotpath in query string: need to include the path ($0) in the replacement, to avoid reverting to the root dir. (mod_rewrite replacements are not like relative URLs, which would interpret ?... as being relative to the *current* path, not the root.)
- 07:56 PM Revision 10669: bugfix: web/.htaccess: auto-detect dotpath in query string: added missing $ at end of regexp
08/10/2013
08/09/2013
- 01:57 PM Revision 10667: planning/timeline/timeline.2013.xls: rebalanced between the months. Geoscrubbing re-run: extended to September since it's not clear when Jim will have time for this.
- 01:53 PM Revision 10666: planning/timeline/timeline.2013.xls: split Data provider validations (spot-checking) into separate tasks for VegBIEN and VegCore
- 01:49 PM Revision 10665: planning/timeline/timeline.2013.xls: updated for July progress
- 01:25 PM Revision 10664: schemas/VegCore/VegCore.ERD.mwb: geovalidation: added step at end to georeference all places to corrected_geoplace
- 12:51 PM Revision 10663: schemas/VegCore/VegCore.ERD.mwb: moved all required fields before optional fields
- 12:39 PM Revision 10662: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 12:23 PM Revision 10661: schemas/VegCore/VegCore.ERD.mwb: individual: identifying_place: documented that for specimens, this is the place the specimen was collected at. within_place: documented that this is not specified for specimens since their coordinates are usually not precise enough to identify an individual. specimen: documented that when there are multiple specimen replicates (copies) of a specimen, each gets its own specimen_observation pointing to the same specimen.
- 12:12 PM Revision 10660: schemas/VegCore/VegCore.ERD.mwb: individual: added identifying_place since some individuals may be identified by their coordinates rather than a tag
- 12:06 PM Revision 10659: schemas/VegCore/VegCore.ERD.mwb: added back aggregate_observation as a superclass of taxon_absence, taxon_presence. have only aggregate_observation, not taxon_observation, inherit from taxa_sampling_event because taxa_sampling_event is designed for multiple or aggregate samplings, but taxon_observation can be for just one occurrence. taxa_sampling_event: added required within_place since taxa must be sampled within a particular place. individual.place: renamed to within_place to clarify that each individual does not necessarily need its own place instance. individual_observation: added place_observed_at to serve the role that individual.place previously did (the place of an individual may change between observations).
- 10:41 AM Revision 10658: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: renamed to individual_count for clarity. added count field. renamed traits to aggregating_traits since these are used in uniquely identifying the individual_count. taxon_presence: added cover_percent.
- 10:17 AM Revision 10657: schemas/VegCore/VegCore.ERD.mwb: taxon_presence: documented that this is actually equivalent to VegX's aggregateOrganismObservation ("An observation applying to all occurrences of an organism"), in spite of VegCore having its own aggregate_observation table with a different purpose
- 10:12 AM Revision 10656: schemas/VegX/.htaccess, TCS/.htaccess: prepend / to subpath so that it becomes an absolute XPath. the subpath is now usually provided as a dotpath translated to a dirpath, which would not have a leading / .
- 12:08 AM Revision 10655: fix: schemas/VegCore/VegCore.ERD.mwb.run: comment about why the Mac version does not export the PDF correctly: clarified that it is the *single-page* PDF where the lines take very long to load, and the *printed* PDF where the colors are wrong in Adobe Reader
- 12:02 AM Revision 10654: schemas/VegCore/VegCore.ERD.mwb.run: documented that to edit VegCore.ERD.mwb, you *must* use the Linux version of MySQL Workbench. this is because the Mac version does not export the PDF correctly: the colors are wrong in Adobe Reader and the lines take very long to load. unfortunately, you also cannot switch back and forth between the two versions, because they render the lines with a different spacing, which makes the tables different sizes and misaligns the connector lines. it is especially unfortunate that the Mac version messes up the PDFs, because it uses a smaller line spacing and therefore smaller tables which allows the tables to be spaced farther apart for better readability.
08/08/2013
- 11:49 PM Revision 10653: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation- and stratum-related tables: documented VegBank equivalents
- 10:20 PM Revision 10652: schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements
- 10:16 PM Revision 10651: schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements
- 10:14 PM Revision 10650: bugfix: schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make(): use new create_db=1 option to mysqldump_local so that the dumpfile contains the necessary schema-qualifiers to be syncable with the ERD (i.e. to allow editing the ERD by changing the dumpfile)
- 07:15 PM Revision 10649: lib/sh/db.sh: mysqldump(): added create_db=1 flag to print the CREATE DATABASE statement
- 07:15 PM Revision 10648: bugfix: lib/sh/util.sh: pf(): echo func decl to stderr instead of stdout
- 07:10 PM Revision 10647: lib/sh/util.sh: echo_vars(): documented that it only prints vars that are defined
- 07:10 PM Revision 10646: lib/sh/util.sh: local_inv: also echo_vars the new var
- 06:17 PM Revision 10645: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 06:16 PM Revision 10644: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 06:06 PM Revision 10643: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 06:04 PM Revision 10642: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: updated name of sampling_event to taxa_sampling_event
- 06:03 PM Revision 10641: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved place_observation-related tables before sampling_event because plots-related tables should come before occurrence-related tables
- 06:01 PM Revision 10640: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 05:34 PM Revision 10639: schemas/VegCore/VegCore.ERD.mwb: soil_observation: allow multiple soil_observations per place_observation (e.g. at different spots), as VegBank does
- 05:27 PM Revision 10638: schemas/VegCore/VegCore.ERD.mwb: method: spaced more evenly from other tables
- 05:21 PM Revision 10637: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 05:20 PM Revision 10636: schemas/VegCore/VegCore.ERD.mwb: *.footprint_geom_WKT: renamed to boundary_WKT for clarity
- 05:14 PM Revision 10635: schemas/VegCore/VegCore.ERD.mwb: added reobservable_presence, which allows a vouchered taxon_presence (e.g. a set of aggregate_observations) to have taxon redeterminations
- 04:44 PM Revision 10634: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: changed relationship to taxon_presence to many:1 instead of 1:1 inheritance, because there can be multiple aggregate_observations (e.g. size classes) for each aggregated taxon. added optional size_class field to indicate the aggregating group.
- 04:34 PM Revision 10633: bugfix: schemas/VegCore/VegCore.ERD.mwb: method: made parent optional because top-level methods will not have a parent. rearranged to make more room for place_observation, soil_observation.
- 04:18 PM Revision 10632: schemas/VegCore/VegCore.ERD.mwb: sampling_event: renamed to taxa_sampling_event for clarity. (fkey order has reversed.)
- 04:12 PM Revision 10631: schemas/VegCore/VegCore.ERD.mwb: place_observation: don't inherit from sampling_event because sampling_event refers specifically to sampling taxa, and does not have anything to do with environmental measurements on the place itself. size_class: moved to occurrence half since it relates specifically to aggregate_observations, rather than plot strata.
- 03:14 PM Revision 10630: schemas/VegCore/VegCore.ERD.mwb: stratum: added layer, size_class subclasses (with name now only required for layers)
- 02:51 PM Revision 10629: schemas/VegCore/VegCore.ERD.mwb: stratum_event: added fkey to parent sampling_event. stratum: made space for subclasses.
- 02:22 PM Revision 10628: schemas/VegCore/VegCore.ERD.mwb: added stratum_event. (the fkey order also got reversed.)
- 01:26 PM Revision 10627: bugfix: inputs/VegBank/observation_/create.sql: ensure only one row per observation by selecting the first soilobs for each observation
- 12:09 PM Revision 10626: bugfix: inputs/VegBank/stemlocation_/create.sql: include 1 row for each stemlocation rather than for each stemcount, because stemcounts are instead imported as part of stemcount_ (which also includes taxonimportance)
08/07/2013
- 04:51 PM Revision 10625: inputs/TEX/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 04:44 PM Revision 10624: inputs/TEX/Specimen2/: translated new multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:41 PM Revision 10623: bugfix: inputs/TEX/Specimen2/map.csv: DETERMINER FIRST NAME/LAST NAME: combined using _join_words() instead of the default _alt()
- 04:20 PM Revision 10622: inputs/TEX/Specimen2/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:13 PM Revision 10621: inputs/TEX/Specimen2/map.csv: MISC NOTES: remapped to locationRemarks since these primarily relate to the coords. PLANT DATA: remapped to occurrenceRemarks since this is the specimenDescription.
- 03:15 PM Revision 10620: inputs/TEX/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 10:22 AM Revision 10619: inputs/UNCC/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 09:50 AM Revision 10618: inputs/UNCC/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns. reproductiveCondition: use the suggested values at DwC.vegpath.org?reproductiveCondition instead of just joining together the values in flower and fruit.
- 09:19 AM Revision 10617: inputs/UNCC/Specimen/map.csv: usdaRank, infrarank: documented that they should actually be mapped to infraspecificRank*. it is better to map them to taxonRank for now because infraspecificRank is now mapped to VegBIEN.
- 09:07 AM Revision 10616: mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).
- 09:04 AM Revision 10615: inputs/UNCC/Specimen/map.csv: infrarank: remapped to EQUIV (of usdaRank). usdaRank, infrarank: documented the VegCore term that each would map to.
- 08:49 AM Revision 10614: inputs/UNCC/Specimen/postprocess.sql: infrarank: remove mismapped values "L." (belong in authors, and are already there)
- 08:30 AM Revision 10613: schemas/util.sql: added ?*>= operator (is populated more often than) from wiki.vegpath.org/Postgres_queries#is-populated-more-often-than
- 08:12 AM Revision 10612: bugfix: web/.htaccess: parse dotpath in query string: make dotpath replacement start with ./ instead of / (which would incorrectly create an absolute URL path). this bug only appears when there is a trailing /, because this causes the matched text to be empty instead of a filesystem path with leading /.
- 08:04 AM Revision 10611: bugfix: web/.htaccess: parse dotpath in query string: redirect so REQUEST_URI is populated from REQUEST_FILENAME: need to use - instead of $0 as the redirect destination so that REQUEST_FILENAME is used instead of the empty REQUEST_URI (i.e. the redirect vars are left untouched)
- 07:45 AM Revision 10610: bugfix: web/.htaccess: parse dotpath in query string: need to add trailing / if dotpath so that dest path is URL path, not filesystem path (when trailing / missing, matched path is filesystem path)
- 07:44 AM Revision 10609: backups/TNRS.backup.md5: updated for last backup
- 06:33 AM Revision 10608: schemas/util.sql: hstore(text[], anyelement): replaced with just hstore(text[], text) so that bare literals (of type unknown) are auto-cast to text instead of causing a "could not determine polymorphic type because input has type "unknown"" error
- 05:56 AM Revision 10607: inputs/UNCC/Specimen/map.csv: loanto, inorout: removed mapping to occurrenceRemarks because this information does not relate to the occurrence. it is also rarely populated.
08/06/2013
- 05:07 PM Revision 10606: inputs/import.stats.xls: added backup MD5 test time for last import
- 05:03 PM Revision 10605: inputs/import.stats.xls: added backup upload time for last import
- 04:55 PM Revision 10604: inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:53 PM Revision 10603: /README.TXT: Full database import: min disk space: updated import schema size for last import
- 04:30 PM Revision 10602: inputs/import.stats.xls: added backup times from last import
- 01:35 AM Revision 10601: inputs/import.stats.xls: Updated import times
- 01:16 AM Revision 10600: /README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run
- 12:32 AM Revision 10599: bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the *beginning* of the line only
- 12:29 AM Revision 10598: bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list
08/04/2013
- 06:07 PM Revision 10597: inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 06:06 PM Revision 10596: bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)
- 05:58 PM Revision 10595: bugfix: schemas/util.sql: trim(): need to remove *all* added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state
- 05:55 PM Revision 10594: schemas/util.sql: added added_cols()
- 04:06 PM Revision 10593: bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name
- 01:29 AM Revision 10592: inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:23 AM Revision 10591: inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.
- 01:10 AM Revision 10590: inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count
- 12:59 AM Revision 10589: bin/after_import: usage: documented that it requires $version
- 12:58 AM Revision 10588: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background
- 12:57 AM Revision 10587: /README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import
08/03/2013
- 09:14 PM Revision 10586: bin/import_all: use new bin/after_import
- 09:13 PM Revision 10585: added bin/after_import, which performs post-normalized-import actions separately from bin/import_all
- 03:34 PM Revision 10584: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1
- 03:32 PM Revision 10583: /README.TXT: Full database import: added steps to restart an aborted import for a specific table
- 03:28 PM Revision 10582: inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.
- 03:11 PM Revision 10581: inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table
- 12:25 AM Revision 10580: bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately
- 12:24 AM Revision 10579: bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.
- 12:03 AM Revision 10578: bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)
- 12:00 AM Revision 10577: /README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all
08/02/2013
08/01/2013
- 05:02 PM Revision 10575: inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 05:00 PM Revision 10574: inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:56 PM Revision 10573: inputs/U/Specimen/map.csv: Gazetteer/Newgazett, Majorarea: documented that these are the closest equivalent for Guyana/Suriname
- 04:55 PM Revision 10572: inputs/U/Specimen/map.csv: Majorarea: mapped to stateProvince, which is the closest equivalent for Guyana/Suriname
- 04:54 PM Revision 10571: inputs/U/Specimen/map.csv: Gazetteer/Newgazett: remapped to county, which is the closest equivalent for Guyana/Suriname
- 04:37 PM Revision 10570: inputs/U/Specimen/map.csv: Gazetteer, Newgazett: combine them with _alt() instead of _join() because only one of Gazetteer, Newgazett is ever populated
- 04:04 PM Revision 10569: inputs/NCU/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 04:02 PM Revision 10568: bugfix: placed inputs/NCU/Specimen/postprocess.sql under version control
- 03:57 PM Revision 10567: bugfix: inputs/NCU/Specimen/map.csv: CatalogSeriesPrefix: enclosed comment in "
- 03:43 PM Revision 10566: inputs/NCU/Specimen/map.csv: OwnerInstitution: remapped to specimenOwner rather than specimenHolderInstitutions. OwnerInstitution, CatalogSeriesPrefix: documented the VegCore SQL dotpath (http://wiki.vegpath.org/SQL_dotpaths) that would be used to refer to the field. this specifies the destination field at a much finer level of detail than the one-size-fits-all denormalized name.
- 02:56 PM Revision 10565: mappings/VegCore-VegBIEN.csv: mapped municipality
- 02:46 PM Revision 10564: inputs/NCU/Specimen/map.csv: CityLocality: remapped to municipality because this is a placename, not a verbatim locality description
- 02:25 PM Revision 10563: inputs/NCU/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 02:07 PM Revision 10562: inputs/NY/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 02:02 PM Revision 10561: inputs/$dest/$subdir/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:28 PM Revision 10560: inputs/NY/Ecatalog_all/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:11 PM Revision 10559: inputs/bien_web/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 11:37 AM Revision 10558: inputs/UBC/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 11:07 AM Revision 10557: /README.TXT: Full database import: don't exit the screen until after getting $version, which is defined within it
- 09:49 AM Revision 10556: planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Attribution and conditions of use up because it's high priority. deferred Importing to normalized VegCore until after October due to decision to use VegBIEN for the October DB.
- 03:51 AM Revision 10555: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:46 AM Revision 10554: schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: refactored to separate geoscrubbing (matching a named place) from prepping for geovalidation (uniquifying the lat/long and parent place). note that only bare lat/longs without official placenames will be geovalidated.
- 02:36 AM Revision 10553: schemas/VegCore/VegCore.ERD.mwb: geovalidation: inherit from georeferencing, since applying a corrected (or confirmed) lat/long is a form of georeferencing
- 02:29 AM Revision 10552: bugfix: schemas/VegCore/VegCore.ERD.mwb: scrubbed_geoplace: made parent_geoplace nullable to allow a scrubbed root node which has no parent. added scrubbed_name to emphasize that any place.name applied in this table is scrubbed.
- 12:47 AM Revision 10551: inputs/CTFS/StemObservation/unmapped_terms.csv: regenerated
- 12:43 AM Revision 10550: inputs/UNCC/Specimen/new_terms.csv: regenerated
- 12:22 AM Revision 10549: /README.TXT: Full database import: make test by_col=1: documented that if you encounter errors, they are most likely related to the PostgreSQL error parsing in /lib/sql.py parse_exception()
07/31/2013
- 11:58 PM Revision 10548: bugfix: lib/sql.py: parse_exception(): MissingCastException from DoesNotExistException for function: handle overloaded functions where none of the overloads supports the given arg types (so assume text). this may have become a bug from system upgrades?
- 11:35 PM Revision 10547: added inputs/newWorld/iso_code_gadm/VegBIEN.csv, etc., generated by running ./run
- 11:34 PM Revision 10546: added inputs/newWorld/newWorldCountries/map.csv, etc., generated by running ./run
- 10:54 PM Revision 10545: inputs/NY/Ecatalog_all/map.csv: PlantFungDescription: documented that PlantFung confusingly refers to the plant/fungus the specimen came from, rather than to a fungus growing on the plant
- 10:29 PM Revision 10544: schemas/VegCore/VegCore.ERD.mwb: fixed lines, including changing inheritance connectors to 1:1/optional on child table
- 09:21 PM Revision 10543: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that bin/redmine_synonyms can be used as a template
- 09:14 PM Revision 10542: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that the dimensions precede the matched name
- 09:13 PM Revision 10541: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: added reset() stub, with instructions of how to autogenerate this using dimensions in ../document.mwb.xml
- 08:55 PM Revision 10540: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 08:28 PM Revision 10539: bugfix: /Makefile: postgres-Linux: phppgadmin.conf: updated `ln -s` to /etc/apache2/conf-available/ for current name of /etc/apache2/conf.d/phppgadmin.conf, which is now just phppgadmin
- 07:52 PM Revision 10538: added planning/workflow/derived_columns/range_measurements/low-high_vs_midpoint-uncertainty.VegBank.png diagram, generated from a screenshot of the VegBank Data Dictionary (VegBank.vegpath.org/plot#Elevation)
- 05:21 PM Revision 10537: schemas/VegCore/VegCore.ERD.mwb: party category: made it pink to match VegBank (http://vegpath.org/exchange_schemas/VegBank/vegbank.ERD.pdf), swapping colors with source. (note that source isn't red like in VegBank because black text doesn't show up well against it.)
- 05:06 PM Revision 10536: bugfix: schemas/VegCore/VegCore.ERD.pdf: regenerated from schemas/VegCore/VegCore.ERD.mwb, without any tables moused-over. apparently, mousing over a table while saving the *PDF* also saves the highlighting on the fields, not just when saving the PNG.
- 04:56 PM Revision 10535: inputs/HVAA/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 04:52 PM Revision 10534: inputs/HVAA/Specimen/map.csv: fieldNumber: remapped to UNUSED
- 04:32 PM Revision 10533: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:31 PM Revision 10532: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:25 PM Revision 10531: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:04 PM Revision 10530: inputs/SpeciesLink/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 03:59 PM Revision 10529: inputs/SpeciesLink/Specimen/map.csv: renamed DUPLICATE#of:... output columns to be <= 63 chars long, in order to be valid PostgreSQL columns without collisions
- 03:37 PM Revision 10528: inputs/SpeciesLink/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 03:12 AM Revision 10527: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:10 AM Revision 10526: schemas/VegCore/VegCore.ERD.mwb: rel_place: renamed to subplace to clarify that this is any place contained within another place, not just places *with a position* relative to their parent place
- 03:05 AM Revision 10525: schemas/VegCore/VegCore.ERD.mwb: individual.place: don't require this to be a rel_place, because some individuals will be associated with standalone specimens, which don't have a parent place
- 02:55 AM Revision 10524: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 02:49 AM Revision 10523: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 02:45 AM Revision 10522: schemas/VegCore/VegCore.ERD.mwb: require rel_place to have a parent place. merge plot_element into subplot because subplot is the only place the parent_plot constraint is imposed (individuals can now be located in any place, not just a plot).
- 02:38 AM Revision 10521: schemas/VegCore/VegCore.ERD.mwb: individual: require it to have a place (which may be a rel_place within another place). do not require the place to be a plot_element, because individuals can be located in places other than uniformly-shaped plots.
- 02:23 AM Revision 10520: schemas/VegCore/VegCore.ERD.mwb: regenerated exports. this now includes updating the image map for moved and new tables.
- 02:11 AM Revision 10519: schemas/VegCore/VegCore.ERD.mwb: added plot_element to require that the associated parent place of a rel_place be a *plot*. added subplot to require that a subplot be a plot_element (i.e. have a parent_plot) and to show where to put subplot data.
- 01:28 AM Revision 10518: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: ERD.pdf: fixed URL since the map is now in a subdir
- 01:22 AM Revision 10517: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: added URLs, which now autoredirect to the appropriate DB table on vegbiendev
- 01:21 AM Revision 10516: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: cleanup(): trim growing whitespace: remove multiple spaces at a time, in case the user saved and reopened VegCore.ERD.png.map.htm but didn't run this script in between
- 01:01 AM Revision 10515: added schemas/VegCore/ERD/.htaccess to forward unknown subdirs to vegbiendev MySQL as tables
- 01:00 AM Revision 10514: added web/servers/vegbiendev/db/my/, which forwards to MySQL instead of PostgreSQL
- 01:00 AM Revision 10513: web/servers/vegbiendev/db/: moved PostgreSQL engine to separate pg/ subdir to allow for other engines
- 12:23 AM Revision 10512: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: #$AUTHOR: trim growing whitespace which Gimp repeatedly adds on each save
- 12:21 AM Revision 10511: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL
- 12:19 AM Revision 10510: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL
- 12:10 AM Revision 10509: schemas/VegCore/ERD/VegCore.ERD.png.map.htm, index.htm: re-ran VegCore.ERD.png.map.htm.run (it needs to be run every time VegCore.ERD.png.map.htm is edited)
- 12:02 AM Revision 10508: added schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run, which cleans up and formats Gimp's image map for publishing; along with derived file index.htm
07/30/2013
- 11:59 PM Revision 10507: added schemas/VegCore/ERD/VegCore.ERD.png.map.htm image map for VegCore.ERD.png. note that the tables are sorted, and this sort order supersedes the data dictionary sort order (which is somewhat similar). the table URLs have not been added yet.
- 10:07 PM Revision 10506: lib/sh/util.sh: $sed_cmd: added usage
- 09:46 PM Revision 10505: schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: 2. GNRS: split into substeps 2a. unique lat/long, 2b. names, 2c. geovalidatable place for clarity. don't refer to the scrubbed_geoplace as the GADM shape, because only the parent_geoplace has the shape (the scrubbed_geoplace just has the scrubbed names).
- 09:10 PM Revision 10504: added schemas/VegCore/VegCore.ERD.letter_size.pdf. this must be generated in Linux rather than Mac, because the Mac PDF printer messes up the colors in the PDF (missing color profile?).
- 08:34 PM Revision 10503: schemas/VegCore/VegCore.ERD.mwb: added labels for GNRS and geovalidation steps, analogous to the TNRS taxonomic scrubbing steps labels
- 07:25 PM Revision 10502: schemas/VegCore/VegCore.ERD.mwb: regenerated exports. VegCore.ERD.png now gets the sRGB color profile attached in Gimp so that the colors don't look washed out on some LCD screens.
- 07:13 PM Revision 10501: schemas/VegCore/VegCore.ERD.mwb: georeferencing: added hstore extender
- 07:08 PM Revision 10500: schemas/VegCore/VegCore.ERD.mwb: individual: made it a plot element by optionally attaching a plot position (a rel_place whose parent is the containing plot)
- 06:54 PM Revision 10499: schemas/VegCore/VegCore.ERD.mwb: plot: replaced bounding_box_rect with length_m, width_m, since the bounding box was intended to store plot dimensions (along the plot azimuth) rather than an actual bounding box aligned to the compass directions. added azimuth_deg_N, which is used to resolve plot element x/y coordinates to absolute geocoordinates while taking into account the rotation of the plot.
- 06:42 PM Revision 10498: schemas/VegCore/VegCore.ERD.mwb: tables with parent hierarchies: made parent optional, since the root(s) of the hierarchy will not have an entry for this, and any unique constraints that include this column should be ignored (which they will be if the value is NULL instead of a self-pointer)
- 06:37 PM Revision 10497: schemas/VegCore/VegCore.ERD.mwb: subplot: renamed to rel_place and inherit from place directly, in order to store other plot elements that are relative to their containing plot
- 06:26 PM Revision 10496: schemas/VegCore/VegCore.ERD.mwb: taxon_path: converted to an auxiliary table of taxon_name instead of a subclass of it (like geopath for the place table). this causes distinct taxon_paths to be stored only once, instead of repeatedly for each taxon_name.
- 06:16 PM Revision 10495: schemas/VegCore/VegCore.ERD.mwb: place hierarchy: reorganized to store scrubbed geoplaces in a containment hierarchy instead of a denormalized geopath. this allows each source-specific place to be GNRS-scrubbed to a GADM place, and then have its coordinates geovalidated to see if it is within the matched GADM place. this uses the georeferencing table to store the matched GADM *place* (scrubbed_geoplace) for each input *place*, instead of geopath_scrub to store the matched GADM geo*path* for each input geo*path*. (this avoids the need to scrub every combination of place ranks, because just the name of each place is scrubbed relative to its parent place.) geopath instead becomes an auxiliary table to store the place table's verbatim ranks, for easy access and storage.
- 04:51 PM Revision 10494: inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_BoundingBox: remapped to UNUSED
- 03:22 PM Revision 10493: schemas/VegCore/VegCore.ERD.mwb: place: renamed to local_place to distinguish it from geoplace, which is *not* a subclass of place (it is a separate, global table, while local_place is source-specific). note that renames sometimes need to be done manually on vegbiendev, to avoid triggering a MySQL bug that blocks the new table from being created and requires the entire database to be recreated to clear the error.
- 03:02 PM Revision 10492: schemas/VegCore/VegCore.ERD.mwb: stem, stem_observation: made associated individual/individual_observation optional, because some stems (e.g. in VegBank) are not grouped together into individuals. note that a stem is still considered to BE-AN individual, but it is a type of individual which may be grouped under another, plant-level individual.
- 02:47 PM Revision 10491: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 02:45 PM Revision 10490: schemas/VegCore/VegCore.ERD.mwb: specimen_observation: added description (vegcore.vegpath.org?specimenDescription). taxon_presence: added occurrence_status (vegcore.vegpath.org?occurrenceStatus). stem_observation, aggregate_observation: made room for them to expand with additional first-class fields.
- 02:22 PM Revision 10489: schemas/VegCore/VegCore.ERD.mwb: taxon_presence, taxon_absence: inherit from taxon_determination rather than taxon_observation, so that the taxon_determination's taxon can be used as the identifying taxon (i.e. the authorPlantName, VegCore.vegpath.org?authorPlantName)
- 01:41 PM Revision 10488: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: inherit from taxon_observation again because now that redeterminations can only occur on reobservable things, it makes sense to only allow one taxon_determination per observation event. this means that each redetermination on a specimen would get its own taxon_observation (where any additional attributes noted in the reobservation could also be included).
- 01:31 PM Revision 10487: schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: renamed to reobservable to emphasize that this is only for things on which taxon redeterminations can be made, such as individuals and specimens (including voucher specimens). a redetermination on an aggregate_observation would instead be made on its voucher specimen, which is the only reobservable part of it.
- 01:07 PM Revision 10486: schemas/VegCore/VegCore.ERD.mwb: moved taxon_observation subclasses closer to taxon_observation so that it would be clear they were observation-related rather than occurrence-related (e.g. there is no concept of "repeat-sampling" of an aggregate_observation, because each sampling it is the collector's opinion that the plants correspond to a particular taxon)
- 02:00 AM Revision 10485: bugfix: schemas/VegCore/VegCore.ERD.png: switched back to attaching the sRGB color profile directly, because actually, the native->sRGB translation happens in the monitor driver itself (and can be adjusted in System Preferences > Displays > Color), rather than in the specific application. this means that the hex color values color-matched in MySQL Workbench *were* actually sRGB (translated *by the OS* to monitor-native for display), and that the sRGB profile merely needed to be *explicitly indicated* for other monitors that are *not* close to sRGB (and thus need the translation). the closeness of the 27-inch iMac screen to sRGB can be verified by selecting sRGB in System Preferences > Displays > Color, and noting that the desktop background does not change from when the default "iMac" setting is selected.
- 01:38 AM Revision 10484: bugfix: schemas/VegCore/VegCore.ERD.png: convert to sRGB color profile after attaching the *native* monitor profile instead of attaching it directly. this allows the hex colors that were color-matched in MySQL Workbench (which presumably uses raw monitor RGB) to be translated to the universal sRGB space, where they can then be localized to a different monitor's local color space. note that this does not visibly change the image on the 27-inch iMac screen from what was produced via the previous, incorrect method (attaching the sRGB profile without conversion from native), which would imply that the iMac's screen is very close to the sRGB color space already. if this is the case, it is instead older LCDs that have off-white color spaces that need translation from sRGB.
07/29/2013
07/27/2013
- 12:13 PM Revision 10482: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 12:09 PM Revision 10481: schemas/VegCore/VegCore.ERD.mwb: added separate geo category (turquoise) to visually distinguish the broader geoplace tables from the more specific plot tables. (note that georeferencing is actually a plot table despite geo- in its name, because it assigns a geoplace *to a plot*.)
- 11:51 AM Revision 10480: schemas/VegCore/VegCore.ERD.mwb: georeferencing: added georeferenced_by
- 11:46 AM Revision 10479: schemas/VegCore/VegCore.ERD.mwb: added georeferencing table for georeference* DwC fields. this can be used to link a place to a georeferenced geoplace other than (or in addition to) the original geoplace.
- 11:14 AM Revision 10478: schemas/VegCore/VegCore.ERD.mwb: added geopath_scrub for GNRS results (separate from point-in-polygon validation)
- 10:51 AM Revision 10477: schemas/VegCore/VegCore.ERD.mwb: place: factored optional geocoords, geopath out into separate geoplace table (with both nullable), which validatable_geoplace (renamed from geoplace, with both NOT NULL) extends
- 10:25 AM Revision 10476: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: made scrubbed_geoplace optional because not all geoplaces will scrub to a valid geoplace
- 10:22 AM Revision 10475: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: need to inherit from record now that this is source-specific
- 10:17 AM Revision 10474: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: HAVE-AN input geoplace rather than BEING-ONE, to allow multiple geovalidations for a geoplace by different sources
- 10:05 AM Revision 10473: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: changed IS-A relationship with taxon_observation to HAS-A so that a separate taxon_observation doesn't need to be created for each taxon_determination (even though each taxon_determination event is theoretically a reobservation of the specimen, etc.). instead, inherit from sampling_event to include the necessary event-related fields.
- 09:45 AM Revision 10472: bugfix: schemas/VegCore/VegCore.ERD.mwb: geopath: made country NOT NULL so that every geoplace (for input to geovalidation) has something on the geopath side. geocoords: made latitude_deg/longitude_deg NOT NULL so that every geoplace (for input to geovalidation) has something on the geocoords side. added geocoords_unique constraint since this is a global table with one entry for each lat/long.
- 09:30 AM Revision 10471: schemas/VegCore/VegCore.ERD.mwb: place: added coords hstore extender, for verbatim coordinates, etc.
- 09:14 AM Revision 10470: schemas/VegCore/VegCore.ERD.mwb: coordinates: abbreviated to coords (unambiguous abbreviation)
- 08:59 AM Revision 10469: schemas/VegCore/VegCore.ERD.mwb: replaced parsed_taxon_assertion with taxon_scrub, which HAS-A parsed taxon_assertion rather than BEING-A parsed_taxon_assertion. (multiple TNRS results may parse to the same thing.)
- 08:08 AM Revision 10468: schemas/VegCore/VegCore.ERD.mwb: geovalidatable_place: renamed to geoplace, since this uniquification is useful independently of geovalidation. note that the MySQL upgrade on vegbiendev has now reordered the fkeys again, this time in forwards order.
- 08:02 AM Revision 10467: planning/timeline/timeline.2013.xls: updated for July progress
- 07:43 AM Revision 10466: schemas/VegCore/VegCore.ERD.mwb: place tables that are absolute within Earth rather than relative to a parent place: prefixed geo- to table name for clarity
- 07:23 AM Revision 10465: schemas/VegCore/VegCore.ERD.mwb: plot, subplot: added hstore extenders (dimensions, coordinates)
- 07:17 AM Revision 10464: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance connectors to be 1:1, optional on subclass
- 07:11 AM Revision 10463: schemas/VegCore/VegCore.ERD.mwb: plot: added shape. bounding_box: changed units to rect, since this just needs a width/height (the x/y coord is the lat/long).
- 07:05 AM Revision 10462: schemas/VegCore/VegCore.ERD.mwb: plot: added footprint_geom_WKT. bounding_box: added units (WKT).
- 06:51 AM Revision 10461: schemas/VegCore/VegCore.ERD.mwb: back-synced from staging copy on vegbiendev to flush out sync changes that it kept trying to re-make
- 06:47 AM Revision 10460: schemas/VegCore/VegCore.ERD.mwb: event: moved method to separate sampling_event subclass
- 06:28 AM Revision 10459: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 06:25 AM Revision 10458: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: inherit from taxon_presence, since this is a type of taxon_presence and it avoids duplicating the taxon_concept field
- 06:11 AM Revision 10457: schemas/VegCore/VegCore.ERD.mwb: added taxon_absence, to avoid including absence observations in the same table as presence observations (which needlessly complicates queries). note that the fkey order now gets set back to forwards whenever a table is changed.
- 06:07 AM Revision 10456: schemas/VegCore/VegCore.ERD.mwb: re-saved. the fkey order is now apparently reversed for recently-changed tables.
07/26/2013
- 11:07 PM Revision 10455: schemas/VegCore/VegCore.ERD.mwb: collector, identified_by: allow multiple parties for these fields, using the new party_list array table
- 10:44 PM Revision 10454: schemas/VegCore/VegCore.ERD.mwb: party arrays: use new party_list array table instead of adding a separate many:many table for each table that uses a party array. this also allows using the party_list ID in a unique constraint, because it is now a first-class field.
- 10:06 PM Revision 10453: schemas/VegCore/VegCore.ERD.mwb: party: added party_list array table
- 09:45 PM Revision 10452: schemas/VegCore/VegCore.ERD.mwb: party: added optional fkey to organization
- 09:32 PM Revision 10451: schemas/VegCore/VegCore.ERD.mwb: geovalidation: renamed lat_long_in_ranks to lat_long_in_place_ranks for clarity
- 09:12 PM Revision 10450: schemas/VegCore/VegCore.ERD.mwb: individual: added tag_history hstore to store custom identity attributes
- 08:39 PM Revision 10449: schemas/VegCore/VegCore.ERD.mwb: taxon_string: documented that to get the parsed_taxon_assertion (TNRS result) for a taxon_string, you would join using the SQL dotpath taxon_string.string<-taxon_assertion(string)::parsed_taxon_assertion[source='TNRS.version'] (see wiki.vegpath.org/SQL_dotpaths). important how-to comments such as this one are now included in the version-controlled MySQL schema file itself, not just the .mwb file and the staging copy on vegbiendev.
- 08:16 PM Revision 10448: bin/my2pg: use s!...!...! when either the regexp or the replacement contains / , to avoid unnecessary \-s
- 08:09 PM Revision 10447: bin/my2pg: commenting out table options: added explanatory comment, because it is not obvious from the regexp what this does
- 08:06 PM Revision 10446: lib/sh/db.sh: mysqldump(): don't use --compatible=postgresql when the table structure is being exported, because this removes the table options (which include the COMMENT attribute). --compatible=postgresql remains on in data-only mode because embedded ` in data cannot easily be distinguished from ` around column names, so ANSI_QUOTES is needed to do the translation to " (and data sections do not contain table options). note that all --compatible modes that offer ANSI_QUOTES unfortunately exclude the table options, and there is no way to run a SQL query to set the SQL mode before beginning the dump, so ANSI_QUOTES translation must be handled by my2pg instead.
- 06:35 PM Revision 10445: bin/my2pg: comment out table options (http://dev.mysql.com/doc/refman/5.5/en/server-sql-mode.html#sqlmode_no_table_options) instead of removing them, because they include table COMMENTs, which contain important metadata such as table definitions. (note that table COMMENTs use a slightly different syntax than column COMMENTs, so the table COMMENTs will not be commented out twice.)
- 06:19 PM Revision 10444: bin/my2pg: comment out COMMENTs instead of removing them so that they will be included in the PostgreSQL translation. COMMENTs contain important metadata about columns, such as definitions and the meanings of integer flag values.
- 05:58 PM Revision 10443: inputs/{.,}*/*.schema.sql: regenerated using the instructions in bin/my2pg. this primarily replaces timestamp with text/*timestamp*/ (to preserve indefinite dates).
- 05:56 PM Revision 10442: bin/my2pg: added instructions for regenerating *.schema.sql whenever this script is changed
- 05:22 PM Revision 10441: bin/my2pg: COMMENT: also match COMMENTs with embedded ', because there will only be one COMMENT per line, so the contents of the COMMENT can just extend to the last ' on the line
- 05:16 PM Revision 10440: bugfix: lib/sh/util.sh: $sed_cmd: make output unbuffered, so that running e.g. bin/my2pg at the command line produces output as each line is read
- 04:29 PM Revision 10439: bin/my2pg: replace MySQL ` quotes with " quotes to support exports that were generated without ANSI_QUOTES mode. (this replacement only applies to schema exports, not data.) ANSI_QUOTES is only available with mysqldump --compatible modes that also include NO_TABLE_OPTIONS, which omits important table options such as comments. in particular, these comments are part of schemas/VegCore/VegCore.ERD.mwb but were not being included in VegCore.my.sql.
- 01:41 PM Revision 10438: schemas/VegCore/VegCore.ERD.mwb: taxon_string: removed parsed_taxon_assertion field, since there may be more than one parsing (TNRS result) for a given taxon_string. the parsing relationship can better be represented by adding a parsed_taxon_assertion whose taxon_assertion.string points to the parsed taxon_string. getting the parsed_taxon_assertion for a taxon_string now requires joining on parsed_taxon_assertion using a backwards instead of forwards fkey, and filtering the corresponding assertions to include only the ones for TNRS (of the desired TNRS version). documented that taxon_assertion.string was previously the concatenated matched name, but is now the TNRS input name. the concatenated matched name is still in parsed_taxon_assertion.matched_taxon_concept->:taxon_name.unique_name.
- 01:22 PM Revision 10437: schemas/VegCore/VegCore.my.sql: regenerated from .mwb schema, which apparently reverses the order of the fkeys (possibly a Linux MySQL bug?)
- 12:26 PM Revision 10436: inputs/SpeciesLink/Specimen/map.csv: remapped Darwin Core synonyms to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic _alts to perform to resolve filter-less collisions. the svn diff was verified by replacing DUPLICATE#of:dwc_terms_<term>#... with <term>, removing the comment, and checking that this removes the diff (except where VegCore has renamed a DwC term).
- 12:17 PM Revision 10435: bugfix: inputs/SpeciesLink/Specimen/map.csv: *scientificName: remapped to scientificName instead of taxonName to match the DwC term's name (this is the same dwc_terms_scientificName mismapping that was fixed in r10434)
- 11:56 AM Revision 10434: bugfix: inputs/SpeciesLink/Specimen/map.csv: dwc_terms_scientificName: remapped to scientificName instead of taxonName to match that DwC term name, as well as the mappings of other *scientificName terms
- 11:06 AM Revision 10433: inputs/SpeciesLink/Specimen/map.csv: marked dwc_geospatial_VerbatimLatitude,Longitude as exact duplicates of dwc_terms_*
- 10:52 AM Revision 10432: inputs/SpeciesLink/Specimen/map.csv: remapped identical _alt-ed fields to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic _alts to perform to resolve filter-less collisions.
- 10:06 AM Revision 10431: bugfix: inputs/SpeciesLink/Specimen/map.csv: *CollectorNumber: moved these to the same _alt group as recordNumber, because they are actually duplicates
- 09:43 AM Revision 10430: correction: inputs/SpeciesLink/Specimen/map.csv: *FieldNumber: fixed incorrect comment that these fields are identical to recordNumber, when instead they have the same *meaning* but not the same values. instead, values are stored under *either* of the two terms. the previous conclusion had been based on an incorrect query, which used != instead of the NULL-sensitive IS NOT DISTINCT FROM.
07/25/2013
- 08:14 PM Revision 10429: planning/timeline/timeline.2013.xls: Adding derived columns: extended to overlap with all subtasks
- 08:12 PM Revision 10428: planning/timeline/timeline.2013.xls: Geoscrubbing: split into separate re-run and automated pipeline tasks
- 08:09 PM Revision 10427: planning/timeline/timeline.2013.xls: moved Data provider validations before Adding derived columns because ensuring that the source data is in the database is more important than the derived data, which can always be added later
- 08:00 PM Revision 10426: planning/timeline/timeline.2013.xls: Data provider validations: added dot in July because some amount of datasource-level validation happens when mappings issues are discovered during the refactoring
- 07:34 PM Revision 10425: bugfix: inputs/*/*/map.csv for specimen tables: remapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself rather than to any parent event (specimens don't have a parent event)
- 07:34 PM Revision 10424: inputs/*/*/map.csv for IndividualObservation tables: also mapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself in addition to any parent event which it may be a part of
- 06:27 PM Revision 10423: bugfix: inputs/XAL/Specimen/, NY/Ecatalog_all/: *JulianDay: remapped to dayOfYear instead of day (the day of the month)
- 05:08 PM Revision 10422: inputs/SpeciesLink/Specimen/map.csv: remapped *dayOfYear-related terms to UNUSED
- 04:53 PM Revision 10421: bugfix: inputs/SpeciesLink/Specimen/map.csv: remapped conceptual_darwin_2003_1_0_JulianDay, dwc_dwcore_DayOfYear to dayOfYear instead of day (the day of the month)
- 04:43 PM Revision 10420: mappings/VegCore.htm: regenerated from wiki. added dayOfYear (=julianDay), which is different from startDayOfYear/endDayOfYear.
- 01:59 PM Revision 10419: inputs/CTFS/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 01:50 PM Revision 10418: inputs/CTFS/StemObservation/: translated collisions (missing filters) to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 10:57 AM Revision 10417: planning/timeline/timeline.2013.xls: rebalanced tasks across the remaining months, taking into account priority changes made in the conference call (e.g. that we should not be handling people's individual data requests (Brad, wiki.vegpath.org/2013-07-25_conference_call#Decisions-made))
- 10:50 AM Revision 10416: planning/timeline/timeline.2013.xls: updated with additional tasks added in conference call: translate source-specific derived columns to plain SQL, flatten the datasources, automated geoscrubbing pipeline
- 08:43 AM Revision 10415: planning/goals/BIEN_3_derived_data_products_NormalizedDB_only.docx: removed BIEN species-level phylogeny, which Brad says is out of scope for the BIEN DB
- 08:24 AM Revision 10414: removed planning/workflow/bien3_architecture.odp because the current version is now in bien3_architecture.pptx
- 08:13 AM Revision 10413: added planning/workflow/validation/TNRS_results.ppt symlink to inputs/test_taxonomic_names/_scrub/TNRS_results.ppt
- 08:10 AM Revision 10412: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: highlighted the sample row and related rows
- 08:04 AM Revision 10411: inputs/test_taxonomic_names/_scrub/TNRS_results.xls: moved arrows to TNRS_results.ppt so they can be changed more easily
- 07:51 AM Revision 10410: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: TNRS.tnrs: added diagram labels for the various names and steps
- 07:32 AM Revision 10409: inputs/test_taxonomic_names/_scrub/TNRS_results.xls: use "Poa annua var. eriolepis"->"Poaceae Poa annua L." as the synonym example instead of "Poa annua fo. lanuginosa"->"Poaceae Poa annua var. annua" because the input name is simpler and it's closer to the beginning of the list
- 07:20 AM Revision 10408: inputs/test_taxonomic_names/_scrub/run: exports/make(): tnrs.csv: include Name_matched instead of Genus_matched+Specific_epithet_matched because this also contains lower ranks, which are used in the TNRS synonymizing
- 07:06 AM Revision 10407: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: added annotations explaining the import steps
- 06:36 AM Revision 10406: added inputs/test_taxonomic_names/_scrub/TNRS_results.ppt, containing the *.png screenshots with tables labeled
- 06:35 AM Revision 10405: added inputs/test_taxonomic_names/_scrub/*.png, screenshots of the TNRS_results.xls tabs (LibreOffice does not preserve the formatting when pasting a spreadsheet to a PowerPoint as a table, and the table editing options are limited)
- 06:31 AM Revision 10404: added inputs/test_taxonomic_names/_scrub/TNRS_results.xls with formatted versions of the *.csv tables
Also available in: Atom