Project

General

Profile

Activity

From 09/18/2013 to 10/17/2013

10/17/2013

11:59 PM Revision 11317: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
10:32 PM Revision 11316: planning/timeline/timeline.2013.xls: "flatten the datasources to a common schema": moved to later column because the complex tasks "switching to new-style import" and "create interactive scripts for each import step" are also scheduled then. (it's unlikely we would have much time over winter break anyway, considering that there is ~1 week's worth of holidays then.)
Aaron Marcuse-Kubitza
10:24 PM Revision 11315: planning/timeline/timeline.2013.xls: scheduled "simplify import process for easier maintainability"
Aaron Marcuse-Kubitza
10:20 PM Revision 11314: planning/timeline/timeline.2013.xls: tasks performed by someone else (geoscrubbing re-run): changed solid check marks ✓ to open check marks ☑ to match the solid • vs. open ◦ dot convention
Aaron Marcuse-Kubitza
10:17 PM Revision 11313: planning/timeline/timeline.2013.xls: documentation testing: added supertask dots. removed later dots for scheduled tasks.
Aaron Marcuse-Kubitza
10:15 PM Revision 11312: planning/timeline/timeline.2013.xls: scheduled "documentation testing"
Aaron Marcuse-Kubitza
10:02 PM Revision 11311: planning/timeline/timeline.2013.xls: scheduled "simplify process of mapping/adding a new datasource"
Aaron Marcuse-Kubitza
10:01 PM Revision 11310: planning/timeline/timeline.2013.xls: "add globally-unique occurrenceID": moved it up to the first week when we're no longer fixing existing issues in datasources, since this has similar priority to adding missing columns discovered during usability testing (which is scheduled as an ongoing task)
Aaron Marcuse-Kubitza
09:54 PM Revision 11309: planning/timeline/timeline.2013.xls: usability testing: did task breakdown (find scientists who want to use BIEN3 data, etc.) and scheduled subtasks
Aaron Marcuse-Kubitza
09:34 PM Revision 11308: planning/timeline/timeline.2013.xls: moved "add missing columns" to its own supertask. used outline check mark ☑ (analogous to open circle ◦) to mark supertasks as completed which were split up into subtasks.
Aaron Marcuse-Kubitza
09:29 PM Revision 11307: planning/timeline/timeline.2013.xls: later column: removed dots from scheduled items
Aaron Marcuse-Kubitza
09:06 PM Revision 11306: planning/timeline/timeline.2013.xls: moved "switching to new-style import"-related steps (other than for CVS) to separate "simplify import process for easier maintainability" supertask, since this is not part of the "simplify process of mapping/adding a new datasource" task
Aaron Marcuse-Kubitza
08:39 PM Revision 11305: planning/timeline/timeline.2013.xls: add any missing columns: added and scheduled step to add globally-unique occurrenceID
Aaron Marcuse-Kubitza
08:34 PM Revision 11304: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added dots ◦ for this for the time when it can be worked on asynchronously by Paul Sarando
Aaron Marcuse-Kubitza
08:30 PM Revision 11303: planning/timeline/timeline.2013.xls: data provider metadata: added dots ◦ for the portion of "attribution and conditions of use" that can be worked on asynchronously by Brad/Brian/Bob
Aaron Marcuse-Kubitza
08:24 PM Revision 11302: planning/timeline/timeline.2013.xls: scheduled "aggregated validations" during the last 2 weeks of "datasource validations (spot-checking)", because these weeks are only spent fixing issues uncovered in the remaining datasources, so there may be extra time then
Aaron Marcuse-Kubitza
07:20 PM Revision 11301: planning/timeline/timeline.2013.xls: scheduled other tasks after "datasource validations (spot-checking)" is complete
Aaron Marcuse-Kubitza
07:14 PM Revision 11300: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): each datasource's validation supertask: added open circles ◦ spanning the length of the subtasks
Aaron Marcuse-Kubitza
07:12 PM Revision 11299: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for supertasks that have been fully split into subtasks (not just itemizing a few subtasks), so that these don't count towards the bullets (estimated workload) in each week
Aaron Marcuse-Kubitza
07:09 PM Revision 11298: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for tasks that are performed by someone other than me, so that these don't count towards the bullets (estimated workload) in each week
Aaron Marcuse-Kubitza
07:04 PM Revision 11297: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): split each datasource into subtasks and scheduled them
Aaron Marcuse-Kubitza
06:00 PM Revision 11296: planning/timeline/timeline.2013.xls: moved "move denormalized validations to stage II", "move stage III validations to stage II" outside of "switching to new-style import" because the "switching to new-style import" step refers just to the per-datasource switching steps, not to the additional refactorings that would be needed to avoid dependency on the complex XPath mappings (mappings/VegCore-VegBIEN.csv)
Aaron Marcuse-Kubitza
05:57 PM Revision 11295: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): added subtasks for each of the remaining datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order)
Aaron Marcuse-Kubitza
05:43 PM Revision 11294: planning/timeline/timeline.2013.xls: moved non-validation-related tasks after the 10/31 deadline so that these are not taking time away from the validation
Aaron Marcuse-Kubitza
05:40 PM Revision 11293: planning/timeline/timeline.2013.xls: moved "flatten the datasources to a common schema" under "simplify process of mapping/adding a new datasource" because this is also needed separately for datasources where the left-joining is not part of the validation
Aaron Marcuse-Kubitza
05:30 PM Revision 11292: planning/timeline/timeline.2013.xls: extended "revisions to VegBIEN schema" to length of "datasource validations (spot-checking)" because schema changes are expected as we add missing fields
Aaron Marcuse-Kubitza
05:23 PM Revision 11291: planning/timeline/timeline.2013.xls: crossed out and hid completed tasks ("find out amount remaining in BIEN3 budget")
Aaron Marcuse-Kubitza
05:17 PM Revision 11290: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): extended through the end of November because data providers' fixes on the remaining 10 datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order) are likely to add significantly to the issues and feature requests associated with these datasources (e.g. the 2nd-round VegBank validation added 4 issues and 5 feature requests). there is also expected to be wait time while data providers are responding (most likely in multiple rounds of feedback).
Aaron Marcuse-Kubitza
04:59 PM Revision 11289: planning/timeline/timeline.2013.xls: data provider metadata: removed "iPlant can do" because this actually requires Brad/Brian/Bob/other data providers to provide this info. however, this info may be findable on the web for some datasources.
Aaron Marcuse-Kubitza
04:53 PM Revision 11288: planning/timeline/timeline.2013.xls: moved "data provider metadata" right after "datasource validations" because this is part of the completed database itself rather than the tools to maintain it
Aaron Marcuse-Kubitza
04:47 PM Revision 11287: planning/timeline/timeline.2013.xls: split "revisions to schema" into "revisions to VegBIEN schema" (part of datasource validations) and "revisions to normalized VegCore" (part of documentation)
Aaron Marcuse-Kubitza
04:44 PM Revision 11286: bin/import_all: use just import_scrub, not reimport_scrub, because import_scrub now automatically publishes the datasource's import (i.e. removes the temp suffix)
Aaron Marcuse-Kubitza
04:43 PM Revision 11285: bugfix: inputs/input.Makefile: import: remove the temp suffix once the import is done, so that the full database import doesn't keep the suffix attached to the datasources that import_all didn't import with reimport. removed unused import_publish target (instead use import_temp to invoke just the import without the temp suffix removal).
Aaron Marcuse-Kubitza
04:27 PM Revision 11284: planning/timeline/timeline.2013.xls: moved part of "switching to new-style import" under "datasource validations (spot-checking)" because this is necessary to validate CVS
Aaron Marcuse-Kubitza
04:24 PM Revision 11283: planning/timeline/timeline.2013.xls: moved "simplify process of mapping/adding a new datasource" and "documentation testing" after "usability testing" because these tasks were there to make it possible for people other than me to reload/add to the database, which we have now decided is a lower priority than creating the validated database itself
Aaron Marcuse-Kubitza
04:14 PM Revision 11282: planning/timeline/timeline.2013.xls: added weeks through the end of the year (12/31)
Aaron Marcuse-Kubitza
02:00 PM Revision 11281: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: changed dataset definition to the definition in normalized VegCore ("a collection of records from the same place, with the same attribution requirements"), following discussion with Ramona
Aaron Marcuse-Kubitza
01:13 PM Revision 11280: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
12:34 PM Revision 11279: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: updated to Ramona's commented version
Aaron Marcuse-Kubitza
12:19 AM Revision 11278: inputs/CVS/plot_/map.csv: realLatitude, realLongitude: remapped to UNUSED because these columns are actually empty
Aaron Marcuse-Kubitza

10/13/2013

07:32 PM Revision 11277: inputs/CVS/taxonObservation_/map.csv: collector_ID: remapped it to UNUSED and removed the join to party via it, like in VegBank
Aaron Marcuse-Kubitza
07:03 PM Revision 11276: inputs/CVS/: deleted stemLocation_, because the CVS stemLocation table is empty (unlike VegBank)
Aaron Marcuse-Kubitza
05:26 PM Revision 11275: inputs/CVS/import_order.txt: added plantConcept_/ so it would get automapped after switching to new-style import
Aaron Marcuse-Kubitza
04:24 PM Revision 11274: inputs/CVS/taxonObservation_/map.csv: denorm_{tri,quad}*: mapped to infraspecificRank*, infraspecificEpithet*
Aaron Marcuse-Kubitza
04:02 PM Revision 11273: inputs/CVS/taxonObservation_/map.csv: infraspecific ranks: remapped to EQUIV#to:species (which is the speciesBinomial), because these actually contain the full taxonomic name at that rank, like VegBank
Aaron Marcuse-Kubitza
03:50 PM Revision 11272: inputs/CVS/taxonObservation_/map.csv: genus: documented that unlike VegBank, does *not* include genus author
Aaron Marcuse-Kubitza
03:47 PM Revision 11271: inputs/CVS/taxonObservation_/map.csv: denorm_* terms _alt-ed with normalized terms: use DUPLICATE#of instead where possible. documented where and why _alt was necessary (this applies to a few rows for division, genus).
Aaron Marcuse-Kubitza
03:42 PM Revision 11270: bugfix: inputs/CVS/taxonObservation_/map.csv: species: remapped to speciesBinomial, not specificEpithet (like for VegBank). however, note that denorm_species *is* in fact the epithet, unlike VegBank.
Aaron Marcuse-Kubitza
03:19 PM Revision 11269: fix: inputs/CVS/taxonObservation_/postprocess.sql: removed {} around denorm_genus to match the normalized genus
Aaron Marcuse-Kubitza
02:28 PM Revision 11268: inputs/CVS/taxonObservation_/map.csv: removed unnecessary _alts for terms that don't have a duplicate denorm_* or hierarchical field
Aaron Marcuse-Kubitza
02:06 PM Revision 11267: fix: inputs/CVS/taxonObservation_/postprocess.sql: fix 1 row that has denorm_kingdom != Kingdom (i.e. both NOT NULL but not the same)
Aaron Marcuse-Kubitza
12:46 PM Revision 11266: bugfix: lib/common.Makefile: $(subMake): don't enclose the target in "" because sometimes the target is empty (i.e. `all`), and nothing should be passed to the sub-make
Aaron Marcuse-Kubitza
12:10 AM Revision 11265: inputs/VegBank/plot_/create.sql: documented runtime (5 min)
Aaron Marcuse-Kubitza
12:10 AM Revision 11264: bugfix: inputs/CVS/plot_/create.sql: like for VegBank, need to compare place.*PLOT_ID*, not PLOTPLACE_ID, with plot.PLOT_ID
Aaron Marcuse-Kubitza
12:02 AM Revision 11263: /README.TXT: Single datasource import: added pointer to instructions to remake the analytical DB (also required after single datasource import)
Aaron Marcuse-Kubitza

10/12/2013

04:59 PM Revision 11262: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
04:20 PM Revision 11261: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
Aaron Marcuse-Kubitza
04:16 PM Revision 11260: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
Aaron Marcuse-Kubitza
03:49 PM Revision 11259: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: run all sync_uploads on the svn working copy using --size-only, because the mtimes are based on when the files were last updated by svn and are not meaningful
Aaron Marcuse-Kubitza
03:46 PM Revision 11258: /README.TXT: Full database import: On local machine: do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine": removed no longer accurate indicator that these steps are above Full database import, since Full database import is now at the beginning of the file
Aaron Marcuse-Kubitza
03:05 PM Revision 11257: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
Aaron Marcuse-Kubitza
03:03 PM Revision 11256: bugfix: inputs/VegBank/+taxon_observation.**.sample/: renamed to ^taxon_observation.**.sample because a leading + has a special meaning to bash (it indicates a shell option, and you will get an error "invalid option name"), as well as to make (it indicates that a recipe command invokes make recursively)
Aaron Marcuse-Kubitza
02:27 PM Task #767 (New): autodetect column mismatches
* if @map.csv@ does not contain the @header.csv@ columns in exactly the right order, this will mess up the import in ... Aaron Marcuse-Kubitza
02:14 PM Revision 11255: bugfix: inputs/VegBank/taxon_observation.**/header.csv: updated for observation_/map.csv bugfix, which added new hasobservationsynonym field. this fixes a strange test bug caused by the taxon_observation.**/map.csv column list being mismatched/misaligned with what was in the underlying tables. (column mismatches will often cause unexplainable errors in unrelated sections of code the same way that buffer overflows do in C++.)
Aaron Marcuse-Kubitza
02:01 PM Revision 11254: bugfix: inputs/VegBank/taxon_observation.**.sample/: renamed to +taxon_observation.**.sample so that the *-expansion of taxon_observation.** doesn't add taxon_observation.**.sample (which causes it to attempt to install taxon_observation.**.sample before taxon_observation.** is installed)
Aaron Marcuse-Kubitza
12:48 PM Revision 11253: bugfix: *Makefile: recursive invocation of $(MAKE): enclose targets in "" in case they contain *
Aaron Marcuse-Kubitza
12:12 PM Revision 11252: bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable
Aaron Marcuse-Kubitza
12:11 PM Revision 11251: bugfix: inputs/input.Makefile: %/uninstall: allow user to set is_view=1 flag to use DROP VIEW instead of DROP TABLE
Aaron Marcuse-Kubitza
12:08 PM Revision 11250: lib/sh/util.sh: added instructions for making an export only visible locally
Aaron Marcuse-Kubitza

10/10/2013

06:50 PM Revision 11249: bugfix: inputs/VegBank/observation_/header.csv, map.csv: updated for refresh, which inserts hasobservationsynonym at the end of the observation table
Aaron Marcuse-Kubitza
05:46 PM Revision 11248: inputs/VegBank/taxon_observation.**.sample/create.sql: reordered columns in the same order as analytical_plot, for easier validation
Aaron Marcuse-Kubitza
05:46 PM Revision 11247: bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed
Aaron Marcuse-Kubitza
03:07 PM Revision 11246: web/links/index.htm: updated to Firefox bookmarks. updated favicons.
Aaron Marcuse-Kubitza
02:57 PM Revision 11245: web/links/index.htm: updated to Firefox bookmarks. sudo: added instructions to turn off incorrect password e-mails.
Aaron Marcuse-Kubitza
02:40 PM Revision 11244: inputs/VegBank/taxon_observation.**.sample/create.sql: include only the subset of columns that is imported to VegBIEN
Aaron Marcuse-Kubitza
02:32 PM Revision 11243: inputs/VegBank/taxon_observation.**.sample/test.xml.ref: updated inserted row count (which was most likely generated before the output column names had been set to the input column names)
Aaron Marcuse-Kubitza
01:55 PM Revision 11242: added inputs/VegBank/verify/input_cols.include.txt, with runscript to generate it
Aaron Marcuse-Kubitza
01:26 PM Revision 11241: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
Aaron Marcuse-Kubitza
01:18 PM Revision 11240: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
Aaron Marcuse-Kubitza
01:11 PM Revision 11239: inputs/VegBank/verify/input_cols.unmapped.txt.run: remove unmapped join columns, since these *would* be included in the extract
Aaron Marcuse-Kubitza
01:09 PM Revision 11238: inputs/VegBank/verify/input_cols.unmapped.txt.run: take input directly from input_cols.txt to avoid needing to first copy and paste it into input_cols.unmapped.txt
Aaron Marcuse-Kubitza
01:03 PM Revision 11237: inputs/VegBank/verify/input_cols.unmapped.txt.run: added back deliberately excluded columns (DUPLICATE#of:..., etc.) so that the # of rows in the file can be subtracted from the total # of columns to get the # of input columns that would be included in the extract
Aaron Marcuse-Kubitza
12:43 PM Revision 11236: bugfix: inputs/input.Makefile: %/VegBIEN.csv: `ln -s` to create VegBIEN.csv: enclose the filenames in "" since they may contain * (e.g. taxon_observation.**)
Aaron Marcuse-Kubitza
12:23 PM Revision 11235: added inputs/VegBank/verify/input_cols.txt, input_cols.unmapped.txt (with runscript to filter input_cols.unmapped.txt)
Aaron Marcuse-Kubitza
11:53 AM Revision 11234: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: made Ramona's corrections with track changes turned on. note that you have to use MS Word for this, not LibreOffice, because LibreOffice can't save the table of contents properly in .docx or .doc format (although it can save it in .odt format).
Aaron Marcuse-Kubitza
08:18 AM Revision 11233: inputs/VegBank/stratum/postprocess.sql: added pkey
Aaron Marcuse-Kubitza
08:05 AM Revision 11232: inputs/VegBank/taxonobservation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
Aaron Marcuse-Kubitza
07:54 AM Revision 11231: inputs/VegBank/observation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
Aaron Marcuse-Kubitza
07:45 AM Revision 11230: inputs/VegBank/import_order.txt: added taxon_observation.**.sample so it will automatically be kept up to date
Aaron Marcuse-Kubitza
07:32 AM Revision 11229: inputs/VegBank/taxon_observation.**.sample/create.sql: set runtime (1 s)
Aaron Marcuse-Kubitza
07:30 AM Revision 11228: inputs/VegBank/: added taxon_observation.**.sample subset of plots to use in the validation. this avoids the need to import all of VegBank just to validate a few of the plots.
Aaron Marcuse-Kubitza

10/09/2013

10:12 PM Revision 11227: bin/map: usage: documented that verbosity > 3 in commit mode turns on debug_temp mode, which creates real tables instead of temp tables
Aaron Marcuse-Kubitza
09:41 PM Revision 11226: schemas/VegCore/ERD/VegCore.ERD.mwb: georeferencing: documented that this stores location determinations (= VegBIEN.vegpath.org?public.locationplace)
Aaron Marcuse-Kubitza
06:31 PM Revision 11225: inputs/VegBank/taxon_observation.**/: updated for data refresh
Aaron Marcuse-Kubitza
06:25 PM Revision 11224: inputs/VegBank/plantconcept_/: mapped columns, since this is now included in import_order.txt and therefore gets processed by the column-renaming runscripts. note that this means that in taxonobservation_/map.csv, the plantconcept_ input column names need to be changed to what they are mapped to.
Aaron Marcuse-Kubitza
06:16 PM Revision 11223: inputs/VegBank/taxonobservation_/create.sql: updated runtime (20 s)
Aaron Marcuse-Kubitza
05:54 PM Revision 11222: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
05:51 PM Revision 11221: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: project: allow it to *have* a dataset rather than *being* one, because projects do not always have separate attribution requirements (e.g. in Madidi they don't). this also fixes a bug where projects whose datasets had a parent dataset (e.g. VegBank) would need a dummy event for that parent dataset so that the fkeys on both dataset.parent->dataset *and* event.parent->event would be satisfied. added project__unique constraint on name within dataset.
Aaron Marcuse-Kubitza
05:38 PM Revision 11220: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event.parent (inherited): require this to be a taxa_sampling_event. documented that this is the taxa_sampling_event for the parent stratum.
Aaron Marcuse-Kubitza
05:15 PM Revision 11219: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
05:03 PM Revision 11218: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
05:00 PM Revision 11217: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
04:55 PM Revision 11216: schemas/VegCore/ERD/VegCore.ERD.mwb: place_visit.parent (inherited): added fkey constraint that this must be a place_visit. documented that this is the concurrent observation of the parent place.
Aaron Marcuse-Kubitza
04:47 PM Revision 11215: schemas/VegCore/ERD/VegCore.ERD.mwb: project: inherit from dataset, since projects generally confer attribution and conditions of use
Aaron Marcuse-Kubitza
04:26 PM Revision 11214: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: place_visit: fixed synonyms list to "VegX.vegpath.org?plotObservation, VegBank.vegpath.org?observation (which was confusingly named) + some of VegBank.vegpath.org?plot (which is actually 1:1 with observation)"
Aaron Marcuse-Kubitza
04:22 PM Revision 11213: schemas/VegCore/ERD/VegCore.ERD.mwb: place_observation: renamed to place_visit for clarity, since other tables that point to this use it as a general container, not just a link to the observations about the place
Aaron Marcuse-Kubitza
04:16 PM Revision 11212: schemas/VegCore/ERD/VegCore.ERD.mwb: merged stratum_event into taxa_sampling_event since they serve the same purpose (separating taxa samplings within the same place_observation by subsetting methodology). merged layer into stratum since strata as we think of them are horizontal, although there are other definitions of stratum (e.g. size class) which are more similar to our method table.
Aaron Marcuse-Kubitza
03:51 PM Revision 11211: schemas/VegCore/ERD/VegCore.ERD.mwb: event subclasses: subject: added applicable fkey constraints to the inherited field. renamed existing table-specific subject fields to subject to merge with the inherited field.
Aaron Marcuse-Kubitza
03:38 PM Revision 11210: schemas/VegCore/ERD/VegCore.ERD.mwb: event: added unique constraints for every way that the scoping fields can meaningfully be combined. these correspond to the various unique constraints offered by VegBIEN.locationevent. (the event table's constraints are always tricky because there are many ways to define an event. it is important to include all of them so that there is almost always a way to generate a natural key, without resorting to an artificial key from traceable.id_by_source.)
Aaron Marcuse-Kubitza
03:28 PM Revision 11209: schemas/VegCore/ERD/VegCore.ERD.mwb: event.place: renamed to subject and allow this to be any traceable, not just a place. this allows taxon_observations to be scoped within their taxon_occurrences just as subplot events are scoped within parent plot events.
Aaron Marcuse-Kubitza
02:46 PM Revision 11208: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.authors: documented that this makes traceable mutually recursive with party_list
Aaron Marcuse-Kubitza
02:45 PM Revision 11207: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.authors: documented that this makes traceable mutually recursive with party_list
Aaron Marcuse-Kubitza
02:42 PM Revision 11206: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed inheritance connectors
Aaron Marcuse-Kubitza
02:38 PM Revision 11205: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
02:35 PM Revision 11204: schemas/VegCore/ERD/VegCore.ERD.mwb: record.authors: moved to traceable because actually, different sections of a denormalized source record may have separate authors (e.g. participants for taxa_sampling_event, identifiers for taxon_determination)
Aaron Marcuse-Kubitza
02:24 PM Revision 11203: schemas/VegCore/ERD/VegCore.ERD.mwb: record.dataset: split this into the scoping_dataset (which scopes the id_within_dataset) and the attribution_dataset (which provides attribution information)
Aaron Marcuse-Kubitza
01:55 PM Revision 11202: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
01:45 PM Revision 11201: schemas/VegCore/ERD/VegCore.ERD.mwb: record.authors: changed this to refer *only* to original data creators such as collectors and identifiers. this way, there is no need to generate a combined party_list of owners+authors for every record, because the record-specific authors list can just point to the same party_list as the collectors or event participants. this is also closer to Brad's meaning, which is specifically "a person or institution responsible for *creating* the dataset" [+emph].
Aaron Marcuse-Kubitza
01:31 PM Revision 11200: schemas/VegCore/ERD/VegCore.ERD.mwb: party_list: added array field, which stores an array combining all the list entries. if specified, a trigger will autogenerate party_list_entries from this field. if not specified, it will be autopopulated whenever a party_list_entry is added.
Aaron Marcuse-Kubitza
12:59 PM Revision 11199: schemas/VegCore/ERD/VegCore.ERD.mwb: added data_owner subclass of party_list_entry, which stores use_conditions for each data_owner of a dataset. dataset: added access_rights (public, private, etc.), use_conditions (citation requirements, etc.).
Aaron Marcuse-Kubitza
12:25 PM Revision 11198: schemas/VegCore/ERD/VegCore.ERD.mwb.run: record.authors: clarified that this should include *all* parties who helped create the data record, making it a *superset* of dataset.data_owners, who receive attribution. clarified that *record-specific* authors such as collectors and identifiers generally do not receive attribution, although *dataset-level* authors do (this field includes both).
Aaron Marcuse-Kubitza
12:07 PM Revision 11197: schemas/VegCore/ERD/VegCore.ERD.mwb.run: record.id_within_dataset: removed instruction to also put this in source.name, because source.name is now optional, and this would not make sense because source does not have a parent pointer
Aaron Marcuse-Kubitza
12:03 PM Revision 11196: schemas/VegCore/ERD/VegCore.ERD.mwb.run: documented that schema COMMENTs cannot contain ' (used in contractions such as "don't"), because MySQL Workbench cannot escape these properly
Aaron Marcuse-Kubitza
12:02 PM Revision 11195: schemas/VegCore/ERD/VegCore.ERD.mwb: record: added authors, which stores the people who helped create the data record. this is Brad's data author term (Brad.vegpath.org?attribution.data_author), but actually belongs to the individual data *record* rather than to an entire data*set* because records may have different authors (collectors, identifiers, etc.). note that schema COMMENTs apparently cannot contain ' (used in contractions such as "don't"), because MySQL Workbench cannot escape these properly.
Aaron Marcuse-Kubitza
11:38 AM Revision 11194: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.data_owners: documented that this includes all aggregators through which the data has passed, so that this is equivalent to Brad's data indexers (Brad.vegpath.org?attribution.data_indexer)
Aaron Marcuse-Kubitza
11:31 AM Revision 11193: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.parent: fixed example to indicate that the GBIF.ARIZ dataset is contained within a specific GBIF *dataset* (the parent), but was obtained from the GBIF *organization* (the dataset_source). ARIZ is then the first_publisher, not the dataset_source.
Aaron Marcuse-Kubitza
11:22 AM Revision 11192: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: renamed data_source to dataset_source for clarity
Aaron Marcuse-Kubitza
11:14 AM Revision 11191: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.contacts: documented that this is the contact person(s) to ask about attribution questions. there should always be at least one contact person. if unknown, the person loading the data should set this to themselves.
Aaron Marcuse-Kubitza
11:12 AM Revision 11190: added schemas/VegCore/Brad_Boyle/BIEN 3 data use and attribution.docx symlink to schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx, so that all of Brad's attribution-related terms are in one place
Aaron Marcuse-Kubitza
11:07 AM Revision 11189: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.contacts: make this required, so that anyone adding data to a VegCore DB always provides at least one contact person, possibly themselves, to ask about attribution questions (as suggested by Brad in BIEN 3 data use and attribution.docx)
Aaron Marcuse-Kubitza
11:00 AM Revision 11188: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.first_publisher: allow this to be any kind of source, rather than just a party (i.e. an organization). this is possible now that party inherits from source.
Aaron Marcuse-Kubitza
10:52 AM Revision 11187: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: linked to Brad's synonyms for dataset_source (Brad.vegpath.org?attribution.proximate_data_provider), first_publisher (Brad.vegpath.org?attribution.data_publisher, Brad.vegpath.org?data_provenance.primarydataprovider)
Aaron Marcuse-Kubitza
10:43 AM Revision 11186: bugfix: web/people/Brad_Boyle/.htaccess: only forward requests for a term defined by Brad to the attribution/ subdir, so that Brad.vegpath.org shows a directory listing instead of forwarding right to the attribution doc
Aaron Marcuse-Kubitza
10:33 AM Revision 11185: bugfix: web/people/Brad_Boyle/.htaccess: set %{ENV:innermost_dir} to Brad_Boyle
Aaron Marcuse-Kubitza
10:28 AM Revision 11184: added web/people/Brad_Boyle/data_provenance/ (BIEN database entities related to data provenance and ownership.docx)
Aaron Marcuse-Kubitza
09:56 AM Revision 11183: bugfix: web/people/Brad_Boyle/attribution/.htaccess: \-escape the %s so that they aren't treated as backreferences. set the [redirect] flag so that the downloaded file has the correct filename (BIEN 3 data use and attribution.docx instead of attribution).
Aaron Marcuse-Kubitza
09:49 AM Revision 11182: bugfix: web/people/Brad_Boyle/attribution/.htaccess: use %20 instead of + because the Apache server apparently doesn't translate + to space correctly
Aaron Marcuse-Kubitza
09:47 AM Revision 11181: bugfix: web/people/Brad_Boyle/attribution/.htaccess: link to the VegBIEN subdir in /exchange_schemas instead of the main /VegBIEN because there are actually two VegBIEN subdirs, and the main one instead points to /datasources/VegBIEN
Aaron Marcuse-Kubitza
09:45 AM Revision 11180: added web/people/Brad_Boyle/ with attribution/ subdir for BIEN 3 data use and attribution.docx
Aaron Marcuse-Kubitza
09:15 AM Revision 11179: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: renamed data_source to dataset_source for clarity
Aaron Marcuse-Kubitza
08:54 AM Revision 11178: inputs/VegBank/plantconcept_/create.sql: documented runtime (21 min)
Aaron Marcuse-Kubitza
08:28 AM Revision 11177: bugfix: inputs/VegBank/plantconcept_/: added new-style import files
Aaron Marcuse-Kubitza
08:27 AM Revision 11176: bugfix: inputs/VegBank/import_order.txt: added plantconcept_, because new-style import needs it to be explicitly listed in import_order.txt in order to run it
Aaron Marcuse-Kubitza
08:24 AM Revision 11175: inputs/VegBank/run: refresh(): added usage
Aaron Marcuse-Kubitza
08:22 AM Revision 11174: inputs/VegBank/run: refresh(): documented that this should be run on *vegbiendev*
Aaron Marcuse-Kubitza
07:58 AM Revision 11173: inputs/VegBank/_archive/2012-8-30/: svn:ignore the data exports
Aaron Marcuse-Kubitza
07:57 AM Revision 11172: inputs/VegBank/run: added refresh() target
Aaron Marcuse-Kubitza
07:54 AM Revision 11171: inputs/VegBank/: refreshed VegBank so that all of Mike Lee's sample plots would be included in the input data. (VegBank was last refreshed from the live DB on 2012-8-30.) split vegbank.sql into vegbank.schema.sql and vegbank.data.sql so that the schema can be examined and imported separately, like for MySQL datasources. inputs/VegBank/vegbank.~.clean_up.sql: commented out setting comminterpretation.commname to NOT NULL, because after the refresh it is now NULL in 10 rows, where commconcept_id is also NULL.
Aaron Marcuse-Kubitza

10/08/2013

05:31 PM Revision 11170: lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions
Aaron Marcuse-Kubitza
05:29 PM Revision 11169: lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir
Aaron Marcuse-Kubitza
12:36 PM Revision 11168: inputs/.geoscrub/_src/README.TXT: added e-mail from John Donoghue with general description of the BIEN2 geovalidation workflow
Aaron Marcuse-Kubitza

10/03/2013

04:04 PM Revision 11167: bugfix: schemas/vegbien.sql: plot.**: removed ORDER BY because a SELECT on the view with its own ORDER BY will actually sort the rows *twice* instead of optimizing away plot.**'s default ORDER BY
Aaron Marcuse-Kubitza
03:44 PM Revision 11166: schemas/vegbien.sql: added plot.** subset view of analytical_stem_view, used for VegBank validation
Aaron Marcuse-Kubitza
01:41 PM Revision 11165: planning/timeline/timeline.2013.xls: moved reload core & analytical database out a week because it shouldn't be done until after CVS validation is complete
Aaron Marcuse-Kubitza
01:10 PM Task #294 (Resolved): find plot data source provider to work with Aaron
Aaron Marcuse-Kubitza
01:10 PM Task #297 (Resolved): Request new data dump of specimen data from GBIF, this time in DwC format
Aaron Marcuse-Kubitza
01:10 PM Task #301 (Resolved): RAINFOR data
part of SALVIAS Aaron Marcuse-Kubitza
01:09 PM Task #305 (Resolved): Complete full-dataset validations for NYBG & SALVIAS
Aaron Marcuse-Kubitza
01:09 PM Task #306 (Resolved): Acquire additional plot data sets from providers willing to work with Aaron on mappings and validations
Aaron Marcuse-Kubitza
01:09 PM Task #307 (Resolved): Acquire additional specimen data sets in both DwC and DwCA format, esp. GBIF
Aaron Marcuse-Kubitza
01:09 PM Task #312 (Resolved): Finish importing SALVIAS data
Aaron Marcuse-Kubitza
01:08 PM Task #320 (Rejected): Convert user-defined VegX fields to first-class fields
using [[VegCore]] instead Aaron Marcuse-Kubitza
01:08 PM Task #324 (Resolved): NYBG validations
Aaron Marcuse-Kubitza
01:07 PM Task #333 (Resolved): add traits table to VegBIEN
Aaron Marcuse-Kubitza
01:07 PM Task #344 (Resolved): contact John about GNRS/georeferencing
Aaron Marcuse-Kubitza
01:07 PM Task #345: integrate GNRS into VegBIEN
prototyped in [[VegCore]] Aaron Marcuse-Kubitza
01:06 PM Task #346: add georeferencing support to schema
prototyped in [[VegCore]] Aaron Marcuse-Kubitza
01:06 PM Task #377 (Resolved): ask NYBG for direct access to server
Aaron Marcuse-Kubitza
01:03 PM Task #466 (Rejected): column-based import: avoid most exceptions by introspecting the DB structure instead of waiting for a parseable exception to be raised
easier to handle exceptions than trying to anticipate them Aaron Marcuse-Kubitza
01:02 PM Task #467: underline all to do items in meetings notes
completed items have been crossed out; remaining tasks can then be underlined Aaron Marcuse-Kubitza
01:01 PM Task #482: translate README.TXT to wiki page
see [[Import process for Madidi]] Aaron Marcuse-Kubitza
01:00 PM Task #544: integrate creation of analytical DB into automated testing
it's part of the extract creation for datasource validations Aaron Marcuse-Kubitza
01:00 PM Task #560: move VegCore data dictionary to a phpPgAdmin-accessible database
see "VegCore ERD":http://vegbiendev.nceas.ucsb.edu/VegCore/ERD/ Aaron Marcuse-Kubitza
12:58 PM Task #562 (Rejected): flatten the mappings
autopopulated natural pkeys actually do require the nested mappings, but these can be part of the schema instead Aaron Marcuse-Kubitza
12:56 PM Task #577: use views instead of map spreadsheets to store the datasource mappings
map spreadsheets copied to tables in [[new-style import]] Aaron Marcuse-Kubitza
12:25 PM Revision 11164: inputs/.geoscrub/_src/README.TXT: added link to geovalidation description in wiki
Aaron Marcuse-Kubitza
11:03 AM Revision 11163: added schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx from Brad
Aaron Marcuse-Kubitza
10:03 AM Revision 11162: planning/timeline/timeline.2013.xls: updated to reflect that the next week should be spent doing data validations, as decided in the conference call
Aaron Marcuse-Kubitza

10/02/2013

05:59 AM Revision 11161: bugfix: schemas/vegbien.sql: locationevent__communities(): need to use commdescription instead of commname as the communityName because that's where communityName has been mapped to in mappings/VegCore-VegBIEN.csv. (commname instead stores the communityID.)
Aaron Marcuse-Kubitza
05:56 AM Revision 11160: inputs/VegBank/: added observation__community/
Aaron Marcuse-Kubitza
05:49 AM Revision 11159: inputs/VegBank/vegbank.~.clean_up.sql: commclass.commcode,commname: rename to prevent collisions
Aaron Marcuse-Kubitza
04:26 AM Revision 11158: inputs/VegBank/vegbank.~.clean_up.sql: indicate required column comminterpretation.commname
Aaron Marcuse-Kubitza
04:20 AM Revision 11157: inputs/VegBank/vegbank.~.clean_up.sql: commconcept.commname: rename to prevent collision with commname.commname
Aaron Marcuse-Kubitza
04:15 AM Revision 11156: planning/timeline/: svn:ignore: added .~* (lock files for .xls)
Aaron Marcuse-Kubitza
03:58 AM Revision 11155: schemas/vegbien.sql: analytical_stem_view: added instructions for what to do after updating this
Aaron Marcuse-Kubitza
03:55 AM Revision 11154: schemas/vegbien.sql: analytical_stem_view, analytical_plot: added communities using new locationevent__communities__array()
Aaron Marcuse-Kubitza
03:52 AM Revision 11153: schemas/vegbien.sql: added locationevent__communities*()
Aaron Marcuse-Kubitza
03:10 AM Revision 11152: bugfix: schemas/vegbien.sql: locationevent: locationevent__pull_forward_from_parent() trigger: only pull forward the location_id when there is a stratum_id, so that empty locationevents that should be auto-forwarded are not unintentionally made valid by populating the location_id field (which causes a check constraint to become satisfied, and prevents auto-forwarding from working). this fixes a bug in some datasources' automated tests, such as `make inputs/SALVIAS/plotObservations/test`.
Aaron Marcuse-Kubitza
02:45 AM Revision 11151: lib/sql_io.py: put_table(): default param: documented that this will be used for *all* missing rows, regardless of which error caused them not to be inserted. this means that auto-forwarding (wiki.vegpath.org/Auto-forwarding) can be used with *any* type of constraint violation, not just NOT NULL constraints (which it is typically used with).
Aaron Marcuse-Kubitza

10/01/2013

10:57 PM Revision 11150: /Makefile: mk_db: config/users.sql: unset ON_ERROR_STOP so user-exists errors don't prevent further user-adds
Aaron Marcuse-Kubitza

09/30/2013

07:47 AM Revision 11149: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
07:38 AM Revision 11148: bugfix: schemas/vegbien.sql: analytical_stem_view: plot__collectors: to include plot__collectors for nested stratum events, need to calculate the locationevent__contributors__array() for the locationevent and parent_event *separately*, and then COALESCE those, instead of just running locationevent__contributors__array() once on the COALESCE of the two IDs
Aaron Marcuse-Kubitza
07:34 AM Revision 11147: bugfix: schemas/vegbien.sql: locationevent: added pull-forward trigger to populate location_id from the parent event. a pull-forward trigger *must* be used for this (not a COALESCE() in analytical_stem_view) because the locationevent cannot even be found by analytical_stem_view if it does not have an associated location.
Aaron Marcuse-Kubitza
06:37 AM Revision 11146: schemas/VegCore/ERD/VegCore.ERD.mwb: source: added required url field, which points to the source data and uniquely identifies the source. note that this differs from the id, which is a *self*-refential pkey (as described at wiki.vegpath.org/Web_interface#Clickable-fkeys).
Aaron Marcuse-Kubitza
05:47 AM Revision 11145: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
05:38 AM Revision 11144: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
05:32 AM Revision 11143: schemas/VegCore/ERD/VegCore.ERD.mwb: source: removed canon, because this would create too much complexity as a result of having multiple copies of the same source with different names. instead, source names will be scrubbed *upon insert* so that the row always contains the most accurate data, and there is only one copy of each source. this scrubbing may use the assistance of a thesaurus table w/ standard abbreviations, etc.
Aaron Marcuse-Kubitza
05:07 AM Revision 11142: schemas/VegCore/ERD/VegCore.ERD.mwb: plot.name: merged with inherited name field
Aaron Marcuse-Kubitza
04:43 AM Revision 11141: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
04:40 AM Revision 11140: schemas/VegCore/ERD/VegCore.ERD.mwb: source: added canon field. when there are multiple names for a source, this points to the canonically-named source. by default, points recursively to itself.
Aaron Marcuse-Kubitza
04:04 AM Revision 11139: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.taxon_occurrence: documented that when not provided (and not fillable from another field), a taxon_occurrence will be created with the same id as the taxon_observation. this operation should eventually be perfomed by a trigger.
Aaron Marcuse-Kubitza
03:58 AM Revision 11138: schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place.parent_geoplace: renamed to parent so it will merge with the inherited parent field. this is the parent geoplace.
Aaron Marcuse-Kubitza
03:27 AM Revision 11137: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
03:20 AM Revision 11136: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
03:18 AM Revision 11135: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that instead, the taxon_occurrence for the individual_observation should have an autogenerated current_observation that merges together all the individual_observations for these institutions' specimens. in this case, each institution's specimen will still have its *own* individual_observation containing what *it* believes to be the specimenholder institutions.
Aaron Marcuse-Kubitza
03:03 AM Revision 11134: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that when provided, the source (i.e. traceable.source) should be set to the value of this field, because these institutions together provided the data for this entity
Aaron Marcuse-Kubitza
02:59 AM Revision 11133: schemas/VegCore/ERD/VegCore.ERD.mwb: party_list: inherit from source so that a party_list can be used as the source for a row (e.g. a list of institutions that together provided the data for an individual_observation that has specimens at multiple institutions)
Aaron Marcuse-Kubitza
02:55 AM Revision 11132: schemas/VegCore/ERD/VegCore.ERD.mwb: reobservable_presence: merged into taxon_determination, so that voucher is directly available to taxon_determination to autopopulate fields when a redetermination is made on the voucher
Aaron Marcuse-Kubitza
02:37 AM Revision 11131: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
02:36 AM Revision 11130: schemas/VegCore/ERD/VegCore.ERD.mwb: aggregate_observation.sampling_event: merged with inherited sampling_event field
Aaron Marcuse-Kubitza
02:33 AM Revision 11129: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.parent: require that this is an fkey to another taxon_observation, since this stores the *taxon_observation* of the same taxon_occurrence in the parent sampling_event
Aaron Marcuse-Kubitza
02:22 AM Revision 11128: schemas/VegCore/ERD/VegCore.ERD.mwb: subplot.parent_plot: renamed to parent so it will merge with the inherited parent field. this is the parent plot.
Aaron Marcuse-Kubitza
02:16 AM Revision 11127: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_count.taxon_presence: renamed to parent so it will merge with the inherited parent field. this is the taxon_presence for the taxon as a whole (not subdivided by size_class, etc.).
Aaron Marcuse-Kubitza
02:13 AM Revision 11126: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event.method: renamed to subsetting_method for clarity
Aaron Marcuse-Kubitza
02:06 AM Revision 11125: schemas/VegCore/ERD/VegCore.ERD.mwb: stratum_event.parent_event: renamed to parent so it will merge with the inherited parent field
Aaron Marcuse-Kubitza
01:58 AM Revision 11124: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable: added alternate pkey id_by_source, which allows referencing a traceable by its source information even when the subclass chose a different value for the pkey
Aaron Marcuse-Kubitza
01:49 AM Revision 11123: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: merged with inherited parent field, documenting that for this table, this contains the observation of the same taxon_occurrence in the parent sampling_event
Aaron Marcuse-Kubitza
01:33 AM Revision 11122: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
01:24 AM Revision 11121: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.place_observed_at: merged with inherited place field, documenting that for this table, this contains the place the individual was observed at
Aaron Marcuse-Kubitza
01:16 AM Revision 11120: schemas/VegCore/ERD/VegCore.ERD.mwb: event: reordered date_range, participants, place in the order of how likely they are to be provided. (place was considered more optional than the other two because a project does not usually have a place.)
Aaron Marcuse-Kubitza
01:09 AM Revision 11119: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.sampling_event: made it required so that there is always a sampling_event to scope the primary_collector/collector_number when provided. (for specimens, this will have the same event information (date_range, participants, place) as the taxon_observation.)
Aaron Marcuse-Kubitza
12:35 AM Revision 11118: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added unique constraint on sampling_event, primary_collector, collector_number. reordered these columns in constraint order (path order).
Aaron Marcuse-Kubitza

09/29/2013

11:35 PM Revision 11117: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added primary_collector, to scope the collector_number. collectors: renamed to all_collectors for clarity, now that there is a primary_collector field.
Aaron Marcuse-Kubitza
11:14 PM Revision 11116: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.current_collection, owner_collection: moved to specimen_observation because these may differ each time the specimen is reobserved
Aaron Marcuse-Kubitza
11:08 PM Revision 11115: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.code_in_individual: renamed to id_within_individual to match the id_within_* naming convention using elsewhere
Aaron Marcuse-Kubitza
10:58 PM Revision 11114: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation: removed code because this duplicates taxon_observation.collector_number and record.id_within_dataset
Aaron Marcuse-Kubitza
10:53 PM Revision 11113: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.specimenholder_institutions: moved to individual_observation because the list of institutions which store a specimen from the plant actually relates to the plant, not a particular specimen (this list should be the same for all specimens taken from the plant)
Aaron Marcuse-Kubitza
10:34 PM Revision 11112: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed collection_event because this is the same as individual_observation
Aaron Marcuse-Kubitza
10:11 PM Revision 11111: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed separate fkey to individual because this is linked to from individual_observation
Aaron Marcuse-Kubitza
09:55 PM Revision 11110: schemas/VegCore/ERD/VegCore.ERD.mwb: place_observation: link to project. taxa_sampling_event: removed link to project since this now happens via place_observation.
Aaron Marcuse-Kubitza
09:35 PM Revision 11109: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: merged required within_place into place_observation, which already contains a required fkey to place
Aaron Marcuse-Kubitza
09:29 PM Revision 11108: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: added place_observation fkey for the concurrent place_observation, which contains all the info about the place (communities, soil data, etc.). this allows a taxon_observation to be traced to its associated location measurements.
Aaron Marcuse-Kubitza
08:58 PM Revision 11107: bugfix: mappings/VegCore-VegBIEN.csv: nest all taxonoccurrences inside a stratum event, so that the parent locationevent is *always* fully populated before child locationevents point to it. (previously, a stub parent event was created when the child event was imported first, which blocked the fully-populated parent event from being inserted later on.) this uses auto-folding (for VegBank/CVS) and auto-forwarding (for other datasources) to prune empty stratum events for taxonoccurrences that don't have strata. (see wiki.vegpath.org/Auto-folding, wiki.vegpath.org/Auto-forwarding for more info about these normalization techniques.) note that the inserted row counts stay *exactly the same* for all datasources except VegBank (which was being fixed), indicating that this signficant change to the mappings did *not* change the semantics of the import of taxonoccurrences.
Aaron Marcuse-Kubitza
08:37 PM Revision 11106: inputs/VegBank/observationcontributor_/test.xml.ref: updated inserted row count
Aaron Marcuse-Kubitza

09/28/2013

10:40 PM Revision 11105: bugfix: mappings/VegCore-VegBIEN.csv: stratum's locationevent: link this to the parent locationevent, so that the parent locationevent's information (such as locationeventcontributors) is accessible to the stratum's locationevent
Aaron Marcuse-Kubitza
09:08 PM Revision 11104: bugfix: inputs/VegBank/taxon_observation.**/postprocess.sql: inlined _join() so that taxon_observation.** wouldn't get cascadingly deleted whenever the util schema (where this normally resides) gets reinstalled
Aaron Marcuse-Kubitza
08:23 PM Revision 11103: bugfix: schemas/vegbien.sql: locationeventcontributor: added missing unique constraint (locationeventcontributor_unique)
Aaron Marcuse-Kubitza
08:21 PM Revision 11102: bugfix: schemas/vegbien.sql: locationeventcontributor.role: made it optional because some locationeventcontributors may not have an associated role
Aaron Marcuse-Kubitza
06:45 PM Revision 11101: schemas/vegbien.sql: analytical_stem_view, analytical_plot: added plot__collectors, which uses locationevent__contributors__array()
Aaron Marcuse-Kubitza
06:42 PM Revision 11100: schemas/vegbien.sql: locationevent__contributors(): split into set-returning and array-returning functions. sort results by fullname so there is a predictable ordering (VegBank does not include a sort_order in the observationContributor table).
Aaron Marcuse-Kubitza
05:56 PM Revision 11099: schemas/vegbien.sql: added locationevent__contributors() aggregating function for use in analytical_stem_view
Aaron Marcuse-Kubitza
06:53 AM Revision 11098: added inputs/VegBank/observationcontributor_/
Aaron Marcuse-Kubitza
06:51 AM Revision 11097: mappings/VegCore-VegBIEN.csv: mapped event__participant
Aaron Marcuse-Kubitza
05:51 AM Revision 11096: mappings/VegCore.htm: regenerated from wiki. added EventParticipant table containing event__participant.
Aaron Marcuse-Kubitza
05:04 AM Revision 11095: bugfix: inputs/VegBank/: taxonOccurrenceID: include the aggregateOrganismObservationID in this so that there is one taxonoccurrence for *each* stratum's taxonImportance. this allows the different strata to have separate taxonoccurrences that are associated with the stratum-specific locationevents, rather than all being lumped into one taxonoccurrence, with only one locationevent.
Aaron Marcuse-Kubitza
12:06 AM Revision 11094: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: renamed to observation_in_parent_event because the taxon_observation might be connected to a stratum_event which is nested within another sampling_event before it's connected to any sampling_event for a parent plot. this is the case for VegBank data with associated strata.
Aaron Marcuse-Kubitza

09/27/2013

11:14 PM Revision 11093: bugfix: schemas/vegbien.sql: locationevent: locationevent_unique_within_creator unique index on sourceaccessioncode: need to add stratum_id, because the addition of stratum_id to *this* table instead of a separate stratum_event table (which would have required more schema changes) means we need to use the awkward construction of appending stratum_id to *every* unique constraint, including ones that ordinarily would just pass through the datasource's pkey
Aaron Marcuse-Kubitza
06:31 PM Revision 11092: web/links/index.htm: updated to Firefox bookmarks. added Java client- and server-side links. web: split into client- and server-side folders.
Aaron Marcuse-Kubitza
06:29 PM Revision 11091: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
03:43 PM Revision 11090: /README.TXT: Datasource setup: added link to Example steps for a datasource (wiki.vegpath.org/Import_process_for_Madidi)
Aaron Marcuse-Kubitza
03:23 PM Revision 11089: /README.TXT: Full database import: To remake analytical DB: added runtime (13 h)
Aaron Marcuse-Kubitza

09/26/2013

10:15 AM Revision 11088: bugfix: schemas/postgresql.Mac.conf: allow external client logins like for vegbiendev, by listening on all addresses rather than just localhost
Aaron Marcuse-Kubitza

09/24/2013

05:53 PM Revision 11087: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
04:40 PM Revision 11086: schemas/vegbien.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
02:54 PM Revision 11085: schemas/vegbien.sql: analytical_plot: added stratum__name
Aaron Marcuse-Kubitza
02:48 PM Revision 11084: schemas/vegbien.ERD.mwb: updated to PostgreSQL schema
Aaron Marcuse-Kubitza
02:47 PM Revision 11083: schemas/vegbien.sql: analytical_stem_view: added stratum table (stratum__name)
Aaron Marcuse-Kubitza
02:14 PM Revision 11082: mappings/VegCore-VegBIEN.csv: mapped stratum__name
Aaron Marcuse-Kubitza
02:09 PM Revision 11081: schemas/vegbien.sql: added source_id to allow different datasources to have their own strata
Aaron Marcuse-Kubitza
02:03 PM Revision 11080: schemas/vegbien.ERD.mwb: updated to PostgreSQL schema
Aaron Marcuse-Kubitza
01:51 PM Revision 11079: schemas/vegbien.sql: stratum: added unique constraint on source_id/stratumname, and require stratumname, so that strata are identified by name within their datasource
Aaron Marcuse-Kubitza
01:45 PM Revision 11078: schemas/vegbien.sql: added source_id to allow different datasources to have their own strata
Aaron Marcuse-Kubitza
01:22 PM Revision 11077: schemas/vegbien.ERD.mwb: regenerated exports
Aaron Marcuse-Kubitza
01:18 PM Revision 11076: schemas/vegbien.ERD.mwb: updated to PostgreSQL schema
Aaron Marcuse-Kubitza
01:15 PM Revision 11075: schemas/vegbien.sql: locationevent: added stratum_id to unique constraints, so that there can be multiple locationevents for the same underlying event but with different strata
Aaron Marcuse-Kubitza
01:01 PM Revision 11074: schemas/vegbien.sql: link locationevent to stratum instead of the other way around (similar to how it's represented in normalized VegCore as stratum_event->stratum, rather than stratum->taxa_sampling_event). this works around column-based import's lack of support for linking two tables together via two separate routes (in this case locationevent->taxonoccurrence->stratum and locationevent->stratum).
Aaron Marcuse-Kubitza
12:54 PM Revision 11073: web/links/index.htm: updated to Firefox bookmarks. added PostgreSQL links.
Aaron Marcuse-Kubitza
12:09 PM Revision 11072: schemas/vegbien.sql: stratum: added stratumname
Aaron Marcuse-Kubitza

09/22/2013

08:25 PM Revision 11071: mappings/VegCore.htm: regenerated from wiki. added Stratum table.
Aaron Marcuse-Kubitza
07:57 PM Revision 11070: schemas/vegbien.ERD.mwb: updated to PostgreSQL schema
Aaron Marcuse-Kubitza
06:25 PM Revision 11069: schemas/VegCore/VegCore.ERD.mwb: model options: template for new fkey names: use the PostgreSQL naming convention so that new fkeys will sync properly with equivalent fkeys in the Postgres schema that have had their names reset
Aaron Marcuse-Kubitza
06:18 PM Revision 11068: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
06:13 PM Revision 11067: schemas/VegCore/VegCore.ERD.mwb: source: made name optional because some sources (e.g. records) don't have a name, and are instead identified differently. instead require name in applicable subclasses. source subclasses: added unique constraints on parent/name since the name is presumed to be identifying within the parent entity.
Aaron Marcuse-Kubitza
05:55 PM Revision 11066: schemas/VegCore/VegCore.ERD.mwb: party: added separate person subclass, which has the organization pointer. organization instead has a recursive parent pointer, to avoid mutual recursion between party and organization.
Aaron Marcuse-Kubitza
09:00 AM Revision 11065: schemas/VegCore/VegCore.ERD.mwb: dataset.parent: updated comment to refer to data_source, not traceable.source (which is now used for a different purpose)
Aaron Marcuse-Kubitza
08:56 AM Revision 11064: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
08:49 AM Revision 11063: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
08:46 AM Revision 11062: schemas/VegCore/ERD/VegCore.ERD.mwb: removed unnecessary hstore extenders that duplicate an extender of the same name in a superclass
Aaron Marcuse-Kubitza
08:25 AM Revision 11061: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: replaced metadata_source with data_source, since the source field inherited from traceable is actually for where the information in the dataset record itself came from
Aaron Marcuse-Kubitza
07:57 AM Revision 11060: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.parent: documented that this provides a containment relationship, which differs from the attribution relationship provided by source (inherited from traceable). e.g. the GBIF.ARIZ dataset is contained within the GBIF dataset (the parent), but was obtained from ARIZ (the source).
Aaron Marcuse-Kubitza
07:37 AM Revision 11059: schemas/VegCore/ERD/VegCore.ERD.mwb: source: moved parent pointer to dataset, since the containment relationship is intended for datasets (there may be other types of sources for which this makes sense, but not ones that are used in our data)
Aaron Marcuse-Kubitza
07:07 AM Revision 11058: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.table_fragment: renamed to id_within_source, since this can also identify e.g. a dataset within the source that it came from
Aaron Marcuse-Kubitza
06:47 AM Revision 11057: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: added metadata_source, which stores where the information *about* the dataset came from (as opposed to where the dataset came from)
Aaron Marcuse-Kubitza
06:31 AM Revision 11056: schemas/VegCore/ERD/VegCore.ERD.mwb: party: inherit from source because a person (or organization) can directly be a source of data. it should not be necessary to link them to a dataset in order to attribute rows to them.
Aaron Marcuse-Kubitza
06:25 AM Revision 11055: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.table_fragment: made this optional, since some traceables may have a source but are not *uniquely identifiable* within that source. the ID would in this case need to be populated by a subclass table, since traceable can't generate a unique ID without the fragment.
Aaron Marcuse-Kubitza
04:33 AM Revision 11054: schemas/VegCore/ERD/VegCore.ERD.mwb: record: clarified that "an actual data record" means "a row in a table". dataset: made definition more specific ("a collection of records from the same place, with the same attribution requirements").
Aaron Marcuse-Kubitza
04:08 AM Revision 11053: schemas/VegCore/ERD/VegCore.ERD.mwb: record: added definition (a collection of data relating to the same entity). note that this doesn't have to be an actual data record, it can also be a paragraph of text relating to the same thing (e.g. a method description in a PDF).
Aaron Marcuse-Kubitza
02:52 AM Revision 11052: schemas/VegCore/ERD/VegCore.ERD.mwb: method: added optional name field
Aaron Marcuse-Kubitza
02:49 AM Revision 11051: schemas/VegCore/ERD/VegCore.ERD.mwb: stratum: added definition (a non-horizontal way of subdividing a plot)
Aaron Marcuse-Kubitza
02:32 AM Revision 11050: schemas/VegCore/ERD/: svn:ignore *.bak
Aaron Marcuse-Kubitza
02:14 AM Revision 11049: schemas/VegCore/ERD/VegCore.ERD.mwb: stratum, stratum_event: added VegX synonyms
Aaron Marcuse-Kubitza
02:00 AM Revision 11048: schemas/VegCore/ERD/VegCore.ERD.mwb: synonyms with other exchange schemas: put the exchange schema name first, so that the first word in the URL is the source of that synonym (which is most important), while the schema-specific synonym name (which is less important) comes after it. this matches the VegCore data dictionary convention where the text of the URL (i.e. the most important part) is the exchange schema name.
Aaron Marcuse-Kubitza
01:53 AM Revision 11047: schemas/VegCore/ERD/VegCore.ERD.mwb: synonyms with other exchange schemas: put the exchange schema name first, so that the first word in the URL is the source of that synonym (which is most important), while the schema-specific synonym name (which is less important) comes after it. this matches the VegCore data dictionary convention where the text of the URL (i.e. the most important part) is the exchange schema name.
Aaron Marcuse-Kubitza
01:09 AM Revision 11046: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_count.taxon: renamed to taxon_presence to clarify that this is an fkey to taxon_presence
Aaron Marcuse-Kubitza

09/21/2013

11:01 PM Revision 11045: schemas/VegCore/: moved ERD-related files to ERD/ subdir
Aaron Marcuse-Kubitza
10:58 PM Revision 11044: added schemas/VegCore/ERD/SQL symlink to the dir with the VegCore SQL schemas
Aaron Marcuse-Kubitza
10:52 PM Revision 11043: schemas/VegCore/: moved ERD-related files to ERD/ subdir
Aaron Marcuse-Kubitza
10:45 PM Revision 11042: schemas/VegCore/: moved ERD-related files to ERD/ subdir
Aaron Marcuse-Kubitza
10:44 PM Revision 11041: schemas/VegCore/: moved ERD-related files to ERD/ subdir
Aaron Marcuse-Kubitza
10:27 PM Revision 11040: bugfix: schemas/VegCore/ERD/.htaccess: mod_autoindex: clear the IndexHeadInsert message "some listed files are not web-accessible" so that it is not displayed above the ERD image, which looks strange because the files are below it (this message can be removed for this dir because all files in it are web-accessible)
Aaron Marcuse-Kubitza
10:17 PM Revision 11039: web/.htaccess: mod_autoindex: IndexHeadInsert: use IndexStyleSheet for stylesheet link instead
Aaron Marcuse-Kubitza
10:12 PM Revision 11038: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore logo: hyperlink this to VegCore.vegpath.org as well, instead of to the data dictionary
Aaron Marcuse-Kubitza
10:10 PM Revision 11037: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
10:00 PM Revision 11036: schemas/VegCore/VegCore.ERD.mwb: moved VegCore.vegpath.org link next to logo so it can be found more easily
Aaron Marcuse-Kubitza
09:45 PM Revision 11035: schemas/VegCore/ERD/.htaccess: display the index page (containing the ERD) as part of the mod_autoindex directory listing, so that the directory listing is included when you go to the VegCore.vegpath.org/ERD/ URL. this will allow the ERD-related files to be moved to the ERD/ subdir, and still be visible in the directory listing.
Aaron Marcuse-Kubitza
09:42 PM Revision 11034: bugfix: web/.htaccess: mod_autoindex: IndexOptions: use +HTMLTable to ensure that the contents are always displayed in an HTML table, even when a header file is provided with HeaderName (which turns off the default HTML table for some reason)
Aaron Marcuse-Kubitza
09:01 PM Revision 11033: lib/sql_io.py: put_table(): added link to new INSERT ON DUPLICATE SELECT wiki page, which now contains the explanation in the doc comment
Aaron Marcuse-Kubitza
07:29 PM Revision 11032: schemas/VegCore/VegCore.ERD.mwb: ERD link: changed to VegCore.vegpath.org so that the user can access *all* additional files for VegCore, not just the PDF version of the ERD
Aaron Marcuse-Kubitza
07:24 PM Revision 11031: schemas/VegCore/VegCore.ERD.mwb: source.name: require this field (for records, populate it from id_within_dataset)
Aaron Marcuse-Kubitza
07:01 PM Revision 11030: schemas/VegCore/VegCore.ERD.mwb: record.dataset_record_id: renamed to id_within_dataset for clarity
Aaron Marcuse-Kubitza
06:59 PM Revision 11029: schemas/VegCore/VegCore.ERD.mwb: traceable: added table_fragment field, which is the portion of the source record that this traceable refers to. (denormalized source data often contains data for many VegCore tables in the same row, and the traceables for each of these table entries must be distinguished *from each other* since they share the same source. this is usually just the VegCore table name, sometimes with a distinguishing prefix (e.g. collector.party/identified_by.party; current_observation.taxon_determination/orig_observation.taxon_determination).) this field is needed *in addition to the source record* to form a unique ID for the traceable.
Aaron Marcuse-Kubitza
06:24 PM Revision 11028: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
06:22 PM Revision 11027: schemas/VegCore/VegCore.ERD.mwb: dataset: inherit from traceable because the information *about* the dataset (data_owners, contacts, etc.) might have been obtained from an external source, such as IH, a web page, or even an e-mail conversation
Aaron Marcuse-Kubitza
05:47 PM Revision 11026: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
05:45 PM Revision 11025: schemas/VegCore/VegCore.ERD.mwb: taxon_observation: draw inheritance connector to event to clarify what table this inherits from. reobservable.original_observation: renamed to orig_observation to match the naming convention of specimen.orig_collection. table relationships legend: updated "inherits from record" label to refer to the traceable table, which now serves this purpose.
Aaron Marcuse-Kubitza

09/19/2013

06:49 PM Revision 11024: bugfix: inputs/VegBank/import_order.txt: added stratum
Aaron Marcuse-Kubitza
06:48 PM Revision 11023: inputs/VegBank/taxon_observation.**/postprocess.sql: added stratum, stratumtype to the left-join
Aaron Marcuse-Kubitza
06:46 PM Revision 11022: inputs/VegBank/stemcount_/map.csv: stratum_id: removed table prefix so it can be used as a join column
Aaron Marcuse-Kubitza
06:45 PM Revision 11021: inputs/VegBank/: mapped stratum
Aaron Marcuse-Kubitza
06:39 PM Revision 11020: inputs/VegBank/: mapped stratumtype
Aaron Marcuse-Kubitza
06:27 PM Revision 11019: /README.TXT: Datasource setup: additional steps for new-style datasources: added steps not present in http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource because they were performed all at once for all datasources
Aaron Marcuse-Kubitza
06:24 PM Revision 11018: /README.TXT: Datasource setup: added additional steps for new-style datasources, from http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
Aaron Marcuse-Kubitza
04:18 PM Revision 11017: bugfix: schemas/vegbien.sql: taxondetermination_set_iscurrent(): is_datasource_current: accept any determinationtype other than the TNRS ones (accepted, matched), so that datasources that provide a custom value for this field (such as VegBank) don't have their taxondeterminations incorrectly treated as non-datasource. this fixes the VegBank bug where datasource taxondeterminations were not being joined on in analytical_stem_view, because the join included a filter for only datasource taxondeterminations, but these taxondeterminations were not being properly treated as such. **this should fix the missing taxonomic information in the VegBank validation extract.**
Aaron Marcuse-Kubitza
11:31 AM Revision 11016: inputs/VegBank/taxonobservation_/map.csv: taxonomic ranks not in VegCore: removed table prefix so they will be automapped (they are globally unique)
Aaron Marcuse-Kubitza
10:33 AM Revision 11015: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
10:31 AM Revision 11014: planning/timeline/timeline.2013.xls: updated as decided in the conference call
Aaron Marcuse-Kubitza
02:55 AM Revision 11013: inputs//: don't import joined tables, because they are now imported in the taxon_observation.** left-join instead
Aaron Marcuse-Kubitza
01:22 AM Revision 11012: inputs/VegBank/taxon_observation.**/postprocess.sql: run mk_subset_by_row_num_func() to add a subset function that uses sort_col. this is used by column-based import, and also provides a common subsetting/sorting API for all the left-joined views. test.xml.ref: the inserted row count most likely changes because the sort order changes.
Aaron Marcuse-Kubitza
01:05 AM Revision 11011: schemas/util.sql: added mk_subset_by_row_num_func(regclass), which uses a sort_col instead of a row_num column
Aaron Marcuse-Kubitza
12:10 AM Revision 11010: schemas/util.sql: mk_subset_by_row_num_func(): factored creation of no-sort subset function into mk_subset_by_row_num_no_sort_func() so it can be used by other functions
Aaron Marcuse-Kubitza

09/18/2013

11:53 PM Revision 11009: bugfix: inputs/VegBank/stemlocation_/map.csv: remapped stemcount-related fields to OMIT, so that these don't collide with fields of the same name in stemcount_ when they are left-joined together in taxon_observation.** . having the same name causes these to be incorrectly interpreted as shared fkey columns in the NATURAL JOIN (and without the NATURAL JOIN, they would instead be collision errors).
Aaron Marcuse-Kubitza
10:35 PM Revision 11008: bugfix: inputs/VegBank/stemlocation_/postprocess.sql: added missing index on aggregateOrganismObservationID, needed for the 1:many portion of the taxon_observation.** left-join
Aaron Marcuse-Kubitza
10:28 PM Revision 11007: inputs/VegBank/stemcount_/postprocess.sql: moved stemcount___parent index before the derived columns section because it does not depend on them
Aaron Marcuse-Kubitza
10:26 PM Revision 11006: bugfix: inputs/VegBank/stemcount_/postprocess.sql: added missing index on taxonOccurrenceID, needed for the 1:many portion of the taxon_observation.** left-join
Aaron Marcuse-Kubitza
10:14 PM Revision 11005: schemas/util.sql: added ||% operator to append to escaped strings (the % indicates an identifier, as in Perl hashes and one of the x86 assembler syntaxes for registers
Aaron Marcuse-Kubitza
03:50 PM Revision 11004: bugfix: inputs/VegBank/taxon_observation.**/postprocess.sql: added sort_col (=identificationID) at beginning because column-based import will always sort a view by the first column, which may lead to slow query plans if the first column is not a joined table's pkey
Aaron Marcuse-Kubitza
02:04 PM Revision 11003: inputs/VegBank/taxon_observation.**/postprocess.sql: documented that there is no row_num because left-join to stemcount_, stemlocation_ adds rows to each taxonobservation_
Aaron Marcuse-Kubitza
02:03 PM Revision 11002: bugfix: inputs/VegBank/taxon_observation.**/postprocess.sql: removed row_num (=identificationID), because there is actually more than one row per VegBank taxonobservation_, so this does not properly enumerate the view rows. this is because there is a 1:many left-join to stemcount_, stemlocation_ which adds rows to each taxonobservation_. since the row_num is gone, any row-subsetting of the view using OFFSET will always need to materialize the entire view up to the OFFSET value. this works for smaller datasources like VegBank that fit almost entirely into one column-based import chunk (1 million rows), but not for larger datasources like FIA where it would be much slower to materialize all preceding 16 million rows on the last chunk (which is what OFFSET normally does with left-joins).
Aaron Marcuse-Kubitza
01:51 PM Revision 11001: bugfix: inputs/VegBank/taxon_observation.**/: generated header.csv and related files, which were previously not generated because the error in `rm header.csv` aborted the runscript
Aaron Marcuse-Kubitza
 

Also available in: Atom