schemas/VegCore/VegCore.ERD.mwb: source: allow multiple data owners

schemas/VegCore/VegCore.ERD.mwb: documented that record is the record in the source data, and traceable is a row that can have associated source information

schemas/VegCore/VegCore.ERD.mwb: removed relationship table, because this is too general, and custom relationships between entities are better indicated in the applicable hstore extender field for the table in question. record: split into record and traceable tables, where record is instead the record in the source data, and traceable indicates that a row has associated source information.

bugfix: schemas/VegCore/VegCore.ERD.mwb: party_list_entry: use just a single id field as the primary key, because the pkey must be a single text string in order to uniquely identify the row by a global ID

schemas/VegCore/VegCore.ERD.mwb: party_list_entry: added role field

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

schemas/VegCore/VegCore.ERD.mwb: taxa_sampling_event: link directly to project, so that you don't have to search the parent event hierarchy to find the associated project

schemas/VegCore/VegCore.ERD.mwb: fixed table positions

schemas/VegCore/VegCore.ERD.mwb: spaced tables out more

schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: require all taxon_occurrences to have an associated within_place (which may be set to Earth if the coordinates/placename is unknown)

schemas/VegCore/VegCore.ERD.mwb: moved the plot table next the the section label for the plots half of the page

schemas/VegCore/VegCore.ERD.mwb: taxon_observation: allow any taxon_observation to link directly to a taxa_sampling_event, not just aggregate_observation. this allows any taxon_observation's methodology information to be found directly by joining to taxa_sampling_event. previously, one had to join to event.parent and then taxa_sampling_event, and it was not clear that event.parent would even be a taxa_sampling_event.

schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: link to rather than being a taxa_sampling_event, because taxa_sampling_event was actually intended to be the container event in which various types of taxon_observations occur, rather than a type of taxon_observation itself. rearranged tables to avoid them covering any lines.

schemas/VegCore/VegCore.ERD.mwb: renamed GNRS & geovalidation steps to just geoscrubbing steps for clarity

schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: added within_place to allow easily locating all the taxon_occurrences in a plot. (some taxon_occurrences require within_place, but there was no common superclass field to allow joining to just a single table.)

schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved reobservable tables (individual, specimen, etc.) before aggregate_observation tables because these are less abstract

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

schemas/VegCore/VegCore.ERD.mwb: individual_count: inherit from taxon_presence because this is also a type of taxon_observation (and specifically, a taxon_presence), in addition to the general taxon_presence it's associated with. it is possible e.g. for different diameter classes to be counted by different collectors, and thus to need separate taxon_observations to store the different collectors.

schemas/VegCore/VegCore.ERD.mwb: added taxon_occurrence to store the underlying occurrence for each taxon_observation. we cannot use reobservable for this because not all taxon_observations are reobservable, but the one-time observations still have an associated occurrence. moved reobservable.current_determination to taxon_occurrence and renamed it to current_observation (this can be used to find the latest observation for each occurrence).

planning/timeline/timeline.2013.xls: added spacing between later column and weeks to make clear that dots in the later column do not correspond to a week

inputs/FIA/occurrence_all/: renamed to taxon_observation.** to clarify what an occurrence is. taxon_observation has the same meaning as in VegCore, where each taxon_determination is considered a separate taxon_observation of the associated specimen or vouchered plant (vegpath.org/VegCore/ERD/). (note that having multiple taxon_determinations only makes sense when there is something to reobserve.) the .** is SQL dotpath syntax (wiki.vegpath.org/SQL_dotpaths) for the recursive expansion of all tables to which taxon_observation has forward fkeys (i.e. "the left-join").

inputs/FIA/occurrence_all/postprocess.sql: use :table_str var instead of hardcoding the table name as occurrence_all, to avoid needing to change this file when renaming the view

inputs/FIA/occurrence_all/postprocess.sql: renamed occurrence_all--row_num to just row_num because this is for the entire view, rather than a specific table in it, and thus does not need a disambiguating table prefix. this also avoids embedding the view name in its own columns.

inputs/FIA/*/map.csv: use -- to separate the table and column name instead of - , to conform with the u-name format (wiki.vegpath.org/u-name#format), which works even when only one of _- can be used in the name. -s are needed in this case to linewrap the column on a separate line as the table in phpPgAdmin.

lib/runscripts/datasrc_dir.run: added postprocess target to run postprocess in just the table subdirs, skipping any additional subdirs that don't have this target

lib/runscripts/datasrc_dir.run: @subdirs: moved import_order.txt subdirs into separate @table_subdirs, which provides access to just the table subdirs when the user adds other dirs to @subdirs

inputs/VegBank/: prepended the table name to each column name to prevent column collisions, using the steps at http://wiki.vegpath.org/Left-joining_a_datasource

inputs/VegBank/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource

bugfix: inputs/VegBank/stemlocation_/map.csv: put columns in table order, which is needed by new-style import

inputs/VegBank/stemlocation_/: translated one-to-many mappings to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

bugfix: inputs/VegBank/taxonobservation_/map.csv: put columns in table order, which is needed by new-style import

bugfix: inputs/VegBank/plot_/postprocess.sql: coordinateUncertaintyInMeters: need to use GREATEST instead of _alt() to handle cases where the coordinate uncertainty is > than the fuzzing uncertainty, where you wouldn't want to just use the smaller fuzzing uncertainty

inputs/VegBank/plot_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/VegBank/plot_/postprocess.sql: map_*() derived cols: updated runtime

inputs/VegBank/plot_/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/VegBank/stemcount_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/VegBank/stemlocation_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/VegBank/taxonobservation_/postprocess.sql: scientificName: recorded runtime (15 s)

inputs/VegBank/taxonobservation_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/VegBank/taxonobservation_/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

inputs/FIA/occurrence_all/postprocess.sql: use much simpler LEFT JOINs instead of nested RIGHT JOINs, which required lots of () to get them to happen in the right order. note that the columns are now provided in reverse instead of forwards path order, but this is still much clearer than the nested mess of RIGHT JOINs. this approach can also be used to simplify VegBank's joins.

bugfix: lib/runscripts/view.run: remake_VegBIEN_mappings(): also need to remake header.csv, not just map.csv as for tables, because view columns may change when the view is regenerated

schemas/VegCore/VegCore.ERD.mwb: specimen: changed definition to "something collected from a plant" rather than just "a physical part of a plant", to support using this table for identifying pictures and descriptions of a plant (as DwC does)

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

schemas/VegCore/VegCore.ERD.mwb: reobservable_presence: allow it to be vouchered by any reobservable element (including a tagged individual), not just a specimen

schemas/VegCore/VegCore.ERD.mwb: specimen.defining_data: clarified that the observations in this are actually a subset of individual_observation.traits (specifically, the subset that can be used to make a taxonomic redetermination). information in this field should therefore always also be stored in individual_observation.traits.

schemas/VegCore/VegCore.ERD.mwb: specimen: added specimen_unique_in_individual_observation unique constraint, analogous to specimen_unique_in_individual

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

schemas/VegCore/VegCore.ERD.mwb: specimen: added defining_data, which for a digital-only specimen, stores the information that comprises the specimen. note that a taxon_presence without a physical voucher can still qualify as reobservable if a detailed description of it is provided in this field, to make taxonomic redeterminations on. for datasources like VegBank, which incorrectly allow multiple taxon_determinations for any type of taxon_observation, their taxonomic redeterminations would actually be considered invalid if made on a purely taxon_presence observation (i.e. just a taxon name) without a detailed description that could be used to make a redetermination. this is different than the scrubbing of a taxon name, which relates a taxon name to another taxon name, rather than a taxon_observation to a completely different taxon name.

bugfix: lib/sh/util.sh: set_fds(): don't add surrounding quotes to empty redirect dest

bugfix: lib/sh/util.sh: set_fds(): need to check if redirect is empty before escaping it with `printf %q`, which may add surrounding quotes to an empty string

planning/timeline/timeline.2013.xls: attribution and conditions of use: documented that Brad/Brian/Bob should work on this, as decided in the conference call (wiki.vegpath.org/2013-09-12_conference_call#data-provider-metadata)

planning/timeline/timeline.2013.xls: reformatted to fit all rows and all per-week columns on one page

planning/timeline/timeline.2013.xls: streamline process of mapping and adding a new datasource: added subtask to create interactive scripts for each import step

planning/timeline/timeline.2013.xls: improve and complete data provider metadata: moved to end because this can also been added manually to the source table, and does not have to be in place before running column-based import

planning/timeline/timeline.2013.xls: flatten the datasources to a common schema: added subtask to left-join unvalidated datasources since they need the flattening in order to validate them properly

planning/timeline/timeline.2013.xls: rebalanced dots

planning/timeline/timeline.2013.xls: moved items marked later to separate section at bottom

planning/timeline/timeline.2013.xls: moved revisions to schema under datasource validations because schema changes are largely driven by validations problems uncovered

planning/timeline/timeline.2013.xls: split tasks into weeks

planning/timeline/timeline.2013.xls: updated for progress

planning/timeline/timeline.2013.xls: split months into (currently identical) weeks

planning/timeline/timeline.2013.xls: added During month of label above months

planning/timeline/timeline.2013.xls: switched to portrait mode to better fit the new format, which hides columns for past months

planning/timeline/timeline.2013.xls: hid crossed out rows to show just the remaining tasks

planning/timeline/timeline.2013.xls: crossed out avoid DB restructuring when ingesting a new datasource, because FIA (which is flattened before import) does properly support optional subplots and diamond linking of subplots to parent plot events, which were necessary to ingest an arbitrary flattened plots datasource

planning/timeline/timeline.2013.xls: crossed out fully-completed tasks. rebalanced dots.

planning/timeline/timeline.2013.xls: moved switching to new-style import to top of streamline process of mapping and adding a new datasource because this puts all the datasource adding steps (except filling in the mappings) into one rerunnable script

planning/timeline/timeline.2013.xls: hid columns for past months so that the current and future months are right next to each task

planning/timeline/timeline.2013.xls: moved streamline process of mapping and adding a new datasource before documentation testing because this will assist the documentation tester in running the import process

planning/timeline/timeline.2013.xls: moved geoscrubbing re-run under add any missing columns because this is needed to fully populate the geoscrubbing columns

planning/timeline/timeline.2013.xls: added documentation testing, usability testing priority tasks (wiki.vegpath.org/Priority_tasks). lowercased tasks for consistency with the wiki and to avoid needing to sentence case new subtasks.

planning/timeline/timeline.2013.xls: moved Flatten the datasources to a common schema under Datasource validations because the query left-joining the tables is needed for validation, and it is much easier to validate datasources when there is only one input table to validate

added derived/biengeo/Geovalidation_and_geoscrubbing_update.presentation.url

added BIEN2/traits_observation_counts.xls

/README.TXT: Single datasource import: removed rescrub step because this is not needed by the current TNRS process

web/links/index.htm: updated to Firefox bookmarks. MySQL: added steps to add a user if you are not root but have sudo access.

BIEN2/country_species/: svn:ignore the .tsv exports

BIEN2/country_species/run: documented runtime (1 min)

added BIEN2/country_species/run, which exports each BIEN2 country's species list

bugfix: lib/sh/util.sh: set_fds(): need to escape redirect destinations which are files, because they may contain special shell characters

lib/sh/util.sh: added rm_prefix()

lib/sh/db.sh: mysql_cmd(): added caller usage with connection/login opts

lib/sh/db.sh: mysql(), mysql_export(): usage: added database=...

planning/timeline/timeline.2013.xls: Data provider validations: renamed to Datasource validations to clarify that this is a validation of the datasources, but not necessarily by the data providers

/README.TXT: Full database import: added Running individual steps separately label for the section that is not part of the main import, but is useful if the import is aborted part of the way through

/README.TXT: moved Single datasource import, Datasource setup to top since these are the most important howtos

bugfix: schemas/Makefile: enclose schema names in "" so that they won't be lowercased

bugfix: schemas/Makefile, lib/common.Makefile: enclose schema names in "" so that they won't be lowercased

/run: geoscrub_input/make(): updated runtime (20 s)

planning/timeline/timeline.2013.xls: Data provider validations (spot-checking): moved ahead of Individual datasource refresh as decided in conference call

schemas/vegbien.sql: analytical_plot: added aggregateOrganismObservationID from analytical_stem

planning/timeline/timeline.2013.xls: updated for progress

planning/timeline/timeline.2013.xls: Data provider validations: added subtask for Aggregated validations (counts)

inputs/import.stats.xls: analytical DB: updated rowcount

inputs/import.stats.xls: updated import times

inputs/input.Makefile: reimport: don't remove the existing import first, because it will instead be removed by the publish step. this ensures there is always one complete copy of the datasource in the DB.

added backups/vegbien.r10848.backup.md5

backups/TNRS.backup.md5: updated

bugfix: bin/import_all: use reimport_scrub instead of import_scrub so that the temp suffix of the datasource name is removed