Project

General

Profile

Statistics
| Revision:

# Date Author Comment
11183 10/09/2013 09:56 AM Aaron Marcuse-Kubitza

bugfix: web/people/Brad_Boyle/attribution/.htaccess: \-escape the %s so that they aren't treated as backreferences. set the [redirect] flag so that the downloaded file has the correct filename (BIEN 3 data use and attribution.docx instead of attribution).

11182 10/09/2013 09:49 AM Aaron Marcuse-Kubitza

bugfix: web/people/Brad_Boyle/attribution/.htaccess: use %20 instead of + because the Apache server apparently doesn't translate + to space correctly

11181 10/09/2013 09:47 AM Aaron Marcuse-Kubitza

bugfix: web/people/Brad_Boyle/attribution/.htaccess: link to the VegBIEN subdir in /exchange_schemas instead of the main /VegBIEN because there are actually two VegBIEN subdirs, and the main one instead points to /datasources/VegBIEN

11180 10/09/2013 09:45 AM Aaron Marcuse-Kubitza

added web/people/Brad_Boyle/ with attribution/ subdir for BIEN 3 data use and attribution.docx

11179 10/09/2013 09:15 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: renamed data_source to dataset_source for clarity

11178 10/09/2013 08:54 AM Aaron Marcuse-Kubitza

inputs/VegBank/plantconcept_/create.sql: documented runtime (21 min)

11177 10/09/2013 08:28 AM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/plantconcept_/: added new-style import files

11176 10/09/2013 08:27 AM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/import_order.txt: added plantconcept_, because new-style import needs it to be explicitly listed in import_order.txt in order to run it

11175 10/09/2013 08:24 AM Aaron Marcuse-Kubitza

inputs/VegBank/run: refresh(): added usage

11174 10/09/2013 08:22 AM Aaron Marcuse-Kubitza

inputs/VegBank/run: refresh(): documented that this should be run on vegbiendev

11173 10/09/2013 07:58 AM Aaron Marcuse-Kubitza

inputs/VegBank/_archive/2012-8-30/: svn:ignore the data exports

11172 10/09/2013 07:57 AM Aaron Marcuse-Kubitza

inputs/VegBank/run: added refresh() target

11171 10/09/2013 07:54 AM Aaron Marcuse-Kubitza

inputs/VegBank/: refreshed VegBank so that all of Mike Lee's sample plots would be included in the input data. (VegBank was last refreshed from the live DB on 2012-8-30.) split vegbank.sql into vegbank.schema.sql and vegbank.data.sql so that the schema can be examined and imported separately, like for MySQL datasources. inputs/VegBank/vegbank.~.clean_up.sql: commented out setting comminterpretation.commname to NOT NULL, because after the refresh it is now NULL in 10 rows, where commconcept_id is also NULL.

11170 10/08/2013 05:31 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions

11169 10/08/2013 05:29 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir

11168 10/08/2013 12:36 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/_src/README.TXT: added e-mail from John Donoghue with general description of the BIEN2 geovalidation workflow

11167 10/03/2013 04:04 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: plot.**: removed ORDER BY because a SELECT on the view with its own ORDER BY will actually sort the rows twice instead of optimizing away plot.**'s default ORDER BY

11166 10/03/2013 03:44 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: added plot.** subset view of analytical_stem_view, used for VegBank validation

11165 10/03/2013 01:41 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved reload core & analytical database out a week because it shouldn't be done until after CVS validation is complete

11164 10/03/2013 12:25 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/_src/README.TXT: added link to geovalidation description in wiki

11163 10/03/2013 11:03 AM Aaron Marcuse-Kubitza

added schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx from Brad

11162 10/03/2013 10:03 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated to reflect that the next week should be spent doing data validations, as decided in the conference call

11161 10/02/2013 05:59 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationevent__communities(): need to use commdescription instead of commname as the communityName because that's where communityName has been mapped to in mappings/VegCore-VegBIEN.csv. (commname instead stores the communityID.)

11160 10/02/2013 05:56 AM Aaron Marcuse-Kubitza

inputs/VegBank/: added observation__community/

11159 10/02/2013 05:49 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: commclass.commcode,commname: rename to prevent collisions

11158 10/02/2013 04:26 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: indicate required column comminterpretation.commname

11157 10/02/2013 04:20 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: commconcept.commname: rename to prevent collision with commname.commname

11156 10/02/2013 04:15 AM Aaron Marcuse-Kubitza

planning/timeline/: svn:ignore: added .~* (lock files for .xls)

11155 10/02/2013 03:58 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: added instructions for what to do after updating this

11154 10/02/2013 03:55 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view, analytical_plot: added communities using new locationevent__communities__array()

11153 10/02/2013 03:52 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: added locationevent__communities*()

11152 10/02/2013 03:10 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationevent: locationevent__pull_forward_from_parent() trigger: only pull forward the location_id when there is a stratum_id, so that empty locationevents that should be auto-forwarded are not unintentionally made valid by populating the location_id field (which causes a check constraint to become satisfied, and prevents auto-forwarding from working). this fixes a bug in some datasources' automated tests, such as `make inputs/SALVIAS/plotObservations/test`.

11151 10/02/2013 02:45 AM Aaron Marcuse-Kubitza

lib/sql_io.py: put_table(): default param: documented that this will be used for all missing rows, regardless of which error caused them not to be inserted. this means that auto-forwarding (wiki.vegpath.org/Auto-forwarding) can be used with any type of constraint violation, not just NOT NULL constraints (which it is typically used with).

11150 10/01/2013 10:57 PM Aaron Marcuse-Kubitza

/Makefile: mk_db: config/users.sql: unset ON_ERROR_STOP so user-exists errors don't prevent further user-adds

11149 09/30/2013 07:47 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11148 09/30/2013 07:38 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: analytical_stem_view: plot__collectors: to include plot__collectors for nested stratum events, need to calculate the locationevent__contributors__array() for the locationevent and parent_event separately, and then COALESCE those, instead of just running locationevent__contributors__array() once on the COALESCE of the two IDs

11147 09/30/2013 07:34 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationevent: added pull-forward trigger to populate location_id from the parent event. a pull-forward trigger must be used for this (not a COALESCE in analytical_stem_view) because the locationevent cannot even be found by analytical_stem_view if it does not have an associated location.

11146 09/30/2013 06:37 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: source: added required url field, which points to the source data and uniquely identifies the source. note that this differs from the id, which is a self-refential pkey (as described at wiki.vegpath.org/Web_interface#Clickable-fkeys).

11145 09/30/2013 05:47 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines

11144 09/30/2013 05:38 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11143 09/30/2013 05:32 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: source: removed canon, because this would create too much complexity as a result of having multiple copies of the same source with different names. instead, source names will be scrubbed upon insert so that the row always contains the most accurate data, and there is only one copy of each source. this scrubbing may use the assistance of a thesaurus table w/ standard abbreviations, etc.

11142 09/30/2013 05:07 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: plot.name: merged with inherited name field

11141 09/30/2013 04:43 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11140 09/30/2013 04:40 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: source: added canon field. when there are multiple names for a source, this points to the canonically-named source. by default, points recursively to itself.

11139 09/30/2013 04:04 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.taxon_occurrence: documented that when not provided (and not fillable from another field), a taxon_occurrence will be created with the same id as the taxon_observation. this operation should eventually be perfomed by a trigger.

11138 09/30/2013 03:58 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place.parent_geoplace: renamed to parent so it will merge with the inherited parent field. this is the parent geoplace.

11137 09/30/2013 03:27 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11136 09/30/2013 03:20 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines

11135 09/30/2013 03:18 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that instead, the taxon_occurrence for the individual_observation should have an autogenerated current_observation that merges together all the individual_observations for these institutions' specimens. in this case, each institution's specimen will still have its own individual_observation containing what it believes to be the specimenholder institutions.

11134 09/30/2013 03:03 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that when provided, the source (i.e. traceable.source) should be set to the value of this field, because these institutions together provided the data for this entity

11133 09/30/2013 02:59 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: party_list: inherit from source so that a party_list can be used as the source for a row (e.g. a list of institutions that together provided the data for an individual_observation that has specimens at multiple institutions)

11132 09/30/2013 02:55 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: reobservable_presence: merged into taxon_determination, so that voucher is directly available to taxon_determination to autopopulate fields when a redetermination is made on the voucher

11131 09/30/2013 02:37 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines

11130 09/30/2013 02:36 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: aggregate_observation.sampling_event: merged with inherited sampling_event field

11129 09/30/2013 02:33 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.parent: require that this is an fkey to another taxon_observation, since this stores the taxon_observation of the same taxon_occurrence in the parent sampling_event

11128 09/30/2013 02:22 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: subplot.parent_plot: renamed to parent so it will merge with the inherited parent field. this is the parent plot.

11127 09/30/2013 02:16 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: individual_count.taxon_presence: renamed to parent so it will merge with the inherited parent field. this is the taxon_presence for the taxon as a whole (not subdivided by size_class, etc.).

11126 09/30/2013 02:13 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event.method: renamed to subsetting_method for clarity

11125 09/30/2013 02:06 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: stratum_event.parent_event: renamed to parent so it will merge with the inherited parent field

11124 09/30/2013 01:58 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: traceable: added alternate pkey id_by_source, which allows referencing a traceable by its source information even when the subclass chose a different value for the pkey

11123 09/30/2013 01:49 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: merged with inherited parent field, documenting that for this table, this contains the observation of the same taxon_occurrence in the parent sampling_event

11122 09/30/2013 01:33 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines

11121 09/30/2013 01:24 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.place_observed_at: merged with inherited place field, documenting that for this table, this contains the place the individual was observed at

11120 09/30/2013 01:16 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: event: reordered date_range, participants, place in the order of how likely they are to be provided. (place was considered more optional than the other two because a project does not usually have a place.)

11119 09/30/2013 01:09 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.sampling_event: made it required so that there is always a sampling_event to scope the primary_collector/collector_number when provided. (for specimens, this will have the same event information (date_range, participants, place) as the taxon_observation.)

11118 09/30/2013 12:35 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added unique constraint on sampling_event, primary_collector, collector_number. reordered these columns in constraint order (path order).

11117 09/29/2013 11:35 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added primary_collector, to scope the collector_number. collectors: renamed to all_collectors for clarity, now that there is a primary_collector field.

11116 09/29/2013 11:14 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.current_collection, owner_collection: moved to specimen_observation because these may differ each time the specimen is reobserved

11115 09/29/2013 11:08 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.code_in_individual: renamed to id_within_individual to match the id_within_* naming convention using elsewhere

11114 09/29/2013 10:58 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation: removed code because this duplicates taxon_observation.collector_number and record.id_within_dataset

11113 09/29/2013 10:53 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.specimenholder_institutions: moved to individual_observation because the list of institutions which store a specimen from the plant actually relates to the plant, not a particular specimen (this list should be the same for all specimens taken from the plant)

11112 09/29/2013 10:34 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed collection_event because this is the same as individual_observation

11111 09/29/2013 10:11 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed separate fkey to individual because this is linked to from individual_observation

11110 09/29/2013 09:55 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place_observation: link to project. taxa_sampling_event: removed link to project since this now happens via place_observation.

11109 09/29/2013 09:35 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: merged required within_place into place_observation, which already contains a required fkey to place

11108 09/29/2013 09:29 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: added place_observation fkey for the concurrent place_observation, which contains all the info about the place (communities, soil data, etc.). this allows a taxon_observation to be traced to its associated location measurements.

11107 09/29/2013 08:58 PM Aaron Marcuse-Kubitza

bugfix: mappings/VegCore-VegBIEN.csv: nest all taxonoccurrences inside a stratum event, so that the parent locationevent is always fully populated before child locationevents point to it. (previously, a stub parent event was created when the child event was imported first, which blocked the fully-populated parent event from being inserted later on.) this uses auto-folding (for VegBank/CVS) and auto-forwarding (for other datasources) to prune empty stratum events for taxonoccurrences that don't have strata. (see wiki.vegpath.org/Auto-folding, wiki.vegpath.org/Auto-forwarding for more info about these normalization techniques.) note that the inserted row counts stay exactly the same for all datasources except VegBank (which was being fixed), indicating that this signficant change to the mappings did not change the semantics of the import of taxonoccurrences.

11106 09/29/2013 08:37 PM Aaron Marcuse-Kubitza

inputs/VegBank/observationcontributor_/test.xml.ref: updated inserted row count

11105 09/28/2013 10:40 PM Aaron Marcuse-Kubitza

bugfix: mappings/VegCore-VegBIEN.csv: stratum's locationevent: link this to the parent locationevent, so that the parent locationevent's information (such as locationeventcontributors) is accessible to the stratum's locationevent

11104 09/28/2013 09:08 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/taxon_observation.**/postprocess.sql: inlined _join() so that taxon_observation.** wouldn't get cascadingly deleted whenever the util schema (where this normally resides) gets reinstalled

11103 09/28/2013 08:23 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationeventcontributor: added missing unique constraint (locationeventcontributor_unique)

11102 09/28/2013 08:21 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationeventcontributor.role: made it optional because some locationeventcontributors may not have an associated role

11101 09/28/2013 06:45 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view, analytical_plot: added plot__collectors, which uses locationevent__contributors__array()

11100 09/28/2013 06:42 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: locationevent__contributors(): split into set-returning and array-returning functions. sort results by fullname so there is a predictable ordering (VegBank does not include a sort_order in the observationContributor table).

11099 09/28/2013 05:56 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: added locationevent__contributors() aggregating function for use in analytical_stem_view

11098 09/28/2013 06:53 AM Aaron Marcuse-Kubitza

added inputs/VegBank/observationcontributor_/

11097 09/28/2013 06:51 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: mapped event__participant

11096 09/28/2013 05:51 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: regenerated from wiki. added EventParticipant table containing event__participant.

11095 09/28/2013 05:04 AM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/: taxonOccurrenceID: include the aggregateOrganismObservationID in this so that there is one taxonoccurrence for each stratum's taxonImportance. this allows the different strata to have separate taxonoccurrences that are associated with the stratum-specific locationevents, rather than all being lumped into one taxonoccurrence, with only one locationevent.

11094 09/28/2013 12:06 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: renamed to observation_in_parent_event because the taxon_observation might be connected to a stratum_event which is nested within another sampling_event before it's connected to any sampling_event for a parent plot. this is the case for VegBank data with associated strata.

11093 09/27/2013 11:14 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: locationevent: locationevent_unique_within_creator unique index on sourceaccessioncode: need to add stratum_id, because the addition of stratum_id to this table instead of a separate stratum_event table (which would have required more schema changes) means we need to use the awkward construction of appending stratum_id to every unique constraint, including ones that ordinarily would just pass through the datasource's pkey

11092 09/27/2013 06:31 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added Java client- and server-side links. web: split into client- and server-side folders.

11091 09/27/2013 06:29 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11090 09/27/2013 03:43 PM Aaron Marcuse-Kubitza

/README.TXT: Datasource setup: added link to Example steps for a datasource (wiki.vegpath.org/Import_process_for_Madidi)

11089 09/27/2013 03:23 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: To remake analytical DB: added runtime (13 h)

11088 09/26/2013 10:15 AM Aaron Marcuse-Kubitza

bugfix: schemas/postgresql.Mac.conf: allow external client logins like for vegbiendev, by listening on all addresses rather than just localhost

11087 09/24/2013 05:53 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11086 09/24/2013 04:40 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: fixed lines

11085 09/24/2013 02:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_plot: added stratum__name

11084 09/24/2013 02:48 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: updated to PostgreSQL schema