Activity
From 09/27/2013 to 10/26/2013
10/25/2013
- 06:15 PM Revision 11450: Added db user and host to load-geoscrub-input.sh
- The psql commands in load-geoscrub-input.sh will now connect with a
specific user on a specific host.
Updated the 'CO... - 04:51 PM Revision 11449: derived/biengeo/README.txt: geoscrub new data: steps that use .sql scripts: added the psql commands to run these
- 04:22 PM Revision 11448: Updated install instructions in the README.
- 03:00 PM Revision 11447: derived/biengeo/README.txt: geoscrub new data: noted that this now deletes any previous geoscrubbing results
- 02:58 PM Revision 11446: derived/biengeo/README.txt: added steps to set the working dir for each set of steps
- 02:54 PM Revision 11445: derived/biengeo/README.txt: added section on obtaining source code, including path to Paul's in-progress files on vegbiendev (not sure whether the in-progress files are needed to run the core scripts in steps 1-6)
- 02:44 PM Revision 11444: derived/biengeo/README.txt: moved commands to run to the top of the README. flagged commands-sections with ***** and an identifying label.
- 02:04 PM Revision 11443: Initial checkin of geoscrub install SQL files.
- Added install.*.sql files that will do initial table creation for all
required tables.
Added a truncate.vegbien_geosc... - 02:04 PM Revision 11442: Update load-geoscrub-input.sh to download from URL.
- Removed logic to dump input data directly from the vegbien database and
to download the input from a URL provided by ... - 11:56 AM Revision 11441: planning/timeline/timeline.2013.xls: reload core & analytical database scheduled for this week: postponed to give us additional time to do datasource validations
- 09:58 AM Revision 11440: inputs/input.Makefile: added %/import_temp alias for %/import, to mirror the presence of import_temp for import
- 09:24 AM Revision 11439: fix: inputs/VegBank/taxonobservation_/map.csv: remapped authorplantname to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
- 09:22 AM Revision 11438: fix: inputs/VegBank/taxonobservation_/map.csv: remapped int_* to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
10/24/2013
- 07:09 PM Revision 11437: exports/2013-10-18.Brian_Enquist.Canadensys.csv.run: inherit from new import_subset.run (which uses extract.run)
- 07:08 PM Revision 11436: added lib/runscripts/import_subset.run, extract.run
- 05:21 PM Revision 11435: added exports/2013-10-18.Brian_Enquist.Canadensys.csv.run
- 05:07 PM Revision 11434: bin/make_analytical_db: removed no longer needed setting of $schema to $public, because this is now done by psql()
- 05:06 PM Revision 11433: lib/sh/local.sh: psql(): also accept $public as the $schema param, since this is used by a lot of import scripts
- 04:24 PM Revision 11432: lib/sh/util.sh: added require_dot_script()
- 04:13 PM Revision 11431: bugfix: lib/sh/util.sh: $top_script: use @BASH_SOURCE instead of $0, because this is also defined for .-scripts
- 04:03 PM Revision 11430: bugfix: bin/import_all: restore the working dir when main() is done, in case it started as something other than the root dir
- 03:49 PM Revision 11429: bin/after_import: support turning off the end-of-import backup for imports that are not the full database
- 03:26 PM Revision 11428: bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.
- 03:21 PM Revision 11427: bugfix: lib/runscripts/util.run: if is_dot_script, fix $@ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $@ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).
- 03:17 PM Revision 11426: bugfix: lib/sh/util.sh: is_dot_script(): need to subtract 1 from ${#BASH_LINENO[@]}, because this is the array length rather than the index of the last element as in Perl
- 02:58 PM Revision 11425: lib/sh/util.sh: added is_dot_script()
- 01:15 PM Revision 11424: bugfix: schemas/vegbien.sql: taxondetermination_set_iscurrent(): is_datasource_current (used by analytical_stem_view): need to separately check if `determinationtype IS NULL`, because `determinationtype NOT IN (accepted, matched))` will return NULL (false) if determinationtype is NULL, causing no match
- 01:11 PM Revision 11423: bugfix: bin/make_analytical_db: when running into a public schema other than "public", also pass this to `/run export_` (which currently uses $schema instead of $public)
- 01:10 PM Revision 11422: bugfix: bin/import_all: fix $@ when .-included without args (which causes bash to put the wrong values in $@ instead of leaving it empty)
- 01:09 PM Revision 11421: bin/import_all: `make schemas/$version/install`: reinstall instead to allow re-running the import to the same custom schema (e.g. 2013-10-18.Brian_Enquist.Canadensys)
- 01:07 PM Revision 11420: bin/import_all: `make schemas/$version/install`: ignore errors if schema exists, to support running with -e
10/23/2013
- 11:10 PM Revision 11419: bugfix: bin/import_all: removing inputs/.TNRS/tnrs/tnrs.make.lock: use `"rm" -f` instead of plain "rm" to avoid having an error exit status, which will abort the script if run with the -e flag (as runscripts are)
- 11:02 PM Revision 11418: lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args
- 11:00 PM Revision 11417: lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)
- 10:34 PM Revision 11416: bin/*_all: *_main(): renamed to just main() because it does not matter that other shell-includes' main() methods will clobber this, because it is only executed once
- 10:29 PM Revision 11415: bugfix: bin/import_all: Source tables: use .../import instead of import_temp because import_temp is only needed when importing all tables, to prevent the temp suffix from being removed yet
- 10:17 PM Revision 11414: lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV
- 09:11 PM Revision 11413: bugfix: lib/sh/util.sh: alias_append(): need to enclose $(alias) call in "" because its result may contain separator chars (i.e. whitespace) that will be parsed incorrectly. this appears to only be a bug when runscripts are run as shell-includes, with a leading ".".
- 01:49 PM Revision 11412: schemas/VegCore/ERD/VegCore.ERD.mwb: connecting lines: inherits from traceable: added arrow to indicate what this label refers to
- 01:43 PM Revision 11411: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 01:37 PM Revision 11410: schemas/VegCore/ERD/VegCore.ERD.mwb: HAS-A/IS-A box: renamed to "connecting lines" for clarity
10/22/2013
- 10:22 PM Revision 11409: schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: HAS-A: added HAS-MANY going in the opposite direction, because every HAS-A has an opposite HAS-MANY
- 10:17 PM Revision 11408: schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: IS-A, HAS-A: added directional arrows
- 10:10 PM Revision 11407: schemas/VegCore/ERD/VegCore.ERD.mwb: field order box: removed spacing between top of text box and bottom of outer box label
- 10:04 PM Revision 11406: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 10:00 PM Revision 11405: schemas/VegCore/ERD/VegCore.ERD.mwb: reordered columns according to the field order convention
- 09:49 PM Revision 11404: schemas/VegCore/ERD/VegCore.ERD.mwb: added label documenting the field order convention:
- 1) inherited
2) required
3) identifying
4) foreign key
5) extenders
6) others - 08:34 PM Revision 11403: web/links/index.htm: updated to Firefox bookmarks. added links for EER models, data management plans. put PostgreSQL before MySQL because we have found PostgreSQL to be a much more capable database system, even though it lacks some of MySQL's user-friendly features.
- 06:38 PM Revision 11402: planning/timeline/timeline.2013.xls: updated for progress
- 06:21 PM Revision 11401: fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names
- 06:16 PM Revision 11400: schemas/vegbien.sql: regenerated using `make schemas/remake`. note that analytical_stem_view column renamings need this step after a search-and-replace of the column names, in order to remove excess "" around all-lowercase names and reset generated index names.
- 06:10 PM Revision 11399: fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names
- 01:20 PM Revision 11398: added planning/goals/web_interface/phpPgAdmin.select_interface.png for use at wiki.vegpath.org/Proposed_enhancements
- 09:39 AM Revision 11397: inputs/CVS/_src/: added refresh from Mike Lee
10/21/2013
- 07:14 PM Revision 11396: fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error
- 05:16 PM Revision 11395: planning/timeline/timeline.2013.xls: usability testing: added subtask to provide scientists with their requested data
10/20/2013
- 05:51 PM Revision 11394: planning/timeline/timeline.2013.xls: updated for progress
- 05:21 PM Revision 11393: bugfix: bin/import_all: need to publish datasources that won't be published by `make .../import`, so that the per-datasource import XPaths that refer to TNRS/geoscrub will link up with the TNRS/geoscrub source entry instead of creating a new entry without the metadata (because the entry with the metadata was named TNRS.new/geoscrub.new)
- 05:09 PM Revision 11392: schemas/vegbien.sql: datasource_publish(): use parameter names instead of $# because this is a PL/pgSQL function
- 05:07 PM Revision 11391: bugfix: schemas/vegbien.sql: datasource_publish(): if the datasource to publish already has the published name, don't datasource_rm() it
- 04:55 PM Revision 11390: bin/import_all: removed no longer needed import of geoscrub data, because analytical_stem_view is now joined to the geoscrub_output table directly, instead of using the imported canon_place entries
- 04:52 PM Revision 11389: schemas/vegbien.sql: analytical_stem_view: join to the geoscrub_output table directly, instead of using the imported canon_place entries. this avoids the need to import geoscrub_output into VegBIEN (which is expected to take 2+ hours after the refresh), as well as the need to then refresh any datasources whose geoscrubbing input data has changed.
- 04:37 PM Revision 11388: inputs/.geoscrub/geoscrub_output/postprocess.sql: added nullable unique index on the inputs, for use by analytical_stem_view. note that it must be nullable in order to create a match when not all of the input fields are populated. this uses array[] to create a nullable index, which is much better than column-based import and VegBIEN's use of COALESCE() because the expression is the same for every type and no NULL sentinel value is needed.
- 03:50 PM Revision 11387: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 03:41 PM Revision 11386: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:34 PM Revision 11385: schemas/VegCore/ERD/VegCore.ERD.mwb: person: allow to have multiple organizations
- 03:28 PM Revision 11384: schemas/VegCore/ERD/VegCore.ERD.mwb: split "2b. GNRS" label into two labels, one for each table GNRS is applied to
- 03:23 PM Revision 11383: schemas/VegCore/ERD/VegCore.ERD.mwb: georeferencing: merged into geoplace, since this is actually information attached to a specific plot, etc. relating to the coordinates used in its geoplace subclass
- 02:59 PM Revision 11382: schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place: changed parent geoplace pointer to parent_boundary_WKT, since the immediate parent may not have an associated boundary to use for geovalidation (i.e. it may not be an official GADM geoplace), although ancestors further up likely will be
- 02:40 PM Revision 11381: schemas/VegCore/ERD/VegCore.ERD.mwb: place.name: made it required because it's needed for the unique constraint to be populated properly (including for subclasses such as geoplace, which need to generate this from the coordinates)
- 02:30 PM Revision 11380: schemas/VegCore/ERD/VegCore.ERD.mwb: place.rank: made it required, because every place should have some kind of rank indicating what type of place it is, including lower ranks (e.g. plot, individual)
- 02:24 PM Revision 11379: schemas/VegCore/ERD/VegCore.ERD.mwb: place: added unique constraint on parent, rank, name
- 02:24 PM Revision 11378: schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have
- 02:07 PM Revision 11377: schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have
- 01:46 PM Revision 11376: schemas/VegCore/ERD/VegCore.ERD.mwb: geoplace.official_name: renamed to name to merge with inherited field from place. documented that for geoplaces, this is the official, scrubbed name.
- 12:48 PM Revision 11375: inputs/.geoscrub/geoscrub_output/postprocess.sql: added geovalid derived column, for use by analytical_stem_view
10/19/2013
- 06:56 PM Revision 11374: bin/with_all: $all: renamed to $hidden_srcs for clarity, since this now just adds the hidden (.*) datasources, rather than always using all datasources
- 06:50 PM Revision 11373: bugfix: bin/with_all: in $all mode, just prepend the .* datasources to the user-selected (or default) @inputs, so that using $all to add these datasources doesn't inadvertently cause the action to be performed for *all* datasources
- 04:33 PM Revision 11372: web/links/index.htm: updated to Firefox bookmarks. PostgreSQL: ALTER TABLE: added documentation about disabling of foreign key triggers, which is only possible by the superuser. note that marking a foreign key constraint as NOT VALID does *not* disable the trigger, so NOT VALID cannot be used for this purpose. this would be used to add fkeys from core VegBIEN tables to validation results tables such as the geoscrubbing results, without needing to import the validation results directly into core VegBIEN (which is time-consuming and currently must be done *before* input data is loaded, requiring a datasource reload to add geoscrubbing results).
- 02:15 PM Revision 11371: bin/import_all: usage: documented that this can now be run with a custom datasources list (each of the form inputs/src/)
- 02:02 PM Revision 11370: bin/with_all: added support for providing a custom list of inputs to run the command on
- 01:29 PM Revision 11369: inputs/.geoscrub/geoscrub_output/postprocess.sql, run: updated runtimes
- 01:13 AM Revision 11368: inputs/.geoscrub/geoscrub_output/run: documented full load_data() runtime (9 min @starscream)
- 01:12 AM Revision 11367: inputs/.geoscrub/geoscrub_output/postprocess.sql: updated runtimes for refreshed data, which now has 4x as many rows (1,707,970->6,747,650)
- 12:54 AM Revision 11366: inputs/.geoscrub/geoscrub_output/: refreshed geoscrub data. removed +header.csv because the extract now contains the header in the first row of the file.
- 12:52 AM Revision 11365: bugfix: lib/sh/local.sh: psql(): $is_root: use `` around case statement instead of $(), because it contains an embedded unbalanced )
- 12:27 AM Revision 11364: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: include only the columns that Jim provided in his extract (the geoscrub table contains additional internal columns that are not part of the geovalidation data for VegBIEN). documented runtime (30 s) and upload time (1.5 min).
10/18/2013
- 10:33 PM Revision 11363: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: removed no longer needed setting of $local_server, $local_user (and use of $local_pg_database instead of $database) because the use_local bug in local.sh has been fixed
- 10:32 PM Revision 11362: bugfix: lib/sh/local.sh: psql(): don't default the connection vars using use_local if running as the postgres user. in that case, connection must happen via a socket, with server="", and as the user running the command (postgres), with user="".
- 09:55 PM Revision 11361: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to manually set local_server, local_user to "" so that they do not default to their bien-user values
- 09:54 PM Revision 11360: bugfix: lib/sh/db.sh: avoid outputting to /dev/fd/# when running as sudo on Linux, because this causes a "Permission denied" error (due to the /dev/fd/# file being owned by a different user). this is not a problem with normal redirects (>&#), because they do not use /dev/fd/# files which can have access permissions.
- 09:52 PM Revision 11359: bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file
- 08:17 PM Revision 11358: lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command
- 07:50 PM Revision 11357: lib/sh/db.sh: pg_as_root(): run sudo with echo_run to help debug
- 06:29 PM Revision 11356: bugfix: lib/sh/db.sh: pg_cmd(): only set PG* connection/login env vars when the corresponding var is *non-empty*. there are some situations in which these must be unset (in order to use the default value), and other situations when the var must be set to something (i.e. "") to avoid it being defaulted to a value in local.sh > connection vars.
- 06:27 PM Revision 11355: backups/TNRS.backup.md5: updated
- 06:13 PM Revision 11354: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to set $local_pg_database instead of $database because use_local (in psql()) does not currently avoid clobbering already-set versions of the applicable env vars
- 06:11 PM Revision 11353: bugfix: lib/sh/local.sh: pg_as_root(): need to use -E (preserve environment) option to sudo, so that $schema, $table get passed through
- 06:05 PM Revision 11352: bugfix: lib/sh/local.sh: psql(): only \set schema, table if $schema, $table are non-empty, because otherwise, you will get a "zero-length delimited identifier" error
- 05:30 PM Revision 11351: added inputs/.geoscrub/geoscrub_output/geoscrub.csv.run to export the geoscrub table (must be run on vegbiendev)
- 05:29 PM Revision 11350: lib/sh/local.sh: added require_remote()
- 05:29 PM Revision 11349: lib/sh/db.sh: added pg_as_root()
- 05:28 PM Revision 11348: lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.
- 05:23 PM Revision 11347: Added instructions for dependencies in the README.
- 05:23 PM Revision 11346: Added indexes to speed up geonames-to-gadm.sql.
- Without these indexes, these queries could take hours to complete.
With them, the times more closely matched the time... - 05:23 PM Revision 11345: Fixed a couple of syntax errors in geovalidate.sh.
- Fixed a sql syntax error and a bash syntax error in the next line.
- 04:03 PM Revision 11344: planning/timeline/timeline.2013.xls: "geoscrubbing automated pipeline": scheduled for after Paul's current set of tasks on the geoscrubbing re-run is complete. i'm budgeting several weeks for this since my understanding is that Paul is doing this part-time.
- 03:57 PM Revision 11343: planning/timeline/timeline.2013.xls: moved "geoscrubbing automated pipeline" under "simplify import process for easier maintainability"
- 03:53 PM Revision 11342: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added subtask to spot-check reloaded geoscrubbing data
- 03:52 PM Revision 11341: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added separate subtask for "geoscrubbing data reload", since apparently it was not clear that of course the new data will need to be imported into VegBIEN before the results of the re-run are available. this is currently scheduled to happen in the next full-database import, which is the week of 10/28 in order to include further validations fixes.
- 02:06 PM Revision 11340: planning/timeline/timeline.2013.xls: CVS validation: use timespan dot ◦ for supertask
- 02:05 PM Revision 11339: planning/timeline/timeline.2013.xls: CVS validation: added subtasks that are similar to for FIA validation (create validation subset, create extract)
- 02:03 PM Revision 11338: planning/timeline/timeline.2013.xls: FIA validation: split apart into subtasks, including "decide which columns to validate", which has to happen ahead of time before the extract can be generated
- 01:15 PM Revision 11337: planning/timeline/timeline.2013.xls: fixed check marks for past (hidden) weeks, which had gotten duplicated when rows were copied together with their check marks
- 01:04 PM Revision 11336: planning/timeline/timeline.2013.xls: fixed line heights
- 12:57 PM Revision 11335: planning/timeline/timeline.2013.xls: fixed column width so the dates display properly in MS Excel
- 12:48 PM Revision 11334: planning/timeline/timeline.2013.xls: right-aligned legend so it isn't too close to the "During week of:" label
- 12:46 PM Revision 11333: planning/timeline/timeline.2013.xls: added legend:
- • task
◦ timespan
✓ task progress
☑ timespan progress - 10:25 AM Revision 11332: planning/timeline/timeline.2013.xls: attribution/conditions of use: made it a subtask of "add missing columns" because this is related to data needed for published analyses. added dots because this is an ongoing task, that depends on data providers getting their use conditions to us.
- 10:08 AM Revision 11331: planning/timeline/timeline.2013.xls: reload core & analytical database: moved next reload ahead to last week of October so that we can include the updated geovalidation data for the 10/31 deadline. added additional reload so that they are spaced <= 1 month apart.
- 09:54 AM Revision 11330: planning/timeline/timeline.2013.xls: receive feedback from documentation tester: added an extra week to receive additional feedback from them in response to documentation fixes made
- 09:51 AM Revision 11329: planning/timeline/timeline.2013.xls: attribution/conditions of use: made this a top-level task instead of a subtask of "data provider metadata", to avoid including lower-priority tasks (i.e. in the later column) in the same section as higher-priority tasks
- 09:47 AM Revision 11328: planning/timeline/timeline.2013.xls: datasource validations: regrouped by subtask instead of by datasource, so that the high-priority subtasks get done for all datasources before moving on to lower-priority subtasks for any datasources
- 08:54 AM Revision 11327: planning/timeline/timeline.2013.xls: reduced width of Milestone column to make room to fit an additional week on the printed page
- 08:42 AM Revision 11326: planning/timeline/timeline.2013.xls: attribution/conditions of use: removed "(Brad/Brian/Bob/etc.)" because these are from everyone who provided or obtained data, not just Brad/Brian/Bob
- 08:40 AM Revision 11325: planning/timeline/timeline.2013.xls: rescheduled tasks to accommodate the separate non-critical feature requests subtasks
- 08:37 AM Revision 11324: planning/timeline/timeline.2013.xls: datasource validations: split "fix feature requests" into separate "fix critical feature requests" and "fix non-critical feature requests" tasks. rescheduled non-critical feature requests until after the other validation tasks have been completed.
- 08:11 AM Revision 11323: planning/timeline/timeline.2013.xls: add globally-unique occurrenceID: moved up to next week because we would like to be able to get this done for the 10/31 deadline
- 07:52 AM Revision 11322: planning/timeline/timeline.2013.xls: updated for progress
- 07:48 AM Revision 11321: planning/timeline/timeline.2013.xls: moved "data provider metadata" before "datasource validations (spot-checking)" because conditions of use are necessary for scientists who want to publish papers based on the data (which is a key use case)
- 07:43 AM Revision 11320: planning/timeline/timeline.2013.xls: moved "usability testing" before "datasource validations (spot-checking)" because this is most important towards reaching our goal of a useful information resource
- 07:41 AM Revision 11319: planning/timeline/timeline.2013.xls: moved "geoscrubbing re-run", "add globally-unique occurrenceID" back under "usability testing" > "add missing columns" because these are in fact part of the usability testing
- 01:26 AM Revision 11318: planning/timeline/timeline.2013.xls: updated for progress
10/17/2013
- 11:59 PM Revision 11317: planning/timeline/timeline.2013.xls: updated for progress
- 10:32 PM Revision 11316: planning/timeline/timeline.2013.xls: "flatten the datasources to a common schema": moved to later column because the complex tasks "switching to new-style import" and "create interactive scripts for each import step" are also scheduled then. (it's unlikely we would have much time over winter break anyway, considering that there is ~1 week's worth of holidays then.)
- 10:24 PM Revision 11315: planning/timeline/timeline.2013.xls: scheduled "simplify import process for easier maintainability"
- 10:20 PM Revision 11314: planning/timeline/timeline.2013.xls: tasks performed by someone else (geoscrubbing re-run): changed solid check marks ✓ to open check marks ☑ to match the solid • vs. open ◦ dot convention
- 10:17 PM Revision 11313: planning/timeline/timeline.2013.xls: documentation testing: added supertask dots. removed later dots for scheduled tasks.
- 10:15 PM Revision 11312: planning/timeline/timeline.2013.xls: scheduled "documentation testing"
- 10:02 PM Revision 11311: planning/timeline/timeline.2013.xls: scheduled "simplify process of mapping/adding a new datasource"
- 10:01 PM Revision 11310: planning/timeline/timeline.2013.xls: "add globally-unique occurrenceID": moved it up to the first week when we're no longer fixing existing issues in datasources, since this has similar priority to adding missing columns discovered during usability testing (which is scheduled as an ongoing task)
- 09:54 PM Revision 11309: planning/timeline/timeline.2013.xls: usability testing: did task breakdown (find scientists who want to use BIEN3 data, etc.) and scheduled subtasks
- 09:34 PM Revision 11308: planning/timeline/timeline.2013.xls: moved "add missing columns" to its own supertask. used outline check mark ☑ (analogous to open circle ◦) to mark supertasks as completed which were split up into subtasks.
- 09:29 PM Revision 11307: planning/timeline/timeline.2013.xls: later column: removed dots from scheduled items
- 09:06 PM Revision 11306: planning/timeline/timeline.2013.xls: moved "switching to new-style import"-related steps (other than for CVS) to separate "simplify import process for easier maintainability" supertask, since this is not part of the "simplify process of mapping/adding a new datasource" task
- 08:39 PM Revision 11305: planning/timeline/timeline.2013.xls: add any missing columns: added and scheduled step to add globally-unique occurrenceID
- 08:34 PM Revision 11304: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added dots ◦ for this for the time when it can be worked on asynchronously by Paul Sarando
- 08:30 PM Revision 11303: planning/timeline/timeline.2013.xls: data provider metadata: added dots ◦ for the portion of "attribution and conditions of use" that can be worked on asynchronously by Brad/Brian/Bob
- 08:24 PM Revision 11302: planning/timeline/timeline.2013.xls: scheduled "aggregated validations" during the last 2 weeks of "datasource validations (spot-checking)", because these weeks are only spent fixing issues uncovered in the remaining datasources, so there may be extra time then
- 07:20 PM Revision 11301: planning/timeline/timeline.2013.xls: scheduled other tasks after "datasource validations (spot-checking)" is complete
- 07:14 PM Revision 11300: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): each datasource's validation supertask: added open circles ◦ spanning the length of the subtasks
- 07:12 PM Revision 11299: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for supertasks that have been fully split into subtasks (not just itemizing a few subtasks), so that these don't count towards the bullets (estimated workload) in each week
- 07:09 PM Revision 11298: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for tasks that are performed by someone other than me, so that these don't count towards the bullets (estimated workload) in each week
- 07:04 PM Revision 11297: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): split each datasource into subtasks and scheduled them
- 06:00 PM Revision 11296: planning/timeline/timeline.2013.xls: moved "move denormalized validations to stage II", "move stage III validations to stage II" outside of "switching to new-style import" because the "switching to new-style import" step refers just to the per-datasource switching steps, not to the additional refactorings that would be needed to avoid dependency on the complex XPath mappings (mappings/VegCore-VegBIEN.csv)
- 05:57 PM Revision 11295: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): added subtasks for each of the remaining datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order)
- 05:43 PM Revision 11294: planning/timeline/timeline.2013.xls: moved non-validation-related tasks after the 10/31 deadline so that these are not taking time away from the validation
- 05:40 PM Revision 11293: planning/timeline/timeline.2013.xls: moved "flatten the datasources to a common schema" under "simplify process of mapping/adding a new datasource" because this is also needed separately for datasources where the left-joining is not part of the validation
- 05:30 PM Revision 11292: planning/timeline/timeline.2013.xls: extended "revisions to VegBIEN schema" to length of "datasource validations (spot-checking)" because schema changes are expected as we add missing fields
- 05:23 PM Revision 11291: planning/timeline/timeline.2013.xls: crossed out and hid completed tasks ("find out amount remaining in BIEN3 budget")
- 05:17 PM Revision 11290: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): extended through the end of November because data providers' fixes on the remaining 10 datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order) are likely to add significantly to the issues and feature requests associated with these datasources (e.g. the 2nd-round VegBank validation added 4 issues and 5 feature requests). there is also expected to be wait time while data providers are responding (most likely in multiple rounds of feedback).
- 04:59 PM Revision 11289: planning/timeline/timeline.2013.xls: data provider metadata: removed "iPlant can do" because this actually requires Brad/Brian/Bob/other data providers to provide this info. however, this info may be findable on the web for some datasources.
- 04:53 PM Revision 11288: planning/timeline/timeline.2013.xls: moved "data provider metadata" right after "datasource validations" because this is part of the completed database itself rather than the tools to maintain it
- 04:47 PM Revision 11287: planning/timeline/timeline.2013.xls: split "revisions to schema" into "revisions to VegBIEN schema" (part of datasource validations) and "revisions to normalized VegCore" (part of documentation)
- 04:44 PM Revision 11286: bin/import_all: use just import_scrub, not reimport_scrub, because import_scrub now automatically publishes the datasource's import (i.e. removes the temp suffix)
- 04:43 PM Revision 11285: bugfix: inputs/input.Makefile: import: remove the temp suffix once the import is done, so that the full database import doesn't keep the suffix attached to the datasources that import_all didn't import with reimport. removed unused import_publish target (instead use import_temp to invoke just the import without the temp suffix removal).
- 04:27 PM Revision 11284: planning/timeline/timeline.2013.xls: moved part of "switching to new-style import" under "datasource validations (spot-checking)" because this is necessary to validate CVS
- 04:24 PM Revision 11283: planning/timeline/timeline.2013.xls: moved "simplify process of mapping/adding a new datasource" and "documentation testing" after "usability testing" because these tasks were there to make it possible for people other than me to reload/add to the database, which we have now decided is a lower priority than creating the validated database itself
- 04:14 PM Revision 11282: planning/timeline/timeline.2013.xls: added weeks through the end of the year (12/31)
- 02:00 PM Revision 11281: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: changed dataset definition to the definition in normalized VegCore ("a collection of records from the same place, with the same attribution requirements"), following discussion with Ramona
- 01:13 PM Revision 11280: planning/timeline/timeline.2013.xls: updated for progress
- 12:34 PM Revision 11279: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: updated to Ramona's commented version
- 12:19 AM Revision 11278: inputs/CVS/plot_/map.csv: realLatitude, realLongitude: remapped to UNUSED because these columns are actually empty
10/13/2013
- 07:32 PM Revision 11277: inputs/CVS/taxonObservation_/map.csv: collector_ID: remapped it to UNUSED and removed the join to party via it, like in VegBank
- 07:03 PM Revision 11276: inputs/CVS/: deleted stemLocation_, because the CVS stemLocation table is empty (unlike VegBank)
- 05:26 PM Revision 11275: inputs/CVS/import_order.txt: added plantConcept_/ so it would get automapped after switching to new-style import
- 04:24 PM Revision 11274: inputs/CVS/taxonObservation_/map.csv: denorm_{tri,quad}*: mapped to infraspecificRank*, infraspecificEpithet*
- 04:02 PM Revision 11273: inputs/CVS/taxonObservation_/map.csv: infraspecific ranks: remapped to EQUIV#to:species (which is the speciesBinomial), because these actually contain the full taxonomic name at that rank, like VegBank
- 03:50 PM Revision 11272: inputs/CVS/taxonObservation_/map.csv: genus: documented that unlike VegBank, does *not* include genus author
- 03:47 PM Revision 11271: inputs/CVS/taxonObservation_/map.csv: denorm_* terms _alt-ed with normalized terms: use DUPLICATE#of instead where possible. documented where and why _alt was necessary (this applies to a few rows for division, genus).
- 03:42 PM Revision 11270: bugfix: inputs/CVS/taxonObservation_/map.csv: species: remapped to speciesBinomial, not specificEpithet (like for VegBank). however, note that denorm_species *is* in fact the epithet, unlike VegBank.
- 03:19 PM Revision 11269: fix: inputs/CVS/taxonObservation_/postprocess.sql: removed {} around denorm_genus to match the normalized genus
- 02:28 PM Revision 11268: inputs/CVS/taxonObservation_/map.csv: removed unnecessary _alts for terms that don't have a duplicate denorm_* or hierarchical field
- 02:06 PM Revision 11267: fix: inputs/CVS/taxonObservation_/postprocess.sql: fix 1 row that has denorm_kingdom != Kingdom (i.e. both NOT NULL but not the same)
- 12:46 PM Revision 11266: bugfix: lib/common.Makefile: $(subMake): don't enclose the target in "" because sometimes the target is empty (i.e. `all`), and nothing should be passed to the sub-make
- 12:10 AM Revision 11265: inputs/VegBank/plot_/create.sql: documented runtime (5 min)
- 12:10 AM Revision 11264: bugfix: inputs/CVS/plot_/create.sql: like for VegBank, need to compare place.*PLOT_ID*, not PLOTPLACE_ID, with plot.PLOT_ID
- 12:02 AM Revision 11263: /README.TXT: Single datasource import: added pointer to instructions to remake the analytical DB (also required after single datasource import)
10/12/2013
- 04:59 PM Revision 11262: planning/timeline/timeline.2013.xls: updated for progress
- 04:20 PM Revision 11261: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 04:16 PM Revision 11260: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 03:49 PM Revision 11259: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: run all sync_uploads on the svn working copy using --size-only, because the mtimes are based on when the files were last updated by svn and are not meaningful
- 03:46 PM Revision 11258: /README.TXT: Full database import: On local machine: do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine": removed no longer accurate indicator that these steps are above Full database import, since Full database import is now at the beginning of the file
- 03:05 PM Revision 11257: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 03:03 PM Revision 11256: bugfix: inputs/VegBank/+taxon_observation.**.sample/: renamed to ^taxon_observation.**.sample because a leading + has a special meaning to bash (it indicates a shell option, and you will get an error "invalid option name"), as well as to make (it indicates that a recipe command invokes make recursively)
- 02:27 PM Task #767 (New): autodetect column mismatches
- * if @map.csv@ does not contain the @header.csv@ columns in exactly the right order, this will mess up the import in ...
- 02:14 PM Revision 11255: bugfix: inputs/VegBank/taxon_observation.**/header.csv: updated for observation_/map.csv bugfix, which added new hasobservationsynonym field. this fixes a strange test bug caused by the taxon_observation.**/map.csv column list being mismatched/misaligned with what was in the underlying tables. (column mismatches will often cause unexplainable errors in unrelated sections of code the same way that buffer overflows do in C++.)
- 02:01 PM Revision 11254: bugfix: inputs/VegBank/taxon_observation.**.sample/: renamed to +taxon_observation.**.sample so that the *-expansion of taxon_observation.** doesn't add taxon_observation.**.sample (which causes it to attempt to install taxon_observation.**.sample before taxon_observation.** is installed)
- 12:48 PM Revision 11253: bugfix: *Makefile: recursive invocation of $(MAKE): enclose targets in "" in case they contain *
- 12:12 PM Revision 11252: bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable
- 12:11 PM Revision 11251: bugfix: inputs/input.Makefile: %/uninstall: allow user to set is_view=1 flag to use DROP VIEW instead of DROP TABLE
- 12:08 PM Revision 11250: lib/sh/util.sh: added instructions for making an export only visible locally
10/10/2013
- 06:50 PM Revision 11249: bugfix: inputs/VegBank/observation_/header.csv, map.csv: updated for refresh, which inserts hasobservationsynonym at the end of the observation table
- 05:46 PM Revision 11248: inputs/VegBank/taxon_observation.**.sample/create.sql: reordered columns in the same order as analytical_plot, for easier validation
- 05:46 PM Revision 11247: bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed
- 03:07 PM Revision 11246: web/links/index.htm: updated to Firefox bookmarks. updated favicons.
- 02:57 PM Revision 11245: web/links/index.htm: updated to Firefox bookmarks. sudo: added instructions to turn off incorrect password e-mails.
- 02:40 PM Revision 11244: inputs/VegBank/taxon_observation.**.sample/create.sql: include only the subset of columns that is imported to VegBIEN
- 02:32 PM Revision 11243: inputs/VegBank/taxon_observation.**.sample/test.xml.ref: updated inserted row count (which was most likely generated before the output column names had been set to the input column names)
- 01:55 PM Revision 11242: added inputs/VegBank/verify/input_cols.include.txt, with runscript to generate it
- 01:26 PM Revision 11241: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
- 01:18 PM Revision 11240: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
- 01:11 PM Revision 11239: inputs/VegBank/verify/input_cols.unmapped.txt.run: remove unmapped join columns, since these *would* be included in the extract
- 01:09 PM Revision 11238: inputs/VegBank/verify/input_cols.unmapped.txt.run: take input directly from input_cols.txt to avoid needing to first copy and paste it into input_cols.unmapped.txt
- 01:03 PM Revision 11237: inputs/VegBank/verify/input_cols.unmapped.txt.run: added back deliberately excluded columns (DUPLICATE#of:..., etc.) so that the # of rows in the file can be subtracted from the total # of columns to get the # of input columns that would be included in the extract
- 12:43 PM Revision 11236: bugfix: inputs/input.Makefile: %/VegBIEN.csv: `ln -s` to create VegBIEN.csv: enclose the filenames in "" since they may contain * (e.g. taxon_observation.**)
- 12:23 PM Revision 11235: added inputs/VegBank/verify/input_cols.txt, input_cols.unmapped.txt (with runscript to filter input_cols.unmapped.txt)
- 11:53 AM Revision 11234: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: made Ramona's corrections with track changes turned on. note that you have to use MS Word for this, not LibreOffice, because LibreOffice can't save the table of contents properly in .docx or .doc format (although it can save it in .odt format).
- 08:18 AM Revision 11233: inputs/VegBank/stratum/postprocess.sql: added pkey
- 08:05 AM Revision 11232: inputs/VegBank/taxonobservation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
- 07:54 AM Revision 11231: inputs/VegBank/observation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
- 07:45 AM Revision 11230: inputs/VegBank/import_order.txt: added taxon_observation.**.sample so it will automatically be kept up to date
- 07:32 AM Revision 11229: inputs/VegBank/taxon_observation.**.sample/create.sql: set runtime (1 s)
- 07:30 AM Revision 11228: inputs/VegBank/: added taxon_observation.**.sample subset of plots to use in the validation. this avoids the need to import all of VegBank just to validate a few of the plots.
10/09/2013
- 10:12 PM Revision 11227: bin/map: usage: documented that verbosity > 3 in commit mode turns on debug_temp mode, which creates real tables instead of temp tables
- 09:41 PM Revision 11226: schemas/VegCore/ERD/VegCore.ERD.mwb: georeferencing: documented that this stores location determinations (= VegBIEN.vegpath.org?public.locationplace)
- 06:31 PM Revision 11225: inputs/VegBank/taxon_observation.**/: updated for data refresh
- 06:25 PM Revision 11224: inputs/VegBank/plantconcept_/: mapped columns, since this is now included in import_order.txt and therefore gets processed by the column-renaming runscripts. note that this means that in taxonobservation_/map.csv, the plantconcept_ input column names need to be changed to what they are mapped to.
- 06:16 PM Revision 11223: inputs/VegBank/taxonobservation_/create.sql: updated runtime (20 s)
- 05:54 PM Revision 11222: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 05:51 PM Revision 11221: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: project: allow it to *have* a dataset rather than *being* one, because projects do not always have separate attribution requirements (e.g. in Madidi they don't). this also fixes a bug where projects whose datasets had a parent dataset (e.g. VegBank) would need a dummy event for that parent dataset so that the fkeys on both dataset.parent->dataset *and* event.parent->event would be satisfied. added project__unique constraint on name within dataset.
- 05:38 PM Revision 11220: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event.parent (inherited): require this to be a taxa_sampling_event. documented that this is the taxa_sampling_event for the parent stratum.
- 05:15 PM Revision 11219: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 05:03 PM Revision 11218: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 05:00 PM Revision 11217: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 04:55 PM Revision 11216: schemas/VegCore/ERD/VegCore.ERD.mwb: place_visit.parent (inherited): added fkey constraint that this must be a place_visit. documented that this is the concurrent observation of the parent place.
- 04:47 PM Revision 11215: schemas/VegCore/ERD/VegCore.ERD.mwb: project: inherit from dataset, since projects generally confer attribution and conditions of use
- 04:26 PM Revision 11214: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: place_visit: fixed synonyms list to "VegX.vegpath.org?plotObservation, VegBank.vegpath.org?observation (which was confusingly named) + some of VegBank.vegpath.org?plot (which is actually 1:1 with observation)"
- 04:22 PM Revision 11213: schemas/VegCore/ERD/VegCore.ERD.mwb: place_observation: renamed to place_visit for clarity, since other tables that point to this use it as a general container, not just a link to the observations about the place
- 04:16 PM Revision 11212: schemas/VegCore/ERD/VegCore.ERD.mwb: merged stratum_event into taxa_sampling_event since they serve the same purpose (separating taxa samplings within the same place_observation by subsetting methodology). merged layer into stratum since strata as we think of them are horizontal, although there are other definitions of stratum (e.g. size class) which are more similar to our method table.
- 03:51 PM Revision 11211: schemas/VegCore/ERD/VegCore.ERD.mwb: event subclasses: subject: added applicable fkey constraints to the inherited field. renamed existing table-specific subject fields to subject to merge with the inherited field.
- 03:38 PM Revision 11210: schemas/VegCore/ERD/VegCore.ERD.mwb: event: added unique constraints for every way that the scoping fields can meaningfully be combined. these correspond to the various unique constraints offered by VegBIEN.locationevent. (the event table's constraints are always tricky because there are many ways to define an event. it is important to include all of them so that there is almost always a way to generate a natural key, without resorting to an artificial key from traceable.id_by_source.)
- 03:28 PM Revision 11209: schemas/VegCore/ERD/VegCore.ERD.mwb: event.place: renamed to subject and allow this to be any traceable, not just a place. this allows taxon_observations to be scoped within their taxon_occurrences just as subplot events are scoped within parent plot events.
- 02:46 PM Revision 11208: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.authors: documented that this makes traceable mutually recursive with party_list
- 02:45 PM Revision 11207: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.authors: documented that this makes traceable mutually recursive with party_list
- 02:42 PM Revision 11206: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed inheritance connectors
- 02:38 PM Revision 11205: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 02:35 PM Revision 11204: schemas/VegCore/ERD/VegCore.ERD.mwb: record.authors: moved to traceable because actually, different sections of a denormalized source record may have separate authors (e.g. participants for taxa_sampling_event, identifiers for taxon_determination)
- 02:24 PM Revision 11203: schemas/VegCore/ERD/VegCore.ERD.mwb: record.dataset: split this into the scoping_dataset (which scopes the id_within_dataset) and the attribution_dataset (which provides attribution information)
- 01:55 PM Revision 11202: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 01:45 PM Revision 11201: schemas/VegCore/ERD/VegCore.ERD.mwb: record.authors: changed this to refer *only* to original data creators such as collectors and identifiers. this way, there is no need to generate a combined party_list of owners+authors for every record, because the record-specific authors list can just point to the same party_list as the collectors or event participants. this is also closer to Brad's meaning, which is specifically "a person or institution responsible for *creating* the dataset" [+emph].
- 01:31 PM Revision 11200: schemas/VegCore/ERD/VegCore.ERD.mwb: party_list: added array field, which stores an array combining all the list entries. if specified, a trigger will autogenerate party_list_entries from this field. if not specified, it will be autopopulated whenever a party_list_entry is added.
- 12:59 PM Revision 11199: schemas/VegCore/ERD/VegCore.ERD.mwb: added data_owner subclass of party_list_entry, which stores use_conditions for each data_owner of a dataset. dataset: added access_rights (public, private, etc.), use_conditions (citation requirements, etc.).
- 12:25 PM Revision 11198: schemas/VegCore/ERD/VegCore.ERD.mwb.run: record.authors: clarified that this should include *all* parties who helped create the data record, making it a *superset* of dataset.data_owners, who receive attribution. clarified that *record-specific* authors such as collectors and identifiers generally do not receive attribution, although *dataset-level* authors do (this field includes both).
- 12:07 PM Revision 11197: schemas/VegCore/ERD/VegCore.ERD.mwb.run: record.id_within_dataset: removed instruction to also put this in source.name, because source.name is now optional, and this would not make sense because source does not have a parent pointer
- 12:03 PM Revision 11196: schemas/VegCore/ERD/VegCore.ERD.mwb.run: documented that schema COMMENTs cannot contain ' (used in contractions such as "don't"), because MySQL Workbench cannot escape these properly
- 12:02 PM Revision 11195: schemas/VegCore/ERD/VegCore.ERD.mwb: record: added authors, which stores the people who helped create the data record. this is Brad's data author term (Brad.vegpath.org?attribution.data_author), but actually belongs to the individual data *record* rather than to an entire data*set* because records may have different authors (collectors, identifiers, etc.). note that schema COMMENTs apparently cannot contain ' (used in contractions such as "don't"), because MySQL Workbench cannot escape these properly.
- 11:38 AM Revision 11194: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.data_owners: documented that this includes all aggregators through which the data has passed, so that this is equivalent to Brad's data indexers (Brad.vegpath.org?attribution.data_indexer)
- 11:31 AM Revision 11193: fix: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.parent: fixed example to indicate that the GBIF.ARIZ dataset is contained within a specific GBIF *dataset* (the parent), but was obtained from the GBIF *organization* (the dataset_source). ARIZ is then the first_publisher, not the dataset_source.
- 11:22 AM Revision 11192: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: renamed data_source to dataset_source for clarity
- 11:14 AM Revision 11191: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.contacts: documented that this is the contact person(s) to ask about attribution questions. there should always be at least one contact person. if unknown, the person loading the data should set this to themselves.
- 11:12 AM Revision 11190: added schemas/VegCore/Brad_Boyle/BIEN 3 data use and attribution.docx symlink to schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx, so that all of Brad's attribution-related terms are in one place
- 11:07 AM Revision 11189: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.contacts: make this required, so that anyone adding data to a VegCore DB always provides at least one contact person, possibly themselves, to ask about attribution questions (as suggested by Brad in BIEN 3 data use and attribution.docx)
- 11:00 AM Revision 11188: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset.first_publisher: allow this to be any kind of source, rather than just a party (i.e. an organization). this is possible now that party inherits from source.
- 10:52 AM Revision 11187: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: linked to Brad's synonyms for dataset_source (Brad.vegpath.org?attribution.proximate_data_provider), first_publisher (Brad.vegpath.org?attribution.data_publisher, Brad.vegpath.org?data_provenance.primarydataprovider)
- 10:43 AM Revision 11186: bugfix: web/people/Brad_Boyle/.htaccess: only forward requests for a term defined by Brad to the attribution/ subdir, so that Brad.vegpath.org shows a directory listing instead of forwarding right to the attribution doc
- 10:33 AM Revision 11185: bugfix: web/people/Brad_Boyle/.htaccess: set %{ENV:innermost_dir} to Brad_Boyle
- 10:28 AM Revision 11184: added web/people/Brad_Boyle/data_provenance/ (BIEN database entities related to data provenance and ownership.docx)
- 09:56 AM Revision 11183: bugfix: web/people/Brad_Boyle/attribution/.htaccess: \-escape the %s so that they aren't treated as backreferences. set the [redirect] flag so that the downloaded file has the correct filename (BIEN 3 data use and attribution.docx instead of attribution).
- 09:49 AM Revision 11182: bugfix: web/people/Brad_Boyle/attribution/.htaccess: use %20 instead of + because the Apache server apparently doesn't translate + to space correctly
- 09:47 AM Revision 11181: bugfix: web/people/Brad_Boyle/attribution/.htaccess: link to the VegBIEN subdir in /exchange_schemas instead of the main /VegBIEN because there are actually two VegBIEN subdirs, and the main one instead points to /datasources/VegBIEN
- 09:45 AM Revision 11180: added web/people/Brad_Boyle/ with attribution/ subdir for BIEN 3 data use and attribution.docx
- 09:15 AM Revision 11179: schemas/VegCore/ERD/VegCore.ERD.mwb: dataset: renamed data_source to dataset_source for clarity
- 08:54 AM Revision 11178: inputs/VegBank/plantconcept_/create.sql: documented runtime (21 min)
- 08:28 AM Revision 11177: bugfix: inputs/VegBank/plantconcept_/: added new-style import files
- 08:27 AM Revision 11176: bugfix: inputs/VegBank/import_order.txt: added plantconcept_, because new-style import needs it to be explicitly listed in import_order.txt in order to run it
- 08:24 AM Revision 11175: inputs/VegBank/run: refresh(): added usage
- 08:22 AM Revision 11174: inputs/VegBank/run: refresh(): documented that this should be run on *vegbiendev*
- 07:58 AM Revision 11173: inputs/VegBank/_archive/2012-8-30/: svn:ignore the data exports
- 07:57 AM Revision 11172: inputs/VegBank/run: added refresh() target
- 07:54 AM Revision 11171: inputs/VegBank/: refreshed VegBank so that all of Mike Lee's sample plots would be included in the input data. (VegBank was last refreshed from the live DB on 2012-8-30.) split vegbank.sql into vegbank.schema.sql and vegbank.data.sql so that the schema can be examined and imported separately, like for MySQL datasources. inputs/VegBank/vegbank.~.clean_up.sql: commented out setting comminterpretation.commname to NOT NULL, because after the refresh it is now NULL in 10 rows, where commconcept_id is also NULL.
10/08/2013
- 05:31 PM Revision 11170: lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions
- 05:29 PM Revision 11169: lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir
- 12:36 PM Revision 11168: inputs/.geoscrub/_src/README.TXT: added e-mail from John Donoghue with general description of the BIEN2 geovalidation workflow
10/03/2013
- 04:04 PM Revision 11167: bugfix: schemas/vegbien.sql: plot.**: removed ORDER BY because a SELECT on the view with its own ORDER BY will actually sort the rows *twice* instead of optimizing away plot.**'s default ORDER BY
- 03:44 PM Revision 11166: schemas/vegbien.sql: added plot.** subset view of analytical_stem_view, used for VegBank validation
- 01:41 PM Revision 11165: planning/timeline/timeline.2013.xls: moved reload core & analytical database out a week because it shouldn't be done until after CVS validation is complete
- 01:10 PM Task #294 (Resolved): find plot data source provider to work with Aaron
- 01:10 PM Task #297 (Resolved): Request new data dump of specimen data from GBIF, this time in DwC format
- 01:10 PM Task #301 (Resolved): RAINFOR data
- part of SALVIAS
- 01:09 PM Task #305 (Resolved): Complete full-dataset validations for NYBG & SALVIAS
- 01:09 PM Task #306 (Resolved): Acquire additional plot data sets from providers willing to work with Aaron on mappings and validations
- 01:09 PM Task #307 (Resolved): Acquire additional specimen data sets in both DwC and DwCA format, esp. GBIF
- 01:09 PM Task #312 (Resolved): Finish importing SALVIAS data
- 01:08 PM Task #320 (Rejected): Convert user-defined VegX fields to first-class fields
- using [[VegCore]] instead
- 01:08 PM Task #324 (Resolved): NYBG validations
- 01:07 PM Task #333 (Resolved): add traits table to VegBIEN
- 01:07 PM Task #344 (Resolved): contact John about GNRS/georeferencing
- 01:07 PM Task #345: integrate GNRS into VegBIEN
- prototyped in [[VegCore]]
- 01:06 PM Task #346: add georeferencing support to schema
- prototyped in [[VegCore]]
- 01:06 PM Task #377 (Resolved): ask NYBG for direct access to server
- 01:03 PM Task #466 (Rejected): column-based import: avoid most exceptions by introspecting the DB structure instead of waiting for a parseable exception to be raised
- easier to handle exceptions than trying to anticipate them
- 01:02 PM Task #467: underline all to do items in meetings notes
- completed items have been crossed out; remaining tasks can then be underlined
- 01:01 PM Task #482: translate README.TXT to wiki page
- see [[Import process for Madidi]]
- 01:00 PM Task #544: integrate creation of analytical DB into automated testing
- it's part of the extract creation for datasource validations
- 01:00 PM Task #560: move VegCore data dictionary to a phpPgAdmin-accessible database
- see "VegCore ERD":http://vegbiendev.nceas.ucsb.edu/VegCore/ERD/
- 12:58 PM Task #562 (Rejected): flatten the mappings
- autopopulated natural pkeys actually do require the nested mappings, but these can be part of the schema instead
- 12:56 PM Task #577: use views instead of map spreadsheets to store the datasource mappings
- map spreadsheets copied to tables in [[new-style import]]
- 12:25 PM Revision 11164: inputs/.geoscrub/_src/README.TXT: added link to geovalidation description in wiki
- 11:03 AM Revision 11163: added schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx from Brad
- 10:03 AM Revision 11162: planning/timeline/timeline.2013.xls: updated to reflect that the next week should be spent doing data validations, as decided in the conference call
10/02/2013
- 05:59 AM Revision 11161: bugfix: schemas/vegbien.sql: locationevent__communities(): need to use commdescription instead of commname as the communityName because that's where communityName has been mapped to in mappings/VegCore-VegBIEN.csv. (commname instead stores the communityID.)
- 05:56 AM Revision 11160: inputs/VegBank/: added observation__community/
- 05:49 AM Revision 11159: inputs/VegBank/vegbank.~.clean_up.sql: commclass.commcode,commname: rename to prevent collisions
- 04:26 AM Revision 11158: inputs/VegBank/vegbank.~.clean_up.sql: indicate required column comminterpretation.commname
- 04:20 AM Revision 11157: inputs/VegBank/vegbank.~.clean_up.sql: commconcept.commname: rename to prevent collision with commname.commname
- 04:15 AM Revision 11156: planning/timeline/: svn:ignore: added .~* (lock files for .xls)
- 03:58 AM Revision 11155: schemas/vegbien.sql: analytical_stem_view: added instructions for what to do after updating this
- 03:55 AM Revision 11154: schemas/vegbien.sql: analytical_stem_view, analytical_plot: added communities using new locationevent__communities__array()
- 03:52 AM Revision 11153: schemas/vegbien.sql: added locationevent__communities*()
- 03:10 AM Revision 11152: bugfix: schemas/vegbien.sql: locationevent: locationevent__pull_forward_from_parent() trigger: only pull forward the location_id when there is a stratum_id, so that empty locationevents that should be auto-forwarded are not unintentionally made valid by populating the location_id field (which causes a check constraint to become satisfied, and prevents auto-forwarding from working). this fixes a bug in some datasources' automated tests, such as `make inputs/SALVIAS/plotObservations/test`.
- 02:45 AM Revision 11151: lib/sql_io.py: put_table(): default param: documented that this will be used for *all* missing rows, regardless of which error caused them not to be inserted. this means that auto-forwarding (wiki.vegpath.org/Auto-forwarding) can be used with *any* type of constraint violation, not just NOT NULL constraints (which it is typically used with).
10/01/2013
09/30/2013
- 07:47 AM Revision 11149: planning/timeline/timeline.2013.xls: updated for progress
- 07:38 AM Revision 11148: bugfix: schemas/vegbien.sql: analytical_stem_view: plot__collectors: to include plot__collectors for nested stratum events, need to calculate the locationevent__contributors__array() for the locationevent and parent_event *separately*, and then COALESCE those, instead of just running locationevent__contributors__array() once on the COALESCE of the two IDs
- 07:34 AM Revision 11147: bugfix: schemas/vegbien.sql: locationevent: added pull-forward trigger to populate location_id from the parent event. a pull-forward trigger *must* be used for this (not a COALESCE() in analytical_stem_view) because the locationevent cannot even be found by analytical_stem_view if it does not have an associated location.
- 06:37 AM Revision 11146: schemas/VegCore/ERD/VegCore.ERD.mwb: source: added required url field, which points to the source data and uniquely identifies the source. note that this differs from the id, which is a *self*-refential pkey (as described at wiki.vegpath.org/Web_interface#Clickable-fkeys).
- 05:47 AM Revision 11145: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 05:38 AM Revision 11144: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 05:32 AM Revision 11143: schemas/VegCore/ERD/VegCore.ERD.mwb: source: removed canon, because this would create too much complexity as a result of having multiple copies of the same source with different names. instead, source names will be scrubbed *upon insert* so that the row always contains the most accurate data, and there is only one copy of each source. this scrubbing may use the assistance of a thesaurus table w/ standard abbreviations, etc.
- 05:07 AM Revision 11142: schemas/VegCore/ERD/VegCore.ERD.mwb: plot.name: merged with inherited name field
- 04:43 AM Revision 11141: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 04:40 AM Revision 11140: schemas/VegCore/ERD/VegCore.ERD.mwb: source: added canon field. when there are multiple names for a source, this points to the canonically-named source. by default, points recursively to itself.
- 04:04 AM Revision 11139: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.taxon_occurrence: documented that when not provided (and not fillable from another field), a taxon_occurrence will be created with the same id as the taxon_observation. this operation should eventually be perfomed by a trigger.
- 03:58 AM Revision 11138: schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place.parent_geoplace: renamed to parent so it will merge with the inherited parent field. this is the parent geoplace.
- 03:27 AM Revision 11137: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:20 AM Revision 11136: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 03:18 AM Revision 11135: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that instead, the taxon_occurrence for the individual_observation should have an autogenerated current_observation that merges together all the individual_observations for these institutions' specimens. in this case, each institution's specimen will still have its *own* individual_observation containing what *it* believes to be the specimenholder institutions.
- 03:03 AM Revision 11134: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.specimenholder_institutions: documented that when provided, the source (i.e. traceable.source) should be set to the value of this field, because these institutions together provided the data for this entity
- 02:59 AM Revision 11133: schemas/VegCore/ERD/VegCore.ERD.mwb: party_list: inherit from source so that a party_list can be used as the source for a row (e.g. a list of institutions that together provided the data for an individual_observation that has specimens at multiple institutions)
- 02:55 AM Revision 11132: schemas/VegCore/ERD/VegCore.ERD.mwb: reobservable_presence: merged into taxon_determination, so that voucher is directly available to taxon_determination to autopopulate fields when a redetermination is made on the voucher
- 02:37 AM Revision 11131: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 02:36 AM Revision 11130: schemas/VegCore/ERD/VegCore.ERD.mwb: aggregate_observation.sampling_event: merged with inherited sampling_event field
- 02:33 AM Revision 11129: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.parent: require that this is an fkey to another taxon_observation, since this stores the *taxon_observation* of the same taxon_occurrence in the parent sampling_event
- 02:22 AM Revision 11128: schemas/VegCore/ERD/VegCore.ERD.mwb: subplot.parent_plot: renamed to parent so it will merge with the inherited parent field. this is the parent plot.
- 02:16 AM Revision 11127: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_count.taxon_presence: renamed to parent so it will merge with the inherited parent field. this is the taxon_presence for the taxon as a whole (not subdivided by size_class, etc.).
- 02:13 AM Revision 11126: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event.method: renamed to subsetting_method for clarity
- 02:06 AM Revision 11125: schemas/VegCore/ERD/VegCore.ERD.mwb: stratum_event.parent_event: renamed to parent so it will merge with the inherited parent field
- 01:58 AM Revision 11124: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable: added alternate pkey id_by_source, which allows referencing a traceable by its source information even when the subclass chose a different value for the pkey
- 01:49 AM Revision 11123: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: merged with inherited parent field, documenting that for this table, this contains the observation of the same taxon_occurrence in the parent sampling_event
- 01:33 AM Revision 11122: schemas/VegCore/ERD/VegCore.ERD.mwb: fixed lines
- 01:24 AM Revision 11121: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation.place_observed_at: merged with inherited place field, documenting that for this table, this contains the place the individual was observed at
- 01:16 AM Revision 11120: schemas/VegCore/ERD/VegCore.ERD.mwb: event: reordered date_range, participants, place in the order of how likely they are to be provided. (place was considered more optional than the other two because a project does not usually have a place.)
- 01:09 AM Revision 11119: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.sampling_event: made it required so that there is always a sampling_event to scope the primary_collector/collector_number when provided. (for specimens, this will have the same event information (date_range, participants, place) as the taxon_observation.)
- 12:35 AM Revision 11118: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added unique constraint on sampling_event, primary_collector, collector_number. reordered these columns in constraint order (path order).
09/29/2013
- 11:35 PM Revision 11117: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation: added primary_collector, to scope the collector_number. collectors: renamed to all_collectors for clarity, now that there is a primary_collector field.
- 11:14 PM Revision 11116: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.current_collection, owner_collection: moved to specimen_observation because these may differ each time the specimen is reobserved
- 11:08 PM Revision 11115: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.code_in_individual: renamed to id_within_individual to match the id_within_* naming convention using elsewhere
- 10:58 PM Revision 11114: schemas/VegCore/ERD/VegCore.ERD.mwb: individual_observation: removed code because this duplicates taxon_observation.collector_number and record.id_within_dataset
- 10:53 PM Revision 11113: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen.specimenholder_institutions: moved to individual_observation because the list of institutions which store a specimen from the plant actually relates to the plant, not a particular specimen (this list should be the same for all specimens taken from the plant)
- 10:34 PM Revision 11112: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed collection_event because this is the same as individual_observation
- 10:11 PM Revision 11111: schemas/VegCore/ERD/VegCore.ERD.mwb: specimen: removed separate fkey to individual because this is linked to from individual_observation
- 09:55 PM Revision 11110: schemas/VegCore/ERD/VegCore.ERD.mwb: place_observation: link to project. taxa_sampling_event: removed link to project since this now happens via place_observation.
- 09:35 PM Revision 11109: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: merged required within_place into place_observation, which already contains a required fkey to place
- 09:29 PM Revision 11108: schemas/VegCore/ERD/VegCore.ERD.mwb: taxa_sampling_event: added place_observation fkey for the concurrent place_observation, which contains all the info about the place (communities, soil data, etc.). this allows a taxon_observation to be traced to its associated location measurements.
- 08:58 PM Revision 11107: bugfix: mappings/VegCore-VegBIEN.csv: nest all taxonoccurrences inside a stratum event, so that the parent locationevent is *always* fully populated before child locationevents point to it. (previously, a stub parent event was created when the child event was imported first, which blocked the fully-populated parent event from being inserted later on.) this uses auto-folding (for VegBank/CVS) and auto-forwarding (for other datasources) to prune empty stratum events for taxonoccurrences that don't have strata. (see wiki.vegpath.org/Auto-folding, wiki.vegpath.org/Auto-forwarding for more info about these normalization techniques.) note that the inserted row counts stay *exactly the same* for all datasources except VegBank (which was being fixed), indicating that this signficant change to the mappings did *not* change the semantics of the import of taxonoccurrences.
- 08:37 PM Revision 11106: inputs/VegBank/observationcontributor_/test.xml.ref: updated inserted row count
09/28/2013
- 10:40 PM Revision 11105: bugfix: mappings/VegCore-VegBIEN.csv: stratum's locationevent: link this to the parent locationevent, so that the parent locationevent's information (such as locationeventcontributors) is accessible to the stratum's locationevent
- 09:08 PM Revision 11104: bugfix: inputs/VegBank/taxon_observation.**/postprocess.sql: inlined _join() so that taxon_observation.** wouldn't get cascadingly deleted whenever the util schema (where this normally resides) gets reinstalled
- 08:23 PM Revision 11103: bugfix: schemas/vegbien.sql: locationeventcontributor: added missing unique constraint (locationeventcontributor_unique)
- 08:21 PM Revision 11102: bugfix: schemas/vegbien.sql: locationeventcontributor.role: made it optional because some locationeventcontributors may not have an associated role
- 06:45 PM Revision 11101: schemas/vegbien.sql: analytical_stem_view, analytical_plot: added plot__collectors, which uses locationevent__contributors__array()
- 06:42 PM Revision 11100: schemas/vegbien.sql: locationevent__contributors(): split into set-returning and array-returning functions. sort results by fullname so there is a predictable ordering (VegBank does not include a sort_order in the observationContributor table).
- 05:56 PM Revision 11099: schemas/vegbien.sql: added locationevent__contributors() aggregating function for use in analytical_stem_view
- 06:53 AM Revision 11098: added inputs/VegBank/observationcontributor_/
- 06:51 AM Revision 11097: mappings/VegCore-VegBIEN.csv: mapped event__participant
- 05:51 AM Revision 11096: mappings/VegCore.htm: regenerated from wiki. added EventParticipant table containing event__participant.
- 05:04 AM Revision 11095: bugfix: inputs/VegBank/: taxonOccurrenceID: include the aggregateOrganismObservationID in this so that there is one taxonoccurrence for *each* stratum's taxonImportance. this allows the different strata to have separate taxonoccurrences that are associated with the stratum-specific locationevents, rather than all being lumped into one taxonoccurrence, with only one locationevent.
- 12:06 AM Revision 11094: schemas/VegCore/ERD/VegCore.ERD.mwb: taxon_observation.observation_in_parent_place: renamed to observation_in_parent_event because the taxon_observation might be connected to a stratum_event which is nested within another sampling_event before it's connected to any sampling_event for a parent plot. this is the case for VegBank data with associated strata.
09/27/2013
- 11:14 PM Revision 11093: bugfix: schemas/vegbien.sql: locationevent: locationevent_unique_within_creator unique index on sourceaccessioncode: need to add stratum_id, because the addition of stratum_id to *this* table instead of a separate stratum_event table (which would have required more schema changes) means we need to use the awkward construction of appending stratum_id to *every* unique constraint, including ones that ordinarily would just pass through the datasource's pkey
- 06:31 PM Revision 11092: web/links/index.htm: updated to Firefox bookmarks. added Java client- and server-side links. web: split into client- and server-side folders.
- 06:29 PM Revision 11091: planning/timeline/timeline.2013.xls: updated for progress
- 03:43 PM Revision 11090: /README.TXT: Datasource setup: added link to Example steps for a datasource (wiki.vegpath.org/Import_process_for_Madidi)
- 03:23 PM Revision 11089: /README.TXT: Full database import: To remake analytical DB: added runtime (13 h)
Also available in: Atom