Activity
From 10/10/2013 to 11/08/2013
11/08/2013
- 10:57 PM Revision 11603: bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)
- 10:51 PM Revision 11602: lib/runscripts/table.run: reordered functions in the order they are called by import()
- 10:28 PM Revision 11601: bugfix: inputs/VegBank/: need to remove inter-datasource duplicates from plot instead of the left-joined plot_ table, because the fkeys needed to do the cascading deletes are all to the plot table. this requires doing the column-renaming and postprocessing on plot *before* it's left-joined.
- 10:10 PM Task #815 (New): auto-prepend table name to columns
- * this avoids the need to add manual renames for every column just to prepend the table name
- 10:08 PM Task #814 (New): rename columns by name, not position
- * this avoids frameshift bugs when columns are added/removed
** this is especially a problem when adding source-spec... - 09:57 PM Revision 11600: inputs/VegBank/plot_/create.sql: updated runtime (5 s) for previous bugfix
- 07:50 PM Revision 11599: exports/2013-7-10.Naia.range_limiting_factors.csv.run: updated export_() runtime and rowcount (~ the same)
- 04:26 PM Revision 11598: bugfix: schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: filter by coordinateUncertaintyInMeters filter: assume true for rows with no coordinateUncertaintyInMeters
- 03:43 PM Revision 11597: schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: filter by coordinateUncertaintyInMeters <= 10 km
11/07/2013
- 04:41 PM Revision 11596: planning/timeline/timeline.2013.xls: updated for progress
- 04:00 PM Revision 11595: inputs/.geoscrub/geoscrub_output/run: load_data(): updated runtime (4 min)
- 08:42 AM Revision 11594: planning/timeline/timeline.2013.xls: updated for progress
- 08:34 AM Revision 11593: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: invoking derived/biengeo/geoscrub.sh: need to split the input file into separate dir and filename parts, because $DATAFILE actually is just the filename, not the entire path, and will otherwise get prepended with the default value for $DATADIR
11/06/2013
- 04:57 PM Revision 11592: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: also run geoscrub.sh. added export_() target to run just the export of the result table separately.
- 04:39 PM Revision 11591: derived/biengeo/load-geoscrub-input.sh: allow the caller to override $DATAFILE in the environment, to use a file named other than "geoscrub-corpus.csv"
- 02:41 PM Revision 11590: /run: use new exports/geoscrub_input.csv.run
- 02:40 PM Revision 11589: added exports/geoscrub_input.csv.run
- 02:39 PM Revision 11588: bugfix: lib/sh/make.sh: $remake: need to explicitly propagate this to invoked commands if it was set from $rm
- 12:34 PM Revision 11587: derived/biengeo/load-geoscrub-input.sh: updated $DATA_URL for new input filename
- 12:27 PM Revision 11586: /run geoscrub_input/make(): include a header on the CSV file, so that the column names don't risk getting spliced from the data (and to shorten the CSV filename, which had to contain the column names instead). this requires changing the geoscrubbing scripts to accept a CSV header.
- 11:22 AM Revision 11585: planning/timeline/timeline.2013.xls: updated for progress
- 10:14 AM Revision 11584: exports/2013-7-10.Naia.range_limiting_factors.csv.run: added rowcount (40 million of 80 million observations, filtered w/ cultivated, geovalid, and various fields NOT NULL)
- 01:46 AM Revision 11583: exports/2013-7-10.Naia.range_limiting_factors.csv.run: updated export_() runtime
- 01:30 AM Revision 11582: schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: don't sort the results by occurrence_id, because this is not a meaningful ordering and prevents incremental output from the query
- 01:09 AM Revision 11581: schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: also filter out rows without species
- 12:59 AM Revision 11580: exports/2013-7-10.Naia.range_limiting_factors.csv.run: export_(): documented runtime (10 min)
11/05/2013
- 11:13 PM Revision 11579: lib/sh/db.sh: mk_select(): usage: documented that this also takes a $limit/$n param
- 11:12 PM Revision 11578: lib/sh/db.sh: limit(): also support using $n as the limit param, since this var name is used by other parts of the import process
- 11:08 PM Revision 11577: added backups/vegbien.r11549.backup.md5
- 11:07 PM Revision 11576: lib/sh/db.sh: limit(): usage: documented that this also need a $limit param
- 11:06 PM Revision 11575: backups/TNRS.backup.md5: updated
- 10:47 PM Revision 11574: lib/runscripts/extract.run: added export_sample()
- 10:31 PM Revision 11573: /README.TXT: Full database import: after import: record the import times in inputs/import.stats.xls: documented that this should be run on the local machine, because it needs the Mac filename ordering
- 10:30 PM Revision 11572: planning/timeline/timeline.2013.xls: updated for progress
- 10:19 PM Revision 11571: inputs/import.stats.xls: updated import times
- 08:54 PM Revision 11570: /README.TXT: Full database import: after import: removed step to install analytical_stem on nimoy because the import mechanism is not set up to do this (we don't generate CSV exports of the full analytical_stem table because they take up a lot of space and are not currently used for anything)
- 08:32 PM Revision 11569: /README.TXT: Full database import: after import: In PostgreSQL: added step to check that analytical_stem contains the expected # of rows
- 08:16 PM Revision 11568: /README.TXT: Full database import: after import: In PostgreSQL: added specific instructions for determining which/how many datasources are expected to be included in the provider_count and source tables
- 08:05 PM Revision 11567: added inputs/analytical_db/_archive/
- 07:46 PM Revision 11566: inputs/analytical_db/: removed import-related files (Source/, etc.), since this is actually just a folder used to store make_analytical_db.log.sql, so that it will be synced along with the other logs
- 07:43 PM Revision 11565: inputs/analytical_db/: added _no_import to prevent this from incorrectly being included in the source table
- 07:27 PM Revision 11564: inputs/input.Makefile: $(_svnFilesGlob): also svn-add _no_import in the top-level datasrc dir. (this requires using add! , because the presence of a _no_import file there will normally turn off adding by svnFilesGlob.)
- 11:49 AM Revision 11563: Added an output CSV file option to geoscrub.sh.
11/04/2013
10/31/2013
- 05:35 PM Revision 11561: Added biengeo script options for data directories.
- Added GADM and geonames.org data dir options to
update_validation_data.sh scripts.
Added geoscrub input data dir opti... - 05:35 PM Revision 11560: Added update options to biengeo update_validation_data.sh
- Added options to update only GADM data, only Geonames.org data, or
neither. In every case, the geonames-to-gadm scrip... - 05:35 PM Revision 11559: Added cmd-line options to biengeo bash scripts.
- All biengeo bash scripts now accept command line options to specify psql
user, host, and database values.
These optio... - 05:35 PM Revision 11558: Fix biengeo script password prompt for postgres user.
- Changed the DB_HOST variables in the biengeo bash scripts to a
DB_HOST_OPT variable that is blank by default.
Updated... - 05:35 PM Revision 11557: Fixed TRUNCATE statement in truncate.geonames.sql.
- Fixed the biengeo truncate.geonames.sql script to include all tables in
one TRUNCATE statement that have foreign-key ... - 05:35 PM Revision 11556: Added more approx. runtimes to biengeo README.
- 05:35 PM Revision 11555: Renamed biengeo install scripts to setup scripts.
- It seems to make more sense to call these setup scripts, since they are
only setting up the database and tables, and ... - 12:29 PM Revision 11554: planning/timeline/timeline.2013.xls: updated for progress
- 12:24 PM Revision 11553: planning/timeline/timeline.2013.xls: datasource validations: CVS: left-join it: moved under "fix issues and critical feature requests" instead of "prepare 1st-round extracts" because the left-joining is actually part of getting it in the same format as VegBank
- 11:12 AM Revision 11552: inputs/CTFS/StemObservation/test.xml.ref: updated inserted row count
- 10:30 AM Revision 11551: planning/timeline/timeline.2013.xls: datasource validations: rescheduled CVS before other datasources, as decided in the conference call
- 10:27 AM Revision 11550: schemas/Makefile: $(confirmRmPublicSchema0): use "any ... schema" instead of "the ... schema" because the schema in question may not exist
- 08:53 AM Revision 11549: planning/timeline/timeline.2013.xls: datasource validations: rescheduled tasks for new order
- 08:42 AM Revision 11548: planning/timeline/timeline.2013.xls: datasource validations: reordered to put plots before specimens, as requested by Brad (wiki.vegpath.org/2013-10-25_conference_call#validation-order)
- 08:24 AM Revision 11547: planning/timeline/timeline.2013.xls: updated for progress
- 08:21 AM Revision 11546: planning/timeline/timeline.2013.xls: hid previous weeks
- 08:20 AM Revision 11545: planning/timeline/timeline.2013.xls: crossed out and hid completed tasks
- 08:17 AM Revision 11544: fix: planning/timeline/timeline.2013.xls: datasource validations: prepare 2nd-round extracts: VegBank: corrected check mark week, based on date of extract
- 08:14 AM Revision 11543: planning/timeline/timeline.2013.xls: datasource validations: added "prepare 3rd-round extracts" subtask, which currently applies to VegBank. updated for progress.
- 08:08 AM Revision 11542: planning/timeline/timeline.2013.xls: "datasource validations (spot-checking)": renamed to just "datasource validations" because that's what we've been calling it
- 08:08 AM Revision 11541: planning/timeline/timeline.2013.xls: datasource validations: CVS: added "VegBank-related changes" subtask
- 08:05 AM Revision 11540: planning/timeline/timeline.2013.xls: updated for progress and revised schedule
- 07:51 AM Revision 11539: bugfix: inputs/VegBank/import_order.txt: updated name of ^taxon_observation.**.sample table
- 07:16 AM Revision 11538: fix: inputs/VegBank/^taxon_observation.**.sample/create.sql: moved continent before country
- 06:54 AM Revision 11537: inputs/VegBank/^taxon_observation.**.sample/create.sql: added missing columns that were recently mapped to VegBIEN (identifiedBy)
- 06:52 AM Revision 11536: inputs/VegBank/^taxon_observation.**.sample/create.sql: synced column order to analytical_plot
- 06:49 AM Revision 11535: inputs/VegBank/^taxon_observation.**.sample/create.sql: synced column order to analytical_plot
- 06:47 AM Revision 11534: inputs/VegBank/taxonobservation_/map.csv, postprocess.sql: mapped identifiedBy (the _join_words() of identifiedBy__first, etc.)
- 06:22 AM Revision 11533: fix: schemas/vegbien.sql: analytical_plot, analytical_specimen: removed derived columns that are not part of the validation
- 06:15 AM Revision 11532: fix: schemas/vegbien.sql: analytical_plot, analytical_specimen: removed internal ID columns that are not part of the validation
- 05:46 AM Revision 11531: schemas/vegbien.sql: analytical_plot: removed derived columns that should not be validated by data providers
- 05:42 AM Revision 11530: schemas/vegbien.sql: analytical_specimen: synced to analytical_stem
- 05:36 AM Revision 11529: schemas/vegbien.sql: analytical_plot: documented that this contains all of the analytical_stem columns, minus specimenHolderInstitutions, collection, accessionNumber, occurrenceID
- 05:34 AM Revision 11528: schemas/vegbien.sql: analytical_plot: synced to analytical_stem
- 05:29 AM Revision 11527: schemas/vegbien.sql: analytical_stem_view: added individualCount
- 04:42 AM Revision 11526: schemas/vegbien.sql: plot.**, analytical_stem_view: added slopeAspect, slopeGradient
- 03:41 AM Revision 11525: schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.id_by_source: support multiple ids_by_source per traceable, because the same entity may be present in multiple datasources (e.g. if one got data from the other), and we would like to remove that duplicate
- 02:46 AM Revision 11524: inputs/VegBank/taxonobservation_/map.csv, postprocess.sql: mapped identifiedBy (the _join_words() of identifiedBy__first, etc.)
- 02:34 AM Revision 11523: inputs/VegBank/taxonobservation_/create.sql: also join party_id to get the identifiedBy (not mapped yet). note that the inserted row count changes, because taxonobservation_ does not yet have a pkey to do a stable ordering with.
- 02:16 AM Revision 11522: bugfix: inputs/input.Makefile: %/install: don't run map_table, because this instead done by the runscript. although it does not hurt to do it twice, invoking load_data by itself should not run map_table at all, so that the original column names can be inspected in the table and map.csv reordered to match.
- 02:06 AM Revision 11521: inputs/VegBank/vegbank.~.clean_up.sql: taxoninterpretation.party_id: don't rename to taxoninterpretation_party_id, so that this can be used directly in taxonobservation_/create.sql with a USING join
- 01:52 AM Revision 11520: inputs/VegBank/taxonobservation_/create.sql: join taxonobservation to taxoninterpretation (as in CVS) instead of vice versa, since taxonobservation is the primary, operative table. having VegBank and CVS do things the same way helps ensure that fixes in one can transfer easily to the other.
- 01:51 AM Revision 11519: bugfix: inputs/input.Makefile: %/install: don't run map_table, because this instead done by the runscript. although it does not hurt to do it twice, invoking load_data by itself should not run map_table at all, so that the original column names can be inspected in the table and map.csv reordered to match.
- 01:30 AM Revision 11518: inputs/VegBank/^taxon_observation.**.sample/create.sql: synced with taxon_observation.**
- 01:22 AM Revision 11517: (for r11396) fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error
- 12:50 AM Revision 11516: /README.TXT: for each task, documented which machine it's run on. for tasks run on vegbiendev, added pointer to "Connecting to vegbiendev" steps.
- 12:19 AM Revision 11515: /README.TXT: added instructions for connecting to vegbiendev
10/30/2013
- 11:03 PM Revision 11514: mappings/VegCore-VegBIEN.csv: mapped taxon_determination__is_current, taxon_determination__is_original
- 09:49 PM Revision 11513: mappings/VegCore-VegBIEN.csv: mapped taxon_determination__is_current, taxon_determination__is_original
- 09:46 PM Revision 11512: bugfix: mappings/VegCore-VegBIEN.csv: main taxondetermination: use [!isoriginal=true] instead of [!isoriginal] so that adding a manual isoriginal field does not prevent this selector from matching
- 09:07 PM Revision 11511: inputs/VegBank/taxonobservation_/map.csv: originalinterpretation, currentinterpretation: removed table name prefix so these would automap
- 09:06 PM Revision 11510: mappings/VegCore.htm: regenerated from wiki. added taxon_determination__is_current, taxon_determination__is_original.
- 09:02 PM Revision 11509: mappings/VegCore.htm: regenerated from wiki. added taxon_determination__is_current, taxon_determination__is_original.
- 08:07 PM Revision 11508: planning/timeline/timeline.2013.xls: geoscrubbing automated pipeline: split into subtasks "build pipeline", "test pipeline", and "integrate pipeline into import process"
- 08:04 PM Revision 11507: planning/timeline/timeline.2013.xls: geoscrubbing re-run: moved recent checkmarks to "geoscrubbing automated pipeline" since the work on these actually relates to *automating* the geoscrubbing, not the one-time reload (which was already completed)
- 08:02 PM Revision 11506: planning/timeline/timeline.2013.xls: geoscrubbing: made "geoscrubbing re-run" a subtask of the main geoscrubbing task, instead of geoscrubbing re-run being the supertask. updated for Paul's progresss.
- 07:23 PM Revision 11505: schemas/vegbien.sql: taxondetermination_set_iscurrent(): include new iscurrent__verbatim, so that taxondeterminations the datasource marks as current are always considered first. this currently applies to VegBank and CVS.
- 07:17 PM Revision 11504: schemas/vegbien.sql: taxondetermination.isoriginal: made it nullable like iscurrent__verbatim, because this is populated from the datasource. taxondetermination_set_iscurrent() now supports isoriginal=NULL, so this is not a problem.
- 07:08 PM Revision 11503: schemas/vegbien.sql: taxondetermination.is_datasource_current: renamed to iscurrent__verbatim and made it nullable, so that this can be used to store the verbatim iscurrent status
- 07:04 PM Revision 11502: schemas/vegbien.sql: taxondetermination_set_iscurrent(): removed setting of is_datasource_current (which is now the same as iscurrent), so that this can be used to store the verbatim iscurrent status
- 06:59 PM Revision 11501: schemas/vegbien.sql: taxondetermination_set_iscurrent(): isoriginal: make sure it is always either true or false, so that if the NOT NULL constraint on this is ever removed you don't end up with the incorrect sort order false, true, NULL (it should be false=NULL, true)
- 06:42 PM Revision 11500: schemas/vegbien.sql: use plain taxondetermination.iscurrent instead of is_datasource_current since these are now the same
- 06:38 PM Revision 11499: schemas/vegbien.sql: taxondetermination_set_iscurrent(): is_datasource_current: set to the same value as iscurrent, since these now have the same formula
- 06:34 PM Revision 11498: schemas/vegbien.sql: taxondetermination_set_iscurrent(): removed no longer used accepted, matched determinationtypes (for these determinations, left-join to TNRS.ScrubbedTaxon)
- 06:24 PM Revision 11497: Updated biengeo README with new script workflow.
- 06:24 PM Revision 11496: Split geovalidate.sh into install and update scripts.
- Split geovalidate.sh into install.sh and update_gadm_data.sh scripts.
The install.sh script creates the databse and u... - 06:24 PM Revision 11495: Refactored geonames.sh to update_geonames_data.sh
- Renamed geonames.sh to update_geonames_data.sh and moved many of the SQL
statements from the bash script into support... - 06:24 PM Revision 11494: Split up geonames-to-gadm.sql into 3 scripts.
- Each script only operates on one table within a transaction.
These scripts now assume the tables have already been cr... - 06:24 PM Revision 11493: Added geoscrub.sh script.
- This script runs the load-geoscrub-input.sh, geonames.sql, and
geovalidate.sql scripts in order to load and scrub veg... - 06:03 PM Revision 11492: inputs/SALVIAS/projects/postprocess.sql: remove institutions that we have direct data for: documented that most of the 13139 removed plots are from duplicates (where we have direct data). this leaves only 560 of SALVIAS's original 13699 plots.
- 05:53 PM Revision 11491: inputs/SALVIAS/projects/postprocess.sql: remove example data
- 05:48 PM Revision 11490: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible (this was probably already removed by the plotMetadata.AccessCode filter in salvias_plots.~.clean_up.sql)
- 05:44 PM Revision 11489: inputs/SALVIAS/projects/postprocess.sql: remove institutions that we have direct data for (Madidi, VegBank)
- 04:23 PM Revision 11488: bugfix: inputs/VegBank/plot_/postprocess.sql: coordinateUncertaintyInMeters__from_fuzzing: need to convert km to m in the fuzzing radii. updated derived cols runtimes.
- 04:05 PM Revision 11487: inputs/VegBank/plot_/postprocess.sql: remove duplicated CVS plots (2323 of 7079 CVS plots are removed by this)
- 03:54 PM Revision 11486: planning/timeline/timeline.2013.xls: updated for progress
- 03:22 PM Revision 11485: added exports/2013-7-10.Naia.range_limiting_factors.csv.run
- 03:04 PM Revision 11484: bugfix: exports/2013-10-18.Brian_Enquist.Canadensys.csv.run: do not override the table to analytical_stem, because the extract-specific view should be used instead. this was actually benign, because extract.run export_() always sets $table to the extract-specific view.
- 02:57 PM Revision 11483: schemas/vegbien.sql: added 2013-7-10.Naia.range_limiting_factors
- 02:45 PM Revision 11482: schemas/vegbien.sql: sync_analytical_stem_to_view(): row_num: renamed to taxon_occurrence__pkey because previous taxon determinations have been removed, so each row is in fact a taxon_occurrence (~= VegCore.vegpath.org?ERD.taxon_occurrence)
- 02:20 PM Revision 11481: fix: schemas/vegbien.sql: analytical_stem_view: don't ORDER BY datasource, because this requires a slow full-table sort after the hash joins. (when selecting a subset of analytical_stem_view, nested loops are used automatically without needing an ORDER BY to force this.) to get the datasource-sorted order (plus a sort-order guarantee), you can still add a manual `ORDER BY datasource`, which will use a fast index scan on one of the datasource indexes.
- 01:58 PM Revision 11480: schemas/vegbien.sql: analytical_stem: added row_num, which can serve as the taxon_observation ID (DwC occurrenceID)
- 01:53 PM Revision 11479: Updated load-geoscrub script with configurable db.
- load-geoscrub-input.sh now uses a variable with the db name defined at
the top of the script.
Updated the default db ... - 12:11 PM Revision 11478: schemas/vegbien.sql: analytical_stem: locationID... index: use eventDate instead of dateCollected since it's now eventDate that identifies the locationevent
- 12:11 PM Revision 11477: schemas/vegbien.sql: analytical_stem: locationID... index: use eventDate instead of dateCollected since it's now eventDate that identifies the locationevent
- 04:41 AM Revision 11476: schemas/vegbien.sql: analytical_stem_view: use plot.** to obtain plot-related fields, so that the same code does not need to be maintained in both analytical_stem_view and plot.**
- 04:32 AM Revision 11475: schemas/vegbien.sql: analytical_stem_view: moved specimen-specific fields to occurrence section
- 03:50 AM Revision 11474: schemas/vegbien.sql: analytical_stem_view, plot.**: added separate location__cultivated__bien
- 03:11 AM Revision 11473: schemas/vegbien.sql: added separate eventDate, in addition to dateCollected
- 02:59 AM Revision 11472: fix: schemas/vegbien.sql: dateCollected: use aggregateoccurrence.collectiondate *before* locationevent.obsstartdate rather than after, because this is more accurate. it was previously the other way around to allow dateCollected to be the pkey for the row's locationevent (for plots data).
- 02:38 AM Revision 11471: schemas/vegbien.sql: analytical_stem_view, plot.**: locationevent__pkey: moved to right before the locationevent-related fields
10/29/2013
- 06:53 PM Revision 11470: schemas/vegbien.sql: analytical_stem_view: changed column order, etc. to match plot.**
- 06:52 PM Revision 11469: schemas/vegbien.sql: analytical_stem_view: changed column order, etc. to match plot.**
- 06:46 PM Revision 11468: schemas/vegbien.sql: plot.**: added locationevent__pkey so that this view can be joined to other VegBIEN tables, which require the internal pkey
- 06:29 PM Revision 11467: derived/biengeo/README.txt: geoscrub new data: geovalidate.sql: added runtime from Paul
- 09:05 AM Revision 11466: schemas/vegbien.sql: sync_analytical_stem_to_view(): speciesBinomialWithMorphospecies index: documented runtime (1 h)
- 08:56 AM Revision 11465: schemas/vegbien.sql: plot.**: updated to use the same column formulas as analytical_stem_view
- 08:19 AM Revision 11464: planning/timeline/timeline.2013.xls: add globally-unique occurrenceID: removed "globally-unique" because Naia is actually OK with this being numeric (i.e. unique within our DB)
- 08:19 AM Revision 11463: planning/timeline/timeline.2013.xls: updated for progress
- 07:46 AM Revision 11462: lib/runscripts/import_subset.run: $version: use new $extract_view, which is set to the same value that this was
- 07:45 AM Revision 11461: lib/runscripts/extract.run: use the extract-specific view instead of all of analytical_stem
- 07:42 AM Revision 11460: schemas/vegbien.sql: added 2013-10-18.Brian_Enquist.Canadensys view
- 06:51 AM Revision 11459: schemas/vegbien.sql: sync_analytical_stem_to_view(): added index on speciesBinomialWithMorphospecies for Brian Enquist's Canadensys request
- 06:19 AM Revision 11458: planning/timeline/timeline.2013.xls: updated for progress
- 04:16 AM Revision 11457: exports/2013-10-18.Brian_Enquist.Canadensys.csv.run: documented runtime (35 min, now that bugs have been fixed)
- 03:33 AM Revision 11456: bugfix: bin/with_all: @inputs default value: use `local`, so that the default value is only set for the current function and doesn't leak back out into the caller. this fixes a bug in subset imports where import_all's Source/import call to with_all would add the .* datasources, but these would then stay in for the import_scrub call, causing extra .* datasources to incorrectly be imported.
- 02:22 AM Revision 11455: planning/timeline/timeline.2013.xls: usability testing: added additional subtask to validate the scientists' extracts (i.e. check that the extract fulfills their request)
- 02:17 AM Revision 11454: planning/timeline/timeline.2013.xls: provide scientists with their requested data: added separate subtask for Brian Enquist's Canadensys extract
- 02:12 AM Revision 11453: planning/timeline/timeline.2013.xls: updated for progress and revised schedule
- 01:53 AM Revision 11452: bugfix: schemas/pg_hba.Mac.conf: made same change for Mac as was made for Linux in r11451
- 01:22 AM Revision 11451: bugfix: schemas/pg_hba.conf: don't allow ident authentication for Unix socket connections, because this apparently prevents having normal, password-based connections ("md5"). note that just switching the order of the ident and md5 entries is not useful, because whichever authentication type comes second will be ignored completely. this problem was previously worked around by just not using Unix socket connections at all, and always specifying "localhost" as the host to force a hostname-based connection. this does not affect the postgres superuser, because they have their own ident line in pg_hba.conf.
10/25/2013
- 06:15 PM Revision 11450: Added db user and host to load-geoscrub-input.sh
- The psql commands in load-geoscrub-input.sh will now connect with a
specific user on a specific host.
Updated the 'CO... - 04:51 PM Revision 11449: derived/biengeo/README.txt: geoscrub new data: steps that use .sql scripts: added the psql commands to run these
- 04:22 PM Revision 11448: Updated install instructions in the README.
- 03:00 PM Revision 11447: derived/biengeo/README.txt: geoscrub new data: noted that this now deletes any previous geoscrubbing results
- 02:58 PM Revision 11446: derived/biengeo/README.txt: added steps to set the working dir for each set of steps
- 02:54 PM Revision 11445: derived/biengeo/README.txt: added section on obtaining source code, including path to Paul's in-progress files on vegbiendev (not sure whether the in-progress files are needed to run the core scripts in steps 1-6)
- 02:44 PM Revision 11444: derived/biengeo/README.txt: moved commands to run to the top of the README. flagged commands-sections with ***** and an identifying label.
- 02:04 PM Revision 11443: Initial checkin of geoscrub install SQL files.
- Added install.*.sql files that will do initial table creation for all
required tables.
Added a truncate.vegbien_geosc... - 02:04 PM Revision 11442: Update load-geoscrub-input.sh to download from URL.
- Removed logic to dump input data directly from the vegbien database and
to download the input from a URL provided by ... - 11:56 AM Revision 11441: planning/timeline/timeline.2013.xls: reload core & analytical database scheduled for this week: postponed to give us additional time to do datasource validations
- 09:58 AM Revision 11440: inputs/input.Makefile: added %/import_temp alias for %/import, to mirror the presence of import_temp for import
- 09:24 AM Revision 11439: fix: inputs/VegBank/taxonobservation_/map.csv: remapped authorplantname to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
- 09:22 AM Revision 11438: fix: inputs/VegBank/taxonobservation_/map.csv: remapped int_* to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.
10/24/2013
- 07:09 PM Revision 11437: exports/2013-10-18.Brian_Enquist.Canadensys.csv.run: inherit from new import_subset.run (which uses extract.run)
- 07:08 PM Revision 11436: added lib/runscripts/import_subset.run, extract.run
- 05:21 PM Revision 11435: added exports/2013-10-18.Brian_Enquist.Canadensys.csv.run
- 05:07 PM Revision 11434: bin/make_analytical_db: removed no longer needed setting of $schema to $public, because this is now done by psql()
- 05:06 PM Revision 11433: lib/sh/local.sh: psql(): also accept $public as the $schema param, since this is used by a lot of import scripts
- 04:24 PM Revision 11432: lib/sh/util.sh: added require_dot_script()
- 04:13 PM Revision 11431: bugfix: lib/sh/util.sh: $top_script: use @BASH_SOURCE instead of $0, because this is also defined for .-scripts
- 04:03 PM Revision 11430: bugfix: bin/import_all: restore the working dir when main() is done, in case it started as something other than the root dir
- 03:49 PM Revision 11429: bin/after_import: support turning off the end-of-import backup for imports that are not the full database
- 03:26 PM Revision 11428: bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.
- 03:21 PM Revision 11427: bugfix: lib/runscripts/util.run: if is_dot_script, fix $@ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $@ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).
- 03:17 PM Revision 11426: bugfix: lib/sh/util.sh: is_dot_script(): need to subtract 1 from ${#BASH_LINENO[@]}, because this is the array length rather than the index of the last element as in Perl
- 02:58 PM Revision 11425: lib/sh/util.sh: added is_dot_script()
- 01:15 PM Revision 11424: bugfix: schemas/vegbien.sql: taxondetermination_set_iscurrent(): is_datasource_current (used by analytical_stem_view): need to separately check if `determinationtype IS NULL`, because `determinationtype NOT IN (accepted, matched))` will return NULL (false) if determinationtype is NULL, causing no match
- 01:11 PM Revision 11423: bugfix: bin/make_analytical_db: when running into a public schema other than "public", also pass this to `/run export_` (which currently uses $schema instead of $public)
- 01:10 PM Revision 11422: bugfix: bin/import_all: fix $@ when .-included without args (which causes bash to put the wrong values in $@ instead of leaving it empty)
- 01:09 PM Revision 11421: bin/import_all: `make schemas/$version/install`: reinstall instead to allow re-running the import to the same custom schema (e.g. 2013-10-18.Brian_Enquist.Canadensys)
- 01:07 PM Revision 11420: bin/import_all: `make schemas/$version/install`: ignore errors if schema exists, to support running with -e
10/23/2013
- 11:10 PM Revision 11419: bugfix: bin/import_all: removing inputs/.TNRS/tnrs/tnrs.make.lock: use `"rm" -f` instead of plain "rm" to avoid having an error exit status, which will abort the script if run with the -e flag (as runscripts are)
- 11:02 PM Revision 11418: lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args
- 11:00 PM Revision 11417: lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)
- 10:34 PM Revision 11416: bin/*_all: *_main(): renamed to just main() because it does not matter that other shell-includes' main() methods will clobber this, because it is only executed once
- 10:29 PM Revision 11415: bugfix: bin/import_all: Source tables: use .../import instead of import_temp because import_temp is only needed when importing all tables, to prevent the temp suffix from being removed yet
- 10:17 PM Revision 11414: lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV
- 09:11 PM Revision 11413: bugfix: lib/sh/util.sh: alias_append(): need to enclose $(alias) call in "" because its result may contain separator chars (i.e. whitespace) that will be parsed incorrectly. this appears to only be a bug when runscripts are run as shell-includes, with a leading ".".
- 01:49 PM Revision 11412: schemas/VegCore/ERD/VegCore.ERD.mwb: connecting lines: inherits from traceable: added arrow to indicate what this label refers to
- 01:43 PM Revision 11411: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 01:37 PM Revision 11410: schemas/VegCore/ERD/VegCore.ERD.mwb: HAS-A/IS-A box: renamed to "connecting lines" for clarity
10/22/2013
- 10:22 PM Revision 11409: schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: HAS-A: added HAS-MANY going in the opposite direction, because every HAS-A has an opposite HAS-MANY
- 10:17 PM Revision 11408: schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: IS-A, HAS-A: added directional arrows
- 10:10 PM Revision 11407: schemas/VegCore/ERD/VegCore.ERD.mwb: field order box: removed spacing between top of text box and bottom of outer box label
- 10:04 PM Revision 11406: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 10:00 PM Revision 11405: schemas/VegCore/ERD/VegCore.ERD.mwb: reordered columns according to the field order convention
- 09:49 PM Revision 11404: schemas/VegCore/ERD/VegCore.ERD.mwb: added label documenting the field order convention:
- 1) inherited
2) required
3) identifying
4) foreign key
5) extenders
6) others - 08:34 PM Revision 11403: web/links/index.htm: updated to Firefox bookmarks. added links for EER models, data management plans. put PostgreSQL before MySQL because we have found PostgreSQL to be a much more capable database system, even though it lacks some of MySQL's user-friendly features.
- 06:38 PM Revision 11402: planning/timeline/timeline.2013.xls: updated for progress
- 06:21 PM Revision 11401: fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names
- 06:16 PM Revision 11400: schemas/vegbien.sql: regenerated using `make schemas/remake`. note that analytical_stem_view column renamings need this step after a search-and-replace of the column names, in order to remove excess "" around all-lowercase names and reset generated index names.
- 06:10 PM Revision 11399: fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names
- 01:20 PM Revision 11398: added planning/goals/web_interface/phpPgAdmin.select_interface.png for use at wiki.vegpath.org/Proposed_enhancements
- 09:39 AM Revision 11397: inputs/CVS/_src/: added refresh from Mike Lee
10/21/2013
- 07:14 PM Revision 11396: fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error
- 05:16 PM Revision 11395: planning/timeline/timeline.2013.xls: usability testing: added subtask to provide scientists with their requested data
10/20/2013
- 05:51 PM Revision 11394: planning/timeline/timeline.2013.xls: updated for progress
- 05:21 PM Revision 11393: bugfix: bin/import_all: need to publish datasources that won't be published by `make .../import`, so that the per-datasource import XPaths that refer to TNRS/geoscrub will link up with the TNRS/geoscrub source entry instead of creating a new entry without the metadata (because the entry with the metadata was named TNRS.new/geoscrub.new)
- 05:09 PM Revision 11392: schemas/vegbien.sql: datasource_publish(): use parameter names instead of $# because this is a PL/pgSQL function
- 05:07 PM Revision 11391: bugfix: schemas/vegbien.sql: datasource_publish(): if the datasource to publish already has the published name, don't datasource_rm() it
- 04:55 PM Revision 11390: bin/import_all: removed no longer needed import of geoscrub data, because analytical_stem_view is now joined to the geoscrub_output table directly, instead of using the imported canon_place entries
- 04:52 PM Revision 11389: schemas/vegbien.sql: analytical_stem_view: join to the geoscrub_output table directly, instead of using the imported canon_place entries. this avoids the need to import geoscrub_output into VegBIEN (which is expected to take 2+ hours after the refresh), as well as the need to then refresh any datasources whose geoscrubbing input data has changed.
- 04:37 PM Revision 11388: inputs/.geoscrub/geoscrub_output/postprocess.sql: added nullable unique index on the inputs, for use by analytical_stem_view. note that it must be nullable in order to create a match when not all of the input fields are populated. this uses array[] to create a nullable index, which is much better than column-based import and VegBIEN's use of COALESCE() because the expression is the same for every type and no NULL sentinel value is needed.
- 03:50 PM Revision 11387: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 03:41 PM Revision 11386: schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:34 PM Revision 11385: schemas/VegCore/ERD/VegCore.ERD.mwb: person: allow to have multiple organizations
- 03:28 PM Revision 11384: schemas/VegCore/ERD/VegCore.ERD.mwb: split "2b. GNRS" label into two labels, one for each table GNRS is applied to
- 03:23 PM Revision 11383: schemas/VegCore/ERD/VegCore.ERD.mwb: georeferencing: merged into geoplace, since this is actually information attached to a specific plot, etc. relating to the coordinates used in its geoplace subclass
- 02:59 PM Revision 11382: schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place: changed parent geoplace pointer to parent_boundary_WKT, since the immediate parent may not have an associated boundary to use for geovalidation (i.e. it may not be an official GADM geoplace), although ancestors further up likely will be
- 02:40 PM Revision 11381: schemas/VegCore/ERD/VegCore.ERD.mwb: place.name: made it required because it's needed for the unique constraint to be populated properly (including for subclasses such as geoplace, which need to generate this from the coordinates)
- 02:30 PM Revision 11380: schemas/VegCore/ERD/VegCore.ERD.mwb: place.rank: made it required, because every place should have some kind of rank indicating what type of place it is, including lower ranks (e.g. plot, individual)
- 02:24 PM Revision 11379: schemas/VegCore/ERD/VegCore.ERD.mwb: place: added unique constraint on parent, rank, name
- 02:24 PM Revision 11378: schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have
- 02:07 PM Revision 11377: schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have
- 01:46 PM Revision 11376: schemas/VegCore/ERD/VegCore.ERD.mwb: geoplace.official_name: renamed to name to merge with inherited field from place. documented that for geoplaces, this is the official, scrubbed name.
- 12:48 PM Revision 11375: inputs/.geoscrub/geoscrub_output/postprocess.sql: added geovalid derived column, for use by analytical_stem_view
10/19/2013
- 06:56 PM Revision 11374: bin/with_all: $all: renamed to $hidden_srcs for clarity, since this now just adds the hidden (.*) datasources, rather than always using all datasources
- 06:50 PM Revision 11373: bugfix: bin/with_all: in $all mode, just prepend the .* datasources to the user-selected (or default) @inputs, so that using $all to add these datasources doesn't inadvertently cause the action to be performed for *all* datasources
- 04:33 PM Revision 11372: web/links/index.htm: updated to Firefox bookmarks. PostgreSQL: ALTER TABLE: added documentation about disabling of foreign key triggers, which is only possible by the superuser. note that marking a foreign key constraint as NOT VALID does *not* disable the trigger, so NOT VALID cannot be used for this purpose. this would be used to add fkeys from core VegBIEN tables to validation results tables such as the geoscrubbing results, without needing to import the validation results directly into core VegBIEN (which is time-consuming and currently must be done *before* input data is loaded, requiring a datasource reload to add geoscrubbing results).
- 02:15 PM Revision 11371: bin/import_all: usage: documented that this can now be run with a custom datasources list (each of the form inputs/src/)
- 02:02 PM Revision 11370: bin/with_all: added support for providing a custom list of inputs to run the command on
- 01:29 PM Revision 11369: inputs/.geoscrub/geoscrub_output/postprocess.sql, run: updated runtimes
- 01:13 AM Revision 11368: inputs/.geoscrub/geoscrub_output/run: documented full load_data() runtime (9 min @starscream)
- 01:12 AM Revision 11367: inputs/.geoscrub/geoscrub_output/postprocess.sql: updated runtimes for refreshed data, which now has 4x as many rows (1,707,970->6,747,650)
- 12:54 AM Revision 11366: inputs/.geoscrub/geoscrub_output/: refreshed geoscrub data. removed +header.csv because the extract now contains the header in the first row of the file.
- 12:52 AM Revision 11365: bugfix: lib/sh/local.sh: psql(): $is_root: use `` around case statement instead of $(), because it contains an embedded unbalanced )
- 12:27 AM Revision 11364: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: include only the columns that Jim provided in his extract (the geoscrub table contains additional internal columns that are not part of the geovalidation data for VegBIEN). documented runtime (30 s) and upload time (1.5 min).
10/18/2013
- 10:33 PM Revision 11363: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: removed no longer needed setting of $local_server, $local_user (and use of $local_pg_database instead of $database) because the use_local bug in local.sh has been fixed
- 10:32 PM Revision 11362: bugfix: lib/sh/local.sh: psql(): don't default the connection vars using use_local if running as the postgres user. in that case, connection must happen via a socket, with server="", and as the user running the command (postgres), with user="".
- 09:55 PM Revision 11361: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to manually set local_server, local_user to "" so that they do not default to their bien-user values
- 09:54 PM Revision 11360: bugfix: lib/sh/db.sh: avoid outputting to /dev/fd/# when running as sudo on Linux, because this causes a "Permission denied" error (due to the /dev/fd/# file being owned by a different user). this is not a problem with normal redirects (>&#), because they do not use /dev/fd/# files which can have access permissions.
- 09:52 PM Revision 11359: bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file
- 08:17 PM Revision 11358: lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command
- 07:50 PM Revision 11357: lib/sh/db.sh: pg_as_root(): run sudo with echo_run to help debug
- 06:29 PM Revision 11356: bugfix: lib/sh/db.sh: pg_cmd(): only set PG* connection/login env vars when the corresponding var is *non-empty*. there are some situations in which these must be unset (in order to use the default value), and other situations when the var must be set to something (i.e. "") to avoid it being defaulted to a value in local.sh > connection vars.
- 06:27 PM Revision 11355: backups/TNRS.backup.md5: updated
- 06:13 PM Revision 11354: bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to set $local_pg_database instead of $database because use_local (in psql()) does not currently avoid clobbering already-set versions of the applicable env vars
- 06:11 PM Revision 11353: bugfix: lib/sh/local.sh: pg_as_root(): need to use -E (preserve environment) option to sudo, so that $schema, $table get passed through
- 06:05 PM Revision 11352: bugfix: lib/sh/local.sh: psql(): only \set schema, table if $schema, $table are non-empty, because otherwise, you will get a "zero-length delimited identifier" error
- 05:30 PM Revision 11351: added inputs/.geoscrub/geoscrub_output/geoscrub.csv.run to export the geoscrub table (must be run on vegbiendev)
- 05:29 PM Revision 11350: lib/sh/local.sh: added require_remote()
- 05:29 PM Revision 11349: lib/sh/db.sh: added pg_as_root()
- 05:28 PM Revision 11348: lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.
- 05:23 PM Revision 11347: Added instructions for dependencies in the README.
- 05:23 PM Revision 11346: Added indexes to speed up geonames-to-gadm.sql.
- Without these indexes, these queries could take hours to complete.
With them, the times more closely matched the time... - 05:23 PM Revision 11345: Fixed a couple of syntax errors in geovalidate.sh.
- Fixed a sql syntax error and a bash syntax error in the next line.
- 04:03 PM Revision 11344: planning/timeline/timeline.2013.xls: "geoscrubbing automated pipeline": scheduled for after Paul's current set of tasks on the geoscrubbing re-run is complete. i'm budgeting several weeks for this since my understanding is that Paul is doing this part-time.
- 03:57 PM Revision 11343: planning/timeline/timeline.2013.xls: moved "geoscrubbing automated pipeline" under "simplify import process for easier maintainability"
- 03:53 PM Revision 11342: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added subtask to spot-check reloaded geoscrubbing data
- 03:52 PM Revision 11341: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added separate subtask for "geoscrubbing data reload", since apparently it was not clear that of course the new data will need to be imported into VegBIEN before the results of the re-run are available. this is currently scheduled to happen in the next full-database import, which is the week of 10/28 in order to include further validations fixes.
- 02:06 PM Revision 11340: planning/timeline/timeline.2013.xls: CVS validation: use timespan dot ◦ for supertask
- 02:05 PM Revision 11339: planning/timeline/timeline.2013.xls: CVS validation: added subtasks that are similar to for FIA validation (create validation subset, create extract)
- 02:03 PM Revision 11338: planning/timeline/timeline.2013.xls: FIA validation: split apart into subtasks, including "decide which columns to validate", which has to happen ahead of time before the extract can be generated
- 01:15 PM Revision 11337: planning/timeline/timeline.2013.xls: fixed check marks for past (hidden) weeks, which had gotten duplicated when rows were copied together with their check marks
- 01:04 PM Revision 11336: planning/timeline/timeline.2013.xls: fixed line heights
- 12:57 PM Revision 11335: planning/timeline/timeline.2013.xls: fixed column width so the dates display properly in MS Excel
- 12:48 PM Revision 11334: planning/timeline/timeline.2013.xls: right-aligned legend so it isn't too close to the "During week of:" label
- 12:46 PM Revision 11333: planning/timeline/timeline.2013.xls: added legend:
- • task
◦ timespan
✓ task progress
☑ timespan progress - 10:25 AM Revision 11332: planning/timeline/timeline.2013.xls: attribution/conditions of use: made it a subtask of "add missing columns" because this is related to data needed for published analyses. added dots because this is an ongoing task, that depends on data providers getting their use conditions to us.
- 10:08 AM Revision 11331: planning/timeline/timeline.2013.xls: reload core & analytical database: moved next reload ahead to last week of October so that we can include the updated geovalidation data for the 10/31 deadline. added additional reload so that they are spaced <= 1 month apart.
- 09:54 AM Revision 11330: planning/timeline/timeline.2013.xls: receive feedback from documentation tester: added an extra week to receive additional feedback from them in response to documentation fixes made
- 09:51 AM Revision 11329: planning/timeline/timeline.2013.xls: attribution/conditions of use: made this a top-level task instead of a subtask of "data provider metadata", to avoid including lower-priority tasks (i.e. in the later column) in the same section as higher-priority tasks
- 09:47 AM Revision 11328: planning/timeline/timeline.2013.xls: datasource validations: regrouped by subtask instead of by datasource, so that the high-priority subtasks get done for all datasources before moving on to lower-priority subtasks for any datasources
- 08:54 AM Revision 11327: planning/timeline/timeline.2013.xls: reduced width of Milestone column to make room to fit an additional week on the printed page
- 08:42 AM Revision 11326: planning/timeline/timeline.2013.xls: attribution/conditions of use: removed "(Brad/Brian/Bob/etc.)" because these are from everyone who provided or obtained data, not just Brad/Brian/Bob
- 08:40 AM Revision 11325: planning/timeline/timeline.2013.xls: rescheduled tasks to accommodate the separate non-critical feature requests subtasks
- 08:37 AM Revision 11324: planning/timeline/timeline.2013.xls: datasource validations: split "fix feature requests" into separate "fix critical feature requests" and "fix non-critical feature requests" tasks. rescheduled non-critical feature requests until after the other validation tasks have been completed.
- 08:11 AM Revision 11323: planning/timeline/timeline.2013.xls: add globally-unique occurrenceID: moved up to next week because we would like to be able to get this done for the 10/31 deadline
- 07:52 AM Revision 11322: planning/timeline/timeline.2013.xls: updated for progress
- 07:48 AM Revision 11321: planning/timeline/timeline.2013.xls: moved "data provider metadata" before "datasource validations (spot-checking)" because conditions of use are necessary for scientists who want to publish papers based on the data (which is a key use case)
- 07:43 AM Revision 11320: planning/timeline/timeline.2013.xls: moved "usability testing" before "datasource validations (spot-checking)" because this is most important towards reaching our goal of a useful information resource
- 07:41 AM Revision 11319: planning/timeline/timeline.2013.xls: moved "geoscrubbing re-run", "add globally-unique occurrenceID" back under "usability testing" > "add missing columns" because these are in fact part of the usability testing
- 01:26 AM Revision 11318: planning/timeline/timeline.2013.xls: updated for progress
10/17/2013
- 11:59 PM Revision 11317: planning/timeline/timeline.2013.xls: updated for progress
- 10:32 PM Revision 11316: planning/timeline/timeline.2013.xls: "flatten the datasources to a common schema": moved to later column because the complex tasks "switching to new-style import" and "create interactive scripts for each import step" are also scheduled then. (it's unlikely we would have much time over winter break anyway, considering that there is ~1 week's worth of holidays then.)
- 10:24 PM Revision 11315: planning/timeline/timeline.2013.xls: scheduled "simplify import process for easier maintainability"
- 10:20 PM Revision 11314: planning/timeline/timeline.2013.xls: tasks performed by someone else (geoscrubbing re-run): changed solid check marks ✓ to open check marks ☑ to match the solid • vs. open ◦ dot convention
- 10:17 PM Revision 11313: planning/timeline/timeline.2013.xls: documentation testing: added supertask dots. removed later dots for scheduled tasks.
- 10:15 PM Revision 11312: planning/timeline/timeline.2013.xls: scheduled "documentation testing"
- 10:02 PM Revision 11311: planning/timeline/timeline.2013.xls: scheduled "simplify process of mapping/adding a new datasource"
- 10:01 PM Revision 11310: planning/timeline/timeline.2013.xls: "add globally-unique occurrenceID": moved it up to the first week when we're no longer fixing existing issues in datasources, since this has similar priority to adding missing columns discovered during usability testing (which is scheduled as an ongoing task)
- 09:54 PM Revision 11309: planning/timeline/timeline.2013.xls: usability testing: did task breakdown (find scientists who want to use BIEN3 data, etc.) and scheduled subtasks
- 09:34 PM Revision 11308: planning/timeline/timeline.2013.xls: moved "add missing columns" to its own supertask. used outline check mark ☑ (analogous to open circle ◦) to mark supertasks as completed which were split up into subtasks.
- 09:29 PM Revision 11307: planning/timeline/timeline.2013.xls: later column: removed dots from scheduled items
- 09:06 PM Revision 11306: planning/timeline/timeline.2013.xls: moved "switching to new-style import"-related steps (other than for CVS) to separate "simplify import process for easier maintainability" supertask, since this is not part of the "simplify process of mapping/adding a new datasource" task
- 08:39 PM Revision 11305: planning/timeline/timeline.2013.xls: add any missing columns: added and scheduled step to add globally-unique occurrenceID
- 08:34 PM Revision 11304: planning/timeline/timeline.2013.xls: geoscrubbing re-run: added dots ◦ for this for the time when it can be worked on asynchronously by Paul Sarando
- 08:30 PM Revision 11303: planning/timeline/timeline.2013.xls: data provider metadata: added dots ◦ for the portion of "attribution and conditions of use" that can be worked on asynchronously by Brad/Brian/Bob
- 08:24 PM Revision 11302: planning/timeline/timeline.2013.xls: scheduled "aggregated validations" during the last 2 weeks of "datasource validations (spot-checking)", because these weeks are only spent fixing issues uncovered in the remaining datasources, so there may be extra time then
- 07:20 PM Revision 11301: planning/timeline/timeline.2013.xls: scheduled other tasks after "datasource validations (spot-checking)" is complete
- 07:14 PM Revision 11300: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): each datasource's validation supertask: added open circles ◦ spanning the length of the subtasks
- 07:12 PM Revision 11299: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for supertasks that have been fully split into subtasks (not just itemizing a few subtasks), so that these don't count towards the bullets (estimated workload) in each week
- 07:09 PM Revision 11298: planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for tasks that are performed by someone other than me, so that these don't count towards the bullets (estimated workload) in each week
- 07:04 PM Revision 11297: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): split each datasource into subtasks and scheduled them
- 06:00 PM Revision 11296: planning/timeline/timeline.2013.xls: moved "move denormalized validations to stage II", "move stage III validations to stage II" outside of "switching to new-style import" because the "switching to new-style import" step refers just to the per-datasource switching steps, not to the additional refactorings that would be needed to avoid dependency on the complex XPath mappings (mappings/VegCore-VegBIEN.csv)
- 05:57 PM Revision 11295: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): added subtasks for each of the remaining datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order)
- 05:43 PM Revision 11294: planning/timeline/timeline.2013.xls: moved non-validation-related tasks after the 10/31 deadline so that these are not taking time away from the validation
- 05:40 PM Revision 11293: planning/timeline/timeline.2013.xls: moved "flatten the datasources to a common schema" under "simplify process of mapping/adding a new datasource" because this is also needed separately for datasources where the left-joining is not part of the validation
- 05:30 PM Revision 11292: planning/timeline/timeline.2013.xls: extended "revisions to VegBIEN schema" to length of "datasource validations (spot-checking)" because schema changes are expected as we add missing fields
- 05:23 PM Revision 11291: planning/timeline/timeline.2013.xls: crossed out and hid completed tasks ("find out amount remaining in BIEN3 budget")
- 05:17 PM Revision 11290: planning/timeline/timeline.2013.xls: datasource validations (spot-checking): extended through the end of November because data providers' fixes on the remaining 10 datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order) are likely to add significantly to the issues and feature requests associated with these datasources (e.g. the 2nd-round VegBank validation added 4 issues and 5 feature requests). there is also expected to be wait time while data providers are responding (most likely in multiple rounds of feedback).
- 04:59 PM Revision 11289: planning/timeline/timeline.2013.xls: data provider metadata: removed "iPlant can do" because this actually requires Brad/Brian/Bob/other data providers to provide this info. however, this info may be findable on the web for some datasources.
- 04:53 PM Revision 11288: planning/timeline/timeline.2013.xls: moved "data provider metadata" right after "datasource validations" because this is part of the completed database itself rather than the tools to maintain it
- 04:47 PM Revision 11287: planning/timeline/timeline.2013.xls: split "revisions to schema" into "revisions to VegBIEN schema" (part of datasource validations) and "revisions to normalized VegCore" (part of documentation)
- 04:44 PM Revision 11286: bin/import_all: use just import_scrub, not reimport_scrub, because import_scrub now automatically publishes the datasource's import (i.e. removes the temp suffix)
- 04:43 PM Revision 11285: bugfix: inputs/input.Makefile: import: remove the temp suffix once the import is done, so that the full database import doesn't keep the suffix attached to the datasources that import_all didn't import with reimport. removed unused import_publish target (instead use import_temp to invoke just the import without the temp suffix removal).
- 04:27 PM Revision 11284: planning/timeline/timeline.2013.xls: moved part of "switching to new-style import" under "datasource validations (spot-checking)" because this is necessary to validate CVS
- 04:24 PM Revision 11283: planning/timeline/timeline.2013.xls: moved "simplify process of mapping/adding a new datasource" and "documentation testing" after "usability testing" because these tasks were there to make it possible for people other than me to reload/add to the database, which we have now decided is a lower priority than creating the validated database itself
- 04:14 PM Revision 11282: planning/timeline/timeline.2013.xls: added weeks through the end of the year (12/31)
- 02:00 PM Revision 11281: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: changed dataset definition to the definition in normalized VegCore ("a collection of records from the same place, with the same attribution requirements"), following discussion with Ramona
- 01:13 PM Revision 11280: planning/timeline/timeline.2013.xls: updated for progress
- 12:34 PM Revision 11279: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: updated to Ramona's commented version
- 12:19 AM Revision 11278: inputs/CVS/plot_/map.csv: realLatitude, realLongitude: remapped to UNUSED because these columns are actually empty
10/13/2013
- 07:32 PM Revision 11277: inputs/CVS/taxonObservation_/map.csv: collector_ID: remapped it to UNUSED and removed the join to party via it, like in VegBank
- 07:03 PM Revision 11276: inputs/CVS/: deleted stemLocation_, because the CVS stemLocation table is empty (unlike VegBank)
- 05:26 PM Revision 11275: inputs/CVS/import_order.txt: added plantConcept_/ so it would get automapped after switching to new-style import
- 04:24 PM Revision 11274: inputs/CVS/taxonObservation_/map.csv: denorm_{tri,quad}*: mapped to infraspecificRank*, infraspecificEpithet*
- 04:02 PM Revision 11273: inputs/CVS/taxonObservation_/map.csv: infraspecific ranks: remapped to EQUIV#to:species (which is the speciesBinomial), because these actually contain the full taxonomic name at that rank, like VegBank
- 03:50 PM Revision 11272: inputs/CVS/taxonObservation_/map.csv: genus: documented that unlike VegBank, does *not* include genus author
- 03:47 PM Revision 11271: inputs/CVS/taxonObservation_/map.csv: denorm_* terms _alt-ed with normalized terms: use DUPLICATE#of instead where possible. documented where and why _alt was necessary (this applies to a few rows for division, genus).
- 03:42 PM Revision 11270: bugfix: inputs/CVS/taxonObservation_/map.csv: species: remapped to speciesBinomial, not specificEpithet (like for VegBank). however, note that denorm_species *is* in fact the epithet, unlike VegBank.
- 03:19 PM Revision 11269: fix: inputs/CVS/taxonObservation_/postprocess.sql: removed {} around denorm_genus to match the normalized genus
- 02:28 PM Revision 11268: inputs/CVS/taxonObservation_/map.csv: removed unnecessary _alts for terms that don't have a duplicate denorm_* or hierarchical field
- 02:06 PM Revision 11267: fix: inputs/CVS/taxonObservation_/postprocess.sql: fix 1 row that has denorm_kingdom != Kingdom (i.e. both NOT NULL but not the same)
- 12:46 PM Revision 11266: bugfix: lib/common.Makefile: $(subMake): don't enclose the target in "" because sometimes the target is empty (i.e. `all`), and nothing should be passed to the sub-make
- 12:10 AM Revision 11265: inputs/VegBank/plot_/create.sql: documented runtime (5 min)
- 12:10 AM Revision 11264: bugfix: inputs/CVS/plot_/create.sql: like for VegBank, need to compare place.*PLOT_ID*, not PLOTPLACE_ID, with plot.PLOT_ID
- 12:02 AM Revision 11263: /README.TXT: Single datasource import: added pointer to instructions to remake the analytical DB (also required after single datasource import)
10/12/2013
- 04:59 PM Revision 11262: planning/timeline/timeline.2013.xls: updated for progress
- 04:20 PM Revision 11261: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 04:16 PM Revision 11260: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 03:49 PM Revision 11259: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: run all sync_uploads on the svn working copy using --size-only, because the mtimes are based on when the files were last updated by svn and are not meaningful
- 03:46 PM Revision 11258: /README.TXT: Full database import: On local machine: do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine": removed no longer accurate indicator that these steps are above Full database import, since Full database import is now at the beginning of the file
- 03:05 PM Revision 11257: inputs/VegBank/verify/input_cols.txt, inputs/VegBank/+taxon_observation.**.sample/create.sql: updated to match taxon_observation.** columns
- 03:03 PM Revision 11256: bugfix: inputs/VegBank/+taxon_observation.**.sample/: renamed to ^taxon_observation.**.sample because a leading + has a special meaning to bash (it indicates a shell option, and you will get an error "invalid option name"), as well as to make (it indicates that a recipe command invokes make recursively)
- 02:27 PM Task #767 (New): autodetect column mismatches
- * if @map.csv@ does not contain the @header.csv@ columns in exactly the right order, this will mess up the import in ...
- 02:14 PM Revision 11255: bugfix: inputs/VegBank/taxon_observation.**/header.csv: updated for observation_/map.csv bugfix, which added new hasobservationsynonym field. this fixes a strange test bug caused by the taxon_observation.**/map.csv column list being mismatched/misaligned with what was in the underlying tables. (column mismatches will often cause unexplainable errors in unrelated sections of code the same way that buffer overflows do in C++.)
- 02:01 PM Revision 11254: bugfix: inputs/VegBank/taxon_observation.**.sample/: renamed to +taxon_observation.**.sample so that the *-expansion of taxon_observation.** doesn't add taxon_observation.**.sample (which causes it to attempt to install taxon_observation.**.sample before taxon_observation.** is installed)
- 12:48 PM Revision 11253: bugfix: *Makefile: recursive invocation of $(MAKE): enclose targets in "" in case they contain *
- 12:12 PM Revision 11252: bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable
- 12:11 PM Revision 11251: bugfix: inputs/input.Makefile: %/uninstall: allow user to set is_view=1 flag to use DROP VIEW instead of DROP TABLE
- 12:08 PM Revision 11250: lib/sh/util.sh: added instructions for making an export only visible locally
10/10/2013
- 06:50 PM Revision 11249: bugfix: inputs/VegBank/observation_/header.csv, map.csv: updated for refresh, which inserts hasobservationsynonym at the end of the observation table
- 05:46 PM Revision 11248: inputs/VegBank/taxon_observation.**.sample/create.sql: reordered columns in the same order as analytical_plot, for easier validation
- 05:46 PM Revision 11247: bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed
- 03:07 PM Revision 11246: web/links/index.htm: updated to Firefox bookmarks. updated favicons.
- 02:57 PM Revision 11245: web/links/index.htm: updated to Firefox bookmarks. sudo: added instructions to turn off incorrect password e-mails.
- 02:40 PM Revision 11244: inputs/VegBank/taxon_observation.**.sample/create.sql: include only the subset of columns that is imported to VegBIEN
- 02:32 PM Revision 11243: inputs/VegBank/taxon_observation.**.sample/test.xml.ref: updated inserted row count (which was most likely generated before the output column names had been set to the input column names)
- 01:55 PM Revision 11242: added inputs/VegBank/verify/input_cols.include.txt, with runscript to generate it
- 01:26 PM Revision 11241: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
- 01:18 PM Revision 11240: inputs/VegBank/verify/input_cols.unmapped.txt*: renamed to input_cols.exclude.txt* because this now includes mapped columns as well
- 01:11 PM Revision 11239: inputs/VegBank/verify/input_cols.unmapped.txt.run: remove unmapped join columns, since these *would* be included in the extract
- 01:09 PM Revision 11238: inputs/VegBank/verify/input_cols.unmapped.txt.run: take input directly from input_cols.txt to avoid needing to first copy and paste it into input_cols.unmapped.txt
- 01:03 PM Revision 11237: inputs/VegBank/verify/input_cols.unmapped.txt.run: added back deliberately excluded columns (DUPLICATE#of:..., etc.) so that the # of rows in the file can be subtracted from the total # of columns to get the # of input columns that would be included in the extract
- 12:43 PM Revision 11236: bugfix: inputs/input.Makefile: %/VegBIEN.csv: `ln -s` to create VegBIEN.csv: enclose the filenames in "" since they may contain * (e.g. taxon_observation.**)
- 12:23 PM Revision 11235: added inputs/VegBank/verify/input_cols.txt, input_cols.unmapped.txt (with runscript to filter input_cols.unmapped.txt)
- 11:53 AM Revision 11234: schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: made Ramona's corrections with track changes turned on. note that you have to use MS Word for this, not LibreOffice, because LibreOffice can't save the table of contents properly in .docx or .doc format (although it can save it in .odt format).
- 08:18 AM Revision 11233: inputs/VegBank/stratum/postprocess.sql: added pkey
- 08:05 AM Revision 11232: inputs/VegBank/taxonobservation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
- 07:54 AM Revision 11231: inputs/VegBank/observation_/postprocess.sql: added __parent index on locationID to facilitate the LEFT JOINs used to create the validation input
- 07:45 AM Revision 11230: inputs/VegBank/import_order.txt: added taxon_observation.**.sample so it will automatically be kept up to date
- 07:32 AM Revision 11229: inputs/VegBank/taxon_observation.**.sample/create.sql: set runtime (1 s)
- 07:30 AM Revision 11228: inputs/VegBank/: added taxon_observation.**.sample subset of plots to use in the validation. this avoids the need to import all of VegBank just to validate a few of the plots.
Also available in: Atom