Project

General

Profile

Statistics
| Revision:

# Date Author Comment
11363 10/18/2013 10:33 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: removed no longer needed setting of $local_server, $local_user (and use of $local_pg_database instead of $database) because the use_local bug in local.sh has been fixed

11362 10/18/2013 10:32 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): don't default the connection vars using use_local if running as the postgres user. in that case, connection must happen via a socket, with server="", and as the user running the command (postgres), with user="".

11361 10/18/2013 09:55 PM Aaron Marcuse-Kubitza

bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to manually set local_server, local_user to "" so that they do not default to their bien-user values

11360 10/18/2013 09:54 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: avoid outputting to /dev/fd/# when running as sudo on Linux, because this causes a "Permission denied" error (due to the /dev/fd/# file being owned by a different user). this is not a problem with normal redirects (>&#), because they do not use /dev/fd/# files which can have access permissions.

11359 10/18/2013 09:52 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file

11358 10/18/2013 08:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command

11357 10/18/2013 07:50 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_as_root(): run sudo with echo_run to help debug

11356 10/18/2013 06:29 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_cmd(): only set PG* connection/login env vars when the corresponding var is non-empty. there are some situations in which these must be unset (in order to use the default value), and other situations when the var must be set to something (i.e. "") to avoid it being defaulted to a value in local.sh > connection vars.

11355 10/18/2013 06:27 PM Aaron Marcuse-Kubitza

backups/TNRS.backup.md5: updated

11354 10/18/2013 06:13 PM Aaron Marcuse-Kubitza

bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to set $local_pg_database instead of $database because use_local (in psql()) does not currently avoid clobbering already-set versions of the applicable env vars

11353 10/18/2013 06:11 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: pg_as_root(): need to use -E (preserve environment) option to sudo, so that $schema, $table get passed through

11352 10/18/2013 06:05 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): only \set schema, table if $schema, $table are non-empty, because otherwise, you will get a "zero-length delimited identifier" error

11351 10/18/2013 05:30 PM Aaron Marcuse-Kubitza

added inputs/.geoscrub/geoscrub_output/geoscrub.csv.run to export the geoscrub table (must be run on vegbiendev)

11350 10/18/2013 05:29 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: added require_remote()

11349 10/18/2013 05:29 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_as_root()

11348 10/18/2013 05:28 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.

11347 10/18/2013 05:23 PM Paul Sarando

Added instructions for dependencies in the README.

11346 10/18/2013 05:23 PM Paul Sarando

Added indexes to speed up geonames-to-gadm.sql.

Without these indexes, these queries could take hours to complete.
With them, the times more closely matched the times Jim noted in the sql
comments.

11345 10/18/2013 05:23 PM Paul Sarando

Fixed a couple of syntax errors in geovalidate.sh.

Fixed a sql syntax error and a bash syntax error in the next line.

11344 10/18/2013 04:03 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: "geoscrubbing automated pipeline": scheduled for after Paul's current set of tasks on the geoscrubbing re-run is complete. i'm budgeting several weeks for this since my understanding is that Paul is doing this part-time.

11343 10/18/2013 03:57 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "geoscrubbing automated pipeline" under "simplify import process for easier maintainability"

11342 10/18/2013 03:53 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: geoscrubbing re-run: added subtask to spot-check reloaded geoscrubbing data

11341 10/18/2013 03:52 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: geoscrubbing re-run: added separate subtask for "geoscrubbing data reload", since apparently it was not clear that of course the new data will need to be imported into VegBIEN before the results of the re-run are available. this is currently scheduled to happen in the next full-database import, which is the week of 10/28 in order to include further validations fixes.

11340 10/18/2013 02:06 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: CVS validation: use timespan dot ◦ for supertask

11339 10/18/2013 02:05 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: CVS validation: added subtasks that are similar to for FIA validation (create validation subset, create extract)

11338 10/18/2013 02:03 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: FIA validation: split apart into subtasks, including "decide which columns to validate", which has to happen ahead of time before the extract can be generated

11337 10/18/2013 01:15 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: fixed check marks for past (hidden) weeks, which had gotten duplicated when rows were copied together with their check marks

11336 10/18/2013 01:04 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: fixed line heights

11335 10/18/2013 12:57 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: fixed column width so the dates display properly in MS Excel

11334 10/18/2013 12:48 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: right-aligned legend so it isn't too close to the "During week of:" label

11333 10/18/2013 12:46 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: added legend:
• task
◦ timespan
✓ task progress
☑ timespan progress

11332 10/18/2013 10:25 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: attribution/conditions of use: made it a subtask of "add missing columns" because this is related to data needed for published analyses. added dots because this is an ongoing task, that depends on data providers getting their use conditions to us.

11331 10/18/2013 10:08 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: reload core & analytical database: moved next reload ahead to last week of October so that we can include the updated geovalidation data for the 10/31 deadline. added additional reload so that they are spaced <= 1 month apart.

11330 10/18/2013 09:54 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: receive feedback from documentation tester: added an extra week to receive additional feedback from them in response to documentation fixes made

11329 10/18/2013 09:51 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: attribution/conditions of use: made this a top-level task instead of a subtask of "data provider metadata", to avoid including lower-priority tasks (i.e. in the later column) in the same section as higher-priority tasks

11328 10/18/2013 09:47 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: regrouped by subtask instead of by datasource, so that the high-priority subtasks get done for all datasources before moving on to lower-priority subtasks for any datasources

11327 10/18/2013 08:54 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: reduced width of Milestone column to make room to fit an additional week on the printed page

11326 10/18/2013 08:42 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: attribution/conditions of use: removed "(Brad/Brian/Bob/etc.)" because these are from everyone who provided or obtained data, not just Brad/Brian/Bob

11325 10/18/2013 08:40 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: rescheduled tasks to accommodate the separate non-critical feature requests subtasks

11324 10/18/2013 08:37 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: split "fix feature requests" into separate "fix critical feature requests" and "fix non-critical feature requests" tasks. rescheduled non-critical feature requests until after the other validation tasks have been completed.

11323 10/18/2013 08:11 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: add globally-unique occurrenceID: moved up to next week because we would like to be able to get this done for the 10/31 deadline

11322 10/18/2013 07:52 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11321 10/18/2013 07:48 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "data provider metadata" before "datasource validations (spot-checking)" because conditions of use are necessary for scientists who want to publish papers based on the data (which is a key use case)

11320 10/18/2013 07:43 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "usability testing" before "datasource validations (spot-checking)" because this is most important towards reaching our goal of a useful information resource

11319 10/18/2013 07:41 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "geoscrubbing re-run", "add globally-unique occurrenceID" back under "usability testing" > "add missing columns" because these are in fact part of the usability testing

11318 10/18/2013 01:26 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11317 10/17/2013 11:59 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11316 10/17/2013 10:32 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: "flatten the datasources to a common schema": moved to later column because the complex tasks "switching to new-style import" and "create interactive scripts for each import step" are also scheduled then. (it's unlikely we would have much time over winter break anyway, considering that there is ~1 week's worth of holidays then.)

11315 10/17/2013 10:24 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: scheduled "simplify import process for easier maintainability"

11314 10/17/2013 10:20 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: tasks performed by someone else (geoscrubbing re-run): changed solid check marks ✓ to open check marks ☑ to match the solid • vs. open ◦ dot convention

11313 10/17/2013 10:17 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: documentation testing: added supertask dots. removed later dots for scheduled tasks.

11312 10/17/2013 10:15 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: scheduled "documentation testing"

11311 10/17/2013 10:02 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: scheduled "simplify process of mapping/adding a new datasource"

11310 10/17/2013 10:01 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: "add globally-unique occurrenceID": moved it up to the first week when we're no longer fixing existing issues in datasources, since this has similar priority to adding missing columns discovered during usability testing (which is scheduled as an ongoing task)

11309 10/17/2013 09:54 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: usability testing: did task breakdown (find scientists who want to use BIEN3 data, etc.) and scheduled subtasks

11308 10/17/2013 09:34 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "add missing columns" to its own supertask. used outline check mark ☑ (analogous to open circle ◦) to mark supertasks as completed which were split up into subtasks.

11307 10/17/2013 09:29 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: later column: removed dots from scheduled items

11306 10/17/2013 09:06 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "switching to new-style import"-related steps (other than for CVS) to separate "simplify import process for easier maintainability" supertask, since this is not part of the "simplify process of mapping/adding a new datasource" task

11305 10/17/2013 08:39 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: add any missing columns: added and scheduled step to add globally-unique occurrenceID

11304 10/17/2013 08:34 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: geoscrubbing re-run: added dots ◦ for this for the time when it can be worked on asynchronously by Paul Sarando

11303 10/17/2013 08:30 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: data provider metadata: added dots ◦ for the portion of "attribution and conditions of use" that can be worked on asynchronously by Brad/Brian/Bob

11302 10/17/2013 08:24 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: scheduled "aggregated validations" during the last 2 weeks of "datasource validations (spot-checking)", because these weeks are only spent fixing issues uncovered in the remaining datasources, so there may be extra time then

11301 10/17/2013 07:20 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: scheduled other tasks after "datasource validations (spot-checking)" is complete

11300 10/17/2013 07:14 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations (spot-checking): each datasource's validation supertask: added open circles ◦ spanning the length of the subtasks

11299 10/17/2013 07:12 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for supertasks that have been fully split into subtasks (not just itemizing a few subtasks), so that these don't count towards the bullets (estimated workload) in each week

11298 10/17/2013 07:09 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: use an open circle ◦ instead of a bullet • for tasks that are performed by someone other than me, so that these don't count towards the bullets (estimated workload) in each week

11297 10/17/2013 07:04 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations (spot-checking): split each datasource into subtasks and scheduled them

11296 10/17/2013 06:00 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "move denormalized validations to stage II", "move stage III validations to stage II" outside of "switching to new-style import" because the "switching to new-style import" step refers just to the per-datasource switching steps, not to the additional refactorings that would be needed to avoid dependency on the complex XPath mappings (mappings/VegCore-VegBIEN.csv)

11295 10/17/2013 05:57 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations (spot-checking): added subtasks for each of the remaining datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order)

11294 10/17/2013 05:43 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved non-validation-related tasks after the 10/31 deadline so that these are not taking time away from the validation

11293 10/17/2013 05:40 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "flatten the datasources to a common schema" under "simplify process of mapping/adding a new datasource" because this is also needed separately for datasources where the left-joining is not part of the validation

11292 10/17/2013 05:30 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: extended "revisions to VegBIEN schema" to length of "datasource validations (spot-checking)" because schema changes are expected as we add missing fields

11291 10/17/2013 05:23 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: crossed out and hid completed tasks ("find out amount remaining in BIEN3 budget")

11290 10/17/2013 05:17 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations (spot-checking): extended through the end of November because data providers' fixes on the remaining 10 datasources (wiki.vegpath.org/2013-10-17_conference_call#validation-order) are likely to add significantly to the issues and feature requests associated with these datasources (e.g. the 2nd-round VegBank validation added 4 issues and 5 feature requests). there is also expected to be wait time while data providers are responding (most likely in multiple rounds of feedback).

11289 10/17/2013 04:59 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: data provider metadata: removed "iPlant can do" because this actually requires Brad/Brian/Bob/other data providers to provide this info. however, this info may be findable on the web for some datasources.

11288 10/17/2013 04:53 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "data provider metadata" right after "datasource validations" because this is part of the completed database itself rather than the tools to maintain it

11287 10/17/2013 04:47 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: split "revisions to schema" into "revisions to VegBIEN schema" (part of datasource validations) and "revisions to normalized VegCore" (part of documentation)

11286 10/17/2013 04:44 PM Aaron Marcuse-Kubitza

bin/import_all: use just import_scrub, not reimport_scrub, because import_scrub now automatically publishes the datasource's import (i.e. removes the temp suffix)

11285 10/17/2013 04:43 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: import: remove the temp suffix once the import is done, so that the full database import doesn't keep the suffix attached to the datasources that import_all didn't import with reimport. removed unused import_publish target (instead use import_temp to invoke just the import without the temp suffix removal).

11284 10/17/2013 04:27 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved part of "switching to new-style import" under "datasource validations (spot-checking)" because this is necessary to validate CVS

11283 10/17/2013 04:24 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved "simplify process of mapping/adding a new datasource" and "documentation testing" after "usability testing" because these tasks were there to make it possible for people other than me to reload/add to the database, which we have now decided is a lower priority than creating the validated database itself

11282 10/17/2013 04:14 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: added weeks through the end of the year (12/31)

11281 10/17/2013 02:00 PM Aaron Marcuse-Kubitza

schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: changed dataset definition to the definition in normalized VegCore ("a collection of records from the same place, with the same attribution requirements"), following discussion with Ramona

11280 10/17/2013 01:13 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11279 10/17/2013 12:34 PM Aaron Marcuse-Kubitza

schemas/VegBIEN/attribution/BIEN 3 data use and attribution.docx: updated to Ramona's commented version

11278 10/17/2013 12:19 AM Aaron Marcuse-Kubitza

inputs/CVS/plot_/map.csv: realLatitude, realLongitude: remapped to UNUSED because these columns are actually empty

11277 10/13/2013 07:32 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: collector_ID: remapped it to UNUSED and removed the join to party via it, like in VegBank

11276 10/13/2013 07:03 PM Aaron Marcuse-Kubitza

inputs/CVS/: deleted stemLocation_, because the CVS stemLocation table is empty (unlike VegBank)

11275 10/13/2013 05:26 PM Aaron Marcuse-Kubitza

inputs/CVS/import_order.txt: added plantConcept_/ so it would get automapped after switching to new-style import

11274 10/13/2013 04:24 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: denorm_{tri,quad}*: mapped to infraspecificRank*, infraspecificEpithet*

11273 10/13/2013 04:02 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: infraspecific ranks: remapped to EQUIV#to:species (which is the speciesBinomial), because these actually contain the full taxonomic name at that rank, like VegBank

11272 10/13/2013 03:50 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: genus: documented that unlike VegBank, does not include genus author

11271 10/13/2013 03:47 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: denorm_* terms _alt-ed with normalized terms: use DUPLICATE#of instead where possible. documented where and why _alt was necessary (this applies to a few rows for division, genus).

11270 10/13/2013 03:42 PM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/taxonObservation_/map.csv: species: remapped to speciesBinomial, not specificEpithet (like for VegBank). however, note that denorm_species is in fact the epithet, unlike VegBank.

11269 10/13/2013 03:19 PM Aaron Marcuse-Kubitza

fix: inputs/CVS/taxonObservation_/postprocess.sql: removed {} around denorm_genus to match the normalized genus

11268 10/13/2013 02:28 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: removed unnecessary alts for terms that don't have a duplicate denorm* or hierarchical field

11267 10/13/2013 02:06 PM Aaron Marcuse-Kubitza

fix: inputs/CVS/taxonObservation_/postprocess.sql: fix 1 row that has denorm_kingdom != Kingdom (i.e. both NOT NULL but not the same)

11266 10/13/2013 12:46 PM Aaron Marcuse-Kubitza

bugfix: lib/common.Makefile: $(subMake): don't enclose the target in "" because sometimes the target is empty (i.e. `all`), and nothing should be passed to the sub-make

11265 10/13/2013 12:10 AM Aaron Marcuse-Kubitza

inputs/VegBank/plot_/create.sql: documented runtime (5 min)

11264 10/13/2013 12:10 AM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/plot_/create.sql: like for VegBank, need to compare place.*PLOT_ID*, not PLOTPLACE_ID, with plot.PLOT_ID