Project

General

Profile

Statistics
| Revision:

# Date Author Comment
11454 10/29/2013 02:17 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: provide scientists with their requested data: added separate subtask for Brian Enquist's Canadensys extract

11453 10/29/2013 02:12 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress and revised schedule

11452 10/29/2013 01:53 AM Aaron Marcuse-Kubitza

bugfix: schemas/pg_hba.Mac.conf: made same change for Mac as was made for Linux in r11451

11451 10/29/2013 01:22 AM Aaron Marcuse-Kubitza

bugfix: schemas/pg_hba.conf: don't allow ident authentication for Unix socket connections, because this apparently prevents having normal, password-based connections ("md5"). note that just switching the order of the ident and md5 entries is not useful, because whichever authentication type comes second will be ignored completely. this problem was previously worked around by just not using Unix socket connections at all, and always specifying "localhost" as the host to force a hostname-based connection. this does not affect the postgres superuser, because they have their own ident line in pg_hba.conf.

11450 10/25/2013 06:15 PM Paul Sarando

Added db user and host to load-geoscrub-input.sh

The psql commands in load-geoscrub-input.sh will now connect with a
specific user on a specific host.
Updated the 'COPY' sql statement to a '\COPY' statement, so that the
psql user does not have to be a PostgreSQL superuser.

11449 10/25/2013 04:51 PM Aaron Marcuse-Kubitza

derived/biengeo/README.txt: geoscrub new data: steps that use .sql scripts: added the psql commands to run these

11448 10/25/2013 04:22 PM Paul Sarando

Updated install instructions in the README.

11447 10/25/2013 03:00 PM Aaron Marcuse-Kubitza

derived/biengeo/README.txt: geoscrub new data: noted that this now deletes any previous geoscrubbing results

11446 10/25/2013 02:58 PM Aaron Marcuse-Kubitza

derived/biengeo/README.txt: added steps to set the working dir for each set of steps

11445 10/25/2013 02:54 PM Aaron Marcuse-Kubitza

derived/biengeo/README.txt: added section on obtaining source code, including path to Paul's in-progress files on vegbiendev (not sure whether the in-progress files are needed to run the core scripts in steps 1-6)

11444 10/25/2013 02:44 PM Aaron Marcuse-Kubitza

derived/biengeo/README.txt: moved commands to run to the top of the README. flagged commands-sections with *** and an identifying label.

11443 10/25/2013 02:04 PM Paul Sarando

Initial checkin of geoscrub install SQL files.

Added install.*.sql files that will do initial table creation for all
required tables.
Added a truncate.vegbien_geoscrub.sql script that will clear tables related to
data downloaded in load-geoscrub-input.sh....

11442 10/25/2013 02:04 PM Paul Sarando

Update load-geoscrub-input.sh to download from URL.

Removed logic to dump input data directly from the vegbien database and
to download the input from a URL provided by AMK instead.
Also updated this script to download the file into an input data
directory, rather than just into the current working directory.

11441 10/25/2013 11:56 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: reload core & analytical database scheduled for this week: postponed to give us additional time to do datasource validations

11440 10/25/2013 09:58 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: added %/import_temp alias for %/import, to mirror the presence of import_temp for import

11439 10/25/2013 09:24 AM Aaron Marcuse-Kubitza

fix: inputs/VegBank/taxonobservation_/map.csv: remapped authorplantname to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.

11438 10/25/2013 09:22 AM Aaron Marcuse-Kubitza

fix: inputs/VegBank/taxonobservation_/map.csv: remapped int_* to OMIT because these are not specific to the taxoninterpretation row (this is in a separate taxoninterpretation for the original determination instead). see wiki.vegpath.org/Spot-checking#2013-10-10 > Mike Lee's conference call feedback.

11437 10/24/2013 07:09 PM Aaron Marcuse-Kubitza

exports/2013-10-18.Brian_Enquist.Canadensys.csv.run: inherit from new import_subset.run (which uses extract.run)

11436 10/24/2013 07:08 PM Aaron Marcuse-Kubitza

added lib/runscripts/import_subset.run, extract.run

11435 10/24/2013 05:21 PM Aaron Marcuse-Kubitza

added exports/2013-10-18.Brian_Enquist.Canadensys.csv.run

11434 10/24/2013 05:07 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: removed no longer needed setting of $schema to $public, because this is now done by psql()

11433 10/24/2013 05:06 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: psql(): also accept $public as the $schema param, since this is used by a lot of import scripts

11432 10/24/2013 04:24 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added require_dot_script()

11431 10/24/2013 04:13 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: $top_script: use @BASH_SOURCE instead of $0, because this is also defined for .-scripts

11430 10/24/2013 04:03 PM Aaron Marcuse-Kubitza

bugfix: bin/import_all: restore the working dir when main() is done, in case it started as something other than the root dir

11429 10/24/2013 03:49 PM Aaron Marcuse-Kubitza

bin/after_import: support turning off the end-of-import backup for imports that are not the full database

11428 10/24/2013 03:26 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.

11427 10/24/2013 03:21 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: if is_dot_script, fix $ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).

11426 10/24/2013 03:17 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: is_dot_script(): need to subtract 1 from ${#BASH_LINENO[@]}, because this is the array length rather than the index of the last element as in Perl

11425 10/24/2013 02:58 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added is_dot_script()

11424 10/24/2013 01:15 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: taxondetermination_set_iscurrent(): is_datasource_current (used by analytical_stem_view): need to separately check if `determinationtype IS NULL`, because `determinationtype NOT IN (accepted, matched))` will return NULL (false) if determinationtype is NULL, causing no match

11423 10/24/2013 01:11 PM Aaron Marcuse-Kubitza

bugfix: bin/make_analytical_db: when running into a public schema other than "public", also pass this to `/run export_` (which currently uses $schema instead of $public)

11422 10/24/2013 01:10 PM Aaron Marcuse-Kubitza

bugfix: bin/import_all: fix $ when .-included without args (which causes bash to put the wrong values in $ instead of leaving it empty)

11421 10/24/2013 01:09 PM Aaron Marcuse-Kubitza

bin/import_all: `make schemas/$version/install`: reinstall instead to allow re-running the import to the same custom schema (e.g. 2013-10-18.Brian_Enquist.Canadensys)

11420 10/24/2013 01:07 PM Aaron Marcuse-Kubitza

bin/import_all: `make schemas/$version/install`: ignore errors if schema exists, to support running with -e

11419 10/23/2013 11:10 PM Aaron Marcuse-Kubitza

bugfix: bin/import_all: removing inputs/.TNRS/tnrs/tnrs.make.lock: use `"rm" -f` instead of plain "rm" to avoid having an error exit status, which will abort the script if run with the -e flag (as runscripts are)

11418 10/23/2013 11:02 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args

11417 10/23/2013 11:00 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)

11416 10/23/2013 10:34 PM Aaron Marcuse-Kubitza

bin/*_all: *_main(): renamed to just main() because it does not matter that other shell-includes' main() methods will clobber this, because it is only executed once

11415 10/23/2013 10:29 PM Aaron Marcuse-Kubitza

bugfix: bin/import_all: Source tables: use .../import instead of import_temp because import_temp is only needed when importing all tables, to prevent the temp suffix from being removed yet

11414 10/23/2013 10:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV

11413 10/23/2013 09:11 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: alias_append(): need to enclose $(alias) call in "" because its result may contain separator chars (i.e. whitespace) that will be parsed incorrectly. this appears to only be a bug when runscripts are run as shell-includes, with a leading ".".

11412 10/23/2013 01:49 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: connecting lines: inherits from traceable: added arrow to indicate what this label refers to

11411 10/23/2013 01:43 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11410 10/23/2013 01:37 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: HAS-A/IS-A box: renamed to "connecting lines" for clarity

11409 10/22/2013 10:22 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: HAS-A: added HAS-MANY going in the opposite direction, because every HAS-A has an opposite HAS-MANY

11408 10/22/2013 10:17 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: relationships: IS-A, HAS-A: added directional arrows

11407 10/22/2013 10:10 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: field order box: removed spacing between top of text box and bottom of outer box label

11406 10/22/2013 10:04 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11405 10/22/2013 10:00 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: reordered columns according to the field order convention

11404 10/22/2013 09:49 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: added label documenting the field order convention:
1) inherited
2) required
3) identifying
4) foreign key
5) extenders
6) others

11403 10/22/2013 08:34 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added links for EER models, data management plans. put PostgreSQL before MySQL because we have found PostgreSQL to be a much more capable database system, even though it lacks some of MySQL's user-friendly features.

11402 10/22/2013 06:38 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11401 10/22/2013 06:21 PM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names

11400 10/22/2013 06:16 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: regenerated using `make schemas/remake`. note that analytical_stem_view column renamings need this step after a search-and-replace of the column names, in order to remove excess "" around all-lowercase names and reset generated index names.

11399 10/22/2013 06:10 PM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: analytical_stem_view: renamed specimens columns to use the VegCore names, where these differ from DwC, so that the now-VegCore staging table column names are the same as the analytical_stem_view column names

11398 10/22/2013 01:20 PM Aaron Marcuse-Kubitza

added planning/goals/web_interface/phpPgAdmin.select_interface.png for use at wiki.vegpath.org/Proposed_enhancements

11397 10/22/2013 09:39 AM Aaron Marcuse-Kubitza

inputs/CVS/_src/: added refresh from Mike Lee

11396 10/21/2013 07:14 PM Aaron Marcuse-Kubitza

fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error

11395 10/21/2013 05:16 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: usability testing: added subtask to provide scientists with their requested data

11394 10/20/2013 05:51 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11393 10/20/2013 05:21 PM Aaron Marcuse-Kubitza

bugfix: bin/import_all: need to publish datasources that won't be published by `make .../import`, so that the per-datasource import XPaths that refer to TNRS/geoscrub will link up with the TNRS/geoscrub source entry instead of creating a new entry without the metadata (because the entry with the metadata was named TNRS.new/geoscrub.new)

11392 10/20/2013 05:09 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: datasource_publish(): use parameter names instead of $# because this is a PL/pgSQL function

11391 10/20/2013 05:07 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: datasource_publish(): if the datasource to publish already has the published name, don't datasource_rm() it

11390 10/20/2013 04:55 PM Aaron Marcuse-Kubitza

bin/import_all: removed no longer needed import of geoscrub data, because analytical_stem_view is now joined to the geoscrub_output table directly, instead of using the imported canon_place entries

11389 10/20/2013 04:52 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: join to the geoscrub_output table directly, instead of using the imported canon_place entries. this avoids the need to import geoscrub_output into VegBIEN (which is expected to take 2+ hours after the refresh), as well as the need to then refresh any datasources whose geoscrubbing input data has changed.

11388 10/20/2013 04:37 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/postprocess.sql: added nullable unique index on the inputs, for use by analytical_stem_view. note that it must be nullable in order to create a match when not all of the input fields are populated. this uses array[] to create a nullable index, which is much better than column-based import and VegBIEN's use of COALESCE because the expression is the same for every type and no NULL sentinel value is needed.

11387 10/20/2013 03:50 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

11386 10/20/2013 03:41 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: regenerated exports and udpated image map

11385 10/20/2013 03:34 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: person: allow to have multiple organizations

11384 10/20/2013 03:28 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: split "2b. GNRS" label into two labels, one for each table GNRS is applied to

11383 10/20/2013 03:23 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: georeferencing: merged into geoplace, since this is actually information attached to a specific plot, etc. relating to the coordinates used in its geoplace subclass

11382 10/20/2013 02:59 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: geovalidatable_place: changed parent geoplace pointer to parent_boundary_WKT, since the immediate parent may not have an associated boundary to use for geovalidation (i.e. it may not be an official GADM geoplace), although ancestors further up likely will be

11381 10/20/2013 02:40 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place.name: made it required because it's needed for the unique constraint to be populated properly (including for subclasses such as geoplace, which need to generate this from the coordinates)

11380 10/20/2013 02:30 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place.rank: made it required, because every place should have some kind of rank indicating what type of place it is, including lower ranks (e.g. plot, individual)

11379 10/20/2013 02:24 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place: added unique constraint on parent, rank, name

11378 10/20/2013 02:24 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have

11377 10/20/2013 02:07 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: place.locality: moved to geopath, because this is actually a rank of place (i.e. below municipality) rather than a field that every place could have

11376 10/20/2013 01:46 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: geoplace.official_name: renamed to name to merge with inherited field from place. documented that for geoplaces, this is the official, scrubbed name.

11375 10/20/2013 12:48 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/postprocess.sql: added geovalid derived column, for use by analytical_stem_view

11374 10/19/2013 06:56 PM Aaron Marcuse-Kubitza

bin/with_all: $all: renamed to $hidden_srcs for clarity, since this now just adds the hidden (.*) datasources, rather than always using all datasources

11373 10/19/2013 06:50 PM Aaron Marcuse-Kubitza

bugfix: bin/with_all: in $all mode, just prepend the .* datasources to the user-selected (or default) @inputs, so that using $all to add these datasources doesn't inadvertently cause the action to be performed for all datasources

11372 10/19/2013 04:33 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. PostgreSQL: ALTER TABLE: added documentation about disabling of foreign key triggers, which is only possible by the superuser. note that marking a foreign key constraint as NOT VALID does not disable the trigger, so NOT VALID cannot be used for this purpose. this would be used to add fkeys from core VegBIEN tables to validation results tables such as the geoscrubbing results, without needing to import the validation results directly into core VegBIEN (which is time-consuming and currently must be done before input data is loaded, requiring a datasource reload to add geoscrubbing results).

11371 10/19/2013 02:15 PM Aaron Marcuse-Kubitza

bin/import_all: usage: documented that this can now be run with a custom datasources list (each of the form inputs/src/)

11370 10/19/2013 02:02 PM Aaron Marcuse-Kubitza

bin/with_all: added support for providing a custom list of inputs to run the command on

11369 10/19/2013 01:29 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/postprocess.sql, run: updated runtimes

11368 10/19/2013 01:13 AM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/run: documented full load_data() runtime (9 min @starscream)

11367 10/19/2013 01:12 AM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/postprocess.sql: updated runtimes for refreshed data, which now has 4x as many rows (1,707,970->6,747,650)

11366 10/19/2013 12:54 AM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/: refreshed geoscrub data. removed +header.csv because the extract now contains the header in the first row of the file.

11365 10/19/2013 12:52 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): $is_root: use `` around case statement instead of $(), because it contains an embedded unbalanced )

11364 10/19/2013 12:27 AM Aaron Marcuse-Kubitza

bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: include only the columns that Jim provided in his extract (the geoscrub table contains additional internal columns that are not part of the geovalidation data for VegBIEN). documented runtime (30 s) and upload time (1.5 min).

11363 10/18/2013 10:33 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: removed no longer needed setting of $local_server, $local_user (and use of $local_pg_database instead of $database) because the use_local bug in local.sh has been fixed

11362 10/18/2013 10:32 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): don't default the connection vars using use_local if running as the postgres user. in that case, connection must happen via a socket, with server="", and as the user running the command (postgres), with user="".

11361 10/18/2013 09:55 PM Aaron Marcuse-Kubitza

bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: need to manually set local_server, local_user to "" so that they do not default to their bien-user values

11360 10/18/2013 09:54 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: avoid outputting to /dev/fd/# when running as sudo on Linux, because this causes a "Permission denied" error (due to the /dev/fd/# file being owned by a different user). this is not a problem with normal redirects (>&#), because they do not use /dev/fd/# files which can have access permissions.

11359 10/18/2013 09:52 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file

11358 10/18/2013 08:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command

11357 10/18/2013 07:50 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_as_root(): run sudo with echo_run to help debug

11356 10/18/2013 06:29 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_cmd(): only set PG* connection/login env vars when the corresponding var is non-empty. there are some situations in which these must be unset (in order to use the default value), and other situations when the var must be set to something (i.e. "") to avoid it being defaulted to a value in local.sh > connection vars.

11355 10/18/2013 06:27 PM Aaron Marcuse-Kubitza

backups/TNRS.backup.md5: updated