Project

General

Profile

Statistics
| Revision:

# Date Author Comment
11616 11/09/2013 04:16 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: schema comment: added steps to determine what changes need to be made on vegbiendev

11615 11/09/2013 04:01 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_fields(): regenerate the derived cols: updated runtimes (~same)

11614 11/09/2013 03:54 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: moved instructions to apply schema changes on vegbiendev to the TNRS schema, because this applies to all elements in the TNRS schema, not just the tnrs table

11613 11/09/2013 03:30 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: score_ok(): don't make it STRICT because this prevents it from being inlined

11612 11/09/2013 03:24 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: removed no longer used tnrs_score_ok index. use tnrs__valid_match instead.

11611 11/09/2013 03:09 PM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): is_valid_match: documented that this excludes homonyms because these are not valid matches (i.e. TNRS provides a name, but the name is not meaningful because it is not unambiguous)

11610 11/09/2013 03:07 PM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: ValidMatchedTaxon: exclude inter-kingdom homonyms because these are not valid matches (i.e. TNRS provides a name, but the name is not meaningful because it is not unambiguous). this uses taxon_scrub__is_valid_match instead of score_ok() to determine whether the result should be included.

11609 11/09/2013 02:56 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: ValidMatchedTaxon: synced to MatchedTaxon

11608 11/09/2013 02:55 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: added is_valid_match

11607 11/09/2013 02:52 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: added tnrs__valid_match index to facilitate joining to only valid matches

11606 11/09/2013 02:48 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: added is_valid_match derived column, to make it easier to select from only those TNRS results that can safely be used as a scrubbed name

11605 11/09/2013 02:02 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: already_exists_msg(): added instructions on how to force-remake when the file already exists (prepend `rm=1` to the command)

11604 11/09/2013 02:20 AM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/test.xml.ref: updated inserted row count, now that CVS plots have been removed

11603 11/08/2013 10:57 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)

11602 11/08/2013 10:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: reordered functions in the order they are called by import()

11601 11/08/2013 10:28 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/: need to remove inter-datasource duplicates from plot instead of the left-joined plot_ table, because the fkeys needed to do the cascading deletes are all to the plot table. this requires doing the column-renaming and postprocessing on plot before it's left-joined.

11600 11/08/2013 09:57 PM Aaron Marcuse-Kubitza

inputs/VegBank/plot_/create.sql: updated runtime (5 s) for previous bugfix

11599 11/08/2013 07:50 PM Aaron Marcuse-Kubitza

exports/2013-7-10.Naia.range_limiting_factors.csv.run: updated export_() runtime and rowcount (~ the same)

11598 11/08/2013 04:26 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: filter by coordinateUncertaintyInMeters filter: assume true for rows with no coordinateUncertaintyInMeters

11597 11/08/2013 03:43 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: filter by coordinateUncertaintyInMeters <= 10 km

11596 11/07/2013 04:41 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11595 11/07/2013 04:00 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/run: load_data(): updated runtime (4 min)

11594 11/07/2013 08:42 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11593 11/07/2013 08:34 AM Aaron Marcuse-Kubitza

bugfix: inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: invoking derived/biengeo/geoscrub.sh: need to split the input file into separate dir and filename parts, because $DATAFILE actually is just the filename, not the entire path, and will otherwise get prepended with the default value for $DATADIR

11592 11/06/2013 04:57 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/geoscrub.csv.run: also run geoscrub.sh. added export_() target to run just the export of the result table separately.

11591 11/06/2013 04:39 PM Aaron Marcuse-Kubitza

derived/biengeo/load-geoscrub-input.sh: allow the caller to override $DATAFILE in the environment, to use a file named other than "geoscrub-corpus.csv"

11590 11/06/2013 02:41 PM Aaron Marcuse-Kubitza

/run: use new exports/geoscrub_input.csv.run

11589 11/06/2013 02:40 PM Aaron Marcuse-Kubitza

added exports/geoscrub_input.csv.run

11588 11/06/2013 02:39 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: $remake: need to explicitly propagate this to invoked commands if it was set from $rm

11587 11/06/2013 12:34 PM Aaron Marcuse-Kubitza

derived/biengeo/load-geoscrub-input.sh: updated $DATA_URL for new input filename

11586 11/06/2013 12:27 PM Aaron Marcuse-Kubitza

/run geoscrub_input/make(): include a header on the CSV file, so that the column names don't risk getting spliced from the data (and to shorten the CSV filename, which had to contain the column names instead). this requires changing the geoscrubbing scripts to accept a CSV header.

11585 11/06/2013 11:22 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11584 11/06/2013 10:14 AM Aaron Marcuse-Kubitza

exports/2013-7-10.Naia.range_limiting_factors.csv.run: added rowcount (40 million of 80 million observations, filtered w/ cultivated, geovalid, and various fields NOT NULL)

11583 11/06/2013 01:46 AM Aaron Marcuse-Kubitza

exports/2013-7-10.Naia.range_limiting_factors.csv.run: updated export_() runtime

11582 11/06/2013 01:30 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: don't sort the results by occurrence_id, because this is not a meaningful ordering and prevents incremental output from the query

11581 11/06/2013 01:09 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: 2013-7-10.Naia.range_limiting_factors: also filter out rows without species

11580 11/06/2013 12:59 AM Aaron Marcuse-Kubitza

exports/2013-7-10.Naia.range_limiting_factors.csv.run: export_(): documented runtime (10 min)

11579 11/05/2013 11:13 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select(): usage: documented that this also takes a $limit/$n param

11578 11/05/2013 11:12 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: limit(): also support using $n as the limit param, since this var name is used by other parts of the import process

11577 11/05/2013 11:08 PM Aaron Marcuse-Kubitza

added backups/vegbien.r11549.backup.md5

11576 11/05/2013 11:07 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: limit(): usage: documented that this also need a $limit param

11575 11/05/2013 11:06 PM Aaron Marcuse-Kubitza

backups/TNRS.backup.md5: updated

11574 11/05/2013 10:47 PM Aaron Marcuse-Kubitza

lib/runscripts/extract.run: added export_sample()

11573 11/05/2013 10:31 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: after import: record the import times in inputs/import.stats.xls: documented that this should be run on the local machine, because it needs the Mac filename ordering

11572 11/05/2013 10:30 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11571 11/05/2013 10:19 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

11570 11/05/2013 08:54 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: after import: removed step to install analytical_stem on nimoy because the import mechanism is not set up to do this (we don't generate CSV exports of the full analytical_stem table because they take up a lot of space and are not currently used for anything)

11569 11/05/2013 08:32 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: after import: In PostgreSQL: added step to check that analytical_stem contains the expected # of rows

11568 11/05/2013 08:16 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: after import: In PostgreSQL: added specific instructions for determining which/how many datasources are expected to be included in the provider_count and source tables

11567 11/05/2013 08:05 PM Aaron Marcuse-Kubitza

added inputs/analytical_db/_archive/

11566 11/05/2013 07:46 PM Aaron Marcuse-Kubitza

inputs/analytical_db/: removed import-related files (Source/, etc.), since this is actually just a folder used to store make_analytical_db.log.sql, so that it will be synced along with the other logs

11565 11/05/2013 07:43 PM Aaron Marcuse-Kubitza

inputs/analytical_db/: added _no_import to prevent this from incorrectly being included in the source table

11564 11/05/2013 07:27 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(_svnFilesGlob): also svn-add _no_import in the top-level datasrc dir. (this requires using add! , because the presence of a _no_import file there will normally turn off adding by svnFilesGlob.)

11563 11/05/2013 11:49 AM Paul Sarando

Added an output CSV file option to geoscrub.sh.

11562 11/04/2013 03:25 PM Paul Sarando

Added notes on running biengeo scripts to README.

11561 10/31/2013 05:35 PM Paul Sarando

Added biengeo script options for data directories.

Added GADM and geonames.org data dir options to
update_validation_data.sh scripts.
Added geoscrub input data dir option to geoscrub.sh scripts.

11560 10/31/2013 05:35 PM Paul Sarando

Added update options to biengeo update_validation_data.sh

Added options to update only GADM data, only Geonames.org data, or
neither. In every case, the geonames-to-gadm scripts are always run.

11559 10/31/2013 05:35 PM Paul Sarando

Added cmd-line options to biengeo bash scripts.

All biengeo bash scripts now accept command line options to specify psql
user, host, and database values.
These options are the same as those defined by the psql command.
If an invalid option is given to a script, a usage message is printed...

11558 10/31/2013 05:35 PM Paul Sarando

Fix biengeo script password prompt for postgres user.

Changed the DB_HOST variables in the biengeo bash scripts to a
DB_HOST_OPT variable that is blank by default.
Updated all psql calls that used "-h $DB_HOST" to use just $DB_HOST_OPT
instead.
This means that to specify a different db host, the DB_HOST_OPT...

11557 10/31/2013 05:35 PM Paul Sarando

Fixed TRUNCATE statement in truncate.geonames.sql.

Fixed the biengeo truncate.geonames.sql script to include all tables in
one TRUNCATE statement that have foreign-key references to geonames and
country tables.

11556 10/31/2013 05:35 PM Paul Sarando

Added more approx. runtimes to biengeo README.

11555 10/31/2013 05:35 PM Paul Sarando

Renamed biengeo install scripts to setup scripts.

It seems to make more sense to call these setup scripts, since they are
only setting up the database and tables, and not actually installing any
files anywhere on the OS.

11554 10/31/2013 12:29 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11553 10/31/2013 12:24 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: CVS: left-join it: moved under "fix issues and critical feature requests" instead of "prepare 1st-round extracts" because the left-joining is actually part of getting it in the same format as VegBank

11552 10/31/2013 11:12 AM Aaron Marcuse-Kubitza

inputs/CTFS/StemObservation/test.xml.ref: updated inserted row count

11551 10/31/2013 10:30 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: rescheduled CVS before other datasources, as decided in the conference call

11550 10/31/2013 10:27 AM Aaron Marcuse-Kubitza

schemas/Makefile: $(confirmRmPublicSchema0): use "any ... schema" instead of "the ... schema" because the schema in question may not exist

11549 10/31/2013 08:53 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: rescheduled tasks for new order

11548 10/31/2013 08:42 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: reordered to put plots before specimens, as requested by Brad (wiki.vegpath.org/2013-10-25_conference_call#validation-order)

11547 10/31/2013 08:24 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress

11546 10/31/2013 08:21 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: hid previous weeks

11545 10/31/2013 08:20 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: crossed out and hid completed tasks

11544 10/31/2013 08:17 AM Aaron Marcuse-Kubitza

fix: planning/timeline/timeline.2013.xls: datasource validations: prepare 2nd-round extracts: VegBank: corrected check mark week, based on date of extract

11543 10/31/2013 08:14 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: added "prepare 3rd-round extracts" subtask, which currently applies to VegBank. updated for progress.

11542 10/31/2013 08:08 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: "datasource validations (spot-checking)": renamed to just "datasource validations" because that's what we've been calling it

11541 10/31/2013 08:08 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: datasource validations: CVS: added "VegBank-related changes" subtask

11540 10/31/2013 08:05 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for progress and revised schedule

11539 10/31/2013 07:51 AM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/import_order.txt: updated name of ^taxon_observation.**.sample table

11538 10/31/2013 07:16 AM Aaron Marcuse-Kubitza

fix: inputs/VegBank/^taxon_observation.**.sample/create.sql: moved continent before country

11537 10/31/2013 06:54 AM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql: added missing columns that were recently mapped to VegBIEN (identifiedBy)

11536 10/31/2013 06:52 AM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql: synced column order to analytical_plot

11535 10/31/2013 06:49 AM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql: synced column order to analytical_plot

11534 10/31/2013 06:47 AM Aaron Marcuse-Kubitza

inputs/VegBank/taxonobservation_/map.csv, postprocess.sql: mapped identifiedBy (the join_words() of identifiedBy_first, etc.)

11533 10/31/2013 06:22 AM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: analytical_plot, analytical_specimen: removed derived columns that are not part of the validation

11532 10/31/2013 06:15 AM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: analytical_plot, analytical_specimen: removed internal ID columns that are not part of the validation

11531 10/31/2013 05:46 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_plot: removed derived columns that should not be validated by data providers

11530 10/31/2013 05:42 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_specimen: synced to analytical_stem

11529 10/31/2013 05:36 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_plot: documented that this contains all of the analytical_stem columns, minus specimenHolderInstitutions, collection, accessionNumber, occurrenceID

11528 10/31/2013 05:34 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_plot: synced to analytical_stem

11527 10/31/2013 05:29 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: added individualCount

11526 10/31/2013 04:42 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: plot.**, analytical_stem_view: added slopeAspect, slopeGradient

11525 10/31/2013 03:41 AM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.mwb: traceable.id_by_source: support multiple ids_by_source per traceable, because the same entity may be present in multiple datasources (e.g. if one got data from the other), and we would like to remove that duplicate

11524 10/31/2013 02:46 AM Aaron Marcuse-Kubitza

inputs/VegBank/taxonobservation_/map.csv, postprocess.sql: mapped identifiedBy (the join_words() of identifiedBy_first, etc.)

11523 10/31/2013 02:34 AM Aaron Marcuse-Kubitza

inputs/VegBank/taxonobservation_/create.sql: also join party_id to get the identifiedBy (not mapped yet). note that the inserted row count changes, because taxonobservation_ does not yet have a pkey to do a stable ordering with.

11522 10/31/2013 02:16 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/install: don't run map_table, because this instead done by the runscript. although it does not hurt to do it twice, invoking load_data by itself should not run map_table at all, so that the original column names can be inspected in the table and map.csv reordered to match.

11521 10/31/2013 02:06 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: taxoninterpretation.party_id: don't rename to taxoninterpretation_party_id, so that this can be used directly in taxonobservation_/create.sql with a USING join

11520 10/31/2013 01:52 AM Aaron Marcuse-Kubitza

inputs/VegBank/taxonobservation_/create.sql: join taxonobservation to taxoninterpretation (as in CVS) instead of vice versa, since taxonobservation is the primary, operative table. having VegBank and CVS do things the same way helps ensure that fixes in one can transfer easily to the other.

11519 10/31/2013 01:51 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/install: don't run map_table, because this instead done by the runscript. although it does not hurt to do it twice, invoking load_data by itself should not run map_table at all, so that the original column names can be inspected in the table and map.csv reordered to match.

11518 10/31/2013 01:30 AM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql: synced with taxon_observation.**

11517 10/31/2013 01:22 AM Aaron Marcuse-Kubitza

(for r11396) fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error