Project

General

Profile

Statistics
| Revision:

# Date Author Comment
7097 01/07/2013 05:39 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Use `SELECT source_id FROM source WHERE shortname = ...` instead of source_by_shortname() so that the source table is updated to point to the same schema as the view rather than pointing to whichever version (usually public) is first in the search_path

7096 01/07/2013 05:23 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Fixed bug where need to include only those taxondeterminations that already have a match in TNRS.tnrs, to avoid adding empty TNRS taxondeterminations. As the concurrent tnrs daemon runs, these taxondeterminations will gradually acquire matches in tnrs and then be processed by scrub.

7095 01/07/2013 05:00 PM Aaron Marcuse-Kubitza

import_all: Make $import_source a local var, so it will be automatically unset if the script is aborted

7094 01/07/2013 04:49 PM Aaron Marcuse-Kubitza

vegbien_dest: Schema override for referring to a table in the $public schema: Only process the override when $!schemaVar and $!tableVar are non-*empty*, to allow setting $schema=""

7093 01/07/2013 04:47 PM Aaron Marcuse-Kubitza

schemas/Makefile: DDL generation: vegbien.sql: Unset $dump_opts so that pg_dump does not use env vars left after running import_all

7092 01/07/2013 04:44 PM Aaron Marcuse-Kubitza

schemas/Makefile: DDL generation: vegbien.sql: Unset $version so that pg_dump always uses the public schema, even after running import_all

7091 01/07/2013 04:13 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7090 01/07/2013 04:13 PM Aaron Marcuse-Kubitza

README.TXT: Testing: Added commands to put in .profile on a development machine

7089 01/07/2013 04:10 PM Aaron Marcuse-Kubitza

import_all: Added command to add scrubbed taxondeterminations

7088 01/07/2013 04:09 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7087 01/07/2013 04:08 PM Aaron Marcuse-Kubitza

import_all: Start tnrs-remake after starting the inputs, so that for subset imports (e.g. n=2), there will already be names to scrub when tnrs-remake starts up and it won't enter pause mode to wait for new rows (the pause is calibrated for full imports, and is too long for subset imports)

7086 01/07/2013 04:01 PM Aaron Marcuse-Kubitza

with_all: Also exclude .archive/ from the subdirs to forward commands to

7085 01/07/2013 03:40 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Added option to wait for new rows, in the same way tnrs_db does

7084 01/07/2013 03:38 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Factored new rows added test out into rowsAdded() function

7083 01/07/2013 03:09 PM Aaron Marcuse-Kubitza

Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make, which adds scrubbed taxondeterminations to VegBIEN

7082 01/07/2013 02:00 PM Aaron Marcuse-Kubitza

root Makefile: Removed $(subMake), which is now defined properly by lib/common.Makefile

7081 01/07/2013 01:59 PM Aaron Marcuse-Kubitza

lib/common.Makefile: $(subMake): Removed `--makefile=../input.Makefile`, which is specific just to inputs/Makefile

7080 01/07/2013 01:43 PM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: $(import): Print the date at the beginning of the import, so successive imports to the same version can be distinguished

7079 01/07/2013 01:37 PM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: \$(import): Fixed bug where 2>&1 needs to come after >>$(log_) rather than before

7078 01/07/2013 01:35 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/tnrs.make: Usage: Added tnrs_db's $wait flag

7077 01/07/2013 01:34 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/tnrs.make: Fixed Usage message to use make, which is needed to set the PATH correctly

7076 01/07/2013 11:47 AM Aaron Marcuse-Kubitza

Makefiles: Changed "Usage: `make -s ...`" to "Run with `make -s` to avoid echoing make commands"

7075 01/07/2013 11:44 AM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: Added %/log_file to view the import log file path

7074 01/07/2013 11:28 AM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: $(import): Append to the log file instead of replacing it, to avoid overwriting the log for a previous import to the same versioned schema. This allows a datasource to be (re-)imported multiple times, and is needed by the new method for linking taxonoccurrences to scrubbed taxonomic names.

7073 01/07/2013 11:22 AM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: $(import): Always output just to log file if $(log) is on, rather than also copying output to the terminal when $(n) is set. When $(log) is on, the output can still be viewed by tailing the log.

7072 01/07/2013 11:16 AM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(n),...)

7071 01/07/2013 11:14 AM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(log),...)

7070 01/07/2013 11:07 AM Aaron Marcuse-Kubitza

Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/

7069 01/07/2013 11:05 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped taxonOccurrencePkey

7068 01/07/2013 10:58 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7067 01/07/2013 10:41 AM Aaron Marcuse-Kubitza

input.Makefile: Staging tables installation: Added %_view/install, to prevent trying to edit a view during installation

7066 01/07/2013 10:31 AM Aaron Marcuse-Kubitza

vegbien_dest: Added schema override support for referring to a table in the $public schema

7065 01/07/2013 10:29 AM Aaron Marcuse-Kubitza

input.Makefile: Staging tables installation: $(cleanup): Moved setting of $schema, $table before vegbien_dest is run, so it can modify them if needed

7064 01/07/2013 09:42 AM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/tnrs.make: Removed unnecessary setting of $prefix, which now defaults to ""

7063 01/07/2013 09:40 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added unscrubbed_taxondetermination_view

7062 01/04/2013 10:10 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Moved CTFS to Deleted section

7061 01/04/2013 10:03 PM Aaron Marcuse-Kubitza

make_analytical_db: ANALYZE each table after its created so that queries use index scans instead of seq scans

7060 01/04/2013 09:40 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sync_analytical_*_to_view(): Added datasource fkey to source.shortname so removing a datasource will also remove the corresponding rows in the analytical views

7059 01/04/2013 09:36 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Regenerated analytical_stem using sync_analytical_stem_to_view()

7058 01/04/2013 09:19 PM Aaron Marcuse-Kubitza

input.Makefile: Editing import: rm: Time the command

7057 01/04/2013 09:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>

7056 01/04/2013 09:11 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Fixed fkey constraint names

7055 01/04/2013 09:09 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>

7054 01/04/2013 06:59 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: fkeys to source: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>

7053 01/04/2013 06:22 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: commconcept: Renamed source_id back to reference_id (it was previously renamed to source_id in a bulk rename)

7052 01/04/2013 06:20 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxondetermination: Added back reference_id, which is different than the scoping source_id (reference_id was previously renamed to source_id in a bulk rename)

7051 01/04/2013 06:04 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Renamed taxonconcept_concept_source_id_fkey back to taxonconcept_concept_reference_id_fkey

7050 01/04/2013 06:02 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Renamed *_reference_id_fkey fkeys to *_source_id_fkey

7049 01/04/2013 05:32 PM Aaron Marcuse-Kubitza

inputs/CTFS/_no_import: Temporarily remove CTFS from the public DB per Rick Condit's request (due to validation issues)

7048 01/04/2013 05:25 PM Aaron Marcuse-Kubitza

import_all: Run import with $public_import set in order to exclude excluded datasources

7047 01/04/2013 05:23 PM Aaron Marcuse-Kubitza

input.Makefile: Import to VegBIEN: %/import: Don't run the import if $public_import flag is set and the datasource contains a _no_import file. This allows just excluding a datasource from the public DB, without also removing it from automated testing.

7046 01/04/2013 05:17 PM Aaron Marcuse-Kubitza

lib/common.Makefile: Added $(and), $(or), $(not)

7045 01/04/2013 04:30 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7044 01/04/2013 04:12 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxondetermination: Added scoping source_id field to allow other datasources (e.g. TNRS) to make taxondeterminations. (Repurposed existing non-scoping source_id.)

7043 01/03/2013 08:25 AM Aaron Marcuse-Kubitza

make_analytical_db: Fixed bug where can't give public_ select access to all analytical_db views because this apparently adds access rather than passing through the underlying table's permissions

7042 01/03/2013 08:18 AM Aaron Marcuse-Kubitza

make_analytical_db: Give public_ select access to analytical_db views. This causes the actual access to depend on the underlying table's permissions.

7041 01/03/2013 07:43 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7040 01/03/2013 07:39 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Include only terms that start with a lowercase letter or are all caps. This also avoids the need to filter out VegCore.tables.csv.

7039 01/03/2013 07:31 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Changed line endings to \n in preparation for not running filter_out_cs on the file (which changes line endings to \r\n)

7038 01/03/2013 02:31 AM Aaron Marcuse-Kubitza

import_all: `make backups/vegbien.$version.backup/test`: Documented that this uses $dump_opts. $dump_opts must be manually set when running this command outside of import_all.

7037 01/03/2013 02:21 AM Aaron Marcuse-Kubitza

backups/Makefile: Synchronization: %/download: Download the .md5 file first, so that the user is prompted right away for their password rather than after the main file has finished downloading, by which time the password prompt has timed out

7036 01/03/2013 12:02 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7035 01/02/2013 11:34 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7034 01/02/2013 11:32 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Fixed bug where need to filter out VegCore.tables.csv case-sensitively so that field names which are the same as a table name don't get filtered out

7033 01/02/2013 11:23 PM Aaron Marcuse-Kubitza

Added filter_out_cs

7032 01/02/2013 09:21 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Added step to ensure there are no local modifications using `svn st`

7031 01/02/2013 08:56 PM Aaron Marcuse-Kubitza

make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_

7030 01/02/2013 07:14 PM Aaron Marcuse-Kubitza

README.TXT: Data import: After import: Also check that the provider_count table contains entries for all inputs

7029 01/02/2013 07:03 PM Aaron Marcuse-Kubitza

Added inputs/.geoscrub/_src/geovalidity-table.txt, which was attached to Jim's geovalidation e-mail (provided in README.TXT)

7028 01/02/2013 06:43 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

7027 01/02/2013 06:39 PM Aaron Marcuse-Kubitza

README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated column group header to "By column"

7026 01/02/2013 06:36 PM Aaron Marcuse-Kubitza

backups/Makefile: Removed no longer used $(psqlVerbose)

7025 01/02/2013 06:36 PM Aaron Marcuse-Kubitza

backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB

7024 01/02/2013 06:31 PM Aaron Marcuse-Kubitza

backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien

7023 12/21/2012 03:34 PM Aaron Marcuse-Kubitza

import_all: Allow caller to override $dump_opts

7022 12/21/2012 03:33 PM Aaron Marcuse-Kubitza

pg_dump_vegbien: Renamed $opts env var to $dump_opts to avoid conflicting with other commands' vars of the same name

7021 12/21/2012 03:22 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7020 12/21/2012 03:20 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7019 12/21/2012 02:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: location: Removed location_unique_within_parent_by_sourceaccessioncode, which duplicates location_unique_within_creator_by_sourceaccessioncode because the sourceaccessioncode is globally unique

7018 12/21/2012 02:10 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: projectID: Use project.projectname if project.sourceaccessioncode isn't provided

7017 12/21/2012 02:02 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: location: location_unique_within_parent: Split into *_by_sourceaccessioncode and *_by_authorlocationcode_position, with each ID being matched separately. This way, if the initial import of a subplot's location provides both fields, but fkey references use only one field, the fkey references will still match the existing location because only one of the fields needs to match.

7016 12/21/2012 01:26 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: elevationInMeters: Use parent_location.elevation_m when location.elevation_m not provided

7015 12/21/2012 01:17 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: scientificName: Fixed bug where need to use accepted_taxon*label*.taxonomicname instead of accepted_taxonverbatim.taxonomicname, because taxonverbatim's name component fields aren't populated if the name doesn't match a scrubbed name. The datasource's own taxonverbatim can't be used for this because the canon_label_id refers to the concatenated taxonomic name owned by the TNRS datasource.

7014 12/21/2012 01:00 PM Aaron Marcuse-Kubitza

inputs/NVS/Plot/map.csv: Corrected Plot ID mapping to go to subplotID instead of locationID, because each subplot gets its own ID in this field

7013 12/21/2012 12:50 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: location: location_unique_within_parent: Also apply this constraint when sourceaccessioncode is provided, because it may be a concatenated value populated for use by the analytical DB but which is not used as an fkey by the datasource itself

7012 12/21/2012 12:30 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: locationID: Concatenate parent location's and subplot's IDs using '; ' instead of ' '

7011 12/21/2012 12:22 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_*: Renamed locationName to locationID because it's now globally unique (within the datasource) and can be used as a sourceaccessioncode

7010 12/21/2012 12:19 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: locationName: For subplots without their own sourceaccessioncode (globally unique ID), prepend the parent location's unique ID so that locationName is globally unique

7009 12/21/2012 12:07 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: locationID/locationName + subplot -> location.sourceaccessioncode mapping: Fixed bug where subplot was incorrectly being mapped to this field even when there was no location*. (This field can only be populated if both location* and subplot are specified.) Also only map locationID for this, to avoid inconsistencies where one table supplies locationID+subplot, while another table supplies locationName+subplot, but they both get mapped to the same field, preventing plots from being matched up with their observations when creating the analytical_stem.

7008 12/21/2012 11:31 AM Aaron Marcuse-Kubitza

xml_func.py: Simplifying functions: Logic: _and(), _or(): Evaluate an expression of only constant values

7007 12/21/2012 11:30 AM Aaron Marcuse-Kubitza

lists.py: Added and_(), or_()

7006 12/21/2012 11:28 AM Aaron Marcuse-Kubitza

xml_func.py: is_scalar(): Fixed bug where need to check if value is a string before calling is_var_name()

7005 12/21/2012 10:15 AM Aaron Marcuse-Kubitza

inputs/NVS/StemObservation/map.csv: Remapped Verbatim Code to authorTaxonCode, because as it's used this is actually an identifier for the taxon, not the stem, despite Nick Spencer's revised mapping

7004 12/20/2012 05:21 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Regenerated exports

7003 12/20/2012 05:21 PM Aaron Marcuse-Kubitza

README.TXT: Schema changes: Update graphical ERD exports: Added step to commit changes

7002 12/20/2012 05:02 PM Aaron Marcuse-Kubitza

inputs/NVS/*/map.csv: Remapped with Nick Spencer's suggested changes

7001 12/20/2012 04:41 PM Aaron Marcuse-Kubitza

xml_func.py: _first(): Fixed bug where need to choose the first non-empty param, by first pruning empty child nodes

7000 12/20/2012 04:38 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode

6999 12/20/2012 04:03 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode

6998 12/20/2012 03:59 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_*: Renamed individualID to individualObservationID because this actually corresponds to plantobservation.sourceaccessioncode, which is an observation of an individual