Project

General

Profile

Statistics
| Revision:

# Date Author Comment
7182 01/11/2013 06:08 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed no longer used mappings for verbatimScientificName in _if conditions

7181 01/11/2013 06:08 AM Aaron Marcuse-Kubitza

inputs/.NCBI/nodes/test.xml.ref: Restored inserted row counts, which had gotten auto-accepted from a test run on a non-empty DB

7180 01/11/2013 06:06 AM Aaron Marcuse-Kubitza

sql_io.py: put_table(): DuplicateKeyException: Uniquifying input table to avoid internal duplicate keys: Also filter out duplicate rows in the out_table, so that they don't create duplicate key errors and the resulting index holes

7179 01/11/2013 06:01 AM Aaron Marcuse-Kubitza

sql.py: distinct_table(): Added support for custom joins used in creating the new table. This can then be used by sql_io.put_table() to filter out duplicate rows in the out_table, so that they don't create duplicate key errors and the resulting index holes.

7178 01/11/2013 05:53 AM Aaron Marcuse-Kubitza

README.TXT: Documentation: Redmine-formatted list of steps for column-based import: Added step to reinstall public schema first, to reset the sequences so that they don't create a diff when the new steps.by_col.log.sql is committed

7177 01/11/2013 05:48 AM Aaron Marcuse-Kubitza

Added inputs/ACAD/Specimen/logs/steps.by_col.log.sql

7176 01/11/2013 05:45 AM Aaron Marcuse-Kubitza

sql_gen.py: Join: Added support for mapping values which are lists, for use in USING joins

7175 01/11/2013 05:40 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/*/test.xml.ref: Restored SALVIAS* inserted row counts, which had gotten auto-accepted from a test run on a non-empty DB

7174 01/11/2013 05:01 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem: Added locationName (authorPlotCode), subplot, individualCode (authorPlantCode) for use in validation

7173 01/11/2013 04:57 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sync_analytical_stem_to_view(): Drop and re-create dependent objects to avoid errors that analytical_stem can't be dropped because of dependents

7172 01/11/2013 04:56 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sync_analytical_stem_to_view(): Changed to PL/pgSQL function to allow adding PL/pgSQL commands

7171 01/11/2013 03:26 AM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Moved family_higher_plant_group to leave room for analytical_stem to expand

7170 01/11/2013 03:08 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed no longer used mappings for verbatimScientificName in _if conditions

7169 01/11/2013 02:59 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed taxonlabel for original taxondetermination, because the original taxondetermination is not scrubbed by scrub.make (only the most current taxondetermination gets scrubbed, because only a single scrubbed determination is added by scrub.make). This still leaves the original taxondetermination's taxonverbatim, which stores the taxonomic information for historical purposes.

7168 01/11/2013 02:44 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed no longer used accepted and verbatim (parsed) taxonlabels, which have been replaced by a single accepted or matched taxondetermination created by scrub.make

7167 01/11/2013 02:34 AM Aaron Marcuse-Kubitza

Removed no longer used inputs/.TNRS/tnrs_accepted, tnrs_other. Use the tnrs_canon view instead.

7166 01/11/2013 02:22 AM Aaron Marcuse-Kubitza

Removed no longer used inputs/.TNRS/tnrs_accepted, tnrs_other. Use the tnrs_canon view instead.

7165 01/11/2013 02:18 AM Aaron Marcuse-Kubitza

Added inputs/.TNRS/_archive/

7164 01/11/2013 02:18 AM Aaron Marcuse-Kubitza

Added inputs/.TNRS/tnrs/cleanup.sql to prevent running the default cleanup operations, which don't work on tables which have views referencing them (as is the case for tnrs, which is referenced by tnrs_canon)

7163 01/11/2013 02:07 AM Aaron Marcuse-Kubitza

import_all: Removed no longer needed TNRS import, which has been replaced by scrub.make (which adds TNRS taxondeterminations after the import instead of creating taxonlabel links before it)

7162 01/11/2013 02:03 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed TNRS input taxonlabels meant to cross-link to taxonlabels added by the TNRS import, because TNRS taxondeterminations are now created instead

7161 01/11/2013 01:42 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Use just the main taxonlabel created by scrub.make instead of all the additional taxonlabels created by the TNRS import

7160 01/11/2013 01:11 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: main taxonverbatim.morphospecies "if has verbatim name" condition: Fixed bug where need to remove the taxonIsCanonical flag, because the TNRS.public.unscrubbed_taxondetermination_view table (which uses this flag) should include this field (although not other places where the morphospecies is stored by other TNRS tables)

7159 01/11/2013 12:49 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent() trigger: Also run on delete, to mark another taxondetermination as the current one when a current taxondetermination is deleted

7158 01/11/2013 12:18 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_canon: Annotations: Always use value from the matched name, because the accepted name does not have this

7157 01/11/2013 12:05 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: primary taxonlabel's parent taxonlabel: Fixed bug where a taxonverbatim was incorrectly being created solely to store the taxonRank, even though it was already stored in the taxonlabel's rank field

7156 01/10/2013 11:52 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Don't map morphospecies to the parsed taxonlabel's taxonepithet, because this causes an extra, parsed taxonlabel to be created for TNRS.public.unscrubbed_taxondetermination_view. It is not needed by the other TNRS tables.

7155 01/10/2013 11:45 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Omit Infraspecific_rank to help avoid creating a separate, parsed taxonlabel. Don't map to taxonRank because Name_matched_rank is populated more often.

7154 01/10/2013 11:34 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Reduced $maxPause to 4 hr, because new taxondeterminations are being added throughout the import, so it is unlikely that more than more than 4 hr would pass between successive imports of taxondeterminations (causing scrub.make to stop prematurely)

7153 01/10/2013 11:23 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Removed no longer used tnrs+accepted. Use tnrs_canon or a self-join of tnrs instead

7152 01/10/2013 11:22 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: Use TNRS.tnrs directly instead of the now-deprecated tnrs+accepted

7151 01/10/2013 11:12 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Use new TNRS.tnrs_canon instead of tnrs+accepted to avoid creating additional taxonlabels for the parsed, matched, and accepted names and instead just use the most-canonicalized name of the names output by TNRS (the accepted name if available, or the matched name otherwise)

7150 01/10/2013 10:50 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: "if has verbatim name" _if statements that filter something out for TNRS mappings: Also assume true if taxonIsCanonical is specified, because some TNRS tables (eventually such as public.unscrubbed_taxondetermination_view) do not specify a separate "verbatim" taxondetermination but do provide taxonIsCanonical as a flag to turn various mappings on and off

7149 01/10/2013 09:06 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Remapped matched*Fit_fraction to taxondetermination.taxonfit when a taxondetermination, not just a taxonlabel, is provided

7148 01/10/2013 09:03 PM Aaron Marcuse-Kubitza

bin/map: map_table(): Resolving prefixes: Fixed bug where need to use list instead of tuple for metadata value mappings

7147 01/10/2013 08:16 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxondetermination: Added CHECK constraint to allow only taxondeterminations with a minimum fit fraction of 80%, analogous to taxonlabel's taxonlabel_1_matched_label_min_fit() trigger

7146 01/09/2013 05:34 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Don't create a separate TNRS input taxonlabel if taxonIsCanonical exists

7145 01/09/2013 05:24 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_canon: Fixed bug where need to always use Unmatched_terms from tnrs rather than tnrs_accepted

7144 01/09/2013 05:07 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Added tnrs_canon, which stores the most canonicalized name output by TNRS

7143 01/09/2013 04:17 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: accepted_taxonverbatim: Fixed bug where need to join only to the taxonverbatim whose morphospecies is NULL, to avoid joining to multiple taxonverbatims at once. This extra filter is now needed because there can be multiple taxonverbatims for a taxonlabel with different morphospecies.

7142 01/09/2013 03:59 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonlabel.taxonomicname: Prepend the family to the rest of the name using new _merge_prefix() instead of _join_words()/_nullIf(), so that any input taxonomic name that includes the family will not have the family duplicated in the combined taxonomic name. Previously, the duplication was removed only when the rest of the input name was equal to the family. This change fixes a bug in the new TNRS import where a pre-concatenated taxonomic name (Accepted_scientific_name) which includes the family is now used instead of Accepted_name, which only includes it when it's equal to the family.

7141 01/09/2013 03:52 PM Aaron Marcuse-Kubitza

xml_func.py: Simplifying functions: Merging: Added _merge_prefix() passthru

7140 01/09/2013 03:33 PM Aaron Marcuse-Kubitza

schemas/functions.sql: Added _merge_prefix()

7139 01/09/2013 02:42 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_accepted_scientific_name(): Fixed bug where Accepted_name_family shouldn't be prefixed to Accepted_name if Accepted_name is itself the family, to avoid duplicating the family in the Accepted_scientific_name

7138 01/09/2013 02:18 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs+accepted: Added new Accepted_scientific_name column and mapped it in public.unscrubbed_taxondetermination_view

7137 01/09/2013 11:06 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: Fixed bug where need to filter out tnrs+accepted rows with NULL Accepted_scientific_name, because inputs to tnrs_db must be strings

7136 01/09/2013 10:53 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: Prepend TNRS accepted names that have not yet been parsed. This allows parsing TNRS accepted names without first needing to import them into taxonlabels, which may not occur until the next import.

7135 01/09/2013 10:09 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs+accepted: Use new Accepted_scientific_name to join to tnrs_accepted.Name_submitted

7134 01/09/2013 10:05 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: Added tnrs_populate_accepted_scientific_name() trigger

7133 01/09/2013 09:57 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: Added Accepted_scientific_name field which will contain the joined-together accepted name that gets re-parsed by TNRS

7132 01/09/2013 09:13 AM Aaron Marcuse-Kubitza

inputs/.TNRS/: Changed tnrs+accepted to a view (defined in schema.sql) so accepted names would automatically be populated as they are parsed by TNRS, rather than needing to run `make inputs/.TNRS/tnrs+accepted/reinstall` to populate them

7131 01/09/2013 08:16 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Also map the morphospecies to the accepted taxonverbatim when an accepted name is provided

7130 01/09/2013 08:01 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonverbatim: taxonverbatim_unique: Added morphospecies so that there can be multiple taxonverbatims for the same taxonlabel, each with different morphospecies suffixes

7129 01/09/2013 04:17 AM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Mapped Accepted_name.*

7128 01/09/2013 03:02 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Use new tnrs+accepted instead of tnrs so that the accepted name can be imported at the same time

7127 01/09/2013 02:23 AM Aaron Marcuse-Kubitza

import_all: Reinstall tnrs+accepted, for eventual use by unscrubbed_taxondetermination_view

7126 01/09/2013 02:20 AM Aaron Marcuse-Kubitza

Added inputs/.TNRS/tnrs+accepted/, which self-joins the TNRS results to their parsed accepted names

7125 01/09/2013 02:02 AM Aaron Marcuse-Kubitza

import_all: Directly import just the TNRS tables that should be imported, because some TNRS tables are included in import_order.txt so that they are part of the automated testing, but should not be imported at the same time as tnrs_accepted/tnrs_other

7124 01/09/2013 12:45 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

7123 01/08/2013 11:24 PM Aaron Marcuse-Kubitza

with_all: $all mode: Fixed bug where need " " before # for it to be interpreted as a comment (unlike in a Makefile, where the " " often needs to be left out to avoid it being treated as part of a variable value)

7122 01/08/2013 10:55 PM Aaron Marcuse-Kubitza

bin/map: Made $redo flag default to off, because redo mode is slow (all tables have to be truncated) and is only needed when running tests on a public schema with data in it, which would not be the case on a development machine where tests are usually run

7121 01/08/2013 10:19 PM Aaron Marcuse-Kubitza

import_all: Made temporary vars local, so they wouldn't affect the calling shell

7120 01/08/2013 09:45 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Sort by taxondetermination.taxonoccurrence_id instead of taxondetermination_id to allow scanning the taxondetermination_single_current_determination index, which includes only current determinations and avoids needing to scan past many non-current determinations. Note that using taxonoccurrence_id does not create sort order ambiguity between taxondeterminations with the same taxonoccurrence_id, because there is only one current determination per taxonoccurrence.

7119 01/08/2013 09:32 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Inner-join to taxonverbatim and taxonlabel instead of LEFT JOINing, because only taxondeterminations with a taxonlabel can have accepted taxondeterminations (otherwise there would be no name to scrub)

7118 01/08/2013 09:30 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Inner-join to tnrs instead of LEFT JOINing, because only taxondeterminations whose taxonlabels have already been scrubbed by TNRS should have accepted taxondeterminations added. Removed now-unneeded filter by tnrs.Name_submitted IS NOT NULL, which is replaced by the inner join.

7117 01/08/2013 08:46 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): ensure_cond(): Fixed bug where need to wrap strings used in the tracked error message in strings.ustr()

7116 01/08/2013 08:33 PM Aaron Marcuse-Kubitza

xml_dom.py: replace_with_text(): Fixed bug where need to use scalar.is_nonnull_scalar() instead of is_scalar() to avoid converting None values to the string 'None'

7115 01/08/2013 08:32 PM Aaron Marcuse-Kubitza

scalar.py: Added is_nonnull_scalar()

7114 01/07/2013 08:17 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Fixed bug where `make inputs/upload` needs to be run on local machine, not vegbiendev

7113 01/07/2013 08:16 PM Aaron Marcuse-Kubitza

sql.py: create_table(): Support creating a table like a view

7112 01/07/2013 08:04 PM Aaron Marcuse-Kubitza

sql.py: Added InvalidTypeException and parse it in parse_exception()

7111 01/07/2013 07:39 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7110 01/07/2013 07:34 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxondetermination_set_iscurrent(): Fixed bug where need to sort scrubbed determinations first for scrub.make to work. (Otherwise, a datasource determination might remain iscurrent even after a scrubbed determination was added, causing scrub.make to repeatedly attempt to re-add it.)

7109 01/07/2013 07:20 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set dateIdentified to _now()

7108 01/07/2013 07:20 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Unset $n to avoid limiting the # rows/iteration

7107 01/07/2013 07:15 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: parse_date_range(): Don't parse strings containing a time, because - and ' ' don't have the same meaning as in a date range

7106 01/07/2013 07:03 PM Aaron Marcuse-Kubitza

xml_dom.py: replace_with_text(): Support any scalar type recognized by scalar.is_scalar()

7105 01/07/2013 06:54 PM Aaron Marcuse-Kubitza

scalar.py: is_scalar(): Added datetime.datetime

7104 01/07/2013 06:43 PM Aaron Marcuse-Kubitza

schemas/functions.sql: Added _now()

7103 01/07/2013 06:39 PM Aaron Marcuse-Kubitza

import_all: Make $dump_opts, $public_import local vars, so they will be automatically unset if the script is aborted

7102 01/07/2013 06:31 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: identificationType: Fixed bug in mapping where extra *_id/ needed to be removed

7101 01/07/2013 06:25 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set taxonOccurrenceID to dummy value 0 to enable the taxonoccurrence CHECK constraint to pass. This is needed because the constraint must pass before the pkey (which should already exist) is even checked.

7100 01/07/2013 06:19 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set identificationType to computer

7099 01/07/2013 06:18 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped identificationType

7098 01/07/2013 06:15 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7097 01/07/2013 05:39 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Use `SELECT source_id FROM source WHERE shortname = ...` instead of source_by_shortname() so that the source table is updated to point to the same schema as the view rather than pointing to whichever version (usually public) is first in the search_path

7096 01/07/2013 05:23 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: unscrubbed_taxondetermination_view: Fixed bug where need to include only those taxondeterminations that already have a match in TNRS.tnrs, to avoid adding empty TNRS taxondeterminations. As the concurrent tnrs daemon runs, these taxondeterminations will gradually acquire matches in tnrs and then be processed by scrub.

7095 01/07/2013 05:00 PM Aaron Marcuse-Kubitza

import_all: Make $import_source a local var, so it will be automatically unset if the script is aborted

7094 01/07/2013 04:49 PM Aaron Marcuse-Kubitza

vegbien_dest: Schema override for referring to a table in the $public schema: Only process the override when $!schemaVar and $!tableVar are non-*empty*, to allow setting $schema=""

7093 01/07/2013 04:47 PM Aaron Marcuse-Kubitza

schemas/Makefile: DDL generation: vegbien.sql: Unset $dump_opts so that pg_dump does not use env vars left after running import_all

7092 01/07/2013 04:44 PM Aaron Marcuse-Kubitza

schemas/Makefile: DDL generation: vegbien.sql: Unset $version so that pg_dump always uses the public schema, even after running import_all

7091 01/07/2013 04:13 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7090 01/07/2013 04:13 PM Aaron Marcuse-Kubitza

README.TXT: Testing: Added commands to put in .profile on a development machine

7089 01/07/2013 04:10 PM Aaron Marcuse-Kubitza

import_all: Added command to add scrubbed taxondeterminations

7088 01/07/2013 04:09 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7087 01/07/2013 04:08 PM Aaron Marcuse-Kubitza

import_all: Start tnrs-remake after starting the inputs, so that for subset imports (e.g. n=2), there will already be names to scrub when tnrs-remake starts up and it won't enter pause mode to wait for new rows (the pause is calibrated for full imports, and is too long for subset imports)

7086 01/07/2013 04:01 PM Aaron Marcuse-Kubitza

with_all: Also exclude .archive/ from the subdirs to forward commands to

7085 01/07/2013 03:40 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Added option to wait for new rows, in the same way tnrs_db does

7084 01/07/2013 03:38 PM Aaron Marcuse-Kubitza

inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Factored new rows added test out into rowsAdded() function

7083 01/07/2013 03:09 PM Aaron Marcuse-Kubitza

Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make, which adds scrubbed taxondeterminations to VegBIEN