Project

General

Profile

Statistics
| Revision:

# Date Author Comment
5352 10/09/2012 11:25 PM Aaron Marcuse-Kubitza

regexp.py: Added sub_recursive()

5351 10/09/2012 11:21 PM Aaron Marcuse-Kubitza

sql.py: simplify_expr(): Use new simplify_parens()

5350 10/09/2012 11:20 PM Aaron Marcuse-Kubitza

sql.py: Added simplify_parens()

5349 10/09/2012 11:14 PM Aaron Marcuse-Kubitza

sql.py: simplify_expr(): Use new regexp.sub_nested()

5348 10/09/2012 11:14 PM Aaron Marcuse-Kubitza

Added regexp.py

5347 10/09/2012 10:46 PM Aaron Marcuse-Kubitza

sql.py: simplify_expr(): Use new logic_op_re()

5346 10/09/2012 10:46 PM Aaron Marcuse-Kubitza

sql.py: Added logic_op_re()

5345 10/09/2012 10:40 PM Aaron Marcuse-Kubitza

sql.py: bool_re: Use new true_re, false_re

5344 10/09/2012 10:40 PM Aaron Marcuse-Kubitza

sql.py: Added true_re, false_re

5343 10/09/2012 10:37 PM Aaron Marcuse-Kubitza

sql.py: bool_re: Use new paren_re()

5342 10/09/2012 10:36 PM Aaron Marcuse-Kubitza

sql.py: bool_re: Use new paren_re()

5341 10/09/2012 10:36 PM Aaron Marcuse-Kubitza

sql.py: Added paren_re()

5340 10/09/2012 10:31 PM Aaron Marcuse-Kubitza

sql.py: simplify_expr(): Combined replacements of bool_re+' OR ' with the value in either order into one replacement

5339 10/09/2012 10:27 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: verbatim* taxonconcept: Don't store Name_submitted in taxonomicnamewithauthor in addition to identifyingtaxonomicname, because the fields other than identifyingtaxonomicname are meant to store parsed values rather than raw, unscrubbed values and TNRS does not directly provide a concatenated taxonomic name with author

5338 10/09/2012 10:23 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: verbatim* taxonconcept: Don't create hierarchy of parent taxonconcepts, because the parsed names (rather than the names for the matched taxonconcept) are from the input taxonomic name, rather than from the official tree of life used by TNRS. Otherwise, if a taxonomic name provides e.g. no family (common), a separate genus taxonconcept would have been created with no parent_id, which would not compare equal to the matched taxonconcept's genus with a parent_id. Continue to store the parsed family, genus, species in the family, genus, species cached fields, because the parsed family is often different from the matched taxonconcept's family when e.g. no family is provided in the taxonomic name.

5337 10/09/2012 10:16 PM Aaron Marcuse-Kubitza

sql.py: Renamed table_cols() to table_col_names() for clarity, because it does not return sql_gen.Col objects

5336 10/09/2012 10:12 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/test.xml.ref: Accepted new inserted row count. The change is most likely from several revisions back, but the cause of the change is unknown (it is not due to the updated TNRS.tnrs table, which is still sorted with the same rows first).

5335 10/09/2012 09:09 PM Aaron Marcuse-Kubitza

sql_gen.py: is_text_col(): Use new is_text_type()

5334 10/09/2012 09:09 PM Aaron Marcuse-Kubitza

sql_gen.py: Added is_text_type()

5333 10/09/2012 09:05 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Documented that NULL has no type, hence the NoUnderlyingTableException being re-raised

5332 10/09/2012 09:04 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Just store the column type in col_type, instead of storing typed_col and using typed_col.type, now that other info in typed_col is no longer needed

5331 10/09/2012 09:02 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Use is_nullable() instead of determining nullability itself, for clarity

5330 10/09/2012 08:59 PM Aaron Marcuse-Kubitza

sql_gen.py: is_nullable(): Fixed bug where non-columns could not be sent to db.col_info()

5329 10/09/2012 08:53 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Always remove_col_rename() the column to ensure that it is acceptable by helper functions like is_nullable()

5328 10/09/2012 08:11 PM Aaron Marcuse-Kubitza

lib/PostgreSQL-MySQL.csv: COMMENT statement: Fixed bug where ending ; could match only when preceded by ' and followed by a newline, to avoid matching ; embedded in the comment

5327 10/09/2012 08:07 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: family, genus, species comments: Changed "scoping" to "identifying" for clarity

5326 10/09/2012 08:06 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: family, genus, species: Added comment that each is a cached field for easy querying and the scoping version of it is stored in the chain of parent_id ancestors

5325 10/09/2012 08:03 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_unique: Removed family, genus, species because these are now just cached fields for analytical_db_view rather than scoping fields. The scoping versions of these fields are stored in the chain of parent_id ancestors.

5324 10/09/2012 07:42 PM Aaron Marcuse-Kubitza

tnrs_db: Moved "Processing # taxonconcepts" log message to before waiting or exiting if no taxonconcepts left, so that it would be printed right after the query is run and say that no taxonconcepts were found

5323 10/09/2012 07:39 PM Aaron Marcuse-Kubitza

tnrs_db: Updated comments and log messages for schema changes

5322 10/09/2012 07:33 PM Aaron Marcuse-Kubitza

tnrs_db: Updated query for schema changes

5321 10/09/2012 07:33 PM Aaron Marcuse-Kubitza

README.TXT: Schema changes: files to update with renamings: Added bin/tnrs_db

5320 10/09/2012 07:25 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

5319 10/09/2012 07:04 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Changed `inputs/*/*/logs` to `inputs/{.,}*/*/logs` to also include the TNRS names import log

5318 10/08/2012 09:58 PM Aaron Marcuse-Kubitza

import_all: Added commands to import TNRS names so the user doesn't have to do this manually

5317 10/08/2012 09:55 PM Aaron Marcuse-Kubitza

sql.py: map_expr(): Fixed bug where names were being matched inside punctuated names replaced in previous calls of map_expr()

5316 10/08/2012 09:45 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: party: party_required_key: Only allow NULL organizationname if party is not a root party (i.e. creator_id != party_id)

5315 10/08/2012 09:39 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped to new taxonconcept.creationdate

5314 10/08/2012 09:37 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Added creationdate as an allowable minimum field when parent_id (containing the associated hierarchical concept) is specified

5313 10/08/2012 09:30 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Removed family and genus because these are now cached fields only, and are not used for scoping a taxonconcept. Instead, *taxonomicname and taxonname+parent_id are used for this purpose. This removes several leaf taxonconcepts with insufficient scoping information to create a taxonconcept separate from the main tree. With the upcoming population of creationdate, some of these taxonconcepts will reappear due to the date's additional distinguishing information.

5312 10/08/2012 09:16 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Added creationdate (the date the taxonconcept was created or defined), and include it in the taxonconcept_unique unique index

5311 10/08/2012 09:05 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Added comment with the definition of a taxon: "a group of one (or more) populations of organism(s), which a taxonomist adjudges to be a unit" (http://en.wikipedia.org/wiki/Taxon). This is useful in clarifying that our taxon concepts are intended to serve a similar purpose, by storing one person's defined taxon.

5310 10/08/2012 08:58 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Removed family and genus because these are now cached fields only, and are not used for scoping a taxonconcept. Instead, *taxonomicname and taxonname+parent_id are used for this purpose.

5309 10/08/2012 08:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Moved identifyingtaxonomicname near other full-taxonomic-name-related fields, after the fields that contain just the current level's component of the full name

5308 10/08/2012 08:48 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id: Changed four-level hierarchy to use "parsed concept" and "matched concept" instead of concatenated and parsed, because the directly-parsed name components actually go in level 2 of the hierarchy (the TNRS input name), while the name components based on the matched taxon concept go in level 3

5307 10/08/2012 08:44 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept.parent_id: Documented that while a taxon name may have multiple parents, a taxon concept has only one, based on the creator's opinion of where that taxonconcept goes in the taxonomic hierarchy

5306 10/08/2012 08:38 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonconcept: Moved infraspecific taxonconcept to its own level, rather than combining it with the level that contains the full taxonomic name and author (as well as any morphospecies), for consistency with the storage of other ranked taxonomic name components, which each get their own taxonconcept. The infraspecific taxon concept is general to all parties making idenfitications (within a datasource), while the concatenated name and author and any morphospecies are specific to the person who defined the taxonconcept used by a taxondetermination.

5305 10/08/2012 08:05 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Removed no longer used higher- and infraspecific taxonomic rank fields because these terms are now stored in their own taxonconcepts. family, genus, and species have not been removed because these are used to cache names of parent taxa for fast access by analytical_db_view.

5304 10/08/2012 07:57 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_db_view: Changed taxonMorphospecies to use taxonconcept.taxonname, where any morphospecies is now stored

5303 10/08/2012 07:53 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: infraspecific taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts, or in the lowest-level taxonconcept as the taxonname and rank

5302 10/08/2012 07:43 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: higher-level taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts

5301 10/08/2012 07:41 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Merged taxonconcept_unique_within_creator_by_name unique index into taxonconcept_unique_within_parent, placed parent_id first, and removed index condition, so that this index can be used as a lookup index by taxonconcept_update_ancestors() (which requires no index condition in order to apply to all taxonconcepts) in addition to as a unique index. Note that an index condition should not be necessary for the index's uniquifying task, because if a set of taxonconcepts provides only the identifyingtaxonomicname, that should collide in the taxonconcept_unique_within_creator_by_identifying_name unique index before this index collides. This assumes that the collision order when multiple indexes collide is alphabetical by the index name.

5300 10/08/2012 06:38 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Also allow a taxonconcept to have just an author when it has a parent_id, so that an author can uniquely identify a taxon within a more general taxon, such as a species name, that has no author

5299 10/08/2012 06:22 PM Aaron Marcuse-Kubitza

strings.py: concat(): Fixed bug where end index of returned str0 portion would wrap around to a negative number if str1 itself was too long, causing incorrect truncation

5298 10/08/2012 05:44 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_parent to taxonconcept_unique because the index does not apply only to taxonconcepts with a parent, and because it's the primary unique index for taxonconcept

5297 10/08/2012 05:42 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_creator_by_identifying_name to taxonconcept_0_unique_identifying_name to ensure that it is always applied before taxonconcept_unique_within_parent if both collide

5296 10/08/2012 05:36 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Merged taxonconcept_unique_within_creator_by_name unique index into taxonconcept_unique_within_parent, placed parent_id first, and removed index condition, so that this index can be used as a lookup index by taxonconcept_update_ancestors() (which requires no index condition in order to apply to all taxonconcepts) in addition to as a unique index. Note that an index condition should not be necessary for the index's uniquifying task, because if a set of taxonconcepts provides only the identifyingtaxonomicname, that should collide in the taxonconcept_unique_within_creator_by_identifying_name unique index before this index collides. This assumes that the collision order when multiple indexes collide is alphabetical by the index name.

5295 10/08/2012 04:47 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree for taxonconcepts created from original*, verbatim*, and accepted* taxonomic terms

5294 10/08/2012 04:35 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree if datasource provided separated components of the taxonomic name and/or its own tree of life with higher classifications. This enables storing the datasource's own tree of life to supplement any official tree (TROPICOS, USDA, etc.).

5293 10/08/2012 04:25 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonconcept tree: Don't map infraspecificEpithet+taxonRank to a taxonconcept in the tree of parent concepts because it has already been mapped to the primary, lowest-level taxonconcept

5292 10/08/2012 04:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Fixed bug where index filter overlapped with taxonconcept_unique_within_parent's index filter, causing these unique indexes to sometimes both apply at the same time and prevent column-based import from correctly choosing which index to use for each taxonconcept import

5291 10/08/2012 01:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Fixed lines

5290 10/08/2012 01:02 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id comment: Changed comment to use "concept" rather than "name" where applicable. Documented that a synonym between taxonconcepts of different sources is indicated by choosing one taxonconcept to be authoritative and pointing the other taxonconcept to it using this field.

5289 10/05/2012 10:52 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): Resolving default value column: Fixed bug where the default value col needed to have its table changed from in_table to full_in_table if it's a table column, and needed to have any column rename removed if it's a literal value

5288 10/05/2012 10:29 PM Aaron Marcuse-Kubitza

Regenerated vegbien.ERD exports

5287 10/05/2012 10:28 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Fixed lines

5286 10/05/2012 10:23 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Renamed plant* taxonomic tables -> taxon*, as part of the taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>

5285 10/05/2012 10:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Rearranged to fit more of location table on the diagram, using the newly available space from taxon

5284 10/05/2012 10:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Fixed lines

5283 10/05/2012 09:59 PM Aaron Marcuse-Kubitza

schemas/tree_cross-links.sql: Synced with schema, updating with new table names

5282 10/05/2012 09:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Removed no longer used taxon table. Use taxonconcept instead.

5281 10/05/2012 09:51 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept.taxonname: comment: Stated that this is the name of the taxon within its parent taxon

5280 10/05/2012 09:48 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: comment: Removed no longer accurate comment that an accepted taxonconcept points to the identified taxon in the tree of life, because it is the identified taxon in the tree of life

5279 10/05/2012 09:39 PM Aaron Marcuse-Kubitza

schemas/filter_ERD.csv: Changed the table with the visible fkey from plant* to taxon* to be plantstatus rather than plantusage, since it contains more core fields

5278 10/05/2012 09:25 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Removed taxon_id, since taxonconcept now contains all the information needed to represent a taxonomic hierarchy, including both conceptual and nomenclature information

5277 10/05/2012 09:20 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: plantusage: Point just to taxonconcept instead of both to taxonconcept and taxon

5276 10/05/2012 09:16 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: rank, verbatimrank comments: Added info from corresponding fields in taxon that also applies to taxonconcept

5275 10/05/2012 09:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: comment: Added info from taxon that also applies to taxonconcept

5274 10/05/2012 09:06 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added taxonconcept_ancestor cross-link table

5273 10/05/2012 08:40 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Added description field

5272 10/05/2012 08:38 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Remapped taxon hierarchy for accepted taxonconcepts to taxonconcept parent_id hierarchy

5271 10/05/2012 08:12 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Fixed bug where taxonconcept.parent_id was missing a foreign key constraint

5270 10/05/2012 08:10 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Changed instructions for including a taxon name at a rank with no explicit column to create a parent taxonconcept for it and point to it using parent_id instead of using otherranks. Removed no longer used otherranks field.

5269 10/05/2012 08:05 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Added taxonname

5268 10/05/2012 07:58 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Removed duplicate entry for creator_id

5267 10/05/2012 07:57 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Added parent_id to point to the parent taxonconcept

5266 10/05/2012 07:56 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Added 'unknown' for taxonrank

5265 10/05/2012 07:44 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonrank: Added 'unknown'

5264 10/05/2012 07:30 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Also map *taxonRank to taxonconcept.rank, so that if it's in the taxonrank enum, it will automatically populate this field

5263 10/05/2012 07:14 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Remapped *infraspecificEpithet to new taxonconcept.taxonname rather than placing it in subspecies prefixed with the taxonRank, because it isn't necessarily the subspecies and because taxonname is defined to contain the lowest-rank portion of the taxonomic name. Note that when both morphospecies and infraspecificEpithet are provided, infraspecificEpithet takes priority for the taxonname field, because if TNRS leaves unmatched terms (which are tentatively mapped to morphospecies) but also matches an infraspecificEpithet, then the unmatched terms can't be for a morphospecies (because an infraspecificEpithet and therefore also a specificEpithet was matched, so the species is definite and formally named).

5262 10/05/2012 06:45 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: Renamed morphospecies to taxonname since it's used in the same way as taxon.taxonname: to store the lowest-rank portion of the taxonomic name, such as the morphospecies suffix

5261 10/05/2012 06:21 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/map.csv: Mapped *_matched terms that are both matched in the input name and which correspond to the matched taxonconcept (Genus_matched, Specific_epithet_matched, etc.) to both the input and matched taxonconcepts

5260 10/05/2012 06:09 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/map.csv: Mapped terms matched in the original string (rather than deduced from the matched taxonconcept) to new verbatim* taxonomic terms

5259 10/05/2012 06:03 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped verbatim* taxonomic terms to the TNRS input taxonconcept

5258 10/05/2012 05:48 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: TNRS input taxonconcept: Split single _if statement controlling where morphospecies goes into two _if statements for each case, so that other verbatim* terms don't need to have an _if statement in their mapping to the input taxonconcept

5257 10/05/2012 05:29 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added back verbatim* taxonomic terms, which will now be used for the TNRS input taxonconcept. Note that they will have a different meaning than the original* taxonomic terms that they were renamed to in r5062.

5256 10/05/2012 05:22 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: In TNRS mode, remapped morphospecies (Unmatched_terms) to the input name's taxonconcept, because this does not relate to the matched taxon concept

5255 10/05/2012 05:12 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: TNRS-only mappings: Switch them on when verbatimScientificNameWithAuthorship is provided rather than when acceptedScientificNameWithAuthorship is provided, because it's the presence of a separate TNRS input name that really determines when TNRS is being mapped

5254 10/05/2012 05:07 PM Aaron Marcuse-Kubitza

Makefiles: .last_cleanup targets: Also make the file that's being cleaned up .PRECIOUS so it doesn't get deleted if the .last_cleanup target has an error

5253 10/05/2012 05:04 PM Aaron Marcuse-Kubitza

Makefiles: .last_cleanup targets: Make each individual target .PRECIOUS (don't delete on error) because just making %.last_cleanup precious doesn't seem to prevent deletion