Project

General

Profile

Statistics
| Revision:
Name Size Revision Age Author Comment
  _archive 1598 almost 13 years Aaron Marcuse-Kubitza Moved _archive/tapir2flatClient/trunk/client/ t...
  analysis 3076 over 12 years Aaron Marcuse-Kubitza Added top-level analysis dir for range modeling
  backups 4751 over 12 years Aaron Marcuse-Kubitza backups/Makefile: Backups: Full DB: Specify the...
  bin 5503 over 12 years Aaron Marcuse-Kubitza import_all: Also import the NCBI tree of life, ...
  config 272 about 13 years Aaron Marcuse-Kubitza Moved bien_password to new config dir
  inputs 5515 over 12 years Aaron Marcuse-Kubitza inputs/.TNRS/tnrs/test.xml.ref: Updated inserte...
  lib 5520 over 12 years Aaron Marcuse-Kubitza sql.py: Added index_exprs() and use it in index...
  mappings 5513 over 12 years Aaron Marcuse-Kubitza mappings/VegCore-VegBIEN.csv: taxonconcept.pare...
  schemas 5518 over 12 years Aaron Marcuse-Kubitza schemas/vegbien.sql: taxonconcept: taxonconcept...
  to_do 4524 over 12 years Aaron Marcuse-Kubitza to_do/timeline.doc: Updated to reflect addition...
  validation 4523 over 12 years Aaron Marcuse-Kubitza Added validation/
Makefile 9.86 KB 5459 over 12 years Aaron Marcuse-Kubitza Makefile: Moved setting of $(root) before inclu...
README.TXT 12.9 KB 5519 over 12 years Aaron Marcuse-Kubitza README.TXT: Data import: On local machine: Adde...
map 989 Bytes 5158 over 12 years Aaron Marcuse-Kubitza root map: Removed no longer needed public schem...
new_terms.csv 30.4 KB 4887 over 12 years Aaron Marcuse-Kubitza Regenerated root unmapped_terms.csv, new_terms.csv
unmapped_terms.csv 5.8 KB 4887 over 12 years Aaron Marcuse-Kubitza Regenerated root unmapped_terms.csv, new_terms.csv

Latest revisions

# Date Author Comment
5520 10/15/2012 01:10 PM Aaron Marcuse-Kubitza

sql.py: Added index_exprs() and use it in index_cols()

5519 10/15/2012 01:08 PM Aaron Marcuse-Kubitza

README.TXT: Data import: On local machine: Added `make inputs/.TNRS/cleanup`, which is necessary because the PostgreSQL collation may differ between vegbiendev's and your DB

5518 10/15/2012 12:24 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_update_ancestors(): Use matched_concept_id's ancestors instead if available. (Recursively applied, this will use the ancestors of the accepted concept.) This facilitates finding all children of and matches to an accepted concept, which will all have an entry for that concept in taxonconcept_ancestor. Note that the concept's own parents will not be indexed in taxonconcept_ancestor, because only accepted ancestors are now stored in taxonconcept_ancestor. Documented that taxonconcept_ancestor now stores the accepted ancestors of a taxonconcept.

5517 10/15/2012 12:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonconcept: taxonconcept_2_propagate_accepted_concept_id(): Also update accepted_concept_id on concepts that resolve to this concept, which may have been created before this concept was marked as accepted if concepts are not imported in dependency order (accepted concepts first). Added index on matched_concept_id to speed up finding concepts that resolve to this concept.

5516 10/15/2012 12:10 PM Aaron Marcuse-Kubitza

sql.py: mk_select(): order_by is order_by_pkey: Only order by the table's actual pkey, if it has one, rather than using the first column if it doesn't

5515 10/15/2012 12:08 PM Aaron Marcuse-Kubitza

inputs/.TNRS/tnrs/test.xml.ref: Updated inserted row count

5514 10/15/2012 10:21 AM Aaron Marcuse-Kubitza

db_xml.py: partition_size: Increased to 1,000,000 (>= NCBI.higher_taxa's size) so NCBI.higher_taxa can be imported completely in one partition. This is necessary because NCBI's taxonconcepts are not in dependency order (parents first), so a later partition cannot rely on the parents of its taxonconcepts having already been imported. Instead, all taxonconcepts must be imported at once and then separately, the parents of all taxonconcepts must be set.

5513 10/15/2012 10:08 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: taxonconcept.parent_id when explicit parent provided: Set taxonconcept.parent_id using new _taxonconcept_set_parent_id() after creating the child taxonconcept, so that the parent_id will point to the already-inserted parent taxonconcept instead of creating a new, empty parent taxonconcept. This creates a two-step import, where first the taxonconcepts are imported, and then the parent_ids are matched up. This is necessary for column-based import because all the parent taxonconcepts are imported in a separate iteration from the child taxonconcepts with only their sourceaccessioncode, so this iteration must occur after the child taxonconcept iteration in order to match up with fully-populated taxonconcepts. Row-based import, on the other hand, does not require _taxonconcept_set_parent_id() but does require the taxonconcepts to be provided in dependency order (parents first), which is unfortunately not the case for NCBI.

5512 10/15/2012 09:57 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: *_update_ancestors(): Telling immediate children to update their ancestors lists: Exclude self to avoid infinite recursion

5511 10/15/2012 09:57 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: *_update_ancestors(): Telling immediate children to update their ancestors lists: Exclude self to avoid infinite recursion

View all revisions | View revisions

Also available in: Atom