Activity
From 09/28/2012 to 10/27/2012
10/26/2012
- 08:18 PM Revision 5823: sql.py: parse_exception(): "could not create unique index" DuplicateKeyException: Fixed bug where can't use make_DuplicateKeyException() because it tries to retrieve information about the index in question, but the index it was trying to create doesn't exist
- 08:10 PM Revision 5822: schemas/vegbien.sql: analytical_db_view: Renamed datasource's taxonverbatim to datasource_taxonverbatim to distinguish it from the other taxonverbatims that are joined on (parsed_taxonverbatim, accepted_taxonverbatim)
- 07:18 PM Revision 5821: inputs/.NCBI/nodes/create.sql: Make genus (mostly) globally unique by removing kingdom Animalia, which has significant genus overlap with plants. This reduces the number of duplicated genera from 578 to 65 (determined with `SELECT name_txt, count(*), array_agg(rank) FROM "NCBI".nodes GROUP BY name_txt HAVING count(*) > 1 AND 'genus' = ALL (array_agg(rank))`).
- 07:08 PM Revision 5820: inputs/.NCBI/nodes/create.sql: Added foreign key on parent tax_id with covering index
- 07:06 PM Revision 5819: input.Makefile: Staging tables installation: Added %/uninstall, %/reinstall to allow reinstalling individual tables
- 06:00 PM Revision 5818: sql_io.py: put_table(): ensure_cond(): When adding the failed condition to the errors table, also include the original, untranslated condition from the DB schema in addition to the translation of the condition into the input schema
- 05:45 PM Revision 5817: sql_io.py: track_data_error(): Fixed bug where errors whose column had no srcs (indicated by () ) were incorrectly being ignored. This affected NOT NULL exceptions where the column was not provided by the dataset.
- 05:38 PM Revision 5816: sql_gen.py: If no cols had srcs, return [] instead of the [()] that itertools.product() would have returned
- 05:38 PM Revision 5815: sql_io.py: track_data_error(): Support errors with no columns by inserting a single entry with column set to NULL
- 05:35 PM Revision 5814: strings.py: Added join()
- 05:00 PM Revision 5813: sql_io.py: mk_errors_table(): Made "column" column nullable, because some errors (such as check constraint violations) don't have any corresponding columns if its columns weren't provided in the input data
- 04:35 PM Revision 5812: inputs/test_taxonomic_names/test_scrub: `make inputs/.TNRS/reinstall`: Use new $schema_only option so that an empty TNRS schema is installed rather than one containing inputs/.TNRS/data.sql
- 04:34 PM Revision 5811: inputs/.TNRS/: Added data.sql containing the test_taxonomic_names TNRS results, so that a new installation of VegBIEN will contain the necessary data to make the tests pass, including the TNRS import test
- 04:32 PM Revision 5810: input.Makefile: Staging tables installation: If $schema_only option is set, only install .sql files ending in schema.sql
- 04:24 PM Revision 5809: inputs/Makefile: $(rsyncLogs): Use $(rsync) instead of $(rsync*) now that it supports excluding just temp files and .svn rather than all .*
- 04:21 PM Revision 5808: lib/common.Makefile: rsync: $(rsync): Exclude .svn, *#, and .DS_Store rather than all .* because dirs beginning with . created by the user (such as .NCBI, .TNRS) should be included in the sync
- 04:18 PM Revision 5807: Added inputs/REMIB/Specimen.src/.map.csv.last_cleanup
- 04:10 PM Revision 5806: Added inputs/bien_web/observation/+header.csv
- 04:09 PM Revision 5805: input.Makefile: Staging tables installation: $(dbExports): When putting schemas first, don't require a . before "schema" to allow the entire filename to be schema.sql
- 03:44 PM Revision 5804: inputs/test_taxonomic_names/_scrub/public.test_taxonomic_names.sql, TNRS.sql: Regenerated with schema and mappings changes
- 03:42 PM Revision 5803: inputs/.TNRS/tnrs/map.csv: Added _nullIf filter to remove "Unknown" values for Accepted_name_family
- 03:35 PM Revision 5802: README.TXT: Generate the local TNRS cache from the test_taxonomic_names rather than syncing it with the vegbiendev TNRS cache, so that the automated test's inserted row count stays the same regardless of the contents of the full-DB TNRS cache
- 03:34 PM Revision 5801: README.TXT: Backups: Added TNRS cache section
- 03:12 PM Revision 5800: inputs/.TNRS/tnrs/test.xml.ref: Accepted inserted row count using TNRS cache created from test_taxonomic_names. Using a standard set of names for the test ensures that the inserted row count will not change when the full-DB TNRS cache changes.
- 02:48 PM Revision 5799: inputs/.TNRS/schema.sql: tnrs_accepted_names: Prepend the Accepted_name_family to the taxonomic name that will be submitted back to TNRS for parsing, because TNRS input names now always include the family when it's provided
- 02:46 PM Revision 5798: inputs/.TNRS/schema.sql: tnrs_accepted_names: Use simpler array_to_string() instead of || and COALESCE() to put together the taxonomic name that will be submitted back to TNRS for parsing. Note that this requires defining an IMMUTABLE wrapper function for array_to_string(), because pg_catalog.array_to_string() is declared STABLE but indexes require functions to be IMMUTABLE (http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg156323.html).
- 02:42 PM Revision 5797: inputs/.TNRS/schema.sql: Don't hardcode the schema name
- 02:40 PM Revision 5796: input.Makefile: Staging tables installation: sql/install: Provide the datasource's schema to the script in :schema, so it can refer to its own elements explicitly when it's not possible to rely on the search_path. This is the case for functions that have the same signature as (and are intended to replace) a pg_catalog function, because the pg_catalog function will be used in preference to the datasource function regardless of the search_path.
- 02:06 PM Revision 5795: input.Makefile: Staging tables installation: $(cleanup): If a cleanup.sql is provided, *only* run it and don't do default cleanup, to allow tables to override rather than just add to default cleanup operations. This prevents the automatic replacement of certain strings (sql_io.null_strs) with NULL on TNRS, and keeps the TNRS cache mostly as it was output by the TNRS service. Note that empty strings are still replaced with NULL by COPY FROM in sql_io.append_csv(). This is necessary for TNRS import to work properly, because although '' generally means NULL, it is not treated that way by PostgreSQL.
- 01:53 PM Revision 5794: input.Makefile: Staging tables installation: Moved custom cleanup.sql cleanup operations to main $(cleanup) function, so custom cleanup operations would run whenever any target (such as %/install) invokes $(cleanup), not just manually through %/cleanup
- 01:52 PM Revision 5793: input.Makefile: Staging tables installation: Moved custom cleanup.sql cleanup operations to main $(cleanup) function, so custom cleanup operations would run whenever any target (such as %/install) invokes $(cleanup), not just manually through %/cleanup
10/25/2012
- 05:16 PM Revision 5792: sql.py: parse_exception(): function MissingCastException: If first param's type is anyelement (for polymorphic function, which had mismatched arg types), use type text, as all types can cast to it
- 05:12 PM Revision 5791: sql_io.py: cast(): Set the created function's value param type to anyelement to support any input type, not just text
- 04:53 PM Revision 5790: mappings/VegCore-VegBIEN.csv: Only prepend the family to the concatenated scientificName for TNRS if it ends in -aceae (using _taxon_family_require_std()), to avoid sending unsupported, nonstandard families to TNRS which it will place in Unmatched_terms
- 04:48 PM Revision 5789: schemas/vegbien.sql: Added _taxon_family_require_std()
- 04:09 PM Revision 5788: mappings/VegCore-VegBIEN.csv: Prepend the family to the concatenated scientificName input to TNRS, so that TNRS can use it to disambiguate the genus
- 03:50 PM Revision 5787: tnrs_db: Making TNRS request: Fixed bug where needed to remove else block now that there is no except block
- 03:45 PM Revision 5786: tnrs.py: retrieval_request_template: Turn on taxonomic_constraint (to match family before genus) and source_sorting (to always return any result from the first source before returning results from any other sources, regardless of match %)
- 12:50 PM Revision 5785: Regenerated vegbien.ERD exports
- 12:46 PM Revision 5784: mappings/VegCore.csv: speciesBinomial: Changed definition to genus+specificEpithet, not genus+species, to match the scientific meaning of specificEpithet vs. species
- 12:45 PM Revision 5783: schemas/vegbien.sql: taxonverbatim: Renamed species to specific_epithet to avoid confusion with the scientific meaning of species (genus+specificEpithet), since this field contains just the specific epithet
- 12:30 PM Revision 5782: input.Makefile: Verification of import: verify: Use tables from the verify/*.ref files themselves rather than from the datasource's subdirs, in order to match the tables in mappings/verify.*.sql
- 11:52 AM Revision 5781: schemas/vegbien.sql: analytical_db_view: Added stemobservation.tag, stemobservation.height_m for use in plot change over time analysis <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Plot_change_over_time_analysis>
- 11:45 AM Revision 5780: schemas/vegbien.sql: analytical_db_view: Fixed typo in scientificNameWithMorphospecies
- 11:41 AM Revision 5779: schemas/vegbien.sql: analytical_db_view: Renamed columns to VegCore names (https://projects.nceas.ucsb.edu/nceas/projects/bien/repository/raw/mappings/VegCore.csv)
- 11:36 AM Revision 5778: mappings/VegCore.csv: Added cultivatedBasis
- 11:34 AM Revision 5777: mappings/VegCore.csv: Added scientificNameWithMorphospecies
- 10:55 AM Revision 5776: mappings/VegCore.csv: Added speciesBinomial
- 10:49 AM Revision 5775: schemas/vegbien.sql: analytical_db_view: Generate species by concatenating genus and specific epithet, since according to Brad this field is actually the binomial, not the specificEpithet
- 10:41 AM Revision 5774: schemas/vegbien.sql: Removed no longer used plot_change_over_time view. Use one of the queries at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Plot_change_over_time_analysis> instead.
- 10:36 AM Revision 5773: mappings/VegCore-VegBIEN.csv: location: Populate sourceaccessioncode with locationID + subplot when subplot is unique only within the parent plot, so that location always has a sourceaccessioncode to use as the plotCode in analytical_db_view
- 10:07 AM Revision 5772: lib/PostgreSQL-MySQL.csv: Remove views because they can contain arbitrary expressions, whose syntax may not be compatible with MySQL
- 10:04 AM Revision 5771: schemas/vegbien.sql: analytical_db_view: Use location.sourceaccessioncode as plotCode instead of authorlocationcode because authorlocationcode isn't globally unique (for subplots, it's only unique within the parent plot)
- 09:48 AM Revision 5770: schemas/vegbien.sql: plantobservation: Made taxonoccurrence_id optional when sourceaccessioncode is specified, so that aggregateoccurrence doesn't get pruned away in datasource tables that link just a stemobservation to a plantobservation (and therefore don't provide a taxonoccurrence to satisfy the previous taxonoccurrence_id NOT NULL constraint)
- 09:47 AM Revision 5769: schemas/vegbien.sql: aggregateoccurrence: Made taxonoccurrence_id optional when sourceaccessioncode is specified, so that aggregateoccurrence doesn't get pruned away in datasource tables that link just a stemobservation to a plantobservation (and therefore don't provide a taxonoccurrence to satisfy the previous taxonoccurrence_id NOT NULL constraint)
- 09:42 AM Revision 5768: schemas/vegbien.sql: aggregateoccurrence: Made taxonoccurrence_id optional when sourceaccessioncode is specified, so that aggregateoccurrence doesn't get pruned away in datasource tables that link just a stemobservation to a plantobservation (and therefore don't provide a taxonoccurrence to satisfy the previous taxonoccurrence_id NOT NULL constraint)
- 09:31 AM Revision 5767: schemas/vegbien.sql: taxonoccurrence: Added taxonoccurrence_required_key check constraint to ensure that all taxonoccurrences are properly identified, and empty taxonoccurrences are properly pruned. This fixes a bug where taxon-only and stem-only data did not properly prune the taxonoccurrence that would otherwise get created because it's included in the mappings.
- 07:51 AM Revision 5766: sql_io.py: put_table(): insert_into_pkeys(): Use new sql.add_pkey_or_index() instead of sql.add_pkey() in order to just print a warning if for some reason there were duplicate entries for an input row in the iteration's pkeys table. This should provide a workaround for bugs (often in the schema itself, related to its unique indexes) that cause an input row to match multiple output rows when joining on the output table using the unique constraint's columns.
- 07:44 AM Revision 5765: sql.py: Added add_pkey_or_index()
- 07:32 AM Revision 5764: sql.py: parse_exception(): Parse "could not create unique index ... Key is duplicated" errors as DuplicateKeyException
- 07:27 AM Revision 5763: sql.py: parse_exception(): DuplicateKeyException: Factored out creation of DuplicateKeyException into helper function
- 07:20 AM Revision 5762: inputs/import.stats.xls: Updated import times
10/24/2012
- 06:31 PM Revision 5761: tnrs_db: Removed tnrs.InvalidResponse exception handler that retries the query because the current query does not track which names have been submitted to but not processed by TNRS, so the error would continue to happen repeatedly
- 06:13 PM Revision 5760: schemas/vegbien.sql: location: Added index on parent_id to speed up plot change over time joins
- 05:45 PM Revision 5759: schemas/vegbien.sql: location: Added index on creator_id to speed up analytical_db_view joins
- 05:15 PM Revision 5758: schemas/vegbien.sql: stemobservation: Added index on plantobservation_id to speed up analytical_db_view joins
10/23/2012
- 01:08 PM Revision 5757: schemas/vegbien.sql: Added initial plot_change_over_time view
- 12:53 PM Revision 5756: Added inputs/bien_web/
- 12:43 PM Revision 5755: schemas/vegbien.sql: analytical_db_view: Reordered taxonoccurrence.growthform to put if after the bien_web.observation fields
- 12:32 PM Revision 5754: schemas/vegbien.sql: analytical_db_view: Include taxonoccurrence.growthform
- 12:27 PM Revision 5753: schemas/vegbien.sql: analytical_db_view: Generate taxonMorphospecies by concatenating the scientificName to the morphospecies
- 12:23 PM Revision 5752: schemas/vegbien.sql: analytical_db_view: Fixed bug where needed to take taxonomic name components from the accepted taxonlabel's taxonverbatim instead of the datasource's taxonverbatim, which does not contain the accepted name
- 12:19 PM Revision 5751: schemas/vegbien.sql: analytical_db_view: identifiedBy: Added NULLIF() to keep empty strings out of the analytical DB
- 12:13 PM Revision 5750: schemas/vegbien.sql: analytical_db_view: Fixed bug where needed to take taxonomic name components from the accepted taxonlabel's taxonverbatim instead of the datasource's taxonverbatim, which does not contain the accepted name
- 12:10 PM Revision 5749: schemas/vegbien.sql: analytical_db_view: Fixed bug where needed to take morphospecies from the parsed taxonlabel's taxonverbatim, where it has been parsed out, instead of the datasource's taxonverbatim, which has it as part of the verbatim input name
- 11:58 AM Revision 5748: analytical_db_view: Added stemobservation.xposition_m, yposition_m
- 11:46 AM Revision 5747: inputs/.TNRS/tnrs/map.csv: Added new Time_submitted field
- 11:45 AM Revision 5746: inputs/REMIB/Specimen/header.csv: Regenerated for new staging tables format
- 11:41 AM Revision 5745: inputs/.TNRS/tnrs/test.xml.ref: Accepted correct inserted row count, which most likely became detached from the primary row count when the TNRS cache was cleared and repopulated with test data
- 11:28 AM Revision 5744: schemas/vegbien.sql: analytical_db_view: Reordered joins in path order, putting datasource before location. This will enable more naturally reusing the SELECT query for other analyses.
- 11:15 AM Revision 5743: mappings/VegCore-VegBIEN.csv: TNRS<->NCBI attachment: Do not include rank in the mapping because taxonomicname is globally unique, and thus it isn't used in looking up the NCBI taxonlabel
- 11:05 AM Revision 5742: inputs/test_taxonomic_names/_scrub/public.test_taxonomic_names.sql, TNRS.sql: Regenerated with schema and mappings changes
- 10:49 AM Revision 5741: mappings/VegCore-VegBIEN.csv: TNRS<->NCBI attachment: Also attach TNRS genus to NCBI backbone. This causes attachment to be made with as many of family and genus as are provided and have an entry in NCBI.
- 10:45 AM Revision 5740: mappings/VegCore-VegBIEN.csv: family -> NCBI backbone: Removed extra path after _if statement's cond/_exists
- 10:39 AM Revision 5739: mappings/VegCore-VegBIEN.csv: Instead of connecting the acceptedFamily to the NCBI backbone, connect the family for the TNRS matched taxonlabel. This connects more families and also connects the same set of fields as will be connected for the genus.
- 10:01 AM Revision 5738: mappings/VegCore-VegBIEN.csv: TNRS<->NCBI attachment: Fixed bug where needed to attach accepted family to NCBI using taxonomicname, which is globally unique, rather than taxonepithet, which is only unique within the parent taxon
- 09:34 AM Revision 5737: inputs/.TNRS/tnrs/: Added Time_submitted column at beginning and populate it in tnrs_db with the time the batch TNRS request was submitted
- 09:08 AM Revision 5736: csvs.py: RowNumFilter: Use new ColInsertFilter
- 09:08 AM Revision 5735: csvs.py: Added ColInsertFilter
- 08:43 AM Revision 5734: schemas/vegbien.sql: Removed no longer used _is_higher_taxon(). Use _has_taxonomic_name() or _taxonomic_name_is_epithet() instead.
- 08:42 AM Revision 5733: mappings/VegCore-VegBIEN.csv: taxonName->taxonepithet: Use new _taxonomic_name_is_epithet() instead of _is_higher_taxon(), because it's more specific to the filtering task for this field
- 08:36 AM Revision 5732: schemas/vegbien.sql: Added _taxonomic_name_is_epithet()
- 08:33 AM Revision 5731: mappings/VegCore-VegBIEN.csv: taxonName->taxonomicname: Use new _has_taxonomic_name() instead of _is_higher_taxon(), because it's more specific to the filtering task for this field
- 08:30 AM Revision 5730: mappings/VegCore-VegBIEN.csv: taxonName->taxonomicname: Use new _has_taxonomic_name() instead of _is_higher_taxon(), because it's more specific to the filtering task for this field
- 08:25 AM Revision 5729: schemas/vegbien.sql: Added _has_taxonomic_name() for lower taxon ranks that typically have a globally unique taxonomic name
- 08:10 AM Revision 5728: schemas/functions.sql: Removed unit conversion functions that take a text input, since casts to the parameter type (double precision) are now automatically performed by sql_io.put_table(), using sql.parse_exception()'s function MissingCastException parsing
- 08:01 AM Revision 5727: mappings/VegCore-VegBIEN.csv: _is_higher_taxon() calls: Default to true if the rank can't be parsed to a taxonrank enum value
- 07:56 AM Revision 5726: sql_io.py: put_table(): is_function: Moved definition of wrapper function inside try block of main loop because the creation of the empty pkeys table (whose row type is needed for the wrapper function) can itself produce MissingCastExceptions, which must be thrown inside the loop in order to be handled properly
- 07:05 AM Revision 5725: db_xml.py: put(): Indicate no parent_ids_loc using no_parent_ids_loc sentinel instead of None to support parent_ids_locs that are equal to None (e.g. if the parent node had an error). Always forward parent_ids_loc to children with fkeys to parent, even on error, because the parent table may not be required for the child tables to be valid, such as for taxonomic-data-only datasets that nevertheless have nodes for the non-taxonomic tables in their mappings.
- 06:38 AM Revision 5724: sql.py: parse_exception(): types cannot be matched MissingCastException: Use the first type as the type to cast to instead of text
- 05:59 AM Revision 5723: sql.py: parse_exception(): InvalidValueException: Fixed bug in regexp where can't use .*? before (?:...)? surrounding matched value, because it prevents the value from being matched now that it is optional
- 05:52 AM Revision 5722: inputs/.NCBI/nodes/header.csv: Updated for new staging table format, which includes a row_num column in each joined table
- 05:51 AM Revision 5721: inputs/.NCBI/nodes/create.sql: Updated for new src table names
- 05:36 AM Revision 5720: xml_func.py: process(): Pass on_error through to sql_io.put(). This fixes a bug in row-based import where DB errors in the xml_func.process() phase would abort the entire import instead of being tracked and having the return value set to None.
- 05:33 AM Revision 5719: sql_io.py: put(): Pass on_error through to put_table()
- 05:19 AM Revision 5718: sql_io.py: put_table(): log_exc(): Return False if removing all rows and have callers break the main loop so that no further exception-handling code is processed before the main loop is exited
- 05:17 AM Revision 5717: sql.py: parse_exception(): InvalidValueException: Also match exceptions which don't provide a specific value but just indicate that a value was invalid, such as PL/Python's "day is out of range for month"
- 04:39 AM Revision 5716: db_xml.py: put(): Inserting children with fkeys to parent: Don't do this if this node had an error and sql_io.put_table() returned None as the generated pkey. This fixes a bug where a node with an error will still try to create children with fkeys to parent, but pass None as the fkey to parent, which the recursive put() call will then incorrectly treat as there being no field with an fkey to parent at all rather than a field whose value is NULL. This causes function overload resolution to be unable to find the intended function, because it is missing a parameter.
- 04:34 AM Revision 5715: sql.py: parse_exception(): function MissingCastException: Return the actual type of the function's 1st param, using new function_param0_type(), rather than just text
- 04:31 AM Revision 5714: sql.py: parse_exception(): function MissingCastException: Fixed bug where can't return the function name as the name of what was missing the cast, because this must be a column
- 04:28 AM Revision 5713: sql.py: Added function_param0_type()
- 04:26 AM Revision 5712: sql.py: parse_exception(): function MissingCastException: Only treat DoesNotExistException as a MissingCastException if the query that was run did not already include a cast, to avoid infinite exception-handling recursion
- 04:24 AM Revision 5711: sql.py: parse_exception(): function MissingCastException: Fixed bug where determining whether the exception is a MissingCastException rather than a DoesNotExistException needs to check whether the function exists rather than whether it's the same in the exception message as in the query that was run. The exception message will of course copy the function name verbatim from the query, so there is no information in the exception message itself to indicate whether the DoesNotExistException was caused by a missing cast or by a nonexistent function.
- 04:19 AM Revision 5710: sql.py: parse_exception(): function MissingCastException: Documented that the regexp match to extract the function name also checks that a function signature with param types was matched, indicating a function call rather than cast to regproc. This check will also help avoid infinite recursion when function MissingCastException parsing calls database structure introspection functions.
- 04:15 AM Revision 5709: sql.py: parse_exception(): function MissingCastException: Don't match quotes around the function name because this particular exception (incorrect param type) does not include them. Casts to regproc, which also produce a DoesNotExistException, include the quotes but do not indicate a MissingCastException.
- 04:12 AM Revision 5708: sql.py: parse_exception(): function MissingCastException: Fixed bug where the 1st param's type in the exception's function signature is not actually the type the argument needs to have, because this is just the argument's current type
- 04:04 AM Revision 5707: sql.py: parse_exception(): typed_name_re: Also match identifiers without quotes, such as functions in "No function matches the given name and argument types" errors. This fixes a bug where DoesNotExistExceptions could not be parsed as MissingCastExceptions when applicable because the DoesNotExistException pattern would not even match.
- 03:57 AM Revision 5706: inputs/.NCBI/: Renamed higher_taxa to nodes because it currently doesn't just contain the higher taxa
- 01:24 AM Revision 5705: schemas/vegbien.sql: taxonlabel: taxonlabel_2_set_canon_label_id(): Only run if matched_label_id has actually changed, to avoid infinite recursion when updating canon_label_id on labels that resolve to this label when there are cycles in the data
- 01:21 AM Revision 5704: inputs/.NCBI/: Renamed higher_taxa to nodes because it currently doesn't just contain the higher taxa
- 12:57 AM Revision 5703: inputs/.NCBI/: Renamed higher_taxa to nodes because it currently doesn't just contain the higher taxa
- 12:49 AM Revision 5702: inputs/.NCBI/: Renamed names, nodes to *.src so they wouldn't get an automatic row_num column and can be used in higher_taxa's join
- 12:38 AM Revision 5701: inputs/NCU-NCSC/Specimen/+header.csv: Fixed bug where needed ! at beginning to indicate a header override file, which prevents the following row from being treated as data
- 12:36 AM Revision 5700: units.py: MissingUnitsException: Fixed bug where quantity is a Quantity object, not a string, and thus needs to be converted to a string using strings.ustr()
- 12:25 AM Revision 5699: inputs/FIA/Organism/test.xml.ref: Accepted new test output now that FIA table is sorted in the order of the original CSV after staging table reinstallation
- 12:24 AM Revision 5698: inputs/VegBank/taxonobservation_/create.sql: Removed dropping of row_num column, which is no longer added on non-CSV tables
- 12:22 AM Revision 5697: input.Makefile: Staging tables installation: %/install: Moved "table-scope src table's row_num col" comment outside of define block so it wouldn't be echoed to stdout even when the table is not a src table
- 12:17 AM Revision 5696: Added inputs/NCU-NCSC/Specimen/+header.csv header override to remove empty, unnamed column at end
- 12:05 AM Revision 5695: inputs/*/*/header.csv: Regenerated for new staging tables format (which now includes a row_num column on every CSV table), as part of reinstalling staging tables
10/22/2012
- 11:59 PM Revision 5694: inputs/VegBank/vegbank.~.clean_up.sql: Fixed bug where DROP VIEW statements needed IF EXISTS because CASCADEs on previous DROP VIEWs may have already dropped the view in question
- 11:57 PM Revision 5693: input.Makefile: Staging tables installation: %/install: Fixed bug where a .src table's row_num column needed to have the table name prefixed (making it globally unique) to allow joining the table with other tables
- 11:31 PM Revision 5692: input.Makefile: Staging tables installation: sql/install: Fixed bug where $(logInstall) needed to be called with arguments, so that either > or >> would be used before the install log's filename
- 08:22 PM Revision 5691: tnrs.py: submission_request_template: Use just Tropicos as the name source, as Brad says "GCC is for only one family (Asteraceae)" and USDA's "taxonomy is of lower quality and sometimes conflicts with Tropicos"
10/19/2012
- 06:20 PM Revision 5690: sql.py: parse_exception(): function MissingCastException: Support functions with named parameters
- 06:18 PM Revision 5689: sql.py: parse_exception(): function MissingCastException: Support function names enclosed in quotes on the context line
- 06:15 PM Revision 5688: mappings/VegCore-VegBIEN.csv: taxonName: Place it in taxonomicname instead of taxonepithet for lower taxa, because the only datasource that currently provides this field (NCBI) actually provides the full taxonomicname instead of the epithet at the current rank for lower taxa. (taxonomicname is not applicable to higher taxa because their names are not guaranteed to be globally unique.) taxonName may need to be renamed and/or redefined to account for this ambiguity in NCBI's usage.
- 06:14 PM Revision 5687: mappings/VegCore-VegBIEN.csv: Do not include the taxonName in the concatenated taxonomicname because it is NOT globally unique. The same name may be used at different taxonomic ranks and mean different things, and lower taxa may have the name appear in multiple genuses or species, meaning different things.
- 06:04 PM Revision 5686: mappings/VegCore-VegBIEN.csv: Do not include the taxonName in the concatenated taxonomicname because it is NOT globally unique. The same name may be used at different taxonomic ranks and mean different things, and lower taxa may have the name appear in multiple genuses or species, meaning different things.
- 05:57 PM Revision 5685: schemas/vegbien.sql: Added _is_higher_taxon()
- 05:52 PM Revision 5684: README.TXT: Documentation: To import and scrub just the test taxonomic names: Added `make inputs/.TNRS/cleanup` after `make backups/TNRS.backup/restore` because the PostgreSQL collation may differ between vegbiendev's and the user's DB
- 05:50 PM Revision 5683: sql.py: parse_exception(): DoesNotExistException: If item not found was a function and not found only because of a missing cast, raise MissingCastException instead. This should allow automatic casts to be added on function parameters as well as table columns.
- 05:28 PM Revision 5682: schemas/vegbien.sql: analytical_db_view: Fixed bug where needed to join to taxonverbatim on taxonverbatim_id (the pkey) instead of taxonlabel_id, which used to be the pkey but is now an fkey
- 05:22 PM Revision 5681: inputs/test_taxonomic_names/test_scrub: Remove any previous version of public.test_taxonomic_names before renaming public to it
- 05:19 PM Revision 5680: inputs/test_taxonomic_names/test_scrub: Fixed bug where public.sql export did not include the "CREATE SCHEMA public" statement, because pg_dump doesn't add it to backups, by using new schemas/rename/% make target to first rename the public schema and then exporting it
- 05:12 PM Revision 5679: root Makefile: VegBIEN DB: Schemas: schemas/rotate: Use new schemas/rename/%
- 05:12 PM Revision 5678: root Makefile: VegBIEN DB: Schemas: Added schemas/rename/% to rename the public schema
- 04:54 PM Revision 5677: mappings/VegCore-VegBIEN.csv: Removed filter preventing taxonomicStatus from being placed in taxonlabel if a morphospecies was provided, because the morphospecies actually never goes in the *matched* taxonlabel, only the *verbatim* taxonlabel
- 04:50 PM Revision 5676: mappings/VegCore-VegBIEN.csv: morphospecies: Also place it in the verbatim (input name's) taxonlabel. Note that it does not go in the matched name's taxonlabel, because that contains only fields from the matched name. The verbatim taxonlabel is thus a synonym of the matched taxonlabel where there is no morphospecies, or a child of it if there is a morphospecies.
- 04:36 PM Revision 5675: mappings/VegCore-VegBIEN.csv: Do not place taxonomicStatus in taxonlabel if a morphospecies was provided, to prevent it from being incorrectly marked as accepted
- 04:25 PM Revision 5674: mappings/VegCore-VegBIEN.csv: morphospecies -> taxonverbatim.morphospecies: Fixed bug where needed suffix with _if statement then clause
- 04:23 PM Revision 5673: inputs/test_taxonomic_names/_scrub/public.sql, TNRS.sql: Regenerated with schema changes
- 03:45 PM Revision 5672: pg_dump_vegbien: Added opts env var to allow specifying options to a Makefile command, which does not take positional arguments
- 03:37 PM Revision 5671: README.TXT: Schema changes: files to update with any renamings: Removed tnrs_db because that is now abstracted from the schema through the tnrs_input_name view. Note that PostgreSQL will automatically update tnrs_input_name with any table or column renames, which is the significant advantage of using a view rather than a hardcoded query.
- 03:35 PM Revision 5670: schemas/vegbien.sql: tnrs_input_name: Use DISTINCT instead of DISTINCT ON because there is only one column
- 03:34 PM Revision 5669: tnrs_db: Use new tnrs_input_name view to avoid hardcoding changing schema information
- 03:25 PM Revision 5668: inputs/test_taxonomic_names/test_scrub, README.TXT: Documented that `make schemas/public/reinstall` must come after TNRS restore to recreate the tnrs_input_name view, which has a dependency on the TNRS schema
- 03:23 PM Revision 5667: schemas/vegbien.sql: Added tnrs_input_name view for use by tnrs_db
- 12:53 PM Revision 5666: schemas/vegbien.sql: taxonlabel, taxonverbatim: Updated comments for new taxonlabel/taxonverbatim split
- 12:42 PM Revision 5665: schemas/vegbien.sql: taxonlabel_update_ancestors(): Use aliased types (http://www.postgresql.org/docs/8.3/static/plpgsql-declarations.html#PLPGSQL-DECLARATION-TYPE) where possible
- 12:37 PM Revision 5664: schemas/vegbien.sql: taxonlabel_update_ancestors(): Adding new parent's ancestors: Change unique_violations to warnings so they don't abort the import. unique_violations should never happen unless there are cycles of two or mode nodes, but they seem to be happening nevertheless, so this will provide a workaround to that problem.
- 12:18 PM Revision 5663: inputs/import.stats.xls: Updated import times
10/18/2012
- 04:58 PM Revision 5662: Regenerated vegbien.ERD exports
- 04:55 PM Revision 5661: tnrs_db: Updated with schema changes
- 04:54 PM Revision 5660: schemas/vegbien.sql: taxonverbatim: Removed subclass relationship to taxonlabel in order to allow multiple taxonverbatims to point to the same taxonlabel. This involves adding a taxonverbatim_id serial column and pointing all fkeys to taxonverbatim to that column.
- 04:43 PM Revision 5659: schemas/vegbien.sql: analytical_db_view: Fixed bug where needed to join on taxonverbatim before joining on taxonlabel, now that taxondetermination is linked directly to taxonverbatim. Interestingly, PostgreSQL did not flag this error when the schema was changed, but only when the schema was reloaded from the DDL.
- 04:30 PM Revision 5658: schemas/vegbien.ERD.mwb: Moved taxonlabel to the right of taxonverbatim to make room for taxonverbatim to expand
- 04:21 PM Revision 5657: schemas/vegbien.sql: Link taxondetermination to taxonverbatim (which is a subclass of taxonlabel) instead of directly to taxonlabel. This will enable later having multiple taxonverbatims for one taxonlabel.
- 04:04 PM Revision 5656: schemas/vegbien.sql: taxonlabel: Renamed identifyingtaxonomicname to taxonomicname because the taxonomicname provided by the datasource is now in taxonverbatim, so there is no name collision. Note that both of these fields store the same type of information, but taxonlabel's is autogenerated while taxonverbatim's is verbatim (and is only set if provided by the datasource).
- 03:57 PM Revision 5655: schemas/vegbien.sql: taxonlabel: Moved non-scoping fields to new taxonverbatim subclass table, which contains the component parts of the taxonlabel
- 03:06 PM Revision 5654: schemas/vegbien.sql: taxonlabel: Renamed taxonlabel_2_propagate_canon_label_id() to taxonlabel_2_set_canon_label_id() for clarity
- 03:04 PM Revision 5653: schemas/vegbien.sql: taxonlabel_2_propagate_canon_label_id(): If no matched taxonlabel, make self-reference. This fixes a bug in analytical_db_view where rows without a canon_label_id were excluded because they did not have a corresponding canonical taxonlabel.
- 02:53 PM Revision 5652: schemas/vegbien.sql: taxonlabel_unique unique index: Removed binomial, author, taxonomicname, and morphospecies because these are now part of the identifyingtaxonomicname, which is also in the unique index
- 02:44 PM Revision 5651: schemas/vegbien.sql: taxonlabel: Require either an identifyingtaxonomicname or a taxonepithet. The NCBI inserted row count decreases by one because this prunes off a taxonlabel created for a parent node which was not contained in the first two rows (remember that NCBI taxa are not in dependency order, so parents are often imported after children).
- 02:41 PM Revision 5650: mappings/VegCore-VegBIEN.csv: Also generate the identifyingtaxonomicname for the original* taxondetermination's taxonlabel
- 02:31 PM Revision 5649: schemas/vegbien.sql: taxonlabel: Renamed taxonomicnamewithauthor to taxonomicname because it is equivalent to Darwin Core's scientificName
- 02:25 PM Revision 5648: mappings/VegCore-VegBIEN.csv: Also include morphospecies in the identifyingtaxonomicname, except for the matched TNRS taxonlabel, which should not contain morphospecies information
- 02:14 PM Revision 5647: mappings/VegCore-VegBIEN.csv: Mapped acceptedScientificName
- 01:51 PM Revision 5646: mappings/VegCore-VegBIEN.csv: Also create the identifyingtaxonomicname on the verbatim taxonlabel supplied by the datasource, in addition to on the TNRS input taxonlabel that the verbatim taxonlabel is matched up with
- 01:46 PM Revision 5645: mappings/VegCore-VegBIEN.csv: Expanded brace expressions for putting together the identifyingtaxonomicname
- 01:21 PM Revision 5644: mappings/VegCore-VegBIEN.csv: Always generate the concatenated identifyingtaxonomicname, even for higher taxa, to ensure that this field is always populated. Note that this will cause names of higher taxa to be scrubbed by TNRS, but this is usually not a problem because such names either have no match or not a close enough match based on the name only. Naming conventions generally cause names at different ranks to be different, so that collisions with lower ranks should not be a problem.
- 01:05 PM Revision 5643: tnrs_db: Fixed bug where needed to remove internal identifyingtaxonomicname duplicates as well as duplicates with existing Name_submitted values, to avoid violating the TNRS.tnrs pkey constraint when the scrubbed names are later inserted. Note that the taxonlabel_0_unique_identifying_name unique index is not sufficient to prevent internal duplicates, because it includes the creator_id (and thus allows multiple instances of the same name defined by different creators).
- 01:01 PM Revision 5642: sql.py: mk_select(): Don't add table0 to order_by with no table, because this could cause it not to match a corresponding DISTINCT ON column with no explicit table. PostgreSQL apparently does not treat a column with no explicit table and a column with the applicable table as identical for purposes of ORDER BY/DISTINCT ON checking, even when they refer to the same physical column.
- 12:53 PM Revision 5641: sql.py: mk_select(): order_by defaults to first distinct_on column when distinct_on provided
- 12:36 PM Revision 5640: tnrs_db: Updated with schema changes
- 12:33 PM Revision 5639: schemas/vegbien.sql: taxonlabel: Renamed taxonomicnamewithauthor to taxonomicname because it is equivalent to Darwin Core's scientificName
- 12:25 PM Revision 5638: schemas/vegbien.sql: taxonlabel: Renamed taxonomicname to binomial because it excludes the author
- 12:15 PM Revision 5637: schemas/vegbien.sql: taxonlabel.taxonomicname, taxonomicnamewithauthor comments: Corrected to show that taxonomicnamewithauthor is actually scientificName, while taxonomicname does not directly correspond to a DwC term (but would be the binomial)
- 12:13 PM Revision 5636: schemas/vegbien.sql: taxonlabel.taxonomicnamewithauthor comment: Removed no longer applicable 'Equivalent to "Name sec. x"'. The "sec" is now stored in taxonconcept.concept_reference_id.
- 12:10 PM Revision 5635: mappings/Makefile: .VegCore.csv.last_cleanup: Remove duplicate entries using uniq
- 12:09 PM Revision 5634: mappings/VegCore.csv: Removed duplicate entries using uniq
- 12:06 PM Revision 5633: mappings/VegCore.csv: Removed *scientificNameWithAuthorship, which are now represented by *scientificName
- 12:04 PM Revision 5632: mappings: Renamed *scientificNameWithAuthorship to *scientificName because scientificNameWithAuthorship is actually a synonym of DwC's scientificName ("The full scientific name, with authorship and date information if known" <http://rs.tdwg.org/dwc/terms/#scientificName>)
- 11:57 AM Revision 5631: mappings: Renamed *scientificName to *binomial because DwC defines the scientificName as "The full scientific name, with authorship and date information if known", but many datasources do not include the author in their scientific name, and the fields scientificName is mapped to in VegBIEN assume it does not include the author
- 11:44 AM Revision 5630: mappings/VegCore.csv: Added verbatimBinomial
- 11:41 AM Revision 5629: mappings/VegCore.csv: Redefined *binomial to "Taxonomic name without author", rather than genus+species
- 11:32 AM Revision 5628: schemas/vegbien.sql: taxonconcept.taxonlabel_id: Changed type from serial to integer because this is a subclass, and therefore each taxonconcept must first have a corresponding entry in taxonlabel
- 11:29 AM Revision 5627: schemas/vegbien.sql: Moved taxonlabel.concept_reference_id to new taxonconcept table, which is a subclass of taxonlabel that adds information about who the taxon concept is according to
- 11:13 AM Revision 5626: taxonlabel: Renamed accepted_label_id to canon_label_id to allow any taxonlabel to be the canonical taxonlabel for this taxonlabel, whether or not its status is accepted
- 11:01 AM Revision 5625: schemas/filter_ERD.csv: Remove the methodtaxonclass.submethod_id fkey to taxonlabel, to make room in the ERD for additional taxon tables
- 10:52 AM Revision 5624: schemas/vegbien.sql: establishmentmeans_dwc: Corrected source comment
- 10:51 AM Revision 5623: schemas/vegbien.sql: taxonomic_status enum: Added source comment
- 10:49 AM Revision 5622: schemas/vegbien.sql: taxonlabel_relationship: Added relationship, with relationship enum
- 10:21 AM Revision 5621: mappings/VegCore-VegBIEN.csv: Mapped taxonomicStatus
- 10:20 AM Revision 5620: inputs/.TNRS/tnrs/test.xml.ref: Updated inserted row count
- 10:15 AM Revision 5619: mappings/VegCore.csv: Removed duplicate entry for taxonomicStatus, which is also a DwC term
- 10:14 AM Revision 5618: mappings/VegCore.csv: Added taxonomicStatus
- 10:13 AM Revision 5617: schemas/vegbien.sql: taxonlabel: Added taxonstatus, with taxonomic_status enum
- 09:40 AM Revision 5616: schemas/vegbien.sql: taxonlabel.creator_id comment: Removed no longer accurate comment that this is the "according to" and "Name sec. x", which is now stored in concept_reference_id
- 09:37 AM Revision 5615: schemas/vegbien.sql: taxonlabel: Added concept_reference_id, which is the entity that defined the taxon concept (who the taxon label is according to)
- 09:22 AM Revision 5614: schemas/vegbien.ERD.mwb: Moved taxonlabel_relationship to the right of taxonlabel to provide room for taxonlabel to grow
10/17/2012
- 04:27 PM Revision 5613: Regenerated vegbien.ERD exports
- 04:25 PM Revision 5612: mappings/VegCore-VegBIEN.csv: Remapped morphospecies to new taxonlabel.morphospecies per today's conference call
- 04:23 PM Revision 5611: schemas/vegbien.sql: taxonlabel: Added separate morphospecies field per today's conference call, where it was decided it could not go in taxonepithet (the lowest-rank component of the name)
- 04:17 PM Revision 5610: schemas/vegbien.sql: Deleted taxonusage table per today's conference call, where it was decided that it was not needed
- 04:14 PM Revision 5609: schemas/vegbien.sql: Renamed taxonlabel_ancestor to taxonlabel_relationship per today's conference call, where it was decided that it would eventually contain asserted relationships (such as synonym and parent) in addition to autopopulated ancestor relationships
- 04:12 PM Revision 5608: schemas/vegbien.sql: Renamed taxonconcept to taxonlabel per today's conference call, where it was decided that taxonconcept contained too many unrelated fields to be purely a taxon concept
- 04:01 PM Revision 5607: inputs/import.stats.xls: Updated import times
- 04:01 PM Revision 5606: inputs/test_taxonomic_names/_scrub/public.sql, TNRS.sql: Regenerated with schema changes
- 01:47 PM Revision 5605: schemas/vegbien.ERD.mwb: Fixed lines
- 01:45 PM Revision 5604: Regenerated vegbien.ERD exports
- 01:44 PM Revision 5603: schemas/vegbien.sql: taxonconcept_ancestor: Renamed taxonconcept_id to descendant_id to emphasize the direction of the relationship between the two taxonconcepts
- 01:35 PM Revision 5602: schemas/vegbien.ERD.mwb: Added taxonconcept_ancestor to the diagram since it is now a core table for storing taxonomic information
- 01:15 PM Revision 5601: mappings/VegCore-VegBIEN.csv: Mapped accordingTo to taxonconcept.creator_id, and have it take the place of identifiedBy when both are present
- 01:12 PM Revision 5600: mappings/VegCore-VegBIEN.csv: Remapped people's names split apart into name components in party to new party.fullname, which does not require splitting or make assumptions about the number of people who may be listed in a particular name field and which components of their name(s) are present
- 01:02 PM Revision 5599: schemas/vegbien.sql: party: Added fullname
- 12:55 PM Revision 5598: mappings/VegCore.csv: Added accordingTo
- 12:47 PM Revision 5597: inputs/.TNRS/tnrs/map.csv: Mapped Name_matched_url to scientificNameID, since the URL uniquely identifies the matched taxonconcept
- 12:43 PM Revision 5596: schemas/vegbien.sql: taxonconcept: Renamed taxonname to taxonepithet for clarity and to be consistent with TCS's use of "epithet" to denote what the taxonname was intended to be (http://www.tdwg.org/standards/117/download/#/UserGuidev_1.3.pdf)
- 12:18 PM Revision 5595: schemas/vegbien.sql: taxonconcept.creator_id: Documented that this is the concept reference for a taxon concept with an "according to", or the identifier's name for a nominal concept, and is equivalent to "Name sec. x"
- 12:02 PM Task #497 (Resolved): create examples of taxonomic names to test the limits of the new taxonomic schema
- See "*testNames.txt*":https://projects.nceas.ucsb.edu/nceas/attachments/download/377/testNames.txt
- 11:50 AM Revision 5594: sql_io.py: import_csv(): Add a row_num column at the beginning of the table, which is autopopulated by csvs.RowNumFilter (it cannot be autopopulated by the serial datatype, because this does not support COPY FROM with a NULL-equivalent value in the serial field). This fixes a bug in csv2db where rows would not stay in inserted order upon querying the table, and would be returned in a different order each query, which prevented LIMIT/OFFSET based subsetting from returning consistent, nonoverlapping results. This occurs because PostgreSQL unfortunately does not return rows in inserted order (or any stable order: "If sorting is not chosen, the rows will be returned in an unspecified order [which] must not be relied on" <http://www.postgresql.org/docs/8.3/static/queries-order.html>), so an explicit ORDER BY is always needed to ensure staging table rows are retrievable in the order they were inserted.
- 11:43 AM Revision 5593: csvs.py: Added RowNumFilter, which adds a row # column at the beginning of each row
- 11:42 AM Revision 5592: streams.py: LineCountStream, LineCountInputStream: Fixed bug where line_num was 1 too high because it started at 1 *and* was incremented *before* each line is returned. It now properly starts at 1, but the initial line_num value is 0 to increment to 1 upon encountering the first line. This off-by-one behavior may have been needed for code that associates an error message with a line #, but such code should add 1 to the line_num to get the line # of the error *if* the error prevents the next line from being read by the LineCount*Stream.
- 11:04 AM Revision 5591: sql_io.py: import_csv(): Take a reader and header rather than a stream to allow callers to pass in a wrapped CSV reader for filtering, etc.
- 11:00 AM Revision 5590: sql_io.py: append_csv(): Take a reader and header rather than a stream_info and stream to allow callers to use the simpler csvs.reader_and_header() function. This also allows callers to pass in a wrapped CSV reader for filtering, etc.
- 10:44 AM Revision 5589: csv2db, tnrs_db: Removed ProgressInputStream wrapper around input stream, which is no longer needed (and causes overlapping output) now that sql_io.append_csv() prints # rows read
- 10:42 AM Revision 5588: sql_io.py: append_csv(): Wrap input stream in a ProgressInputStream that reports rows (rather than lines) read
- 10:40 AM Revision 5587: csvs.py: InputRewriter: Use new StreamFilter to translate StopIteration EOF to ''
- 10:36 AM Revision 5586: csvs.py: Added StreamFilter
- 10:36 AM Revision 5585: csvs.py: InputRewriter: Also support stream inputs which report EOF as '' instead of StopIteration
- 09:55 AM Revision 5584: sql_io.py: append_csv(): Removed no longer used INSERT mode, since all callers now use the default COPY FROM
- 09:53 AM Revision 5583: sql_io.py: import_csv(): Removed no longer needed manual setting of use_copy_from, which defaults to True in append_csv()
- 09:50 AM Revision 5582: csv2db: Removed no longer needed manual setting of use_copy_from, which defaults to True in sql_io.import_csv()
- 09:49 AM Revision 5581: csv2db: Removed no longer needed separate handling of sql.DatabaseErrors, because all recoverable errors caused by COPY FROM (EncodingException and ragged rows) are now handled or avoided
- 09:46 AM Revision 5580: csv2db: Handle EncodingException separately by changing the connection encoding to LATIN1 and retrying
- 09:45 AM Revision 5579: sql.py: DbConn: Added set_encoding()
- 09:32 AM Revision 5578: sql_io.py: append_csv(): Parse any exceptions generated by the COPY FROM using new sql.parse_exception()
- 09:28 AM Revision 5577: sql.py: run_query(): Factored exception parsing out into new parse_exception()
- 09:22 AM Revision 5576: sql.py: Added EncodingException and parse it in run_query()
- 09:14 AM Revision 5575: sql.py: Removed no longer used NameException
- 09:14 AM Revision 5574: csvs.py: Filter: Added empty close() method to support using it as a stream (such as with streams.ProgressInputStream)
- 09:01 AM Revision 5573: sql_io.py: append_csv(): Don't disable COPY FROM for TSVs, which are now supported using csvs.InputRewriter
- 08:59 AM Revision 5572: sql_io.py: append_csv(): COPY FROM: Wrap provided stream in standardizing stream to fix ragged rows (with unequal # columns) and nonstandard CSV dialects (such as TSV with \-escaped newlines)
- 08:56 AM Revision 5571: csvs.py: Added InputRewriter, which wraps a reader, writing each row back to CSV
- 08:54 AM Revision 5570: csvs.py: Added ColCtFilter, which gives all rows the same # columns
- 07:25 AM Revision 5569: sql_io.py: row_num_col_def: Changed type to integer so the row_num can be populated directly by the insert process
- 07:19 AM Revision 5568: sql_io.py: Added row_num_col_def for use by import_csv(). The row_num column will be necessary again because PostgreSQL unfortunately does not return rows in inserted order (or any stable order: "If sorting is not chosen, the rows will be returned in an unspecified order [which] must not be relied on" <http://www.postgresql.org/docs/8.3/static/queries-order.html>), so an explicit ORDER BY is always needed to ensure staging table rows are retrievable in the order they were inserted.
10/16/2012
- 10:58 PM Revision 5567: mappings/VegCore.csv: Removed unit-ambiguous height. Use height_m, height_ft instead.
- 10:57 PM Revision 5566: mappings/Veg+-VegCore.csv: Added height
- 10:57 PM Revision 5565: mappings/Veg+-VegCore.csv: Added height
- 10:52 PM Revision 5564: mappings/VegCore-VegBIEN.csv: Removed no longer used height mapping. Use height_m, height_ft instead.
- 10:39 PM Revision 5563: README.TXT: Data import: import_all: Added NCBI backbone to note about import_all not immediately returning control to the shell
- 10:30 PM Revision 5562: inputs/FIA/Organism/map.csv: Height: Remapped to height_ft, assuming units based on the range of values, the height of the tallest tree, and location inside the U.S.
- 10:23 PM Revision 5561: inputs/FIA/Organism/test.xml.ref: Accepted new inserted row count
- 10:01 PM Revision 5560: mappings/VegCore-VegBIEN.csv: Mapped height_ft
- 09:58 PM Revision 5559: schemas/functions.sql: Added _ft_to_m()
- 09:52 PM Revision 5558: mappings/VegCore.csv: Added height_ft
- 09:38 PM Revision 5557: inputs/SALVIAS/stems/map.csv: stem_height_m: Remapped to height_m using units from <http://salvias.net/Documents/salvias_data_dictionary.html#Plot+data>
- 09:37 PM Revision 5556: inputs/SALVIAS-CSV/Organism/map.csv: stem_height_m: Re-sourced units to stem_height_m rather than height_m definition in SALVIAS data dictionary
- 09:29 PM Revision 5555: Regenerated vegbien.ERD exports
- 09:23 PM Revision 5554: schemas/vegbien.sql: taxonconcept: taxonconcept_update_ancestors() trigger: Fixed bug where matched_concept_id needed to be changed to NULL when equal to taxonconcept_id, to avoid including the node itself with its parent's ancestors (which would violate the taxonconcept_ancestor pkey)
- 09:19 PM Revision 5553: sql_io.py: put_table(): Ensuring into's out_pkey is different from in_pkey: Prepend "out." instead of out_table to avoid long column names for the output pkey
- 09:18 PM Revision 5552: sql_gen.py: concat(): Allow multiple "column" suffixes with "." when matching the existing suffix
- 08:47 PM Revision 5551: schemas/vegbien.sql: taxonconcept: taxonconcept_update_ancestors() trigger: Corrected comment explaining why we don't need an ON DELETE trigger to say that this is because the foreign key for *taxonconcept_ancestor.ancestor_id*, not taxonconcept.parent_id, is ON DELETE CASCADE. The auto-deletion will also occur if taxonconcept.parent_id is ON DELETE CASCADE, because taxonconcept_ancestor.taxonconcept_id is ON DELETE CASCADE, but it is not actually necessary to have cascading deletes on taxonconcept.parent_id (and SET NULL may in fact sometimes be more appropriate).
- 08:33 PM Revision 5550: schemas/tree_cross-links.sql: Removed header comments added by pgAdmin
- 08:30 PM Revision 5549: schemas/tree_cross-links.sql: Updated for new taxonconcept_update_ancestors() trigger
- 08:21 PM Revision 5548: schemas/vegbien.sql: taxonconcept: Rewrote taxonconcept() trigger to avoid completely reinserting the taxonconcept_ancestor entries of all descendants every time taxonconcept changes or using trigger recursion to find descendants. Instead, just delete the old parent's ancestors from and add the new parent's ancestors to each descendant, using taxonconcept_ancestor itself (with the new taxonconcept_ancestor_descendants index) to find all descendants. As an additional optimization, only update taxonconcept_ancestor if the parent_id or matched_concept_id has actually changed. This fixes a bug in NCBI where inserting taxonconcepts out of dependency order caused taxonconcept_ancestor entries to be repeatedly regenerated, slowing the import down to a crawl.
- 07:42 PM Revision 5547: schemas/vegbien.sql: taxonconcept: Added taxonconcept_3_parent_id_avoid_self_ref() trigger to avoid recursive references in root taxonconcepts (taxonconcepts with no parent). This will simplify the new taxonconcept_update_ancestors() trigger.
- 06:32 PM Revision 5546: schemas/vegbien.sql: taxonconcept_ancestor: Added taxonconcept_ancestor_descendants index to support looking up all the descendants for a taxonconcept. This will be used by the new taxonconcept_update_ancestors() trigger, which will support inserting taxonconcepts out of dependency order (such as for NCBI).
- 04:35 PM Revision 5545: schemas/vegbien.sql: *_update_ancestors(): Made trigger deferred, so that it would run after all rows have been inserted in a bulk insert, such as during column-based import. This ensures that ancestors lists are not populated until all parents are inserted, which may occur out of order for datasources (such as NCBI) whose nodes are not in dependency order. (A node that newly acquires a parent will have to update all its descendants, which will then be updated again when its parent acquires its own parent.)
- 04:28 PM Revision 5544: lib/PostgreSQL-MySQL.csv: Also filter out constraint triggers in addition to regular triggers
10/15/2012
- 05:37 PM Revision 5543: inputs/Madidi/Organism/map.csv: Total height: Remapped to height_m, assuming units based on the range and precision of values
- 05:33 PM Revision 5542: inputs/VegBank/stemcount/map.csv: stemheight: Remapped to height_m using units from <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename>
- 05:29 PM Revision 5541: inputs/SALVIAS/plotObservations/map.csv, inputs/SALVIAS-CSV/Organism/map.csv: height_m, stem_height_m: Remapped to height_m using units from <http://salvias.net/Documents/salvias_data_dictionary.html#Plot+data>
- 05:24 PM Revision 5540: mappings/VegCore-VegBIEN.csv: Mapped height_m
- 05:15 PM Revision 5539: mappings/VegCore.csv: Added height_m
- 04:20 PM Revision 5538: mappings/VegCore.csv, VegCore-VegBIEN.csv: Removed no longer used and unit-ambiguous organismX, organismY. Use organismX_m, organismY_m instead.
- 04:18 PM Revision 5537: inputs/VegBank/stemlocation/map.csv: stemxposition, stemyposition: Remapped to organismX_m/organismY_m using units from <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemlocation&entity=dba_tabledescription&where=where_tablename>
- 04:06 PM Revision 5536: inputs/TEAM/*/map.csv: 1ha Plot X Coordinate, 1ha Plot Y Coordinate: Remapped to organismX_m/organismY_m using units from <https://projects.nceas.ucsb.edu/nceas/projects/bien/repository/raw/inputs/TEAM/_src/TEAM-DataPackage-20120920191251_3859/Vegetation+-+Trees+&+Lianas/Vegetation-Tree-and-Liana-Metadata-1.5.pdf>
- 03:59 PM Revision 5535: inputs/SALVIAS/plotObservations/map.csv, inputs/SALVIAS-CSV/Organism/map.csv: x_position, y_position: Remapped to organismX_m/organismY_m using units from <http://salvias.net/Documents/salvias_data_dictionary.html#Plot+data>
- 03:51 PM Revision 5534: inputs/Madidi/Organism/map.csv: Subplot X, Subplot Y: Remapped to organismX_m/organismY_m, assuming units based on the size of values relative to the plot area, which has units of ha
- 03:44 PM Revision 5533: inputs/CTFS/StemObservation/map.csv: x, y: Remapped to organismX_m/organismY_m, assuming units based on the size of values relative to plot area, which has units of ha
- 03:30 PM Revision 5532: mappings/VegCore-VegBIEN.csv: Mapped organismX_m, organismY_m
- 03:29 PM Revision 5531: mappings/VegCore.csv: Added organismX_m, organismY_m
- 03:23 PM Revision 5530: sql_io.py: put_table(): full_in_table: Create it using new sql.copy_table() instead of sql.run_query_into()
- 03:23 PM Revision 5529: sql.py: Added copy_table()
- 03:14 PM Revision 5528: sql.mk_select() calls: Removed no longer needed order_by=None when limit=0
- 03:11 PM Revision 5527: sql.py: mk_select(): Set order_by to None if limit == 0
- 03:09 PM Revision 5526: inputs/.TNRS/schema.sql: Documented that accepted names must be processed before any names that resolve to them, because the entry for the accepted name contains all the ranks parsed out but the resolved name of another entry contains just some ranks and the taxonomic name. Column-based import will do this automatically when the total # of rows is <= the partition_size (because _taxonconcept_set_matched_concept_id()'s accepted taxonconcept is created after the main taxonconcept), but TNRS has more rows than this so sorting is needed to ensure that all the accepted names are processed in the first partitions.
- 02:52 PM Revision 5525: sql.py: table_order_by(): Cache the order_by in table.order_by and propagate it when a LIKE table is created
- 02:51 PM Revision 5524: sql_gen.py: Table: Added order_by attr to cache the results of table_order_by()
- 02:36 PM Revision 5523: sql.select() calls: Removed order_by=None everywhere that a stable row order is required (i.e. consistent between selects, or consistent between table transformations). This causes several tests to return different inserted row counts, because the input table is now being accessed in pkey order instead of in table order. This fixes a bug where tables with more rows than ~100 would return different results for repeated calls of the same non-ordered select.
- 02:27 PM Revision 5522: sql.py: mk_select(): Use table_order_by() instead of table_pkey_col() to determine what column(s) to order by if order_by is set to order_by_pkey
- 02:26 PM Revision 5521: sql.py: Added table_pkey_index(), index_order_by(), table_cluster_on(), table_order_by()
- 01:10 PM Revision 5520: sql.py: Added index_exprs() and use it in index_cols()
- 01:08 PM Revision 5519: README.TXT: Data import: On local machine: Added `make inputs/.TNRS/cleanup`, which is necessary because the PostgreSQL collation may differ between vegbiendev's and your DB
- 12:24 PM Revision 5518: schemas/vegbien.sql: taxonconcept: taxonconcept_update_ancestors(): Use matched_concept_id's ancestors instead if available. (Recursively applied, this will use the ancestors of the accepted concept.) This facilitates finding all children of and matches to an accepted concept, which will all have an entry for that concept in taxonconcept_ancestor. Note that the concept's own parents will not be indexed in taxonconcept_ancestor, because only accepted ancestors are now stored in taxonconcept_ancestor. Documented that taxonconcept_ancestor now stores the *accepted* ancestors of a taxonconcept.
- 12:14 PM Revision 5517: schemas/vegbien.sql: taxonconcept: taxonconcept_2_propagate_accepted_concept_id(): Also update accepted_concept_id on concepts that resolve to this concept, which may have been created before this concept was marked as accepted if concepts are not imported in dependency order (accepted concepts first). Added index on matched_concept_id to speed up finding concepts that resolve to this concept.
- 12:10 PM Revision 5516: sql.py: mk_select(): order_by is order_by_pkey: Only order by the table's actual pkey, if it has one, rather than using the first column if it doesn't
- 12:08 PM Revision 5515: inputs/.TNRS/tnrs/test.xml.ref: Updated inserted row count
- 10:21 AM Revision 5514: db_xml.py: partition_size: Increased to 1,000,000 (>= NCBI.higher_taxa's size) so NCBI.higher_taxa can be imported completely in one partition. This is necessary because NCBI's taxonconcepts are not in dependency order (parents first), so a later partition cannot rely on the parents of its taxonconcepts having already been imported. Instead, all taxonconcepts must be imported at once and then separately, the parents of all taxonconcepts must be set.
- 10:08 AM Revision 5513: mappings/VegCore-VegBIEN.csv: taxonconcept.parent_id when explicit parent provided: Set taxonconcept.parent_id using new _taxonconcept_set_parent_id() *after* creating the child taxonconcept, so that the parent_id will point to the already-inserted parent taxonconcept instead of creating a new, empty parent taxonconcept. This creates a two-step import, where first the taxonconcepts are imported, and then the parent_ids are matched up. This is necessary for column-based import because all the parent taxonconcepts are imported in a separate iteration from the child taxonconcepts with only their sourceaccessioncode, so this iteration must occur after the child taxonconcept iteration in order to match up with fully-populated taxonconcepts. Row-based import, on the other hand, does not require _taxonconcept_set_parent_id() but does require the taxonconcepts to be provided in dependency order (parents first), which is unfortunately not the case for NCBI.
- 09:57 AM Revision 5512: schemas/vegbien.sql: *_update_ancestors(): Telling immediate children to update their ancestors lists: Exclude self to avoid infinite recursion
- 09:57 AM Revision 5511: schemas/vegbien.sql: *_update_ancestors(): Telling immediate children to update their ancestors lists: Exclude self to avoid infinite recursion
- 09:41 AM Revision 5510: schemas/vegbien.sql: Added _taxonconcept_set_parent_id()
- 09:37 AM Revision 5509: schemas/vegbien.sql: Renamed _set_matched_taxonconcept() to _taxonconcept_set_matched_concept_id() so that the function name is prefixed with the table it applies to
- 09:35 AM Revision 5508: db_xml.py: put(): Treat a child node which is a function (starts with _) as a child with fkey to parent rather than as a field in the table. Such a function accepts the table's pkey as one of its arguments.
- 09:05 AM Revision 5507: sql_gen.py: map_expr(): Don't replace an unquoted name when followed by ",", as it would be in an into table name for a function with multiple arguments (e.g. family in "_join_words(1=Field family, 2=Field name)")
- 08:49 AM Revision 5506: schemas/vegbien.sql: locationevent: Moved obsstartdate, obsenddate to top of table so they would be visible in the ERD
- 08:45 AM Revision 5505: sql_io.py: put_table(): ensure_cond(): track_data_error(): Concatenate the columns in the constraint together using , rather than adding a separate entry for each column, because the constraint is applicable to all columns together rather than to each column separately
- 08:26 AM Revision 5504: sql_io.py: put_table(): Renamed ignore_cond() to ensure_cond() for clarity
- 08:22 AM Revision 5503: import_all: Also import the NCBI tree of life, before the TNRS names
- 08:17 AM Revision 5502: mappings/VegCore-VegBIEN.csv: Also map acceptedFamily to the corresponding NCBI family
- 08:07 AM Revision 5501: lib/PostgreSQL-MySQL.csv: custom types: Also exclude time. Reordered excluded (built-in) types by name.
- 07:57 AM Revision 5500: inputs/import.stats.xls: Updated import times
- 07:50 AM Revision 5499: schemas/vegbien.sql: Changed `timestamp with time zone` fields to `date` because time information is not stored in these fields, and it's confusing to have an arbitrary timezone (the server's timezone) and an arbitrary time (midnight) set for input data that only has a precision to the nearest day
- 07:43 AM Revision 5498: sql_gen.py: null_sentinels: Added entry for date
- 07:40 AM Revision 5497: lib/PostgreSQL-MySQL.csv: custom types: Also exclude date, datetime
- 07:11 AM Revision 5496: README.TXT: Documentation: To import and scrub just the test taxonomic names: Run `make backups/TNRS.backup/restore` in the background because it takes awhile
- 06:45 AM Revision 5495: mappings/VegCore.csv: Re-sourced TaxonomicRankEnum fields to the official TCS schema rather than the TCS version in VegX
- 06:42 AM Revision 5494: schemas/vegbien.sql: taxonrank: Updated source to the TCS schema (rather than VegBank) for the new, expanded list. Note that although the list itself was compiled from the TCS version in VegX, the official TCS download does not differ from the VegX TCS in the TaxonomicRankEnum fields (the xs: namespace has just been replaced with xsd: by VegX).
10/12/2012
- 05:21 PM Revision 5493: schemas/vegbien.sql: analytical_db_view: taxonconcept: Join again on the accepted_concept_id in order to use the accepted taxonconcept rather than the verbatim taxonconcept from the datasource
- 05:14 PM Revision 5492: schemas/: svn:ignore log files
- 05:11 PM Revision 5491: Added inputs/.NCBI/. This uses many of the new schema and mappings features, such as taxonconcept.sourceaccessioncode and parentTaxonID
- 05:07 PM Revision 5490: mappings/VegCore-VegBIEN.csv: identifyingtaxonomicname: Don't create if taxonconcept has an explicit parent, because the taxonName (which is generally only a component of the full taxonomic name, e.g. specificEpithet) is not globally unique. Datasources that provide name components in such a way that levels at or below family can't be directly concatenated cannot currently receive an identifyingtaxonomicname for input to TNRS.
- 04:54 PM Revision 5489: mappings/VegCore-VegBIEN.csv: taxonName->identifyingtaxonomicname: Don't include the rank with the taxonName, because TNRS only allows the rank to be included in the taxonomic name if it's infraspecific (otherwise, it returns no or an invalid match due to the presence of what it sees as an invalid term or a name component)
- 04:48 PM Revision 5488: mappings/VegCore-VegBIEN.csv: Mapped taxonName to the TNRS input taxonconcept's identifyingtaxonomicname
- 04:28 PM Revision 5487: mappings/VegCore-VegBIEN.csv: Only forward taxonRank to the parent taxonconcept (which stores the infraspecific taxonconcept when the infraspecificEpithet is provided) if there is no explicit parent provided via parentTaxonID/etc.
- 04:09 PM Revision 5486: mappings/VegCore-VegBIEN.csv: Mapped parentScientificNameID, parentTaxonConceptID, parentTaxonID
- 04:03 PM Revision 5485: mappings/VegCore.csv: Added parentScientificNameID, parentTaxonConceptID, parentTaxonID
- 03:53 PM Revision 5484: input.Makefile: $(inDatasrc): Also include the vegbien_dest $schemas in the search_path, so that the datasource's SQL scripts (create.sql, etc.) can use VegBIEN functions and types
- 03:44 PM Revision 5483: lib/common.Makefile: Added $(comma)
- 02:41 PM Revision 5482: inputs/test_taxonomic_names/_scrub/public.sql: Regenerated with schema changes
- 02:38 PM Revision 5481: input.Makefile: Maps building: %/.map.csv.last_cleanup: Fixed bug where needed to include $(coreMap) as a prerequisite, because even though it is not used directly in this target's recipe, it is used by targets invoked via recursive make after the main recipe runs. In general, whenever targets forward commands to a recursive make target, they also need to forward those recursive targets' prerequisites by including them in their own prerequisites list.
- 02:29 PM Revision 5480: mappings/VegCore-VegBIEN.csv: Mapped taxonConceptID, taxonID, scientificNameID to taxonconcept.sourceaccessioncode. Note that taxonconcept stores all of these taxonomic entities, using creator_id+creationdate, taxonname+rank+parent_id, and identifyingtaxonomicname, respectively.
- 02:28 PM Revision 5479: mappings/VegCore-VegBIEN.csv: Mapped taxonConceptID, taxonID, scientificNameID to taxonconcept.sourceaccessioncode. Note that taxonconcept stores all of these taxonomic entities, using creator_id+creationdate, taxonname+rank+parent_id, and identifyingtaxonomicname, respectively.
- 02:13 PM Revision 5478: mappings/VegCore-VegBIEN.csv: Mapped taxonName
- 02:11 PM Revision 5477: mappings/VegCore.csv: Added taxonName
- 02:05 PM Revision 5476: schemas/vegbien.ERD.mwb: Fixed lines
- 01:55 PM Revision 5475: schemas/vegbien.sql: Copied functions in the functions schema that are also used by the public schema to the public schema, so that reinstalling the functions schema would not cause anything that depends on a function in it to be cascadingly deleted. Currently, this just affects analytical_db_view, which uses _fraction_to_percent().
- 01:44 PM Revision 5474: inputs/test_taxonomic_names/_scrub/public.sql: Regenerated with schema changes
- 01:36 PM Revision 5473: schemas/vegbien.sql: taxonconcept: Added taxonconcept_2_propagate_accepted_concept_id() trigger to auto-populate the accepted_concept_id
- 12:53 PM Revision 5472: schemas/vegbien.sql: taxonconcept.sourceaccessioncode: Added descriptive comment
- 12:53 PM Revision 5471: schemas/vegbien.sql: taxonconcept.accepted_concept_id: Added descriptive comment
- 12:48 PM Revision 5470: Regenerated vegbien.ERD exports
- 12:47 PM Revision 5469: schemas/vegbien.sql: taxonconcept: Added sourceaccessioncode, and allow it to scope the taxonconcept when provided
- 12:33 PM Revision 5468: inputs/test_taxonomic_names/_scrub/public.sql: Regenerated with schema changes
- 12:29 PM Revision 5467: schemas/vegbien.sql: taxonconcept: Renamed canon_concept_id to matched_concept_id, because this is actually the closest-match taxonconcept in the match hierarchy (datasource concept -> parsed concept -> matched concept -> accepted concept) rather than the accepted synonym, which goes in accepted_concept_id
- 05:51 AM Revision 5466: Regenerated vegbien.ERD exports
- 05:47 AM Revision 5465: schemas/vegbien.sql: taxonconcept: Renamed canon_concept_id to matched_concept_id, because this is actually the closest-match taxonconcept in the match hierarchy (datasource concept -> parsed concept -> matched concept -> accepted concept) rather than the accepted synonym, which goes in accepted_concept_id
- 05:34 AM Revision 5464: schemas/vegbien.sql: taxonconcept: Added accepted_concept_id
- 05:27 AM Revision 5463: schemas/vegbien.sql: taxonconcept.canon_concept_id: comment: Changed "accepted synonym" to "closest match", since canon_concept_id is actually a hierarchy from datasource concept -> parsed concept -> matched concept -> accepted concept
- 05:22 AM Revision 5462: schemas/vegbien.sql: taxonconcept: Added order # to trigger names so they run in a defined order (triggers are run in alphabetical order)
- 05:01 AM Task #498 (Resolved): add definitions to columns in "green tables"
- Definitions have been added on all new tables (currently taxonconcept and taxonconcept_ancestor). Other tables' colum...
- 04:56 AM Task #462 (Resolved): name backups according to svn revision instead of or in addition to the date
- This now happens for backups as well as log files
- 04:53 AM Revision 5461: README.TXT: Use new revision # in log filenames to get all the logs for an import. Changed <datetime> to <version> because the rotated public schema now also includes the svn revision.
- 04:44 AM Revision 5460: lib/common.Makefile: $(version): Include both the svn revision when make was started as well as the svn revision when the command is actually run (when these values differ), in case svn was updated between the time an import was started and the time a particular table started being imported. Because tables within a datasource are imported sequentially, it is possible that an update would have happened before the last table started importing.
- 04:23 AM Revision 5459: Makefile: Moved setting of $(root) before include of lib/common.Makefile because it's used by lib/common.Makefile
- 04:21 AM Revision 5458: Factored OS section out from Makefile, input.Makefile into lib/common.Makefile
- 04:13 AM Revision 5457: Makefile, input.Makefile: Use new $(version), which unlike $(date) also includes the svn revision, to version log files, etc. This way, the working copy can be put back to the way it was at the time of a given import (excluding changes to nonversioned files). This also makes it easier to get all the log files for a particular import when different tables' imports started at different times.
- 04:08 AM Revision 5456: Makefile: Added $(root) for use with $(rootRevision)
- 04:08 AM Revision 5455: lib/common.Makefile: Added $(version), to replace $(date) for versioning log files, etc., and helper function $(rootRevision)
- 04:07 AM Revision 5454: lib/common.Makefile: Added $(revision)
- 04:04 AM Revision 5453: input.Makefile: Removed no longer used $(SED)
- 04:03 AM Revision 5452: lib/common.Makefile: Added $(sed)
- 03:58 AM Revision 5451: Factored $(date) out from Makefile, input.Makefile into lib/common.Makefile
- 03:18 AM Revision 5450: sql_io.py: put_table(): DuplicateKeyException: Fixed bug where indexes with conditions needed to have the input rows filtered by the condition, to prevent trying to retrieve an existing/inserted row using a join on the index columns when the index in fact does not apply. This fixes a bug in the import of taxonconcept where the taxonconcept_0_unique_identifying_name unique index has a condition which was not satisfied for input rows with no identifyingtaxonomicname, causing any input row with NULL in this column to match *all* taxonconcepts with a NULL identifyingtaxonomicname. This uses ignore_cond()'s new support for constraints that did not fail at least once.
- 03:12 AM Revision 5449: sql_io.py: put_table(): ignore_cond(): Added support for constraints that did not fail at least once, and therefore should not be required to simplify to a non-false value. As part of this, only track the failed constraint in the errors table if it actually failed at least once based on the deleted row count or the `failed` param.
- 03:05 AM Revision 5448: sql_gen.py: map_expr(): Fixed bug where names were being replaced when they were inside another name. This occurred with combined names created by sql_io.into_table_name().
- 01:11 AM Revision 5447: sql.py: ConstraintException: message: Wrap condition in strings.as_tt()
- 01:06 AM Task #522 (New): fix deadlock when multiple testers are running simultaneously
- Commands running simultaneously:
* @make inputs/.TNRS/tnrs/test.by_col.xml verbosity=3@
* @make test by_col=1@
... - 12:30 AM Revision 5446: sql.py: run_query(): DuplicateKeyException: Also retrieve the index's condition using new index_cond()
- 12:28 AM Revision 5445: sql.py: Added index_cond()
- 12:11 AM Revision 5444: sql_io.py: put_table(): insert_into_pkeys(): Take a query as the param instead of sql.mk_select()'s params, to allow the caller to pass in any query without needing insert_into_pkeys() to manually pass through those args
10/11/2012
- 11:40 PM Revision 5443: sql.py: constraint_cond(): Fixed NotImplementedError message to apply to this function
- 09:53 PM Task #521 (Resolved): make place* tables use a structure similar to taxonconcept
- 09:36 PM Revision 5442: sql_io.py: put_table(): ignore_cond(): Log message: Replaced don't with do not so it wouldn't mess up syntax highlighting when viewing the log file in a text editor
- 09:07 PM Revision 5441: input.Makefile: Staging tables installation: Don't delete %/header.csv on error, because header.csv is a byproduct rather than the primary output and is created roughly atomically
- 08:40 PM Revision 5440: schemas/vegbien.sql: *_ancestor tables: Added descriptive comment that these are ancestor cross link tables
- 08:23 PM Revision 5439: csvs.py: sniff(): Support multi-char delims using \t, such as \t|\t used by NCBI. Support custom line suffixes, such as \t| used by NCBI.
- 08:18 PM Revision 5438: csvs.py: TsvReader.next(): Remove only the autodetected line ending instead of any standard line ending. Note that this requires all header override files to use the same line ending as the CSV they override, which is now the case.
- 08:15 PM Revision 5437: csvs.py: is_tsv(): Support multi-char delimiters by checking only the first char of the delimiter
- 08:12 PM Revision 5436: csvs.py: sniff(): Also autodetect the line ending
- 08:11 PM Revision 5435: csvs.py: sniff(): Also autodetect the line ending
- 08:02 PM Revision 5434: inputs/test_taxonomic_names/Taxon/+header.txt: Changed line endings to \r\n to match testNames.txt line endings. This will be necessary when the line ending is autodetected by csvs.sniff().
- 07:59 PM Revision 5433: csvs.py: TsvReader.next(): Renamed raw_contents var to line, since this is just the line with the ending removed
- 07:36 PM Revision 5432: strings.py: Replaced no longer used contains_any() with find_any(), which returns any found substring, or None if none of the substrings were found
- 07:22 PM Revision 5431: csvs.py: Modify csv.Dialect._validate() to ignore "delimiter must be a 1-character string" errors, in order to support multi-char delimiters used by TsvReader
- 07:21 PM Revision 5430: csvs.py: Modify csv.Dialect._validate() to ignore "delimiter must be a 1-character string" errors, in order to support multi-char delimiters used by TsvReader
- 06:58 PM Revision 5429: csvs.py: TsvReader: Use str.split() instead of csv.reader().next() to parse the row, for efficiency and to support multi-char delimiters. This is possible because the TSV dialect doesn't use CSV parsing features other than the delimiter and newline-escaping (which is handled separately).
- 06:02 PM Revision 5428: Regenerated vegbien.ERD exports
10/10/2012
- 11:43 AM Revision 5427: input.Makefile: $(exts): Added .dmp
- 11:43 AM Revision 5426: csvs.py: delims: Added |
- 11:28 AM Revision 5425: Removed no longer used inputs/.public/. Use inputs/.TNRS/ and inputs/.TNRS/tnrs/tnrs.make instead.
- 11:23 AM Revision 5424: README.TXT: Documentation: To import and scrub just the test taxonomic names: Added steps to restore the original DB when the test scrub is complete
- 11:22 AM Revision 5423: inputs/test_taxonomic_names/test_scrub: Also export the results to inputs/test_taxonomic_names/_scrub/
- 11:06 AM Revision 5422: inputs/test_taxonomic_names/test_scrub: Use regular for .. in loop with a list of what's being processed in each iteration (match_input_names, parse_accepted_names)
- 10:58 AM Revision 5421: inputs/.TNRS/tnrs/map.csv: Mapped Genus_score, Specific_epithet_score
- 10:56 AM Revision 5420: mappings/VegCore-VegBIEN.csv: Mapped matchedGenusFit_fraction, matchedSpeciesFit_fraction. Reordered canon_concept_fit_fraction _maxs in the order they would be used if _alt were being used instead.
- 10:52 AM Revision 5419: mappings/VegCore.csv: Added matchedSpeciesFit_fraction
- 10:47 AM Revision 5418: mappings/VegCore.csv: matchedFamilyFit_fraction: Source the "matched" to Family_matched, which is a closer fit than Name_matched. matchedGenusFit_fraction: Fixed Genus_matched source to use #detailed_download instead of #simple_download.
- 10:42 AM Revision 5417: mappings/VegCore.csv: Added matchedGenusFit_fraction
- 10:18 AM Revision 5416: README.TXT: Removed extra trailing whitespace
- 10:18 AM Revision 5415: README.TXT: Documentation: To import and scrub just the test taxonomic names: Use new inputs/test_taxonomic_names/test_scrub
- 10:17 AM Revision 5414: Added inputs/test_taxonomic_names/test_scrub
- 10:01 AM Revision 5413: schemas/vegbien.sql: taxonconcept: Renamed canon_taxonconcept_id to canon_concept_id to shorten the name, which is used often
- 09:45 AM Revision 5412: schemas/vegbien.sql: taxonconcept: Added taxonconcept_canon_concept_min_fit() trigger to remove the canon_concept_id link from insufficient matches. These occur when e.g. a name in another language is approximated to a latin name or when the input name is not a proper taxon but TNRS provides a best-guess match anyway.
- 09:42 AM Revision 5411: inputs/.TNRS/tnrs/map.csv: Mapped Family_score to new matchedFamilyFit_fraction
- 09:39 AM Revision 5410: mappings/VegCore-VegBIEN.csv: Use matchedFamilyFit_fraction as canon_concept_fit_fraction when greater than matchedTaxonFit_fraction, because if there is at least a matched family, there is a valid taxonconcept to attach to
- 09:39 AM Revision 5409: xml_func.py: Simplifying functions: Added _min, _max as passthroughs
- 09:34 AM Revision 5408: schemas/functions.sql: Added _max(), _min()
- 09:21 AM Revision 5407: mappings/VegCore.csv: Added matchedFamilyFit_fraction
- 09:04 AM Revision 5406: mappings/VegCore-VegBIEN.csv: Remapped matchedTaxonFit_fraction to the verbatim* taxonconcept, because this is actually for the verbatim* concept's fit to the matched concept, not the matched concept's fit to the accepted concept
- 08:59 AM Revision 5405: inputs/.TNRS/tnrs/map.csv: Restored *-prefixed output terms for unmapped terms that had initially been mapped to OMIT but could reasonably match to something in the future. Continue mapping Name_number to OMIT because it isn't globally unique (it identifies the name only within one TNRS batch).
- 08:45 AM Revision 5404: inputs/.TNRS/tnrs/map.csv: Mapped Overall_score to new matchedTaxonFit_fraction
- 08:44 AM Revision 5403: mappings/VegCore-VegBIEN.csv: Mapped matchedTaxonFit_fraction to _set_canon_taxonconcept(canon_concept_fit_fraction)
- 08:37 AM Revision 5402: mappings/VegCore.csv: Added matchedTaxonFit_fraction
- 08:20 AM Revision 5401: schemas/vegbien.sql: _set_canon_taxonconcept(): Also set the canon_concept_fit_fraction
- 08:10 AM Revision 5400: schemas/vegbien.sql: taxonconcept: Added canon_concept_fit_fraction to store the closeness of fit of the canon_concept
- 07:55 AM Revision 5399: schemas/vegbien.sql: taxonconcept: Renamed canon_taxonconcept_id to canon_concept_id to shorten the name, which is used often
- 07:10 AM Revision 5398: sql.py: mk_update(): in_place: Convert columns of type character varying to text so that they can be merge-joined with text columns. Note that these two types are equivalent but not aliases of one another, so the explicit type change is needed.
- 07:07 AM Revision 5397: sql_gen.py: Added canon_type()
- 06:52 AM Revision 5396: sql.py: mk_update(): in_place: Factored retrieval of column type out into separate statement for clarity
- 06:27 AM Revision 5395: schemas/functions.sql: _join*(): Fixed bug where was returning '' instead of NULL when only NULL inputs were provided, because array_to_string() always returns a non-NULL string. Functions must always return NULL in place of '' to ensure that empty strings do not find their way into VegBIEN, and to prevent inconsistencies between row-based and column-based import (row-based import folds empty strings to NULL while column-based import relies on having a clean input table).
- 06:10 AM Revision 5394: sql_io.py: cleanup_table(): Use sql.table_pkey_col() instead of sql.pkey_col() so that only an actual pkey column is removed from the list of columns to clean. This fixes a bug where the first column in the table was not cleaned up if there was no pkey. Note that this bug only affected newly re-created staging tables, because staging tables previously had a special row_num pkey column added if they did not already have a pkey. The row_num column is now added by column-based import instead.
- 05:51 AM Revision 5393: sql.py: table_pkey_col(): Raise a DoesNotExistException if the table has no pkey
- 05:23 AM Revision 5392: sql.py: pkey_col(): Call table_pkey_col() directly rather than via pkey_name(). pkey_name(): Call pkey_col() instead of table_pkey_col() now that pkey_col() calls table_pkey_col().
- 05:14 AM Revision 5391: sql.py: pkey_col(): Documented that if there is no pkey, returns the first column in the table
- 05:13 AM Revision 5390: sql.py: pkey_col(): Specify recover directly as a kw_arg because it's the only kw_arg passed to pkey_name()
- 05:10 AM Revision 5389: sql.py: Added table_pkey_col() and use it in pkey_name()
- 05:01 AM Revision 5388: sql.py: Renamed pkey() to pkey_name()
- 04:45 AM Revision 5387: sql.py: Renamed pkey_col_() to pkey_col()
- 04:43 AM Revision 5386: sql.py: Removed no longer used pkey_col
- 04:43 AM Revision 5385: db_xml.py: cleanup_table(): Inline sql.pkey_col ('row_num') because this is the only place it's used
- 04:37 AM Revision 5384: cleanup_table(): Use new sql.table_cols() instead of sql.table_col_names()
- 04:36 AM Revision 5383: sql.py: Added table_cols()
- 04:16 AM Revision 5382: db_xml.py: put(): Fixed bug where needed to avoid truncating the pkeys_loc table, in case it's the same as one of the in_tables. This occurs now that sql_io.put_table() passes through the actual input column instead of the joined-together input table's column when ignoring all rows.
- 03:33 AM Revision 5381: sql_io.py: put_table(): Resolving default value column: If ignoring all rows, use input cols directly instead of cols from joined-together input table. In addition to being simpler, this prevents the returned column's name from growing longer and longer as each iteration prepends its input table table name to the default value column name.
- 03:07 AM Revision 5380: sql_io.py: put_table(): Moved changing the table of the default value column from Resolving the default value column to Setting pkeys of missing rows, because the table change is only needed in this section
- 03:04 AM Revision 5379: sql_io.py: put_table(): Resolving default value column: Always call sql_gen.remove_col_rename() because it will just pass the value through if it's not a column
- 02:41 AM Revision 5378: sql_gen.py: simplify_parens(): Removed extra simplify_parens() at end because it is done in the final iteration that performs no other replacements, so it is not necessary to also do it explicitly
- 02:30 AM Revision 5377: sql_io.py: put_table(): Replaced limit_ref integer with ignore_all_ref boolean, because it is no longer used as a select statement limit
- 02:29 AM Revision 5376: sql_io.py: put_table(): remove_all_rows(): Corrected "just create an empty pkeys table" comment to "just return the default value column"
- 02:27 AM Revision 5375: sql_io.py: put_table(): mk_main_select(): Removed setting limit to limit_ref[0], because an empty pkeys table is no longer created when ignoring all rows
- 02:19 AM Revision 5374: sql_io.py: put_table(): Setting pkeys of missing rows: Removed "limit_ref[0] == 0" check because this code is never reached in that case
- 02:16 AM Revision 5373: sql_io.py: put_table(): Ignoring all rows for unrecoverable errors: Even in multi-row mode, just return whatever the default value or column was, instead of creating an output table containing the default value filled in for every row. This also assists the optimization to skip empty levels of taxonconcepts, because it folds the empty level to that level's parent level rather than creating a whole new temp table with ultimately the same contents.
- 01:57 AM Revision 5372: sql_gen.py: not_false_re, not_true_re: Appended \b to ensure that true/false is only matched as a single word
- 01:56 AM Revision 5371: sql_gen.py: simplify_expr(): Also simplify "NOT false" to true
- 01:53 AM Revision 5370: sql_gen.py: simplify_expr(): Also simplify "NOT true" to false
- 01:24 AM Revision 5369: sql_io.py: put_table(): ignore_cond(): Changed "Ignoring rows where" message with the negated (filter-out) condition to "Ignoring rows that don't satisfy" with the filter condition for clarity
- 01:22 AM Revision 5368: sql_io.py: put_table(): ignore_cond(): If cond simplifies to false, remove all rows instead of filtering out individual rows which will all be filtered out. This optimization should improve import times of tables, such as taxonconcept, which use a check constraint instead of NOT NULL constraints to prevent empty rows. The taxonomic schema refactoring caused the creation of many more levels of taxonconcepts, many of which (such as variety, forma, cultivar) are empty for most datasources, so this optimization should also reduce overall import times for datasources that have any empty levels of taxonconcept. Note that this optimization is only possible now that sql_gen.simplify_expr() is able to simplify all the way to a single boolean value for the taxonconcept_required_key constraint.
- 12:55 AM Revision 5367: Moved expression transforming functions from sql.py to sql_gen.py because they do not manipulate an actual database and merely generate SQL
- 12:38 AM Revision 5366: sql.py: Added true_expr, false_expr and use them where their values are used
- 12:34 AM Revision 5365: sql.py: simplify_expr(): Also simplify "AND true" expressions
- 12:30 AM Revision 5364: sql.py: simplify_expr(): Also simplify "AND false" expressions
- 12:19 AM Revision 5363: sql.py: Added atom_re and use it in simplify_parens()
- 12:19 AM Revision 5362: sql.py: Added or_re and use it in simplify_expr()
- 12:18 AM Revision 5361: sql.py: logic_op_re(): Added expr_re param for an expr on the other side of the operator
10/09/2012
- 11:54 PM Revision 5360: sql.py: simplify_parens(): Use bool_re
- 11:54 PM Revision 5359: sql.py: Removed no longer needed paren_re()
- 11:53 PM Revision 5358: sql.py: true_re, false_re: Removed no longer needed paren_re() because simplify_parens() now handles this
- 11:50 PM Revision 5357: sql.py: simplify_expr(): Removed final simplify_parens() because this is now done by simplify_recursive()
- 11:49 PM Revision 5356: sql.py: simplify_expr(): Use new simplify_recursive(). This also fixes a bug where some logic expressions are not simplified because of extra parens.
- 11:48 PM Revision 5355: sql.py: Added simplify_recursive()
- 11:31 PM Revision 5354: sql.py: simplify_parens(): Also remove parens around true and false
- 11:26 PM Revision 5353: regexp.py: sub_nested(): Use new sub_recursive()
- 11:25 PM Revision 5352: regexp.py: Added sub_recursive()
- 11:21 PM Revision 5351: sql.py: simplify_expr(): Use new simplify_parens()
- 11:20 PM Revision 5350: sql.py: Added simplify_parens()
- 11:14 PM Revision 5349: sql.py: simplify_expr(): Use new regexp.sub_nested()
- 11:14 PM Revision 5348: Added regexp.py
- 10:46 PM Revision 5347: sql.py: simplify_expr(): Use new logic_op_re()
- 10:46 PM Revision 5346: sql.py: Added logic_op_re()
- 10:40 PM Revision 5345: sql.py: bool_re: Use new true_re, false_re
- 10:40 PM Revision 5344: sql.py: Added true_re, false_re
- 10:37 PM Revision 5343: sql.py: bool_re: Use new paren_re()
- 10:36 PM Revision 5342: sql.py: bool_re: Use new paren_re()
- 10:36 PM Revision 5341: sql.py: Added paren_re()
- 10:31 PM Revision 5340: sql.py: simplify_expr(): Combined replacements of bool_re+' OR ' with the value in either order into one replacement
- 10:27 PM Revision 5339: mappings/VegCore-VegBIEN.csv: verbatim* taxonconcept: Don't store Name_submitted in taxonomicnamewithauthor in addition to identifyingtaxonomicname, because the fields other than identifyingtaxonomicname are meant to store parsed values rather than raw, unscrubbed values and TNRS does not directly provide a concatenated taxonomic name with author
- 10:23 PM Revision 5338: mappings/VegCore-VegBIEN.csv: verbatim* taxonconcept: Don't create hierarchy of parent taxonconcepts, because the parsed names (rather than the names for the matched taxonconcept) are from the input taxonomic name, rather than from the official tree of life used by TNRS. Otherwise, if a taxonomic name provides e.g. no family (common), a separate genus taxonconcept would have been created with no parent_id, which would not compare equal to the matched taxonconcept's genus *with* a parent_id. Continue to store the parsed family, genus, species in the family, genus, species cached fields, because the parsed family is often different from the matched taxonconcept's family when e.g. no family is provided in the taxonomic name.
- 10:16 PM Revision 5337: sql.py: Renamed table_cols() to table_col_names() for clarity, because it does not return sql_gen.Col objects
- 10:12 PM Revision 5336: inputs/.TNRS/tnrs/test.xml.ref: Accepted new inserted row count. The change is most likely from several revisions back, but the cause of the change is unknown (it is not due to the updated TNRS.tnrs table, which is still sorted with the same rows first).
- 09:09 PM Revision 5335: sql_gen.py: is_text_col(): Use new is_text_type()
- 09:09 PM Revision 5334: sql_gen.py: Added is_text_type()
- 09:05 PM Revision 5333: sql_gen.py: ensure_not_null(): Documented that NULL has no type, hence the NoUnderlyingTableException being re-raised
- 09:04 PM Revision 5332: sql_gen.py: ensure_not_null(): Just store the column type in col_type, instead of storing typed_col and using typed_col.type, now that other info in typed_col is no longer needed
- 09:02 PM Revision 5331: sql_gen.py: ensure_not_null(): Use is_nullable() instead of determining nullability itself, for clarity
- 08:59 PM Revision 5330: sql_gen.py: is_nullable(): Fixed bug where non-columns could not be sent to db.col_info()
- 08:53 PM Revision 5329: sql_gen.py: ensure_not_null(): Always remove_col_rename() the column to ensure that it is acceptable by helper functions like is_nullable()
- 08:11 PM Revision 5328: lib/PostgreSQL-MySQL.csv: COMMENT statement: Fixed bug where ending ; could match only when preceded by ' and followed by a newline, to avoid matching ; embedded in the comment
- 08:07 PM Revision 5327: schemas/vegbien.sql: taxonconcept: family, genus, species comments: Changed "scoping" to "identifying" for clarity
- 08:06 PM Revision 5326: schemas/vegbien.sql: taxonconcept: family, genus, species: Added comment that each is a cached field for easy querying and the scoping version of it is stored in the chain of parent_id ancestors
- 08:03 PM Revision 5325: schemas/vegbien.sql: taxonconcept: taxonconcept_unique: Removed family, genus, species because these are now just cached fields for analytical_db_view rather than scoping fields. The scoping versions of these fields are stored in the chain of parent_id ancestors.
- 07:42 PM Revision 5324: tnrs_db: Moved "Processing # taxonconcepts" log message to before waiting or exiting if no taxonconcepts left, so that it would be printed right after the query is run and say that no taxonconcepts were found
- 07:39 PM Revision 5323: tnrs_db: Updated comments and log messages for schema changes
- 07:33 PM Revision 5322: tnrs_db: Updated query for schema changes
- 07:33 PM Revision 5321: README.TXT: Schema changes: files to update with renamings: Added bin/tnrs_db
- 07:25 PM Revision 5320: inputs/import.stats.xls: Updated import times
- 07:04 PM Revision 5319: README.TXT: Data import: Changed `inputs/*/*/logs` to `inputs/{.,}*/*/logs` to also include the TNRS names import log
10/08/2012
- 09:58 PM Revision 5318: import_all: Added commands to import TNRS names so the user doesn't have to do this manually
- 09:55 PM Revision 5317: sql.py: map_expr(): Fixed bug where names were being matched inside punctuated names replaced in previous calls of map_expr()
- 09:45 PM Revision 5316: schemas/vegbien.sql: party: party_required_key: Only allow NULL organizationname if party is not a root party (i.e. creator_id != party_id)
- 09:39 PM Revision 5315: mappings/VegCore-VegBIEN.csv: Mapped to new taxonconcept.creationdate
- 09:37 PM Revision 5314: schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Added creationdate as an allowable minimum field when parent_id (containing the associated hierarchical concept) is specified
- 09:30 PM Revision 5313: schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Removed family and genus because these are now cached fields only, and are not used for scoping a taxonconcept. Instead, *taxonomicname and taxonname+parent_id are used for this purpose. This removes several leaf taxonconcepts with insufficient scoping information to create a taxonconcept separate from the main tree. With the upcoming population of creationdate, some of these taxonconcepts will reappear due to the date's additional distinguishing information.
- 09:16 PM Revision 5312: schemas/vegbien.sql: taxonconcept: Added creationdate (the date the taxonconcept was created or defined), and include it in the taxonconcept_unique unique index
- 09:05 PM Revision 5311: schemas/vegbien.sql: taxonconcept: Added comment with the definition of a taxon: "a group of one (or more) populations of organism(s), which a taxonomist adjudges to be a unit" (http://en.wikipedia.org/wiki/Taxon). This is useful in clarifying that our taxon concepts are intended to serve a similar purpose, by storing one person's defined taxon.
- 08:58 PM Revision 5310: schemas/vegbien.sql: taxonconcept: taxonconcept_required_key: Removed family and genus because these are now cached fields only, and are not used for scoping a taxonconcept. Instead, *taxonomicname and taxonname+parent_id are used for this purpose.
- 08:54 PM Revision 5309: schemas/vegbien.sql: taxonconcept: Moved identifyingtaxonomicname near other full-taxonomic-name-related fields, after the fields that contain just the current level's component of the full name
- 08:48 PM Revision 5308: schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id: Changed four-level hierarchy to use "parsed concept" and "matched concept" instead of concatenated and parsed, because the directly-parsed name components actually go in level 2 of the hierarchy (the TNRS input name), while the name components based on the matched taxon concept go in level 3
- 08:44 PM Revision 5307: schemas/vegbien.sql: taxonconcept.parent_id: Documented that while a taxon *name* may have multiple parents, a taxon *concept* has only one, based on the creator's opinion of where that taxonconcept goes in the taxonomic hierarchy
- 08:38 PM Revision 5306: mappings/VegCore-VegBIEN.csv: taxonconcept: Moved infraspecific taxonconcept to its own level, rather than combining it with the level that contains the full taxonomic name and author (as well as any morphospecies), for consistency with the storage of other ranked taxonomic name components, which each get their own taxonconcept. The infraspecific taxon concept is general to all parties making idenfitications (within a datasource), while the concatenated name and author and any morphospecies are specific to the person who defined the taxonconcept used by a taxondetermination.
- 08:05 PM Revision 5305: schemas/vegbien.sql: taxonconcept: Removed no longer used higher- and infraspecific taxonomic rank fields because these terms are now stored in their own taxonconcepts. family, genus, and species have not been removed because these are used to cache names of parent taxa for fast access by analytical_db_view.
- 07:57 PM Revision 5304: schemas/vegbien.sql: analytical_db_view: Changed taxonMorphospecies to use taxonconcept.taxonname, where any morphospecies is now stored
- 07:53 PM Revision 5303: mappings/VegCore-VegBIEN.csv: infraspecific taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts, or in the lowest-level taxonconcept as the taxonname and rank
- 07:43 PM Revision 5302: mappings/VegCore-VegBIEN.csv: higher-level taxonomic terms: Removed mappings to first-class taxonconcept fields because these terms are now stored in their own taxonconcepts
- 07:41 PM Revision 5301: schemas/vegbien.sql: taxonconcept: Merged taxonconcept_unique_within_creator_by_name unique index into taxonconcept_unique_within_parent, placed parent_id first, and removed index condition, so that this index can be used as a lookup index by taxonconcept_update_ancestors() (which requires no index condition in order to apply to *all* taxonconcepts) in addition to as a unique index. Note that an index condition should not be necessary for the index's uniquifying task, because if a set of taxonconcepts provides only the identifyingtaxonomicname, that should collide in the taxonconcept_unique_within_creator_by_identifying_name unique index before this index collides. This assumes that the collision order when multiple indexes collide is alphabetical by the index name.
- 07:16 PM Task #486: add unit-conversion mechanism
- All applicable VegBIEN fields have unit suffixes. Most corresponding VegCore terms also have unit suffixes.
- 07:15 PM Task #499 (Resolved): map example terms into the taxonomic schema
- See "README.TXT":https://projects.nceas.ucsb.edu/nceas/projects/bien/repository/entry/README.TXT section "To import a...
- 06:38 PM Revision 5300: schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Also allow a taxonconcept to have just an author when it has a parent_id, so that an author can uniquely identify a taxon within a more general taxon, such as a species name, that has no author
- 06:22 PM Revision 5299: strings.py: concat(): Fixed bug where end index of returned str0 portion would wrap around to a negative number if str1 itself was too long, causing incorrect truncation
- 05:44 PM Revision 5298: schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_parent to taxonconcept_unique because the index does not apply only to taxonconcepts with a parent, and because it's the primary unique index for taxonconcept
- 05:42 PM Revision 5297: schemas/vegbien.sql: taxonconcept: Renamed taxonconcept_unique_within_creator_by_identifying_name to taxonconcept_0_unique_identifying_name to ensure that it is always applied before taxonconcept_unique_within_parent if both collide
- 05:36 PM Revision 5296: schemas/vegbien.sql: taxonconcept: Merged taxonconcept_unique_within_creator_by_name unique index into taxonconcept_unique_within_parent, placed parent_id first, and removed index condition, so that this index can be used as a lookup index by taxonconcept_update_ancestors() (which requires no index condition in order to apply to *all* taxonconcepts) in addition to as a unique index. Note that an index condition should not be necessary for the index's uniquifying task, because if a set of taxonconcepts provides only the identifyingtaxonomicname, that should collide in the taxonconcept_unique_within_creator_by_identifying_name unique index before this index collides. This assumes that the collision order when multiple indexes collide is alphabetical by the index name.
- 04:47 PM Revision 5295: mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree for taxonconcepts created from original*, verbatim*, and accepted* taxonomic terms
- 04:35 PM Revision 5294: mappings/VegCore-VegBIEN.csv: taxonconcepts: Also create the taxonconcept tree if datasource provided separated components of the taxonomic name and/or its own tree of life with higher classifications. This enables storing the datasource's own tree of life to supplement any official tree (TROPICOS, USDA, etc.).
- 04:25 PM Revision 5293: mappings/VegCore-VegBIEN.csv: taxonconcept tree: Don't map infraspecificEpithet+taxonRank to a taxonconcept in the tree of parent concepts because it has already been mapped to the primary, lowest-level taxonconcept
- 04:00 PM Revision 5292: schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Fixed bug where index filter overlapped with taxonconcept_unique_within_parent's index filter, causing these unique indexes to sometimes both apply at the same time and prevent column-based import from correctly choosing which index to use for each taxonconcept import
- 01:15 PM Revision 5291: schemas/vegbien.ERD.mwb: Fixed lines
- 01:02 PM Revision 5290: schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id comment: Changed comment to use "concept" rather than "name" where applicable. Documented that a synonym between taxonconcepts of different sources is indicated by choosing one taxonconcept to be authoritative and pointing the other taxonconcept to it using this field.
10/05/2012
- 10:52 PM Revision 5289: sql_io.py: put_table(): Resolving default value column: Fixed bug where the default value col needed to have its table changed from in_table to full_in_table if it's a table column, and needed to have any column rename removed if it's a literal value
- 10:29 PM Revision 5288: Regenerated vegbien.ERD exports
- 10:28 PM Revision 5287: schemas/vegbien.ERD.mwb: Fixed lines
- 10:23 PM Revision 5286: schemas/vegbien.sql: Renamed plant* taxonomic tables -> taxon*, as part of the taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
- 10:15 PM Revision 5285: schemas/vegbien.ERD.mwb: Rearranged to fit more of location table on the diagram, using the newly available space from taxon
- 10:00 PM Revision 5284: schemas/vegbien.ERD.mwb: Fixed lines
- 09:59 PM Revision 5283: schemas/tree_cross-links.sql: Synced with schema, updating with new table names
- 09:54 PM Revision 5282: schemas/vegbien.sql: Removed no longer used taxon table. Use taxonconcept instead.
- 09:51 PM Revision 5281: schemas/vegbien.sql: taxonconcept.taxonname: comment: Stated that this is the name of the taxon within its parent taxon
- 09:48 PM Revision 5280: schemas/vegbien.sql: taxonconcept: comment: Removed no longer accurate comment that an accepted taxonconcept points to the identified taxon in the tree of life, because it *is* the identified taxon in the tree of life
- 09:39 PM Revision 5279: schemas/filter_ERD.csv: Changed the table with the visible fkey from plant* to taxon* to be plantstatus rather than plantusage, since it contains more core fields
- 09:25 PM Revision 5278: schemas/vegbien.sql: taxonconcept: Removed taxon_id, since taxonconcept now contains all the information needed to represent a taxonomic hierarchy, including both conceptual and nomenclature information
- 09:20 PM Revision 5277: schemas/vegbien.sql: plantusage: Point just to taxonconcept instead of both to taxonconcept and taxon
- 09:16 PM Revision 5276: schemas/vegbien.sql: taxonconcept: rank, verbatimrank comments: Added info from corresponding fields in taxon that also applies to taxonconcept
- 09:14 PM Revision 5275: schemas/vegbien.sql: taxonconcept: comment: Added info from taxon that also applies to taxonconcept
- 09:06 PM Revision 5274: schemas/vegbien.sql: Added taxonconcept_ancestor cross-link table
- 08:40 PM Revision 5273: schemas/vegbien.sql: taxonconcept: Added description field
- 08:38 PM Revision 5272: mappings/VegCore-VegBIEN.csv: Remapped taxon hierarchy for accepted taxonconcepts to taxonconcept parent_id hierarchy
- 08:12 PM Revision 5271: schemas/vegbien.sql: Fixed bug where taxonconcept.parent_id was missing a foreign key constraint
- 08:10 PM Revision 5270: schemas/vegbien.sql: taxonconcept: Changed instructions for including a taxon name at a rank with no explicit column to create a parent taxonconcept for it and point to it using parent_id instead of using otherranks. Removed no longer used otherranks field.
- 08:05 PM Revision 5269: schemas/vegbien.sql: taxonconcept: taxonconcept_required_key check constraint: Added taxonname
- 07:58 PM Revision 5268: schemas/vegbien.sql: taxonconcept: taxonconcept_unique_within_creator_by_name unique index: Removed duplicate entry for creator_id
- 07:57 PM Revision 5267: schemas/vegbien.sql: taxonconcept: Added parent_id to point to the parent taxonconcept
- 07:56 PM Revision 5266: sql_gen.py: null_sentinels: Added 'unknown' for taxonrank
- 07:44 PM Revision 5265: schemas/vegbien.sql: taxonrank: Added 'unknown'
- 07:30 PM Revision 5264: mappings/VegCore-VegBIEN.csv: Also map *taxonRank to taxonconcept.rank, so that if it's in the taxonrank enum, it will automatically populate this field
- 07:14 PM Revision 5263: mappings/VegCore-VegBIEN.csv: Remapped *infraspecificEpithet to new taxonconcept.taxonname rather than placing it in subspecies prefixed with the taxonRank, because it isn't necessarily the subspecies and because taxonname is defined to contain the lowest-rank portion of the taxonomic name. Note that when both morphospecies and infraspecificEpithet are provided, infraspecificEpithet takes priority for the taxonname field, because if TNRS leaves unmatched terms (which are tentatively mapped to morphospecies) but also matches an infraspecificEpithet, then the unmatched terms can't be for a morphospecies (because an infraspecificEpithet and therefore also a specificEpithet was matched, so the species is definite and formally named).
- 06:45 PM Revision 5262: schemas/vegbien.sql: taxonconcept: Renamed morphospecies to taxonname since it's used in the same way as taxon.taxonname: to store the lowest-rank portion of the taxonomic name, such as the morphospecies suffix
- 06:21 PM Revision 5261: inputs/.TNRS/tnrs/map.csv: Mapped *_matched terms that are both matched in the input name and which correspond to the matched taxonconcept (Genus_matched, Specific_epithet_matched, etc.) to both the input and matched taxonconcepts
- 06:09 PM Revision 5260: inputs/.TNRS/tnrs/map.csv: Mapped terms matched in the original string (rather than deduced from the matched taxonconcept) to new verbatim* taxonomic terms
- 06:03 PM Revision 5259: mappings/VegCore-VegBIEN.csv: Mapped verbatim* taxonomic terms to the TNRS input taxonconcept
- 05:48 PM Revision 5258: mappings/VegCore-VegBIEN.csv: TNRS input taxonconcept: Split single _if statement controlling where morphospecies goes into two _if statements for each case, so that other verbatim* terms don't need to have an _if statement in their mapping to the input taxonconcept
- 05:29 PM Revision 5257: mappings/VegCore.csv: Added back verbatim* taxonomic terms, which will now be used for the TNRS input taxonconcept. Note that they will have a different meaning than the original* taxonomic terms that they were renamed to in r5062.
- 05:22 PM Revision 5256: mappings/VegCore-VegBIEN.csv: In TNRS mode, remapped morphospecies (Unmatched_terms) to the input name's taxonconcept, because this does not relate to the matched taxon concept
- 05:12 PM Revision 5255: mappings/VegCore-VegBIEN.csv: TNRS-only mappings: Switch them on when verbatimScientificNameWithAuthorship is provided rather than when acceptedScientificNameWithAuthorship is provided, because it's the presence of a separate TNRS input name that really determines when TNRS is being mapped
- 05:07 PM Revision 5254: Makefiles: .last_cleanup targets: Also make the file that's being cleaned up .PRECIOUS so it doesn't get deleted if the .last_cleanup target has an error
- 05:04 PM Revision 5253: Makefiles: .last_cleanup targets: Make each individual target .PRECIOUS (don't delete on error) because just making %.last_cleanup precious doesn't seem to prevent deletion
10/04/2012
- 11:19 PM Revision 5252: mappings/VegCore-VegBIEN.csv: Mapped *taxonRank to new taxonconcept.verbatimrank
- 11:15 PM Revision 5251: schemas/vegbien.sql: taxonconcept: Added rank, verbatimrank analogous to those fields in taxon
- 09:59 PM Revision 5250: Makefiles: Don't delete %.last_cleanup on error because it's a mod time record rather than a generated file, and so that it's left at the last successful cleanup time when a cleanup operation is cancelled
- 09:52 PM Revision 5249: input.Makefile: Maps building: %/.map.csv.last_cleanup: Removed no longer accurate comment about mappings being autoremoved
- 09:34 PM Revision 5248: inputs/.TNRS/tnrs/map.csv: Remapped Name_submitted to new verbatimScientificNameWithAuthorship to create an additional level of taxonconcept for the concatenated (TNRS input) name separate from the parsed (TNRS output) name
- 09:33 PM Revision 5247: mappings/VegCore-VegBIEN.csv: Mapped verbatimScientificNameWithAuthorship as an additional level of taxonconcept for the concatenated (TNRS input) name separate from the parsed (TNRS output) name
- 09:26 PM Revision 5246: schemas/vegbien.sql: taxonconcept.canon_taxonconcept_id: comment: Changed three-level hierarchy to four-level hierarchy which separates the concatenated (TNRS input) name from the parsed (TNRS output) name
- 09:22 PM Revision 5245: mappings/VegCore.csv: Added back verbatimScientificNameWithAuthorship, which will now be used to store the TNRS input name
- 08:45 PM Revision 5244: schemas/filter_ERD.csv: Removed no longer used table taxonscope
- 08:32 PM Revision 5243: schemas/vegbien.sql: voucher: Removed accessioncode because this table has no sourceaccessioncode which it would be generated from (it just links a taxonoccurrence to a vouchering specimenreplicate)
- 08:26 PM Revision 5242: schemas/vegbien.sql: Renamed datasource_id to creator_id so it can apply generally to any entity (such as a person), not just an aggregated datasource. This also enables taxonconcept.datasource_id to merge with creator_id, which now serves the same purpose.
- 08:05 PM Revision 5241: schemas/vegbien.sql: taxonconcept: Renamed definer_id to creator_id to allow merging with datasource_id when datasource_id is renamed to creator_id
- 07:50 PM Revision 5240: mappings/VegCore-VegBIEN.csv: Populated new taxonconcept.definer_id from identifiedBy, or when no identifiedBy is specified, from the datasource itself (using _simplifyPath:[next=datasource_id])
- 07:43 PM Revision 5239: sql_io.py: put_table(): Resolve default value column *after* the main loop (inserts and selects), so that the default value column can refer to an output column that is not in the original mapping but is added to the mapping from a col_defaults entry. This requires deferring the "Missing mapping for NOT NULL column" warning until the default value column is resolved, and including all columns in the full_in_table since the default value input column is not yet known.
- 06:59 PM Revision 5238: schemas/vegbien.sql: taxonconcept: comment: Changed definition to "A taxon concept defined by an entity" to correspond with the table's new name and usage
- 06:51 PM Revision 5237: mappings/VegCore-VegBIEN.csv: Fixed bug where needed to set datasource_id=0 on the TNRS party (which concatenated names/TNRS inputs are owned by) in order to make it a datasource (a root party)
- 06:44 PM Revision 5236: schemas/vegbien.sql: party: Fixed bug where needed separate unique index for roots (datasources), whose organizationnames must be globally unique rather than unique within a datasource
- 06:28 PM Revision 5235: schemas/vegbien.sql: taxonconcept: Renamed concept_reference_id to definer_id because this is a clearer name and because this will allow merging with datasource_id, which serves the same purpose
- 06:15 PM Revision 5234: schemas/vegbien.sql: party: Made it datasource-scoped. Since this creates a recursive fkey, a datasource (a root party) should point to itself in this field, which will happen automatically by setting it to the special value 0.
- 05:51 PM Revision 5233: lib/PostgreSQL-MySQL.csv: Changed translation of fulltext to quote the identifier instead of appending characters to make it not a reserved word
- 05:36 PM Revision 5232: schemas/vegbien.sql: taxonconcept: Moved concept_reference_id to the top of the table because it is now a key scoping field
- 05:30 PM Revision 5231: schemas/vegbien.sql: concept_reference_id: Made it an fkey to party instead of taxonscope, because this is now the entity that defined the taxon concept, and is no longer specific to morphospecies. Removed no longer used table taxonscope.
- 05:13 PM Revision 5230: schemas/vegbien.sql: taxonconcept: Documented that it's equivalent to VegBank's plantConcept table
- 04:56 PM Revision 5229: schemas/filter_ERD.csv: taxonconcept inward fkeys: Removed not applicable taxon filtered table, since the fkey points in the opposite direction and thus is not part of this filter
- 04:52 PM Revision 5228: schemas/vegbien.sql: taxonconcept: Renamed scope_id -> concept_reference_id as part of taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
- 04:47 PM Revision 5227: README.TXT: Schema changes: Moved "update the following files with any renamings" out of "Sync ERD with vegbien.sql schema" because this is needed for any schema changes, not just as part of syncing the ERD
- 04:42 PM Revision 5226: README.TXT: Schema changes: Added Refactoring tips section with steps to rename a table and a column
- 04:23 PM Revision 5225: schemas/vegbien.sql: Renamed taxonpath -> taxonconcept as part of taxonomic schema refactoring at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2012-10-03_conference_call#Taxonomic-schema-refactoring>
- 04:17 PM Revision 5224: README.TXT: Schema changes: Syncing ERD with vegbien.sql schema: Added step to update mappings/VegCore-VegBIEN.csv with any renamings
- 04:10 PM Revision 5223: README.TXT: Schema changes: Syncing ERD with vegbien.sql schema: Added step to update schemas/filter_ERD.csv with any table renamings
- 03:58 PM Revision 5222: inputs/import.stats.xls: Updated import times. This now includes the half-hour-long pre-import of the TNRS taxonomic names (which the datasources then match up with), as well as the concatenation of the datasource's taxonomic name components to create or match up with the TNRS input name.
- 03:54 PM Revision 5221: README.TXT: Data import: make backups/TNRS.backup/restore: Run it in the background because it takes awhile
- 03:53 PM Revision 5220: README.TXT: Data import: Added steps to sync the TNRS schema to the latest version on vegbiendev
- 03:38 PM Revision 5219: README.TXT: Data import: make inputs/download-logs: Added tnrs_log=1 so the TNRS daemon log is downloaded as well
10/03/2012
- 01:55 PM Revision 5218: Added inputs/test_taxonomic_names/Taxon/testNames.txt since this is test data, and thus can be under version control
- 01:55 PM Revision 5217: Added inputs/test_taxonomic_names/README.TXT with Bob's comments
- 01:41 PM Revision 5216: schemas/vegbien.sql: taxonpath.taxon_id: Changed comment to indicate that this used for parsed, not just accepted names. Parsed names have been standardized by TNRS but may be synonyms.
- 01:27 PM Revision 5215: README.TXT: Documentation: To import and scrub just the test taxonomic names: Added `yes|` before make schemas/public/reinstall so the user isn't prompted to confirm the reinstallation a second time, and can just copy and paste the set of 5 commands directly into the terminal
- 01:11 PM Revision 5214: tnrs_db: Made wait option default to off to facilitate running tnrs_db by itself, rather than as part of an import
- 01:08 PM Revision 5213: tnrs_db: Added wait option to have tnrs_db exit as soon as no more names are available. This is useful for running tnrs_db when there is no concurrent import running, and therefore no need to wait for new data.
- 01:00 PM Revision 5212: tnrs_db: Fixed the time of the "Waited" message so it that the total_pause (containing the next wait) would be incremented *after* the message was displayed. Split the "Waited" and "Waiting" messages into two separate messages.
- 12:51 PM Revision 5211: README.TXT: Data import: Added steps to back up the TNRS cache, since it takes a long time to recreate. This also enables syncing it with a local machine when `make backups/download` is run.
- 12:47 PM Revision 5210: README.TXT: Documentation: Added instructions to import and scrub just the test taxonomic names
- 12:41 PM Revision 5209: input.Makefile: Staging tables installation: uninstall: For the TNRS datasource, prompt the user before deleting the schema, since the data in it is not easily reconstructible from a flat file
- 11:41 AM Revision 5208: sql.py: map_expr(): When matching without quotes, support names containing spaces by not matching words when preceded or followed by quotes
- 11:24 AM Revision 5207: sql.py: Expressions: bool_re: Also match parentheses surrounding the boolean value
- 08:57 AM Revision 5206: README.TXT: Data import: import_all: Don't run with & because this prevents the created jobs from being owned by the calling shell. Instead, import the TNRS names as a separate backgrounded step and wait for it to finish before starting import_all. Removed TNRS import steps from import_all since these are now invoked separately.
- 08:35 AM Revision 5205: README.TXT: Data import: Run import_all in the background, because it needs to import all the taxonomic names synchronously before it can start the datasource import in the background
- 08:19 AM Revision 5204: Regenerated vegbien.ERD exports
- 08:14 AM Revision 5203: inputs/.TNRS/tnrs/map.csv: Mapped Unmatched_terms to morphospecies because the morphospecies is what's left once named ranks are matched
- 08:11 AM Revision 5202: mappings/VegCore-VegBIEN.csv: Mapped morphospecies
- 08:08 AM Revision 5201: mappings/VegCore.csv: Added morphospecies
- 08:04 AM Revision 5200: schemas/vegbien.sql: taxonpath: Added morphospecies
- 07:43 AM Revision 5199: inputs/.TNRS/tnrs/test.xml.ref: Updated for latest TNRS output
- 06:40 AM Revision 5198: inputs/.TNRS/tnrs/map.csv: Infraspecific_rank_2, Infraspecific_epithet_2_*: Mapped to UNUSED because they do not appear to be provided by TNRS (it just puts additional infraspecific names in Unmatched_terms)
- 06:34 AM Revision 5197: inputs/.TNRS/tnrs/map.csv: Omit Infraspecific_rank because Name_matched_rank contains the unabbreviated rank and is provided more often
- 06:29 AM Revision 5196: mappings/VegCore-VegBIEN.csv: Also map TNRS-parsed infraspecificEpithet (Infraspecific_epithet_matched) to taxon at the infraspecies rank
- 06:07 AM Revision 5195: mappings/VegCore-VegBIEN.csv: Also map TNRS-parsed taxonomic ranks to the tree of life in the taxon table
- 05:18 AM Revision 5194: schemas/vegbien.sql: taxon: Added comment that this table stores the tree of life
- 05:00 AM Revision 5193: mappings/VegCore-VegBIEN.csv: accepted taxonomic terms: Use new _set_canon_taxonpath() to set the canon_taxonpath_id *after* the taxonpath has been inserted, so that if the taxonpath is an accepted name (scrubs to itself), it will link up to the just-inserted taxonpath with the taxonomic ranks parsed out, rather than to a new taxonpath containing only the few taxonomic ranks of the accepted name that TNRS provides. In particular, this (together with the tnrs_accepted_names sorting index on TNRS.tnrs) ensures that an accepted name is imported with its genus and species parsed out by TNRS instead of concatenated together in the Accepted_name_species field (genus+species). This enables the individual taxonomic ranks to be used in constructing the leaves of the tree of life (the taxon table).
- 04:50 AM Revision 5192: sql_io.py: put_table(): Fixed bug where row_ct_ref was incorrectly being incremented when the iteration is a function call. This bug only occurred in row-based mode, because the DB cursor for a function call is not stored in column-based mode.
- 04:30 AM Revision 5191: inputs/.TNRS/tnrs/map.csv: Use Name_matched_author/Name_matched_accepted_family instead of Author_matched/Family_matched because these fields are provided more often, due to being determined from the matched name itself rather than from the original string. This helps to fill in as many fields as possible. For accepted names (which scrub to themselves), this is especially important, because it adds the accepted name's family, which is not present in the input taxonomic name.
- 03:58 AM Revision 5190: xml_func.py: process(): Fixed bug where need to preserve complex functions that have unevaluated XML nodes as arguments, because XML nodes are not accepted by sql_io.put() (they are handled by db_xml.put())
- 03:08 AM Revision 5189: schemas/vegbien.sql: Renamed set_canon_taxonpath() to _set_canon_taxonpath() (adding _ prefix) so that db_xml.put() treats its arguments as arguments rather than as children with fkeys to parent
- 03:02 AM Revision 5188: schemas/vegbien.sql: Added set_canon_taxonpath() to set a taxonpath's canon_taxonpath_id after it has been created
- 02:48 AM Revision 5187: Added inputs/.TNRS/tnrs/cleanup.sql to cluster TNRS.tnrs on tnrs_accepted_names. This keeps TNRS.tnrs sorted with the accepted names first.
- 02:46 AM Revision 5186: input.Makefile: Staging tables installation: %/cleanup: Also run any custom cleanup.sql provided in the subdir. %/install: Removed processing of postprocess.sql because no datasources are using it and because cleanup.sql can now be used for this purpose.
- 02:39 AM Revision 5185: inputs/.TNRS/schema.sql: tnrs: Added tnrs_accepted_names index, which sorts accepted names first, and cluster the table on this index. This ensures that the component-parsed entries for accepted names are created before any verbatim names that point to them.
- 02:37 AM Revision 5184: input.Makefile: Staging tables installation: %/cleanup: Documented that this removes any index comments, due to a PostgreSQL bug. (This occurs because ALTER TABLE recreates the index but not its comment.)
- 01:55 AM Revision 5183: inputs/.TNRS/schema.sql: Removed hardcoded schema name
- 01:18 AM Revision 5182: inputs/.TNRS/tnrs/map.csv: Changed Name_matched_accepted_family comment to match analogous Name_matched_author comment
- 01:17 AM Revision 5181: inputs/.TNRS/tnrs/map.csv: Remapped Author_matched as the scientificNameAuthorship instead of Name_matched_author, because Name_matched_author contains the author based on the matched name, not the author in the original string, so it's not strictly from the original name
- 12:33 AM Revision 5180: mappings/VegCore.csv: Added acceptedBinomial, originalBinomial
- 12:29 AM Revision 5179: mappings/VegCore.csv: Added binomial
- 12:03 AM Revision 5178: inputs/.TNRS/tnrs/map.csv: Mapped Specific_epithet_matched
10/02/2012
- 11:53 PM Revision 5177: Added inputs/test_taxonomic_names/
- 11:37 PM Revision 5176: mappings/VegCore-VegBIEN.csv: taxonoccurrence.authortaxoncode: Only populate if needed to distinguish the taxonoccurrence within a plot
- 11:24 PM Revision 5175: schemas/vegbien.sql: placepath: Removed no longer used placepath_unique constraint on place_id. Removed place_id from placepath_unique_within_datasource_by_name unique index because otherranks is now used to store custom ranks.
- 11:23 PM Revision 5174: schemas/vegbien.sql: placepath: Removed no longer used placepath_unique constraint on place_id. Removed place_id from placepath_unique_within_datasource_by_name unique index because otherranks is now used to store custom ranks.
- 11:14 PM Revision 5173: schemas/vegbien.sql: taxonpath, placepath: Added *_required_key check constraints to ensure that empty entries are not created when a row does not have taxonpath/placepath data
- 10:35 PM Revision 5172: import_all: Use new dedicated cleanup make target to clean up TNRS.tnrs
- 09:54 PM Revision 5171: tnrs.py: encode_map: Added hidden minus sign, which TNRS removes
- 09:44 PM Revision 5170: csvs.py: tsv_encode_map: Escape \n as \n (instead of as a \ followed by a newline) for clarity. Added escape for \r by using strings.json_encode_map. TsvReader: Decode all escapes in tsv_encode_map.
- 09:25 PM Revision 5169: tnrs.py: encode_map: Added × (times), which TNRS replaces with x
- 09:18 PM Revision 5168: tnrs.py: encode_map: Added " and ', which TNRS removes when at the beginning or end
- 09:12 PM Revision 5167: tnrs.py: encode_map: Documented why each character needs to be encoded
- 09:04 PM Revision 5166: tnrs.py: encode_map: Removed '&', which is actually not a special character for TNRS (although ';' is)
- 09:02 PM Revision 5165: tnrs.py: encode_map: Added '_', which TNRS replaces with space
- 08:56 PM Revision 5164: sql_io.py: append_csv(): In INSERT mode, print # rows read (different from # lines read if some fields contained embedded newlines) and # rows inserted (different from # rows read if some violated a constraint)
- 08:42 PM Revision 5163: sql.py: insert(): Explicitly return None if the insert failed and a DuplicateKeyException or NullValueException was suppressed
- 07:13 PM Revision 5162: input.Makefile: Staging tables installation: $(logInstall*Add): Fixed bug where the existing install log would be overwritten in quiet mode, even though this function should append its output to the log. Note that plain $(logInstall*) always overwrites the existing install log because it is used by the first install command.
- 06:53 PM Revision 5161: strings.py: json_encode(): Fixed bug where '\n' and '\r' also needed to be encoded
- 06:50 PM Revision 5160: tnrs.py: repeated_tnrs_request(): Also retry request in debug mode if an HTTPError is thrown, so that debugging info can also be obtained if there is a bug in the TNRS client
10/01/2012
- 10:44 PM Revision 5159: tnrs_db: Updated query for new three-level taxonpath hierarchy, where the concatenated name is now stored in identifyingtaxonomicname instead of taxonomicnamewithauthor
- 10:41 PM Revision 5158: root map: Removed no longer needed public schema override, which is now handled by vegbien_dest
- 10:40 PM Revision 5157: vegbien_dest: Allow user to specify a custom public schema in the $public env var. This makes custom public schema functionality available to all VegBIEN-accessing scripts, not just map.
- 10:12 PM Revision 5156: tnrs_db: Adjusted pause, max_pause so the daemon waits longer before exiting, because after the initial TNRS run, most names have already been scrubbed and new names may not be added until the end of the import (in the case of a very large new datasource)
- 09:44 PM Revision 5155: input.Makefile: Staging tables installation: Added cleanup, %/cleanup to clean up already-installed tables
- 09:36 PM Revision 5154: tnrs.py: encode(): Also prepend special padding string to empty and whitespace-only strings because these names are otherwise ignored by TNRS (no response row)
- 09:15 PM Revision 5153: tnrs_db: pause: Increased to 30 min because if no new names are available in TNRS.tnrs, there is no need to check every minute for new names (which clutters up the log file output). The pause feature is designed to allow tnrs_db to run in parallel with the import process, and process new names as they are made available, which only happens once for each partition of each datasource.
- 09:11 PM Revision 5152: tnrs_db: Fixed bug where the new filtering out of already-scrubbed names caused names to be skipped, because the loop would both advance by the number of rows found *and* those rows would no longer be returned by the query, causing only every other set of rows to be processed
- 08:58 PM Revision 5151: tnrs.py: tnrs_request(): Rewrapped lines (became >80 chars after adding profiling)
- 08:52 PM Revision 5150: tnrs.py: tnrs_request(): Use new encode() and TnrsOutputStream to escape TNRS-invalid characters
- 08:51 PM Revision 5149: tnrs.py: Added encode(), decode(), decode_for_tsv(), and TnrsOutputStream to handle escaping TNRS-invalid characters
- 08:48 PM Revision 5148: strings.py: Added regexp_repl_esc()
- 08:47 PM Revision 5147: strings.py: Added replace_all() and replace_all_re(), as well as flip_map() for use with maps for these functions
- 08:46 PM Revision 5146: csvs.py: Added tsv_encode_map for use in creating TSVs parsed by TsvReader
- 06:42 PM Revision 5145: csvs.py: TsvReader: Also interpret '\t' as a tab, to provide a mechanism for encoding embedded tabs
- 05:47 PM Revision 5144: tnrs.py: gwt_encode(): Escape special characters in the string instead of removing them, so that TNRS receives the original name rather than a modified version. This will help make the submitted names match up with the returned Name_submitted.
- 05:45 PM Revision 5143: strings.py: Added json_encode()
- 05:44 PM Revision 5142: strings.py: Added esc_quotes()
- 04:52 PM Revision 5141: schemas/vegbien.sql: placepath.canon_placepath_id: Changed hierarchy comment to match the taxonpath.canon_taxonpath_id comment, but with a two-level hierarchy of datasource name -> accepted name. This may later be changed to a three-level hierarchy like taxonpath.canon_taxonpath_id depending on how GNRS works.
- 04:49 PM Revision 5140: schemas/vegbien.sql: taxonpath.canon_taxonpath_id: Changed comment to specify that taxonpaths should now be linked in a three-level hierarchy of datasource name -> concatenated name -> accepted name
- 04:45 PM Revision 5139: schemas/vegbien.sql: taxonpath, placepath: Changed "scrubbed" to "accepted" to emphasize that the name is the accepted name returned by TNRS or GNRS, rather than merely the matched name
- 04:38 PM Revision 5138: mappings/VegCore-VegBIEN.csv: non-TNRS taxonpaths: Store the concatenated identifyingtaxonomicname in a separate taxonpath owned by the TNRS datasource, so that it will match up with (and create a link to) the corresponding submitted TNRS name's taxonpath. This in turn is linked to the TNRS-determined accepted name, thus creating a three-level hierarchy of datasource name -> concatenated name -> accepted name.
- 03:59 PM Revision 5137: mappings/VegCore-VegBIEN.csv: taxonomic terms: Remapped the concatenated taxonomic name to new identifyingtaxonomicname to use it directly to match up with the TNRS submitted name. Continue to map scientificNameWithAuthorship to taxonomicnamewithauthor.
- 03:56 PM Revision 5136: schemas/vegbien.sql: taxonpath: Renamed plantcode to identifyingtaxonomicname so that it can be used to store the concatenated taxonomicname that gets scrubbed. This enables ignoring the name components when the full name is specified, so that when a TNRS submitted name's matched components are included in its taxonpath, this will not prevent a datasource's concatenated name (without the matched components) from matching up with the corresponding TNRS submitted name.
- 03:25 PM Revision 5135: schemas/vegbien.sql: taxonpath: Made taxonomicnamewithauthor optional again and include all columns in the taxonpath_unique_within_datasource_by_name unique index so that the original name components can be stored in a separate taxonpath from the taxonpath with the concatenated taxonomic name. (The datasource's taxonpath would not always contain an entry for taxonomicnamewithauthor, so the other columns also need to be used in the unique index.)
- 02:57 PM Revision 5134: schemas/vegbien.sql: taxonpath: Added back datasource_id, plantcode to make taxonpath datasource-specific again. This way, the original name components can still be stored in taxonpath, in addition to storing the concatenated name in a datasource-general taxonpath for use by TNRS.
09/28/2012
- 03:46 PM Revision 5133: inputs/.TNRS/tnrs/map.csv: Mapped columns for components of original, submitted name
- 03:33 PM Revision 5132: mappings/VegCore-VegBIEN.csv, VegCore.csv: Removed no longer used verbatimScientificNameWithAuthorship. Use scientificNameWithAuthorship instead, and map accepted (scrubbed) names to acceptedScientificNameWithAuthorship to create the canon_taxonpath_id link.
- 03:28 PM Revision 5131: inputs/.TNRS/tnrs/map.csv: Remapped to new accepted* taxonomic terms
- 03:23 PM Revision 5130: mappings/VegCore-VegBIEN.csv: Mapped accepted* taxonomic terms
- 03:00 PM Revision 5129: sql_io.py: cleanup_table(): Don't clean up the pkey, because the canonicalization involved may produce collisions (as it does for TNRS.tnrs)
- 02:58 PM Revision 5128: sql.py: Added pkey_col_()
- 02:31 PM Revision 5127: tnrs.py: tnrs_request(): Added comment that names containing only whitespace characters are ignored by TNRS and do not receive a response row. Our tnrs_db and reimport pipeline handles the necessary re-matching-up by just creating taxonpaths for each Name_submitted, and then letting the data import process on the following import attach to the prepopulated taxonpaths.
- 02:17 PM Revision 5126: tnrs_db: Exclude taxonomic names which have already been scrubbed, by using a filter-out LEFT JOIN on TNRS.tnrs
- 02:02 PM Revision 5125: tnrs.py: max_pause: Changed to 30 min because TNRS sometimes freezes for ~10 min. The freezing usually happens while the data is being uploaded rather than when it's being retrieved, so that the max_pause would not apply, but to be on the safe side, requests should not time out unnecessarily.
- 01:27 PM Revision 5124: tnrs_db: tnrs_profiler: Use iter_text='name' for consistency with tnrs.tnrs_request()'s own profiler's iter_text
- 01:25 PM Revision 5123: tnrs_db: Print cumulative profiling information after every TNRS request, rather than just at the end
- 01:22 PM Revision 5122: inputs/.TNRS/tnrs/tnrs.make: Append to the log file instead of overwriting it, so that the TNRS scrubbing of each import's new taxonomic names can be included in one log file. Echo the command to the log file to identify separate runs.
- 01:15 PM Revision 5121: TNRS-related programs: Use "names" instead of "taxons" for variable names because what's being submitted are actually verbatim taxonomic names, not official references to specific taxa
- 01:08 PM Revision 5120: tnrs.py: tnrs_request(): Profile the TNRS request
- 12:58 PM Revision 5119: tnrs.py: tnrs_request(): Fixed bug where initial_headers needed to be copied instead of just assigned to headers, because initial_headers is a global constant and should not be changed when the Cookie header is added
- 12:17 PM Revision 5118: mappings/VegCore.csv: originalTaxonRank, acceptedTaxonRank: Fixed sources to use verbatimTaxonRank, not taxonRank
- 12:15 PM Revision 5117: mappings/VegCore.csv: originalTaxonRank: Added source of the original* prefix
- 12:14 PM Revision 5116: mappings/VegCore.csv: acceptedTaxonRank: Added source of the accepted prefix
- 12:12 PM Revision 5115: mappings/VegCore.csv: accepted* taxonomic terms: Fixed sources of the accepted prefix to use acceptedNameUsage, not acceptedNameUsageID
- 12:09 PM Revision 5114: mappings/VegCore.csv: original* taxonomic terms: Source the original prefix to DwC originalNameUsage, which is a more offical source than SALVIAS orig_species
- 12:09 PM Revision 5113: mappings/VegCore.csv: original* taxonomic terms: Source the original prefix to DwC originalNameUsage, which is a more offical source than SALVIAS orig_species
- 11:56 AM Revision 5112: mappings/VegCore.csv: Added accepted* taxonomic terms to store the scrubbed name
- 11:42 AM Revision 5111: import_all: Clean up any new TNRS.tnrs entries before importing the TNRS data
- 11:36 AM Revision 5110: inputs/.TNRS/tnrs/: Create using datasource schema.sql file instead of text header and postprocess.sql, for clarity and to enable using `make inputs/.TNRS/tnrs/install` to clean up the tnrs entries populated by tnrs_db
- 11:21 AM Revision 5109: mappings/VegCore-VegBIEN.csv: Don't combine taxonRank with infraspecificEpithet if there is no infraspecificEpithet, because the taxonRank is only the infraspecificEpithet's prefix when there is an actual infraspecificEpithet. Often, taxonRank contains values like "genus" or "species" which cannot be used for this purpose.
- 10:54 AM Revision 5108: tnrs.py: repeated_tnrs_request(): Just retry the request once with with debug turned on, to avoid cluttering the log output with the verbose debug info of multiple failed requests if the error is not resolved on retry
- 10:47 AM Revision 5107: tnrs.py: tnrs_request(): repeated_tnrs_request(): Print all suppressed exceptions to stderr
- 10:41 AM Revision 5106: tnrs.py: tnrs_request(): parse_response(): Include both the response headers and the response body in the InvalidResponse message
- 10:23 AM Revision 5105: inputs/import.stats.xls: Updated import times
- 10:15 AM Revision 5104: profiling.py: Profiler: Fixed bug where instance variable start had the same name as method start()
- 10:08 AM Revision 5103: mappings/VegCore-VegBIEN.csv: verbatimScientificNameWithAuthorship: Set canon_taxonpath_id to 0 on the first, scrubbed taxonpath to auto-create the self reference that indicates a scrubbed taxonpath
- 10:03 AM Revision 5102: mappings/VegCore-VegBIEN.csv: Don't forward scientificName to taxonoccurrence.authortaxoncode when importing just taxonpaths, as for TNRS
- 09:51 AM Revision 5101: tnrs_db: Moved lower max_taxons limit to tnrs.py because it's really required to avoid crashing the TNRS server and should apply to all callers
- 09:35 AM Revision 5100: tnrs_db: Print log message with # of taxonpaths being sent to TNRS
- 09:30 AM Revision 5099: tnrs_db: Fixed bug where InvalidResponse was missing module name
- 09:29 AM Revision 5098: tnrs_db: Profile the TNRS requests. This involves using a finally block to ensure that the profiling stats are printed even if the program exits with an error.
- 09:13 AM Revision 5097: tnrs_db: Reduced the chunk size to avoid slowing down the TNRS server
- 09:07 AM Revision 5096: inputs/.TNRS/tnrs/tnrs.make: Added log option which outputs to the terminal instead when set to ""
- 09:01 AM Revision 5095: tnrs_db: Added log messages for Making TNRS request and Storing TNRS response data so that if the TNRS daemon pauses, it's obvious which step it's waiting on
- 08:58 AM Revision 5094: sql.py: insert(): ignore optimization: Fixed bug where needed to run insert_select() recoverably so that the aborted transaction is rolled back after a DuplicateKeyException or NullValueException
- 08:43 AM Revision 5093: tnrs_db: If tnrs.repeated_tnrs_request() stil throws InvalidResponse, skip the current set in case its data caused the error. Note that it will still be tried again the next time tnrs_db is run.
- 08:34 AM Revision 5092: mappings/VegCore-VegBIEN.csv: Don't forward scientificName to taxonoccurrence.authortaxoncode when importing just taxonpaths, as for TNRS
- 08:30 AM Revision 5091: repeated_tnrs_request(): When retrying after an invalid response, output protocol info for debugging
- 08:29 AM Revision 5090: inputs/Makefile: Import logs: Don't download .TNRS/tnrs/tnrs.make.log by default because it changes each time `make inputs/.TNRS/tnrs/tnrs-remake` is run, and any version downloaded for debugging should be preserved. It can still be downloaded by setting the tnrs_log env var.
- 08:17 AM Revision 5089: tnrs_client, tnrs_db: Use new tnrs.repeated_tnrs_request()
- 08:16 AM Revision 5088: tnrs.py: Added repeated_tnrs_request() to retry a TNRS request which returned an invalid response
- 08:05 AM Revision 5087: db_xml.py: put_table(): Fixed bug where pkeys_loc needed to be initialized. Note that this bug was only triggered when importing a table with zero rows (in this case, the initial empty TNRS.tnrs table), because otherwise it would be set in the loop.
- 07:57 AM Revision 5086: inputs/Makefile: Import logs: Also download inputs/.TNRS/tnrs/tnrs.make.log
- 07:56 AM Revision 5085: inputs/Makefile: Import logs: Use new $(rsync*) to also sync datasources starting with ., such as .TNRS
- 07:55 AM Revision 5084: lib/common.Makefile: rsync: Added $(rsync*) to rsync all files, including those starting with "."
- 07:43 AM Revision 5083: tnrs.py: parse_response(): Raise custom InvalidResponse exception instead of SystemExit, so callers can catch the exception and respond to it
- 07:38 AM Revision 5082: mappings/VegCore-VegBIEN.csv: taxonpath.taxonomicnamewithauthor _join_words mappings: Added space after taxon rank prefix (var., etc.) for infraspecific ranks
Also available in: Atom