07/30/2012

07:49 PM Task #459 (New): data provider feedback: propagate srcs (input columns for an intermediate column) for all tables, not just SQL functions?

* but would not want to propagate them for hierarchical tables (plantname, namedplace), because each level should be ... Aaron Marcuse-Kubitza

07:31 PM Task #458 (Resolved): map all VegX sources to stems table

* facilitates migration to VegCSV because won't need to support multiple tables for it
** but then have to deal with... Aaron Marcuse-Kubitza

07:16 PM Task #457 (New): use driver-native autocommit mode instead of our custom autocommit implementation

* Should improve performance by avoiding creating unnecessary transactions
* In commit mode, also removes the need t... Aaron Marcuse-Kubitza

07:07 PM Task #456 (New): test that querying the DB uses the unique indexes instead of a seq scan

Sample query:... Aaron Marcuse-Kubitza

06:59 PM Task #424: Finish translating XML functions to SQL functions for column-based import

translate _map() first because that processes various formats of cultivated flags
* but this requires handling arbitr... Aaron Marcuse-Kubitza

06:55 PM Task #455 (Resolved): change summarizing queries to use vegbien staging tables

* Will remove the need to maintain MySQL databases for each input that should be verified Aaron Marcuse-Kubitza

06:54 PM Task #454 (Resolved): update summarizing queries for current schema

* See @mappings/verify.*.sql@ and @verify@ subdir of NY, SALVIAS inputs Aaron Marcuse-Kubitza

06:50 PM Task #453 (New): cluster vegbien tables periodically

Needs cluster() function added to sql.py:... Aaron Marcuse-Kubitza

06:46 PM Task #452 (Resolved): add column-based import to automated testing

* Use row-based accepted test outputs (*.ref) as accepted outputs, to ensure row-based and column-based import produc... Aaron Marcuse-Kubitza

06:25 PM Task #451 (New): verify database contents as part of automated testing

* Use @make backups/public.<date>.sql@ after rotating the public schema?
* Requires truncating all tables to remove ... Aaron Marcuse-Kubitza

06:19 PM Task #374 (Resolved): mechanism to export VegBIEN data to flat file

Run @make backups/public.<date>.sql@ Aaron Marcuse-Kubitza

04:24 PM Task #450 (New): cast all untyped literals as text using cast_literal()

* but need to deal with SQL function _nullIf() param type mismatch
sql_gen.py code to use:... Aaron Marcuse-Kubitza

04:15 PM Task #449 (New): optimize JOINs on existing rows

vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-17-23-12-36.log.sql:
[2] DB query: non-cacheab... Aaron Marcuse-Kubitza

04:15 PM Task #448 (New): timeout ANALYZE queries

vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-12-13-34-57.log.sql:
[3] DB query: non-cacheab... Aaron Marcuse-Kubitza

07/20/2012

06:32 AM Task #424: Finish translating XML functions to SQL functions for column-based import

Core XML functions have been translated.
Sandboxing has been added to allow SQL functions to throw exceptions. Aaron Marcuse-Kubitza

06:31 AM Task #378: create automated feedback mechanism

SQL function errors are now also saved using this mechanism. Aaron Marcuse-Kubitza

07/19/2012

05:56 PM Task #446 (New): fix deadlock in INSERT IGNORE replacement

h3. Possible solutions
# -retry query on TransactionRollbackError-: _done, but doesn't fix the underlying problem ... Aaron Marcuse-Kubitza

07/17/2012

03:58 PM Task #443 (Resolved): back up staging tables

They have now been backed up as part of the full DB backup[1].
fn1. @make backups/vegbien.backup &@ Aaron Marcuse-Kubitza

03:51 PM Task #442 (Resolved): test restoring of import backups

I tested restoring by extracting the compressed, custom-format backup to a plain text SQL script[1]. This is much fas... Aaron Marcuse-Kubitza

03:39 PM Task #444 (Resolved): test backing up entire DB

Aaron Marcuse-Kubitza

03:35 PM Task #444: test backing up entire DB

Full DB backup is 2.5 GB (compressed) and takes ~45 min[1]. Testing takes 3.5 min[2].
fn1. @make backups/vegbien.bac... Aaron Marcuse-Kubitza

07/16/2012

04:20 PM Task #442: test restoring of import backups

Options:
* "testing if the export can be restored":http://www.linuxquestions.org/questions/linux-software-2/testing-... Aaron Marcuse-Kubitza

02:53 PM Task #442: test restoring of import backups

Google techniques for this Aaron Marcuse-Kubitza

02:52 PM Task #442 (Resolved): test restoring of import backups

Aaron Marcuse-Kubitza

02:52 PM Task #444 (Resolved): test backing up entire DB

* run time
* storage space Aaron Marcuse-Kubitza

02:52 PM Task #443 (Resolved): back up staging tables

Aaron Marcuse-Kubitza

02:51 PM Task #441 (Resolved): import CTFS data using JOINs from DB export, not VegX

Aaron Marcuse-Kubitza

02:50 PM Task #440: aggregating validations of imports

Have the # of:
* records
* plots
* specimens
* taxonomic categories
* species
* locations Aaron Marcuse-Kubitza

02:49 PM Task #440: aggregating validations of imports

update for current schema Aaron Marcuse-Kubitza

02:48 PM Task #440 (New): aggregating validations of imports

Aaron Marcuse-Kubitza

07/02/2012

12:03 PM Task #426 (Resolved): Reload DB using column-based import

Has been reloaded once, and is being reloaded again Aaron Marcuse-Kubitza