Activity
From 07/05/2012 to 08/03/2012
08/03/2012
- 08:49 PM Task #465 (New): move issue tracker comments to wiki pages to ensure version history
- replace comments with:
@[Comment moved to wiki to ensure version history]@ - 06:45 PM Task #464: reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
- [Comment moved to description to ensure version history]
- 06:44 PM Task #464 (Resolved): reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
- For steps to reverse, see [[XPath Reversing]]
- 06:38 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
- Both non-derived core maps (@mappings/DwC2-VegBIEN.specimens.csv@, @VegX-VegBIEN.stems.csv@) have had @expand_braces@...
08/02/2012
- 06:39 PM Task #463: allow an input to appear multiple times in the same map spreadsheet
- Already multi-safe:
* all line operations
* set operations:
** @union@
*** Note that if there is _any_ mapping for a... - 06:18 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
- * This will allow an input to map to multiple outputs without needing to use the @{}@ custom XPath syntax
- 05:50 PM Task #462 (Resolved): name backups according to svn revision instead of or in addition to the date
08/01/2012
- 06:57 AM Task #461 (New): don't display warning when nullable boolean columns are handled without using COALESCE()
- ...
07/30/2012
- 07:49 PM Task #459 (New): data provider feedback: propagate srcs (input columns for an intermediate column) for all tables, not just SQL functions?
- * but would not want to propagate them for hierarchical tables (plantname, namedplace), because each level should be ...
- 07:31 PM Task #458 (Resolved): map all VegX sources to stems table
- * facilitates migration to VegCSV because won't need to support multiple tables for it
** but then have to deal with... - 07:16 PM Task #457 (New): use driver-native autocommit mode instead of our custom autocommit implementation
- * Should improve performance by avoiding creating unnecessary transactions
* In commit mode, also removes the need t... - 07:07 PM Task #456 (New): test that querying the DB uses the unique indexes instead of a seq scan
- Sample query:...
- 06:59 PM Task #424: Finish translating XML functions to SQL functions for column-based import
- translate _map() first because that processes various formats of cultivated flags
* but this requires handling arbitr... - 06:55 PM Task #455 (Resolved): change summarizing queries to use vegbien staging tables
- * Will remove the need to maintain MySQL databases for each input that should be verified
- 06:54 PM Task #454 (Resolved): update summarizing queries for current schema
- * See @mappings/verify.*.sql@ and @verify@ subdir of NY, SALVIAS inputs
- 06:50 PM Task #453 (New): cluster vegbien tables periodically
- Needs cluster() function added to sql.py:...
- 06:46 PM Task #452 (Resolved): add column-based import to automated testing
- * Use row-based accepted test outputs (*.ref) as accepted outputs, to ensure row-based and column-based import produc...
- 06:25 PM Task #451 (New): verify database contents as part of automated testing
- * Use @make backups/public.<date>.sql@ after rotating the public schema?
* Requires truncating all tables to remove ... - 06:19 PM Task #374 (Resolved): mechanism to export VegBIEN data to flat file
- Run @make backups/public.<date>.sql@
- 04:24 PM Task #450 (New): cast all untyped literals as text using cast_literal()
- * but need to deal with SQL function _nullIf() param type mismatch
sql_gen.py code to use:... - 04:15 PM Task #449 (New): optimize JOINs on existing rows
- vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-17-23-12-36.log.sql:
[2] DB query: non-cacheab... - 04:15 PM Task #448 (New): timeout ANALYZE queries
- vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-12-13-34-57.log.sql:
[3] DB query: non-cacheab...
07/20/2012
- 06:32 AM Task #424: Finish translating XML functions to SQL functions for column-based import
- Core XML functions have been translated.
Sandboxing has been added to allow SQL functions to throw exceptions. - 06:31 AM Task #378: create automated feedback mechanism
- SQL function errors are now also saved using this mechanism.
07/19/2012
- 05:56 PM Task #446 (New): fix deadlock in INSERT IGNORE replacement
- h3. Possible solutions
# -retry query on TransactionRollbackError-: _done, but doesn't fix the underlying problem ...
07/17/2012
- 03:58 PM Task #443 (Resolved): back up staging tables
- They have now been backed up as part of the full DB backup[1].
fn1. @make backups/vegbien.backup &@ - 03:51 PM Task #442 (Resolved): test restoring of import backups
- I tested restoring by extracting the compressed, custom-format backup to a plain text SQL script[1]. This is much fas...
- 03:39 PM Task #444 (Resolved): test backing up entire DB
- 03:35 PM Task #444: test backing up entire DB
- Full DB backup is 2.5 GB (compressed) and takes ~45 min[1]. Testing takes 3.5 min[2].
fn1. @make backups/vegbien.bac...
07/16/2012
- 04:20 PM Task #442: test restoring of import backups
- Options:
* "testing if the export can be restored":http://www.linuxquestions.org/questions/linux-software-2/testing-... - 02:53 PM Task #442: test restoring of import backups
- Google techniques for this
- 02:52 PM Task #442 (Resolved): test restoring of import backups
- 02:52 PM Task #444 (Resolved): test backing up entire DB
- * run time
* storage space - 02:52 PM Task #443 (Resolved): back up staging tables
- 02:51 PM Task #441 (Resolved): import CTFS data using JOINs from DB export, not VegX
- 02:50 PM Task #440: aggregating validations of imports
- Have the # of:
* records
* plots
* specimens
* taxonomic categories
* species
* locations - 02:49 PM Task #440: aggregating validations of imports
- update for current schema
- 02:48 PM Task #440 (New): aggregating validations of imports
Also available in: Atom