Activity
From 07/12/2012 to 08/10/2012
08/09/2012
- 05:02 PM Task #458: map all VegX sources to stems table
- * VegX sources have been mapped to @VegCSV-VegBIEN.specimens.csv@, which for now is copied to produce the other table...
- 03:58 PM Task #383: convert VegBank data dictionary to database comments
- Have URLs to VegBank data dictionary pages for *[[VegCSV]]* terms in "*mappings/VegCSV-VegBIEN.specimens.csv*":https:...
- 12:18 PM Task #472 (Rejected): replace accessioncodes with datasource_id+sourceaccessioncode
- * If we autopopulate the sourceaccessioncode using other input data (as some mappings do already), the datasource_id+...
- 11:59 AM Task #471 (Resolved): add make actions so new dependent maps are rebuilt automatically when their source map changes
- * Currently, you have to run @make mappings/@ in addition to @make test@ after any mappings changes
* Before VegCSV,...
08/08/2012
- 08:39 PM Task #470 (Resolved): source terms from old versions of DwC to the DwC history page
- * "DwC history page":http://rs.tdwg.org/dwc/terms/history/index.htm
* This will provide an automatic link to the des... - 04:27 PM Task #469 (New): give everyone their own account on vegbien
- * Replaces less secure shared password
* Ideally, get PostgreSQL to use the Linux password hashes on vegbiendev:
**... - 04:25 PM Task #468 (Resolved): rename/delete and recreate public schema in one transaction
- * This ensures that an instance of the current public schema is always available
08/06/2012
- 11:15 PM Task #366 (Rejected): refactor VegX
- We're switching to *[[VegCSV]]*, so this is no longer necessary
- 11:14 PM Task #332 (Rejected): switch to VegX 1.5.3
- We're switching to *[[VegCSV]]* instead
- 11:13 PM Task #467 (New): underline all to do items in meetings notes
- * This will make it easy to see not only which items are done (crossed out), but also which remain
- 09:33 PM Task #464: reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
- All XPaths have been reversed. However, CTFS's two test rows needed to be disabled because they require a transformat...
- 09:24 PM Task #466 (Rejected): column-based import: avoid most exceptions by introspecting the DB structure instead of waiting for a parseable exception to be raised
08/03/2012
- 08:49 PM Task #465 (New): move issue tracker comments to wiki pages to ensure version history
- replace comments with:
@[Comment moved to wiki to ensure version history]@ - 06:45 PM Task #464: reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
- [Comment moved to description to ensure version history]
- 06:44 PM Task #464 (Resolved): reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
- For steps to reverse, see [[XPath Reversing]]
- 06:38 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
- Both non-derived core maps (@mappings/DwC2-VegBIEN.specimens.csv@, @VegX-VegBIEN.stems.csv@) have had @expand_braces@...
08/02/2012
- 06:39 PM Task #463: allow an input to appear multiple times in the same map spreadsheet
- Already multi-safe:
* all line operations
* set operations:
** @union@
*** Note that if there is _any_ mapping for a... - 06:18 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
- * This will allow an input to map to multiple outputs without needing to use the @{}@ custom XPath syntax
- 05:50 PM Task #462 (Resolved): name backups according to svn revision instead of or in addition to the date
08/01/2012
- 06:57 AM Task #461 (New): don't display warning when nullable boolean columns are handled without using COALESCE()
- ...
07/30/2012
- 07:49 PM Task #459 (New): data provider feedback: propagate srcs (input columns for an intermediate column) for all tables, not just SQL functions?
- * but would not want to propagate them for hierarchical tables (plantname, namedplace), because each level should be ...
- 07:31 PM Task #458 (Resolved): map all VegX sources to stems table
- * facilitates migration to VegCSV because won't need to support multiple tables for it
** but then have to deal with... - 07:16 PM Task #457 (New): use driver-native autocommit mode instead of our custom autocommit implementation
- * Should improve performance by avoiding creating unnecessary transactions
* In commit mode, also removes the need t... - 07:07 PM Task #456 (New): test that querying the DB uses the unique indexes instead of a seq scan
- Sample query:...
- 06:59 PM Task #424: Finish translating XML functions to SQL functions for column-based import
- translate _map() first because that processes various formats of cultivated flags
* but this requires handling arbitr... - 06:55 PM Task #455 (Resolved): change summarizing queries to use vegbien staging tables
- * Will remove the need to maintain MySQL databases for each input that should be verified
- 06:54 PM Task #454 (Resolved): update summarizing queries for current schema
- * See @mappings/verify.*.sql@ and @verify@ subdir of NY, SALVIAS inputs
- 06:50 PM Task #453 (New): cluster vegbien tables periodically
- Needs cluster() function added to sql.py:...
- 06:46 PM Task #452 (Resolved): add column-based import to automated testing
- * Use row-based accepted test outputs (*.ref) as accepted outputs, to ensure row-based and column-based import produc...
- 06:25 PM Task #451 (New): verify database contents as part of automated testing
- * Use @make backups/public.<date>.sql@ after rotating the public schema?
* Requires truncating all tables to remove ... - 06:19 PM Task #374 (Resolved): mechanism to export VegBIEN data to flat file
- Run @make backups/public.<date>.sql@
- 04:24 PM Task #450 (New): cast all untyped literals as text using cast_literal()
- * but need to deal with SQL function _nullIf() param type mismatch
sql_gen.py code to use:... - 04:15 PM Task #449 (New): optimize JOINs on existing rows
- vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-17-23-12-36.log.sql:
[2] DB query: non-cacheab... - 04:15 PM Task #448 (New): timeout ANALYZE queries
- vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-12-13-34-57.log.sql:
[3] DB query: non-cacheab...
07/20/2012
- 06:32 AM Task #424: Finish translating XML functions to SQL functions for column-based import
- Core XML functions have been translated.
Sandboxing has been added to allow SQL functions to throw exceptions. - 06:31 AM Task #378: create automated feedback mechanism
- SQL function errors are now also saved using this mechanism.
07/19/2012
- 05:56 PM Task #446 (New): fix deadlock in INSERT IGNORE replacement
- h3. Possible solutions
# -retry query on TransactionRollbackError-: _done, but doesn't fix the underlying problem ...
07/17/2012
- 03:58 PM Task #443 (Resolved): back up staging tables
- They have now been backed up as part of the full DB backup[1].
fn1. @make backups/vegbien.backup &@ - 03:51 PM Task #442 (Resolved): test restoring of import backups
- I tested restoring by extracting the compressed, custom-format backup to a plain text SQL script[1]. This is much fas...
- 03:39 PM Task #444 (Resolved): test backing up entire DB
- 03:35 PM Task #444: test backing up entire DB
- Full DB backup is 2.5 GB (compressed) and takes ~45 min[1]. Testing takes 3.5 min[2].
fn1. @make backups/vegbien.bac...
07/16/2012
- 04:20 PM Task #442: test restoring of import backups
- Options:
* "testing if the export can be restored":http://www.linuxquestions.org/questions/linux-software-2/testing-... - 02:53 PM Task #442: test restoring of import backups
- Google techniques for this
- 02:52 PM Task #442 (Resolved): test restoring of import backups
- 02:52 PM Task #444 (Resolved): test backing up entire DB
- * run time
* storage space - 02:52 PM Task #443 (Resolved): back up staging tables
- 02:51 PM Task #441 (Resolved): import CTFS data using JOINs from DB export, not VegX
- 02:50 PM Task #440: aggregating validations of imports
- Have the # of:
* records
* plots
* specimens
* taxonomic categories
* species
* locations - 02:49 PM Task #440: aggregating validations of imports
- update for current schema
- 02:48 PM Task #440 (New): aggregating validations of imports
Also available in: Atom