Project

General

Profile

Activity

From 07/15/2012 to 08/13/2012

08/09/2012

05:02 PM Task #458: map all VegX sources to stems table
* VegX sources have been mapped to @VegCSV-VegBIEN.specimens.csv@, which for now is copied to produce the other table... Aaron Marcuse-Kubitza
03:58 PM Task #383: convert VegBank data dictionary to database comments
Have URLs to VegBank data dictionary pages for *[[VegCSV]]* terms in "*mappings/VegCSV-VegBIEN.specimens.csv*":https:... Aaron Marcuse-Kubitza
12:18 PM Task #472 (Rejected): replace accessioncodes with datasource_id+sourceaccessioncode
* If we autopopulate the sourceaccessioncode using other input data (as some mappings do already), the datasource_id+... Aaron Marcuse-Kubitza
11:59 AM Task #471 (Resolved): add make actions so new dependent maps are rebuilt automatically when their source map changes
* Currently, you have to run @make mappings/@ in addition to @make test@ after any mappings changes
* Before VegCSV,...
Aaron Marcuse-Kubitza

08/08/2012

08:39 PM Task #470 (Resolved): source terms from old versions of DwC to the DwC history page
* "DwC history page":http://rs.tdwg.org/dwc/terms/history/index.htm
* This will provide an automatic link to the des...
Aaron Marcuse-Kubitza
04:27 PM Task #469 (New): give everyone their own account on vegbien
* Replaces less secure shared password
* Ideally, get PostgreSQL to use the Linux password hashes on vegbiendev:
**...
Aaron Marcuse-Kubitza
04:25 PM Task #468 (Resolved): rename/delete and recreate public schema in one transaction
* This ensures that an instance of the current public schema is always available Aaron Marcuse-Kubitza

08/06/2012

11:15 PM Task #366 (Rejected): refactor VegX
We're switching to *[[VegCSV]]*, so this is no longer necessary Aaron Marcuse-Kubitza
11:14 PM Task #332 (Rejected): switch to VegX 1.5.3
We're switching to *[[VegCSV]]* instead Aaron Marcuse-Kubitza
11:13 PM Task #467 (New): underline all to do items in meetings notes
* This will make it easy to see not only which items are done (crossed out), but also which remain Aaron Marcuse-Kubitza
09:33 PM Task #464: reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
All XPaths have been reversed. However, CTFS's two test rows needed to be disabled because they require a transformat... Aaron Marcuse-Kubitza
09:24 PM Task #466 (Rejected): column-based import: avoid most exceptions by introspecting the DB structure instead of waiting for a parseable exception to be raised
Aaron Marcuse-Kubitza

08/03/2012

08:49 PM Task #465 (New): move issue tracker comments to wiki pages to ensure version history
replace comments with:
@[Comment moved to wiki to ensure version history]@
Aaron Marcuse-Kubitza
06:45 PM Task #464: reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
[Comment moved to description to ensure version history] Aaron Marcuse-Kubitza
06:44 PM Task #464 (Resolved): reverse XPaths so that they start with location instead of plantobservation or specimenreplicate
For steps to reverse, see [[XPath Reversing]] Aaron Marcuse-Kubitza
06:38 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
Both non-derived core maps (@mappings/DwC2-VegBIEN.specimens.csv@, @VegX-VegBIEN.stems.csv@) have had @expand_braces@... Aaron Marcuse-Kubitza

08/02/2012

06:39 PM Task #463: allow an input to appear multiple times in the same map spreadsheet
Already multi-safe:
* all line operations
* set operations:
** @union@
*** Note that if there is _any_ mapping for a...
Aaron Marcuse-Kubitza
06:18 PM Task #463 (Resolved): allow an input to appear multiple times in the same map spreadsheet
* This will allow an input to map to multiple outputs without needing to use the @{}@ custom XPath syntax Aaron Marcuse-Kubitza
05:50 PM Task #462 (Resolved): name backups according to svn revision instead of or in addition to the date
Aaron Marcuse-Kubitza

08/01/2012

06:57 AM Task #461 (New): don't display warning when nullable boolean columns are handled without using COALESCE()
... Aaron Marcuse-Kubitza

07/30/2012

07:49 PM Task #459 (New): data provider feedback: propagate srcs (input columns for an intermediate column) for all tables, not just SQL functions?
* but would not want to propagate them for hierarchical tables (plantname, namedplace), because each level should be ... Aaron Marcuse-Kubitza
07:31 PM Task #458 (Resolved): map all VegX sources to stems table
* facilitates migration to VegCSV because won't need to support multiple tables for it
** but then have to deal with...
Aaron Marcuse-Kubitza
07:16 PM Task #457 (New): use driver-native autocommit mode instead of our custom autocommit implementation
* Should improve performance by avoiding creating unnecessary transactions
* In commit mode, also removes the need t...
Aaron Marcuse-Kubitza
07:07 PM Task #456 (New): test that querying the DB uses the unique indexes instead of a seq scan
Sample query:... Aaron Marcuse-Kubitza
06:59 PM Task #424: Finish translating XML functions to SQL functions for column-based import
translate _map() first because that processes various formats of cultivated flags
* but this requires handling arbitr...
Aaron Marcuse-Kubitza
06:55 PM Task #455 (Resolved): change summarizing queries to use vegbien staging tables
* Will remove the need to maintain MySQL databases for each input that should be verified Aaron Marcuse-Kubitza
06:54 PM Task #454 (Resolved): update summarizing queries for current schema
* See @mappings/verify.*.sql@ and @verify@ subdir of NY, SALVIAS inputs Aaron Marcuse-Kubitza
06:50 PM Task #453 (New): cluster vegbien tables periodically
Needs cluster() function added to sql.py:... Aaron Marcuse-Kubitza
06:46 PM Task #452 (Resolved): add column-based import to automated testing
* Use row-based accepted test outputs (*.ref) as accepted outputs, to ensure row-based and column-based import produc... Aaron Marcuse-Kubitza
06:25 PM Task #451 (New): verify database contents as part of automated testing
* Use @make backups/public.<date>.sql@ after rotating the public schema?
* Requires truncating all tables to remove ...
Aaron Marcuse-Kubitza
06:19 PM Task #374 (Resolved): mechanism to export VegBIEN data to flat file
Run @make backups/public.<date>.sql@ Aaron Marcuse-Kubitza
04:24 PM Task #450 (New): cast all untyped literals as text using cast_literal()
* but need to deal with SQL function _nullIf() param type mismatch
sql_gen.py code to use:...
Aaron Marcuse-Kubitza
04:15 PM Task #449 (New): optimize JOINs on existing rows
vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-17-23-12-36.log.sql:
[2] DB query: non-cacheab...
Aaron Marcuse-Kubitza
04:15 PM Task #448 (New): timeout ANALYZE queries
vegbiendev:/home/bien/svn/inputs/SpeciesLink/import/specimens.2012-07-12-13-34-57.log.sql:
[3] DB query: non-cacheab...
Aaron Marcuse-Kubitza

07/20/2012

06:32 AM Task #424: Finish translating XML functions to SQL functions for column-based import
Core XML functions have been translated.
Sandboxing has been added to allow SQL functions to throw exceptions.
Aaron Marcuse-Kubitza
06:31 AM Task #378: create automated feedback mechanism
SQL function errors are now also saved using this mechanism. Aaron Marcuse-Kubitza

07/19/2012

05:56 PM Task #446 (New): fix deadlock in INSERT IGNORE replacement
h3. Possible solutions
# -retry query on TransactionRollbackError-: _done, but doesn't fix the underlying problem ...
Aaron Marcuse-Kubitza

07/17/2012

03:58 PM Task #443 (Resolved): back up staging tables
They have now been backed up as part of the full DB backup[1].
fn1. @make backups/vegbien.backup &@
Aaron Marcuse-Kubitza
03:51 PM Task #442 (Resolved): test restoring of import backups
I tested restoring by extracting the compressed, custom-format backup to a plain text SQL script[1]. This is much fas... Aaron Marcuse-Kubitza
03:39 PM Task #444 (Resolved): test backing up entire DB
Aaron Marcuse-Kubitza
03:35 PM Task #444: test backing up entire DB
Full DB backup is 2.5 GB (compressed) and takes ~45 min[1]. Testing takes 3.5 min[2].
fn1. @make backups/vegbien.bac...
Aaron Marcuse-Kubitza

07/16/2012

04:20 PM Task #442: test restoring of import backups
Options:
* "testing if the export can be restored":http://www.linuxquestions.org/questions/linux-software-2/testing-...
Aaron Marcuse-Kubitza
02:53 PM Task #442: test restoring of import backups
Google techniques for this Aaron Marcuse-Kubitza
02:52 PM Task #442 (Resolved): test restoring of import backups
Aaron Marcuse-Kubitza
02:52 PM Task #444 (Resolved): test backing up entire DB
* run time
* storage space
Aaron Marcuse-Kubitza
02:52 PM Task #443 (Resolved): back up staging tables
Aaron Marcuse-Kubitza
02:51 PM Task #441 (Resolved): import CTFS data using JOINs from DB export, not VegX
Aaron Marcuse-Kubitza
02:50 PM Task #440: aggregating validations of imports
Have the # of:
* records
* plots
* specimens
* taxonomic categories
* species
* locations
Aaron Marcuse-Kubitza
02:49 PM Task #440: aggregating validations of imports
update for current schema Aaron Marcuse-Kubitza
02:48 PM Task #440 (New): aggregating validations of imports
Aaron Marcuse-Kubitza
 

Also available in: Atom