Project

General

Profile

Statistics
| Revision:
Name Size Revision Age Author Comment
  _archive 1598 almost 13 years Aaron Marcuse-Kubitza Moved _archive/tapir2flatClient/trunk/client/ t...
  analysis 3076 over 12 years Aaron Marcuse-Kubitza Added top-level analysis dir for range modeling
  bin 3271 over 12 years Aaron Marcuse-Kubitza csv2db: verbosity defaults to 3 so that detaile...
  config 272 about 13 years Aaron Marcuse-Kubitza Moved bien_password to new config dir
  inputs 3261 over 12 years Aaron Marcuse-Kubitza inputs/import.stats.xls: Updated with stats fro...
  lib 3274 over 12 years Aaron Marcuse-Kubitza sql_io.py: put_table(): DuplicateKeyException: ...
  mappings 3229 over 12 years Aaron Marcuse-Kubitza mappings/VegX-VegBIEN.stems.csv: Sort the plant...
  schemas 3278 over 12 years Aaron Marcuse-Kubitza schemas/py_functions.sql: Removed per-column in...
  to_do 2547 over 12 years Aaron Marcuse-Kubitza to_do/timeline.doc: Updated to reflect the mont...
Makefile 10.5 KB 3249 over 12 years Aaron Marcuse-Kubitza root Makefile: VegBIEN DB: Schemas: Added schem...
README.TXT 2.96 KB 3205 over 12 years Aaron Marcuse-Kubitza README.TXT: Data import: Import data into VegBI...
map 1.21 KB 3140 over 12 years Aaron Marcuse-Kubitza top-level map: Added support for custom public ...

Latest revisions

# Date Author Comment
3278 07/09/2012 04:32 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: Removed per-column indexes on relational functions, which are no longer needed by row-based import because it is able to do a merge join-style lookup using the table's UNIQUE INDEX. (Note that column-based import doesn't use the (slower) relational functions at all anymore, and instead calls the corresponding SQL function directly using named arguments.)

3277 07/09/2012 04:31 PM Aaron Marcuse-Kubitza

schemas/functions.sql: Removed per-column indexes on relational functions, which are no longer needed by row-based import because it is able to do a merge join-style lookup using the table's UNIQUE INDEX. (Note that column-based import doesn't use the (slower) relational functions at all anymore, and instead calls the corresponding SQL function directly using named arguments.)

3276 07/09/2012 04:26 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: plantname: plantname_unique UNIQUE INDEX: Moved scope_id to the back so that the index can easily be used for lookup queries (not just column-based import) without having to explicitly specify NULL for that field. This takes advantage of a btree sorting feature where a broader lookup can be done using just the first n columns of the index.

3275 07/09/2012 04:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: locationevent, specimenreplicate: Turned UNIQUE CONSTRAINTs and UNIQUE INDEXes with nullable fields into partial UNIQUE INDEXes with IS NOT NULL filter conditions, in order to work automatically with sql_gen without requiring a separate covering lookup index. Removed no longer needed covering lookup indexes.

3274 07/09/2012 03:07 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): DuplicateKeyException: Fixed bug where combining multiple unique constraints was incorrectly allowed, when in fact the constraints need to be separately applied to the different rows that violate them, which is not currently supported

3273 07/09/2012 03:02 PM Aaron Marcuse-Kubitza

sql.py: DbConn.with_savepoint(): Log transaction profiling info with level=4 like the rest of the transaction commands, so that it isn't output when the transaction itself should be hidden (e.g. for name versioning or internal commands)

3272 07/09/2012 02:16 PM Aaron Marcuse-Kubitza

sql.py: DbConn.with_savepoint(): Profile (nested) transactions so that the run time for groups of commands (e.g. csv2db INSERTs) is known

3271 07/09/2012 02:04 PM Aaron Marcuse-Kubitza

csv2db: verbosity defaults to 3 so that detailed queries with profiling stats are included in the log file, to assist in optimization

3270 07/09/2012 02:01 PM Aaron Marcuse-Kubitza

csv2db: Don't cache per-row INSERT queries because this bloats the cache (there aren't repeated identical INSERTs that shouldn't be re-run like in row-based import)

3269 07/09/2012 01:57 PM Aaron Marcuse-Kubitza

sql.py with_explain_comment(), DbConn: Fixed bug where with_explain_comment() was being run in per-row imports (row-based import and csv2db with INSERT), causing the overhead of an EXPLAIN query for every single INSERT and filling up the cache with EXPLAIN query results, by adding autoexplain mode, only running with_explain_comment() in autoexplain mode, and only enabling autoexplain mode for column-based import

View all revisions | View revisions

Also available in: Atom