Project

General

Profile

« Previous | Next » 

Revision 5368

sql_io.py: put_table(): ignore_cond(): If cond simplifies to false, remove all rows instead of filtering out individual rows which will all be filtered out. This optimization should improve import times of tables, such as taxonconcept, which use a check constraint instead of NOT NULL constraints to prevent empty rows. The taxonomic schema refactoring caused the creation of many more levels of taxonconcepts, many of which (such as variety, forma, cultivar) are empty for most datasources, so this optimization should also reduce overall import times for datasources that have any empty levels of taxonconcept. Note that this optimization is only possible now that sql_gen.simplify_expr() is able to simplify all the way to a single boolean value for the taxonconcept_required_key constraint.

View differences:

lib/sql_io.py
481 481
            
482 482
            not_cond = sql_gen.NotCond(sql_gen.CustomCode(cond))
483 483
            log_debug('Ignoring rows where '+strings.as_tt(not_cond.to_str(db)))
484
            sql.delete(db, insert_in_table, not_cond)
484
            assert cond != sql_gen.true_expr # we know the constraint failed
485
            if cond == sql_gen.false_expr: remove_all_rows()
486
            else: sql.delete(db, insert_in_table, not_cond)
485 487
    
486 488
    not_null_cols = set()
487 489
    def ignore(in_col, value, e):

Also available in: Unified diff