Revision 8335
Added by Aaron Marcuse-Kubitza over 11 years ago
| lib/sql.py | ||
|---|---|---|
| 778 | 778 |
if limit == 0: order_by = None |
| 779 | 779 |
if order_by in (None, order_by_pkey) and has_subset_func(db, table0): |
| 780 | 780 |
# can use subset function for fast querying at large OFFSET values |
| 781 |
run_query(db, 'SET LOCAL enable_sort TO off') |
|
| 781 | 782 |
table0 = sql_gen.FunctionCall(table0, limit_=limit, offset_=start) |
| 782 | 783 |
if limit != 0: limit = None # done by function |
| 783 | 784 |
start = None # done by function |
Also available in: Unified diff
lib/sql.py: mk_select(): using subset function: Turn off enable_sort (within the transaction) to avoid unwanted slow sorts. This change (along with the subset functions themselves) should significantly reduce the long FIA.occurrence_all table subset time (~8 hours altogether) and with it the total import time, which had more than doubled as a result of the FIA refresh. Note that this issue would have been even more pronounced for larger datasets, such as the GBIF refresh, which would have taken ~2.5 days longer (400 million rows * ~30% are plants * (FIA: ~8 hours/16.7 million rows) * 1 day/24 hours).