Task #883
Updated by Aaron Marcuse-Kubitza about 10 years ago
* due to recent bugs in Postgres[1], full-database import now causes all running into the available disk space to be used up, and crashes the import * there is no soft limit on disk space inside Postgres, so the hard limit gets reached instead, causing an error which ricochets across the system causes crashes, and crashes various processes (similar to an out-of-memory condition caused by kernel overcommit, except that Postgres already has a throttle for that problem) should be avoided if at all possible * since Postgres itself does not have a disk space throttle[2], mechanism to do this[1], but our scripts need to could do this instead fn1. this is a recent problem in Postgres, because we used to need only 100 GB of free disk space for the import (in r6802/2012-12-12), but now we need 1 _TB_ (10x as much). although it only became an issue on the last import, it may have been a latent problem that just never crossed the disk space limit. fn2. the @temp_file_limit@ config param seems to be intended to do this, but "throws an error":http://www.postgresql.org/docs/9.3/static/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-DISK instead of handling the problem by pausing (self-throttling) h3. implementation * @lib/sql.py@ should trap @"OperationalError: could not extend file "...": No space left on device"@ and handle it by pausing until the disk space goes back down, and then retrying the last SQL command ** the last SQL command does not need to be idempotent, because the error means that it was rolled back fn1. the @temp_file_limit@ config param seems to be intended to do this, but "throws an error":http://www.postgresql.org/docs/9.3/static/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-DISK instead of handling the problem by pausing (self-throttling)