issue¶

• there is no soft limit on disk space inside Postgres, so the hard limit gets reached instead, causing an error which ricochets across the system and crashes various processes (similar to an out-of-memory condition caused by kernel overcommit, except that Postgres already has a throttle for that problem)
• since Postgres does not have a disk space throttle¹, our scripts need to do this instead

¹ the temp_file_limit config param seems to be intended to do this, but throws an error instead of handling the problem by pausing (self-throttling)
.

implementation¶

• lib/sql.py run_query() (the global function, not the method of DbConn) should trap "OperationalError: could not extend file "...": No space left on device" and handle it by pausing until the disk space goes back down, and then retrying the last SQL command
  • the last SQL command does not need to be idempotent, because the error means that it was rolled back