Task #887
Updated by Aaron Marcuse-Kubitza about 10 years ago
h3. issue * in importing even moderate-sized datasources (eg. NY, SALVIAS) causes all the last full-database import, this caused available disk space errors in 29 of [[VegBIEN_contents#datasources|41 datasources]]: to be used up, and crashes the import: <pre> ssh -t vegbiendev.nceas.ucsb.edu exec sudo su - aaronmk export version=r13016 version=test_import grep --files-with-matches -F "No space left on device" inputs/{.,}*/*/logs/$version.log.sql # and uniqify by datasource the list includes all the datasources in the test run: NY, SALVIAS </pre> * in a test run, importing even moderate-sized datasources (eg. NY, SALVIAS) causes all the available last full-database import, this caused disk space to be used up, and crashes the import: errors in 29 of [[VegBIEN_contents#datasources|41 datasources]]: <pre> ssh -t vegbiendev.nceas.ucsb.edu exec sudo su - aaronmk export version=test_import version=r13016 grep --files-with-matches -F "No space left on device" inputs/{.,}*/*/logs/$version.log.sql # the list includes all the datasources in the test run: NY, SALVIAS and uniqify by datasource </pre> * because the test run crashed, as well, the problem this is most more likely a Postgres bug, but could also be a bug in Postgres or Linux itself. this unfortunately means that we *can't import datasources individually, either*, until we find any other part of the Postgres/Linux bug that is causing the problem. VM * this is not necessarily caused by #884, because sort temp files consume disk space only in proportion to the table size, which for a small import would not be nearly enough to fill the disk (and yet the disk does fill up) h3. fix * roll back Postgres to the version it was at in the last successful import, re-run import, and see if problem goes away ** this may require building Postgres from source, because past _revisions_ of the same numeric version might only be available in version control, not in binary form via apt-get (which numbers packages by numeric version) * if this isn't possible, it may be necessary to downgrade to Postgres 9.2 (which will unfortunately be missing some features that we now use)