Task #905
Updated by Aaron Marcuse-Kubitza over 10 years ago
_see #887, #902_ h3. -alternate OS approach- p(. _tried, and problem also occurs on Mac, so using other approaches_ # -fix @`make install`@, which sets up the entire BIEN installation and dependencies- # -test the import on the local testing machine (a Mac), which already has most of the dependencies- # -if that doesn't work, try the other approaches below- h3. working-backup approach p(. _restoring _in theory, restoring to a working backup allows should allow us to successfully run the import, so that import. however, this requires a working backup, which we can then test system changes to see what broke things_ only partially have because it's missing the database._ # -restore vegbiendev to last working configuration- # -get restored VM to work on VirtualBox- ## -install bootloader- ## -install device drivers- _the Linux VM configuration does not support the VirtualBox ethernet device natively, so it must be configured manually_ # -set up database configuration[1] (the files and data that are not part of a Postgres-level backup)- # -install database contents[1]- there are 2 possible strategies for this: ** -a) fix the bugs in the version of @make install@ available in that backup, and run it to build the database from the flat files. the advantage of this approach is that @make install@ is very fast (~15 min) when restoring only a single datasource in addition to the metadata datasources, while @pg_restore@ would take a day or more or require a custom script to do a selective restore.- _not using this method_ ** -b) find the nearest database backup, restore it, and undo/redo modifications between the time of the VM backup and the time of the database backup. this involves either restoring the full DB (~1 day on vegbiendev, likely slower on an external USB drive) or writing a script to do a selective restore of the data needed for the test import; and then figuring out which modifications were made to the DB before/since the VM backup.- _see "README.TXT":http://vegpath.org/fs/config/VirtualBox_VMs/vegbiendev/README.TXT > to set up the VM_ # -test import and verify that it works- # -perform system upgrades and see if import still works- _see "README.TXT":http://vegpath.org/fs/config/VirtualBox_VMs/vegbiendev/README.TXT > to install system updates_ ** if not, selectively install system upgrades to narrow down the problem # update svn and see if import still works ** if not, test different revisions to narrow down the problem fn1(. the backups from that time did not include the database because of "incorrect information in the Postgres documentation":http://www.postgresql.org/docs/9.3/static/backup-file.html stating that a filesystem-level backup of the live database would not be useful (in fact Postgres keeps around enough WAL files, and can be configured to keep around even more of them, so that a filesystem-level backup would work perfectly fine). this incorrect information (essentially a disclaimer) is likely there because the Postgres developers do not want users being upset at them if their filesystem-level backup happens to be unrestorable. h3. tracing approach # attempt to trace where the incorrect columns list is being introduced ** generally time-consuming to trace data through complex code ** narrowing down the affected section of code (using the working-backup approach) would reduce the amount of code that needs to be traced h3. rewriting approach # rewrite the section of code with the bug ** in this case, that code section is the most complex part of our codebase, so this would be potentially very time-consuming ** narrowing down the affected section of code (using the working-backup approach) would reduce the amount of code that needs to be rewritten --- h3. -clean VM approach- p(. _not using this approach because we have a backup from the time of the last successful import, so we don't need to obtain/install dependencies in the versions they had in the last successful import_ # -prepare clean VMs- # -fix @`make install`@, which sets up the entire BIEN installation and dependencies- ** normally, we do not reinstall the DB from scratch, so the bugs in @`make install`@ only become apparent when it is run on a partial installation # install the database from scratch on a clean VM (VirtualBox) ** this would involve adding any missing dependencies to our install scripts # test the import in the clean VM with a sample datasource to see if that reproduces the problem ** if it does, we know it's a bug in Postgres/Ubuntu and can troubleshoot using VM images with different Postgres/Ubuntu versions ** if it doesn't, it's a problem specific to just the vegbiendev VM and we would reset the vegbiendev VM to a clean Ubuntu install and reinstall our dependencies h3. -Postgres rollback approach- p(. _not using this approach because it only works if the problem is with Postgres, but it could also be a problem in the other import code (eg. Python)_ # roll back Postgres to the version it was at in the last successful import ** this may require building Postgres from source, because past _revisions_ of the same numeric version might only be available in version control, not in binary form via apt-get (which numbers packages by numeric version) ** if this isn't possible, it may be necessary to downgrade to Postgres 9.2 (which will unfortunately be missing some features that we now use) # see if this fixes the problem