/README.TXT: Single datasource import: renamed to Single datasource refresh since it works on existing datasources
/README.TXT: Single datasource import: also need to reload staging tables
/README.TXT: Single datasource import: added steps to re-run geoscrubbing and back up the vegbiendev database
: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE since one might not see the SAVE file in it
/README.TXT: to back up the local machine's hard drive: also exclude *-files indicating the (differing) retention statuses of the partitions involved
/README.TXT: to back up the local machine's hard drive: renamed backup partition to BIEN3 to make clear what the backup drive contains
fix: /README.TXT: to back up the local machine's hard drive: updated location of `screen` for added commands
/README.TXT: added trailing / on dirs to make clear that they're dirs
fix: /README.TXT: to back up vegbiendev: also back up /home/aaronmk/bien/ (instead of just symlinking to the local copy), since this can be done space-efficiently with hardlinks. this ensures that the vegbiendev backup will not be modified when the local copy of bien/ is.
/README.TXT: to backup files not in Time Machine: don't need to review diff because command is unidirectional
fix: /README.TXT: to back up the local machine's hard drive: "repeat until only minimal changes" should refer to the first sync command
/README.TXT: to back up vegbiendev: use inplace=1 to speed stopping and resuming transfer
fix: /README.TXT: to back up the local machine's hard drive: removed --extended-attributes (after initial sync) because rsync apparently has to visit every file for this
fix: /README.TXT: to back up the local machine's hard drive: also need --extended-attributes
/README.TXT: to back up the local machine's hard drive: removed --delete-before now that that partition has been expanded
fix: /README.TXT: to back up vegbiendev: exclude /var/lib/mysql.bak,postgresql.bak because the local machine doesn't need 2 copies of this information
/README.TXT: to back up vegbiendev: removed no longer needed exclude of Dropbox subdir backup
fix: /README.TXT: to back up vegbiendev: also need to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine" because /home/aaronmk/bien is not synced here
bugfix: /README.TXT: to back up vegbiendev: need `overwrite=1`
/README.TXT: to back up the version history: don't also need this on vegbiendev because it's already on jupiter and the local machine
bugfix: /README.TXT: to back up vegbiendev: need to include Postgres config files
/README.TXT: to back up the local machine's hard drive: don't back up temp files: added /.fseventsd/
fix: /README.TXT: to back up the local machine's hard drive: initial runtime: use range instead because some of the later runtime might have been from the same files
/README.TXT: to back up the local machine's hard drive: updated initial runtime to include additional transferred files (17 h)
fix: /README.TXT: to back up the local machine's hard drive: need to use --delete-before because the backup partition is near capacity
/README.TXT: to back up the local machine's hard drive: don't back up temp files such as /private/var/vm/*
fix: /README.TXT: to back up the local machine's hard drive: back up most Dropbox/Postgres files before stopping processes, to minimize downtime
bugfix: /README.TXT: to back up the local machine's hard drive: can't use ~ with --exclude
/README.TXT: to back up the local machine's hard drive: back up the non-Dropbox, non-Postgres files separately to minimize the Dropbox and Postgres downtime
/README.TXT: to back up the vegbiendev databases: don't need to review diff for these as it's always unidirectional
/README.TXT: added instructions to back up vegbiendev
fix: /README.TXT: to back up the local machine's hard drive: also need to repeat backup command until only minimal changes
/README.TXT: to back up the local machine's hard drive: added step to stop Postgres
bugfix: /README.TXT: to back up the local machine's hard drive: also need to stop Dropbox
/README.TXT: to back up the local machine's settings: added step to remove .DS_Store
fix: /README.TXT: to back up the local machine's settings: Dropbox: shoudl not run with `del=`, because the backup should be an exact replica
/README.TXT: To re-run geoscrubbing: updated runtimes
/README.TXT: Full database import: converted database commands to command-line commands to make them easier to run
/README.TXT: Full database import: added steps to re-run geoscrubbing
/README.TXT: added steps to restore from Time Machine
/README.TXT: added steps to back up the local machine's hard drive
/README.TXT: to synchronize a Mac's settings with my testing machine's: renamed to "to back up the local machine's settings"
/README.TXT: to synchronize a Mac's settings with my testing machine's: removed download section because we don't use this
bugfix: /README.TXT: Full database import: added warnings that you should not run backups/pg_snapshot while the import is running, nor until the previous import has been replaced
/README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it
bugfix: Full database import: need nested shell to prevent errexit from closing the window
bugfix: /README.TXT: logs glob: don't unintentionally match . or ..
/README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)
bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location
bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location
bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter
/README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi
fix: /README.TXT: Full database import: disk space: removed instructions to rerun the import if the disk space gets used up, because this is actually a bug and is not generally "fixed just by rerunning the import"
/Makefile, /README.TXT: PostgreSQL: Mac OS X: switched to using homebrew, which supports PostGIS
bugfix: /README.TXT: Installation: Check out svn: need to use trunk instead of root
*: use vegbiendev:/home/bien instead of /home/bien/svn
/README.TXT: Mac settings backup: to Dropbox: stop Dropbox beforehand to prevent Dropbox from trying to capture filesystem events while syncing
/README.TXT: Mac settings backup: exclude ~/Library/Thunderbird/Profiles/9oo8rcyn.default/global-messages-db.sqlite because this file does not sync (issue #907)
bugfix: /README.TXT: Full database import: To run TNRS: to rescrub all names: also need to re-create public-schema views that were cascadingly deleted
/README.TXT: Full database import: To run TNRS: added steps to rescrub all names
/README.TXT: to back up the vegbiendev databases: added step to review diff before performing backup, in case the backup scripts break and try to synchronize things incorrectly
/README.TXT: Maintenance: added steps to back up the vegbiendev databases
/README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function
/README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this
bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac
fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files
/README.TXT: Full database import: for test import, added step to turn off DB backup
fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell
/README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)
fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell
fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)
fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)
/README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing
/README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload
fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files
/README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape
bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly
/README.TXT: to back up the version history: use $HOME to make paths platform-independent
/README.TXT: use `sudo -u ... -i` instead of `sudo su - ...` to avoid using two commands to accomplish the login
bugfix: /README.TXT: Full database import: to import just a subset of the datasources: array env var needs to be set after opening the `screen` shell because array vars are apparently not inherited by the `screen` shell
/README.TXT: Full database import: to import just a subset of the datasources: added step to set custom import name
/README.TXT: Full database import: added instructions for importing just a subset of the datasources
/README.TXT: Full database import: disk space: added high-water mark of 1.8 TB @11:15:05
/README.TXT: Full database import: added steps to figure out which datasource tables were not successfully imported due to disk space errors
fix: /README.TXT: Full database import: moved verification of exit statuses before verification of DB contents because there is no point in verifying the DB if the datasources didn't finish importing
/README.TXT: Full database import: disk space: documented that the entire disk again gets used long after the beginning of the import, when only a few datasources are running (ie. it definitely seems to be a recent bug in Postgres, and not a latent problem)
/README.TXT: Maintenance: added task to regularly re-run full-database import so that bugs in it don't pile up. it needs to be kept in working order so that it works when it's needed.
/README.TXT: Full database import: added steps to manually reimport the applicable datasources if there are errors due to exceeding available disk space
/README.TXT: Full database import: removed extra `ssh -t vegbiendev.nceas.ucsb.edu` before "upload logs", because the previous steps also occur on vegbiendev
/README.TXT: Notes on system stability: added recommendation to maintain a snapshot copy of the VM as it was at the last successful import, for fallback use if a system upgrade breaks anything. system upgrades on the snapshot VM should be disabled completely, and because this will also disable security fixes, the snapshot VM should be disconnected from the internet and all networking interfaces. (this is an unfortunate consequence of modern OSes being written in non-memory-safe languages such as C and C++.)
/README.TXT: Full database import: disk space: documented that a higher high-water mark actually occurs later in the import, so that the disk usage issue actually remains a problem after the very beginning
fix: /README.TXT: Full database import: disk space: increased the minimum free space recommendation to 1 TB, because analysis of the disk usage during the beginning of the import shows that actually close to the entire amount is being used. however, this problem is normally undetectable unless the disk space is specifically checked, because it only manifests itself if the available disk space is exceeded completely.
/README.TXT: Full database import: documented that the beginning of the import should be scheduled at a time when the DB will not be needed for other uses, because vegbiendev will be slow for the first few hours of the import due to the import using all the available cores
/README.TXT: Full database import: documented that CPU load warning e-mails can safely be ignored. they happen because the parallel imports use all the available cores.
/README.TXT: Full database import: exiting `screen`: clarify that you must use `exit`, as Ctrl+D gets disabled to prevent accidental exits
/README.TXT: Full database import: added step to restart Postgres to free up any disk space used by temp tables from the last import (this is apparently not automatically reclaimed)
/README.TXT: Full database import: disk space: increased minimum requirement to 500GB (~200GB extra), as the import may use significant additional space for temp tables
/README.TXT: Full database import: documented that env vars set before invoking `screen` will be inherited by it, so these steps will work even if they come before `screen`
