backups/pg_snapshot: documented runtime for initial backup (4 h)

backups/mysql_snapshot: documented initial vegbiendev->jupiter upload time for GBIF/raw_occurrence_record.MYD (7 h for 91 GB = 3.7 MB/s)

added backups/mysql_snapshot, pg_snapshot

added backups/vegbien.r13002.backup.md5, vegbien.r13160.backup.md5

backups/TNRS.backup.md5: updated

added backups/users.sql.run

backups/TNRS.backup.md5: updated

added backups/vegbien.r12025.backup.md5

backups/TNRS.backup.md5: updated

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

updated backups/TNRS.backup.md5

added backups/vegbien.r11786.backup.md5

backups/TNRS.backup: saved copy backups/TNRS.2013-11-18.backup

backups/TNRS.backup.md5: updated

backups/TNRS.backup.md5: updated

added backups/vegbien.r11549.backup.md5

backups/TNRS.backup.md5: updated

backups/TNRS.backup.md5: updated

added backups/vegbien.r10848.backup.md5

backups/TNRS.backup.md5: updated

bugfix: backups/fix_perms: just make the backups themselves private, since the other files are in svn, and their permissions should match their accessibility through Redmine

updated backups/TNRS.backup.md5

added backups/vegbien.r10548.backup.md5

bugfix: backups/: svn:ignore: removed *.md5, which should be under version control

updated backups/TNRS.backup

backups/TNRS.backup.md5: updated for last backup

backups/Makefile: %.backup/restore: documented runtime (11 h; ~5 h to insert data). note that this is still much shorter than re-running column-based import.

backups/Makefile: $(restore): added --verbose to display pg_restore's incremental progress

bugfix: backups/.rsync_filter.download: need to prevent existing backups from being deleted on the local side, too, by changing hide patterns to exclude

backups/Makefile: upload: use bin/sync_upload

added backups/vegbien.r9897.backup.md5

backups/.rsync_ignore: replaced with .rsync_filter.upload to allow uploading new backups but not deleting existing backups if they don't exist on the local (rsync-invoking) side; and .rsync_filter.download to avoid downloading backups to the local side. this allows storing older backups just on jupiter, where there is much more disk space. note that this change must be made on the remote side (jupiter) for it to be effective, because these are remote-side rules and are only processed by the remote-side rsync instance.

updated backups/TNRS.backup.md5

added backups/TNRS.2013-6-17.backup.md5, TNRS.2013-6-22.backup.md5

added backups/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.

backups/TNRS.backup.md5: updated

added backups/TNRS.backup.md5, vegbien.r9459.backup.md5

added backups/*.md5

added backups/TNRS.2013-5-21.backup.md5

lib/common.Makefile: remote server: Use jupiter instead of vegbiendev, to ensure that all files get uploaded there rather than only to vegbiendev. This involves adding an extra database import step to download the uploaded files from jupiter onto vegbiendev.

Moved Checksums from backups/Makefile to lib/common.Makefile so all dirs (including inputs/) can use md5sum testing

lib/common.Makefile: Renamed $(src_server) to $(remote_host) and $(src_user) to $(remote_user) for clarity

backups/Makefile: %.md5/test: Added comment to run with `make -s` to avoid echoing make commands

backups/Makefile: Synchronization: %/download: Download the .md5 file first, so that the user is prompted right away for their password rather than after the main file has finished downloading, by which time the password prompt has timed out

backups/Makefile: Removed no longer used $(psqlVerbose)

backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB

backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien

backups/Makefile: Factored $(isMac) out into lib/common.Makefile

backups/Makefile; $(public*): Don't add a "public." prefix to get the name of the public schema

backups/Makefile: Removed no longer used $(rmSchema)

backups/Makefile: Use \$(rmSchemaCmd) from lib/common.Makefile instead of \$(rmSchema)

backups/Makefile: Removed no longer used $(dateFmt), $(mtime)

backups/Makefile: Removed %.backup/rotate, because this incorrectly causes the current time rather than the version to be used in the backup filename. The version should instead be specified in the backup filename when it's created.

Makefiles: Factored out common vars/functions into lib/common.Makefile

backups/Makefile: Synchronization: $(remote): Fixed bug where need trailing / at end of path

backups/Makefile: Synchronization: $(remote): Updated path to backups

backups/fix_perms: Removed world read permissions from backups dir. Note that this will require superuser permissions to view archived backups on jupiter, because the bien group is not set up with the same members as on vegbiendev. (On jupiter, it contains only stri,regetz,donoghue,naiamh.)

Added backups/fix_perms

backups/Makefile: Synchronization: %/download: Also download any .md5 file for the file

backups/Makefile: Synchronization: Replaced download target, which downloads all backups, with %/download, which downloads just a specific backup, because you would generally only want to extract a single backup from the archive for reinstallation

backups/Makefile: Synchronization: Sync with jupiter instead of vegbiendev. This requires running `make backups/upload` on vegbiendev to archive the files, instead of `make backups/download` to download them to your local machine.

backups/Makefile: Checksums: %.md5/test: Only use md5sum's -v option on Mac, because it's not supported on Linux (there, verbose mode is the default)

backups/Makefile: SQL: Full DB: vegbien.%.backup: Also generate MD5 sum

backups/Makefile: Support removing public schema versions based on the version of a full DB backup

backups/Makefile: Synchronization: Also sync *.md5

backups/Makefile: Checksums: %.md5/test: Made it an _always target

backups/Makefile: Checksums: Added %.md5/test to test generated checksums

backups/Makefile: Moved md5-related targets to separate Checksums section

backups/Makefile: %.md5: Removed not applicable comment which had been copied from %.sql

backups/Makefile: %.md5: Fixed bug where md5sum does not have a -q option like md5

backups/Makefile: %.md5: Fixed bug where need to use md5sum instead of md5 on Linux

backups/Makefile: Added md5s target to generate .md5 files for all backups

backups/Makefile: %.md5: Run with `nice -n +5` to avoid slowing down the UI

backups/: svn:ignore: Added *.md5. Removed no longer applicable *.log.

backups/Makefile: Changed paths to be relative to the Makefile rather than the current directory, so this Makefile can be used in other directories as well (such as jupiter:/aaronmk/VegBIEN.backups/)

backups/Makefile: %.backup: Also create MD5 of backup

backups/Makefile: Added %.md5 target to create checksums of each backup

backups/Makefile: $(rsyncBackups): Added *.csv

backups/: Ignore _* and *.csv

backups/Makefile: Backups: Full DB: Specify the date suffix of the backup when it's created rather than adding it afterwards. This allows the user to specify a suffix that matches the corresponding public-schema backup.

backups/: svn:ignore: Also ignore .*, which includes temp files generated by rsync

backups/Makefile: Added synchronization of backups with vegbiendev. Added downloading backups to After a new import steps.

backups/Makefile: Refactored to include lib/common.Makefile

backups/Makefile: Full DB: Added vegbien.backup/all to run both test and rotate

backups/Makefile: .sql: When testing, turn it off so make won't skip `.sql: %` in favor of it

backups/Makefile: Split %.backup and %.sql into separate targets for clarity

backups/Makefile: Added %.backup/rotate

backups/Makefile: Rearranged sections so that backup targets, which apply to both Archived imports and Full DB, are at the top of a common Backups section

backups/Makefile: Added %.backup/test

backups/Makefile: Full DB backups: Added vegbien.backup

backups/Makefile: Factored backup command into $(backup) for later use by full DB backups. Made Backups, Archived imports sections subsections of Archived imports so Full DB backups can have its own section.

backups/Makefile: Fixed bug where $(SHELL) needed to be set to bash so that $'\n' would be interpreted correctly

backups/Makefile: Fixed bug where *.sql files needed to be restored using psql because pg_restore only supports "non-plain-text formats"

backups/Makefile: Don't log stderr or run the command verbosely and instead just output the command and run time to the terminal. This matches what we do for pg_dump, which works better because it just prints the useful information when it's done running.

backups/Makefile: Remove log files after successful restore/extraction because they are only useful for tail -f when the restore operation is running in the background

backups/Makefile: Backups: Fixed bug where `%.sql: %` needed to come before %.sql with no prerequisites to be matched first

Moved archived imports and make targets to maintain them to new backups dir

Moved archived imports and make targets to maintain them to new backups dir