Project

General

Profile

# Date Author Comment
14447 08/10/2014 05:28 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added steps to re-run geoscrubbing

14401 08/01/2014 04:22 PM Aaron Marcuse-Kubitza

/README.TXT: added steps to restore from Time Machine

14400 08/01/2014 04:15 PM Aaron Marcuse-Kubitza

/README.TXT: added steps to back up the local machine's hard drive

14399 08/01/2014 04:13 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize a Mac's settings with my testing machine's: renamed to "to back up the local machine's settings"

14398 08/01/2014 04:11 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize a Mac's settings with my testing machine's: removed download section because we don't use this

14197 07/20/2014 03:40 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: added warnings that you should not run backups/pg_snapshot while the import is running, nor until the previous import has been replaced

14091 07/16/2014 05:05 PM Aaron Marcuse-Kubitza

/README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it

14087 07/16/2014 03:24 PM Aaron Marcuse-Kubitza

bugfix: Full database import: need nested shell to prevent errexit from closing the window

14079 07/15/2014 11:14 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: logs glob: don't unintentionally match . or ..

14077 07/15/2014 10:41 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)

13942 07/09/2014 02:13 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location

13941 07/09/2014 02:12 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location

13877 07/02/2014 02:17 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter

13876 07/02/2014 02:00 AM Aaron Marcuse-Kubitza

/README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi

13824 06/19/2014 02:33 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: disk space: removed instructions to rerun the import if the disk space gets used up, because this is actually a bug and is not generally "fixed just by rerunning the import"

13797 06/18/2014 03:12 PM Aaron Marcuse-Kubitza

/Makefile, /README.TXT: PostgreSQL: Mac OS X: switched to using homebrew, which supports PostGIS

13764 06/13/2014 05:51 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Installation: Check out svn: need to use trunk instead of root

13763 06/13/2014 05:49 PM Aaron Marcuse-Kubitza

*: use vegbiendev:/home/bien instead of /home/bien/svn

13742 06/13/2014 10:51 AM Aaron Marcuse-Kubitza

/README.TXT: Mac settings backup: to Dropbox: stop Dropbox beforehand to prevent Dropbox from trying to capture filesystem events while syncing

13741 06/13/2014 10:18 AM Aaron Marcuse-Kubitza

/README.TXT: Mac settings backup: exclude ~/Library/Thunderbird/Profiles/9oo8rcyn.default/global-messages-db.sqlite because this file does not sync (issue #907)

13594 06/02/2014 05:25 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: To run TNRS: to rescrub all names: also need to re-create public-schema views that were cascadingly deleted

13593 06/02/2014 05:23 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: To run TNRS: added steps to rescrub all names

13494 05/20/2014 02:39 PM Aaron Marcuse-Kubitza

/README.TXT: to back up the vegbiendev databases: added step to review diff before performing backup, in case the backup scripts break and try to synchronize things incorrectly

13466 05/17/2014 03:40 PM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: added steps to back up the vegbiendev databases

13429 05/09/2014 07:27 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function

13428 05/09/2014 07:24 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this

13427 05/09/2014 02:51 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac

13426 05/09/2014 02:16 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files

13425 05/09/2014 02:05 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: for test import, added step to turn off DB backup

13424 05/09/2014 01:48 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell

13423 05/09/2014 01:46 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)

13422 05/09/2014 01:42 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell

13421 05/09/2014 01:40 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)

13341 04/29/2014 09:33 PM Aaron Marcuse-Kubitza

fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)

13337 04/29/2014 04:42 PM Aaron Marcuse-Kubitza

/README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing

13336 04/29/2014 04:40 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload

13335 04/29/2014 04:40 PM Aaron Marcuse-Kubitza

fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files

13333 04/29/2014 03:26 PM Aaron Marcuse-Kubitza

/README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape

13332 04/29/2014 03:23 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly

13331 04/29/2014 02:36 PM Aaron Marcuse-Kubitza

/README.TXT: to back up the version history: use $HOME to make paths platform-independent

13284 04/22/2014 08:14 PM Aaron Marcuse-Kubitza

/README.TXT: use `sudo -u ... -i` instead of `sudo su - ...` to avoid using two commands to accomplish the login

13119 04/10/2014 03:13 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: to import just a subset of the datasources: array env var needs to be set after opening the `screen` shell because array vars are apparently not inherited by the `screen` shell

13118 04/10/2014 02:42 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: to import just a subset of the datasources: added step to set custom import name

13117 04/10/2014 02:41 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added instructions for importing just a subset of the datasources

13031 04/02/2014 11:17 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: disk space: added high-water mark of 1.8 TB @11:15:05

13030 04/02/2014 10:56 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added steps to figure out which datasource tables were not successfully imported due to disk space errors

13029 04/02/2014 10:45 AM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: moved verification of exit statuses before verification of DB contents because there is no point in verifying the DB if the datasources didn't finish importing

13028 04/02/2014 09:01 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: disk space: documented that the entire disk again gets used long after the beginning of the import, when only a few datasources are running (ie. it definitely seems to be a recent bug in Postgres, and not a latent problem)

13027 04/01/2014 05:40 PM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: added task to regularly re-run full-database import so that bugs in it don't pile up. it needs to be kept in working order so that it works when it's needed.

13026 04/01/2014 04:24 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added steps to manually reimport the applicable datasources if there are errors due to exceeding available disk space

13025 04/01/2014 04:13 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: removed extra `ssh -t vegbiendev.nceas.ucsb.edu` before "upload logs", because the previous steps also occur on vegbiendev

13024 04/01/2014 04:04 PM Aaron Marcuse-Kubitza

/README.TXT: Notes on system stability: added recommendation to maintain a snapshot copy of the VM as it was at the last successful import, for fallback use if a system upgrade breaks anything. system upgrades on the snapshot VM should be disabled completely, and because this will also disable security fixes, the snapshot VM should be disconnected from the internet and all networking interfaces. (this is an unfortunate consequence of modern OSes being written in non-memory-safe languages such as C and C++.)

13023 04/01/2014 03:43 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: disk space: documented that a higher high-water mark actually occurs later in the import, so that the disk usage issue actually remains a problem after the very beginning

13022 04/01/2014 03:37 PM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: disk space: increased the minimum free space recommendation to 1 TB, because analysis of the disk usage during the beginning of the import shows that actually close to the entire amount is being used. however, this problem is normally undetectable unless the disk space is specifically checked, because it only manifests itself if the available disk space is exceeded completely.

13021 04/01/2014 02:04 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: documented that the beginning of the import should be scheduled at a time when the DB will not be needed for other uses, because vegbiendev will be slow for the first few hours of the import due to the import using all the available cores

13020 04/01/2014 01:36 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: documented that CPU load warning e-mails can safely be ignored. they happen because the parallel imports use all the available cores.

13017 04/01/2014 01:14 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: exiting `screen`: clarify that you must use `exit`, as Ctrl+D gets disabled to prevent accidental exits

13016 04/01/2014 12:47 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added step to restart Postgres to free up any disk space used by temp tables from the last import (this is apparently not automatically reclaimed)

13010 04/01/2014 11:40 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: disk space: increased minimum requirement to 500GB (~200GB extra), as the import may use significant additional space for temp tables

13009 04/01/2014 11:37 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: documented that env vars set before invoking `screen` will be inherited by it, so these steps will work even if they come before `screen`

13007 04/01/2014 11:23 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added steps to set a custom version, if the auto-assigned one would cause a collision with the last import

13006 04/01/2014 11:08 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: `unset version`: documented that this is needed because it may have been set in the outer shell

13002 03/30/2014 06:52 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: disk space: updated schema size (315GB)

13001 03/30/2014 06:45 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: removed `up` on jupiter because this is done as part of "do steps under Maintenance > "to synchronize vegbiendev, ..."

13000 03/30/2014 06:44 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: moved "do steps under Maintenance > "to synchronize vegbiendev, ..." outside of "On local machine" because these steps don't only take place on the local machine

12999 03/30/2014 06:41 PM Aaron Marcuse-Kubitza

/README.TXT: use `up` instead of `svn up --force` for consistency

12998 03/30/2014 06:40 PM Aaron Marcuse-Kubitza

fix: /README.TXT: always use `up` instead of `svn up` since this includes --force

12997 03/30/2014 06:39 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: removed unneeded `ssh -t vegbiendev.nceas.ucsb.edu exec sudo su - aaronmk` at beginning since this is performed again the first time it's needed

12996 03/30/2014 06:38 PM Aaron Marcuse-Kubitza

fix: /README.TXT: Full database import: removed erroneous line that resulted from a search-and-replace of connection commands in r12396. (it used to read "Follow the steps under Connecting to vegbiendev above, using jupiter instead". this step is now performed on the line below it.)

12959 03/28/2014 01:31 AM Aaron Marcuse-Kubitza

/README.TXT: moved "to back up e-mails" and "to back up the version history" before settings backup so that the local backup of these is up to date when everything gets backed up

12957 03/28/2014 12:45 AM Aaron Marcuse-Kubitza

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: backups/TNRS.backup: do this before the general sync so that any reverse sync that's needed won't include it

12956 03/28/2014 12:44 AM Aaron Marcuse-Kubitza

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: backups/TNRS.backup: use bin/sync_upload now that this works for rsync-ignored files

12951 03/27/2014 11:13 PM Aaron Marcuse-Kubitza

fix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: run `up` on all machines, not just jupiter, because all must be up-to-date to avoid extraneous diffs

12950 03/27/2014 11:11 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: `svn up` on jupiter: need to use up alias because that adds --force

12949 03/27/2014 11:10 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: added `svn up` on jupiter: needs to be in main dir (~/bien), not ~/Dropbox/svn/

12948 03/27/2014 11:08 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: added `svn up` on jupiter to avoid extraneous diffs when rsyncing

12929 03/27/2014 04:43 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: manually apply schema changes to the live public schema: moved under "update mappings and staging table column names" because this is a necessary part of that step

12928 03/27/2014 04:43 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: manually apply schema changes to the live public schema: moved under "update mappings and staging table column names" because this is a necessary part of that step

12927 03/27/2014 04:40 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: changed "update staging table column names" to "update mappings and staging table column names"

12887 03/24/2014 05:45 PM Aaron Marcuse-Kubitza

/README.TXT: `make inputs/{NVS,SALVIAS,TEAM}/test`: updated runtime (1 min)

12883 03/24/2014 05:04 PM Aaron Marcuse-Kubitza

/README.TXT: calls to `inputs/run postprocess`: direct user to refer to inputs/run for this, so the runtime doesn't have to be updated in multiple places

12881 03/24/2014 05:01 PM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: added steps to update staging table column names on the local machine and vegbiendev

12877 03/24/2014 01:21 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: VegCore data dictionary: `make inputs/{NVS,SALVIAS,TEAM}/test`: recorded runtime (30 s)

12876 03/24/2014 01:17 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: VegCore data dictionary: `make inputs/{NVS,SALVIAS,TEAM}/test`: prepended `time` to enable obtaining the runtime

12875 03/24/2014 01:11 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: VegCore data dictionary: `inputs/run postprocess`: updated runtime (20 min)

12752 03/18/2014 05:34 AM Aaron Marcuse-Kubitza

inputs/run: postprocess(): documented runtime (30 min)

12741 03/18/2014 02:59 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Maintenance: VegCore data dictionary: apply new data dict mappings: need to use postprocess rather than import runscript target, so that the command also works on an svn checkout without the flat files (the flat files are not needed for the staging table renaming)

12718 03/14/2014 09:09 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Maintenance: VegCore data dictionary: apply new data dict mappings: need to use import rather than mappings runscript target, to rename the staging tables

12717 03/14/2014 09:06 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Maintenance: VegCore data dictionary: also need to apply new data dict mappings on vegbiendev

12716 03/14/2014 08:19 PM Aaron Marcuse-Kubitza

fix: /README.TXT: Maintenance: VegCore data dictionary: added steps to apply the new data dictionary mappings to the datasource mappings and staging tables

12548 02/28/2014 10:28 PM Aaron Marcuse-Kubitza

/Makefile: added separate phppgadmin-Linux target to avoid needing to run the entire postgres-Linux target whenever http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down (after some system updates)

12426 02/25/2014 07:56 AM Aaron Marcuse-Kubitza

/README.TXT: use full hostname for jupiter so the commands work outside of the NCEAS network as well

12396 02/23/2014 11:57 PM Aaron Marcuse-Kubitza

fix: /README.TXT: use exact ssh command needed to connect to vegbiendev/jupiter (eg. `ssh -t vegbiendev.nceas.ucsb.edu exec sudo su - aaronmk`) instead of vaguely referring to "on vegbiendev"/"on jupiter"

12395 02/23/2014 11:28 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: screen: run `unset TMOUT` first because it is most important, now that the remote servers have a TMOUT set for extra security

12381 02/23/2014 06:00 PM Aaron Marcuse-Kubitza

/README.TXT: to back up the version history: added steps to sync git to the local machine

12227 02/15/2014 04:08 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: clarified that the staging tables should only be reinstalled if needed

12226 02/15/2014 04:08 AM Aaron Marcuse-Kubitza

/README.TXT: put ... around all uppercased text, for consistency

12148 02/08/2014 11:17 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: Check that source contains [# datasources] rows up through XAL: added alternative verification method when this is not the case (some datasources may be near the end depending on import order)

12128 02/07/2014 08:37 AM Aaron Marcuse-Kubitza

/README.TXT: to back up the version history: added back `git svn fetch` so we keep the git export up-to-date, too

12127 02/07/2014 08:29 AM Aaron Marcuse-Kubitza

/README.TXT: to back up the version history: added runtimes (1.5 h for the initial svnsync)