Revision 10884
Added by Aaron Marcuse-Kubitza about 11 years ago
README.TXT | ||
---|---|---|
7 | 7 |
WARNING: This will delete your entire VegBIEN DB! |
8 | 8 |
This includes all archived imports and staging tables. |
9 | 9 |
|
10 |
Maintenance: |
|
11 |
on a live machine, you should put the following in your .profile: |
|
12 |
-- |
|
13 |
# make svn files web-accessible. this does not affect unversioned files, because |
|
14 |
# these get the right permissions on the local machine instead. |
|
15 |
umask ug=rwx,o=rx |
|
16 |
|
|
17 |
unset TMOUT # TMOUT causes screen to exit even with background processes |
|
18 |
-- |
|
19 |
if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down: |
|
20 |
on vegbiendev: make postgres-Linux |
|
21 |
to synchronize vegbiendev, jupiter, and your local machine: |
|
22 |
WARNING: pay careful attention to all files that will be deleted or |
|
23 |
overwritten! |
|
24 |
install put if needed: |
|
25 |
download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it |
|
26 |
when changes are made on vegbiendev: |
|
27 |
on all machines: ./fix_perms |
|
28 |
on vegbiendev, upload: |
|
29 |
overwrite=1 bin/sync_upload |
|
30 |
then rerun with l=1 ... |
|
31 |
on your machine, download: |
|
32 |
overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki |
|
33 |
then rerun with l=1 ... |
|
34 |
overwrite=1 swap=1 bin/sync_upload |
|
35 |
then rerun with l=1 ... |
|
36 |
make backups/TNRS.backup/download live=1 |
|
37 |
overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms |
|
38 |
then rerun with l=1 ... |
|
39 |
to synchronize a Mac's settings with my testing machine's: |
|
40 |
download: |
|
41 |
WARNING: this will overwrite all your user's settings! |
|
42 |
overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State" |
|
43 |
then rerun with l=1 ... |
|
44 |
upload: |
|
45 |
do step when changes are made on vegbiendev > on your machine, download |
|
46 |
on jupiter: (cd ~/Dropbox/svn/; up) |
|
47 |
on your machine: |
|
48 |
overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes |
|
49 |
then rerun with l=1 ... |
|
50 |
overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc" |
|
51 |
then rerun with l=1 ... |
|
52 |
overwrite=1 sync_local_dir=~ sync_remote_url=~/Dropbox/home bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk" |
|
53 |
then rerun with l=1 ... |
|
54 |
upload just the VirtualBox VMs: |
|
55 |
on your machine: |
|
56 |
overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload ~/"VirtualBox VMs/**" |
|
57 |
then rerun with l=1 ... |
|
58 |
to backup files not in Time Machine: |
|
59 |
pg_ctl. stop # stop the PostgreSQL server |
|
60 |
src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/ |
|
61 |
then rerun with l=1 ... |
|
62 |
pg_ctl. start # start the PostgreSQL server |
|
63 |
VegCore data dictionary: |
|
64 |
Regularly, or whenever the VegCore data dictionary page |
|
65 |
(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore) |
|
66 |
is changed, regenerate mappings/VegCore.csv: |
|
67 |
make mappings/VegCore.htm-remake; make mappings/ |
|
68 |
svn di mappings/VegCore.tables.redmine |
|
69 |
If there are changes, update the data dictionary's Tables section |
|
70 |
When moving terms, check that no terms were lost: svn di |
|
71 |
svn ci -m 'mappings/VegCore.htm: regenerated from wiki' |
|
72 |
Important: Whenever you install a system update that affects PostgreSQL or |
|
73 |
any of its dependencies, such as libc, you should restart the PostgreSQL |
|
74 |
server. Otherwise, you may get strange errors like "the database system |
|
75 |
is in recovery mode" which go away upon reimport, or you may not be able |
|
76 |
to access the database as the postgres superuser. This applies to both |
|
77 |
Linux and Mac OS X. |
|
78 |
|
|
79 | 10 |
Single datasource import: |
80 | 11 |
(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 & |
81 | 12 |
(Re)import only: make inputs/<datasrc>/reimport by_col=1 & |
... | ... | |
221 | 152 |
$version with the appropriate revision |
222 | 153 |
make backups/upload live=1 |
223 | 154 |
|
224 |
Backups: |
|
225 |
Archived imports: |
|
226 |
Back up: make backups/<version>.backup & |
|
227 |
Note: To back up the last import, you must archive it first: |
|
228 |
make schemas/rotate |
|
229 |
Test: make -s backups/<version>.backup/test & |
|
230 |
Restore: make backups/<version>.backup/restore & |
|
231 |
Remove: make backups/<version>.backup/remove |
|
232 |
Download: make backups/<version>.backup/download |
|
233 |
TNRS cache: |
|
234 |
Back up: make backups/TNRS.backup-remake & |
|
235 |
runtime: 3 min ("real 2m48.859s") |
|
236 |
Restore: |
|
237 |
yes|make inputs/.TNRS/uninstall |
|
238 |
make backups/TNRS.backup/restore & |
|
239 |
runtime: 5.5 min ("real 5m35.829s") |
|
240 |
yes|make schemas/public/reinstall |
|
241 |
Must come after TNRS restore to recreate tnrs_input_name view |
|
242 |
Full DB: |
|
243 |
Back up: make backups/vegbien.<version>.backup & |
|
244 |
Test: make -s backups/vegbien.<version>.backup/test & |
|
245 |
Restore: make backups/vegbien.<version>.backup/restore & |
|
246 |
Download: make backups/vegbien.<version>.backup/download |
|
247 |
Import logs: |
|
248 |
Download: make inputs/download-logs live=1 |
|
249 |
|
|
250 | 155 |
Datasource setup: |
251 | 156 |
umask ug=rwx,o= # prevent files from becoming web-accessible |
252 | 157 |
Add a new datasource: make inputs/<datasrc>/add |
... | ... | |
363 | 268 |
make inputs/download live=1 |
364 | 269 |
Follow the steps under Install the staging tables above |
365 | 270 |
|
271 |
Maintenance: |
|
272 |
on a live machine, you should put the following in your .profile: |
|
273 |
-- |
|
274 |
# make svn files web-accessible. this does not affect unversioned files, because |
|
275 |
# these get the right permissions on the local machine instead. |
|
276 |
umask ug=rwx,o=rx |
|
277 |
|
|
278 |
unset TMOUT # TMOUT causes screen to exit even with background processes |
|
279 |
-- |
|
280 |
if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down: |
|
281 |
on vegbiendev: make postgres-Linux |
|
282 |
to synchronize vegbiendev, jupiter, and your local machine: |
|
283 |
WARNING: pay careful attention to all files that will be deleted or |
|
284 |
overwritten! |
|
285 |
install put if needed: |
|
286 |
download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it |
|
287 |
when changes are made on vegbiendev: |
|
288 |
on all machines: ./fix_perms |
|
289 |
on vegbiendev, upload: |
|
290 |
overwrite=1 bin/sync_upload |
|
291 |
then rerun with l=1 ... |
|
292 |
on your machine, download: |
|
293 |
overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki |
|
294 |
then rerun with l=1 ... |
|
295 |
overwrite=1 swap=1 bin/sync_upload |
|
296 |
then rerun with l=1 ... |
|
297 |
make backups/TNRS.backup/download live=1 |
|
298 |
overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms |
|
299 |
then rerun with l=1 ... |
|
300 |
to synchronize a Mac's settings with my testing machine's: |
|
301 |
download: |
|
302 |
WARNING: this will overwrite all your user's settings! |
|
303 |
overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State" |
|
304 |
then rerun with l=1 ... |
|
305 |
upload: |
|
306 |
do step when changes are made on vegbiendev > on your machine, download |
|
307 |
on jupiter: (cd ~/Dropbox/svn/; up) |
|
308 |
on your machine: |
|
309 |
overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes |
|
310 |
then rerun with l=1 ... |
|
311 |
overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc" |
|
312 |
then rerun with l=1 ... |
|
313 |
overwrite=1 sync_local_dir=~ sync_remote_url=~/Dropbox/home bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk" |
|
314 |
then rerun with l=1 ... |
|
315 |
upload just the VirtualBox VMs: |
|
316 |
on your machine: |
|
317 |
overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload ~/"VirtualBox VMs/**" |
|
318 |
then rerun with l=1 ... |
|
319 |
to backup files not in Time Machine: |
|
320 |
pg_ctl. stop # stop the PostgreSQL server |
|
321 |
src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/ |
|
322 |
then rerun with l=1 ... |
|
323 |
pg_ctl. start # start the PostgreSQL server |
|
324 |
VegCore data dictionary: |
|
325 |
Regularly, or whenever the VegCore data dictionary page |
|
326 |
(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore) |
|
327 |
is changed, regenerate mappings/VegCore.csv: |
|
328 |
make mappings/VegCore.htm-remake; make mappings/ |
|
329 |
svn di mappings/VegCore.tables.redmine |
|
330 |
If there are changes, update the data dictionary's Tables section |
|
331 |
When moving terms, check that no terms were lost: svn di |
|
332 |
svn ci -m 'mappings/VegCore.htm: regenerated from wiki' |
|
333 |
Important: Whenever you install a system update that affects PostgreSQL or |
|
334 |
any of its dependencies, such as libc, you should restart the PostgreSQL |
|
335 |
server. Otherwise, you may get strange errors like "the database system |
|
336 |
is in recovery mode" which go away upon reimport, or you may not be able |
|
337 |
to access the database as the postgres superuser. This applies to both |
|
338 |
Linux and Mac OS X. |
|
339 |
|
|
340 |
Backups: |
|
341 |
Archived imports: |
|
342 |
Back up: make backups/<version>.backup & |
|
343 |
Note: To back up the last import, you must archive it first: |
|
344 |
make schemas/rotate |
|
345 |
Test: make -s backups/<version>.backup/test & |
|
346 |
Restore: make backups/<version>.backup/restore & |
|
347 |
Remove: make backups/<version>.backup/remove |
|
348 |
Download: make backups/<version>.backup/download |
|
349 |
TNRS cache: |
|
350 |
Back up: make backups/TNRS.backup-remake & |
|
351 |
runtime: 3 min ("real 2m48.859s") |
|
352 |
Restore: |
|
353 |
yes|make inputs/.TNRS/uninstall |
|
354 |
make backups/TNRS.backup/restore & |
|
355 |
runtime: 5.5 min ("real 5m35.829s") |
|
356 |
yes|make schemas/public/reinstall |
|
357 |
Must come after TNRS restore to recreate tnrs_input_name view |
|
358 |
Full DB: |
|
359 |
Back up: make backups/vegbien.<version>.backup & |
|
360 |
Test: make -s backups/vegbien.<version>.backup/test & |
|
361 |
Restore: make backups/vegbien.<version>.backup/restore & |
|
362 |
Download: make backups/vegbien.<version>.backup/download |
|
363 |
Import logs: |
|
364 |
Download: make inputs/download-logs live=1 |
|
365 |
|
|
366 | 366 |
Datasource refreshing: |
367 | 367 |
VegBank: |
368 | 368 |
make inputs/VegBank/vegbank.sql-remake |
Also available in: Unified diff
/README.TXT: moved Single datasource import, Datasource setup to top since these are the most important howtos