Project

General

Profile

« Previous | Next » 

Revision 10884

/README.TXT: moved Single datasource import, Datasource setup to top since these are the most important howtos

View differences:

README.TXT
7 7
		WARNING: This will delete your entire VegBIEN DB!
8 8
		This includes all archived imports and staging tables.
9 9

  
10
Maintenance:
11
	on a live machine, you should put the following in your .profile:
12
--
13
# make svn files web-accessible. this does not affect unversioned files, because
14
# these get the right permissions on the local machine instead.
15
umask ug=rwx,o=rx
16

  
17
unset TMOUT # TMOUT causes screen to exit even with background processes
18
--
19
	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:
20
		on vegbiendev: make postgres-Linux
21
	to synchronize vegbiendev, jupiter, and your local machine:
22
		WARNING: pay careful attention to all files that will be deleted or
23
			overwritten!
24
		install put if needed:
25
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
26
		when changes are made on vegbiendev:
27
			on all machines: ./fix_perms
28
			on vegbiendev, upload:
29
				overwrite=1 bin/sync_upload
30
					then rerun with l=1 ...
31
			on your machine, download:
32
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
33
					then rerun with l=1 ...
34
				overwrite=1 swap=1 bin/sync_upload
35
					then rerun with l=1 ...
36
				make backups/TNRS.backup/download live=1
37
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
38
					then rerun with l=1 ...
39
	to synchronize a Mac's settings with my testing machine's:
40
		download:
41
			WARNING: this will overwrite all your user's settings!
42
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
43
				then rerun with l=1 ...
44
		upload:
45
			do step when changes are made on vegbiendev > on your machine, download
46
			on jupiter: (cd ~/Dropbox/svn/; up)
47
			on your machine:
48
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
49
					then rerun with l=1 ...
50
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"
51
					then rerun with l=1 ...
52
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
53
					then rerun with l=1 ...
54
		upload just the VirtualBox VMs:
55
			on your machine:
56
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
57
					then rerun with l=1 ...
58
	to backup files not in Time Machine:
59
		pg_ctl. stop # stop the PostgreSQL server
60
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/
61
			then rerun with l=1 ...
62
		pg_ctl. start # start the PostgreSQL server
63
	VegCore data dictionary:
64
		Regularly, or whenever the VegCore data dictionary page
65
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
66
			is changed, regenerate mappings/VegCore.csv:
67
			make mappings/VegCore.htm-remake; make mappings/
68
			svn di mappings/VegCore.tables.redmine
69
			If there are changes, update the data dictionary's Tables section
70
			When moving terms, check that no terms were lost: svn di
71
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
72
	Important: Whenever you install a system update that affects PostgreSQL or
73
		any of its dependencies, such as libc, you should restart the PostgreSQL
74
		server. Otherwise, you may get strange errors like "the database system
75
		is in recovery mode" which go away upon reimport, or you may not be able
76
		to access the database as the postgres superuser. This applies to both
77
		Linux and Mac OS X.
78

  
79 10
Single datasource import:
80 11
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 &
81 12
	(Re)import only: make inputs/<datasrc>/reimport by_col=1 &
......
221 152
			$version with the appropriate revision
222 153
		make backups/upload live=1
223 154

  
224
Backups:
225
	Archived imports:
226
		Back up: make backups/<version>.backup &
227
			Note: To back up the last import, you must archive it first:
228
				make schemas/rotate
229
		Test: make -s backups/<version>.backup/test &
230
		Restore: make backups/<version>.backup/restore &
231
		Remove: make backups/<version>.backup/remove
232
		Download: make backups/<version>.backup/download
233
	TNRS cache:
234
		Back up: make backups/TNRS.backup-remake &
235
			runtime: 3 min ("real 2m48.859s")
236
		Restore:
237
			yes|make inputs/.TNRS/uninstall
238
			make backups/TNRS.backup/restore &
239
				runtime: 5.5 min ("real 5m35.829s")
240
			yes|make schemas/public/reinstall
241
				Must come after TNRS restore to recreate tnrs_input_name view
242
	Full DB:
243
		Back up: make backups/vegbien.<version>.backup &
244
		Test: make -s backups/vegbien.<version>.backup/test &
245
		Restore: make backups/vegbien.<version>.backup/restore &
246
		Download: make backups/vegbien.<version>.backup/download
247
	Import logs:
248
		Download: make inputs/download-logs live=1
249

  
250 155
Datasource setup:
251 156
	umask ug=rwx,o= # prevent files from becoming web-accessible
252 157
	Add a new datasource: make inputs/<datasrc>/add
......
363 268
			make inputs/download live=1
364 269
			Follow the steps under Install the staging tables above
365 270

  
271
Maintenance:
272
	on a live machine, you should put the following in your .profile:
273
--
274
# make svn files web-accessible. this does not affect unversioned files, because
275
# these get the right permissions on the local machine instead.
276
umask ug=rwx,o=rx
277

  
278
unset TMOUT # TMOUT causes screen to exit even with background processes
279
--
280
	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:
281
		on vegbiendev: make postgres-Linux
282
	to synchronize vegbiendev, jupiter, and your local machine:
283
		WARNING: pay careful attention to all files that will be deleted or
284
			overwritten!
285
		install put if needed:
286
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
287
		when changes are made on vegbiendev:
288
			on all machines: ./fix_perms
289
			on vegbiendev, upload:
290
				overwrite=1 bin/sync_upload
291
					then rerun with l=1 ...
292
			on your machine, download:
293
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
294
					then rerun with l=1 ...
295
				overwrite=1 swap=1 bin/sync_upload
296
					then rerun with l=1 ...
297
				make backups/TNRS.backup/download live=1
298
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
299
					then rerun with l=1 ...
300
	to synchronize a Mac's settings with my testing machine's:
301
		download:
302
			WARNING: this will overwrite all your user's settings!
303
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
304
				then rerun with l=1 ...
305
		upload:
306
			do step when changes are made on vegbiendev > on your machine, download
307
			on jupiter: (cd ~/Dropbox/svn/; up)
308
			on your machine:
309
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
310
					then rerun with l=1 ...
311
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"
312
					then rerun with l=1 ...
313
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
314
					then rerun with l=1 ...
315
		upload just the VirtualBox VMs:
316
			on your machine:
317
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
318
					then rerun with l=1 ...
319
	to backup files not in Time Machine:
320
		pg_ctl. stop # stop the PostgreSQL server
321
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/
322
			then rerun with l=1 ...
323
		pg_ctl. start # start the PostgreSQL server
324
	VegCore data dictionary:
325
		Regularly, or whenever the VegCore data dictionary page
326
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
327
			is changed, regenerate mappings/VegCore.csv:
328
			make mappings/VegCore.htm-remake; make mappings/
329
			svn di mappings/VegCore.tables.redmine
330
			If there are changes, update the data dictionary's Tables section
331
			When moving terms, check that no terms were lost: svn di
332
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
333
	Important: Whenever you install a system update that affects PostgreSQL or
334
		any of its dependencies, such as libc, you should restart the PostgreSQL
335
		server. Otherwise, you may get strange errors like "the database system
336
		is in recovery mode" which go away upon reimport, or you may not be able
337
		to access the database as the postgres superuser. This applies to both
338
		Linux and Mac OS X.
339

  
340
Backups:
341
	Archived imports:
342
		Back up: make backups/<version>.backup &
343
			Note: To back up the last import, you must archive it first:
344
				make schemas/rotate
345
		Test: make -s backups/<version>.backup/test &
346
		Restore: make backups/<version>.backup/restore &
347
		Remove: make backups/<version>.backup/remove
348
		Download: make backups/<version>.backup/download
349
	TNRS cache:
350
		Back up: make backups/TNRS.backup-remake &
351
			runtime: 3 min ("real 2m48.859s")
352
		Restore:
353
			yes|make inputs/.TNRS/uninstall
354
			make backups/TNRS.backup/restore &
355
				runtime: 5.5 min ("real 5m35.829s")
356
			yes|make schemas/public/reinstall
357
				Must come after TNRS restore to recreate tnrs_input_name view
358
	Full DB:
359
		Back up: make backups/vegbien.<version>.backup &
360
		Test: make -s backups/vegbien.<version>.backup/test &
361
		Restore: make backups/vegbien.<version>.backup/restore &
362
		Download: make backups/vegbien.<version>.backup/download
363
	Import logs:
364
		Download: make inputs/download-logs live=1
365

  
366 366
Datasource refreshing:
367 367
	VegBank:
368 368
		make inputs/VegBank/vegbank.sql-remake

Also available in: Unified diff