Project

General

Profile

1
Installation:
2
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9

    
10
Connecting to vegbiendev:
11
	ssh vegbiendev.nceas.ucsb.edu
12
	sudo su - aaronmk
13
	cd /home/bien/svn # should happen automatically at login
14

    
15
Single datasource import:
16
	Follow the steps under Connecting to vegbiendev above
17
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 &
18
	(Re)import only: make inputs/<datasrc>/reimport by_col=1 &
19
	Note that these commands also work if the datasource is not yet imported
20
	Remake analytical DB: see Full database import > To remake analytical DB
21

    
22
Full database import:
23
	WARNING: You must perform *every single* step listed below, to avoid
24
		breaking column-based import
25
	Follow the steps under Connecting to vegbiendev above
26
	Follow the steps under Connecting to vegbiendev above, using jupiter instead
27
	On jupiter: svn up --force
28
	On local machine:
29
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
30
			your local machine"
31
		make inputs/upload
32
		make inputs/upload live=1
33
		make test by_col=1
34
			if you encounter errors, they are most likely related to the
35
				PostgreSQL error parsing in /lib/sql.py parse_exception()
36
			See note under Testing below
37
	On vegbiendev:
38
	Ensure there are no local modifications: svn st
39
	svn up
40
	make inputs/download
41
	make inputs/download live=1
42
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
43
	Update the auxiliary schemas: make schemas/reinstall
44
		The public schema will be installed separately by the import process
45
	Delete imports before the last so they won't bloat the full DB backup:
46
		make backups/vegbien.<version>.backup/remove
47
		To keep a previous import other than the public schema:
48
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
49
	Make sure there is at least 300GB of disk space on /: df -h
50
		The import schema is 230GB, and may use additional space for temp tables
51
		To free up space, remove backups that have been archived on jupiter:
52
			List backups/ to view older backups
53
			Check their MD5 sums using the steps under On jupiter below
54
			Remove these backups
55
	screen
56
	Press ENTER
57
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
58
	unset TMOUT # TMOUT causes screen to exit even with background processes
59
	unset version
60
	Start column-based import: . bin/import_all
61
		To use row-based import: . bin/import_all by_col=
62
		To stop all running imports: . bin/stop_imports
63
		WARNING: Do NOT run import_all in the background, or the jobs it creates
64
			won't be owned by your shell.
65
		Note that import_all will take up to an hour to import the NCBI backbone
66
			and other metadata before returning control to the shell.
67
	Wait (4 days) for the import to finish
68
	To recover from a closed terminal window: screen -r
69
	To restart an aborted import for a specific table:
70
		export version=<version>
71
		make inputs/<datasrc>/<table>/import_scrub by_col=1 continue=1 &
72
		bin/after_import $! & # $! can also be obtained from `jobs -l`
73
	Get $version: echo $version
74
	Set $version in all vegbiendev terminals: export version=<version>
75
	When there are no more running jobs, exit the screen
76
	Upload logs (run on vegbiendev): make inputs/upload live=1
77
	On local machine: make inputs/download-logs live=1
78
	In PostgreSQL:
79
		Go to wiki.vegpath.org/VegBIEN_contents
80
		Get the # observations (currently 84 million)
81
		Get the # datasources (currently 42)
82
		Get the # datasources with observations (currently 31)
83
		Check that analytical_stem contains [# observations] rows
84
		Check that source contains [# datasources] rows up through XAL
85
		Check that provider_count contains [# datasources with observations]
86
			rows with dataset="(total)" (at the top when the table is unsorted)
87
	Check that TNRS ran successfully:
88
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
89
		If the log ends in an AssertionError
90
			"assert sql.table_col_names(db, table) == header":
91
			Figure out which TNRS CSV columns have changed
92
			On local machine:
93
				Make the changes in the DB's TNRS and public schemas
94
				rm=1 inputs/.TNRS/schema.sql.run export_
95
				make schemas/remake
96
				inputs/test_taxonomic_names/test_scrub # re-run TNRS
97
				rm=1 inputs/.TNRS/data.sql.run export_
98
				Commit
99
			On vegbiendev:
100
				If dropping a column, save the dependent views
101
				Make the same changes in the live TNRS.tnrs table on vegbiendev
102
				If dropping a column, recreate the dependent views
103
				Restart the TNRS client: make scrub by_col=1 &
104
	tail inputs/{.,}*/*/logs/$version.log.sql
105
	In the output, search for "Command exited with non-zero status"
106
	For inputs that have this, fix the associated bug(s)
107
	If many inputs have errors, discard the current (partial) import:
108
		make schemas/$version/uninstall
109
	Otherwise, continue
110
	Publish the new import:
111
		WARNING: Before proceeding, be sure you have done *every single*
112
			verification step listed above. Otherwise, a previous valid import
113
			could incorrectly be overwritten with a broken one.
114
		make schemas/$version/publish # runtime: 1 min ("real 1m10.451s")
115
	unset version
116
	make backups/upload live=1
117
	On jupiter:
118
		cd /data/dev/aaronmk/bien/backups
119
		For each newly-archived backup:
120
			make -s <backup>.md5/test
121
			Check that "OK" is printed next to the filename
122
	On nimoy:
123
		cd /home/bien/svn/
124
		svn up
125
		export version=<version>
126
		make backups/analytical_stem.$version.csv/download
127
		In the bien_web DB:
128
			Create the analytical_stem_<version> table using its schema
129
				in schemas/vegbien.my.sql
130
		make -s backups/analytical_stem.$version.csv.md5/test
131
		Check that "OK" is printed next to the filename
132
		table=analytical_stem_$version bin/publish_analytical_db \
133
			backups/analytical_stem.$version.csv
134
	If desired, record the import times in inputs/import.stats.xls:
135
		Open inputs/import.stats.xls
136
		If the rightmost import is within 5 columns of column IV:
137
			Copy the current tab to <leftmost-date>~<rightmost-date>
138
			Remove the previous imports from the current tab because they are
139
				now in the copied tab instead
140
		Insert a copy of the leftmost "By column" column group before it
141
		export version=<version>
142
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
143
		Update the import date in the upper-right corner
144
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
145
		Paste the output over the # Rows/Time columns, making sure that the
146
			row counts match up with the previous import's row counts
147
		If the row counts do not match up, insert or reorder rows as needed
148
			until they do. Get the datasource names from the log file footers:
149
			tail inputs/{.,}*/*/logs/$version.log.sql
150
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
151
	Running individual steps separately:
152
	To run TNRS:
153
		To use an import other than public: export version=<version>
154
		make scrub &
155
		To view progress:
156
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
157
	To remake analytical DB:
158
		To use an import other than public: export version=<version>
159
		bin/make_analytical_db & # runtime: 13 h ("12:43:57elapsed")
160
		To view progress:
161
			tail -150 inputs/analytical_db/logs/make_analytical_db.log.sql
162
	To back up DB (staging tables and last import):
163
		To use an import *other than public*: export version=<version>
164
		make backups/TNRS.backup-remake &
165
		dump_opts=--exclude-schema=public make backups/vegbien.$version.backup/test &
166
			If after renaming to public, instead set dump_opts='' and replace
167
			$version with the appropriate revision
168
		make backups/upload live=1
169

    
170
Datasource setup:
171
	On local machine:
172
	Example steps for a datasource: wiki.vegpath.org/Import_process_for_Madidi
173
	umask ug=rwx,o= # prevent files from becoming web-accessible
174
	Add a new datasource: make inputs/<datasrc>/add
175
		<datasrc> may not contain spaces, and should be abbreviated.
176
		If the datasource is a herbarium, <datasrc> should be the herbarium code
177
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
178
	For a new-style datasource (one containing a ./run runscript):
179
		"cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/<datasrc>/
180
	For MySQL inputs (exports and live DB connections):
181
		For .sql exports:
182
			Place the original .sql file in _src/ (*not* in _MySQL/)
183
			Follow the steps starting with Install the staging tables below.
184
				This is for an initial sync to get the file onto vegbiendev.
185
			On vegbiendev:
186
				Create a database for the MySQL export in phpMyAdmin
187
				Give the bien user all database-specific privileges *except*
188
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
189
					import scripts from accidentally deleting data.
190
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
191
		mkdir inputs/<datasrc>/_MySQL/
192
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
193
		Edit _MySQL/*.make for the DB connection
194
			For a .sql export, use server=vegbiendev and --user=bien
195
		Skip the Add input data for each table section
196
	For MS Access databases:
197
		Place the .mdb or .accdb file in _src/
198
		Download and install Access To PostgreSQL from
199
			http://www.bullzip.com/download.php
200
		Use Access To PostgreSQL to export the database:
201
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
202
			Export just the data to inputs/<datasrc>/<file>.data.sql
203
		In <file>.schema.sql, make the following changes:
204
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
205
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
206
		Skip the Add input data for each table section
207
	Add input data for each table present in the datasource:
208
		For .sql exports, you must use the name of the table in the DB export
209
		For CSV files, you can use any name. It's recommended to use a table
210
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
211
		Note that if this table will be joined together with another table, its
212
			name must end in ".src"
213
		make inputs/<datasrc>/<table>/add
214
			Important: DO NOT just create an empty directory named <table>!
215
				This command also creates necessary subdirs, such as logs/.
216
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
217
			Otherwise, place the CSV(s) for the table in
218
			inputs/<datasrc>/<table>/ OR place a query joining other tables
219
			together in inputs/<datasrc>/<table>/create.sql
220
		Important: When exporting relational databases to CSVs, you MUST ensure
221
			that embedded quotes are escaped by doubling them, *not* by
222
			preceding them with a "\" as is the default in phpMyAdmin
223
		If there are multiple part files for a table, and the header is repeated
224
			in each part, make sure each header is EXACTLY the same.
225
			(If the headers are not the same, the CSV concatenation script
226
			assumes the part files don't have individual headers and treats the
227
			subsequent headers as data rows.)
228
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
229
			that depend on it
230
		For a new-style datasource:
231
			"cp" -f inputs/.NCBI/nodes/run inputs/<datasrc>/<table>/
232
			inputs/<datasrc>/<table>/run
233
	Install the staging tables:
234
		make inputs/<datasrc>/reinstall quiet=1 &
235
		For a MySQL .sql export:
236
			At prompt "[you]@vegbiendev's password:", enter your password
237
			At prompt "Enter password:", enter the value in config/bien_password
238
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
239
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
240
			tail provides a header line with the filename
241
			+1 starts at the first line, to show the whole file
242
		For every file with an error 'column "..." specified more than once':
243
			Add a header override file "+header.<ext>" in <table>/:
244
				Note: The leading "+" should sort it before the flat files.
245
					"_" unfortunately sorts *after* capital letters in ASCII.
246
				Create a text file containing the header line of the flat files
247
				Add an ! at the beginning of the line
248
					This signals cat_csv that this is a header override.
249
				For empty names, use their 0-based column # (by convention)
250
				For duplicate names, add a distinguishing suffix
251
				For long names that collided, rename them to <= 63 chars long
252
				Do NOT make readability changes in this step; that is what the
253
					map spreadsheets (below) are for.
254
				Save
255
		If you made any changes, re-run the install command above
256
	Auto-create the map spreadsheets: make inputs/<datasrc>/
257
	Map each table's columns:
258
		In each <table>/ subdir, for each "via map" map.csv:
259
			Open the map in a spreadsheet editor
260
			Open the "core map" /mappings/Veg+-VegBIEN.csv
261
			In each row of the via map, set the right column to a value from the
262
				left column of the core map
263
			Save
264
		Regenerate the derived maps: make inputs/<datasrc>/
265
	Accept the test cases:
266
		For a new-style datasource:
267
			inputs/<datasrc>/run
268
			svn di inputs/<datasrc>/*/test.xml.ref
269
			If you get errors, follow the steps for old-style datasources below
270
		For an old-style datasource:
271
			make inputs/<datasrc>/test
272
			When prompted to "Accept new test output", enter y and press ENTER
273
			If you instead get errors, do one of the following for each one:
274
			-	If the error was due to a bug, fix it
275
			-	Add a SQL function that filters or transforms the invalid data
276
			-	Make an empty mapping for the columns that produced the error.
277
				Put something in the Comments column of the map spreadsheet to
278
				prevent the automatic mapper from auto-removing the mapping.
279
			When accepting tests, it's helpful to use WinMerge
280
				(see WinMerge setup below for configuration)
281
		make inputs/<datasrc>/test by_col=1
282
			If you get errors this time, this always indicates a bug, usually in
283
				the VegBIEN unique constraints or column-based import itself
284
	Add newly-created files: make inputs/<datasrc>/add
285
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
286
	Update vegbiendev:
287
		On jupiter: svn up
288
		On local machine:
289
			./fix_perms
290
			make inputs/upload
291
			make inputs/upload live=1
292
		On vegbiendev:
293
			svn up
294
			make inputs/download
295
			make inputs/download live=1
296
			Follow the steps under Install the staging tables above
297

    
298
Maintenance:
299
	on a live machine, you should put the following in your .profile:
300
--
301
# make svn files web-accessible. this does not affect unversioned files, because
302
# these get the right permissions on the local machine instead.
303
umask ug=rwx,o=rx
304

    
305
unset TMOUT # TMOUT causes screen to exit even with background processes
306
--
307
	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:
308
		on vegbiendev: make postgres-Linux
309
	to synchronize vegbiendev, jupiter, and your local machine:
310
		WARNING: pay careful attention to all files that will be deleted or
311
			overwritten!
312
		install put if needed:
313
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
314
		when changes are made on vegbiendev:
315
			on all machines: ./fix_perms
316
			on vegbiendev, upload:
317
				overwrite=1 bin/sync_upload --size-only
318
					then rerun with l=1 ...
319
			on your machine, download:
320
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
321
					then rerun with l=1 ...
322
				overwrite=1 swap=1 bin/sync_upload --size-only
323
					then rerun with l=1 ...
324
				make backups/TNRS.backup/download live=1
325
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
326
					then rerun with l=1 ...
327
	to synchronize a Mac's settings with my testing machine's:
328
		download:
329
			WARNING: this will overwrite all your user's settings!
330
			on your machine:
331
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
332
				then rerun with l=1 ...
333
		upload:
334
			do step when changes are made on vegbiendev > on your machine, download
335
			on jupiter: (cd ~/Dropbox/svn/; up)
336
			on your machine:
337
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
338
					then rerun with l=1 ...
339
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"
340
					then rerun with l=1 ...
341
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
342
					then rerun with l=1 ...
343
		upload just the VirtualBox VMs:
344
			on your machine:
345
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
346
					then rerun with l=1 ...
347
	to backup files not in Time Machine:
348
		On local machine:
349
		pg_ctl. stop # stop the PostgreSQL server
350
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.1/data/
351
			then rerun with l=1 ...
352
		pg_ctl. start # start the PostgreSQL server
353
	VegCore data dictionary:
354
		Regularly, or whenever the VegCore data dictionary page
355
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
356
			is changed, regenerate mappings/VegCore.csv:
357
			On local machine:
358
			make mappings/VegCore.htm-remake; make mappings/
359
			svn di mappings/VegCore.tables.redmine
360
			If there are changes, update the data dictionary's Tables section
361
			When moving terms, check that no terms were lost: svn di
362
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
363
	Important: Whenever you install a system update that affects PostgreSQL or
364
		any of its dependencies, such as libc, you should restart the PostgreSQL
365
		server. Otherwise, you may get strange errors like "the database system
366
		is in recovery mode" which go away upon reimport, or you may not be able
367
		to access the database as the postgres superuser. This applies to both
368
		Linux and Mac OS X.
369

    
370
Backups:
371
	Archived imports:
372
		Follow the steps under Connecting to vegbiendev above
373
		Back up: make backups/<version>.backup &
374
			Note: To back up the last import, you must archive it first:
375
				make schemas/rotate
376
		Test: make -s backups/<version>.backup/test &
377
		Restore: make backups/<version>.backup/restore &
378
		Remove: make backups/<version>.backup/remove
379
		Download: make backups/<version>.backup/download
380
	TNRS cache:
381
		Follow the steps under Connecting to vegbiendev above
382
		Back up: make backups/TNRS.backup-remake &
383
			runtime: 3 min ("real 2m48.859s")
384
		Restore:
385
			yes|make inputs/.TNRS/uninstall
386
			make backups/TNRS.backup/restore &
387
				runtime: 5.5 min ("real 5m35.829s")
388
			yes|make schemas/public/reinstall
389
				Must come after TNRS restore to recreate tnrs_input_name view
390
	Full DB:
391
		Follow the steps under Connecting to vegbiendev above
392
		Back up: make backups/vegbien.<version>.backup &
393
		Test: make -s backups/vegbien.<version>.backup/test &
394
		Restore: make backups/vegbien.<version>.backup/restore &
395
		Download: make backups/vegbien.<version>.backup/download
396
	Import logs:
397
		On local machine:
398
		Download: make inputs/download-logs live=1
399

    
400
Datasource refreshing:
401
	VegBank:
402
		Follow the steps under Connecting to vegbiendev above
403
		make inputs/VegBank/vegbank.sql-remake
404
		make inputs/VegBank/reinstall quiet=1 &
405

    
406
Schema changes:
407
	On local machine:
408
	When changing the analytical views, run sync_analytical_..._to_view()
409
		to update the corresponding table
410
	Remember to update the following files with any renamings:
411
		schemas/filter_ERD.csv
412
		mappings/VegCore-VegBIEN.csv
413
		mappings/verify.*.sql
414
	Regenerate schema from installed DB: make schemas/remake
415
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
416
		WARNING: This will delete the current public schema of your VegBIEN DB!
417
	Reinstall staging tables:
418
		On local machine:
419
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
420
			make db
421
			. bin/reinstall_all
422
			Fix any bugs and retry until no errors
423
			make schemas/public/install
424
				This must be run *after* the datasources are installed, because
425
				views in public depend on some of the datasources
426
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
427
		On vegbiendev: repeat the above steps
428
			WARNING: Do not run this until reinstall_all runs successfully on
429
			the local machine, or the live DB may be unrestorable!
430
	Sync ERD with vegbien.sql schema:
431
		Run make schemas/vegbien.my.sql
432
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
433
		Go to File > Export > Synchronize With SQL CREATE Script...
434
		For Input File, select schemas/vegbien.my.sql
435
		Click Continue
436
		In the changes list, select each table with an arrow next to it
437
		Click Update Model
438
		Click Continue
439
		Note: The generated SQL script will be empty because we are syncing in
440
			the opposite direction
441
		Click Execute
442
		Reposition any lines that have been reset
443
		Add any new tables by dragging them from the Catalog in the left sidebar
444
			to the diagram
445
		Remove any deleted tables by right-clicking the table's diagram element,
446
			selecting Delete '<table name>', and clicking Delete
447
		Save
448
		If desired, update the graphical ERD exports (see below)
449
	Update graphical ERD exports:
450
		Go to File > Export > Export as PNG...
451
		Select schemas/vegbien.ERD.png and click Save
452
		Go to File > Export > Export as SVG...
453
		Select schemas/vegbien.ERD.svg and click Save
454
		Go to File > Export > Export as Single Page PDF...
455
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
456
		Go to File > Print...
457
		In the lower left corner, click PDF > Save as PDF...
458
		Set the Title and Author to ""
459
		Select schemas/vegbien.ERD.pdf and click Save
460
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
461
	Refactoring tips:
462
		To rename a table:
463
			In vegbien.sql, do the following:
464
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
465
					This is necessary because the table name is *everywhere*
466
				Search for <new>
467
				Manually change back any replacements inside comments
468
		To rename a column:
469
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
470
			Recreate any foreign key for the column, removing CONSTRAINT <name>
471
				This resets the foreign key name using the new column name
472
	Creating a poster of the ERD:
473
		Determine the poster size:
474
			Measure the line height (from the bottom of one line to the bottom
475
				of another): 16.3cm/24 lines = 0.679cm
476
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
477
			Zoom in as far as possible
478
			Measure the height of a capital letter: 3.5mm
479
			Measure the line height: 8.5mm
480
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
481
			Calculate the text height: 0.679cm*0.41 = 0.28cm
482
			Calculate the text height's fraction of the ERD height:
483
				0.28cm/70.8cm = 0.0040
484
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
485
			Calculate the VegBIEN poster height to make the text the same size:
486
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
487
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
488
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
489
			The minimum VegBIEN poster size is 35x54in portrait
490
		Determine the cost:
491
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
492
				charges the following for posters:
493
				base: $7.25/sq ft
494
				lamination: $3/sq ft
495
				mounting on a board: $8/sq ft
496

    
497
Testing:
498
	On a development machine, you should put the following in your .profile:
499
		umask ug=rwx,o= # prevent files from becoming web-accessible
500
		export log= n=2
501
	On local machine:
502
	Mapping process: make test
503
		Including column-based import: make test by_col=1
504
			If the row-based and column-based imports produce different inserted
505
			row counts, this usually means that a table is underconstrained
506
			(the unique indexes don't cover all possible rows).
507
			This can occur if you didn't use COALESCE(field, null_value) around
508
			a nullable field in a unique index. See sql_gen.null_sentinels for
509
			the appropriate null value to use.
510
	Map spreadsheet generation: make remake
511
	Missing mappings: make missing_mappings
512
	Everything (for most complete coverage): make test-all
513

    
514
Debugging:
515
	"Binary chop" debugging:
516
		(This is primarily useful for regressions that occurred in a previous
517
		revision, which was committed without running all the tests)
518
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
519
	.htaccess:
520
		mod_rewrite:
521
			IMPORTANT: whenever you change the DirectorySlash setting for a
522
				directory, you *must* clear your browser's cache to ensure that
523
				a cached redirect is not used. this is because RewriteRule
524
				redirects are (by default) temporary, but DirectorySlash
525
				redirects are permanent.
526
				for Firefox:
527
					press Cmd+Shift+Delete
528
					check only Cache
529
					press Enter or click Clear Now
530

    
531
WinMerge setup:
532
	In a Windows VM:
533
	Install WinMerge from <http://winmerge.org/>
534
	Open WinMerge
535
	Go to Edit > Options and click Compare in the left sidebar
536
	Enable "Moved block detection", as described at
537
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
538
	Set Whitespace to Ignore change, as described at
539
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
540

    
541
Documentation:
542
	To generate a Redmine-formatted list of steps for column-based import:
543
		On local machine:
544
		make schemas/public/reinstall
545
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
546
	To import and scrub just the test taxonomic names:
547
		Follow the steps under Connecting to vegbiendev above
548
		inputs/test_taxonomic_names/test_scrub
549

    
550
General:
551
	To see a program's description, read its top-of-file comment
552
	To see a program's usage, run it without arguments
553
	To remake a directory: make <dir>/remake
554
	To remake a file: make <file>-remake
(5-5/10)