Project

General

Profile

1
Installation:
2
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9

    
10
Connecting to vegbiendev:
11
	ssh vegbiendev.nceas.ucsb.edu
12
	sudo su - aaronmk
13
	cd /home/bien/svn # should happen automatically at login
14

    
15
Single datasource import:
16
	Follow the steps under Connecting to vegbiendev above
17
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 &
18
	(Re)import only: make inputs/<datasrc>/reimport by_col=1 &
19
	Note that these commands also work if the datasource is not yet imported
20
	Remake analytical DB: see Full database import > To remake analytical DB
21

    
22
Full database import:
23
	WARNING: You must perform *every single* step listed below, to avoid
24
		breaking column-based import
25
	Follow the steps under Connecting to vegbiendev above
26
	Follow the steps under Connecting to vegbiendev above, using jupiter instead
27
	On jupiter: svn up --force
28
	On local machine:
29
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
30
			your local machine"
31
		make inputs/upload
32
		make inputs/upload live=1
33
		make test by_col=1
34
			if you encounter errors, they are most likely related to the
35
				PostgreSQL error parsing in /lib/sql.py parse_exception()
36
			See note under Testing below
37
	On vegbiendev:
38
	Ensure there are no local modifications: svn st
39
	svn up
40
	make inputs/download
41
	make inputs/download live=1
42
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
43
	Update the auxiliary schemas: make schemas/reinstall
44
		The public schema will be installed separately by the import process
45
	Delete imports before the last so they won't bloat the full DB backup:
46
		make backups/vegbien.<version>.backup/remove
47
		To keep a previous import other than the public schema:
48
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
49
	Make sure there is at least 300GB of disk space on /: df -h
50
		The import schema is 230GB, and may use additional space for temp tables
51
		To free up space, remove backups that have been archived on jupiter:
52
			List backups/ to view older backups
53
			Check their MD5 sums using the steps under On jupiter below
54
			Remove these backups
55
	screen
56
	Press ENTER
57
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
58
	unset TMOUT # TMOUT causes screen to exit even with background processes
59
	unset version
60
	Start column-based import: . bin/import_all
61
		To use row-based import: . bin/import_all by_col=
62
		To stop all running imports: . bin/stop_imports
63
		WARNING: Do NOT run import_all in the background, or the jobs it creates
64
			won't be owned by your shell.
65
		Note that import_all will take up to an hour to import the NCBI backbone
66
			and other metadata before returning control to the shell.
67
	Wait (4 days) for the import to finish
68
	To recover from a closed terminal window: screen -r
69
	To restart an aborted import for a specific table:
70
		export version=<version>
71
		make inputs/<datasrc>/<table>/import_scrub by_col=1 continue=1 &
72
		bin/after_import $! & # $! can also be obtained from `jobs -l`
73
	Get $version: echo $version
74
	Set $version in all vegbiendev terminals: export version=<version>
75
	When there are no more running jobs, exit the screen
76
	Upload logs (run on vegbiendev): make inputs/upload live=1
77
	On local machine: make inputs/download-logs live=1
78
	In PostgreSQL:
79
		Check that the provider_count and source tables contain entries for all
80
			inputs
81
	Check that TNRS ran successfully:
82
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
83
		If the log ends in an AssertionError
84
			"assert sql.table_col_names(db, table) == header":
85
			Figure out which TNRS CSV columns have changed
86
			On local machine:
87
				Make the changes in the DB's TNRS and public schemas
88
				rm=1 inputs/.TNRS/schema.sql.run export_
89
				make schemas/remake
90
				inputs/test_taxonomic_names/test_scrub # re-run TNRS
91
				rm=1 inputs/.TNRS/data.sql.run export_
92
				Commit
93
			On vegbiendev:
94
				If dropping a column, save the dependent views
95
				Make the same changes in the live TNRS.tnrs table on vegbiendev
96
				If dropping a column, recreate the dependent views
97
				Restart the TNRS client: make scrub by_col=1 &
98
	tail inputs/{.,}*/*/logs/$version.log.sql
99
	In the output, search for "Command exited with non-zero status"
100
	For inputs that have this, fix the associated bug(s)
101
	If many inputs have errors, discard the current (partial) import:
102
		make schemas/$version/uninstall
103
	Otherwise, continue
104
	Publish the new import:
105
		WARNING: Before proceeding, be sure you have done *every single*
106
			verification step listed above. Otherwise, a previous valid import
107
			could incorrectly be overwritten with a broken one.
108
		make schemas/$version/publish # runtime: 1 min ("real 1m10.451s")
109
	unset version
110
	make backups/upload live=1
111
	On jupiter:
112
		cd /data/dev/aaronmk/bien/backups
113
		For each newly-archived backup:
114
			make -s <backup>.md5/test
115
			Check that "OK" is printed next to the filename
116
	On nimoy:
117
		cd /home/bien/svn/
118
		svn up
119
		export version=<version>
120
		make backups/analytical_stem.$version.csv/download
121
		In the bien_web DB:
122
			Create the analytical_stem_<version> table using its schema
123
				in schemas/vegbien.my.sql
124
		make -s backups/analytical_stem.$version.csv.md5/test
125
		Check that "OK" is printed next to the filename
126
		table=analytical_stem_$version bin/publish_analytical_db \
127
			backups/analytical_stem.$version.csv
128
	If desired, record the import times in inputs/import.stats.xls:
129
		Open inputs/import.stats.xls
130
		If the rightmost import is within 5 columns of column IV:
131
			Copy the current tab to <leftmost-date>~<rightmost-date>
132
			Remove the previous imports from the current tab because they are
133
				now in the copied tab instead
134
		Insert a copy of the leftmost "By column" column group before it
135
		export version=<version>
136
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
137
		Update the import date in the upper-right corner
138
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
139
		Paste the output over the # Rows/Time columns, making sure that the
140
			row counts match up with the previous import's row counts
141
		If the row counts do not match up, insert or reorder rows as needed
142
			until they do. Get the datasource names from the log file footers:
143
			tail inputs/{.,}*/*/logs/$version.log.sql
144
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
145
	Running individual steps separately:
146
	To run TNRS:
147
		To use an import other than public: export version=<version>
148
		make scrub &
149
		To view progress:
150
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
151
	To remake analytical DB:
152
		To use an import other than public: export version=<version>
153
		bin/make_analytical_db & # runtime: 13 h ("12:43:57elapsed")
154
		To view progress:
155
			tail -150 inputs/analytical_db/logs/make_analytical_db.log.sql
156
	To back up DB (staging tables and last import):
157
		To use an import *other than public*: export version=<version>
158
		make backups/TNRS.backup-remake &
159
		dump_opts=--exclude-schema=public make backups/vegbien.$version.backup/test &
160
			If after renaming to public, instead set dump_opts='' and replace
161
			$version with the appropriate revision
162
		make backups/upload live=1
163

    
164
Datasource setup:
165
	On local machine:
166
	Example steps for a datasource: wiki.vegpath.org/Import_process_for_Madidi
167
	umask ug=rwx,o= # prevent files from becoming web-accessible
168
	Add a new datasource: make inputs/<datasrc>/add
169
		<datasrc> may not contain spaces, and should be abbreviated.
170
		If the datasource is a herbarium, <datasrc> should be the herbarium code
171
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
172
	For a new-style datasource (one containing a ./run runscript):
173
		"cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/<datasrc>/
174
	For MySQL inputs (exports and live DB connections):
175
		For .sql exports:
176
			Place the original .sql file in _src/ (*not* in _MySQL/)
177
			Follow the steps starting with Install the staging tables below.
178
				This is for an initial sync to get the file onto vegbiendev.
179
			On vegbiendev:
180
				Create a database for the MySQL export in phpMyAdmin
181
				Give the bien user all database-specific privileges *except*
182
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
183
					import scripts from accidentally deleting data.
184
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
185
		mkdir inputs/<datasrc>/_MySQL/
186
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
187
		Edit _MySQL/*.make for the DB connection
188
			For a .sql export, use server=vegbiendev and --user=bien
189
		Skip the Add input data for each table section
190
	For MS Access databases:
191
		Place the .mdb or .accdb file in _src/
192
		Download and install Access To PostgreSQL from
193
			http://www.bullzip.com/download.php
194
		Use Access To PostgreSQL to export the database:
195
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
196
			Export just the data to inputs/<datasrc>/<file>.data.sql
197
		In <file>.schema.sql, make the following changes:
198
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
199
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
200
		Skip the Add input data for each table section
201
	Add input data for each table present in the datasource:
202
		For .sql exports, you must use the name of the table in the DB export
203
		For CSV files, you can use any name. It's recommended to use a table
204
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
205
		Note that if this table will be joined together with another table, its
206
			name must end in ".src"
207
		make inputs/<datasrc>/<table>/add
208
			Important: DO NOT just create an empty directory named <table>!
209
				This command also creates necessary subdirs, such as logs/.
210
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
211
			Otherwise, place the CSV(s) for the table in
212
			inputs/<datasrc>/<table>/ OR place a query joining other tables
213
			together in inputs/<datasrc>/<table>/create.sql
214
		Important: When exporting relational databases to CSVs, you MUST ensure
215
			that embedded quotes are escaped by doubling them, *not* by
216
			preceding them with a "\" as is the default in phpMyAdmin
217
		If there are multiple part files for a table, and the header is repeated
218
			in each part, make sure each header is EXACTLY the same.
219
			(If the headers are not the same, the CSV concatenation script
220
			assumes the part files don't have individual headers and treats the
221
			subsequent headers as data rows.)
222
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
223
			that depend on it
224
		For a new-style datasource:
225
			"cp" -f inputs/.NCBI/nodes/run inputs/<datasrc>/<table>/
226
			inputs/<datasrc>/<table>/run
227
	Install the staging tables:
228
		make inputs/<datasrc>/reinstall quiet=1 &
229
		For a MySQL .sql export:
230
			At prompt "[you]@vegbiendev's password:", enter your password
231
			At prompt "Enter password:", enter the value in config/bien_password
232
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
233
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
234
			tail provides a header line with the filename
235
			+1 starts at the first line, to show the whole file
236
		For every file with an error 'column "..." specified more than once':
237
			Add a header override file "+header.<ext>" in <table>/:
238
				Note: The leading "+" should sort it before the flat files.
239
					"_" unfortunately sorts *after* capital letters in ASCII.
240
				Create a text file containing the header line of the flat files
241
				Add an ! at the beginning of the line
242
					This signals cat_csv that this is a header override.
243
				For empty names, use their 0-based column # (by convention)
244
				For duplicate names, add a distinguishing suffix
245
				For long names that collided, rename them to <= 63 chars long
246
				Do NOT make readability changes in this step; that is what the
247
					map spreadsheets (below) are for.
248
				Save
249
		If you made any changes, re-run the install command above
250
	Auto-create the map spreadsheets: make inputs/<datasrc>/
251
	Map each table's columns:
252
		In each <table>/ subdir, for each "via map" map.csv:
253
			Open the map in a spreadsheet editor
254
			Open the "core map" /mappings/Veg+-VegBIEN.csv
255
			In each row of the via map, set the right column to a value from the
256
				left column of the core map
257
			Save
258
		Regenerate the derived maps: make inputs/<datasrc>/
259
	Accept the test cases:
260
		For a new-style datasource:
261
			inputs/<datasrc>/run
262
			svn di inputs/<datasrc>/*/test.xml.ref
263
			If you get errors, follow the steps for old-style datasources below
264
		For an old-style datasource:
265
			make inputs/<datasrc>/test
266
			When prompted to "Accept new test output", enter y and press ENTER
267
			If you instead get errors, do one of the following for each one:
268
			-	If the error was due to a bug, fix it
269
			-	Add a SQL function that filters or transforms the invalid data
270
			-	Make an empty mapping for the columns that produced the error.
271
				Put something in the Comments column of the map spreadsheet to
272
				prevent the automatic mapper from auto-removing the mapping.
273
			When accepting tests, it's helpful to use WinMerge
274
				(see WinMerge setup below for configuration)
275
		make inputs/<datasrc>/test by_col=1
276
			If you get errors this time, this always indicates a bug, usually in
277
				the VegBIEN unique constraints or column-based import itself
278
	Add newly-created files: make inputs/<datasrc>/add
279
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
280
	Update vegbiendev:
281
		On jupiter: svn up
282
		On local machine:
283
			./fix_perms
284
			make inputs/upload
285
			make inputs/upload live=1
286
		On vegbiendev:
287
			svn up
288
			make inputs/download
289
			make inputs/download live=1
290
			Follow the steps under Install the staging tables above
291

    
292
Maintenance:
293
	on a live machine, you should put the following in your .profile:
294
--
295
# make svn files web-accessible. this does not affect unversioned files, because
296
# these get the right permissions on the local machine instead.
297
umask ug=rwx,o=rx
298

    
299
unset TMOUT # TMOUT causes screen to exit even with background processes
300
--
301
	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:
302
		on vegbiendev: make postgres-Linux
303
	to synchronize vegbiendev, jupiter, and your local machine:
304
		WARNING: pay careful attention to all files that will be deleted or
305
			overwritten!
306
		install put if needed:
307
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
308
		when changes are made on vegbiendev:
309
			on all machines: ./fix_perms
310
			on vegbiendev, upload:
311
				overwrite=1 bin/sync_upload --size-only
312
					then rerun with l=1 ...
313
			on your machine, download:
314
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
315
					then rerun with l=1 ...
316
				overwrite=1 swap=1 bin/sync_upload --size-only
317
					then rerun with l=1 ...
318
				make backups/TNRS.backup/download live=1
319
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
320
					then rerun with l=1 ...
321
	to synchronize a Mac's settings with my testing machine's:
322
		download:
323
			WARNING: this will overwrite all your user's settings!
324
			on your machine:
325
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
326
				then rerun with l=1 ...
327
		upload:
328
			do step when changes are made on vegbiendev > on your machine, download
329
			on jupiter: (cd ~/Dropbox/svn/; up)
330
			on your machine:
331
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
332
					then rerun with l=1 ...
333
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"
334
					then rerun with l=1 ...
335
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
336
					then rerun with l=1 ...
337
		upload just the VirtualBox VMs:
338
			on your machine:
339
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
340
					then rerun with l=1 ...
341
	to backup files not in Time Machine:
342
		On local machine:
343
		pg_ctl. stop # stop the PostgreSQL server
344
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.1/data/
345
			then rerun with l=1 ...
346
		pg_ctl. start # start the PostgreSQL server
347
	VegCore data dictionary:
348
		Regularly, or whenever the VegCore data dictionary page
349
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
350
			is changed, regenerate mappings/VegCore.csv:
351
			On local machine:
352
			make mappings/VegCore.htm-remake; make mappings/
353
			svn di mappings/VegCore.tables.redmine
354
			If there are changes, update the data dictionary's Tables section
355
			When moving terms, check that no terms were lost: svn di
356
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
357
	Important: Whenever you install a system update that affects PostgreSQL or
358
		any of its dependencies, such as libc, you should restart the PostgreSQL
359
		server. Otherwise, you may get strange errors like "the database system
360
		is in recovery mode" which go away upon reimport, or you may not be able
361
		to access the database as the postgres superuser. This applies to both
362
		Linux and Mac OS X.
363

    
364
Backups:
365
	Archived imports:
366
		Follow the steps under Connecting to vegbiendev above
367
		Back up: make backups/<version>.backup &
368
			Note: To back up the last import, you must archive it first:
369
				make schemas/rotate
370
		Test: make -s backups/<version>.backup/test &
371
		Restore: make backups/<version>.backup/restore &
372
		Remove: make backups/<version>.backup/remove
373
		Download: make backups/<version>.backup/download
374
	TNRS cache:
375
		Follow the steps under Connecting to vegbiendev above
376
		Back up: make backups/TNRS.backup-remake &
377
			runtime: 3 min ("real 2m48.859s")
378
		Restore:
379
			yes|make inputs/.TNRS/uninstall
380
			make backups/TNRS.backup/restore &
381
				runtime: 5.5 min ("real 5m35.829s")
382
			yes|make schemas/public/reinstall
383
				Must come after TNRS restore to recreate tnrs_input_name view
384
	Full DB:
385
		Follow the steps under Connecting to vegbiendev above
386
		Back up: make backups/vegbien.<version>.backup &
387
		Test: make -s backups/vegbien.<version>.backup/test &
388
		Restore: make backups/vegbien.<version>.backup/restore &
389
		Download: make backups/vegbien.<version>.backup/download
390
	Import logs:
391
		On local machine:
392
		Download: make inputs/download-logs live=1
393

    
394
Datasource refreshing:
395
	VegBank:
396
		Follow the steps under Connecting to vegbiendev above
397
		make inputs/VegBank/vegbank.sql-remake
398
		make inputs/VegBank/reinstall quiet=1 &
399

    
400
Schema changes:
401
	On local machine:
402
	When changing the analytical views, run sync_analytical_..._to_view()
403
		to update the corresponding table
404
	Remember to update the following files with any renamings:
405
		schemas/filter_ERD.csv
406
		mappings/VegCore-VegBIEN.csv
407
		mappings/verify.*.sql
408
	Regenerate schema from installed DB: make schemas/remake
409
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
410
		WARNING: This will delete the current public schema of your VegBIEN DB!
411
	Reinstall staging tables:
412
		On local machine:
413
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
414
			make db
415
			. bin/reinstall_all
416
			Fix any bugs and retry until no errors
417
			make schemas/public/install
418
				This must be run *after* the datasources are installed, because
419
				views in public depend on some of the datasources
420
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
421
		On vegbiendev: repeat the above steps
422
			WARNING: Do not run this until reinstall_all runs successfully on
423
			the local machine, or the live DB may be unrestorable!
424
	Sync ERD with vegbien.sql schema:
425
		Run make schemas/vegbien.my.sql
426
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
427
		Go to File > Export > Synchronize With SQL CREATE Script...
428
		For Input File, select schemas/vegbien.my.sql
429
		Click Continue
430
		In the changes list, select each table with an arrow next to it
431
		Click Update Model
432
		Click Continue
433
		Note: The generated SQL script will be empty because we are syncing in
434
			the opposite direction
435
		Click Execute
436
		Reposition any lines that have been reset
437
		Add any new tables by dragging them from the Catalog in the left sidebar
438
			to the diagram
439
		Remove any deleted tables by right-clicking the table's diagram element,
440
			selecting Delete '<table name>', and clicking Delete
441
		Save
442
		If desired, update the graphical ERD exports (see below)
443
	Update graphical ERD exports:
444
		Go to File > Export > Export as PNG...
445
		Select schemas/vegbien.ERD.png and click Save
446
		Go to File > Export > Export as SVG...
447
		Select schemas/vegbien.ERD.svg and click Save
448
		Go to File > Export > Export as Single Page PDF...
449
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
450
		Go to File > Print...
451
		In the lower left corner, click PDF > Save as PDF...
452
		Set the Title and Author to ""
453
		Select schemas/vegbien.ERD.pdf and click Save
454
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
455
	Refactoring tips:
456
		To rename a table:
457
			In vegbien.sql, do the following:
458
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
459
					This is necessary because the table name is *everywhere*
460
				Search for <new>
461
				Manually change back any replacements inside comments
462
		To rename a column:
463
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
464
			Recreate any foreign key for the column, removing CONSTRAINT <name>
465
				This resets the foreign key name using the new column name
466
	Creating a poster of the ERD:
467
		Determine the poster size:
468
			Measure the line height (from the bottom of one line to the bottom
469
				of another): 16.3cm/24 lines = 0.679cm
470
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
471
			Zoom in as far as possible
472
			Measure the height of a capital letter: 3.5mm
473
			Measure the line height: 8.5mm
474
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
475
			Calculate the text height: 0.679cm*0.41 = 0.28cm
476
			Calculate the text height's fraction of the ERD height:
477
				0.28cm/70.8cm = 0.0040
478
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
479
			Calculate the VegBIEN poster height to make the text the same size:
480
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
481
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
482
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
483
			The minimum VegBIEN poster size is 35x54in portrait
484
		Determine the cost:
485
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
486
				charges the following for posters:
487
				base: $7.25/sq ft
488
				lamination: $3/sq ft
489
				mounting on a board: $8/sq ft
490

    
491
Testing:
492
	On a development machine, you should put the following in your .profile:
493
		umask ug=rwx,o= # prevent files from becoming web-accessible
494
		export log= n=2
495
	On local machine:
496
	Mapping process: make test
497
		Including column-based import: make test by_col=1
498
			If the row-based and column-based imports produce different inserted
499
			row counts, this usually means that a table is underconstrained
500
			(the unique indexes don't cover all possible rows).
501
			This can occur if you didn't use COALESCE(field, null_value) around
502
			a nullable field in a unique index. See sql_gen.null_sentinels for
503
			the appropriate null value to use.
504
	Map spreadsheet generation: make remake
505
	Missing mappings: make missing_mappings
506
	Everything (for most complete coverage): make test-all
507

    
508
Debugging:
509
	"Binary chop" debugging:
510
		(This is primarily useful for regressions that occurred in a previous
511
		revision, which was committed without running all the tests)
512
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
513
	.htaccess:
514
		mod_rewrite:
515
			IMPORTANT: whenever you change the DirectorySlash setting for a
516
				directory, you *must* clear your browser's cache to ensure that
517
				a cached redirect is not used. this is because RewriteRule
518
				redirects are (by default) temporary, but DirectorySlash
519
				redirects are permanent.
520
				for Firefox:
521
					press Cmd+Shift+Delete
522
					check only Cache
523
					press Enter or click Clear Now
524

    
525
WinMerge setup:
526
	In a Windows VM:
527
	Install WinMerge from <http://winmerge.org/>
528
	Open WinMerge
529
	Go to Edit > Options and click Compare in the left sidebar
530
	Enable "Moved block detection", as described at
531
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
532
	Set Whitespace to Ignore change, as described at
533
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
534

    
535
Documentation:
536
	To generate a Redmine-formatted list of steps for column-based import:
537
		On local machine:
538
		make schemas/public/reinstall
539
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
540
	To import and scrub just the test taxonomic names:
541
		Follow the steps under Connecting to vegbiendev above
542
		inputs/test_taxonomic_names/test_scrub
543

    
544
General:
545
	To see a program's description, read its top-of-file comment
546
	To see a program's usage, run it without arguments
547
	To remake a directory: make <dir>/remake
548
	To remake a file: make <file>-remake
(5-5/10)