Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 3674 aaronmk
Maintenance:
11 9399 aaronmk
	on a live machine, you should put the following in your .profile:
12
		umask ug=rwx,o= # prevent files from becoming web-accessible
13 9400 aaronmk
		unset TMOUT # TMOUT causes screen to exit even with background processes
14 8459 aaronmk
	to synchronize vegbiendev, jupiter, and your local machine:
15 9886 aaronmk
		WARNING: pay careful attention to all files that will be deleted or
16
			overwritten!
17 8460 aaronmk
		install put if needed:
18
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
19 8459 aaronmk
		when changes are made on vegbiendev:
20
			on vegbiendev, upload:
21 8723 aaronmk
				env overwrite=1             src=. dest='aaronmk@jupiter:~/bien' put --no-group --exclude=.svn --exclude=install.log.sql --exclude='*.backup' --exclude='/backups/analytical_aggregate.*.csv' --exclude='inputs/GBIF/**.data.sql' --exclude='bin/dotlockfile'
22 8459 aaronmk
					then rerun with env l=1 ...
23 8723 aaronmk
				env overwrite=1 del=        src=. dest='aaronmk@jupiter:~/bien' put --no-group --exclude=.svn --exclude=install.log.sql --exclude='inputs/GBIF/**.data.sql'
24 8459 aaronmk
					then rerun with env l=1 ...
25
			on your machine, download:
26 8945 aaronmk
				env overwrite=1      swap=1 src=. dest='aaronmk@jupiter:~/bien' put            --exclude=.svn inputs/VegBIEN/TWiki
27
					then rerun with env l=1 ...
28 8723 aaronmk
				env overwrite=1 del= swap=1 src=. dest='aaronmk@jupiter:~/bien' put            --exclude=.svn --exclude=install.log.sql --exclude='*.backup' --exclude='/backups/analytical_aggregate.*.csv' --exclude='inputs/GBIF/**.data.sql' --exclude='bin/dotlockfile'
29 8459 aaronmk
					then rerun with env l=1 ...
30 8462 aaronmk
	to synchronize a Mac's settings with my testing machine's:
31
		download:
32
			WARNING: this will overwrite all your user's settings!
33 8718 aaronmk
			env overwrite=1 swap=1 src=~/Library/ dest='aaronmk@jupiter:~/Library/' put --exclude="/Saved Application State"
34 8462 aaronmk
				then rerun with env l=1 ...
35
		upload:
36 8719 aaronmk
			do step when changes are made on vegbiendev > on your machine, download
37 8724 aaronmk
			env overwrite=1        src=~ dest='aaronmk@jupiter:~' put --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Documents/BIEN/large_files" --exclude="/Documents/BIEN/svn/backups/*.backup" --exclude="/Documents/BIEN/svn/backups/analytical_aggregate.*.csv" --exclude="/Dropbox/home"
38 8462 aaronmk
				then rerun with env l=1 ...
39 8719 aaronmk
			env overwrite=1        src=~ dest=~/Dropbox/home      put --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
40 8717 aaronmk
				then rerun with env l=1 ...
41 8720 aaronmk
	to backup files not in Time Machine:
42
		env inplace=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ put Users/aaronmk/VirtualBox\ VMs/
43
			then rerun with env l=1 ...
44
		stop the PostgreSQL server
45
		sudo env src=/ dest=/Volumes/Time\ Machine\ Backups/ put Library/PostgreSQL/9.1/data/
46
			then rerun with env l=1 ...
47
		start the PostgreSQL server
48 8458 aaronmk
	VegCore data dictionary:
49
		Regularly, or whenever the VegCore data dictionary page
50
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
51
			is changed, regenerate mappings/VegCore.csv:
52
			make mappings/VegCore.htm-remake; make mappings/
53
			svn di mappings/VegCore.tables.redmine
54
			If there are changes, update the data dictionary's Tables section
55
			When moving terms, check that no terms were lost: svn di
56
			svn ci -m "mappings/VegCore.htm: Regenerated from wiki"
57
	Important: Whenever you install a system update that affects PostgreSQL or
58
		any of its dependencies, such as libc, you should restart the PostgreSQL
59
		server. Otherwise, you may get strange errors like "the database system
60
		is in recovery mode" which go away upon reimport, or you may not be able
61
		to access the database as the postgres superuser. This applies to both
62
		Linux and Mac OS X.
63 3674 aaronmk
64 7287 aaronmk
Single datasource import:
65 8458 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1
66
	(Re)import only: make inputs/<datasrc>/reimport by_col=1
67
	(Re)scrub: make inputs/<datasrc>/rescrub by_col=1
68
	Note that these commands also work if the datasource is not yet imported
69 7287 aaronmk
70
Full database import:
71 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
72
		breaking column-based import
73 9883 aaronmk
	On jupiter: svn up --force
74 8458 aaronmk
	On local machine:
75
		./fix_perms
76 9887 aaronmk
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
77
			your local machine" above
78 9884 aaronmk
		make inputs/upload test=1
79 8458 aaronmk
		make inputs/upload
80
		make test by_col=1
81
			See note under Testing below
82
	On vegbiendev:
83
	Ensure there are no local modifications: svn st
84
	svn up
85 9884 aaronmk
	make inputs/download test=1
86 8458 aaronmk
	make inputs/download
87
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
88
	Update the auxiliary schemas: make schemas/reinstall
89
		The public schema will be installed separately by the import process
90
	Delete imports before the last so they won't bloat the full DB backup:
91
		make backups/vegbien.<version>.backup/remove
92
		To keep a previous import other than the public schema:
93
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
94 9970 aaronmk
	Make sure there is at least 300GB of disk space on /: df -h
95
		The import schema is 255GB, and may use additional space for temp tables
96 8458 aaronmk
		To free up space, remove backups that have been archived on jupiter:
97
			List backups/ to view older backups
98
			Check their MD5 sums using the steps under On jupiter below
99
			Remove these backups
100 9899 aaronmk
	Remove any leftover TNRS lockfile: rm inputs/.TNRS/tnrs/tnrs.make.lock
101
		Usually, the PID in it would not exist, but sometimes it now refers to
102
			a different, active process which blocks tnrs.make
103 8458 aaronmk
	screen
104
	Press ENTER
105 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
106 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
107
	unset version
108 8458 aaronmk
	Start column-based import: . bin/import_all by_col=1
109
		To use row-based import: . bin/import_all
110
		To stop all running imports: . bin/stop_imports
111
		WARNING: Do NOT run import_all in the background, or the jobs it creates
112
			won't be owned by your shell.
113
		Note that import_all will take up to an hour to import the NCBI backbone
114
			and other metadata before returning control to the shell.
115
	Wait (overnight) for the import to finish
116
	To recover from a closed terminal window: screen -r
117
	When there are no more running jobs, exit the screen
118
	Get $version: echo $version
119
	Set $version in all vegbiendev terminals: export version=<version>
120
	Upload logs (run on vegbiendev): make inputs/upload
121
	On local machine: make inputs/download-logs
122
	In PostgreSQL:
123
		Check that the provider_count and source tables contain entries for all
124
			inputs
125 9492 aaronmk
	Check that TNRS ran successfully:
126
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
127
		If the log ends in an AssertionError
128
			"assert sql.table_col_names(db, table) == header":
129
			Figure out which TNRS CSV columns have changed
130
			On local machine:
131
				Make the changes in inputs/.TNRS/schema.sql
132
				rm inputs/.TNRS/tnrs/header.csv
133
				make inputs/.TNRS/reinstall schema_only=1
134
				make schemas/public/reinstall
135
				If there are errors "column ... does not exist", etc.:
136
					Make the necessary changes in schemas/vegbien.sql
137
					make schemas/public/reinstall
138
				make schemas/remake
139
				inputs/test_taxonomic_names/test_scrub
140 9528 aaronmk
				In inputs/test_taxonomic_names/_scrub/TNRS.sql, copy the
141
					"COPY tnrs ..." statement to inputs/.TNRS/data.sql
142 9492 aaronmk
				Commit
143
			On vegbiendev:
144
				If dropping a column, save the dependent views
145
				Make the same changes in the live TNRS.tnrs table on vegbiendev
146
				If dropping a column, recreate the dependent views
147
				Restart the TNRS client: make scrub by_col=1 &
148 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
149
	In the output, search for "Command exited with non-zero status"
150
	For inputs that have this, fix the associated bug(s)
151
	If many inputs have errors, discard the current (partial) import:
152
		make schemas/$version/uninstall
153
	Otherwise, continue
154 9498 aaronmk
	Publish the new import:
155
		WARNING: Before proceeding, be sure you have done *every single*
156
			verification step listed above. Otherwise, a previous valid import
157
			could incorrectly be overwritten with a broken one.
158
		make schemas/$version/publish
159 8458 aaronmk
	unset version
160
	backups/fix_perms
161
	make backups/upload
162
	On jupiter:
163
		cd /data/dev/aaronmk/bien/backups
164
		For each newly-archived backup:
165
			make -s <backup>.md5/test
166
			Check that "OK" is printed next to the filename
167
	On nimoy:
168
		cd /home/bien/svn/
169
		svn up
170
		export version=<version>
171
		make backups/analytical_stem.$version.csv/download
172
		In the bien_web DB:
173
			Create the analytical_stem_<version> table using its schema
174
				in schemas/vegbien.my.sql
175
		make -s backups/analytical_stem.$version.csv.md5/test
176
		Check that "OK" is printed next to the filename
177
		env table=analytical_stem_$version bin/publish_analytical_db \
178
			backups/analytical_stem.$version.csv
179
	If desired, record the import times in inputs/import.stats.xls:
180
		Open inputs/import.stats.xls
181
		If the rightmost import is within 5 columns of column IV:
182
			Copy the current tab to <leftmost-date>~<rightmost-date>
183
			Remove the previous imports from the current tab because they are
184
				now in the copied tab instead
185
		Insert a copy of the leftmost "By column" column group before it
186
		export version=<version>
187
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
188
		Update the import date in the upper-right corner
189
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
190
		Paste the output over the # Rows/Time columns, making sure that the
191
			row counts match up with the previous import's row counts
192
		If the row counts do not match up, insert or reorder rows as needed
193
			until they do. Get the datasource names from the log file footers:
194
			tail inputs/{.,}*/*/logs/$version.log.sql
195
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
196 9497 aaronmk
	To run TNRS:
197 9996 aaronmk
		To use an import other than public: export version=<version>
198 9995 aaronmk
		make scrub &
199 8458 aaronmk
		To view progress:
200
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
201 9497 aaronmk
	To remake analytical DB:
202 9996 aaronmk
		To use an import other than public: export version=<version>
203 9497 aaronmk
		bin/make_analytical_db &
204 8458 aaronmk
		To view progress:
205
			tail -100 inputs/analytical_db/logs/make_analytical_db.log.sql
206
	To back up DB (staging tables and last import):
207 9996 aaronmk
		To use an import other than public: export version=<version>
208 8458 aaronmk
		If before renaming to public: export dump_opts=--exclude-schema=public
209
		make backups/vegbien.$version.backup/test &
210 3381 aaronmk
211 3545 aaronmk
Backups:
212 8458 aaronmk
	Archived imports:
213
		Back up: make backups/<version>.backup &
214
			Note: To back up the last import, you must archive it first:
215
				make schemas/rotate
216
		Test: make -s backups/<version>.backup/test &
217
		Restore: make backups/<version>.backup/restore &
218
		Remove: make backups/<version>.backup/remove
219
		Download: make backups/download
220
	TNRS cache:
221
		Back up: make backups/TNRS.backup-remake &
222 10010 aaronmk
			runtime: 3 min ("real 2m48.859s")
223 8458 aaronmk
		Restore:
224
			yes|make inputs/.TNRS/uninstall
225
			make backups/TNRS.backup/restore &
226 10010 aaronmk
				runtime: 5.5 min ("real 5m35.829s")
227 8458 aaronmk
			yes|make schemas/public/reinstall
228
				Must come after TNRS restore to recreate tnrs_input_name view
229
	Full DB:
230
		Back up: make backups/vegbien.<version>.backup &
231
		Test: make -s backups/vegbien.<version>.backup/test &
232
		Restore: make backups/vegbien.<version>.backup/restore &
233
		Download: make backups/download
234
	Import logs:
235
		Download: make inputs/download-logs
236 554 aaronmk
237 1773 aaronmk
Datasource setup:
238 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
239 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
240
		<datasrc> may not contain spaces, and should be abbreviated.
241
		If the datasource is a herbarium, <datasrc> should be the herbarium code
242
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
243
	For MySQL inputs (exports and live DB connections):
244
		For .sql exports:
245
			Place the original .sql file in _src/ (*not* in _MySQL/)
246
			Follow the steps starting with Install the staging tables below.
247
				This is for an initial sync to get the file onto vegbiendev.
248
			On vegbiendev:
249
				Create a database for the MySQL export in phpMyAdmin
250 9494 aaronmk
				Give the bien user all database-specific privileges *except*
251
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
252
					import scripts from accidentally deleting data.
253 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
254
		mkdir inputs/<datasrc>/_MySQL/
255
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
256
		Edit _MySQL/*.make for the DB connection
257
			For a .sql export, use server=vegbiendev and --user=bien
258
		Skip the Add input data for each table section
259
	For MS Access databases:
260
		Place the .mdb or .accdb file in _src/
261
		Download and install Access To PostgreSQL from
262
			http://www.bullzip.com/download.php
263
		Use Access To PostgreSQL to export the database:
264
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
265
			Export just the data to inputs/<datasrc>/<file>.data.sql
266
		In <file>.schema.sql, make the following changes:
267
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
268
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
269
		Skip the Add input data for each table section
270
	Add input data for each table present in the datasource:
271
		For .sql exports, you must use the name of the table in the DB export
272
		For CSV files, you can use any name. It's recommended to use a table
273
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
274
		Note that if this table will be joined together with another table, its
275
			name must end in ".src"
276
		make inputs/<datasrc>/<table>/add
277
			Important: DO NOT just create an empty directory named <table>!
278
				This command also creates necessary subdirs, such as logs/.
279
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
280
			Otherwise, place the CSV(s) for the table in
281
			inputs/<datasrc>/<table>/ OR place a query joining other tables
282
			together in inputs/<datasrc>/<table>/create.sql
283
		Important: When exporting relational databases to CSVs, you MUST ensure
284
			that embedded quotes are escaped by doubling them, *not* by
285
			preceding them with a "\" as is the default in phpMyAdmin
286
		If there are multiple part files for a table, and the header is repeated
287
			in each part, make sure each header is EXACTLY the same.
288 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
289
			assumes the part files don't have individual headers and treats the
290
			subsequent headers as data rows.)
291 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
292
			that depend on it
293
	Install the staging tables:
294
		make inputs/<datasrc>/reinstall quiet=1 &
295
		For a MySQL .sql export:
296
			At prompt "[you]@vegbiendev's password:", enter your password
297
			At prompt "Enter password:", enter the value in config/bien_password
298
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
299
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
300
			tail provides a header line with the filename
301
			+1 starts at the first line, to show the whole file
302
		For every file with an error 'column "..." specified more than once':
303
			Add a header override file "+header.<ext>" in <table>/:
304
				Note: The leading "+" should sort it before the flat files.
305
					"_" unfortunately sorts *after* capital letters in ASCII.
306
				Create a text file containing the header line of the flat files
307
				Add an ! at the beginning of the line
308
					This signals cat_csv that this is a header override.
309
				For empty names, use their 0-based column # (by convention)
310
				For duplicate names, add a distinguishing suffix
311
				For long names that collided, rename them to <= 63 chars long
312
				Do NOT make readability changes in this step; that is what the
313
					map spreadsheets (below) are for.
314
				Save
315
		If you made any changes, re-run the install command above
316
	Auto-create the map spreadsheets: make inputs/<datasrc>/
317
	Map each table's columns:
318
		In each <table>/ subdir, for each "via map" map.csv:
319
			Open the map in a spreadsheet editor
320
			Open the "core map" /mappings/Veg+-VegBIEN.csv
321
			In each row of the via map, set the right column to a value from the
322
				left column of the core map
323
			Save
324
		Regenerate the derived maps: make inputs/<datasrc>/
325
	Accept the test cases:
326
		make inputs/<datasrc>/test
327
			When prompted to "Accept new test output", enter y and press ENTER
328
			If you instead get errors, do one of the following for each one:
329
			-	If the error was due to a bug, fix it
330
			-	Add a SQL function that filters or transforms the invalid data
331
			-	Make an empty mapping for the columns that produced the error.
332
				Put something in the Comments column of the map spreadsheet to
333
				prevent the automatic mapper from auto-removing the mapping.
334
			When accepting tests, it's helpful to use WinMerge
335
				(see WinMerge setup below for configuration)
336
		make inputs/<datasrc>/test by_col=1
337
			If you get errors this time, this always indicates a bug, usually in
338
				the VegBIEN unique constraints or column-based import itself
339
	Add newly-created files: make inputs/<datasrc>/add
340
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
341
	Update vegbiendev:
342
		On jupiter: svn up
343
		On local machine:
344
			./fix_perms
345 9950 aaronmk
			make inputs/upload test=1
346 8458 aaronmk
			make inputs/upload
347
		On vegbiendev:
348
			svn up
349 9950 aaronmk
			make inputs/download test=1
350 8458 aaronmk
			make inputs/download
351
			Follow the steps under Install the staging tables above
352 1773 aaronmk
353 6484 aaronmk
Datasource refreshing:
354 8458 aaronmk
	VegBank:
355
		make inputs/VegBank/vegbank.sql-remake
356
		make inputs/VegBank/reinstall quiet=1 &
357 6484 aaronmk
358 702 aaronmk
Schema changes:
359 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
360
		to update the corresponding table
361
	Remember to update the following files with any renamings:
362
		schemas/filter_ERD.csv
363
		mappings/VegCore-VegBIEN.csv
364
		mappings/verify.*.sql
365
	Regenerate schema from installed DB: make schemas/remake
366
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
367
		WARNING: This will delete the current public schema of your VegBIEN DB!
368 8837 aaronmk
	Reinstall staging tables:
369
		On local machine:
370 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
371 8845 aaronmk
			make db
372 8837 aaronmk
			. bin/reinstall_all
373
			Fix any bugs and retry until no errors
374 8846 aaronmk
			make schemas/public/install
375
				This must be run *after* the datasources are installed, because
376
				views in public depend on some of the datasources
377 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
378 8841 aaronmk
		On vegbiendev: repeat the above steps
379 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
380
			the local machine, or the live DB may be unrestorable!
381 8458 aaronmk
	Sync ERD with vegbien.sql schema:
382
		Run make schemas/vegbien.my.sql
383
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
384
		Go to File > Export > Synchronize With SQL CREATE Script...
385
		For Input File, select schemas/vegbien.my.sql
386
		Click Continue
387
		In the changes list, select each table with an arrow next to it
388
		Click Update Model
389
		Click Continue
390
		Note: The generated SQL script will be empty because we are syncing in
391
			the opposite direction
392
		Click Execute
393
		Reposition any lines that have been reset
394
		Add any new tables by dragging them from the Catalog in the left sidebar
395
			to the diagram
396
		Remove any deleted tables by right-clicking the table's diagram element,
397
			selecting Delete '<table name>', and clicking Delete
398
		Save
399
		If desired, update the graphical ERD exports (see below)
400
	Update graphical ERD exports:
401
		Go to File > Export > Export as PNG...
402
		Select schemas/vegbien.ERD.png and click Save
403
		Go to File > Export > Export as SVG...
404
		Select schemas/vegbien.ERD.svg and click Save
405
		Go to File > Export > Export as Single Page PDF...
406
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
407
		Go to File > Print...
408
		In the lower left corner, click PDF > Save as PDF...
409
		Set the Title and Author to ""
410
		Select schemas/vegbien.ERD.pdf and click Save
411
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
412
	Refactoring tips:
413
		To rename a table:
414
			In vegbien.sql, do the following:
415
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
416
					This is necessary because the table name is *everywhere*
417
				Search for <new>
418
				Manually change back any replacements inside comments
419
		To rename a column:
420
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
421
			Recreate any foreign key for the column, removing CONSTRAINT <name>
422
				This resets the foreign key name using the new column name
423
	Creating a poster of the ERD:
424
		Determine the poster size:
425
			Measure the line height (from the bottom of one line to the bottom
426
				of another): 16.3cm/24 lines = 0.679cm
427
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
428
			Zoom in as far as possible
429
			Measure the height of a capital letter: 3.5mm
430
			Measure the line height: 8.5mm
431
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
432
			Calculate the text height: 0.679cm*0.41 = 0.28cm
433
			Calculate the text height's fraction of the ERD height:
434
				0.28cm/70.8cm = 0.0040
435
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
436
			Calculate the VegBIEN poster height to make the text the same size:
437
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
438
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
439
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
440
			The minimum VegBIEN poster size is 35x54in portrait
441
		Determine the cost:
442
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
443
				charges the following for posters:
444
				base: $7.25/sq ft
445
				lamination: $3/sq ft
446
				mounting on a board: $8/sq ft
447 203 aaronmk
448 1459 aaronmk
Testing:
449 8458 aaronmk
	On a development machine, you should put the following in your .profile:
450 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
451 8458 aaronmk
		export log= n=2
452
	Mapping process: make test
453
		Including column-based import: make test by_col=1
454
			If the row-based and column-based imports produce different inserted
455
			row counts, this usually means that a table is underconstrained
456
			(the unique indexes don't cover all possible rows).
457
			This can occur if you didn't use COALESCE(field, null_value) around
458
			a nullable field in a unique index. See sql_gen.null_sentinels for
459
			the appropriate null value to use.
460
	Map spreadsheet generation: make remake
461
	Missing mappings: make missing_mappings
462
	Everything (for most complete coverage): make test-all
463 702 aaronmk
464 7183 aaronmk
Debugging:
465 8458 aaronmk
	"Binary chop" debugging:
466
		(This is primarily useful for regressions that occurred in a previous
467
		revision, which was committed without running all the tests)
468
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
469 8470 aaronmk
	.htaccess:
470
		mod_rewrite:
471 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
472
				directory, you *must* clear your browser's cache to ensure that
473
				a cached redirect is not used. this is because RewriteRule
474
				redirects are (by default) temporary, but DirectorySlash
475
				redirects are permanent.
476 8470 aaronmk
				for Firefox:
477
					press Cmd+Shift+Delete
478
					check only Cache
479
					press Enter or click Clear Now
480 7183 aaronmk
481 3783 aaronmk
WinMerge setup:
482 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
483
	Open WinMerge
484
	Go to Edit > Options and click Compare in the left sidebar
485
	Enable "Moved block detection", as described at
486
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
487
	Set Whitespace to Ignore change, as described at
488
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
489 3783 aaronmk
490 3133 aaronmk
Documentation:
491 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
492
		make schemas/public/reinstall
493
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
494
	To import and scrub just the test taxonomic names:
495
		inputs/test_taxonomic_names/test_scrub
496 3133 aaronmk
497 702 aaronmk
General:
498 8458 aaronmk
	To see a program's description, read its top-of-file comment
499
	To see a program's usage, run it without arguments
500
	To remake a directory: make <dir>/remake
501
	To remake a file: make <file>-remake