Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 3674 aaronmk
Maintenance:
11 9399 aaronmk
	on a live machine, you should put the following in your .profile:
12
		umask ug=rwx,o= # prevent files from becoming web-accessible
13 9400 aaronmk
		unset TMOUT # TMOUT causes screen to exit even with background processes
14 8459 aaronmk
	to synchronize vegbiendev, jupiter, and your local machine:
15 9886 aaronmk
		WARNING: pay careful attention to all files that will be deleted or
16
			overwritten!
17 8460 aaronmk
		install put if needed:
18
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
19 8459 aaronmk
		when changes are made on vegbiendev:
20
			on vegbiendev, upload:
21 10032 aaronmk
				overwrite=1 bin/sync_upload
22 10029 aaronmk
					then rerun with l=1 ...
23 8459 aaronmk
			on your machine, download:
24 10040 aaronmk
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
25 10029 aaronmk
					then rerun with l=1 ...
26 10032 aaronmk
				overwrite=1 swap=1 bin/sync_upload
27 10029 aaronmk
					then rerun with l=1 ...
28 10093 aaronmk
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
29 10068 aaronmk
					then rerun with l=1 ...
30 8462 aaronmk
	to synchronize a Mac's settings with my testing machine's:
31
		download:
32
			WARNING: this will overwrite all your user's settings!
33 10040 aaronmk
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
34 10029 aaronmk
				then rerun with l=1 ...
35 8462 aaronmk
		upload:
36 8719 aaronmk
			do step when changes are made on vegbiendev > on your machine, download
37 10035 aaronmk
			on jupiter: (cd ~/Dropbox/svn/; svn up)
38
			on your machine:
39 10040 aaronmk
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
40 10035 aaronmk
					then rerun with l=1 ...
41 10040 aaronmk
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home"
42 10035 aaronmk
					then rerun with l=1 ...
43 10040 aaronmk
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
44 10035 aaronmk
					then rerun with l=1 ...
45 10221 aaronmk
		upload just the VirtualBox VMs:
46
			on your machine:
47
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
48
					then rerun with l=1 ...
49 8720 aaronmk
	to backup files not in Time Machine:
50
		stop the PostgreSQL server
51 10029 aaronmk
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/
52
			then rerun with l=1 ...
53 8720 aaronmk
		start the PostgreSQL server
54 8458 aaronmk
	VegCore data dictionary:
55
		Regularly, or whenever the VegCore data dictionary page
56
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
57
			is changed, regenerate mappings/VegCore.csv:
58
			make mappings/VegCore.htm-remake; make mappings/
59
			svn di mappings/VegCore.tables.redmine
60
			If there are changes, update the data dictionary's Tables section
61
			When moving terms, check that no terms were lost: svn di
62 10286 aaronmk
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
63 8458 aaronmk
	Important: Whenever you install a system update that affects PostgreSQL or
64
		any of its dependencies, such as libc, you should restart the PostgreSQL
65
		server. Otherwise, you may get strange errors like "the database system
66
		is in recovery mode" which go away upon reimport, or you may not be able
67
		to access the database as the postgres superuser. This applies to both
68
		Linux and Mac OS X.
69 3674 aaronmk
70 7287 aaronmk
Single datasource import:
71 8458 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1
72
	(Re)import only: make inputs/<datasrc>/reimport by_col=1
73
	(Re)scrub: make inputs/<datasrc>/rescrub by_col=1
74
	Note that these commands also work if the datasource is not yet imported
75 7287 aaronmk
76
Full database import:
77 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
78
		breaking column-based import
79 9883 aaronmk
	On jupiter: svn up --force
80 8458 aaronmk
	On local machine:
81
		./fix_perms
82 9887 aaronmk
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
83
			your local machine" above
84 8458 aaronmk
		make inputs/upload
85 10025 aaronmk
		make inputs/upload live=1
86 8458 aaronmk
		make test by_col=1
87
			See note under Testing below
88
	On vegbiendev:
89
	Ensure there are no local modifications: svn st
90
	svn up
91
	make inputs/download
92 10025 aaronmk
	make inputs/download live=1
93 8458 aaronmk
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
94
	Update the auxiliary schemas: make schemas/reinstall
95
		The public schema will be installed separately by the import process
96
	Delete imports before the last so they won't bloat the full DB backup:
97
		make backups/vegbien.<version>.backup/remove
98
		To keep a previous import other than the public schema:
99
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
100 9970 aaronmk
	Make sure there is at least 300GB of disk space on /: df -h
101
		The import schema is 255GB, and may use additional space for temp tables
102 8458 aaronmk
		To free up space, remove backups that have been archived on jupiter:
103
			List backups/ to view older backups
104
			Check their MD5 sums using the steps under On jupiter below
105
			Remove these backups
106 9899 aaronmk
	Remove any leftover TNRS lockfile: rm inputs/.TNRS/tnrs/tnrs.make.lock
107
		Usually, the PID in it would not exist, but sometimes it now refers to
108
			a different, active process which blocks tnrs.make
109 8458 aaronmk
	screen
110
	Press ENTER
111 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
112 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
113
	unset version
114 8458 aaronmk
	Start column-based import: . bin/import_all by_col=1
115
		To use row-based import: . bin/import_all
116
		To stop all running imports: . bin/stop_imports
117
		WARNING: Do NOT run import_all in the background, or the jobs it creates
118
			won't be owned by your shell.
119
		Note that import_all will take up to an hour to import the NCBI backbone
120
			and other metadata before returning control to the shell.
121
	Wait (overnight) for the import to finish
122
	To recover from a closed terminal window: screen -r
123
	When there are no more running jobs, exit the screen
124
	Get $version: echo $version
125
	Set $version in all vegbiendev terminals: export version=<version>
126 10025 aaronmk
	Upload logs (run on vegbiendev): make inputs/upload live=1
127
	On local machine: make inputs/download-logs live=1
128 8458 aaronmk
	In PostgreSQL:
129
		Check that the provider_count and source tables contain entries for all
130
			inputs
131 9492 aaronmk
	Check that TNRS ran successfully:
132
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
133
		If the log ends in an AssertionError
134
			"assert sql.table_col_names(db, table) == header":
135
			Figure out which TNRS CSV columns have changed
136
			On local machine:
137
				Make the changes in inputs/.TNRS/schema.sql
138
				rm inputs/.TNRS/tnrs/header.csv
139
				make inputs/.TNRS/reinstall schema_only=1
140
				make schemas/public/reinstall
141
				If there are errors "column ... does not exist", etc.:
142
					Make the necessary changes in schemas/vegbien.sql
143
					make schemas/public/reinstall
144
				make schemas/remake
145
				inputs/test_taxonomic_names/test_scrub
146 9528 aaronmk
				In inputs/test_taxonomic_names/_scrub/TNRS.sql, copy the
147
					"COPY tnrs ..." statement to inputs/.TNRS/data.sql
148 9492 aaronmk
				Commit
149
			On vegbiendev:
150
				If dropping a column, save the dependent views
151
				Make the same changes in the live TNRS.tnrs table on vegbiendev
152
				If dropping a column, recreate the dependent views
153
				Restart the TNRS client: make scrub by_col=1 &
154 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
155
	In the output, search for "Command exited with non-zero status"
156
	For inputs that have this, fix the associated bug(s)
157
	If many inputs have errors, discard the current (partial) import:
158
		make schemas/$version/uninstall
159
	Otherwise, continue
160 9498 aaronmk
	Publish the new import:
161
		WARNING: Before proceeding, be sure you have done *every single*
162
			verification step listed above. Otherwise, a previous valid import
163
			could incorrectly be overwritten with a broken one.
164
		make schemas/$version/publish
165 8458 aaronmk
	unset version
166
	backups/fix_perms
167 10027 aaronmk
	make backups/upload live=1
168 8458 aaronmk
	On jupiter:
169
		cd /data/dev/aaronmk/bien/backups
170
		For each newly-archived backup:
171
			make -s <backup>.md5/test
172
			Check that "OK" is printed next to the filename
173
	On nimoy:
174
		cd /home/bien/svn/
175
		svn up
176
		export version=<version>
177
		make backups/analytical_stem.$version.csv/download
178
		In the bien_web DB:
179
			Create the analytical_stem_<version> table using its schema
180
				in schemas/vegbien.my.sql
181
		make -s backups/analytical_stem.$version.csv.md5/test
182
		Check that "OK" is printed next to the filename
183 10029 aaronmk
		table=analytical_stem_$version bin/publish_analytical_db \
184 8458 aaronmk
			backups/analytical_stem.$version.csv
185
	If desired, record the import times in inputs/import.stats.xls:
186
		Open inputs/import.stats.xls
187
		If the rightmost import is within 5 columns of column IV:
188
			Copy the current tab to <leftmost-date>~<rightmost-date>
189
			Remove the previous imports from the current tab because they are
190
				now in the copied tab instead
191
		Insert a copy of the leftmost "By column" column group before it
192
		export version=<version>
193
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
194
		Update the import date in the upper-right corner
195
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
196
		Paste the output over the # Rows/Time columns, making sure that the
197
			row counts match up with the previous import's row counts
198
		If the row counts do not match up, insert or reorder rows as needed
199
			until they do. Get the datasource names from the log file footers:
200
			tail inputs/{.,}*/*/logs/$version.log.sql
201
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
202 9497 aaronmk
	To run TNRS:
203 9996 aaronmk
		To use an import other than public: export version=<version>
204 9995 aaronmk
		make scrub &
205 8458 aaronmk
		To view progress:
206
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
207 9497 aaronmk
	To remake analytical DB:
208 9996 aaronmk
		To use an import other than public: export version=<version>
209 9497 aaronmk
		bin/make_analytical_db &
210 8458 aaronmk
		To view progress:
211
			tail -100 inputs/analytical_db/logs/make_analytical_db.log.sql
212
	To back up DB (staging tables and last import):
213 9996 aaronmk
		To use an import other than public: export version=<version>
214 8458 aaronmk
		If before renaming to public: export dump_opts=--exclude-schema=public
215
		make backups/vegbien.$version.backup/test &
216 3381 aaronmk
217 3545 aaronmk
Backups:
218 8458 aaronmk
	Archived imports:
219
		Back up: make backups/<version>.backup &
220
			Note: To back up the last import, you must archive it first:
221
				make schemas/rotate
222
		Test: make -s backups/<version>.backup/test &
223
		Restore: make backups/<version>.backup/restore &
224
		Remove: make backups/<version>.backup/remove
225 10028 aaronmk
		Download: make backups/<version>.backup/download
226 8458 aaronmk
	TNRS cache:
227
		Back up: make backups/TNRS.backup-remake &
228 10010 aaronmk
			runtime: 3 min ("real 2m48.859s")
229 8458 aaronmk
		Restore:
230
			yes|make inputs/.TNRS/uninstall
231
			make backups/TNRS.backup/restore &
232 10010 aaronmk
				runtime: 5.5 min ("real 5m35.829s")
233 8458 aaronmk
			yes|make schemas/public/reinstall
234
				Must come after TNRS restore to recreate tnrs_input_name view
235
	Full DB:
236
		Back up: make backups/vegbien.<version>.backup &
237
		Test: make -s backups/vegbien.<version>.backup/test &
238
		Restore: make backups/vegbien.<version>.backup/restore &
239 10028 aaronmk
		Download: make backups/vegbien.<version>.backup/download
240 8458 aaronmk
	Import logs:
241 10025 aaronmk
		Download: make inputs/download-logs live=1
242 554 aaronmk
243 1773 aaronmk
Datasource setup:
244 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
245 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
246
		<datasrc> may not contain spaces, and should be abbreviated.
247
		If the datasource is a herbarium, <datasrc> should be the herbarium code
248
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
249
	For MySQL inputs (exports and live DB connections):
250
		For .sql exports:
251
			Place the original .sql file in _src/ (*not* in _MySQL/)
252
			Follow the steps starting with Install the staging tables below.
253
				This is for an initial sync to get the file onto vegbiendev.
254
			On vegbiendev:
255
				Create a database for the MySQL export in phpMyAdmin
256 9494 aaronmk
				Give the bien user all database-specific privileges *except*
257
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
258
					import scripts from accidentally deleting data.
259 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
260
		mkdir inputs/<datasrc>/_MySQL/
261
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
262
		Edit _MySQL/*.make for the DB connection
263
			For a .sql export, use server=vegbiendev and --user=bien
264
		Skip the Add input data for each table section
265
	For MS Access databases:
266
		Place the .mdb or .accdb file in _src/
267
		Download and install Access To PostgreSQL from
268
			http://www.bullzip.com/download.php
269
		Use Access To PostgreSQL to export the database:
270
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
271
			Export just the data to inputs/<datasrc>/<file>.data.sql
272
		In <file>.schema.sql, make the following changes:
273
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
274
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
275
		Skip the Add input data for each table section
276
	Add input data for each table present in the datasource:
277
		For .sql exports, you must use the name of the table in the DB export
278
		For CSV files, you can use any name. It's recommended to use a table
279
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
280
		Note that if this table will be joined together with another table, its
281
			name must end in ".src"
282
		make inputs/<datasrc>/<table>/add
283
			Important: DO NOT just create an empty directory named <table>!
284
				This command also creates necessary subdirs, such as logs/.
285
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
286
			Otherwise, place the CSV(s) for the table in
287
			inputs/<datasrc>/<table>/ OR place a query joining other tables
288
			together in inputs/<datasrc>/<table>/create.sql
289
		Important: When exporting relational databases to CSVs, you MUST ensure
290
			that embedded quotes are escaped by doubling them, *not* by
291
			preceding them with a "\" as is the default in phpMyAdmin
292
		If there are multiple part files for a table, and the header is repeated
293
			in each part, make sure each header is EXACTLY the same.
294 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
295
			assumes the part files don't have individual headers and treats the
296
			subsequent headers as data rows.)
297 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
298
			that depend on it
299
	Install the staging tables:
300
		make inputs/<datasrc>/reinstall quiet=1 &
301
		For a MySQL .sql export:
302
			At prompt "[you]@vegbiendev's password:", enter your password
303
			At prompt "Enter password:", enter the value in config/bien_password
304
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
305
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
306
			tail provides a header line with the filename
307
			+1 starts at the first line, to show the whole file
308
		For every file with an error 'column "..." specified more than once':
309
			Add a header override file "+header.<ext>" in <table>/:
310
				Note: The leading "+" should sort it before the flat files.
311
					"_" unfortunately sorts *after* capital letters in ASCII.
312
				Create a text file containing the header line of the flat files
313
				Add an ! at the beginning of the line
314
					This signals cat_csv that this is a header override.
315
				For empty names, use their 0-based column # (by convention)
316
				For duplicate names, add a distinguishing suffix
317
				For long names that collided, rename them to <= 63 chars long
318
				Do NOT make readability changes in this step; that is what the
319
					map spreadsheets (below) are for.
320
				Save
321
		If you made any changes, re-run the install command above
322
	Auto-create the map spreadsheets: make inputs/<datasrc>/
323
	Map each table's columns:
324
		In each <table>/ subdir, for each "via map" map.csv:
325
			Open the map in a spreadsheet editor
326
			Open the "core map" /mappings/Veg+-VegBIEN.csv
327
			In each row of the via map, set the right column to a value from the
328
				left column of the core map
329
			Save
330
		Regenerate the derived maps: make inputs/<datasrc>/
331
	Accept the test cases:
332
		make inputs/<datasrc>/test
333
			When prompted to "Accept new test output", enter y and press ENTER
334
			If you instead get errors, do one of the following for each one:
335
			-	If the error was due to a bug, fix it
336
			-	Add a SQL function that filters or transforms the invalid data
337
			-	Make an empty mapping for the columns that produced the error.
338
				Put something in the Comments column of the map spreadsheet to
339
				prevent the automatic mapper from auto-removing the mapping.
340
			When accepting tests, it's helpful to use WinMerge
341
				(see WinMerge setup below for configuration)
342
		make inputs/<datasrc>/test by_col=1
343
			If you get errors this time, this always indicates a bug, usually in
344
				the VegBIEN unique constraints or column-based import itself
345
	Add newly-created files: make inputs/<datasrc>/add
346
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
347
	Update vegbiendev:
348
		On jupiter: svn up
349
		On local machine:
350
			./fix_perms
351
			make inputs/upload
352 10025 aaronmk
			make inputs/upload live=1
353 8458 aaronmk
		On vegbiendev:
354
			svn up
355
			make inputs/download
356 10025 aaronmk
			make inputs/download live=1
357 8458 aaronmk
			Follow the steps under Install the staging tables above
358 1773 aaronmk
359 6484 aaronmk
Datasource refreshing:
360 8458 aaronmk
	VegBank:
361
		make inputs/VegBank/vegbank.sql-remake
362
		make inputs/VegBank/reinstall quiet=1 &
363 6484 aaronmk
364 702 aaronmk
Schema changes:
365 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
366
		to update the corresponding table
367
	Remember to update the following files with any renamings:
368
		schemas/filter_ERD.csv
369
		mappings/VegCore-VegBIEN.csv
370
		mappings/verify.*.sql
371
	Regenerate schema from installed DB: make schemas/remake
372
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
373
		WARNING: This will delete the current public schema of your VegBIEN DB!
374 8837 aaronmk
	Reinstall staging tables:
375
		On local machine:
376 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
377 8845 aaronmk
			make db
378 8837 aaronmk
			. bin/reinstall_all
379
			Fix any bugs and retry until no errors
380 8846 aaronmk
			make schemas/public/install
381
				This must be run *after* the datasources are installed, because
382
				views in public depend on some of the datasources
383 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
384 8841 aaronmk
		On vegbiendev: repeat the above steps
385 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
386
			the local machine, or the live DB may be unrestorable!
387 8458 aaronmk
	Sync ERD with vegbien.sql schema:
388
		Run make schemas/vegbien.my.sql
389
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
390
		Go to File > Export > Synchronize With SQL CREATE Script...
391
		For Input File, select schemas/vegbien.my.sql
392
		Click Continue
393
		In the changes list, select each table with an arrow next to it
394
		Click Update Model
395
		Click Continue
396
		Note: The generated SQL script will be empty because we are syncing in
397
			the opposite direction
398
		Click Execute
399
		Reposition any lines that have been reset
400
		Add any new tables by dragging them from the Catalog in the left sidebar
401
			to the diagram
402
		Remove any deleted tables by right-clicking the table's diagram element,
403
			selecting Delete '<table name>', and clicking Delete
404
		Save
405
		If desired, update the graphical ERD exports (see below)
406
	Update graphical ERD exports:
407
		Go to File > Export > Export as PNG...
408
		Select schemas/vegbien.ERD.png and click Save
409
		Go to File > Export > Export as SVG...
410
		Select schemas/vegbien.ERD.svg and click Save
411
		Go to File > Export > Export as Single Page PDF...
412
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
413
		Go to File > Print...
414
		In the lower left corner, click PDF > Save as PDF...
415
		Set the Title and Author to ""
416
		Select schemas/vegbien.ERD.pdf and click Save
417
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
418
	Refactoring tips:
419
		To rename a table:
420
			In vegbien.sql, do the following:
421
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
422
					This is necessary because the table name is *everywhere*
423
				Search for <new>
424
				Manually change back any replacements inside comments
425
		To rename a column:
426
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
427
			Recreate any foreign key for the column, removing CONSTRAINT <name>
428
				This resets the foreign key name using the new column name
429
	Creating a poster of the ERD:
430
		Determine the poster size:
431
			Measure the line height (from the bottom of one line to the bottom
432
				of another): 16.3cm/24 lines = 0.679cm
433
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
434
			Zoom in as far as possible
435
			Measure the height of a capital letter: 3.5mm
436
			Measure the line height: 8.5mm
437
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
438
			Calculate the text height: 0.679cm*0.41 = 0.28cm
439
			Calculate the text height's fraction of the ERD height:
440
				0.28cm/70.8cm = 0.0040
441
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
442
			Calculate the VegBIEN poster height to make the text the same size:
443
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
444
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
445
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
446
			The minimum VegBIEN poster size is 35x54in portrait
447
		Determine the cost:
448
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
449
				charges the following for posters:
450
				base: $7.25/sq ft
451
				lamination: $3/sq ft
452
				mounting on a board: $8/sq ft
453 203 aaronmk
454 1459 aaronmk
Testing:
455 8458 aaronmk
	On a development machine, you should put the following in your .profile:
456 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
457 8458 aaronmk
		export log= n=2
458
	Mapping process: make test
459
		Including column-based import: make test by_col=1
460
			If the row-based and column-based imports produce different inserted
461
			row counts, this usually means that a table is underconstrained
462
			(the unique indexes don't cover all possible rows).
463
			This can occur if you didn't use COALESCE(field, null_value) around
464
			a nullable field in a unique index. See sql_gen.null_sentinels for
465
			the appropriate null value to use.
466
	Map spreadsheet generation: make remake
467
	Missing mappings: make missing_mappings
468
	Everything (for most complete coverage): make test-all
469 702 aaronmk
470 7183 aaronmk
Debugging:
471 8458 aaronmk
	"Binary chop" debugging:
472
		(This is primarily useful for regressions that occurred in a previous
473
		revision, which was committed without running all the tests)
474
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
475 8470 aaronmk
	.htaccess:
476
		mod_rewrite:
477 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
478
				directory, you *must* clear your browser's cache to ensure that
479
				a cached redirect is not used. this is because RewriteRule
480
				redirects are (by default) temporary, but DirectorySlash
481
				redirects are permanent.
482 8470 aaronmk
				for Firefox:
483
					press Cmd+Shift+Delete
484
					check only Cache
485
					press Enter or click Clear Now
486 7183 aaronmk
487 3783 aaronmk
WinMerge setup:
488 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
489
	Open WinMerge
490
	Go to Edit > Options and click Compare in the left sidebar
491
	Enable "Moved block detection", as described at
492
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
493
	Set Whitespace to Ignore change, as described at
494
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
495 3783 aaronmk
496 3133 aaronmk
Documentation:
497 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
498
		make schemas/public/reinstall
499
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
500
	To import and scrub just the test taxonomic names:
501
		inputs/test_taxonomic_names/test_scrub
502 3133 aaronmk
503 702 aaronmk
General:
504 8458 aaronmk
	To see a program's description, read its top-of-file comment
505
	To see a program's usage, run it without arguments
506
	To remake a directory: make <dir>/remake
507
	To remake a file: make <file>-remake