Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 3674 aaronmk
Maintenance:
11 9399 aaronmk
	on a live machine, you should put the following in your .profile:
12
		umask ug=rwx,o= # prevent files from becoming web-accessible
13 9400 aaronmk
		unset TMOUT # TMOUT causes screen to exit even with background processes
14 8459 aaronmk
	to synchronize vegbiendev, jupiter, and your local machine:
15 9886 aaronmk
		WARNING: pay careful attention to all files that will be deleted or
16
			overwritten!
17 8460 aaronmk
		install put if needed:
18
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
19 8459 aaronmk
		when changes are made on vegbiendev:
20
			on vegbiendev, upload:
21 10032 aaronmk
				overwrite=1 bin/sync_upload
22 10029 aaronmk
					then rerun with l=1 ...
23 8459 aaronmk
			on your machine, download:
24 10040 aaronmk
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
25 10029 aaronmk
					then rerun with l=1 ...
26 10032 aaronmk
				overwrite=1 swap=1 bin/sync_upload
27 10029 aaronmk
					then rerun with l=1 ...
28 10093 aaronmk
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
29 10068 aaronmk
					then rerun with l=1 ...
30 8462 aaronmk
	to synchronize a Mac's settings with my testing machine's:
31
		download:
32
			WARNING: this will overwrite all your user's settings!
33 10040 aaronmk
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
34 10029 aaronmk
				then rerun with l=1 ...
35 8462 aaronmk
		upload:
36 8719 aaronmk
			do step when changes are made on vegbiendev > on your machine, download
37 10035 aaronmk
			on jupiter: (cd ~/Dropbox/svn/; svn up)
38
			on your machine:
39 10040 aaronmk
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
40 10035 aaronmk
					then rerun with l=1 ...
41 10040 aaronmk
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home"
42 10035 aaronmk
					then rerun with l=1 ...
43 10040 aaronmk
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
44 10035 aaronmk
					then rerun with l=1 ...
45 8720 aaronmk
	to backup files not in Time Machine:
46 10029 aaronmk
		inplace=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ put Users/aaronmk/VirtualBox\ VMs/
47
			then rerun with l=1 ...
48 8720 aaronmk
		stop the PostgreSQL server
49 10029 aaronmk
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/
50
			then rerun with l=1 ...
51 8720 aaronmk
		start the PostgreSQL server
52 8458 aaronmk
	VegCore data dictionary:
53
		Regularly, or whenever the VegCore data dictionary page
54
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
55
			is changed, regenerate mappings/VegCore.csv:
56
			make mappings/VegCore.htm-remake; make mappings/
57
			svn di mappings/VegCore.tables.redmine
58
			If there are changes, update the data dictionary's Tables section
59
			When moving terms, check that no terms were lost: svn di
60
			svn ci -m "mappings/VegCore.htm: Regenerated from wiki"
61
	Important: Whenever you install a system update that affects PostgreSQL or
62
		any of its dependencies, such as libc, you should restart the PostgreSQL
63
		server. Otherwise, you may get strange errors like "the database system
64
		is in recovery mode" which go away upon reimport, or you may not be able
65
		to access the database as the postgres superuser. This applies to both
66
		Linux and Mac OS X.
67 3674 aaronmk
68 7287 aaronmk
Single datasource import:
69 8458 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1
70
	(Re)import only: make inputs/<datasrc>/reimport by_col=1
71
	(Re)scrub: make inputs/<datasrc>/rescrub by_col=1
72
	Note that these commands also work if the datasource is not yet imported
73 7287 aaronmk
74
Full database import:
75 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
76
		breaking column-based import
77 9883 aaronmk
	On jupiter: svn up --force
78 8458 aaronmk
	On local machine:
79
		./fix_perms
80 9887 aaronmk
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
81
			your local machine" above
82 8458 aaronmk
		make inputs/upload
83 10025 aaronmk
		make inputs/upload live=1
84 8458 aaronmk
		make test by_col=1
85
			See note under Testing below
86
	On vegbiendev:
87
	Ensure there are no local modifications: svn st
88
	svn up
89
	make inputs/download
90 10025 aaronmk
	make inputs/download live=1
91 8458 aaronmk
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
92
	Update the auxiliary schemas: make schemas/reinstall
93
		The public schema will be installed separately by the import process
94
	Delete imports before the last so they won't bloat the full DB backup:
95
		make backups/vegbien.<version>.backup/remove
96
		To keep a previous import other than the public schema:
97
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
98 9970 aaronmk
	Make sure there is at least 300GB of disk space on /: df -h
99
		The import schema is 255GB, and may use additional space for temp tables
100 8458 aaronmk
		To free up space, remove backups that have been archived on jupiter:
101
			List backups/ to view older backups
102
			Check their MD5 sums using the steps under On jupiter below
103
			Remove these backups
104 9899 aaronmk
	Remove any leftover TNRS lockfile: rm inputs/.TNRS/tnrs/tnrs.make.lock
105
		Usually, the PID in it would not exist, but sometimes it now refers to
106
			a different, active process which blocks tnrs.make
107 8458 aaronmk
	screen
108
	Press ENTER
109 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
110 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
111
	unset version
112 8458 aaronmk
	Start column-based import: . bin/import_all by_col=1
113
		To use row-based import: . bin/import_all
114
		To stop all running imports: . bin/stop_imports
115
		WARNING: Do NOT run import_all in the background, or the jobs it creates
116
			won't be owned by your shell.
117
		Note that import_all will take up to an hour to import the NCBI backbone
118
			and other metadata before returning control to the shell.
119
	Wait (overnight) for the import to finish
120
	To recover from a closed terminal window: screen -r
121
	When there are no more running jobs, exit the screen
122
	Get $version: echo $version
123
	Set $version in all vegbiendev terminals: export version=<version>
124 10025 aaronmk
	Upload logs (run on vegbiendev): make inputs/upload live=1
125
	On local machine: make inputs/download-logs live=1
126 8458 aaronmk
	In PostgreSQL:
127
		Check that the provider_count and source tables contain entries for all
128
			inputs
129 9492 aaronmk
	Check that TNRS ran successfully:
130
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
131
		If the log ends in an AssertionError
132
			"assert sql.table_col_names(db, table) == header":
133
			Figure out which TNRS CSV columns have changed
134
			On local machine:
135
				Make the changes in inputs/.TNRS/schema.sql
136
				rm inputs/.TNRS/tnrs/header.csv
137
				make inputs/.TNRS/reinstall schema_only=1
138
				make schemas/public/reinstall
139
				If there are errors "column ... does not exist", etc.:
140
					Make the necessary changes in schemas/vegbien.sql
141
					make schemas/public/reinstall
142
				make schemas/remake
143
				inputs/test_taxonomic_names/test_scrub
144 9528 aaronmk
				In inputs/test_taxonomic_names/_scrub/TNRS.sql, copy the
145
					"COPY tnrs ..." statement to inputs/.TNRS/data.sql
146 9492 aaronmk
				Commit
147
			On vegbiendev:
148
				If dropping a column, save the dependent views
149
				Make the same changes in the live TNRS.tnrs table on vegbiendev
150
				If dropping a column, recreate the dependent views
151
				Restart the TNRS client: make scrub by_col=1 &
152 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
153
	In the output, search for "Command exited with non-zero status"
154
	For inputs that have this, fix the associated bug(s)
155
	If many inputs have errors, discard the current (partial) import:
156
		make schemas/$version/uninstall
157
	Otherwise, continue
158 9498 aaronmk
	Publish the new import:
159
		WARNING: Before proceeding, be sure you have done *every single*
160
			verification step listed above. Otherwise, a previous valid import
161
			could incorrectly be overwritten with a broken one.
162
		make schemas/$version/publish
163 8458 aaronmk
	unset version
164
	backups/fix_perms
165 10027 aaronmk
	make backups/upload live=1
166 8458 aaronmk
	On jupiter:
167
		cd /data/dev/aaronmk/bien/backups
168
		For each newly-archived backup:
169
			make -s <backup>.md5/test
170
			Check that "OK" is printed next to the filename
171
	On nimoy:
172
		cd /home/bien/svn/
173
		svn up
174
		export version=<version>
175
		make backups/analytical_stem.$version.csv/download
176
		In the bien_web DB:
177
			Create the analytical_stem_<version> table using its schema
178
				in schemas/vegbien.my.sql
179
		make -s backups/analytical_stem.$version.csv.md5/test
180
		Check that "OK" is printed next to the filename
181 10029 aaronmk
		table=analytical_stem_$version bin/publish_analytical_db \
182 8458 aaronmk
			backups/analytical_stem.$version.csv
183
	If desired, record the import times in inputs/import.stats.xls:
184
		Open inputs/import.stats.xls
185
		If the rightmost import is within 5 columns of column IV:
186
			Copy the current tab to <leftmost-date>~<rightmost-date>
187
			Remove the previous imports from the current tab because they are
188
				now in the copied tab instead
189
		Insert a copy of the leftmost "By column" column group before it
190
		export version=<version>
191
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
192
		Update the import date in the upper-right corner
193
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
194
		Paste the output over the # Rows/Time columns, making sure that the
195
			row counts match up with the previous import's row counts
196
		If the row counts do not match up, insert or reorder rows as needed
197
			until they do. Get the datasource names from the log file footers:
198
			tail inputs/{.,}*/*/logs/$version.log.sql
199
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
200 9497 aaronmk
	To run TNRS:
201 9996 aaronmk
		To use an import other than public: export version=<version>
202 9995 aaronmk
		make scrub &
203 8458 aaronmk
		To view progress:
204
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
205 9497 aaronmk
	To remake analytical DB:
206 9996 aaronmk
		To use an import other than public: export version=<version>
207 9497 aaronmk
		bin/make_analytical_db &
208 8458 aaronmk
		To view progress:
209
			tail -100 inputs/analytical_db/logs/make_analytical_db.log.sql
210
	To back up DB (staging tables and last import):
211 9996 aaronmk
		To use an import other than public: export version=<version>
212 8458 aaronmk
		If before renaming to public: export dump_opts=--exclude-schema=public
213
		make backups/vegbien.$version.backup/test &
214 3381 aaronmk
215 3545 aaronmk
Backups:
216 8458 aaronmk
	Archived imports:
217
		Back up: make backups/<version>.backup &
218
			Note: To back up the last import, you must archive it first:
219
				make schemas/rotate
220
		Test: make -s backups/<version>.backup/test &
221
		Restore: make backups/<version>.backup/restore &
222
		Remove: make backups/<version>.backup/remove
223 10028 aaronmk
		Download: make backups/<version>.backup/download
224 8458 aaronmk
	TNRS cache:
225
		Back up: make backups/TNRS.backup-remake &
226 10010 aaronmk
			runtime: 3 min ("real 2m48.859s")
227 8458 aaronmk
		Restore:
228
			yes|make inputs/.TNRS/uninstall
229
			make backups/TNRS.backup/restore &
230 10010 aaronmk
				runtime: 5.5 min ("real 5m35.829s")
231 8458 aaronmk
			yes|make schemas/public/reinstall
232
				Must come after TNRS restore to recreate tnrs_input_name view
233
	Full DB:
234
		Back up: make backups/vegbien.<version>.backup &
235
		Test: make -s backups/vegbien.<version>.backup/test &
236
		Restore: make backups/vegbien.<version>.backup/restore &
237 10028 aaronmk
		Download: make backups/vegbien.<version>.backup/download
238 8458 aaronmk
	Import logs:
239 10025 aaronmk
		Download: make inputs/download-logs live=1
240 554 aaronmk
241 1773 aaronmk
Datasource setup:
242 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
243 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
244
		<datasrc> may not contain spaces, and should be abbreviated.
245
		If the datasource is a herbarium, <datasrc> should be the herbarium code
246
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
247
	For MySQL inputs (exports and live DB connections):
248
		For .sql exports:
249
			Place the original .sql file in _src/ (*not* in _MySQL/)
250
			Follow the steps starting with Install the staging tables below.
251
				This is for an initial sync to get the file onto vegbiendev.
252
			On vegbiendev:
253
				Create a database for the MySQL export in phpMyAdmin
254 9494 aaronmk
				Give the bien user all database-specific privileges *except*
255
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
256
					import scripts from accidentally deleting data.
257 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
258
		mkdir inputs/<datasrc>/_MySQL/
259
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
260
		Edit _MySQL/*.make for the DB connection
261
			For a .sql export, use server=vegbiendev and --user=bien
262
		Skip the Add input data for each table section
263
	For MS Access databases:
264
		Place the .mdb or .accdb file in _src/
265
		Download and install Access To PostgreSQL from
266
			http://www.bullzip.com/download.php
267
		Use Access To PostgreSQL to export the database:
268
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
269
			Export just the data to inputs/<datasrc>/<file>.data.sql
270
		In <file>.schema.sql, make the following changes:
271
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
272
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
273
		Skip the Add input data for each table section
274
	Add input data for each table present in the datasource:
275
		For .sql exports, you must use the name of the table in the DB export
276
		For CSV files, you can use any name. It's recommended to use a table
277
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
278
		Note that if this table will be joined together with another table, its
279
			name must end in ".src"
280
		make inputs/<datasrc>/<table>/add
281
			Important: DO NOT just create an empty directory named <table>!
282
				This command also creates necessary subdirs, such as logs/.
283
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
284
			Otherwise, place the CSV(s) for the table in
285
			inputs/<datasrc>/<table>/ OR place a query joining other tables
286
			together in inputs/<datasrc>/<table>/create.sql
287
		Important: When exporting relational databases to CSVs, you MUST ensure
288
			that embedded quotes are escaped by doubling them, *not* by
289
			preceding them with a "\" as is the default in phpMyAdmin
290
		If there are multiple part files for a table, and the header is repeated
291
			in each part, make sure each header is EXACTLY the same.
292 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
293
			assumes the part files don't have individual headers and treats the
294
			subsequent headers as data rows.)
295 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
296
			that depend on it
297
	Install the staging tables:
298
		make inputs/<datasrc>/reinstall quiet=1 &
299
		For a MySQL .sql export:
300
			At prompt "[you]@vegbiendev's password:", enter your password
301
			At prompt "Enter password:", enter the value in config/bien_password
302
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
303
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
304
			tail provides a header line with the filename
305
			+1 starts at the first line, to show the whole file
306
		For every file with an error 'column "..." specified more than once':
307
			Add a header override file "+header.<ext>" in <table>/:
308
				Note: The leading "+" should sort it before the flat files.
309
					"_" unfortunately sorts *after* capital letters in ASCII.
310
				Create a text file containing the header line of the flat files
311
				Add an ! at the beginning of the line
312
					This signals cat_csv that this is a header override.
313
				For empty names, use their 0-based column # (by convention)
314
				For duplicate names, add a distinguishing suffix
315
				For long names that collided, rename them to <= 63 chars long
316
				Do NOT make readability changes in this step; that is what the
317
					map spreadsheets (below) are for.
318
				Save
319
		If you made any changes, re-run the install command above
320
	Auto-create the map spreadsheets: make inputs/<datasrc>/
321
	Map each table's columns:
322
		In each <table>/ subdir, for each "via map" map.csv:
323
			Open the map in a spreadsheet editor
324
			Open the "core map" /mappings/Veg+-VegBIEN.csv
325
			In each row of the via map, set the right column to a value from the
326
				left column of the core map
327
			Save
328
		Regenerate the derived maps: make inputs/<datasrc>/
329
	Accept the test cases:
330
		make inputs/<datasrc>/test
331
			When prompted to "Accept new test output", enter y and press ENTER
332
			If you instead get errors, do one of the following for each one:
333
			-	If the error was due to a bug, fix it
334
			-	Add a SQL function that filters or transforms the invalid data
335
			-	Make an empty mapping for the columns that produced the error.
336
				Put something in the Comments column of the map spreadsheet to
337
				prevent the automatic mapper from auto-removing the mapping.
338
			When accepting tests, it's helpful to use WinMerge
339
				(see WinMerge setup below for configuration)
340
		make inputs/<datasrc>/test by_col=1
341
			If you get errors this time, this always indicates a bug, usually in
342
				the VegBIEN unique constraints or column-based import itself
343
	Add newly-created files: make inputs/<datasrc>/add
344
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
345
	Update vegbiendev:
346
		On jupiter: svn up
347
		On local machine:
348
			./fix_perms
349
			make inputs/upload
350 10025 aaronmk
			make inputs/upload live=1
351 8458 aaronmk
		On vegbiendev:
352
			svn up
353
			make inputs/download
354 10025 aaronmk
			make inputs/download live=1
355 8458 aaronmk
			Follow the steps under Install the staging tables above
356 1773 aaronmk
357 6484 aaronmk
Datasource refreshing:
358 8458 aaronmk
	VegBank:
359
		make inputs/VegBank/vegbank.sql-remake
360
		make inputs/VegBank/reinstall quiet=1 &
361 6484 aaronmk
362 702 aaronmk
Schema changes:
363 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
364
		to update the corresponding table
365
	Remember to update the following files with any renamings:
366
		schemas/filter_ERD.csv
367
		mappings/VegCore-VegBIEN.csv
368
		mappings/verify.*.sql
369
	Regenerate schema from installed DB: make schemas/remake
370
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
371
		WARNING: This will delete the current public schema of your VegBIEN DB!
372 8837 aaronmk
	Reinstall staging tables:
373
		On local machine:
374 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
375 8845 aaronmk
			make db
376 8837 aaronmk
			. bin/reinstall_all
377
			Fix any bugs and retry until no errors
378 8846 aaronmk
			make schemas/public/install
379
				This must be run *after* the datasources are installed, because
380
				views in public depend on some of the datasources
381 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
382 8841 aaronmk
		On vegbiendev: repeat the above steps
383 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
384
			the local machine, or the live DB may be unrestorable!
385 8458 aaronmk
	Sync ERD with vegbien.sql schema:
386
		Run make schemas/vegbien.my.sql
387
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
388
		Go to File > Export > Synchronize With SQL CREATE Script...
389
		For Input File, select schemas/vegbien.my.sql
390
		Click Continue
391
		In the changes list, select each table with an arrow next to it
392
		Click Update Model
393
		Click Continue
394
		Note: The generated SQL script will be empty because we are syncing in
395
			the opposite direction
396
		Click Execute
397
		Reposition any lines that have been reset
398
		Add any new tables by dragging them from the Catalog in the left sidebar
399
			to the diagram
400
		Remove any deleted tables by right-clicking the table's diagram element,
401
			selecting Delete '<table name>', and clicking Delete
402
		Save
403
		If desired, update the graphical ERD exports (see below)
404
	Update graphical ERD exports:
405
		Go to File > Export > Export as PNG...
406
		Select schemas/vegbien.ERD.png and click Save
407
		Go to File > Export > Export as SVG...
408
		Select schemas/vegbien.ERD.svg and click Save
409
		Go to File > Export > Export as Single Page PDF...
410
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
411
		Go to File > Print...
412
		In the lower left corner, click PDF > Save as PDF...
413
		Set the Title and Author to ""
414
		Select schemas/vegbien.ERD.pdf and click Save
415
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
416
	Refactoring tips:
417
		To rename a table:
418
			In vegbien.sql, do the following:
419
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
420
					This is necessary because the table name is *everywhere*
421
				Search for <new>
422
				Manually change back any replacements inside comments
423
		To rename a column:
424
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
425
			Recreate any foreign key for the column, removing CONSTRAINT <name>
426
				This resets the foreign key name using the new column name
427
	Creating a poster of the ERD:
428
		Determine the poster size:
429
			Measure the line height (from the bottom of one line to the bottom
430
				of another): 16.3cm/24 lines = 0.679cm
431
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
432
			Zoom in as far as possible
433
			Measure the height of a capital letter: 3.5mm
434
			Measure the line height: 8.5mm
435
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
436
			Calculate the text height: 0.679cm*0.41 = 0.28cm
437
			Calculate the text height's fraction of the ERD height:
438
				0.28cm/70.8cm = 0.0040
439
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
440
			Calculate the VegBIEN poster height to make the text the same size:
441
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
442
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
443
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
444
			The minimum VegBIEN poster size is 35x54in portrait
445
		Determine the cost:
446
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
447
				charges the following for posters:
448
				base: $7.25/sq ft
449
				lamination: $3/sq ft
450
				mounting on a board: $8/sq ft
451 203 aaronmk
452 1459 aaronmk
Testing:
453 8458 aaronmk
	On a development machine, you should put the following in your .profile:
454 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
455 8458 aaronmk
		export log= n=2
456
	Mapping process: make test
457
		Including column-based import: make test by_col=1
458
			If the row-based and column-based imports produce different inserted
459
			row counts, this usually means that a table is underconstrained
460
			(the unique indexes don't cover all possible rows).
461
			This can occur if you didn't use COALESCE(field, null_value) around
462
			a nullable field in a unique index. See sql_gen.null_sentinels for
463
			the appropriate null value to use.
464
	Map spreadsheet generation: make remake
465
	Missing mappings: make missing_mappings
466
	Everything (for most complete coverage): make test-all
467 702 aaronmk
468 7183 aaronmk
Debugging:
469 8458 aaronmk
	"Binary chop" debugging:
470
		(This is primarily useful for regressions that occurred in a previous
471
		revision, which was committed without running all the tests)
472
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
473 8470 aaronmk
	.htaccess:
474
		mod_rewrite:
475 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
476
				directory, you *must* clear your browser's cache to ensure that
477
				a cached redirect is not used. this is because RewriteRule
478
				redirects are (by default) temporary, but DirectorySlash
479
				redirects are permanent.
480 8470 aaronmk
				for Firefox:
481
					press Cmd+Shift+Delete
482
					check only Cache
483
					press Enter or click Clear Now
484 7183 aaronmk
485 3783 aaronmk
WinMerge setup:
486 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
487
	Open WinMerge
488
	Go to Edit > Options and click Compare in the left sidebar
489
	Enable "Moved block detection", as described at
490
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
491
	Set Whitespace to Ignore change, as described at
492
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
493 3783 aaronmk
494 3133 aaronmk
Documentation:
495 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
496
		make schemas/public/reinstall
497
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
498
	To import and scrub just the test taxonomic names:
499
		inputs/test_taxonomic_names/test_scrub
500 3133 aaronmk
501 702 aaronmk
General:
502 8458 aaronmk
	To see a program's description, read its top-of-file comment
503
	To see a program's usage, run it without arguments
504
	To remake a directory: make <dir>/remake
505
	To remake a file: make <file>-remake