Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 3674 aaronmk
Maintenance:
11 9399 aaronmk
	on a live machine, you should put the following in your .profile:
12
		umask ug=rwx,o= # prevent files from becoming web-accessible
13 9400 aaronmk
		unset TMOUT # TMOUT causes screen to exit even with background processes
14 8459 aaronmk
	to synchronize vegbiendev, jupiter, and your local machine:
15 9886 aaronmk
		WARNING: pay careful attention to all files that will be deleted or
16
			overwritten!
17 8460 aaronmk
		install put if needed:
18
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
19 8459 aaronmk
		when changes are made on vegbiendev:
20
			on vegbiendev, upload:
21 10032 aaronmk
				overwrite=1 bin/sync_upload
22 10029 aaronmk
					then rerun with l=1 ...
23 8459 aaronmk
			on your machine, download:
24 10029 aaronmk
				overwrite=1      swap=1 src=. dest='aaronmk@jupiter:~/bien' put            --exclude=.svn inputs/VegBIEN/TWiki
25
					then rerun with l=1 ...
26 10032 aaronmk
				overwrite=1 swap=1 bin/sync_upload
27 10029 aaronmk
					then rerun with l=1 ...
28 8462 aaronmk
	to synchronize a Mac's settings with my testing machine's:
29
		download:
30
			WARNING: this will overwrite all your user's settings!
31 10034 aaronmk
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/    bin/sync_upload --exclude="/Saved Application State"
32 10029 aaronmk
				then rerun with l=1 ...
33 8462 aaronmk
		upload:
34 8719 aaronmk
			do step when changes are made on vegbiendev > on your machine, download
35 10035 aaronmk
			on jupiter: (cd ~/Dropbox/svn/; svn up)
36
			on your machine:
37
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/    bin/sync_upload --size-only # just update mtimes
38
					then rerun with l=1 ...
39 10037 aaronmk
				overwrite=1      sync_local_dir=~              sync_remote_subdir=                bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home"
40 10035 aaronmk
					then rerun with l=1 ...
41
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home     bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
42
					then rerun with l=1 ...
43 8720 aaronmk
	to backup files not in Time Machine:
44 10029 aaronmk
		inplace=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ put Users/aaronmk/VirtualBox\ VMs/
45
			then rerun with l=1 ...
46 8720 aaronmk
		stop the PostgreSQL server
47 10029 aaronmk
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo put Library/PostgreSQL/9.1/data/
48
			then rerun with l=1 ...
49 8720 aaronmk
		start the PostgreSQL server
50 8458 aaronmk
	VegCore data dictionary:
51
		Regularly, or whenever the VegCore data dictionary page
52
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
53
			is changed, regenerate mappings/VegCore.csv:
54
			make mappings/VegCore.htm-remake; make mappings/
55
			svn di mappings/VegCore.tables.redmine
56
			If there are changes, update the data dictionary's Tables section
57
			When moving terms, check that no terms were lost: svn di
58
			svn ci -m "mappings/VegCore.htm: Regenerated from wiki"
59
	Important: Whenever you install a system update that affects PostgreSQL or
60
		any of its dependencies, such as libc, you should restart the PostgreSQL
61
		server. Otherwise, you may get strange errors like "the database system
62
		is in recovery mode" which go away upon reimport, or you may not be able
63
		to access the database as the postgres superuser. This applies to both
64
		Linux and Mac OS X.
65 3674 aaronmk
66 7287 aaronmk
Single datasource import:
67 8458 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1
68
	(Re)import only: make inputs/<datasrc>/reimport by_col=1
69
	(Re)scrub: make inputs/<datasrc>/rescrub by_col=1
70
	Note that these commands also work if the datasource is not yet imported
71 7287 aaronmk
72
Full database import:
73 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
74
		breaking column-based import
75 9883 aaronmk
	On jupiter: svn up --force
76 8458 aaronmk
	On local machine:
77
		./fix_perms
78 9887 aaronmk
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
79
			your local machine" above
80 8458 aaronmk
		make inputs/upload
81 10025 aaronmk
		make inputs/upload live=1
82 8458 aaronmk
		make test by_col=1
83
			See note under Testing below
84
	On vegbiendev:
85
	Ensure there are no local modifications: svn st
86
	svn up
87
	make inputs/download
88 10025 aaronmk
	make inputs/download live=1
89 8458 aaronmk
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
90
	Update the auxiliary schemas: make schemas/reinstall
91
		The public schema will be installed separately by the import process
92
	Delete imports before the last so they won't bloat the full DB backup:
93
		make backups/vegbien.<version>.backup/remove
94
		To keep a previous import other than the public schema:
95
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
96 9970 aaronmk
	Make sure there is at least 300GB of disk space on /: df -h
97
		The import schema is 255GB, and may use additional space for temp tables
98 8458 aaronmk
		To free up space, remove backups that have been archived on jupiter:
99
			List backups/ to view older backups
100
			Check their MD5 sums using the steps under On jupiter below
101
			Remove these backups
102 9899 aaronmk
	Remove any leftover TNRS lockfile: rm inputs/.TNRS/tnrs/tnrs.make.lock
103
		Usually, the PID in it would not exist, but sometimes it now refers to
104
			a different, active process which blocks tnrs.make
105 8458 aaronmk
	screen
106
	Press ENTER
107 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
108 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
109
	unset version
110 8458 aaronmk
	Start column-based import: . bin/import_all by_col=1
111
		To use row-based import: . bin/import_all
112
		To stop all running imports: . bin/stop_imports
113
		WARNING: Do NOT run import_all in the background, or the jobs it creates
114
			won't be owned by your shell.
115
		Note that import_all will take up to an hour to import the NCBI backbone
116
			and other metadata before returning control to the shell.
117
	Wait (overnight) for the import to finish
118
	To recover from a closed terminal window: screen -r
119
	When there are no more running jobs, exit the screen
120
	Get $version: echo $version
121
	Set $version in all vegbiendev terminals: export version=<version>
122 10025 aaronmk
	Upload logs (run on vegbiendev): make inputs/upload live=1
123
	On local machine: make inputs/download-logs live=1
124 8458 aaronmk
	In PostgreSQL:
125
		Check that the provider_count and source tables contain entries for all
126
			inputs
127 9492 aaronmk
	Check that TNRS ran successfully:
128
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
129
		If the log ends in an AssertionError
130
			"assert sql.table_col_names(db, table) == header":
131
			Figure out which TNRS CSV columns have changed
132
			On local machine:
133
				Make the changes in inputs/.TNRS/schema.sql
134
				rm inputs/.TNRS/tnrs/header.csv
135
				make inputs/.TNRS/reinstall schema_only=1
136
				make schemas/public/reinstall
137
				If there are errors "column ... does not exist", etc.:
138
					Make the necessary changes in schemas/vegbien.sql
139
					make schemas/public/reinstall
140
				make schemas/remake
141
				inputs/test_taxonomic_names/test_scrub
142 9528 aaronmk
				In inputs/test_taxonomic_names/_scrub/TNRS.sql, copy the
143
					"COPY tnrs ..." statement to inputs/.TNRS/data.sql
144 9492 aaronmk
				Commit
145
			On vegbiendev:
146
				If dropping a column, save the dependent views
147
				Make the same changes in the live TNRS.tnrs table on vegbiendev
148
				If dropping a column, recreate the dependent views
149
				Restart the TNRS client: make scrub by_col=1 &
150 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
151
	In the output, search for "Command exited with non-zero status"
152
	For inputs that have this, fix the associated bug(s)
153
	If many inputs have errors, discard the current (partial) import:
154
		make schemas/$version/uninstall
155
	Otherwise, continue
156 9498 aaronmk
	Publish the new import:
157
		WARNING: Before proceeding, be sure you have done *every single*
158
			verification step listed above. Otherwise, a previous valid import
159
			could incorrectly be overwritten with a broken one.
160
		make schemas/$version/publish
161 8458 aaronmk
	unset version
162
	backups/fix_perms
163 10027 aaronmk
	make backups/upload live=1
164 8458 aaronmk
	On jupiter:
165
		cd /data/dev/aaronmk/bien/backups
166
		For each newly-archived backup:
167
			make -s <backup>.md5/test
168
			Check that "OK" is printed next to the filename
169
	On nimoy:
170
		cd /home/bien/svn/
171
		svn up
172
		export version=<version>
173
		make backups/analytical_stem.$version.csv/download
174
		In the bien_web DB:
175
			Create the analytical_stem_<version> table using its schema
176
				in schemas/vegbien.my.sql
177
		make -s backups/analytical_stem.$version.csv.md5/test
178
		Check that "OK" is printed next to the filename
179 10029 aaronmk
		table=analytical_stem_$version bin/publish_analytical_db \
180 8458 aaronmk
			backups/analytical_stem.$version.csv
181
	If desired, record the import times in inputs/import.stats.xls:
182
		Open inputs/import.stats.xls
183
		If the rightmost import is within 5 columns of column IV:
184
			Copy the current tab to <leftmost-date>~<rightmost-date>
185
			Remove the previous imports from the current tab because they are
186
				now in the copied tab instead
187
		Insert a copy of the leftmost "By column" column group before it
188
		export version=<version>
189
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
190
		Update the import date in the upper-right corner
191
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
192
		Paste the output over the # Rows/Time columns, making sure that the
193
			row counts match up with the previous import's row counts
194
		If the row counts do not match up, insert or reorder rows as needed
195
			until they do. Get the datasource names from the log file footers:
196
			tail inputs/{.,}*/*/logs/$version.log.sql
197
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
198 9497 aaronmk
	To run TNRS:
199 9996 aaronmk
		To use an import other than public: export version=<version>
200 9995 aaronmk
		make scrub &
201 8458 aaronmk
		To view progress:
202
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
203 9497 aaronmk
	To remake analytical DB:
204 9996 aaronmk
		To use an import other than public: export version=<version>
205 9497 aaronmk
		bin/make_analytical_db &
206 8458 aaronmk
		To view progress:
207
			tail -100 inputs/analytical_db/logs/make_analytical_db.log.sql
208
	To back up DB (staging tables and last import):
209 9996 aaronmk
		To use an import other than public: export version=<version>
210 8458 aaronmk
		If before renaming to public: export dump_opts=--exclude-schema=public
211
		make backups/vegbien.$version.backup/test &
212 3381 aaronmk
213 3545 aaronmk
Backups:
214 8458 aaronmk
	Archived imports:
215
		Back up: make backups/<version>.backup &
216
			Note: To back up the last import, you must archive it first:
217
				make schemas/rotate
218
		Test: make -s backups/<version>.backup/test &
219
		Restore: make backups/<version>.backup/restore &
220
		Remove: make backups/<version>.backup/remove
221 10028 aaronmk
		Download: make backups/<version>.backup/download
222 8458 aaronmk
	TNRS cache:
223
		Back up: make backups/TNRS.backup-remake &
224 10010 aaronmk
			runtime: 3 min ("real 2m48.859s")
225 8458 aaronmk
		Restore:
226
			yes|make inputs/.TNRS/uninstall
227
			make backups/TNRS.backup/restore &
228 10010 aaronmk
				runtime: 5.5 min ("real 5m35.829s")
229 8458 aaronmk
			yes|make schemas/public/reinstall
230
				Must come after TNRS restore to recreate tnrs_input_name view
231
	Full DB:
232
		Back up: make backups/vegbien.<version>.backup &
233
		Test: make -s backups/vegbien.<version>.backup/test &
234
		Restore: make backups/vegbien.<version>.backup/restore &
235 10028 aaronmk
		Download: make backups/vegbien.<version>.backup/download
236 8458 aaronmk
	Import logs:
237 10025 aaronmk
		Download: make inputs/download-logs live=1
238 554 aaronmk
239 1773 aaronmk
Datasource setup:
240 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
241 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
242
		<datasrc> may not contain spaces, and should be abbreviated.
243
		If the datasource is a herbarium, <datasrc> should be the herbarium code
244
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
245
	For MySQL inputs (exports and live DB connections):
246
		For .sql exports:
247
			Place the original .sql file in _src/ (*not* in _MySQL/)
248
			Follow the steps starting with Install the staging tables below.
249
				This is for an initial sync to get the file onto vegbiendev.
250
			On vegbiendev:
251
				Create a database for the MySQL export in phpMyAdmin
252 9494 aaronmk
				Give the bien user all database-specific privileges *except*
253
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
254
					import scripts from accidentally deleting data.
255 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
256
		mkdir inputs/<datasrc>/_MySQL/
257
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
258
		Edit _MySQL/*.make for the DB connection
259
			For a .sql export, use server=vegbiendev and --user=bien
260
		Skip the Add input data for each table section
261
	For MS Access databases:
262
		Place the .mdb or .accdb file in _src/
263
		Download and install Access To PostgreSQL from
264
			http://www.bullzip.com/download.php
265
		Use Access To PostgreSQL to export the database:
266
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
267
			Export just the data to inputs/<datasrc>/<file>.data.sql
268
		In <file>.schema.sql, make the following changes:
269
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
270
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
271
		Skip the Add input data for each table section
272
	Add input data for each table present in the datasource:
273
		For .sql exports, you must use the name of the table in the DB export
274
		For CSV files, you can use any name. It's recommended to use a table
275
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
276
		Note that if this table will be joined together with another table, its
277
			name must end in ".src"
278
		make inputs/<datasrc>/<table>/add
279
			Important: DO NOT just create an empty directory named <table>!
280
				This command also creates necessary subdirs, such as logs/.
281
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
282
			Otherwise, place the CSV(s) for the table in
283
			inputs/<datasrc>/<table>/ OR place a query joining other tables
284
			together in inputs/<datasrc>/<table>/create.sql
285
		Important: When exporting relational databases to CSVs, you MUST ensure
286
			that embedded quotes are escaped by doubling them, *not* by
287
			preceding them with a "\" as is the default in phpMyAdmin
288
		If there are multiple part files for a table, and the header is repeated
289
			in each part, make sure each header is EXACTLY the same.
290 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
291
			assumes the part files don't have individual headers and treats the
292
			subsequent headers as data rows.)
293 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
294
			that depend on it
295
	Install the staging tables:
296
		make inputs/<datasrc>/reinstall quiet=1 &
297
		For a MySQL .sql export:
298
			At prompt "[you]@vegbiendev's password:", enter your password
299
			At prompt "Enter password:", enter the value in config/bien_password
300
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
301
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
302
			tail provides a header line with the filename
303
			+1 starts at the first line, to show the whole file
304
		For every file with an error 'column "..." specified more than once':
305
			Add a header override file "+header.<ext>" in <table>/:
306
				Note: The leading "+" should sort it before the flat files.
307
					"_" unfortunately sorts *after* capital letters in ASCII.
308
				Create a text file containing the header line of the flat files
309
				Add an ! at the beginning of the line
310
					This signals cat_csv that this is a header override.
311
				For empty names, use their 0-based column # (by convention)
312
				For duplicate names, add a distinguishing suffix
313
				For long names that collided, rename them to <= 63 chars long
314
				Do NOT make readability changes in this step; that is what the
315
					map spreadsheets (below) are for.
316
				Save
317
		If you made any changes, re-run the install command above
318
	Auto-create the map spreadsheets: make inputs/<datasrc>/
319
	Map each table's columns:
320
		In each <table>/ subdir, for each "via map" map.csv:
321
			Open the map in a spreadsheet editor
322
			Open the "core map" /mappings/Veg+-VegBIEN.csv
323
			In each row of the via map, set the right column to a value from the
324
				left column of the core map
325
			Save
326
		Regenerate the derived maps: make inputs/<datasrc>/
327
	Accept the test cases:
328
		make inputs/<datasrc>/test
329
			When prompted to "Accept new test output", enter y and press ENTER
330
			If you instead get errors, do one of the following for each one:
331
			-	If the error was due to a bug, fix it
332
			-	Add a SQL function that filters or transforms the invalid data
333
			-	Make an empty mapping for the columns that produced the error.
334
				Put something in the Comments column of the map spreadsheet to
335
				prevent the automatic mapper from auto-removing the mapping.
336
			When accepting tests, it's helpful to use WinMerge
337
				(see WinMerge setup below for configuration)
338
		make inputs/<datasrc>/test by_col=1
339
			If you get errors this time, this always indicates a bug, usually in
340
				the VegBIEN unique constraints or column-based import itself
341
	Add newly-created files: make inputs/<datasrc>/add
342
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
343
	Update vegbiendev:
344
		On jupiter: svn up
345
		On local machine:
346
			./fix_perms
347
			make inputs/upload
348 10025 aaronmk
			make inputs/upload live=1
349 8458 aaronmk
		On vegbiendev:
350
			svn up
351
			make inputs/download
352 10025 aaronmk
			make inputs/download live=1
353 8458 aaronmk
			Follow the steps under Install the staging tables above
354 1773 aaronmk
355 6484 aaronmk
Datasource refreshing:
356 8458 aaronmk
	VegBank:
357
		make inputs/VegBank/vegbank.sql-remake
358
		make inputs/VegBank/reinstall quiet=1 &
359 6484 aaronmk
360 702 aaronmk
Schema changes:
361 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
362
		to update the corresponding table
363
	Remember to update the following files with any renamings:
364
		schemas/filter_ERD.csv
365
		mappings/VegCore-VegBIEN.csv
366
		mappings/verify.*.sql
367
	Regenerate schema from installed DB: make schemas/remake
368
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
369
		WARNING: This will delete the current public schema of your VegBIEN DB!
370 8837 aaronmk
	Reinstall staging tables:
371
		On local machine:
372 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
373 8845 aaronmk
			make db
374 8837 aaronmk
			. bin/reinstall_all
375
			Fix any bugs and retry until no errors
376 8846 aaronmk
			make schemas/public/install
377
				This must be run *after* the datasources are installed, because
378
				views in public depend on some of the datasources
379 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
380 8841 aaronmk
		On vegbiendev: repeat the above steps
381 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
382
			the local machine, or the live DB may be unrestorable!
383 8458 aaronmk
	Sync ERD with vegbien.sql schema:
384
		Run make schemas/vegbien.my.sql
385
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
386
		Go to File > Export > Synchronize With SQL CREATE Script...
387
		For Input File, select schemas/vegbien.my.sql
388
		Click Continue
389
		In the changes list, select each table with an arrow next to it
390
		Click Update Model
391
		Click Continue
392
		Note: The generated SQL script will be empty because we are syncing in
393
			the opposite direction
394
		Click Execute
395
		Reposition any lines that have been reset
396
		Add any new tables by dragging them from the Catalog in the left sidebar
397
			to the diagram
398
		Remove any deleted tables by right-clicking the table's diagram element,
399
			selecting Delete '<table name>', and clicking Delete
400
		Save
401
		If desired, update the graphical ERD exports (see below)
402
	Update graphical ERD exports:
403
		Go to File > Export > Export as PNG...
404
		Select schemas/vegbien.ERD.png and click Save
405
		Go to File > Export > Export as SVG...
406
		Select schemas/vegbien.ERD.svg and click Save
407
		Go to File > Export > Export as Single Page PDF...
408
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
409
		Go to File > Print...
410
		In the lower left corner, click PDF > Save as PDF...
411
		Set the Title and Author to ""
412
		Select schemas/vegbien.ERD.pdf and click Save
413
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
414
	Refactoring tips:
415
		To rename a table:
416
			In vegbien.sql, do the following:
417
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
418
					This is necessary because the table name is *everywhere*
419
				Search for <new>
420
				Manually change back any replacements inside comments
421
		To rename a column:
422
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
423
			Recreate any foreign key for the column, removing CONSTRAINT <name>
424
				This resets the foreign key name using the new column name
425
	Creating a poster of the ERD:
426
		Determine the poster size:
427
			Measure the line height (from the bottom of one line to the bottom
428
				of another): 16.3cm/24 lines = 0.679cm
429
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
430
			Zoom in as far as possible
431
			Measure the height of a capital letter: 3.5mm
432
			Measure the line height: 8.5mm
433
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
434
			Calculate the text height: 0.679cm*0.41 = 0.28cm
435
			Calculate the text height's fraction of the ERD height:
436
				0.28cm/70.8cm = 0.0040
437
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
438
			Calculate the VegBIEN poster height to make the text the same size:
439
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
440
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
441
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
442
			The minimum VegBIEN poster size is 35x54in portrait
443
		Determine the cost:
444
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
445
				charges the following for posters:
446
				base: $7.25/sq ft
447
				lamination: $3/sq ft
448
				mounting on a board: $8/sq ft
449 203 aaronmk
450 1459 aaronmk
Testing:
451 8458 aaronmk
	On a development machine, you should put the following in your .profile:
452 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
453 8458 aaronmk
		export log= n=2
454
	Mapping process: make test
455
		Including column-based import: make test by_col=1
456
			If the row-based and column-based imports produce different inserted
457
			row counts, this usually means that a table is underconstrained
458
			(the unique indexes don't cover all possible rows).
459
			This can occur if you didn't use COALESCE(field, null_value) around
460
			a nullable field in a unique index. See sql_gen.null_sentinels for
461
			the appropriate null value to use.
462
	Map spreadsheet generation: make remake
463
	Missing mappings: make missing_mappings
464
	Everything (for most complete coverage): make test-all
465 702 aaronmk
466 7183 aaronmk
Debugging:
467 8458 aaronmk
	"Binary chop" debugging:
468
		(This is primarily useful for regressions that occurred in a previous
469
		revision, which was committed without running all the tests)
470
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
471 8470 aaronmk
	.htaccess:
472
		mod_rewrite:
473 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
474
				directory, you *must* clear your browser's cache to ensure that
475
				a cached redirect is not used. this is because RewriteRule
476
				redirects are (by default) temporary, but DirectorySlash
477
				redirects are permanent.
478 8470 aaronmk
				for Firefox:
479
					press Cmd+Shift+Delete
480
					check only Cache
481
					press Enter or click Clear Now
482 7183 aaronmk
483 3783 aaronmk
WinMerge setup:
484 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
485
	Open WinMerge
486
	Go to Edit > Options and click Compare in the left sidebar
487
	Enable "Moved block detection", as described at
488
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
489
	Set Whitespace to Ignore change, as described at
490
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
491 3783 aaronmk
492 3133 aaronmk
Documentation:
493 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
494
		make schemas/public/reinstall
495
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
496
	To import and scrub just the test taxonomic names:
497
		inputs/test_taxonomic_names/test_scrub
498 3133 aaronmk
499 702 aaronmk
General:
500 8458 aaronmk
	To see a program's description, read its top-of-file comment
501
	To see a program's usage, run it without arguments
502
	To remake a directory: make <dir>/remake
503
	To remake a file: make <file>-remake