Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 3674 aaronmk
Maintenance:
11 9399 aaronmk
	on a live machine, you should put the following in your .profile:
12
		umask ug=rwx,o= # prevent files from becoming web-accessible
13 9400 aaronmk
		unset TMOUT # TMOUT causes screen to exit even with background processes
14 8459 aaronmk
	to synchronize vegbiendev, jupiter, and your local machine:
15 8460 aaronmk
		install put if needed:
16
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
17 8459 aaronmk
		when changes are made on vegbiendev:
18
			on vegbiendev, upload:
19 8723 aaronmk
				env overwrite=1             src=. dest='aaronmk@jupiter:~/bien' put --no-group --exclude=.svn --exclude=install.log.sql --exclude='*.backup' --exclude='/backups/analytical_aggregate.*.csv' --exclude='inputs/GBIF/**.data.sql' --exclude='bin/dotlockfile'
20 8459 aaronmk
					then rerun with env l=1 ...
21 8723 aaronmk
				env overwrite=1 del=        src=. dest='aaronmk@jupiter:~/bien' put --no-group --exclude=.svn --exclude=install.log.sql --exclude='inputs/GBIF/**.data.sql'
22 8459 aaronmk
					then rerun with env l=1 ...
23
			on your machine, download:
24 8945 aaronmk
				env overwrite=1      swap=1 src=. dest='aaronmk@jupiter:~/bien' put            --exclude=.svn inputs/VegBIEN/TWiki
25
					then rerun with env l=1 ...
26 8723 aaronmk
				env overwrite=1 del= swap=1 src=. dest='aaronmk@jupiter:~/bien' put            --exclude=.svn --exclude=install.log.sql --exclude='*.backup' --exclude='/backups/analytical_aggregate.*.csv' --exclude='inputs/GBIF/**.data.sql' --exclude='bin/dotlockfile'
27 8459 aaronmk
					then rerun with env l=1 ...
28 8462 aaronmk
	to synchronize a Mac's settings with my testing machine's:
29
		download:
30
			WARNING: this will overwrite all your user's settings!
31 8718 aaronmk
			env overwrite=1 swap=1 src=~/Library/ dest='aaronmk@jupiter:~/Library/' put --exclude="/Saved Application State"
32 8462 aaronmk
				then rerun with env l=1 ...
33
		upload:
34 8719 aaronmk
			do step when changes are made on vegbiendev > on your machine, download
35 8724 aaronmk
			env overwrite=1        src=~ dest='aaronmk@jupiter:~' put --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Documents/BIEN/large_files" --exclude="/Documents/BIEN/svn/backups/*.backup" --exclude="/Documents/BIEN/svn/backups/analytical_aggregate.*.csv" --exclude="/Dropbox/home"
36 8462 aaronmk
				then rerun with env l=1 ...
37 8719 aaronmk
			env overwrite=1        src=~ dest=~/Dropbox/home      put --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
38 8717 aaronmk
				then rerun with env l=1 ...
39 8720 aaronmk
	to backup files not in Time Machine:
40
		env inplace=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ put Users/aaronmk/VirtualBox\ VMs/
41
			then rerun with env l=1 ...
42
		stop the PostgreSQL server
43
		sudo env src=/ dest=/Volumes/Time\ Machine\ Backups/ put Library/PostgreSQL/9.1/data/
44
			then rerun with env l=1 ...
45
		start the PostgreSQL server
46 8458 aaronmk
	VegCore data dictionary:
47
		Regularly, or whenever the VegCore data dictionary page
48
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
49
			is changed, regenerate mappings/VegCore.csv:
50
			make mappings/VegCore.htm-remake; make mappings/
51
			svn di mappings/VegCore.tables.redmine
52
			If there are changes, update the data dictionary's Tables section
53
			When moving terms, check that no terms were lost: svn di
54
			svn ci -m "mappings/VegCore.htm: Regenerated from wiki"
55
	Important: Whenever you install a system update that affects PostgreSQL or
56
		any of its dependencies, such as libc, you should restart the PostgreSQL
57
		server. Otherwise, you may get strange errors like "the database system
58
		is in recovery mode" which go away upon reimport, or you may not be able
59
		to access the database as the postgres superuser. This applies to both
60
		Linux and Mac OS X.
61 3674 aaronmk
62 7287 aaronmk
Single datasource import:
63 8458 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1
64
	(Re)import only: make inputs/<datasrc>/reimport by_col=1
65
	(Re)scrub: make inputs/<datasrc>/rescrub by_col=1
66
	Note that these commands also work if the datasource is not yet imported
67 7287 aaronmk
68
Full database import:
69 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
70
		breaking column-based import
71 8458 aaronmk
	On jupiter: svn up
72
	On local machine:
73
		./fix_perms
74
		make inputs/upload
75
		make test by_col=1
76
			See note under Testing below
77
	On vegbiendev:
78
	Ensure there are no local modifications: svn st
79
	svn up
80
	make inputs/download
81
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
82
	Update the auxiliary schemas: make schemas/reinstall
83
		The public schema will be installed separately by the import process
84
	Delete imports before the last so they won't bloat the full DB backup:
85
		make backups/vegbien.<version>.backup/remove
86
		To keep a previous import other than the public schema:
87
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
88
	Make sure there is at least 150GB of disk space on /: df -h
89
		The import schema is 100GB, and may use additional space for temp tables
90
		To free up space, remove backups that have been archived on jupiter:
91
			List backups/ to view older backups
92
			Check their MD5 sums using the steps under On jupiter below
93
			Remove these backups
94
	screen
95
	Press ENTER
96 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
97 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
98
	unset version
99 8458 aaronmk
	Start column-based import: . bin/import_all by_col=1
100
		To use row-based import: . bin/import_all
101
		To stop all running imports: . bin/stop_imports
102
		WARNING: Do NOT run import_all in the background, or the jobs it creates
103
			won't be owned by your shell.
104
		Note that import_all will take up to an hour to import the NCBI backbone
105
			and other metadata before returning control to the shell.
106
	Wait (overnight) for the import to finish
107
	To recover from a closed terminal window: screen -r
108
	When there are no more running jobs, exit the screen
109
	Get $version: echo $version
110
	Set $version in all vegbiendev terminals: export version=<version>
111
	Upload logs (run on vegbiendev): make inputs/upload
112
	On local machine: make inputs/download-logs
113
	In PostgreSQL:
114
		Check that the provider_count and source tables contain entries for all
115
			inputs
116 9492 aaronmk
	Check that TNRS ran successfully:
117
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
118
		If the log ends in an AssertionError
119
			"assert sql.table_col_names(db, table) == header":
120
			Figure out which TNRS CSV columns have changed
121
			On local machine:
122
				Make the changes in inputs/.TNRS/schema.sql
123
				rm inputs/.TNRS/tnrs/header.csv
124
				make inputs/.TNRS/reinstall schema_only=1
125
				make schemas/public/reinstall
126
				If there are errors "column ... does not exist", etc.:
127
					Make the necessary changes in schemas/vegbien.sql
128
					make schemas/public/reinstall
129
				make schemas/remake
130
				inputs/test_taxonomic_names/test_scrub
131 9528 aaronmk
				In inputs/test_taxonomic_names/_scrub/TNRS.sql, copy the
132
					"COPY tnrs ..." statement to inputs/.TNRS/data.sql
133 9492 aaronmk
				Commit
134
			On vegbiendev:
135
				If dropping a column, save the dependent views
136
				Make the same changes in the live TNRS.tnrs table on vegbiendev
137
				If dropping a column, recreate the dependent views
138
				Restart the TNRS client: make scrub by_col=1 &
139 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
140
	In the output, search for "Command exited with non-zero status"
141
	For inputs that have this, fix the associated bug(s)
142
	If many inputs have errors, discard the current (partial) import:
143
		make schemas/$version/uninstall
144
	Otherwise, continue
145 9498 aaronmk
	Publish the new import:
146
		WARNING: Before proceeding, be sure you have done *every single*
147
			verification step listed above. Otherwise, a previous valid import
148
			could incorrectly be overwritten with a broken one.
149
		make schemas/$version/publish
150 8458 aaronmk
	unset version
151
	backups/fix_perms
152
	make backups/upload
153
	On jupiter:
154
		cd /data/dev/aaronmk/bien/backups
155
		For each newly-archived backup:
156
			make -s <backup>.md5/test
157
			Check that "OK" is printed next to the filename
158
	On nimoy:
159
		cd /home/bien/svn/
160
		svn up
161
		export version=<version>
162
		make backups/analytical_stem.$version.csv/download
163
		In the bien_web DB:
164
			Create the analytical_stem_<version> table using its schema
165
				in schemas/vegbien.my.sql
166
		make -s backups/analytical_stem.$version.csv.md5/test
167
		Check that "OK" is printed next to the filename
168
		env table=analytical_stem_$version bin/publish_analytical_db \
169
			backups/analytical_stem.$version.csv
170
	If desired, record the import times in inputs/import.stats.xls:
171
		Open inputs/import.stats.xls
172
		If the rightmost import is within 5 columns of column IV:
173
			Copy the current tab to <leftmost-date>~<rightmost-date>
174
			Remove the previous imports from the current tab because they are
175
				now in the copied tab instead
176
		Insert a copy of the leftmost "By column" column group before it
177
		export version=<version>
178
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
179
		Update the import date in the upper-right corner
180
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
181
		Paste the output over the # Rows/Time columns, making sure that the
182
			row counts match up with the previous import's row counts
183
		If the row counts do not match up, insert or reorder rows as needed
184
			until they do. Get the datasource names from the log file footers:
185
			tail inputs/{.,}*/*/logs/$version.log.sql
186
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
187 9497 aaronmk
	To run TNRS:
188 8458 aaronmk
		export version=<version>
189 9497 aaronmk
		make scrub by_col=1 &
190 8458 aaronmk
		To view progress:
191
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
192 9497 aaronmk
	To remake analytical DB:
193 8458 aaronmk
		export version=<version>
194 9497 aaronmk
		bin/make_analytical_db &
195 8458 aaronmk
		To view progress:
196
			tail -100 inputs/analytical_db/logs/make_analytical_db.log.sql
197
	To back up DB (staging tables and last import):
198
		export version=<version>
199
		If before renaming to public: export dump_opts=--exclude-schema=public
200
		make backups/vegbien.$version.backup/test &
201 3381 aaronmk
202 3545 aaronmk
Backups:
203 8458 aaronmk
	Archived imports:
204
		Back up: make backups/<version>.backup &
205
			Note: To back up the last import, you must archive it first:
206
				make schemas/rotate
207
		Test: make -s backups/<version>.backup/test &
208
		Restore: make backups/<version>.backup/restore &
209
		Remove: make backups/<version>.backup/remove
210
		Download: make backups/download
211
	TNRS cache:
212
		Back up: make backups/TNRS.backup-remake &
213
		Restore:
214
			yes|make inputs/.TNRS/uninstall
215
			make backups/TNRS.backup/restore &
216
			yes|make schemas/public/reinstall
217
				Must come after TNRS restore to recreate tnrs_input_name view
218
	Full DB:
219
		Back up: make backups/vegbien.<version>.backup &
220
		Test: make -s backups/vegbien.<version>.backup/test &
221
		Restore: make backups/vegbien.<version>.backup/restore &
222
		Download: make backups/download
223
	Import logs:
224
		Download: make inputs/download-logs
225 554 aaronmk
226 1773 aaronmk
Datasource setup:
227 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
228 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
229
		<datasrc> may not contain spaces, and should be abbreviated.
230
		If the datasource is a herbarium, <datasrc> should be the herbarium code
231
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
232
	For MySQL inputs (exports and live DB connections):
233
		For .sql exports:
234
			Place the original .sql file in _src/ (*not* in _MySQL/)
235
			Follow the steps starting with Install the staging tables below.
236
				This is for an initial sync to get the file onto vegbiendev.
237
			On vegbiendev:
238
				Create a database for the MySQL export in phpMyAdmin
239 9494 aaronmk
				Give the bien user all database-specific privileges *except*
240
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
241
					import scripts from accidentally deleting data.
242 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
243
		mkdir inputs/<datasrc>/_MySQL/
244
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
245
		Edit _MySQL/*.make for the DB connection
246
			For a .sql export, use server=vegbiendev and --user=bien
247
		Skip the Add input data for each table section
248
	For MS Access databases:
249
		Place the .mdb or .accdb file in _src/
250
		Download and install Access To PostgreSQL from
251
			http://www.bullzip.com/download.php
252
		Use Access To PostgreSQL to export the database:
253
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
254
			Export just the data to inputs/<datasrc>/<file>.data.sql
255
		In <file>.schema.sql, make the following changes:
256
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
257
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
258
		Skip the Add input data for each table section
259
	Add input data for each table present in the datasource:
260
		For .sql exports, you must use the name of the table in the DB export
261
		For CSV files, you can use any name. It's recommended to use a table
262
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
263
		Note that if this table will be joined together with another table, its
264
			name must end in ".src"
265
		make inputs/<datasrc>/<table>/add
266
			Important: DO NOT just create an empty directory named <table>!
267
				This command also creates necessary subdirs, such as logs/.
268
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
269
			Otherwise, place the CSV(s) for the table in
270
			inputs/<datasrc>/<table>/ OR place a query joining other tables
271
			together in inputs/<datasrc>/<table>/create.sql
272
		Important: When exporting relational databases to CSVs, you MUST ensure
273
			that embedded quotes are escaped by doubling them, *not* by
274
			preceding them with a "\" as is the default in phpMyAdmin
275
		If there are multiple part files for a table, and the header is repeated
276
			in each part, make sure each header is EXACTLY the same.
277 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
278
			assumes the part files don't have individual headers and treats the
279
			subsequent headers as data rows.)
280 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
281
			that depend on it
282
	Install the staging tables:
283
		make inputs/<datasrc>/reinstall quiet=1 &
284
		For a MySQL .sql export:
285
			At prompt "[you]@vegbiendev's password:", enter your password
286
			At prompt "Enter password:", enter the value in config/bien_password
287
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
288
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
289
			tail provides a header line with the filename
290
			+1 starts at the first line, to show the whole file
291
		For every file with an error 'column "..." specified more than once':
292
			Add a header override file "+header.<ext>" in <table>/:
293
				Note: The leading "+" should sort it before the flat files.
294
					"_" unfortunately sorts *after* capital letters in ASCII.
295
				Create a text file containing the header line of the flat files
296
				Add an ! at the beginning of the line
297
					This signals cat_csv that this is a header override.
298
				For empty names, use their 0-based column # (by convention)
299
				For duplicate names, add a distinguishing suffix
300
				For long names that collided, rename them to <= 63 chars long
301
				Do NOT make readability changes in this step; that is what the
302
					map spreadsheets (below) are for.
303
				Save
304
		If you made any changes, re-run the install command above
305
	Auto-create the map spreadsheets: make inputs/<datasrc>/
306
	Map each table's columns:
307
		In each <table>/ subdir, for each "via map" map.csv:
308
			Open the map in a spreadsheet editor
309
			Open the "core map" /mappings/Veg+-VegBIEN.csv
310
			In each row of the via map, set the right column to a value from the
311
				left column of the core map
312
			Save
313
		Regenerate the derived maps: make inputs/<datasrc>/
314
	Accept the test cases:
315
		make inputs/<datasrc>/test
316
			When prompted to "Accept new test output", enter y and press ENTER
317
			If you instead get errors, do one of the following for each one:
318
			-	If the error was due to a bug, fix it
319
			-	Add a SQL function that filters or transforms the invalid data
320
			-	Make an empty mapping for the columns that produced the error.
321
				Put something in the Comments column of the map spreadsheet to
322
				prevent the automatic mapper from auto-removing the mapping.
323
			When accepting tests, it's helpful to use WinMerge
324
				(see WinMerge setup below for configuration)
325
		make inputs/<datasrc>/test by_col=1
326
			If you get errors this time, this always indicates a bug, usually in
327
				the VegBIEN unique constraints or column-based import itself
328
	Add newly-created files: make inputs/<datasrc>/add
329
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
330
	Update vegbiendev:
331
		On jupiter: svn up
332
		On local machine:
333
			./fix_perms
334
			make inputs/upload
335
		On vegbiendev:
336
			svn up
337
			make inputs/download
338
			Follow the steps under Install the staging tables above
339 1773 aaronmk
340 6484 aaronmk
Datasource refreshing:
341 8458 aaronmk
	VegBank:
342
		make inputs/VegBank/vegbank.sql-remake
343
		make inputs/VegBank/reinstall quiet=1 &
344 6484 aaronmk
345 702 aaronmk
Schema changes:
346 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
347
		to update the corresponding table
348
	Remember to update the following files with any renamings:
349
		schemas/filter_ERD.csv
350
		mappings/VegCore-VegBIEN.csv
351
		mappings/verify.*.sql
352
	Regenerate schema from installed DB: make schemas/remake
353
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
354
		WARNING: This will delete the current public schema of your VegBIEN DB!
355 8837 aaronmk
	Reinstall staging tables:
356
		On local machine:
357 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
358 8845 aaronmk
			make db
359 8837 aaronmk
			. bin/reinstall_all
360
			Fix any bugs and retry until no errors
361 8846 aaronmk
			make schemas/public/install
362
				This must be run *after* the datasources are installed, because
363
				views in public depend on some of the datasources
364 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
365 8841 aaronmk
		On vegbiendev: repeat the above steps
366 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
367
			the local machine, or the live DB may be unrestorable!
368 8458 aaronmk
	Sync ERD with vegbien.sql schema:
369
		Run make schemas/vegbien.my.sql
370
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
371
		Go to File > Export > Synchronize With SQL CREATE Script...
372
		For Input File, select schemas/vegbien.my.sql
373
		Click Continue
374
		In the changes list, select each table with an arrow next to it
375
		Click Update Model
376
		Click Continue
377
		Note: The generated SQL script will be empty because we are syncing in
378
			the opposite direction
379
		Click Execute
380
		Reposition any lines that have been reset
381
		Add any new tables by dragging them from the Catalog in the left sidebar
382
			to the diagram
383
		Remove any deleted tables by right-clicking the table's diagram element,
384
			selecting Delete '<table name>', and clicking Delete
385
		Save
386
		If desired, update the graphical ERD exports (see below)
387
	Update graphical ERD exports:
388
		Go to File > Export > Export as PNG...
389
		Select schemas/vegbien.ERD.png and click Save
390
		Go to File > Export > Export as SVG...
391
		Select schemas/vegbien.ERD.svg and click Save
392
		Go to File > Export > Export as Single Page PDF...
393
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
394
		Go to File > Print...
395
		In the lower left corner, click PDF > Save as PDF...
396
		Set the Title and Author to ""
397
		Select schemas/vegbien.ERD.pdf and click Save
398
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
399
	Refactoring tips:
400
		To rename a table:
401
			In vegbien.sql, do the following:
402
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
403
					This is necessary because the table name is *everywhere*
404
				Search for <new>
405
				Manually change back any replacements inside comments
406
		To rename a column:
407
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
408
			Recreate any foreign key for the column, removing CONSTRAINT <name>
409
				This resets the foreign key name using the new column name
410
	Creating a poster of the ERD:
411
		Determine the poster size:
412
			Measure the line height (from the bottom of one line to the bottom
413
				of another): 16.3cm/24 lines = 0.679cm
414
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
415
			Zoom in as far as possible
416
			Measure the height of a capital letter: 3.5mm
417
			Measure the line height: 8.5mm
418
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
419
			Calculate the text height: 0.679cm*0.41 = 0.28cm
420
			Calculate the text height's fraction of the ERD height:
421
				0.28cm/70.8cm = 0.0040
422
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
423
			Calculate the VegBIEN poster height to make the text the same size:
424
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
425
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
426
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
427
			The minimum VegBIEN poster size is 35x54in portrait
428
		Determine the cost:
429
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
430
				charges the following for posters:
431
				base: $7.25/sq ft
432
				lamination: $3/sq ft
433
				mounting on a board: $8/sq ft
434 203 aaronmk
435 1459 aaronmk
Testing:
436 8458 aaronmk
	On a development machine, you should put the following in your .profile:
437 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
438 8458 aaronmk
		export log= n=2
439
	Mapping process: make test
440
		Including column-based import: make test by_col=1
441
			If the row-based and column-based imports produce different inserted
442
			row counts, this usually means that a table is underconstrained
443
			(the unique indexes don't cover all possible rows).
444
			This can occur if you didn't use COALESCE(field, null_value) around
445
			a nullable field in a unique index. See sql_gen.null_sentinels for
446
			the appropriate null value to use.
447
	Map spreadsheet generation: make remake
448
	Missing mappings: make missing_mappings
449
	Everything (for most complete coverage): make test-all
450 702 aaronmk
451 7183 aaronmk
Debugging:
452 8458 aaronmk
	"Binary chop" debugging:
453
		(This is primarily useful for regressions that occurred in a previous
454
		revision, which was committed without running all the tests)
455
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
456 8470 aaronmk
	.htaccess:
457
		mod_rewrite:
458 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
459
				directory, you *must* clear your browser's cache to ensure that
460
				a cached redirect is not used. this is because RewriteRule
461
				redirects are (by default) temporary, but DirectorySlash
462
				redirects are permanent.
463 8470 aaronmk
				for Firefox:
464
					press Cmd+Shift+Delete
465
					check only Cache
466
					press Enter or click Clear Now
467 7183 aaronmk
468 3783 aaronmk
WinMerge setup:
469 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
470
	Open WinMerge
471
	Go to Edit > Options and click Compare in the left sidebar
472
	Enable "Moved block detection", as described at
473
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
474
	Set Whitespace to Ignore change, as described at
475
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
476 3783 aaronmk
477 3133 aaronmk
Documentation:
478 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
479
		make schemas/public/reinstall
480
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
481
	To import and scrub just the test taxonomic names:
482
		inputs/test_taxonomic_names/test_scrub
483 3133 aaronmk
484 702 aaronmk
General:
485 8458 aaronmk
	To see a program's description, read its top-of-file comment
486
	To see a program's usage, run it without arguments
487
	To remake a directory: make <dir>/remake
488
	To remake a file: make <file>-remake