Project

General

Profile

1 702 aaronmk
Installation:
2 8458 aaronmk
	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien
3
	cd bien/
4
	Install: make install
5
		WARNING: This will delete the current public schema of your VegBIEN DB!
6
	Uninstall: make uninstall
7
		WARNING: This will delete your entire VegBIEN DB!
8
		This includes all archived imports and staging tables.
9 554 aaronmk
10 7287 aaronmk
Single datasource import:
11 10804 aaronmk
	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 &
12
	(Re)import only: make inputs/<datasrc>/reimport by_col=1 &
13 8458 aaronmk
	Note that these commands also work if the datasource is not yet imported
14 11263 aaronmk
	Remake analytical DB: see Full database import > To remake analytical DB
15 7287 aaronmk
16
Full database import:
17 9499 aaronmk
	WARNING: You must perform *every single* step listed below, to avoid
18
		breaking column-based import
19 9883 aaronmk
	On jupiter: svn up --force
20 8458 aaronmk
	On local machine:
21 9887 aaronmk
		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and
22 11258 aaronmk
			your local machine"
23 8458 aaronmk
		make inputs/upload
24 10025 aaronmk
		make inputs/upload live=1
25 8458 aaronmk
		make test by_col=1
26 10549 aaronmk
			if you encounter errors, they are most likely related to the
27
				PostgreSQL error parsing in /lib/sql.py parse_exception()
28 8458 aaronmk
			See note under Testing below
29
	On vegbiendev:
30
	Ensure there are no local modifications: svn st
31
	svn up
32
	make inputs/download
33 10025 aaronmk
	make inputs/download live=1
34 8458 aaronmk
	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall
35
	Update the auxiliary schemas: make schemas/reinstall
36
		The public schema will be installed separately by the import process
37
	Delete imports before the last so they won't bloat the full DB backup:
38
		make backups/vegbien.<version>.backup/remove
39
		To keep a previous import other than the public schema:
40
			export dump_opts='--exclude-schema=public --exclude-schema=<version>'
41 9970 aaronmk
	Make sure there is at least 300GB of disk space on /: df -h
42 10603 aaronmk
		The import schema is 230GB, and may use additional space for temp tables
43 8458 aaronmk
		To free up space, remove backups that have been archived on jupiter:
44
			List backups/ to view older backups
45
			Check their MD5 sums using the steps under On jupiter below
46
			Remove these backups
47
	screen
48
	Press ENTER
49 9531 aaronmk
	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs
50 9532 aaronmk
	unset TMOUT # TMOUT causes screen to exit even with background processes
51
	unset version
52 10579 aaronmk
	Start column-based import: . bin/import_all
53
		To use row-based import: . bin/import_all by_col=
54 8458 aaronmk
		To stop all running imports: . bin/stop_imports
55
		WARNING: Do NOT run import_all in the background, or the jobs it creates
56
			won't be owned by your shell.
57
		Note that import_all will take up to an hour to import the NCBI backbone
58
			and other metadata before returning control to the shell.
59 10850 aaronmk
	Wait (4 days) for the import to finish
60 8458 aaronmk
	To recover from a closed terminal window: screen -r
61 10583 aaronmk
	To restart an aborted import for a specific table:
62
		export version=<version>
63 10584 aaronmk
		make inputs/<datasrc>/<table>/import_scrub by_col=1 continue=1 &
64 10588 aaronmk
		bin/after_import $! & # $! can also be obtained from `jobs -l`
65 8458 aaronmk
	Get $version: echo $version
66
	Set $version in all vegbiendev terminals: export version=<version>
67 10557 aaronmk
	When there are no more running jobs, exit the screen
68 10025 aaronmk
	Upload logs (run on vegbiendev): make inputs/upload live=1
69
	On local machine: make inputs/download-logs live=1
70 8458 aaronmk
	In PostgreSQL:
71
		Check that the provider_count and source tables contain entries for all
72
			inputs
73 9492 aaronmk
	Check that TNRS ran successfully:
74
		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
75
		If the log ends in an AssertionError
76
			"assert sql.table_col_names(db, table) == header":
77
			Figure out which TNRS CSV columns have changed
78
			On local machine:
79 10784 aaronmk
				Make the changes in the DB's TNRS and public schemas
80
				rm=1 inputs/.TNRS/schema.sql.run export_
81 9492 aaronmk
				make schemas/remake
82 10785 aaronmk
				inputs/test_taxonomic_names/test_scrub # re-run TNRS
83 10784 aaronmk
				rm=1 inputs/.TNRS/data.sql.run export_
84 9492 aaronmk
				Commit
85
			On vegbiendev:
86
				If dropping a column, save the dependent views
87
				Make the same changes in the live TNRS.tnrs table on vegbiendev
88
				If dropping a column, recreate the dependent views
89
				Restart the TNRS client: make scrub by_col=1 &
90 8458 aaronmk
	tail inputs/{.,}*/*/logs/$version.log.sql
91
	In the output, search for "Command exited with non-zero status"
92
	For inputs that have this, fix the associated bug(s)
93
	If many inputs have errors, discard the current (partial) import:
94
		make schemas/$version/uninstall
95
	Otherwise, continue
96 9498 aaronmk
	Publish the new import:
97
		WARNING: Before proceeding, be sure you have done *every single*
98
			verification step listed above. Otherwise, a previous valid import
99
			could incorrectly be overwritten with a broken one.
100 10864 aaronmk
		make schemas/$version/publish # runtime: 1 min ("real 1m10.451s")
101 8458 aaronmk
	unset version
102 10027 aaronmk
	make backups/upload live=1
103 8458 aaronmk
	On jupiter:
104
		cd /data/dev/aaronmk/bien/backups
105
		For each newly-archived backup:
106
			make -s <backup>.md5/test
107
			Check that "OK" is printed next to the filename
108
	On nimoy:
109
		cd /home/bien/svn/
110
		svn up
111
		export version=<version>
112
		make backups/analytical_stem.$version.csv/download
113
		In the bien_web DB:
114
			Create the analytical_stem_<version> table using its schema
115
				in schemas/vegbien.my.sql
116
		make -s backups/analytical_stem.$version.csv.md5/test
117
		Check that "OK" is printed next to the filename
118 10029 aaronmk
		table=analytical_stem_$version bin/publish_analytical_db \
119 8458 aaronmk
			backups/analytical_stem.$version.csv
120
	If desired, record the import times in inputs/import.stats.xls:
121
		Open inputs/import.stats.xls
122
		If the rightmost import is within 5 columns of column IV:
123
			Copy the current tab to <leftmost-date>~<rightmost-date>
124
			Remove the previous imports from the current tab because they are
125
				now in the copied tab instead
126
		Insert a copy of the leftmost "By column" column group before it
127
		export version=<version>
128
		bin/import_date inputs/{.,}*/*/logs/$version.log.sql
129
		Update the import date in the upper-right corner
130
		bin/import_times inputs/{.,}*/*/logs/$version.log.sql
131
		Paste the output over the # Rows/Time columns, making sure that the
132
			row counts match up with the previous import's row counts
133
		If the row counts do not match up, insert or reorder rows as needed
134
			until they do. Get the datasource names from the log file footers:
135
			tail inputs/{.,}*/*/logs/$version.log.sql
136
		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
137 10885 aaronmk
	Running individual steps separately:
138 9497 aaronmk
	To run TNRS:
139 9996 aaronmk
		To use an import other than public: export version=<version>
140 9995 aaronmk
		make scrub &
141 8458 aaronmk
		To view progress:
142
			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql
143 9497 aaronmk
	To remake analytical DB:
144 9996 aaronmk
		To use an import other than public: export version=<version>
145 11089 aaronmk
		bin/make_analytical_db & # runtime: 13 h ("12:43:57elapsed")
146 8458 aaronmk
		To view progress:
147 10600 aaronmk
			tail -150 inputs/analytical_db/logs/make_analytical_db.log.sql
148 8458 aaronmk
	To back up DB (staging tables and last import):
149 10578 aaronmk
		To use an import *other than public*: export version=<version>
150 10743 aaronmk
		make backups/TNRS.backup-remake &
151 10577 aaronmk
		dump_opts=--exclude-schema=public make backups/vegbien.$version.backup/test &
152 10578 aaronmk
			If after renaming to public, instead set dump_opts='' and replace
153
			$version with the appropriate revision
154 10744 aaronmk
		make backups/upload live=1
155 3381 aaronmk
156 1773 aaronmk
Datasource setup:
157 11090 aaronmk
	Example steps for a datasource: wiki.vegpath.org/Import_process_for_Madidi
158 8469 aaronmk
	umask ug=rwx,o= # prevent files from becoming web-accessible
159 8458 aaronmk
	Add a new datasource: make inputs/<datasrc>/add
160
		<datasrc> may not contain spaces, and should be abbreviated.
161
		If the datasource is a herbarium, <datasrc> should be the herbarium code
162
			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
163 11018 aaronmk
	For a new-style datasource (one containing a ./run runscript):
164 11019 aaronmk
		"cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/<datasrc>/
165 8458 aaronmk
	For MySQL inputs (exports and live DB connections):
166
		For .sql exports:
167
			Place the original .sql file in _src/ (*not* in _MySQL/)
168
			Follow the steps starting with Install the staging tables below.
169
				This is for an initial sync to get the file onto vegbiendev.
170
			On vegbiendev:
171
				Create a database for the MySQL export in phpMyAdmin
172 9494 aaronmk
				Give the bien user all database-specific privileges *except*
173
					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the
174
					import scripts from accidentally deleting data.
175 8458 aaronmk
				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &
176
		mkdir inputs/<datasrc>/_MySQL/
177
		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
178
		Edit _MySQL/*.make for the DB connection
179
			For a .sql export, use server=vegbiendev and --user=bien
180
		Skip the Add input data for each table section
181
	For MS Access databases:
182
		Place the .mdb or .accdb file in _src/
183
		Download and install Access To PostgreSQL from
184
			http://www.bullzip.com/download.php
185
		Use Access To PostgreSQL to export the database:
186
			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql
187
			Export just the data to inputs/<datasrc>/<file>.data.sql
188
		In <file>.schema.sql, make the following changes:
189
			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"
190
			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"
191
		Skip the Add input data for each table section
192
	Add input data for each table present in the datasource:
193
		For .sql exports, you must use the name of the table in the DB export
194
		For CSV files, you can use any name. It's recommended to use a table
195
			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
196
		Note that if this table will be joined together with another table, its
197
			name must end in ".src"
198
		make inputs/<datasrc>/<table>/add
199
			Important: DO NOT just create an empty directory named <table>!
200
				This command also creates necessary subdirs, such as logs/.
201
		If the table is in a .sql export: make inputs/<datasrc>/<table>/install
202
			Otherwise, place the CSV(s) for the table in
203
			inputs/<datasrc>/<table>/ OR place a query joining other tables
204
			together in inputs/<datasrc>/<table>/create.sql
205
		Important: When exporting relational databases to CSVs, you MUST ensure
206
			that embedded quotes are escaped by doubling them, *not* by
207
			preceding them with a "\" as is the default in phpMyAdmin
208
		If there are multiple part files for a table, and the header is repeated
209
			in each part, make sure each header is EXACTLY the same.
210 8466 aaronmk
			(If the headers are not the same, the CSV concatenation script
211
			assumes the part files don't have individual headers and treats the
212
			subsequent headers as data rows.)
213 8458 aaronmk
		Add <table> to inputs/<datasrc>/import_order.txt before other tables
214
			that depend on it
215 11018 aaronmk
		For a new-style datasource:
216
			"cp" -f inputs/.NCBI/nodes/run inputs/<datasrc>/<table>/
217
			inputs/<datasrc>/<table>/run
218 8458 aaronmk
	Install the staging tables:
219
		make inputs/<datasrc>/reinstall quiet=1 &
220
		For a MySQL .sql export:
221
			At prompt "[you]@vegbiendev's password:", enter your password
222
			At prompt "Enter password:", enter the value in config/bien_password
223
		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
224
		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
225
			tail provides a header line with the filename
226
			+1 starts at the first line, to show the whole file
227
		For every file with an error 'column "..." specified more than once':
228
			Add a header override file "+header.<ext>" in <table>/:
229
				Note: The leading "+" should sort it before the flat files.
230
					"_" unfortunately sorts *after* capital letters in ASCII.
231
				Create a text file containing the header line of the flat files
232
				Add an ! at the beginning of the line
233
					This signals cat_csv that this is a header override.
234
				For empty names, use their 0-based column # (by convention)
235
				For duplicate names, add a distinguishing suffix
236
				For long names that collided, rename them to <= 63 chars long
237
				Do NOT make readability changes in this step; that is what the
238
					map spreadsheets (below) are for.
239
				Save
240
		If you made any changes, re-run the install command above
241
	Auto-create the map spreadsheets: make inputs/<datasrc>/
242
	Map each table's columns:
243
		In each <table>/ subdir, for each "via map" map.csv:
244
			Open the map in a spreadsheet editor
245
			Open the "core map" /mappings/Veg+-VegBIEN.csv
246
			In each row of the via map, set the right column to a value from the
247
				left column of the core map
248
			Save
249
		Regenerate the derived maps: make inputs/<datasrc>/
250
	Accept the test cases:
251 11018 aaronmk
		For a new-style datasource:
252
			inputs/<datasrc>/run
253
			svn di inputs/<datasrc>/*/test.xml.ref
254
			If you get errors, follow the steps for old-style datasources below
255
		For an old-style datasource:
256
			make inputs/<datasrc>/test
257 8458 aaronmk
			When prompted to "Accept new test output", enter y and press ENTER
258
			If you instead get errors, do one of the following for each one:
259
			-	If the error was due to a bug, fix it
260
			-	Add a SQL function that filters or transforms the invalid data
261
			-	Make an empty mapping for the columns that produced the error.
262
				Put something in the Comments column of the map spreadsheet to
263
				prevent the automatic mapper from auto-removing the mapping.
264
			When accepting tests, it's helpful to use WinMerge
265
				(see WinMerge setup below for configuration)
266
		make inputs/<datasrc>/test by_col=1
267
			If you get errors this time, this always indicates a bug, usually in
268
				the VegBIEN unique constraints or column-based import itself
269
	Add newly-created files: make inputs/<datasrc>/add
270
	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
271
	Update vegbiendev:
272
		On jupiter: svn up
273
		On local machine:
274
			./fix_perms
275
			make inputs/upload
276 10025 aaronmk
			make inputs/upload live=1
277 8458 aaronmk
		On vegbiendev:
278
			svn up
279
			make inputs/download
280 10025 aaronmk
			make inputs/download live=1
281 8458 aaronmk
			Follow the steps under Install the staging tables above
282 1773 aaronmk
283 10884 aaronmk
Maintenance:
284
	on a live machine, you should put the following in your .profile:
285
--
286
# make svn files web-accessible. this does not affect unversioned files, because
287
# these get the right permissions on the local machine instead.
288
umask ug=rwx,o=rx
289
290
unset TMOUT # TMOUT causes screen to exit even with background processes
291
--
292
	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:
293
		on vegbiendev: make postgres-Linux
294
	to synchronize vegbiendev, jupiter, and your local machine:
295
		WARNING: pay careful attention to all files that will be deleted or
296
			overwritten!
297
		install put if needed:
298
			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it
299
		when changes are made on vegbiendev:
300
			on all machines: ./fix_perms
301
			on vegbiendev, upload:
302 11259 aaronmk
				overwrite=1 bin/sync_upload --size-only
303 10884 aaronmk
					then rerun with l=1 ...
304
			on your machine, download:
305
				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki
306
					then rerun with l=1 ...
307 11259 aaronmk
				overwrite=1 swap=1 bin/sync_upload --size-only
308 10884 aaronmk
					then rerun with l=1 ...
309
				make backups/TNRS.backup/download live=1
310
				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms
311
					then rerun with l=1 ...
312
	to synchronize a Mac's settings with my testing machine's:
313
		download:
314
			WARNING: this will overwrite all your user's settings!
315
			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"
316
				then rerun with l=1 ...
317
		upload:
318
			do step when changes are made on vegbiendev > on your machine, download
319
			on jupiter: (cd ~/Dropbox/svn/; up)
320
			on your machine:
321
				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes
322
					then rerun with l=1 ...
323
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"
324
					then rerun with l=1 ...
325
				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"
326
					then rerun with l=1 ...
327
		upload just the VirtualBox VMs:
328
			on your machine:
329
				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"
330
					then rerun with l=1 ...
331
	to backup files not in Time Machine:
332
		pg_ctl. stop # stop the PostgreSQL server
333 10981 aaronmk
		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.1/data/
334 10884 aaronmk
			then rerun with l=1 ...
335
		pg_ctl. start # start the PostgreSQL server
336
	VegCore data dictionary:
337
		Regularly, or whenever the VegCore data dictionary page
338
			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)
339
			is changed, regenerate mappings/VegCore.csv:
340
			make mappings/VegCore.htm-remake; make mappings/
341
			svn di mappings/VegCore.tables.redmine
342
			If there are changes, update the data dictionary's Tables section
343
			When moving terms, check that no terms were lost: svn di
344
			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'
345
	Important: Whenever you install a system update that affects PostgreSQL or
346
		any of its dependencies, such as libc, you should restart the PostgreSQL
347
		server. Otherwise, you may get strange errors like "the database system
348
		is in recovery mode" which go away upon reimport, or you may not be able
349
		to access the database as the postgres superuser. This applies to both
350
		Linux and Mac OS X.
351
352
Backups:
353
	Archived imports:
354
		Back up: make backups/<version>.backup &
355
			Note: To back up the last import, you must archive it first:
356
				make schemas/rotate
357
		Test: make -s backups/<version>.backup/test &
358
		Restore: make backups/<version>.backup/restore &
359
		Remove: make backups/<version>.backup/remove
360
		Download: make backups/<version>.backup/download
361
	TNRS cache:
362
		Back up: make backups/TNRS.backup-remake &
363
			runtime: 3 min ("real 2m48.859s")
364
		Restore:
365
			yes|make inputs/.TNRS/uninstall
366
			make backups/TNRS.backup/restore &
367
				runtime: 5.5 min ("real 5m35.829s")
368
			yes|make schemas/public/reinstall
369
				Must come after TNRS restore to recreate tnrs_input_name view
370
	Full DB:
371
		Back up: make backups/vegbien.<version>.backup &
372
		Test: make -s backups/vegbien.<version>.backup/test &
373
		Restore: make backups/vegbien.<version>.backup/restore &
374
		Download: make backups/vegbien.<version>.backup/download
375
	Import logs:
376
		Download: make inputs/download-logs live=1
377
378 6484 aaronmk
Datasource refreshing:
379 8458 aaronmk
	VegBank:
380
		make inputs/VegBank/vegbank.sql-remake
381
		make inputs/VegBank/reinstall quiet=1 &
382 6484 aaronmk
383 702 aaronmk
Schema changes:
384 8458 aaronmk
	When changing the analytical views, run sync_analytical_..._to_view()
385
		to update the corresponding table
386
	Remember to update the following files with any renamings:
387
		schemas/filter_ERD.csv
388
		mappings/VegCore-VegBIEN.csv
389
		mappings/verify.*.sql
390
	Regenerate schema from installed DB: make schemas/remake
391
	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall
392
		WARNING: This will delete the current public schema of your VegBIEN DB!
393 8837 aaronmk
	Reinstall staging tables:
394
		On local machine:
395 8840 aaronmk
			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'
396 8845 aaronmk
			make db
397 8837 aaronmk
			. bin/reinstall_all
398
			Fix any bugs and retry until no errors
399 8846 aaronmk
			make schemas/public/install
400
				This must be run *after* the datasources are installed, because
401
				views in public depend on some of the datasources
402 8842 aaronmk
			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'
403 8841 aaronmk
		On vegbiendev: repeat the above steps
404 8837 aaronmk
			WARNING: Do not run this until reinstall_all runs successfully on
405
			the local machine, or the live DB may be unrestorable!
406 8458 aaronmk
	Sync ERD with vegbien.sql schema:
407
		Run make schemas/vegbien.my.sql
408
		Open schemas/vegbien.ERD.mwb in MySQLWorkbench
409
		Go to File > Export > Synchronize With SQL CREATE Script...
410
		For Input File, select schemas/vegbien.my.sql
411
		Click Continue
412
		In the changes list, select each table with an arrow next to it
413
		Click Update Model
414
		Click Continue
415
		Note: The generated SQL script will be empty because we are syncing in
416
			the opposite direction
417
		Click Execute
418
		Reposition any lines that have been reset
419
		Add any new tables by dragging them from the Catalog in the left sidebar
420
			to the diagram
421
		Remove any deleted tables by right-clicking the table's diagram element,
422
			selecting Delete '<table name>', and clicking Delete
423
		Save
424
		If desired, update the graphical ERD exports (see below)
425
	Update graphical ERD exports:
426
		Go to File > Export > Export as PNG...
427
		Select schemas/vegbien.ERD.png and click Save
428
		Go to File > Export > Export as SVG...
429
		Select schemas/vegbien.ERD.svg and click Save
430
		Go to File > Export > Export as Single Page PDF...
431
		Select schemas/vegbien.ERD.1_pg.pdf and click Save
432
		Go to File > Print...
433
		In the lower left corner, click PDF > Save as PDF...
434
		Set the Title and Author to ""
435
		Select schemas/vegbien.ERD.pdf and click Save
436
		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"
437
	Refactoring tips:
438
		To rename a table:
439
			In vegbien.sql, do the following:
440
				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
441
					This is necessary because the table name is *everywhere*
442
				Search for <new>
443
				Manually change back any replacements inside comments
444
		To rename a column:
445
			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
446
			Recreate any foreign key for the column, removing CONSTRAINT <name>
447
				This resets the foreign key name using the new column name
448
	Creating a poster of the ERD:
449
		Determine the poster size:
450
			Measure the line height (from the bottom of one line to the bottom
451
				of another): 16.3cm/24 lines = 0.679cm
452
			Measure the height of the ERD: 35.4cm*2 = 70.8cm
453
			Zoom in as far as possible
454
			Measure the height of a capital letter: 3.5mm
455
			Measure the line height: 8.5mm
456
			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
457
			Calculate the text height: 0.679cm*0.41 = 0.28cm
458
			Calculate the text height's fraction of the ERD height:
459
				0.28cm/70.8cm = 0.0040
460
			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
461
			Calculate the VegBIEN poster height to make the text the same size:
462
				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
463
			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
464
			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
465
			The minimum VegBIEN poster size is 35x54in portrait
466
		Determine the cost:
467
			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
468
				charges the following for posters:
469
				base: $7.25/sq ft
470
				lamination: $3/sq ft
471
				mounting on a board: $8/sq ft
472 203 aaronmk
473 1459 aaronmk
Testing:
474 8458 aaronmk
	On a development machine, you should put the following in your .profile:
475 8469 aaronmk
		umask ug=rwx,o= # prevent files from becoming web-accessible
476 8458 aaronmk
		export log= n=2
477
	Mapping process: make test
478
		Including column-based import: make test by_col=1
479
			If the row-based and column-based imports produce different inserted
480
			row counts, this usually means that a table is underconstrained
481
			(the unique indexes don't cover all possible rows).
482
			This can occur if you didn't use COALESCE(field, null_value) around
483
			a nullable field in a unique index. See sql_gen.null_sentinels for
484
			the appropriate null value to use.
485
	Map spreadsheet generation: make remake
486
	Missing mappings: make missing_mappings
487
	Everything (for most complete coverage): make test-all
488 702 aaronmk
489 7183 aaronmk
Debugging:
490 8458 aaronmk
	"Binary chop" debugging:
491
		(This is primarily useful for regressions that occurred in a previous
492
		revision, which was committed without running all the tests)
493
		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml
494 8470 aaronmk
	.htaccess:
495
		mod_rewrite:
496 8471 aaronmk
			IMPORTANT: whenever you change the DirectorySlash setting for a
497
				directory, you *must* clear your browser's cache to ensure that
498
				a cached redirect is not used. this is because RewriteRule
499
				redirects are (by default) temporary, but DirectorySlash
500
				redirects are permanent.
501 8470 aaronmk
				for Firefox:
502
					press Cmd+Shift+Delete
503
					check only Cache
504
					press Enter or click Clear Now
505 7183 aaronmk
506 3783 aaronmk
WinMerge setup:
507 8458 aaronmk
	Install WinMerge from <http://winmerge.org/>
508
	Open WinMerge
509
	Go to Edit > Options and click Compare in the left sidebar
510
	Enable "Moved block detection", as described at
511
		<http://manual.winmerge.org/Configuration.html#d0e5892>.
512
	Set Whitespace to Ignore change, as described at
513
		<http://manual.winmerge.org/Configuration.html#d0e5758>.
514 3783 aaronmk
515 3133 aaronmk
Documentation:
516 8458 aaronmk
	To generate a Redmine-formatted list of steps for column-based import:
517
		make schemas/public/reinstall
518
		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
519
	To import and scrub just the test taxonomic names:
520
		inputs/test_taxonomic_names/test_scrub
521 3133 aaronmk
522 702 aaronmk
General:
523 8458 aaronmk
	To see a program's description, read its top-of-file comment
524
	To see a program's usage, run it without arguments
525
	To remake a directory: make <dir>/remake
526
	To remake a file: make <file>-remake