Installation:

	Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien

	cd bien/

	Install: make install

		WARNING: This will delete the current public schema of your VegBIEN DB!

	Uninstall: make uninstall

		WARNING: This will delete your entire VegBIEN DB!

		This includes all archived imports and staging tables.

Single datasource import:

	(Re)import and scrub: make inputs/<datasrc>/reimport_scrub by_col=1 &

	(Re)import only: make inputs/<datasrc>/reimport by_col=1 &

	Note that these commands also work if the datasource is not yet imported

	Remake analytical DB: see Full database import > To remake analytical DB

Full database import:

	WARNING: You must perform *every single* step listed below, to avoid

		breaking column-based import

	On jupiter: svn up --force

	On local machine:

		do steps under Maintenance > "to synchronize vegbiendev, jupiter, and

			your local machine"

		make inputs/upload

		make inputs/upload live=1

		make test by_col=1

			if you encounter errors, they are most likely related to the

				PostgreSQL error parsing in /lib/sql.py parse_exception()

			See note under Testing below

	On vegbiendev:

	Ensure there are no local modifications: svn st

	svn up

	make inputs/download

	make inputs/download live=1

	For each newly-uploaded datasource above: make inputs/<datasrc>/reinstall

	Update the auxiliary schemas: make schemas/reinstall

		The public schema will be installed separately by the import process

	Delete imports before the last so they won't bloat the full DB backup:

		make backups/vegbien.<version>.backup/remove

		To keep a previous import other than the public schema:

			export dump_opts='--exclude-schema=public --exclude-schema=<version>'

	Make sure there is at least 300GB of disk space on /: df -h

		The import schema is 230GB, and may use additional space for temp tables

		To free up space, remove backups that have been archived on jupiter:

			List backups/ to view older backups

			Check their MD5 sums using the steps under On jupiter below

			Remove these backups

	screen

	Press ENTER

	set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs

	unset TMOUT # TMOUT causes screen to exit even with background processes

	unset version

	Start column-based import: . bin/import_all

		To use row-based import: . bin/import_all by_col=

		To stop all running imports: . bin/stop_imports

		WARNING: Do NOT run import_all in the background, or the jobs it creates

			won't be owned by your shell.

		Note that import_all will take up to an hour to import the NCBI backbone

			and other metadata before returning control to the shell.

	Wait (4 days) for the import to finish

	To recover from a closed terminal window: screen -r

	To restart an aborted import for a specific table:

		export version=<version>

		make inputs/<datasrc>/<table>/import_scrub by_col=1 continue=1 &

		bin/after_import $! & # $! can also be obtained from `jobs -l`

	Get $version: echo $version

	Set $version in all vegbiendev terminals: export version=<version>

	When there are no more running jobs, exit the screen

	Upload logs (run on vegbiendev): make inputs/upload live=1

	On local machine: make inputs/download-logs live=1

	In PostgreSQL:

		Check that the provider_count and source tables contain entries for all

			inputs

	Check that TNRS ran successfully:

		tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql

		If the log ends in an AssertionError

			"assert sql.table_col_names(db, table) == header":

			Figure out which TNRS CSV columns have changed

			On local machine:

				Make the changes in the DB's TNRS and public schemas

				rm=1 inputs/.TNRS/schema.sql.run export_

				make schemas/remake

				inputs/test_taxonomic_names/test_scrub # re-run TNRS

				rm=1 inputs/.TNRS/data.sql.run export_

				Commit

			On vegbiendev:

				If dropping a column, save the dependent views

				Make the same changes in the live TNRS.tnrs table on vegbiendev

				If dropping a column, recreate the dependent views

				Restart the TNRS client: make scrub by_col=1 &

	tail inputs/{.,}*/*/logs/$version.log.sql

	In the output, search for "Command exited with non-zero status"

	For inputs that have this, fix the associated bug(s)

	If many inputs have errors, discard the current (partial) import:

		make schemas/$version/uninstall

	Otherwise, continue

	Publish the new import:

		WARNING: Before proceeding, be sure you have done *every single*

			verification step listed above. Otherwise, a previous valid import

			could incorrectly be overwritten with a broken one.

		make schemas/$version/publish # runtime: 1 min ("real 1m10.451s")

	unset version

	make backups/upload live=1

	On jupiter:

		cd /data/dev/aaronmk/bien/backups

		For each newly-archived backup:

			make -s <backup>.md5/test

			Check that "OK" is printed next to the filename

	On nimoy:

		cd /home/bien/svn/

		svn up

		export version=<version>

		make backups/analytical_stem.$version.csv/download

		In the bien_web DB:

			Create the analytical_stem_<version> table using its schema

				in schemas/vegbien.my.sql

		make -s backups/analytical_stem.$version.csv.md5/test

		Check that "OK" is printed next to the filename

		table=analytical_stem_$version bin/publish_analytical_db \

			backups/analytical_stem.$version.csv

	If desired, record the import times in inputs/import.stats.xls:

		Open inputs/import.stats.xls

		If the rightmost import is within 5 columns of column IV:

			Copy the current tab to <leftmost-date>~<rightmost-date>

			Remove the previous imports from the current tab because they are

				now in the copied tab instead

		Insert a copy of the leftmost "By column" column group before it

		export version=<version>

		bin/import_date inputs/{.,}*/*/logs/$version.log.sql

		Update the import date in the upper-right corner

		bin/import_times inputs/{.,}*/*/logs/$version.log.sql

		Paste the output over the # Rows/Time columns, making sure that the

			row counts match up with the previous import's row counts

		If the row counts do not match up, insert or reorder rows as needed

			until they do. Get the datasource names from the log file footers:

			tail inputs/{.,}*/*/logs/$version.log.sql

		Commit: svn ci -m "inputs/import.stats.xls: Updated import times"

	Running individual steps separately:

	To run TNRS:

		To use an import other than public: export version=<version>

		make scrub &

		To view progress:

			tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql

	To remake analytical DB:

		To use an import other than public: export version=<version>

		bin/make_analytical_db & # runtime: 13 h ("12:43:57elapsed")

		To view progress:

			tail -150 inputs/analytical_db/logs/make_analytical_db.log.sql

	To back up DB (staging tables and last import):

		To use an import *other than public*: export version=<version>

		make backups/TNRS.backup-remake &

		dump_opts=--exclude-schema=public make backups/vegbien.$version.backup/test &

			If after renaming to public, instead set dump_opts='' and replace

			$version with the appropriate revision

		make backups/upload live=1

Datasource setup:

	Example steps for a datasource: wiki.vegpath.org/Import_process_for_Madidi

	umask ug=rwx,o= # prevent files from becoming web-accessible

	Add a new datasource: make inputs/<datasrc>/add

		<datasrc> may not contain spaces, and should be abbreviated.

		If the datasource is a herbarium, <datasrc> should be the herbarium code

			as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>

	For a new-style datasource (one containing a ./run runscript):

		"cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/<datasrc>/

	For MySQL inputs (exports and live DB connections):

		For .sql exports:

			Place the original .sql file in _src/ (*not* in _MySQL/)

			Follow the steps starting with Install the staging tables below.

				This is for an initial sync to get the file onto vegbiendev.

			On vegbiendev:

				Create a database for the MySQL export in phpMyAdmin

				Give the bien user all database-specific privileges *except*

					UPDATE, DELETE, ALTER, DROP. This prevents bugs in the

					import scripts from accidentally deleting data.

				bin/mysql_bien database <inputs/<datasrc>/_src/export.sql &

		mkdir inputs/<datasrc>/_MySQL/

		cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/

		Edit _MySQL/*.make for the DB connection

			For a .sql export, use server=vegbiendev and --user=bien

		Skip the Add input data for each table section

	For MS Access databases:

		Place the .mdb or .accdb file in _src/

		Download and install Access To PostgreSQL from

			http://www.bullzip.com/download.php

		Use Access To PostgreSQL to export the database:

			Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql

			Export just the data to inputs/<datasrc>/<file>.data.sql

		In <file>.schema.sql, make the following changes:

			Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"

			Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"

		Skip the Add input data for each table section

	Add input data for each table present in the datasource:

		For .sql exports, you must use the name of the table in the DB export

		For CSV files, you can use any name. It's recommended to use a table

			name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>

		Note that if this table will be joined together with another table, its

			name must end in ".src"

		make inputs/<datasrc>/<table>/add

			Important: DO NOT just create an empty directory named <table>!

				This command also creates necessary subdirs, such as logs/.

		If the table is in a .sql export: make inputs/<datasrc>/<table>/install

			Otherwise, place the CSV(s) for the table in

			inputs/<datasrc>/<table>/ OR place a query joining other tables

			together in inputs/<datasrc>/<table>/create.sql

		Important: When exporting relational databases to CSVs, you MUST ensure

			that embedded quotes are escaped by doubling them, *not* by

			preceding them with a "\" as is the default in phpMyAdmin

		If there are multiple part files for a table, and the header is repeated

			in each part, make sure each header is EXACTLY the same.

			(If the headers are not the same, the CSV concatenation script

			assumes the part files don't have individual headers and treats the

			subsequent headers as data rows.)

		Add <table> to inputs/<datasrc>/import_order.txt before other tables

			that depend on it

		For a new-style datasource:

			"cp" -f inputs/.NCBI/nodes/run inputs/<datasrc>/<table>/

			inputs/<datasrc>/<table>/run

	Install the staging tables:

		make inputs/<datasrc>/reinstall quiet=1 &

		For a MySQL .sql export:

			At prompt "[you]@vegbiendev's password:", enter your password

			At prompt "Enter password:", enter the value in config/bien_password

		To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql

		View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql

			tail provides a header line with the filename

			+1 starts at the first line, to show the whole file

		For every file with an error 'column "..." specified more than once':

			Add a header override file "+header.<ext>" in <table>/:

				Note: The leading "+" should sort it before the flat files.

					"_" unfortunately sorts *after* capital letters in ASCII.

				Create a text file containing the header line of the flat files

				Add an ! at the beginning of the line

					This signals cat_csv that this is a header override.

				For empty names, use their 0-based column # (by convention)

				For duplicate names, add a distinguishing suffix

				For long names that collided, rename them to <= 63 chars long

				Do NOT make readability changes in this step; that is what the

					map spreadsheets (below) are for.

				Save

		If you made any changes, re-run the install command above

	Auto-create the map spreadsheets: make inputs/<datasrc>/

	Map each table's columns:

		In each <table>/ subdir, for each "via map" map.csv:

			Open the map in a spreadsheet editor

			Open the "core map" /mappings/Veg+-VegBIEN.csv

			In each row of the via map, set the right column to a value from the

				left column of the core map

			Save

		Regenerate the derived maps: make inputs/<datasrc>/

	Accept the test cases:

		For a new-style datasource:

			inputs/<datasrc>/run

			svn di inputs/<datasrc>/*/test.xml.ref

			If you get errors, follow the steps for old-style datasources below

		For an old-style datasource:

			make inputs/<datasrc>/test

			When prompted to "Accept new test output", enter y and press ENTER

			If you instead get errors, do one of the following for each one:

			-	If the error was due to a bug, fix it

			-	Add a SQL function that filters or transforms the invalid data

			-	Make an empty mapping for the columns that produced the error.

				Put something in the Comments column of the map spreadsheet to

				prevent the automatic mapper from auto-removing the mapping.

			When accepting tests, it's helpful to use WinMerge

				(see WinMerge setup below for configuration)

		make inputs/<datasrc>/test by_col=1

			If you get errors this time, this always indicates a bug, usually in

				the VegBIEN unique constraints or column-based import itself

	Add newly-created files: make inputs/<datasrc>/add

	Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/

	Update vegbiendev:

		On jupiter: svn up

		On local machine:

			./fix_perms

			make inputs/upload

			make inputs/upload live=1

		On vegbiendev:

			svn up

			make inputs/download

			make inputs/download live=1

			Follow the steps under Install the staging tables above

Maintenance:

	on a live machine, you should put the following in your .profile:

--

# make svn files web-accessible. this does not affect unversioned files, because

# these get the right permissions on the local machine instead.

umask ug=rwx,o=rx

unset TMOUT # TMOUT causes screen to exit even with background processes

--

	if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down:

		on vegbiendev: make postgres-Linux

	to synchronize vegbiendev, jupiter, and your local machine:

		WARNING: pay careful attention to all files that will be deleted or

			overwritten!

		install put if needed:

			download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it

		when changes are made on vegbiendev:

			on all machines: ./fix_perms

			on vegbiendev, upload:

				overwrite=1 bin/sync_upload --size-only

					then rerun with l=1 ...

			on your machine, download:

				overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki

					then rerun with l=1 ...

				overwrite=1 swap=1 bin/sync_upload --size-only

					then rerun with l=1 ...

				make backups/TNRS.backup/download live=1

				overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms

					then rerun with l=1 ...

	to synchronize a Mac's settings with my testing machine's:

		download:

			WARNING: this will overwrite all your user's settings!

			overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State"

				then rerun with l=1 ...

		upload:

			do step when changes are made on vegbiendev > on your machine, download

			on jupiter: (cd ~/Dropbox/svn/; up)

			on your machine:

				overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes

					then rerun with l=1 ...

				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc"

					then rerun with l=1 ...

				overwrite=1      sync_local_dir=~              sync_remote_url=~/Dropbox/home  bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk"

					then rerun with l=1 ...

		upload just the VirtualBox VMs:

			on your machine:

				overwrite=1      sync_local_dir=~              sync_remote_subdir=             bin/sync_upload ~/"VirtualBox VMs/**"

					then rerun with l=1 ...

	to backup files not in Time Machine:

		pg_ctl. stop # stop the PostgreSQL server

		src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.1/data/

			then rerun with l=1 ...

		pg_ctl. start # start the PostgreSQL server

	VegCore data dictionary:

		Regularly, or whenever the VegCore data dictionary page

			(https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)

			is changed, regenerate mappings/VegCore.csv:

			make mappings/VegCore.htm-remake; make mappings/

			svn di mappings/VegCore.tables.redmine

			If there are changes, update the data dictionary's Tables section

			When moving terms, check that no terms were lost: svn di

			svn ci -m 'mappings/VegCore.htm: regenerated from wiki'

	Important: Whenever you install a system update that affects PostgreSQL or

		any of its dependencies, such as libc, you should restart the PostgreSQL

		server. Otherwise, you may get strange errors like "the database system

		is in recovery mode" which go away upon reimport, or you may not be able

		to access the database as the postgres superuser. This applies to both

		Linux and Mac OS X.

Backups:

	Archived imports:

		Back up: make backups/<version>.backup &

			Note: To back up the last import, you must archive it first:

				make schemas/rotate

		Test: make -s backups/<version>.backup/test &

		Restore: make backups/<version>.backup/restore &

		Remove: make backups/<version>.backup/remove

		Download: make backups/<version>.backup/download

	TNRS cache:

		Back up: make backups/TNRS.backup-remake &

			runtime: 3 min ("real 2m48.859s")

		Restore:

			yes|make inputs/.TNRS/uninstall

			make backups/TNRS.backup/restore &

				runtime: 5.5 min ("real 5m35.829s")

			yes|make schemas/public/reinstall

				Must come after TNRS restore to recreate tnrs_input_name view

	Full DB:

		Back up: make backups/vegbien.<version>.backup &

		Test: make -s backups/vegbien.<version>.backup/test &

		Restore: make backups/vegbien.<version>.backup/restore &

		Download: make backups/vegbien.<version>.backup/download

	Import logs:

		Download: make inputs/download-logs live=1

Datasource refreshing:

	VegBank:

		make inputs/VegBank/vegbank.sql-remake

		make inputs/VegBank/reinstall quiet=1 &

Schema changes:

	When changing the analytical views, run sync_analytical_..._to_view()

		to update the corresponding table

	Remember to update the following files with any renamings:

		schemas/filter_ERD.csv

		mappings/VegCore-VegBIEN.csv

		mappings/verify.*.sql

	Regenerate schema from installed DB: make schemas/remake

	Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall

		WARNING: This will delete the current public schema of your VegBIEN DB!

	Reinstall staging tables:

		On local machine:

			sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev'

			make db

			. bin/reinstall_all

			Fix any bugs and retry until no errors

			make schemas/public/install

				This must be run *after* the datasources are installed, because

				views in public depend on some of the datasources

			sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev'

		On vegbiendev: repeat the above steps

			WARNING: Do not run this until reinstall_all runs successfully on

			the local machine, or the live DB may be unrestorable!

	Sync ERD with vegbien.sql schema:

		Run make schemas/vegbien.my.sql

		Open schemas/vegbien.ERD.mwb in MySQLWorkbench

		Go to File > Export > Synchronize With SQL CREATE Script...

		For Input File, select schemas/vegbien.my.sql

		Click Continue

		In the changes list, select each table with an arrow next to it

		Click Update Model

		Click Continue

		Note: The generated SQL script will be empty because we are syncing in

			the opposite direction

		Click Execute

		Reposition any lines that have been reset

		Add any new tables by dragging them from the Catalog in the left sidebar

			to the diagram

		Remove any deleted tables by right-clicking the table's diagram element,

			selecting Delete '<table name>', and clicking Delete

		Save

		If desired, update the graphical ERD exports (see below)

	Update graphical ERD exports:

		Go to File > Export > Export as PNG...

		Select schemas/vegbien.ERD.png and click Save

		Go to File > Export > Export as SVG...

		Select schemas/vegbien.ERD.svg and click Save

		Go to File > Export > Export as Single Page PDF...

		Select schemas/vegbien.ERD.1_pg.pdf and click Save

		Go to File > Print...

		In the lower left corner, click PDF > Save as PDF...

		Set the Title and Author to ""

		Select schemas/vegbien.ERD.pdf and click Save

		Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports"

	Refactoring tips:

		To rename a table:

			In vegbien.sql, do the following:

				Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>

					This is necessary because the table name is *everywhere*

				Search for <new>

				Manually change back any replacements inside comments

		To rename a column:

			Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;

			Recreate any foreign key for the column, removing CONSTRAINT <name>

				This resets the foreign key name using the new column name

	Creating a poster of the ERD:

		Determine the poster size:

			Measure the line height (from the bottom of one line to the bottom

				of another): 16.3cm/24 lines = 0.679cm

			Measure the height of the ERD: 35.4cm*2 = 70.8cm

			Zoom in as far as possible

			Measure the height of a capital letter: 3.5mm

			Measure the line height: 8.5mm

			Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41

			Calculate the text height: 0.679cm*0.41 = 0.28cm

			Calculate the text height's fraction of the ERD height:

				0.28cm/70.8cm = 0.0040

			Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm

			Calculate the VegBIEN poster height to make the text the same size:

				0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H

			The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait

			Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W

			The minimum VegBIEN poster size is 35x54in portrait

		Determine the cost:

			The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)

				charges the following for posters:

				base: $7.25/sq ft

				lamination: $3/sq ft

				mounting on a board: $8/sq ft

Testing:

	On a development machine, you should put the following in your .profile:

		umask ug=rwx,o= # prevent files from becoming web-accessible

		export log= n=2

	Mapping process: make test

		Including column-based import: make test by_col=1

			If the row-based and column-based imports produce different inserted

			row counts, this usually means that a table is underconstrained

			(the unique indexes don't cover all possible rows).

			This can occur if you didn't use COALESCE(field, null_value) around

			a nullable field in a unique index. See sql_gen.null_sentinels for

			the appropriate null value to use.

	Map spreadsheet generation: make remake

	Missing mappings: make missing_mappings

	Everything (for most complete coverage): make test-all

Debugging:

	"Binary chop" debugging:

		(This is primarily useful for regressions that occurred in a previous

		revision, which was committed without running all the tests)

		svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml

	.htaccess:

		mod_rewrite:

			IMPORTANT: whenever you change the DirectorySlash setting for a

				directory, you *must* clear your browser's cache to ensure that

				a cached redirect is not used. this is because RewriteRule

				redirects are (by default) temporary, but DirectorySlash

				redirects are permanent.

				for Firefox:

					press Cmd+Shift+Delete

					check only Cache

					press Enter or click Clear Now

WinMerge setup:

	Install WinMerge from <http://winmerge.org/>

	Open WinMerge

	Go to Edit > Options and click Compare in the left sidebar

	Enable "Moved block detection", as described at

		<http://manual.winmerge.org/Configuration.html#d0e5892>.

	Set Whitespace to Ignore change, as described at

		<http://manual.winmerge.org/Configuration.html#d0e5758>.

Documentation:

	To generate a Redmine-formatted list of steps for column-based import:

		make schemas/public/reinstall

		make inputs/ACAD/Specimen/logs/steps.by_col.log.sql

	To import and scrub just the test taxonomic names:

		inputs/test_taxonomic_names/test_scrub

General:

	To see a program's description, read its top-of-file comment

	To see a program's usage, run it without arguments

	To remake a directory: make <dir>/remake

	To remake a file: make <file>-remake