Installation: Check out svn: svn co https://code.nceas.ucsb.edu/code/projects/bien cd bien/ Install: make install **WARNING**: This will delete the public schema of your VegBIEN DB! Uninstall: make uninstall **WARNING**: This will delete your entire VegBIEN DB! This includes all archived imports and staging tables. Connecting to vegbiendev: ssh vegbiendev.nceas.ucsb.edu sudo su - aaronmk cd /home/bien/svn # should happen automatically at login Notes on running programs: **WARNING**: always start with a clean shell, to avoid spurious bugs. the shell should not have changes to the env vars. (there have been bugs that went away after closing and reopening the terminal window.) note that running `exec bash` is not sufficient to *reset* the env vars. Notes on editing files: **WARNING**: shell scripts should always be read-only, so that editing them while an import is in progress will not crash the import (see http://vegpath.org/links/#**%20modifying%20a%20running%20shell%20script) Single datasource import: Follow the steps under Connecting to vegbiendev above (Re)import and scrub: make inputs//reimport_scrub by_col=1 & (Re)import only: make inputs//reimport by_col=1 & Note that these commands also work if the datasource is not yet imported Remake analytical DB: see Full database import > To remake analytical DB Full database import: **WARNING**: You must perform *every single* step listed below, to avoid breaking column-based import **WARNING**: always start with a clean shell, as described above under "Notes on running programs" Follow the steps under Connecting to vegbiendev above Follow the steps under Connecting to vegbiendev above, using jupiter instead On jupiter: svn up --force On local machine: do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine" make inputs/upload make inputs/upload live=1 make test by_col=1 # runtime: 20 min ("4m46.108s" + ("21:50:43" - "21:37:09")) @starscream if you encounter errors, they are most likely related to the PostgreSQL error parsing in /lib/sql.py parse_exception() See note under Testing below On vegbiendev: Ensure there are no local modifications: svn st svn up make inputs/download make inputs/download live=1 For each newly-uploaded datasource above: make inputs//reinstall Update the auxiliary schemas: make schemas/reinstall **WARNING**: requires sudo access! The public schema will be installed separately by the import process Delete imports before the last so they won't bloat the full DB backup: make backups/vegbien..backup/remove To keep a previous import other than the public schema: export dump_opts='--exclude-schema=public --exclude-schema=' Make sure there is at least 300GB of disk space on /: df -h The import schema is 230GB, and may use additional space for temp tables To free up space, remove backups that have been archived on jupiter: List backups/ to view older backups Check their MD5 sums using the steps under On jupiter below Remove these backups screen Press ENTER unset TMOUT # TMOUT causes screen to exit even with background processes set -o ignoreeof #prevent Ctrl+D from exiting `screen` to keep attached jobs unset version Start column-based import: . bin/import_all To use row-based import: . bin/import_all by_col= To stop all running imports: . bin/stop_imports **WARNING**: Do NOT run import_all in the background, or the jobs it creates won't be owned by your shell. Note that import_all will take up to an hour to import the NCBI backbone and other metadata before returning control to the shell. To view progress: tail inputs/{.,}*/*/logs/$version.log.sql Wait (4 days) for the import to finish To recover from a closed terminal window: screen -r To restart an aborted import for a specific table: export version= (set -o errexit; make inputs///import_scrub by_col=1 continue=1; make inputs//publish) & bin/after_import $! & # $! can also be obtained from `jobs -l` Get $version: echo $version Set $version in all vegbiendev terminals: export version= When there are no more running jobs, exit the screen Upload logs (run on vegbiendev): make inputs/upload live=1 On local machine: make inputs/download-logs live=1 In PostgreSQL: Go to wiki.vegpath.org/VegBIEN_contents Get the # observations Get the # datasources Get the # datasources with observations in the r# schema: Check that analytical_stem contains [# observations] rows Check that source contains [# datasources] rows up through XAL. If this is not the case, manually check the entries in source against the datasources list on the wiki page (some datasources may be near the end depending on import order). Check that provider_count contains [# datasources with observations] rows with dataset="(total)" (at the top when the table is unsorted) Check that TNRS ran successfully: tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql If the log ends in an AssertionError "assert sql.table_col_names(db, table) == header": Figure out which TNRS CSV columns have changed On local machine: Make the changes in the DB's TNRS and public schemas rm=1 inputs/.TNRS/schema.sql.run export_ make schemas/remake inputs/test_taxonomic_names/test_scrub # re-run TNRS rm=1 inputs/.TNRS/data.sql.run export_ Commit On vegbiendev: If dropping a column, save the dependent views Make the same changes in the live TNRS.tnrs table on vegbiendev If dropping a column, recreate the dependent views Restart the TNRS client: make scrub by_col=1 & tail inputs/{.,}*/*/logs/$version.log.sql In the output, search for "Command exited with non-zero status" For inputs that have this, fix the associated bug(s) If many inputs have errors, discard the current (partial) import: make schemas/$version/uninstall Otherwise, continue Publish the new import: **WARNING**: Before proceeding, be sure you have done *every single* verification step listed above. Otherwise, a previous valid import could incorrectly be overwritten with a broken one. make schemas/$version/publish # runtime: 1 min ("real 1m10.451s") unset version make backups/upload live=1 on local machine: make backups/vegbien.$version.backup/download live=1 # download backup to local machine On jupiter: cd /data/dev/aaronmk/bien/backups For each newly-archived backup: make -s .md5/test Check that "OK" is printed next to the filename If desired, record the import times in inputs/import.stats.xls: On local machine: Open inputs/import.stats.xls If the rightmost import is within 5 columns of column IV: Copy the current tab to ~ Remove the previous imports from the current tab because they are now in the copied tab instead Insert a copy of the leftmost "By column" column group before it export version= bin/import_date inputs/{.,}*/*/logs/$version.log.sql Update the import date in the upper-right corner bin/import_times inputs/{.,}*/*/logs/$version.log.sql Paste the output over the # Rows/Time columns, making sure that the row counts match up with the previous import's row counts If the row counts do not match up, insert or reorder rows as needed until they do. Get the datasource names from the log file footers: tail inputs/{.,}*/*/logs/$version.log.sql Commit: svn ci -m 'inputs/import.stats.xls: updated import times' Running individual steps separately: To run TNRS: To use an import other than public: export version= make scrub & To view progress: tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql To remake analytical DB: To use an import other than public: export version= bin/make_analytical_db & # runtime: 13 h ("12:43:57elapsed") To view progress: tail -150 inputs/analytical_db/logs/make_analytical_db.log.sql To back up DB (staging tables and last import): To use an import *other than public*: export version= make backups/TNRS.backup-remake & dump_opts=--exclude-schema=public make backups/vegbien.$version.backup/test & If after renaming to public, instead set dump_opts='' and replace $version with the appropriate revision make backups/upload live=1 Datasource setup: On local machine: Example steps for a datasource: wiki.vegpath.org/Import_process_for_Madidi umask ug=rwx,o= # prevent files from becoming web-accessible Add a new datasource: make inputs//add may not contain spaces, and should be abbreviated. If the datasource is a herbarium, should be the herbarium code as defined by the Index Herbariorum For a new-style datasource (one containing a ./run runscript): "cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs// For MySQL inputs (exports and live DB connections): For .sql exports: Place the original .sql file in _src/ (*not* in _MySQL/) Follow the steps starting with Install the staging tables below. This is for an initial sync to get the file onto vegbiendev. On vegbiendev: Create a database for the MySQL export in phpMyAdmin Give the bien user all database-specific privileges *except* UPDATE, DELETE, ALTER, DROP. This prevents bugs in the import scripts from accidentally deleting data. bin/mysql_bien database /_src/export.sql & mkdir inputs//_MySQL/ cp -p lib/MySQL.{data,schema}.sql.make inputs//_MySQL/ Edit _MySQL/*.make for the DB connection For a .sql export, use server=vegbiendev and --user=bien Skip the Add input data for each table section For MS Access databases: Place the .mdb or .accdb file in _src/ Download and install Access To PostgreSQL from http://www.bullzip.com/download.php Use Access To PostgreSQL to export the database: Export just the tables/indexes to inputs//.schema.sql Export just the data to inputs//.data.sql In .schema.sql, make the following changes: Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER" Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION" Skip the Add input data for each table section Add input data for each table present in the datasource: For .sql exports, you must use the name of the table in the DB export For CSV files, you can use any name. It's recommended to use a table name from Note that if this table will be joined together with another table, its name must end in ".src" make inputs//
/add Important: DO NOT just create an empty directory named
! This command also creates necessary subdirs, such as logs/. If the table is in a .sql export: make inputs//
/install Otherwise, place the CSV(s) for the table in inputs//
/ OR place a query joining other tables together in inputs//
/create.sql Important: When exporting relational databases to CSVs, you MUST ensure that embedded quotes are escaped by doubling them, *not* by preceding them with a "\" as is the default in phpMyAdmin If there are multiple part files for a table, and the header is repeated in each part, make sure each header is EXACTLY the same. (If the headers are not the same, the CSV concatenation script assumes the part files don't have individual headers and treats the subsequent headers as data rows.) Add
to inputs//import_order.txt before other tables that depend on it For a new-style datasource: "cp" -f inputs/.NCBI/nodes/run inputs//
/ inputs//
/run Install the staging tables: make inputs//reinstall quiet=1 & For a MySQL .sql export: At prompt "[you]@vegbiendev's password:", enter your password At prompt "Enter password:", enter the value in config/bien_password To view progress: tail -f inputs//
/logs/install.log.sql View the logs: tail -n +1 inputs//*/logs/install.log.sql tail provides a header line with the filename +1 starts at the first line, to show the whole file For every file with an error 'column "..." specified more than once': Add a header override file "+header." in
/: Note: The leading "+" should sort it before the flat files. "_" unfortunately sorts *after* capital letters in ASCII. Create a text file containing the header line of the flat files Add an ! at the beginning of the line This signals cat_csv that this is a header override. For empty names, use their 0-based column # (by convention) For duplicate names, add a distinguishing suffix For long names that collided, rename them to <= 63 chars long Do NOT make readability changes in this step; that is what the map spreadsheets (below) are for. Save If you made any changes, re-run the install command above Auto-create the map spreadsheets: make inputs// Map each table's columns: In each
/ subdir, for each "via map" map.csv: Open the map in a spreadsheet editor Open the "core map" /mappings/Veg+-VegBIEN.csv In each row of the via map, set the right column to a value from the left column of the core map Save Regenerate the derived maps: make inputs// Accept the test cases: For a new-style datasource: inputs//run svn di inputs//*/test.xml.ref If you get errors, follow the steps for old-style datasources below For an old-style datasource: make inputs//test When prompted to "Accept new test output", enter y and press ENTER If you instead get errors, do one of the following for each one: - If the error was due to a bug, fix it - Add a SQL function that filters or transforms the invalid data - Make an empty mapping for the columns that produced the error. Put something in the Comments column of the map spreadsheet to prevent the automatic mapper from auto-removing the mapping. When accepting tests, it's helpful to use WinMerge (see WinMerge setup below for configuration) make inputs//test by_col=1 If you get errors this time, this always indicates a bug, usually in the VegBIEN unique constraints or column-based import itself Add newly-created files: make inputs//add Commit: svn ci -m "Added inputs//" inputs// Update vegbiendev: On jupiter: svn up On local machine: ./fix_perms make inputs/upload make inputs/upload live=1 On vegbiendev: svn up make inputs/download make inputs/download live=1 Follow the steps under Install the staging tables above Maintenance: on a live machine, you should put the following in your .profile: -- # make svn files web-accessible. this does not affect unversioned files, because # these get the right permissions on the local machine instead. umask ug=rwx,o=rx unset TMOUT # TMOUT causes screen to exit even with background processes -- if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down: on vegbiendev: make postgres-Linux to synchronize vegbiendev, jupiter, and your local machine: **WARNING**: pay careful attention to all files that will be deleted or overwritten! install put if needed: download https://uutils.googlecode.com/svn/trunk/bin/put to ~/bin/ and `chmod +x` it when changes are made on vegbiendev: on all machines: ./fix_perms on vegbiendev, upload: overwrite=1 bin/sync_upload --size-only then rerun with l=1 ... on your machine, download: overwrite=1 swap=1 src=. dest='aaronmk@jupiter:~/bien' put --exclude=.svn inputs/VegBIEN/TWiki then rerun with l=1 ... overwrite=1 swap=1 bin/sync_upload --size-only then rerun with l=1 ... make backups/TNRS.backup/download live=1 overwrite=1 sync_remote_url=~/Dropbox/svn/ bin/sync_upload --existing --size-only # just update mtimes/perms then rerun with l=1 ... to synchronize a Mac's settings with my testing machine's: download: **WARNING**: this will overwrite all your user's settings! on your machine: overwrite=1 swap=1 sync_local_dir=~/Library/ sync_remote_subdir=Library/ bin/sync_upload --exclude="/Saved Application State" then rerun with l=1 ... upload: do step when changes are made on vegbiendev > on your machine, download on jupiter: (cd ~/Dropbox/svn/; up) on your machine: rm ~/'Library/Thunderbird/Profiles/9oo8rcyn.default/ImapMail/imap.googlemail.com/[Gmail].sbd/Spam' # remove the downloaded Spam folder, because spam e-mails often contain viruses that would trigger clamscan overwrite=1 del= sync_local_dir=~/Dropbox/svn/ sync_remote_subdir=Dropbox/svn/ bin/sync_upload --size-only # just update mtimes then rerun with l=1 ... overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/bin" --exclude="/bin/pg_ctl" --exclude="/bin/unzip" --exclude="/Dropbox/home" --exclude="/.profile" --exclude="/.shrc" --exclude="/.bashrc" then rerun with l=1 ... overwrite=1 sync_local_dir=~ sync_remote_url=~/Dropbox/home bin/sync_upload --exclude="/Library/Saved Application State" --exclude="/.Trash" --exclude="/.dropbox" --exclude="/Documents/BIEN" --exclude="/Dropbox" --exclude="/software" --exclude="/VirtualBox VMs/**.sav" --exclude="/VirtualBox VMs/**.vdi" --exclude="/VirtualBox VMs/**.vmdk" then rerun with l=1 ... upload just the VirtualBox VMs: on your machine: overwrite=1 sync_local_dir=~ sync_remote_subdir= bin/sync_upload ~/"VirtualBox VMs/**" then rerun with l=1 ... to backup files not in Time Machine: On local machine: overwrite=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.3/data/ then rerun with l=1 ... pg_ctl. stop # stop the PostgreSQL server overwrite=1 src=/ dest=/Volumes/Time\ Machine\ Backups/ sudo -E put Library/PostgreSQL/9.3/data/ then rerun with l=1 ... pg_ctl. start # start the PostgreSQL server to backup e-mails: on local machine: /Applications/gmvault-v1.8.1-beta/bin/gmvault sync --multiple-db-owner --type quick aaronmk.nceas@gmail.com /Applications/gmvault-v1.8.1-beta/bin/gmvault sync --multiple-db-owner --type quick aaronmk@nceas.ucsb.edu open Thunderbird click the All Mail folder for each account and wait for it to download the e-mails in it to back up the version history: on vegbiendev: svnsync sync file:///home/aaronmk/Dropbox/docs/BIEN/svn_repo/ # initial runtime: 1.5 h ("08:21:38" - "06:45:26") @vegbiendev (cd ~/Dropbox/docs/BIEN/git/; git svn fetch) overwrite=1 src=~ dest='aaronmk@jupiter:~' put Dropbox/docs/BIEN/svn_repo/ # runtime: 1 min ("1:05.08") then rerun with l=1 ... overwrite=1 src=~ dest='aaronmk@jupiter:~' put Dropbox/docs/BIEN/git/ then rerun with l=1 ... on local machine: overwrite=1 swap=1 src=~ dest='aaronmk@jupiter:~' put Dropbox/docs/BIEN/svn_repo/ # runtime: 30 s ("36.19") then rerun with l=1 ... overwrite=1 swap=1 src=~ dest='aaronmk@jupiter:~' put Dropbox/docs/BIEN/git/ then rerun with l=1 ... VegCore data dictionary: Regularly, or whenever the VegCore data dictionary page (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore) is changed, regenerate mappings/VegCore.csv: On local machine: make mappings/VegCore.htm-remake; make mappings/ svn di mappings/VegCore.tables.redmine If there are changes, update the data dictionary's Tables section When moving terms, check that no terms were lost: svn di svn ci -m 'mappings/VegCore.htm: regenerated from wiki' Important: Whenever you install a system update that affects PostgreSQL or any of its dependencies, such as libc, you should restart the PostgreSQL server. Otherwise, you may get strange errors like "the database system is in recovery mode" which go away upon reimport, or you may not be able to access the database as the postgres superuser. This applies to both Linux and Mac OS X. Backups: Archived imports: Follow the steps under Connecting to vegbiendev above Back up: make backups/.backup & Note: To back up the last import, you must archive it first: make schemas/rotate Test: make -s backups/.backup/test & Restore: make backups/.backup/restore & Remove: make backups/.backup/remove Download: make backups/.backup/download TNRS cache: Follow the steps under Connecting to vegbiendev above Back up: make backups/TNRS.backup-remake & runtime: 3 min ("real 2m48.859s") Restore: yes|make inputs/.TNRS/uninstall make backups/TNRS.backup/restore & runtime: 5.5 min ("real 5m35.829s") yes|make schemas/public/reinstall Must come after TNRS restore to recreate tnrs_input_name view Full DB: Follow the steps under Connecting to vegbiendev above Back up: make backups/vegbien..backup & Test: make -s backups/vegbien..backup/test & Restore: make backups/vegbien..backup/restore & Download: make backups/vegbien..backup/download Import logs: On local machine: Download: make inputs/download-logs live=1 Datasource refreshing: VegBank: Follow the steps under Connecting to vegbiendev above make inputs/VegBank/vegbank.sql-remake make inputs/VegBank/reinstall quiet=1 & Schema changes: On local machine: When changing the analytical views, run sync_analytical_..._to_view() to update the corresponding table Remember to update the following files with any renamings: schemas/filter_ERD.csv mappings/VegCore-VegBIEN.csv mappings/verify.*.sql Regenerate schema from installed DB: make schemas/remake Reinstall DB from schema: make schemas/public/reinstall schemas/reinstall **WARNING**: This will delete the public schema of your VegBIEN DB! If needed, reinstall staging tables: On local machine: sudo -E -u postgres psql <<<'ALTER DATABASE vegbien RENAME TO vegbien_prev' make db . bin/reinstall_all Fix any bugs and retry until no errors make schemas/public/install This must be run *after* the datasources are installed, because views in public depend on some of the datasources sudo -E -u postgres psql <<<'DROP DATABASE vegbien_prev' On vegbiendev: repeat the above steps **WARNING**: Do not run this until reinstall_all runs successfully on the local machine, or the live DB may be unrestorable! Sync ERD with vegbien.sql schema: Run make schemas/vegbien.my.sql Open schemas/vegbien.ERD.mwb in MySQLWorkbench Go to File > Export > Synchronize With SQL CREATE Script... For Input File, select schemas/vegbien.my.sql Click Continue In the changes list, select each table with an arrow next to it Click Update Model Click Continue Note: The generated SQL script will be empty because we are syncing in the opposite direction Click Execute Reposition any lines that have been reset Add any new tables by dragging them from the Catalog in the left sidebar to the diagram Remove any deleted tables by right-clicking the table's diagram element, selecting Delete '
', and clicking Delete Save If desired, update the graphical ERD exports (see below) Update graphical ERD exports: Go to File > Export > Export as PNG... Select schemas/vegbien.ERD.png and click Save Go to File > Export > Export as SVG... Select schemas/vegbien.ERD.svg and click Save Go to File > Export > Export as Single Page PDF... Select schemas/vegbien.ERD.1_pg.pdf and click Save Go to File > Print... In the lower left corner, click PDF > Save as PDF... Set the Title and Author to "" Select schemas/vegbien.ERD.pdf and click Save Commit: svn ci -m "schemas/vegbien.ERD.mwb: Regenerated exports" Refactoring tips: To rename a table: In vegbien.sql, do the following: Replace regexp (?<=_|\b)(?=_|\b) with This is necessary because the table name is *everywhere* Search for Manually change back any replacements inside comments To rename a column: Rename the column: ALTER TABLE
RENAME TO ; Recreate any foreign key for the column, removing CONSTRAINT This resets the foreign key name using the new column name Creating a poster of the ERD: Determine the poster size: Measure the line height (from the bottom of one line to the bottom of another): 16.3cm/24 lines = 0.679cm Measure the height of the ERD: 35.4cm*2 = 70.8cm Zoom in as far as possible Measure the height of a capital letter: 3.5mm Measure the line height: 8.5mm Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41 Calculate the text height: 0.679cm*0.41 = 0.28cm Calculate the text height's fraction of the ERD height: 0.28cm/70.8cm = 0.0040 Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm Calculate the VegBIEN poster height to make the text the same size: 0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W The minimum VegBIEN poster size is 35x54in portrait Determine the cost: The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101) charges the following for posters: base: $7.25/sq ft lamination: $3/sq ft mounting on a board: $8/sq ft Testing: On a development machine, you should put the following in your .profile: umask ug=rwx,o= # prevent files from becoming web-accessible export log= n=2 For development machine specs, see /planning/resources/dev_machine.specs/ On local machine: Mapping process: make test Including column-based import: make test by_col=1 If the row-based and column-based imports produce different inserted row counts, this usually means that a table is underconstrained (the unique indexes don't cover all possible rows). This can occur if you didn't use COALESCE(field, null_value) around a nullable field in a unique index. See sql_gen.null_sentinels for the appropriate null value to use. Map spreadsheet generation: make remake Missing mappings: make missing_mappings Everything (for most complete coverage): make test-all Debugging: "Binary chop" debugging: (This is primarily useful for regressions that occurred in a previous revision, which was committed without running all the tests) svn up -r ; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make .xml .htaccess: mod_rewrite: **IMPORTANT**: whenever you change the DirectorySlash setting for a directory, you *must* clear your browser's cache to ensure that a cached redirect is not used. this is because RewriteRule redirects are (by default) temporary, but DirectorySlash redirects are permanent. for Firefox: press Cmd+Shift+Delete check only Cache press Enter or click Clear Now WinMerge setup: In a Windows VM: Install WinMerge from Open WinMerge Go to Edit > Options and click Compare in the left sidebar Enable "Moved block detection", as described at . Set Whitespace to Ignore change, as described at . Documentation: To generate a Redmine-formatted list of steps for column-based import: On local machine: make schemas/public/reinstall make inputs/ACAD/Specimen/logs/steps.by_col.log.sql To import and scrub just the test taxonomic names: Follow the steps under Connecting to vegbiendev above inputs/test_taxonomic_names/test_scrub General: To see a program's description, read its top-of-file comment To see a program's usage, run it without arguments To remake a directory: make /remake To remake a file: make -remake