Installation:

    Install: make install

        WARNING: This will delete the current public schema of your VegBIEN DB!

    Uninstall: make uninstall

        WARNING: This will delete your entire VegBIEN DB!

        This includes all archived imports and staging tables.

Maintenance:

    Important: Whenever you install a system update that affects PostgreSQL or

        any of its dependencies, such as libc, you should restart the PostgreSQL

        server. Otherwise, you may get strange errors like "the database system

        is in recovery mode" which go away upon reimport.

Data import:

    On local machine:

        make test by_col=1

            If you get errors (including skipped errors), this always indicates

            a bug, usually in either the unique constraints or column-based

            import itself

            Note that inserted row count diffs of a few rows are benign

    On vegbiendev:

    svn up

    Update the schemas: make schemas/reinstall

        WARNING: This will delete the current public schema of your VegBIEN DB!

        To save it: make schemas/rotate

    Start column-based import: . bin/with_all 'import by_col=1'

        To use row-based import: . bin/import_all

        To stop all running imports: . bin/stop_imports

    Wait (overnight) for the import to finish

    ./bin/make_analytical_db &

    Wait for make_analytical_db to finish

    Archive the last import: make schemas/rotate

    Rename the rotated schema using the date in the first datasource's log

        file name

    tail inputs/*/*/logs/<date>-*.log.sql

    Check that every input's log ends in "Encountered 0 error(s)"

    If many do not, fix the bug and discard the current (partial) import:

        make schemas/public/reinstall

    Otherwise, continue

    Delete previous imports so they won't bloat the full DB backup:

        make backups/public.<datetime>.backup/remove

    make backups/<schema>.backup/test & make backups/vegbien.backup/all &

    On local machine:

        make inputs/download-logs

        make backups/download

    If desired, record the import times in inputs/import.stats.xls:

        Open inputs/import.stats.xls

        Insert a copy of the leftmost Column-based column group before it

        Update the import date in the upper-right corner

        ./bin/import_times inputs/*/*/logs/<date>-*.log.sql

        Paste the output over the # Rows/Time columns, making sure that the

            row counts match up with the previous import's row counts

        If the row counts do not match up, insert or reorder rows as needed

            until they do

        Commit: svn ci -m "inputs/import.stats.xls: Updated with stats from \

latest import" inputs/import.stats.xls

Backups:

    Archived imports:

        Back up: make backups/public.<date>.backup &

            Note: To back up the last import, you must archive it first (above)

        Test: make backups/public.<date>.backup/test &

        Restore: make backups/public.<date>.backup/restore &

        Remove: make backups/public.<date>.backup/remove

        Download: make backups/download

    Full DB:

        Back up, test, and rotate: make backups/vegbien.backup/all &

        Back up and rotate: make backups/vegbien.backup/rotate &

        Test: make backups/vegbien.<date>.backup/test &

        Restore: make backups/vegbien.<date>.backup/restore &

        Download: make backups/download

    Import logs:

        Download: make inputs/download-logs

Datasource setup:

    Add a new datasource: make inputs/<datasrc>/add

        <datasrc> may not contain spaces, and should be abbreviated.

        If the datasource is a herbarium, <datasrc> should be the herbarium code

            as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>

    Install any MySQL export:

        Create database in phpMyAdmin

        mysql -p database <export.sql

    Add input data for each table present in the datasource:

        Choose a table name from <https://projects.nceas.ucsb.edu/nceas/projects

            /bien/wiki/VegCSV#Suggested-table-names>, or use a custom name

        Note that if this table will be joined together with another table, its

            name must end in ".src"

        make inputs/<datasrc>/<table>/add

            Important: DO NOT just create an empty directory named <table>!

                This command also creates necessary subdirs, such as logs/.

        Place the CSV for the table in inputs/<datasrc>/<table>/

            OR place a query joining other tables together in

            inputs/<datasrc>/<table>/create.sql and svn add this file

        Important: When exporting relational databases to CSVs, you MUST ensure

            that embedded quotes are escaped by doubling them, *not* by

            preceding them with a "\" as is the default in phpMyAdmin

        If there are multiple part files for a table, and the header is repeated

            in each part, make sure each header is EXACTLY the same.

             (If the headers are not the same, the CSV concatenation script

             assumes the part files don't have individual headers and treats the

             subsequent headers as data rows.)

        Add <table> to inputs/<datasrc>/import_order.txt before other tables

            that depend on it

    Auto-create the map spreadsheets:

        make inputs/<datasrc>/

        svn add inputs/<datasrc>/*/{,.}{header,map,*terms,VegBIEN}.csv{,.*}

    Install the staging tables:

        make inputs/<datasrc>/reinstall quiet=1 &

        To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql

        View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql

            tail provides a header line with the filename

            +1 starts at the first line, to show the whole file

        For every file with an error 'column "..." specified more than once':

            Add a header override file "+header.<ext>" in <table>/:

                Note: The leading "+" should sort it before the flat files.

                    "_" unfortunately sorts *after* capital letters in ASCII.

                Create a text file containing the header line of the flat files

                Add an ! at the beginning of the line

                    This signals cat_csv that this is a header override.

                For empty names, use their 0-based column # (by convention)

                For duplicate names, add a distinguishing suffix

                For long names that collided, rename them to <= 63 chars long

                Do NOT make readability changes in this step; that is what the

                    map spreadsheets (below) are for.

                Save

                svn add inputs/<datasrc>/<table>/<header_override>

        If you made any changes, re-run the install command above

    Map each table's columns:

        In each <table>/ subdir, for each "via map" map.csv:

            Open the map in a spreadsheet editor

            Open the "core map" /mappings/Veg+-VegBIEN.csv

            In each row of the via map, set the right column to a value from the

                left column of the core map

            Save

        Regenerate the derived maps: make inputs/<datasrc>/

    Accept the test cases:

        make inputs/<datasrc>/test

            When prompted to "Accept new test output", enter y and press ENTER

            If you instead get errors, do one of the following for each one:

            -   If the error was due to a bug, fix it

            -   Add a SQL function that filters or transforms the invalid data

            -   Make an empty mapping for the columns that produced the error.

                Put something in the Comments column of the map spreadsheet to

                prevent the automatic mapper from auto-removing the mapping.

            When accepting tests, it's helpful to use WinMerge

                (see WinMerge setup below for configuration)

        make inputs/<datasrc>/test by_col=1

            If you get errors this time, this always indicates a bug, usually in

                either the unique constraints or column-based import itself

        svn add inputs/<datasrc>/*/test.xml.ref

    Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/

    Update vegbiendev:

        On vegbiendev: svn up

        On local machine: make inputs/upload

        On vegbiendev:

            Follow the steps under Install the staging tables above

            make inputs/<datasrc>/test

Schema changes:

    Regenerate schema from installed DB: make schemas/remake

    Reinstall DB from schema: make schemas/reinstall

        WARNING: This will delete the current public schema of your VegBIEN DB!

    Reinstall staging tables: . bin/reinstall_all

    Sync ERD with vegbien.sql schema:

        Run make schemas/vegbien.my.sql

        Open schemas/vegbien.ERD.mwb in MySQLWorkbench

        Go to File > Export > Synchronize With SQL CREATE Script...

        For Input File, select schemas/vegbien.my.sql

        Click Continue

        Click in the changes list and press Ctrl+A or Apple+A to select all

        Click Update Model

        Click Continue

        Note: The generated SQL script will be empty because we are syncing in

            the opposite direction

        Click Execute

        Reposition any lines that have been reset

        Add any new tables by dragging them from the Catalog in the left sidebar

            to the diagram

        Remove any deleted tables by right-clicking the table's diagram element,

            selecting Delete '<table name>', and clicking Delete

        Save

        If desired, update the graphical ERD exports (see below)

    Update graphical ERD exports:

        Go to File > Export > Export as PNG...

        Select schemas/vegbien.ERD.png and click Save

        Go to File > Export > Export as SVG...

        Select schemas/vegbien.ERD.svg and click Save

        Go to File > Export > Export as Single Page PDF...

        Select schemas/vegbien.ERD.1_pg.pdf and click Save

        Go to File > Print...

        In the lower left corner, click PDF > Save as PDF...

        Set the Title and Author to ""

        Select schemas/vegbien.ERD.pdf and click Save

Testing:

    Mapping process: make test

        Including column-based import: make test by_col=1

    Map spreadsheet generation: make remake

    Missing mappings: make missing_mappings

    Everything (for most complete coverage): make test-all

WinMerge setup:

    Install WinMerge from <http://winmerge.org/>

    Open WinMerge

    Go to Edit > Options and click Compare in the left sidebar

    Enable "Moved block detection", as described at

        <http://manual.winmerge.org/Configuration.html#d0e5892>.

    Set Whitespace to Ignore change, as described at

        <http://manual.winmerge.org/Configuration.html#d0e5758>.

Documentation:

    To generate a Redmine-formatted list of steps for column-based import:

        make inputs/QMOR/Specimen/logs/steps.by_col.log.sql

General:

    To see a program's description, read its top-of-file comment

    To see a program's usage, run it without arguments

    To remake a directory: make <dir>/remake

    To remake a file: make <file>-remake