Installation:

    Install: make install

    Important: Whenever you install a system update that affects PostgreSQL or

        any of its dependencies, such as libc, you should restart the PostgreSQL

        server. Otherwise, you may get strange errors like "the database system

        is in recovery mode" which go away upon reimport.

        make test by_col=1

    Update the schemas: make schemas/reinstall

        WARNING: This will delete the current public schema of your VegBIEN DB!

        To save it: make schemas/rotate

            previous import

        make inputs/.TNRS/cleanup; make inputs/.TNRS/import by_col=1 &

    Wait for the TNRS import to finish: wait

    Start column-based import: . bin/import_all by_col=1

        To use row-based import: . bin/import_all

    Wait (overnight) for the import to finish

    ./bin/make_analytical_db &

    Wait for make_analytical_db to finish

        file name

    tail inputs/*/*/logs/<date>-*.log.sql

    Check that every input's log ends in "Encountered 0 error(s)"

    If many do not, fix the bug and discard the current (partial) import:

        make schemas/public/reinstall

    Otherwise, continue

    Delete previous imports so they won't bloat the full DB backup:

        make backups/public.<datetime>.backup/remove

    make backups/vegbien.<datetime>.backup/test &

        Open inputs/import.stats.xls

        Insert a copy of the leftmost Column-based column group before it

        Update the import date in the upper-right corner

        ./bin/import_times inputs/*/*/logs/<date>-*.log.sql

        Paste the output over the # Rows/Time columns, making sure that the

            row counts match up with the previous import's row counts

        If the row counts do not match up, insert or reorder rows as needed

            until they do

        Back up: make backups/public.<date>.backup &

        Remove: make backups/public.<date>.backup/remove

        Test: make backups/vegbien.<date>.backup/test &

        Restore: make backups/vegbien.<date>.backup/restore &

        Download: make inputs/download-logs

        <datasrc> may not contain spaces, and should be abbreviated.

        If the datasource is a herbarium, <datasrc> should be the herbarium code

            as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>

        Create database in phpMyAdmin

        mysql -p database <export.sql

        Choose a table name from <https://projects.nceas.ucsb.edu/nceas/projects

            /bien/wiki/VegCSV#Suggested-table-names>, or use a custom name

            name must end in ".src"

                This command also creates necessary subdirs, such as logs/.

            inputs/<datasrc>/<table>/create.sql and svn add this file

            that embedded quotes are escaped by doubling them, *not* by

            preceding them with a "\" as is the default in phpMyAdmin

            in each part, make sure each header is EXACTLY the same.

             (If the headers are not the same, the CSV concatenation script

             assumes the part files don't have individual headers and treats the

             subsequent headers as data rows.)

        To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql

        View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql

            +1 starts at the first line, to show the whole file

        For every file with an error 'column "..." specified more than once':

                    "_" unfortunately sorts *after* capital letters in ASCII.

                Create a text file containing the header line of the flat files

                Add an ! at the beginning of the line

                    This signals cat_csv that this is a header override.

                For empty names, use their 0-based column # (by convention)

                For duplicate names, add a distinguishing suffix

                For long names that collided, rename them to <= 63 chars long

                Do NOT make readability changes in this step; that is what the

                    map spreadsheets (below) are for.

                Save

                left column of the core map

            Save

            -   If the error was due to a bug, fix it

            -   Add a SQL function that filters or transforms the invalid data

            -   Make an empty mapping for the columns that produced the error.

                Put something in the Comments column of the map spreadsheet to

                prevent the automatic mapper from auto-removing the mapping.

                (see WinMerge setup below for configuration)

            If you get errors this time, this always indicates a bug, usually in

                either the unique constraints or column-based import itself

    Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/

        On vegbiendev: svn up

        On local machine: make inputs/upload

            Follow the steps under Install the staging tables above

            make inputs/<datasrc>/test

        schemas/filter_ERD.csv

        mappings/VegCore-VegBIEN.csv

        Run make schemas/vegbien.my.sql

        Open schemas/vegbien.ERD.mwb in MySQLWorkbench

        Go to File > Export > Synchronize With SQL CREATE Script...

        For Input File, select schemas/vegbien.my.sql

        Click Continue

        Click in the changes list and press Ctrl+A or Apple+A to select all

        Click Update Model

        Click Continue

        Note: The generated SQL script will be empty because we are syncing in

            the opposite direction

        Click Execute

        Reposition any lines that have been reset

        Add any new tables by dragging them from the Catalog in the left sidebar

            to the diagram

        Remove any deleted tables by right-clicking the table's diagram element,

            selecting Delete '<table name>', and clicking Delete

        Save

    Update graphical ERD exports:

        In the lower left corner, click PDF > Save as PDF...

        Set the Title and Author to ""

        To rename a table:

            In vegbien.sql, do the following:

                Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>

                    This is necessary because the table name is *everywhere*

                Search for <new>

                Manually change back any replacements inside comments

        To rename a column:

            Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;

            Recreate any foreign key for the column, removing CONSTRAINT <name>

                This resets the foreign key name using the new column name

    Mapping process: make test

            row counts, this usually means that a table is underconstrained

            (the unique indexes don't cover all possible rows).

            This can occur if you didn't use COALESCE(field, null_value) around

            a nullable field in a unique index. See sql_gen.null_sentinels for

            the appropriate null value to use.

    Install WinMerge from <http://winmerge.org/>

    Go to Edit > Options and click Compare in the left sidebar

        <http://manual.winmerge.org/Configuration.html#d0e5892>.

        <http://manual.winmerge.org/Configuration.html#d0e5758>.

    To generate a Redmine-formatted list of steps for column-based import:

        make inputs/.TNRS/reinstall

            WARNING: This will delete the TNRS cache!

            To save it: make backups/TNRS.backup-remake

        make schemas/public/reinstall

            WARNING: This will delete the public schema of your VegBIEN DB!

            To save it: make schemas/rotate

        make inputs/test_taxonomic_names/import log=

        Do the following twice: (The 2nd time parses the accepted names.)

            make inputs/.TNRS/tnrs/tnrs-remake log=

            make inputs/.TNRS/import log=

            make inputs/test_taxonomic_names/import log=

    To see a program's description, read its top-of-file comment

    To see a program's usage, run it without arguments

    To remake a file: make <file>-remake