Installation:

        Regularly, or whenever the VegCore data dictionary page

            (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore)

            is changed, regenerate mappings/VegCore.csv:

            If there are changes, update the data dictionary's Tables section

        any of its dependencies, such as libc, you should restart the PostgreSQL

        server. Otherwise, you may get strange errors like "the database system

        Linux and Mac OS X.

    (Re)import only: make inputs/<datasrc>/reimport by_col=1

    (Re)scrub: make inputs/<datasrc>/rescrub by_col=1

Full database import:

        The public schema will be installed separately by the import process

        make backups/vegbien.<version>.backup/remove

            export dump_opts='--exclude-schema=public --exclude-schema=<version>'

            List backups/ to view older backups

            Check their MD5 sums using the steps under On jupiter below

            Remove these backups

    Press ENTER

        To use row-based import: . bin/import_all

            won't be owned by your shell.

            and other metadata before returning control to the shell.

    When there are no more running jobs, exit the screen

    Set $version in all vegbiendev terminals: export version=<version>

        Check that the provider_count and source tables contain entries for all

            inputs

    For inputs that have this, fix the associated bug(s)

    If many inputs have errors, discard the current (partial) import:

        cd /home/bien/svn/

        svn up

        Check that "OK" is printed next to the filename

            backups/analytical_aggregate.$version.csv

        Open inputs/import.stats.xls

            row counts match up with the previous import's row counts

        If the row counts do not match up, insert or reorder rows as needed

            tail inputs/{.,}*/*/logs/$version.log.sql

            tail -100 inputs/.TNRS/tnrs/logs/tnrs.make.log.sql

        export version=<version>

        If before renaming to public: export dump_opts=--exclude-schema=public

        make backups/vegbien.$version.backup/test &

                make schemas/rotate

        Remove: make backups/<version>.backup/remove

        Back up: make backups/TNRS.backup-remake &

        Restore:

            yes|make inputs/.TNRS/uninstall

            make backups/TNRS.backup/restore &

            yes|make schemas/public/reinstall

                Must come after TNRS restore to recreate tnrs_input_name view

        Download: make inputs/download-logs

        <datasrc> may not contain spaces, and should be abbreviated.

        If the datasource is a herbarium, <datasrc> should be the herbarium code

            as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>

        For .sql exports:

            Place the original .sql file in _src/ (*not* in _MySQL/)

                This is for an initial sync to get the file onto vegbiendev.

            On vegbiendev:

                Create a database for the MySQL export in phpMyAdmin

        cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/

        Edit _MySQL/*.make for the DB connection

        Place the .mdb or .accdb file in _src/

        Download and install Access To PostgreSQL from

            http://www.bullzip.com/download.php

        Use Access To PostgreSQL to export the database:

            Export just the tables/indexes to inputs/<datasrc>/<file>.schema.sql

            Export just the data to inputs/<datasrc>/<file>.data.sql

        In <file>.schema.sql, make the following changes:

            Replace text "BOOLEAN" with "/*BOOLEAN*/INTEGER"

            Replace text "DOUBLE PRECISION NULL" with "DOUBLE PRECISION"

        Skip the Add input data for each table section

        For CSV files, you can use any name. It's recommended to use a table

            name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>

            name must end in ".src"

                This command also creates necessary subdirs, such as logs/.

            Otherwise, place the CSV(s) for the table in

            together in inputs/<datasrc>/<table>/create.sql

            that embedded quotes are escaped by doubling them, *not* by

            preceding them with a "\" as is the default in phpMyAdmin

            in each part, make sure each header is EXACTLY the same.

             (If the headers are not the same, the CSV concatenation script

             assumes the part files don't have individual headers and treats the

             subsequent headers as data rows.)

            At prompt "[you]@vegbiendev's password:", enter your password

            At prompt "Enter password:", enter the value in config/bien_password

        View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql

            +1 starts at the first line, to show the whole file

        For every file with an error 'column "..." specified more than once':

                    "_" unfortunately sorts *after* capital letters in ASCII.

                Create a text file containing the header line of the flat files

                Add an ! at the beginning of the line

                    This signals cat_csv that this is a header override.

                For empty names, use their 0-based column # (by convention)

                For duplicate names, add a distinguishing suffix

                For long names that collided, rename them to <= 63 chars long

                Do NOT make readability changes in this step; that is what the

                    map spreadsheets (below) are for.

                Save

        If you made any changes, re-run the install command above

                left column of the core map

            Save

            -   If the error was due to a bug, fix it

            -   Add a SQL function that filters or transforms the invalid data

            -   Make an empty mapping for the columns that produced the error.

                Put something in the Comments column of the map spreadsheet to

                prevent the automatic mapper from auto-removing the mapping.

                (see WinMerge setup below for configuration)

            If you get errors this time, this always indicates a bug, usually in

        On local machine:

            ./fix_perms

            make inputs/upload

            make inputs/download

    VegBank:

        make inputs/VegBank/vegbank.sql-remake

        make inputs/VegBank/reinstall quiet=1 &

        to update the corresponding table

        schemas/filter_ERD.csv

        mappings/VegCore-VegBIEN.csv

        Run make schemas/vegbien.my.sql

        Open schemas/vegbien.ERD.mwb in MySQLWorkbench

        Go to File > Export > Synchronize With SQL CREATE Script...

        For Input File, select schemas/vegbien.my.sql

        Click Continue

        Click Continue

        Note: The generated SQL script will be empty because we are syncing in

            the opposite direction

        Click Execute

        Reposition any lines that have been reset

        Add any new tables by dragging them from the Catalog in the left sidebar

            to the diagram

        Remove any deleted tables by right-clicking the table's diagram element,

            selecting Delete '<table name>', and clicking Delete

        Save

    Update graphical ERD exports:

        In the lower left corner, click PDF > Save as PDF...

        Set the Title and Author to ""

        To rename a table:

            In vegbien.sql, do the following:

                Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>

                    This is necessary because the table name is *everywhere*

                Search for <new>

                Manually change back any replacements inside comments

        To rename a column:

            Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;

            Recreate any foreign key for the column, removing CONSTRAINT <name>

                This resets the foreign key name using the new column name

                of another): 16.3cm/24 lines = 0.679cm

            Measure the height of the ERD: 35.4cm*2 = 70.8cm

            Zoom in as far as possible

            Measure the height of a capital letter: 3.5mm

            Measure the line height: 8.5mm

            Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41

            Calculate the text height: 0.679cm*0.41 = 0.28cm

            Calculate the text height's fraction of the ERD height:

                0.28cm/70.8cm = 0.0040

            Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm

            Calculate the VegBIEN poster height to make the text the same size:

                0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H

            The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait

            Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W

            The minimum VegBIEN poster size is 35x54in portrait

            The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)

                charges the following for posters:

                base: $7.25/sq ft

                lamination: $3/sq ft

                mounting on a board: $8/sq ft

            row counts, this usually means that a table is underconstrained

            (the unique indexes don't cover all possible rows).

            This can occur if you didn't use COALESCE(field, null_value) around

            a nullable field in a unique index. See sql_gen.null_sentinels for

            the appropriate null value to use.

    "Binary chop" debugging:

        (This is primarily useful for regressions that occurred in a previous

        revision, which was committed without running all the tests)

        svn up -r <rev>; make inputs/.TNRS/reinstall; make schemas/public/reinstall; make <failed-test>.xml

    Install WinMerge from <http://winmerge.org/>

    Go to Edit > Options and click Compare in the left sidebar

        <http://manual.winmerge.org/Configuration.html#d0e5892>.

        <http://manual.winmerge.org/Configuration.html#d0e5758>.

    To generate a Redmine-formatted list of steps for column-based import:

    To see a program's description, read its top-of-file comment

    To see a program's usage, run it without arguments

    To remake a file: make <file>-remake