Project

General

Profile

1 702 aaronmk
Installation:
2
    Install: make install
3 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
4 702 aaronmk
    Uninstall: make uninstall
5 3370 aaronmk
        WARNING: This will delete your entire VegBIEN DB!
6 3374 aaronmk
        This includes all archived imports and staging tables.
7 554 aaronmk
8 702 aaronmk
Data import:
9 1550 aaronmk
    Import data into VegBIEN: . bin/import_all
10 3205 aaronmk
        Using column-based import: . bin/with_all 'import by_col=1'
11 1556 aaronmk
    Stop all running imports: . bin/stop_imports
12 2976 aaronmk
    Archive the last import: make schemas/rotate
13 3381 aaronmk
    Remove the last import: make schemas/public/reinstall
14
        WARNING: This will delete the current public schema of your VegBIEN DB!
15 2976 aaronmk
    Re-import data: make schemas/rotate; . bin/import_all
16
        Note: This will archive the last import.
17 3381 aaronmk
18 3545 aaronmk
Backups:
19 3547 aaronmk
    After a new import:
20
        make schemas/rotate
21
        Rename the rotated schema using the date in the first datasource's log
22
            file name
23 3548 aaronmk
        Delete previous imports so they won't bloat the full DB backup:
24
            make backups/public.<date>.backup/remove
25 3547 aaronmk
        make backups/<schema>.backup/test & make backups/vegbien.backup/all &
26 3408 aaronmk
    Archived imports:
27
        Back up: make backups/public.<date>.backup &
28 3546 aaronmk
            Note: To back up the last import, you must archive it first (above)
29 3410 aaronmk
        Test: make backups/public.<date>.backup/test &
30 3408 aaronmk
        Restore: make backups/public.<date>.backup/restore &
31
        Remove: make backups/public.<date>.backup/remove
32
    Full DB:
33 3546 aaronmk
        Back up, test, and rotate: make backups/vegbien.backup/all &
34 3439 aaronmk
        Back up and rotate: make backups/vegbien.backup/rotate &
35
        Test: make backups/vegbien.<date>.backup/test &
36
        Restore: make backups/vegbien.<date>.backup/restore &
37 554 aaronmk
38 1773 aaronmk
Datasource setup:
39 3586 aaronmk
    Add a new datasource: make inputs/<name>/add
40
        <name> may not contain spaces, and should be abbreviated.
41
        If the datasource is a herbarium, <name> should be the herbarium code as
42
        defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
43 3557 aaronmk
    Populate the src/ subdir with input data:
44 3586 aaronmk
        Obtain/create CSVs for the table(s) present in the datasource:
45 3557 aaronmk
            specimens, plots, organisms, stems
46 3612 aaronmk
        If there are multiple part files for a table, and the header is repeated
47
            in each part, make sure each header is EXACTLY the same.
48
             (If the headers are not the same, the CSV concatenation script
49
             assumes the part files don't have individual headers and treats the
50
             subsequent headers as data rows.)
51 3557 aaronmk
        Rename each CSV so it ends in ".<table>.<ext>" (see tables above)
52 3593 aaronmk
    Auto-create the map spreadsheets:
53
        make inputs/<name>/; make inputs/<name>/
54
            Note: Must be run twice to properly bootstrap all maps.
55
        svn add inputs/<name>/maps/{*.csv,.*.last_cleanup}
56 3611 aaronmk
    Install the staging tables:
57
        make inputs/<name>/reinstall quiet=1 &
58
        To view progress: tail -f inputs/<name>/import/install-<table>.log.sql
59 3614 aaronmk
        View the logs: tail -n +1 inputs/<name>/import/install-*.log.sql
60 3611 aaronmk
            tail provides a header line with the filename
61
            +1 starts at the first line, to show the whole file
62
        For every file with an error 'column "..." specified more than once':
63
            Add a header override file "+header.<table>.<ext>" in src/:
64
                Note: The leading "+" should sort it before the flat files.
65
                    "_" unfortunately sorts *after* capital letters in ASCII.
66
                Create a text file containing the header line of the flat files
67
                Add an ! at the beginning of the line
68
                    This signals cat_csv that this is a header override.
69
                For empty names, use their 0-based column # (by convention)
70
                For duplicate names, add a distinguishing suffix
71
                For long names that collided, rename them to <= 63 chars long
72
                Do NOT make readability changes in this step; that is what the
73
                    map spreadsheets (below) are for.
74
                Save
75
                svn add inputs/<name>/src/<header_override>
76
        If you made any changes, re-run the install command above
77 3576 aaronmk
    Map each table's columns:
78
        In the maps/ subdir, for each "via map" of the form "<via>.<table>.csv":
79
            Open the map in a spreadsheet editor
80
            In /mappings, open the corresponding "core map" of the form
81
                "<via>-VegBIEN.<table>.csv"
82
            In each row of the via map, set the right column to a value from the
83
                left column of the core map
84
            Save
85
        Regenerate the derived maps: make inputs/<name>/
86 3593 aaronmk
    Accept the test cases:
87
        make inputs/<name>/test/
88
            When prompted to "Accept new test output", enter y and press ENTER
89
        svn add inputs/<name>/test/*.ref
90 3583 aaronmk
    Commit: svn ci -m "Added inputs/<name>/" inputs/<name>/
91 3585 aaronmk
    Update vegbiendev:
92
        On vegbiendev: svn up
93
        On local machine: make inputs/upload
94 3588 aaronmk
        On vegbiendev: Follow the steps under Install the staging tables above
95 1773 aaronmk
96 702 aaronmk
Schema changes:
97
    Regenerate schema from installed DB: make schemas/remake
98 1967 aaronmk
    Reinstall DB from schema: make schemas/reinstall
99 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
100 3441 aaronmk
    Reinstall errors tables: make inputs/install errors_table_only=1
101 3589 aaronmk
    Reinstall staging tables: . bin/reinstall_all
102 702 aaronmk
    Sync ERD with vegbien.sql schema:
103
        Run make schemas/vegbien.my.sql
104
        Open schemas/vegbien.ERD.mwb in MySQLWorkbench
105
        Go to File > Export > Synchronize With SQL CREATE Script...
106
        For Input File, select schemas/vegbien.my.sql
107
        Click Continue
108
        Click in the changes list and press Ctrl+A or Apple+A to select all
109
        Click Update Model
110
        Click Continue
111
        Note: The generated SQL script will be empty because we are syncing in
112
            the opposite direction
113
        Click Execute
114
        Reposition any lines that have been reset
115
        Add any new tables by dragging them from the Catalog in the left sidebar
116
            to the diagram
117
        Remove any deleted tables by right-clicking the table's diagram element,
118
            selecting Delete '<table name>', and clicking Delete
119
        Save
120 1774 aaronmk
        If desired, update the graphical ERD exports (see below)
121
    Update graphical ERD exports:
122 702 aaronmk
        Go to File > Export > Export as PNG...
123 1774 aaronmk
        Select schemas/vegbien.ERD.png and click Save
124 702 aaronmk
        Go to File > Export > Export as SVG...
125 1774 aaronmk
        Select schemas/vegbien.ERD.svg and click Save
126 702 aaronmk
        Go to File > Export > Export as Single Page PDF...
127 1774 aaronmk
        Select schemas/vegbien.ERD.pdf and click Save
128
        Go to File > Print...
129
        For Pages, choose From 1 To 1
130
        In the lower left corner, click PDF > Save as PDF...
131
        Set the Title and Author to ""
132
        Select schemas/vegbien.ERD.core.pdf and click Save
133 203 aaronmk
134 1459 aaronmk
Testing:
135
    Mapping process: make test
136
    Map spreadsheet generation: make remake
137 1744 aaronmk
    Missing mappings: make missing_mappings
138 1459 aaronmk
    Everything (for most complete coverage): make test-all
139 702 aaronmk
140 3133 aaronmk
Documentation:
141
    To generate a Redmine-formatted list of steps for column-based import:
142
        make inputs/QMOR/import/steps.by_col.sql
143
144 702 aaronmk
General:
145
    To see a program's description, read its top-of-file comment
146
    To see a program's usage, run it without arguments
147 3389 aaronmk
    To remake a directory: make <dir>/remake
148
    To remake a file: make <file>-remake