Project

General

Profile

1 702 aaronmk
Installation:
2
    Install: make install
3 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
4 702 aaronmk
    Uninstall: make uninstall
5 3370 aaronmk
        WARNING: This will delete your entire VegBIEN DB!
6 3374 aaronmk
        This includes all archived imports and staging tables.
7 554 aaronmk
8 702 aaronmk
Data import:
9 1550 aaronmk
    Import data into VegBIEN: . bin/import_all
10 3205 aaronmk
        Using column-based import: . bin/with_all 'import by_col=1'
11 1556 aaronmk
    Stop all running imports: . bin/stop_imports
12 2976 aaronmk
    Archive the last import: make schemas/rotate
13 3381 aaronmk
    Remove the last import: make schemas/public/reinstall
14
        WARNING: This will delete the current public schema of your VegBIEN DB!
15 2976 aaronmk
    Re-import data: make schemas/rotate; . bin/import_all
16
        Note: This will archive the last import.
17 3381 aaronmk
18 3545 aaronmk
Backups:
19 3547 aaronmk
    After a new import:
20
        make schemas/rotate
21
        Rename the rotated schema using the date in the first datasource's log
22
            file name
23 3548 aaronmk
        Delete previous imports so they won't bloat the full DB backup:
24
            make backups/public.<date>.backup/remove
25 3547 aaronmk
        make backups/<schema>.backup/test & make backups/vegbien.backup/all &
26 3408 aaronmk
    Archived imports:
27
        Back up: make backups/public.<date>.backup &
28 3546 aaronmk
            Note: To back up the last import, you must archive it first (above)
29 3410 aaronmk
        Test: make backups/public.<date>.backup/test &
30 3408 aaronmk
        Restore: make backups/public.<date>.backup/restore &
31
        Remove: make backups/public.<date>.backup/remove
32
    Full DB:
33 3546 aaronmk
        Back up, test, and rotate: make backups/vegbien.backup/all &
34 3439 aaronmk
        Back up and rotate: make backups/vegbien.backup/rotate &
35
        Test: make backups/vegbien.<date>.backup/test &
36
        Restore: make backups/vegbien.<date>.backup/restore &
37 554 aaronmk
38 1773 aaronmk
Datasource setup:
39 3586 aaronmk
    Add a new datasource: make inputs/<name>/add
40
        <name> may not contain spaces, and should be abbreviated.
41
        If the datasource is a herbarium, <name> should be the herbarium code as
42
        defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
43 3557 aaronmk
    Populate the src/ subdir with input data:
44 3586 aaronmk
        Obtain/create CSVs for the table(s) present in the datasource:
45 3557 aaronmk
            specimens, plots, organisms, stems
46
        Rename each CSV so it ends in ".<table>.<ext>" (see tables above)
47 3593 aaronmk
    Auto-create the map spreadsheets:
48
        make inputs/<name>/; make inputs/<name>/
49
            Note: Must be run twice to properly bootstrap all maps.
50
        svn add inputs/<name>/maps/{*.csv,.*.last_cleanup}
51 3611 aaronmk
    Install the staging tables:
52
        make inputs/<name>/reinstall quiet=1 &
53
        To view progress: tail -f inputs/<name>/import/install-<table>.log.sql
54
        View the logs: tail +1 inputs/<name>/import/install-*.log.sql
55
            tail provides a header line with the filename
56
            +1 starts at the first line, to show the whole file
57
        For every file with an error 'column "..." specified more than once':
58
            Add a header override file "+header.<table>.<ext>" in src/:
59
                Note: The leading "+" should sort it before the flat files.
60
                    "_" unfortunately sorts *after* capital letters in ASCII.
61
                Create a text file containing the header line of the flat files
62
                Add an ! at the beginning of the line
63
                    This signals cat_csv that this is a header override.
64
                For empty names, use their 0-based column # (by convention)
65
                For duplicate names, add a distinguishing suffix
66
                For long names that collided, rename them to <= 63 chars long
67
                Do NOT make readability changes in this step; that is what the
68
                    map spreadsheets (below) are for.
69
                Save
70
                svn add inputs/<name>/src/<header_override>
71
        If you made any changes, re-run the install command above
72 3576 aaronmk
    Map each table's columns:
73
        In the maps/ subdir, for each "via map" of the form "<via>.<table>.csv":
74
            Open the map in a spreadsheet editor
75
            In /mappings, open the corresponding "core map" of the form
76
                "<via>-VegBIEN.<table>.csv"
77
            In each row of the via map, set the right column to a value from the
78
                left column of the core map
79
            Save
80
        Regenerate the derived maps: make inputs/<name>/
81 3593 aaronmk
    Accept the test cases:
82
        make inputs/<name>/test/
83
            When prompted to "Accept new test output", enter y and press ENTER
84
        svn add inputs/<name>/test/*.ref
85 3583 aaronmk
    Commit: svn ci -m "Added inputs/<name>/" inputs/<name>/
86 3585 aaronmk
    Update vegbiendev:
87
        On vegbiendev: svn up
88
        On local machine: make inputs/upload
89 3588 aaronmk
        On vegbiendev: Follow the steps under Install the staging tables above
90 1773 aaronmk
91 702 aaronmk
Schema changes:
92
    Regenerate schema from installed DB: make schemas/remake
93 1967 aaronmk
    Reinstall DB from schema: make schemas/reinstall
94 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
95 3441 aaronmk
    Reinstall errors tables: make inputs/install errors_table_only=1
96 3589 aaronmk
    Reinstall staging tables: . bin/reinstall_all
97 702 aaronmk
    Sync ERD with vegbien.sql schema:
98
        Run make schemas/vegbien.my.sql
99
        Open schemas/vegbien.ERD.mwb in MySQLWorkbench
100
        Go to File > Export > Synchronize With SQL CREATE Script...
101
        For Input File, select schemas/vegbien.my.sql
102
        Click Continue
103
        Click in the changes list and press Ctrl+A or Apple+A to select all
104
        Click Update Model
105
        Click Continue
106
        Note: The generated SQL script will be empty because we are syncing in
107
            the opposite direction
108
        Click Execute
109
        Reposition any lines that have been reset
110
        Add any new tables by dragging them from the Catalog in the left sidebar
111
            to the diagram
112
        Remove any deleted tables by right-clicking the table's diagram element,
113
            selecting Delete '<table name>', and clicking Delete
114
        Save
115 1774 aaronmk
        If desired, update the graphical ERD exports (see below)
116
    Update graphical ERD exports:
117 702 aaronmk
        Go to File > Export > Export as PNG...
118 1774 aaronmk
        Select schemas/vegbien.ERD.png and click Save
119 702 aaronmk
        Go to File > Export > Export as SVG...
120 1774 aaronmk
        Select schemas/vegbien.ERD.svg and click Save
121 702 aaronmk
        Go to File > Export > Export as Single Page PDF...
122 1774 aaronmk
        Select schemas/vegbien.ERD.pdf and click Save
123
        Go to File > Print...
124
        For Pages, choose From 1 To 1
125
        In the lower left corner, click PDF > Save as PDF...
126
        Set the Title and Author to ""
127
        Select schemas/vegbien.ERD.core.pdf and click Save
128 203 aaronmk
129 1459 aaronmk
Testing:
130
    Mapping process: make test
131
    Map spreadsheet generation: make remake
132 1744 aaronmk
    Missing mappings: make missing_mappings
133 1459 aaronmk
    Everything (for most complete coverage): make test-all
134 702 aaronmk
135 3133 aaronmk
Documentation:
136
    To generate a Redmine-formatted list of steps for column-based import:
137
        make inputs/QMOR/import/steps.by_col.sql
138
139 702 aaronmk
General:
140
    To see a program's description, read its top-of-file comment
141
    To see a program's usage, run it without arguments
142 3389 aaronmk
    To remake a directory: make <dir>/remake
143
    To remake a file: make <file>-remake