Project

General

Profile

1 702 aaronmk
Installation:
2
    Install: make install
3 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
4 702 aaronmk
    Uninstall: make uninstall
5 3370 aaronmk
        WARNING: This will delete your entire VegBIEN DB!
6 3374 aaronmk
        This includes all archived imports and staging tables.
7 554 aaronmk
8 3674 aaronmk
Maintenance:
9
    Important: Whenever you install a system update that affects PostgreSQL or
10
        any of its dependencies, such as libc, you should restart the PostgreSQL
11
        server. Otherwise, you may get strange errors like "the database system
12
        is in recovery mode" which go away upon reimport.
13
14 702 aaronmk
Data import:
15 4487 aaronmk
    On local machine:
16
        make test by_col=1
17 5016 aaronmk
            See note under Testing below
18 4487 aaronmk
    On vegbiendev:
19 4482 aaronmk
    svn up
20 6059 aaronmk
    make inputs/upload
21
    For each newly-uploaded datasource: make inputs/<datasrc>/reinstall
22 4482 aaronmk
    Update the schemas: make schemas/reinstall
23
        WARNING: This will delete the current public schema of your VegBIEN DB!
24
        To save it: make schemas/rotate
25 5206 aaronmk
    Start column-based import: . bin/import_all by_col=1
26
        To use row-based import: . bin/import_all
27 4481 aaronmk
        To stop all running imports: . bin/stop_imports
28 5318 aaronmk
        WARNING: Do NOT run import_all in the background, or the jobs it creates
29
            won't be owned by your shell.
30 5563 aaronmk
        Note that import_all will several hours to import the NCBI backbone and
31
            TNRS names before returning control to the shell.
32 4481 aaronmk
    Wait (overnight) for the import to finish
33 6103 aaronmk
    tail inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
34
    Check that every input's log ends in "Encountered 0 error(s)"
35
    If many do not, fix the bug and discard the current (partial) import:
36
        make schemas/public/reinstall
37
    Otherwise, continue
38 6105 aaronmk
    Determine the import name:
39
        bin/import_name inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
40
    Archive the last import: make schemas/rename/public.<import_name>
41 4481 aaronmk
    Delete previous imports so they won't bloat the full DB backup:
42 6646 aaronmk
        make backups/vegbien.<version>.backup/remove
43 5917 aaronmk
    make backups/TNRS.backup-remake &
44 5461 aaronmk
    make backups/vegbien.<version>.backup/test &
45 6490 aaronmk
    env public=public.<version> bin/export_analytical_db &
46 6666 aaronmk
    make backups/upload
47 6719 aaronmk
    On jupiter:
48
        cd /data/dev/aaronmk/VegBIEN.backups
49
        For each newly-archived backup:
50
            make <backup>.md5/test
51
            Check that "OK" is printed next to the filename
52 6494 aaronmk
    On nimoy:
53
        cd /home/bien/svn/
54
        svn up
55 6722 aaronmk
        make backups/analytical_aggregate.public.<version>.csv/download
56 6723 aaronmk
        make backups/analytical_aggregate.public.<version>.csv.md5/test
57
        Check that "OK" is printed next to the filename
58 6494 aaronmk
        In the bien_web DB:
59 6652 aaronmk
            Create the analytical_aggregate_r<revision> table using its schema
60
                in schemas/vegbien.my.sql
61 6650 aaronmk
        env table=analytical_aggregate_r<revision> bin/publish_analytical_db \
62 6653 aaronmk
            backups/analytical_aggregate.public.<version>.csv
63 6728 aaronmk
    On local machine:
64
        make inputs/download-logs
65 4677 aaronmk
    If desired, record the import times in inputs/import.stats.xls:
66
        Open inputs/import.stats.xls
67
        Insert a copy of the leftmost Column-based column group before it
68
        Update the import date in the upper-right corner
69 5461 aaronmk
        ./bin/import_times inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
70 4677 aaronmk
        Paste the output over the # Rows/Time columns, making sure that the
71
            row counts match up with the previous import's row counts
72
        If the row counts do not match up, insert or reorder rows as needed
73
            until they do
74 4777 aaronmk
        Commit: svn ci -m "inputs/import.stats.xls: Updated import times"
75 6367 aaronmk
    To remake analytical DB: env public=... bin/make_analytical_db &
76
        public should be set to the current import's schema name
77 6212 aaronmk
        To view progress:
78
            tail -f inputs/analytical_db/logs/make_analytical_db.log.sql
79 3381 aaronmk
80 3545 aaronmk
Backups:
81 3408 aaronmk
    Archived imports:
82
        Back up: make backups/public.<date>.backup &
83 6516 aaronmk
            Note: To back up the last import, you must archive it first:
84
                make schemas/rotate
85 3410 aaronmk
        Test: make backups/public.<date>.backup/test &
86 3408 aaronmk
        Restore: make backups/public.<date>.backup/restore &
87
        Remove: make backups/public.<date>.backup/remove
88 3701 aaronmk
        Download: make backups/download
89 5801 aaronmk
    TNRS cache:
90
        Back up: make backups/TNRS.backup-remake &
91
        Restore:
92
            yes|make inputs/.TNRS/uninstall
93
            make backups/TNRS.backup/restore &
94
            yes|make schemas/public/reinstall
95
                Must come after TNRS restore to recreate tnrs_input_name view
96 3408 aaronmk
    Full DB:
97 6517 aaronmk
        Back up: make backups/vegbien.<date>.backup &
98 3439 aaronmk
        Test: make backups/vegbien.<date>.backup/test &
99
        Restore: make backups/vegbien.<date>.backup/restore &
100 3701 aaronmk
        Download: make backups/download
101 3698 aaronmk
    Import logs:
102
        Download: make inputs/download-logs
103 554 aaronmk
104 1773 aaronmk
Datasource setup:
105 4219 aaronmk
    Add a new datasource: make inputs/<datasrc>/add
106
        <datasrc> may not contain spaces, and should be abbreviated.
107
        If the datasource is a herbarium, <datasrc> should be the herbarium code
108
            as defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
109 6366 aaronmk
    For MySQL inputs (exports and live DB connections):
110
        For .sql exports:
111
            Place the original .sql file in _src/ (*not* in _MySQL/)
112
            Create a database for the MySQL export in phpMyAdmin
113
            mysql -p database <inputs/<datasrc>/_src/export.sql
114
        mkdir inputs/<datasrc>/_MySQL/
115
        cp -p lib/MySQL.{data,schema}.sql.make inputs/<datasrc>/_MySQL/
116
        Edit _MySQL/*.make for the DB connection
117
            For a .sql export, use your local MySQL DB
118 6372 aaronmk
        Install the export according to Install the staging tables below
119 4218 aaronmk
    Add input data for each table present in the datasource:
120 6365 aaronmk
        For .sql exports, you must use the name of the table in the DB export
121
        For CSV files, you can use any name. It's recommended to use a table
122
            name from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>
123 4264 aaronmk
        Note that if this table will be joined together with another table, its
124
            name must end in ".src"
125 4219 aaronmk
        make inputs/<datasrc>/<table>/add
126 4342 aaronmk
            Important: DO NOT just create an empty directory named <table>!
127
                This command also creates necessary subdirs, such as logs/.
128 6373 aaronmk
        If the table is in a .sql export: make inputs/<datasrc>/<table>/install
129
            Otherwise, place the CSV(s) for the table in
130 6370 aaronmk
            inputs/<datasrc>/<table>/ OR place a query joining other tables
131
            together in inputs/<datasrc>/<table>/create.sql
132 4212 aaronmk
        Important: When exporting relational databases to CSVs, you MUST ensure
133
            that embedded quotes are escaped by doubling them, *not* by
134
            preceding them with a "\" as is the default in phpMyAdmin
135 3612 aaronmk
        If there are multiple part files for a table, and the header is repeated
136
            in each part, make sure each header is EXACTLY the same.
137
             (If the headers are not the same, the CSV concatenation script
138
             assumes the part files don't have individual headers and treats the
139
             subsequent headers as data rows.)
140 4255 aaronmk
        Add <table> to inputs/<datasrc>/import_order.txt before other tables
141 4220 aaronmk
            that depend on it
142 3611 aaronmk
    Install the staging tables:
143 4219 aaronmk
        make inputs/<datasrc>/reinstall quiet=1 &
144
        To view progress: tail -f inputs/<datasrc>/<table>/logs/install.log.sql
145
        View the logs: tail -n +1 inputs/<datasrc>/*/logs/install.log.sql
146 3611 aaronmk
            tail provides a header line with the filename
147
            +1 starts at the first line, to show the whole file
148
        For every file with an error 'column "..." specified more than once':
149 4182 aaronmk
            Add a header override file "+header.<ext>" in <table>/:
150 3611 aaronmk
                Note: The leading "+" should sort it before the flat files.
151
                    "_" unfortunately sorts *after* capital letters in ASCII.
152
                Create a text file containing the header line of the flat files
153
                Add an ! at the beginning of the line
154
                    This signals cat_csv that this is a header override.
155
                For empty names, use their 0-based column # (by convention)
156
                For duplicate names, add a distinguishing suffix
157
                For long names that collided, rename them to <= 63 chars long
158
                Do NOT make readability changes in this step; that is what the
159
                    map spreadsheets (below) are for.
160
                Save
161
        If you made any changes, re-run the install command above
162 6015 aaronmk
    Auto-create the map spreadsheets: make inputs/<datasrc>/
163 3576 aaronmk
    Map each table's columns:
164 4125 aaronmk
        In each <table>/ subdir, for each "via map" map.csv:
165 3576 aaronmk
            Open the map in a spreadsheet editor
166 4125 aaronmk
            Open the "core map" /mappings/Veg+-VegBIEN.csv
167 3576 aaronmk
            In each row of the via map, set the right column to a value from the
168
                left column of the core map
169
            Save
170 4219 aaronmk
        Regenerate the derived maps: make inputs/<datasrc>/
171 3593 aaronmk
    Accept the test cases:
172 4219 aaronmk
        make inputs/<datasrc>/test
173 3593 aaronmk
            When prompted to "Accept new test output", enter y and press ENTER
174 3690 aaronmk
            If you instead get errors, do one of the following for each one:
175
            -   If the error was due to a bug, fix it
176
            -   Add a SQL function that filters or transforms the invalid data
177
            -   Make an empty mapping for the columns that produced the error.
178
                Put something in the Comments column of the map spreadsheet to
179
                prevent the automatic mapper from auto-removing the mapping.
180 3783 aaronmk
            When accepting tests, it's helpful to use WinMerge
181
                (see WinMerge setup below for configuration)
182 4476 aaronmk
        make inputs/<datasrc>/test by_col=1
183
            If you get errors this time, this always indicates a bug, usually in
184 6368 aaronmk
                the VegBIEN unique constraints or column-based import itself
185 5881 aaronmk
    Add newly-created files: make inputs/<datasrc>/add
186 4219 aaronmk
    Commit: svn ci -m "Added inputs/<datasrc>/" inputs/<datasrc>/
187 3585 aaronmk
    Update vegbiendev:
188
        On vegbiendev: svn up
189
        On local machine: make inputs/upload
190 4291 aaronmk
        On vegbiendev:
191
            Follow the steps under Install the staging tables above
192
            make inputs/<datasrc>/test
193 1773 aaronmk
194 6484 aaronmk
Datasource refreshing:
195
    VegBank:
196
        make inputs/VegBank/vegbank.sql-remake
197
        make inputs/VegBank/reinstall quiet=1 &
198
199 702 aaronmk
Schema changes:
200 5227 aaronmk
    Remember to update the following files with any renamings:
201
        schemas/filter_ERD.csv
202
        mappings/VegCore-VegBIEN.csv
203 6058 aaronmk
        mappings/verify.*.sql
204 702 aaronmk
    Regenerate schema from installed DB: make schemas/remake
205 1967 aaronmk
    Reinstall DB from schema: make schemas/reinstall
206 3370 aaronmk
        WARNING: This will delete the current public schema of your VegBIEN DB!
207 3589 aaronmk
    Reinstall staging tables: . bin/reinstall_all
208 702 aaronmk
    Sync ERD with vegbien.sql schema:
209
        Run make schemas/vegbien.my.sql
210
        Open schemas/vegbien.ERD.mwb in MySQLWorkbench
211
        Go to File > Export > Synchronize With SQL CREATE Script...
212
        For Input File, select schemas/vegbien.my.sql
213
        Click Continue
214 6369 aaronmk
        In the changes list, select each table with an arrow next to it
215 702 aaronmk
        Click Update Model
216
        Click Continue
217
        Note: The generated SQL script will be empty because we are syncing in
218
            the opposite direction
219
        Click Execute
220
        Reposition any lines that have been reset
221
        Add any new tables by dragging them from the Catalog in the left sidebar
222
            to the diagram
223
        Remove any deleted tables by right-clicking the table's diagram element,
224
            selecting Delete '<table name>', and clicking Delete
225
        Save
226 1774 aaronmk
        If desired, update the graphical ERD exports (see below)
227
    Update graphical ERD exports:
228 702 aaronmk
        Go to File > Export > Export as PNG...
229 1774 aaronmk
        Select schemas/vegbien.ERD.png and click Save
230 702 aaronmk
        Go to File > Export > Export as SVG...
231 1774 aaronmk
        Select schemas/vegbien.ERD.svg and click Save
232 702 aaronmk
        Go to File > Export > Export as Single Page PDF...
233 4087 aaronmk
        Select schemas/vegbien.ERD.1_pg.pdf and click Save
234 1774 aaronmk
        Go to File > Print...
235
        In the lower left corner, click PDF > Save as PDF...
236
        Set the Title and Author to ""
237 4087 aaronmk
        Select schemas/vegbien.ERD.pdf and click Save
238 5226 aaronmk
    Refactoring tips:
239
        To rename a table:
240
            In vegbien.sql, do the following:
241
                Replace regexp (?<=_|\b)<old>(?=_|\b) with <new>
242
                    This is necessary because the table name is *everywhere*
243
                Search for <new>
244
                Manually change back any replacements inside comments
245
        To rename a column:
246
            Rename the column: ALTER TABLE <table> RENAME <old> TO <new>;
247
            Recreate any foreign key for the column, removing CONSTRAINT <name>
248
                This resets the foreign key name using the new column name
249 6320 aaronmk
    Creating a poster of the ERD:
250 6326 aaronmk
        Determine the poster size:
251 6320 aaronmk
            Measure the line height (from the bottom of one line to the bottom
252
                of another): 16.3cm/24 lines = 0.679cm
253
            Measure the height of the ERD: 35.4cm*2 = 70.8cm
254
            Zoom in as far as possible
255
            Measure the height of a capital letter: 3.5mm
256
            Measure the line height: 8.5mm
257
            Calculate the text's fraction of the line height: 3.5mm/8.5mm = 0.41
258
            Calculate the text height: 0.679cm*0.41 = 0.28cm
259
            Calculate the text height's fraction of the ERD height:
260
                0.28cm/70.8cm = 0.0040
261
            Measure the text height on the *VegBank* ERD poster: 5.5mm = 0.55cm
262
            Calculate the VegBIEN poster height to make the text the same size:
263
                0.55cm/0.0040 = 137.5cm H; *1in/2.54cm = 54.1in H
264
            The ERD aspect ratio is 11 in W x (2*8.5in H) = 11x17 portrait
265
            Calculate the VegBIEN poster width: 54.1in H*11W/17H = 35.0in W
266
            The minimum VegBIEN poster size is 35x54in portrait
267 6327 aaronmk
        Determine the cost:
268
            The FedEx Kinkos near NCEAS (1030 State St, Santa Barbara, CA 93101)
269
                charges the following for posters:
270
                base: $7.25/sq ft
271
                lamination: $3/sq ft
272
                mounting on a board: $8/sq ft
273 203 aaronmk
274 1459 aaronmk
Testing:
275
    Mapping process: make test
276 4292 aaronmk
        Including column-based import: make test by_col=1
277 4985 aaronmk
            If the row-based and column-based imports produce different inserted
278
            row counts, this usually means that a table is underconstrained
279
            (the unique indexes don't cover all possible rows).
280
            This can occur if you didn't use COALESCE(field, null_value) around
281
            a nullable field in a unique index. See sql_gen.null_sentinels for
282
            the appropriate null value to use.
283 1459 aaronmk
    Map spreadsheet generation: make remake
284 1744 aaronmk
    Missing mappings: make missing_mappings
285 1459 aaronmk
    Everything (for most complete coverage): make test-all
286 702 aaronmk
287 3783 aaronmk
WinMerge setup:
288
    Install WinMerge from <http://winmerge.org/>
289 3785 aaronmk
    Open WinMerge
290
    Go to Edit > Options and click Compare in the left sidebar
291 3783 aaronmk
    Enable "Moved block detection", as described at
292
        <http://manual.winmerge.org/Configuration.html#d0e5892>.
293 3784 aaronmk
    Set Whitespace to Ignore change, as described at
294
        <http://manual.winmerge.org/Configuration.html#d0e5758>.
295 3783 aaronmk
296 3133 aaronmk
Documentation:
297
    To generate a Redmine-formatted list of steps for column-based import:
298 6291 aaronmk
        make inputs/ACAD/Specimen/logs/steps.by_col.log.sql
299 5210 aaronmk
    To import and scrub just the test taxonomic names:
300 5415 aaronmk
        inputs/test_taxonomic_names/test_scrub
301 3133 aaronmk
302 702 aaronmk
General:
303
    To see a program's description, read its top-of-file comment
304
    To see a program's usage, run it without arguments
305 3389 aaronmk
    To remake a directory: make <dir>/remake
306
    To remake a file: make <file>-remake