1 |
702
|
aaronmk
|
Installation:
|
2 |
|
|
Install: make install
|
3 |
3370
|
aaronmk
|
WARNING: This will delete the current public schema of your VegBIEN DB!
|
4 |
702
|
aaronmk
|
Uninstall: make uninstall
|
5 |
3370
|
aaronmk
|
WARNING: This will delete your entire VegBIEN DB!
|
6 |
3374
|
aaronmk
|
This includes all archived imports and staging tables.
|
7 |
554
|
aaronmk
|
|
8 |
702
|
aaronmk
|
Data import:
|
9 |
1550
|
aaronmk
|
Import data into VegBIEN: . bin/import_all
|
10 |
3205
|
aaronmk
|
Using column-based import: . bin/with_all 'import by_col=1'
|
11 |
1556
|
aaronmk
|
Stop all running imports: . bin/stop_imports
|
12 |
2976
|
aaronmk
|
Archive the last import: make schemas/rotate
|
13 |
3381
|
aaronmk
|
Remove the last import: make schemas/public/reinstall
|
14 |
|
|
WARNING: This will delete the current public schema of your VegBIEN DB!
|
15 |
2976
|
aaronmk
|
Re-import data: make schemas/rotate; . bin/import_all
|
16 |
|
|
Note: This will archive the last import.
|
17 |
3381
|
aaronmk
|
|
18 |
3545
|
aaronmk
|
Backups:
|
19 |
3547
|
aaronmk
|
After a new import:
|
20 |
|
|
make schemas/rotate
|
21 |
|
|
Rename the rotated schema using the date in the first datasource's log
|
22 |
|
|
file name
|
23 |
3548
|
aaronmk
|
Delete previous imports so they won't bloat the full DB backup:
|
24 |
|
|
make backups/public.<date>.backup/remove
|
25 |
3547
|
aaronmk
|
make backups/<schema>.backup/test & make backups/vegbien.backup/all &
|
26 |
3408
|
aaronmk
|
Archived imports:
|
27 |
|
|
Back up: make backups/public.<date>.backup &
|
28 |
3546
|
aaronmk
|
Note: To back up the last import, you must archive it first (above)
|
29 |
3410
|
aaronmk
|
Test: make backups/public.<date>.backup/test &
|
30 |
3408
|
aaronmk
|
Restore: make backups/public.<date>.backup/restore &
|
31 |
|
|
Remove: make backups/public.<date>.backup/remove
|
32 |
|
|
Full DB:
|
33 |
3546
|
aaronmk
|
Back up, test, and rotate: make backups/vegbien.backup/all &
|
34 |
3439
|
aaronmk
|
Back up and rotate: make backups/vegbien.backup/rotate &
|
35 |
|
|
Test: make backups/vegbien.<date>.backup/test &
|
36 |
|
|
Restore: make backups/vegbien.<date>.backup/restore &
|
37 |
554
|
aaronmk
|
|
38 |
1773
|
aaronmk
|
Datasource setup:
|
39 |
3586
|
aaronmk
|
Add a new datasource: make inputs/<name>/add
|
40 |
|
|
<name> may not contain spaces, and should be abbreviated.
|
41 |
|
|
If the datasource is a herbarium, <name> should be the herbarium code as
|
42 |
|
|
defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
|
43 |
3557
|
aaronmk
|
Populate the src/ subdir with input data:
|
44 |
3586
|
aaronmk
|
Obtain/create CSVs for the table(s) present in the datasource:
|
45 |
3557
|
aaronmk
|
specimens, plots, organisms, stems
|
46 |
3612
|
aaronmk
|
If there are multiple part files for a table, and the header is repeated
|
47 |
|
|
in each part, make sure each header is EXACTLY the same.
|
48 |
|
|
(If the headers are not the same, the CSV concatenation script
|
49 |
|
|
assumes the part files don't have individual headers and treats the
|
50 |
|
|
subsequent headers as data rows.)
|
51 |
3557
|
aaronmk
|
Rename each CSV so it ends in ".<table>.<ext>" (see tables above)
|
52 |
3593
|
aaronmk
|
Auto-create the map spreadsheets:
|
53 |
|
|
make inputs/<name>/; make inputs/<name>/
|
54 |
|
|
Note: Must be run twice to properly bootstrap all maps.
|
55 |
|
|
svn add inputs/<name>/maps/{*.csv,.*.last_cleanup}
|
56 |
3611
|
aaronmk
|
Install the staging tables:
|
57 |
|
|
make inputs/<name>/reinstall quiet=1 &
|
58 |
|
|
To view progress: tail -f inputs/<name>/import/install-<table>.log.sql
|
59 |
3614
|
aaronmk
|
View the logs: tail -n +1 inputs/<name>/import/install-*.log.sql
|
60 |
3611
|
aaronmk
|
tail provides a header line with the filename
|
61 |
|
|
+1 starts at the first line, to show the whole file
|
62 |
|
|
For every file with an error 'column "..." specified more than once':
|
63 |
|
|
Add a header override file "+header.<table>.<ext>" in src/:
|
64 |
|
|
Note: The leading "+" should sort it before the flat files.
|
65 |
|
|
"_" unfortunately sorts *after* capital letters in ASCII.
|
66 |
|
|
Create a text file containing the header line of the flat files
|
67 |
|
|
Add an ! at the beginning of the line
|
68 |
|
|
This signals cat_csv that this is a header override.
|
69 |
|
|
For empty names, use their 0-based column # (by convention)
|
70 |
|
|
For duplicate names, add a distinguishing suffix
|
71 |
|
|
For long names that collided, rename them to <= 63 chars long
|
72 |
|
|
Do NOT make readability changes in this step; that is what the
|
73 |
|
|
map spreadsheets (below) are for.
|
74 |
|
|
Save
|
75 |
|
|
svn add inputs/<name>/src/<header_override>
|
76 |
|
|
If you made any changes, re-run the install command above
|
77 |
3576
|
aaronmk
|
Map each table's columns:
|
78 |
|
|
In the maps/ subdir, for each "via map" of the form "<via>.<table>.csv":
|
79 |
|
|
Open the map in a spreadsheet editor
|
80 |
|
|
In /mappings, open the corresponding "core map" of the form
|
81 |
|
|
"<via>-VegBIEN.<table>.csv"
|
82 |
|
|
In each row of the via map, set the right column to a value from the
|
83 |
|
|
left column of the core map
|
84 |
|
|
Save
|
85 |
|
|
Regenerate the derived maps: make inputs/<name>/
|
86 |
3593
|
aaronmk
|
Accept the test cases:
|
87 |
|
|
make inputs/<name>/test/
|
88 |
|
|
When prompted to "Accept new test output", enter y and press ENTER
|
89 |
|
|
svn add inputs/<name>/test/*.ref
|
90 |
3583
|
aaronmk
|
Commit: svn ci -m "Added inputs/<name>/" inputs/<name>/
|
91 |
3585
|
aaronmk
|
Update vegbiendev:
|
92 |
|
|
On vegbiendev: svn up
|
93 |
|
|
On local machine: make inputs/upload
|
94 |
3588
|
aaronmk
|
On vegbiendev: Follow the steps under Install the staging tables above
|
95 |
1773
|
aaronmk
|
|
96 |
702
|
aaronmk
|
Schema changes:
|
97 |
|
|
Regenerate schema from installed DB: make schemas/remake
|
98 |
1967
|
aaronmk
|
Reinstall DB from schema: make schemas/reinstall
|
99 |
3370
|
aaronmk
|
WARNING: This will delete the current public schema of your VegBIEN DB!
|
100 |
3441
|
aaronmk
|
Reinstall errors tables: make inputs/install errors_table_only=1
|
101 |
3589
|
aaronmk
|
Reinstall staging tables: . bin/reinstall_all
|
102 |
702
|
aaronmk
|
Sync ERD with vegbien.sql schema:
|
103 |
|
|
Run make schemas/vegbien.my.sql
|
104 |
|
|
Open schemas/vegbien.ERD.mwb in MySQLWorkbench
|
105 |
|
|
Go to File > Export > Synchronize With SQL CREATE Script...
|
106 |
|
|
For Input File, select schemas/vegbien.my.sql
|
107 |
|
|
Click Continue
|
108 |
|
|
Click in the changes list and press Ctrl+A or Apple+A to select all
|
109 |
|
|
Click Update Model
|
110 |
|
|
Click Continue
|
111 |
|
|
Note: The generated SQL script will be empty because we are syncing in
|
112 |
|
|
the opposite direction
|
113 |
|
|
Click Execute
|
114 |
|
|
Reposition any lines that have been reset
|
115 |
|
|
Add any new tables by dragging them from the Catalog in the left sidebar
|
116 |
|
|
to the diagram
|
117 |
|
|
Remove any deleted tables by right-clicking the table's diagram element,
|
118 |
|
|
selecting Delete '<table name>', and clicking Delete
|
119 |
|
|
Save
|
120 |
1774
|
aaronmk
|
If desired, update the graphical ERD exports (see below)
|
121 |
|
|
Update graphical ERD exports:
|
122 |
702
|
aaronmk
|
Go to File > Export > Export as PNG...
|
123 |
1774
|
aaronmk
|
Select schemas/vegbien.ERD.png and click Save
|
124 |
702
|
aaronmk
|
Go to File > Export > Export as SVG...
|
125 |
1774
|
aaronmk
|
Select schemas/vegbien.ERD.svg and click Save
|
126 |
702
|
aaronmk
|
Go to File > Export > Export as Single Page PDF...
|
127 |
1774
|
aaronmk
|
Select schemas/vegbien.ERD.pdf and click Save
|
128 |
|
|
Go to File > Print...
|
129 |
|
|
For Pages, choose From 1 To 1
|
130 |
|
|
In the lower left corner, click PDF > Save as PDF...
|
131 |
|
|
Set the Title and Author to ""
|
132 |
|
|
Select schemas/vegbien.ERD.core.pdf and click Save
|
133 |
203
|
aaronmk
|
|
134 |
1459
|
aaronmk
|
Testing:
|
135 |
|
|
Mapping process: make test
|
136 |
|
|
Map spreadsheet generation: make remake
|
137 |
1744
|
aaronmk
|
Missing mappings: make missing_mappings
|
138 |
1459
|
aaronmk
|
Everything (for most complete coverage): make test-all
|
139 |
702
|
aaronmk
|
|
140 |
3133
|
aaronmk
|
Documentation:
|
141 |
|
|
To generate a Redmine-formatted list of steps for column-based import:
|
142 |
|
|
make inputs/QMOR/import/steps.by_col.sql
|
143 |
|
|
|
144 |
702
|
aaronmk
|
General:
|
145 |
|
|
To see a program's description, read its top-of-file comment
|
146 |
|
|
To see a program's usage, run it without arguments
|
147 |
3389
|
aaronmk
|
To remake a directory: make <dir>/remake
|
148 |
|
|
To remake a file: make <file>-remake
|