Revision 4481
Added by Aaron Marcuse-Kubitza about 12 years ago
| README.TXT | ||
|---|---|---|
| 12 | 12 |
is in recovery mode" which go away upon reimport. |
| 13 | 13 |
|
| 14 | 14 |
Data import: |
| 15 |
Import data into VegBIEN: . bin/import_all |
|
| 16 |
Using column-based import: . bin/with_all 'import by_col=1' |
|
| 17 |
Stop all running imports: . bin/stop_imports |
|
| 15 |
Perform the following steps on vegbiendev: |
|
| 16 |
Start column-based import: . bin/with_all 'import by_col=1' |
|
| 17 |
To use row-based import: . bin/import_all |
|
| 18 |
To stop all running imports: . bin/stop_imports |
|
| 19 |
Wait (overnight) for the import to finish |
|
| 20 |
./bin/make_analytical_db & |
|
| 21 |
Wait for make_analytical_db to finish |
|
| 18 | 22 |
Archive the last import: make schemas/rotate |
| 19 |
Remove the last import: make schemas/public/reinstall |
|
| 20 |
WARNING: This will delete the current public schema of your VegBIEN DB! |
|
| 21 |
Re-import data: make schemas/rotate; . bin/import_all |
|
| 22 |
Note: This will archive the last import. |
|
| 23 |
Rename the rotated schema using the date in the first datasource's log |
|
| 24 |
file name |
|
| 25 |
tail inputs/*/*/logs/<date>-*.log.sql |
|
| 26 |
Check that every input's log ends in "Encountered 0 error(s)" |
|
| 27 |
If many do not, fix the bug and discard the current (partial) import: |
|
| 28 |
make schemas/public/reinstall |
|
| 29 |
Otherwise, continue |
|
| 30 |
Delete previous imports so they won't bloat the full DB backup: |
|
| 31 |
make backups/public.<datetime>.backup/remove |
|
| 32 |
make backups/<schema>.backup/test & make backups/vegbien.backup/all & |
|
| 33 |
On local machine: |
|
| 34 |
make inputs/download-logs |
|
| 35 |
make backups/download & |
|
| 36 |
If desired, record the import times in inputs/import.stats.xls |
|
| 23 | 37 |
|
| 24 | 38 |
Backups: |
| 25 |
After a new import: |
|
| 26 |
On vegbiendev: |
|
| 27 |
./bin/make_analytical_db & |
|
| 28 |
Wait for make_analytical_db to finish |
|
| 29 |
make schemas/rotate |
|
| 30 |
Rename the rotated schema using the date in the first datasource's log |
|
| 31 |
file name |
|
| 32 |
tail inputs/*/*/logs/<date>-*.log.sql |
|
| 33 |
Check that every input's log ends in "Encountered 0 error(s)" |
|
| 34 |
If many do not, fix the bug and discard the current (partial) import |
|
| 35 |
Otherwise, continue |
|
| 36 |
Delete previous imports so they won't bloat the full DB backup: |
|
| 37 |
make backups/public.<datetime>.backup/remove |
|
| 38 |
make backups/<schema>.backup/test & make backups/vegbien.backup/all & |
|
| 39 |
On local machine: |
|
| 40 |
make inputs/download-logs |
|
| 41 |
make backups/download & |
|
| 42 |
If desired, record the import times in inputs/import.stats.xls |
|
| 43 | 39 |
Archived imports: |
| 44 | 40 |
Back up: make backups/public.<date>.backup & |
| 45 | 41 |
Note: To back up the last import, you must archive it first (above) |
Also available in: Unified diff
README.TXT: Merged Backups > After a new import and Data import sections into one Data import section that contains the steps to perform and back up an import. Note that many `svn diff` lines result from a change in indentation.