Revision 6943
Added by Aaron Marcuse-Kubitza almost 12 years ago
| README.TXT | ||
|---|---|---|
| 47 | 47 |
Wait (overnight) for the import to finish |
| 48 | 48 |
On local machine: make inputs/download-logs |
| 49 | 49 |
In PostgreSQL, check that the source table contains entries for all inputs |
| 50 |
tail inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
|
|
| 50 |
tail inputs/{.,}*/*/logs/*.<version>[.-]*log.sql
|
|
| 51 | 51 |
In the output, search for "Command exited with non-zero status" |
| 52 | 52 |
For inputs that have this, fix the associated bug(s) |
| 53 | 53 |
If many inputs have errors, discard the current (partial) import: |
| 54 | 54 |
make schemas/public/reinstall |
| 55 | 55 |
Otherwise, continue |
| 56 | 56 |
Determine the import name: |
| 57 |
bin/import_name inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
|
|
| 57 |
bin/import_name inputs/{.,}*/*/logs/*.<version>[.-]*log.sql
|
|
| 58 | 58 |
Archive the last import: make schemas/rename/<import_name> |
| 59 | 59 |
Delete previous imports so they won't bloat the full DB backup: |
| 60 | 60 |
make backups/vegbien.<version>.backup/remove |
| ... | ... | |
| 75 | 75 |
make backups/analytical_aggregate.<version>.csv.md5/test |
| 76 | 76 |
Check that "OK" is printed next to the filename |
| 77 | 77 |
In the bien_web DB: |
| 78 |
Create the analytical_aggregate_r<revision> table using its schema
|
|
| 78 |
Create the analytical_aggregate_<version> table using its schema
|
|
| 79 | 79 |
in schemas/vegbien.my.sql |
| 80 |
env table=analytical_aggregate_r<revision> bin/publish_analytical_db \
|
|
| 80 |
env table=analytical_aggregate_<version> bin/publish_analytical_db \
|
|
| 81 | 81 |
backups/analytical_aggregate.<version>.csv |
| 82 | 82 |
If desired, record the import times in inputs/import.stats.xls: |
| 83 | 83 |
Open inputs/import.stats.xls |
| 84 | 84 |
Insert a copy of the leftmost Column-based column group before it |
| 85 | 85 |
Update the import date in the upper-right corner |
| 86 |
./bin/import_times inputs/{.,}*/*/logs/*.r<revision>[.-]*log.sql
|
|
| 86 |
./bin/import_times inputs/{.,}*/*/logs/*.<version>[.-]*log.sql
|
|
| 87 | 87 |
Paste the output over the # Rows/Time columns, making sure that the |
| 88 | 88 |
row counts match up with the previous import's row counts |
| 89 | 89 |
If the row counts do not match up, insert or reorder rows as needed |
Also available in: Unified diff
README.TXT: Data import: Replaced r<revision> with <version> because the version string is now equal to r<revision>