Revision 14882
Added by Aaron Marcuse-Kubitza about 10 years ago
| README.TXT | ||
|---|---|---|
| 159 | 159 |
Note that import_all will take up to an hour to import the NCBI backbone |
| 160 | 160 |
and other metadata before returning control to the shell. |
| 161 | 161 |
To view progress: |
| 162 |
tail inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 162 |
tail inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 163 | 163 |
note: at the beginning of the import, the system may send out CPU load |
| 164 | 164 |
warning e-mails. these can safely be ignored. (they happen because the |
| 165 | 165 |
parallel imports use all the available cores.) |
| ... | ... | |
| 181 | 181 |
upload logs: make inputs/upload live=1 |
| 182 | 182 |
On local machine: make inputs/download-logs live=1 |
| 183 | 183 |
check for disk space errors: |
| 184 |
grep --files-with-matches -F 'No space left on device' inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 184 |
grep --files-with-matches -F 'No space left on device' inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 185 | 185 |
if there are any matches: |
| 186 | 186 |
manually reimport these datasources using the steps under |
| 187 | 187 |
Single datasource import |
| 188 | 188 |
bin/after_import & |
| 189 | 189 |
wait for the import to finish |
| 190 |
tail inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 190 |
tail inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 191 | 191 |
In the output, search for "Command exited with non-zero status" |
| 192 | 192 |
For inputs that have this, fix the associated bug(s) |
| 193 | 193 |
If many inputs have errors, discard the current (partial) import: |
| ... | ... | |
| 248 | 248 |
now in the copied tab instead |
| 249 | 249 |
Insert a copy of the leftmost "By column" column group before it |
| 250 | 250 |
export version=<version> |
| 251 |
bin/import_date inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 251 |
bin/import_date inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 252 | 252 |
Update the import date in the upper-right corner |
| 253 |
bin/import_times inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 253 |
bin/import_times inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 254 | 254 |
Paste the output over the # Rows/Time columns, making sure that the |
| 255 | 255 |
row counts match up with the previous import's row counts |
| 256 | 256 |
If the row counts do not match up, insert or reorder rows as needed |
| 257 | 257 |
until they do. Get the datasource names from the log file footers: |
| 258 |
tail inputs/{.,}??*/*/logs/$version.log.sql
|
|
| 258 |
tail inputs/{.[^as.],}*/*/logs/$version.log.sql
|
|
| 259 | 259 |
Commit: svn ci -m 'inputs/import.stats.xls: updated import times' |
| 260 | 260 |
Running individual steps separately: |
| 261 | 261 |
To run TNRS: |
Also available in: Unified diff
bugfix: /README.TXT: globs that list all the datasources: this was incorrectly excluding U, so fixed to match bin/with_all $hidden_srcs