Project

General

Profile

« Previous | Next » 

Revision 6948

Added by Aaron Marcuse-Kubitza about 12 years ago

README.TXT: Data import: Use env var $version, which is now set by import_all, instead of manually inserting the version for <version>

View differences:

README.TXT
47 47 
    Wait (overnight) for the import to finish
48 48 
    On local machine: make inputs/download-logs
49 49 
    In PostgreSQL, check that the source table contains entries for all inputs
50 
    tail inputs/{.,}*/*/logs/*.<version>[.-]*log.sql
50 
    tail inputs/{.,}*/*/logs/*.$version[.-]*log.sql
51 51 
    In the output, search for "Command exited with non-zero status"
52 52 
    For inputs that have this, fix the associated bug(s)
53 53 
    If many inputs have errors, discard the current (partial) import:
54 54 
        make schemas/public/reinstall
55 55 
    Otherwise, continue
56 56 
    Delete previous imports so they won't bloat the full DB backup:
57 
        make backups/vegbien.<version>.backup/remove
57 
        make backups/vegbien.$version.backup/remove
58 58 
    make backups/TNRS.backup-remake &
59 
    make backups/vegbien.<version>.backup/test &
60 
    env public=<version> bin/export_analytical_db &
59 
    make backups/vegbien.$version.backup/test &
60 
    env public=$version bin/export_analytical_db &
61 61 
    sudo backups/fix_perms
62 62 
    make backups/upload
63 63 
    On jupiter:
......
68 68 
    On nimoy:
69 69 
        cd /home/bien/svn/
70 70 
        svn up
71 
        make backups/analytical_aggregate.<version>.csv/download
72 
        make backups/analytical_aggregate.<version>.csv.md5/test
71 
        make backups/analytical_aggregate.$version.csv/download
72 
        make backups/analytical_aggregate.$version.csv.md5/test
73 73 
        Check that "OK" is printed next to the filename
74 74 
        In the bien_web DB:
75 
            Create the analytical_aggregate_<version> table using its schema
75 
            Create the analytical_aggregate_$version table using its schema
76 76 
                in schemas/vegbien.my.sql
77 
        env table=analytical_aggregate_<version> bin/publish_analytical_db \
78 
            backups/analytical_aggregate.<version>.csv
77 
        env table=analytical_aggregate_$version bin/publish_analytical_db \
78 
            backups/analytical_aggregate.$version.csv
79 79 
    If desired, record the import times in inputs/import.stats.xls:
80 80 
        Open inputs/import.stats.xls
81 81 
        Insert a copy of the leftmost Column-based column group before it
82 82 
        Update the import date in the upper-right corner
83 
        ./bin/import_times inputs/{.,}*/*/logs/*.<version>[.-]*log.sql
83 
        ./bin/import_times inputs/{.,}*/*/logs/*.$version[.-]*log.sql
84 84 
        Paste the output over the # Rows/Time columns, making sure that the
85 85 
            row counts match up with the previous import's row counts
86 86 
        If the row counts do not match up, insert or reorder rows as needed

Also available in: Unified diff