Project

General

Profile

« Previous | Next » 

Revision 3611

README.TXT: Datasource setup: Installing the staging tables: Added steps to deal with colliding column names in the flat file headers. Added command to view the logs.

View differences:

README.TXT
48 48
        make inputs/<name>/; make inputs/<name>/
49 49
            Note: Must be run twice to properly bootstrap all maps.
50 50
        svn add inputs/<name>/maps/{*.csv,.*.last_cleanup}
51
    Install the staging tables: make inputs/<name>/install quiet=1 &
52
        To view the log: tail -f inputs/<name>/import/install-<table>.log.sql
51
    Install the staging tables:
52
        make inputs/<name>/reinstall quiet=1 &
53
        To view progress: tail -f inputs/<name>/import/install-<table>.log.sql
54
        View the logs: tail +1 inputs/<name>/import/install-*.log.sql
55
            tail provides a header line with the filename
56
            +1 starts at the first line, to show the whole file
57
        For every file with an error 'column "..." specified more than once':
58
            Add a header override file "+header.<table>.<ext>" in src/:
59
                Note: The leading "+" should sort it before the flat files.
60
                    "_" unfortunately sorts *after* capital letters in ASCII.
61
                Create a text file containing the header line of the flat files
62
                Add an ! at the beginning of the line
63
                    This signals cat_csv that this is a header override.
64
                For empty names, use their 0-based column # (by convention)
65
                For duplicate names, add a distinguishing suffix
66
                For long names that collided, rename them to <= 63 chars long
67
                Do NOT make readability changes in this step; that is what the
68
                    map spreadsheets (below) are for.
69
                Save
70
                svn add inputs/<name>/src/<header_override>
71
        If you made any changes, re-run the install command above
53 72
    Map each table's columns:
54 73
        In the maps/ subdir, for each "via map" of the form "<via>.<table>.csv":
55 74
            Open the map in a spreadsheet editor

Also available in: Unified diff