Project

General

Profile

« Previous | Next » 

Revision 4182

Added by Aaron Marcuse-Kubitza almost 12 years ago

inputs: Move src subdir into main dir, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV_subfolders#Move-src-subdir-into-main-dir>

View differences:

README.TXT
27 27 
        make schemas/rotate
28 28 
        Rename the rotated schema using the date in the first datasource's log
29 29 
            file name
30 
        tail inputs/*/src/*/logs/<date>-*.log.sql
30 
        tail inputs/*/*/logs/<date>-*.log.sql
31 31 
        Check that every input's log ends in "Encountered 0 error(s)"
32 32 
        If many do not, fix the bug and discard the current (partial) import
33 33 
        Otherwise, continue
......
59 59 
        <name> may not contain spaces, and should be abbreviated.
60 60 
        If the datasource is a herbarium, <name> should be the herbarium code as
61 61 
        defined by the Index Herbariorum <http://sweetgum.nybg.org/ih/>
62 
    Populate the src/ subdir with input data:
62 
    Add input data:
63 63 
        Obtain/create CSVs for the table(s) present in the datasource:
64 64 
            specimens, 0.plots, 1.organisms, 2.stems
65 65 
        If there are multiple part files for a table, and the header is repeated
......
71 71 
    Auto-create the map spreadsheets:
72 72 
        make inputs/<name>/; make inputs/<name>/
73 73 
            Note: Must be run twice to properly bootstrap all maps.
74 
        svn add inputs/<name>/src/*/{{src,map*,VegBIEN}.csv,.*.last_cleanup}
74 
        svn add inputs/<name>/*/{{src,map*,VegBIEN}.csv,.*.last_cleanup}
75 75 
    Install the staging tables:
76 76 
        make inputs/<name>/reinstall quiet=1 &
77 
        To view progress: tail -f inputs/<name>/src/<table>/logs/install.log.sql
78 
        View the logs: tail -n +1 inputs/<name>/src/*/logs/install.log.sql
77 
        To view progress: tail -f inputs/<name>/<table>/logs/install.log.sql
78 
        View the logs: tail -n +1 inputs/<name>/*/logs/install.log.sql
79 79 
            tail provides a header line with the filename
80 80 
            +1 starts at the first line, to show the whole file
81 81 
        For every file with an error 'column "..." specified more than once':
82 
            Add a header override file "+header.<ext>" in src/<table>/:
82 
            Add a header override file "+header.<ext>" in <table>/:
83 83 
                Note: The leading "+" should sort it before the flat files.
84 84 
                    "_" unfortunately sorts *after* capital letters in ASCII.
85 85 
                Create a text file containing the header line of the flat files
......
91 91 
                Do NOT make readability changes in this step; that is what the
92 92 
                    map spreadsheets (below) are for.
93 93 
                Save
94 
                svn add inputs/<name>/src/<table>/<header_override>
94 
                svn add inputs/<name>/<table>/<header_override>
95 95 
        If you made any changes, re-run the install command above
96 96 
    Map each table's columns:
97 97 
        In each <table>/ subdir, for each "via map" map.csv:
......
112 112 
                prevent the automatic mapper from auto-removing the mapping.
113 113 
            When accepting tests, it's helpful to use WinMerge
114 114 
                (see WinMerge setup below for configuration)
115 
        svn add inputs/<name>/src/*/test.xml.ref
115 
        svn add inputs/<name>/*/test.xml.ref
116 116 
    Commit: svn ci -m "Added inputs/<name>/" inputs/<name>/
117 117 
    Update vegbiendev:
118 118 
        On vegbiendev: svn up
......
173 173 

  		




  174
  174
  
    
Documentation:


  		




  175
  175
  
    
    To generate a Redmine-formatted list of steps for column-based import:


  		




  176
  
  
    
        make inputs/QMOR/src/specimens/logs/steps.by_col.log.sql


  		




  
  176
  
    
        make inputs/QMOR/specimens/logs/steps.by_col.log.sql


  		




  177
  177
  
    

  		




  178
  178
  
    
General:


  		




  179
  179
  
    
    To see a program's description, read its top-of-file comment


  		












Also available in:  Unified diff