set up datasource folder¶

from README.TXT > Datasource setup

1. add folder for datasource in inputs/$datasrc/:
   

   make inputs/$datasrc/add
   "cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/$datasrc/ # add new-style import files
   mkdir inputs/$datasrc/_src/
   

2. place extract in inputs/$datasrc/_src/

map to VegCore¶

from README.TXT > Datasource setup

map metadata¶

1. add subdir:
   

   echo Source >>inputs/$datasrc/import_order.txt
   "cp" -f inputs/.NCBI/Source/{run,data.csv} inputs/$datasrc/Source/ # add new-style import files
   inputs/$datasrc/Source/run # create map spreadsheet
   

2. fill out metadata in inputs/$datasrc/Source/map.csv

map each table¶

1. add subdir:
   

   make inputs/$datasrc/$table/add
   "cp" -f inputs/.NCBI/nodes/run inputs/$datasrc/$table/ # add new-style import files
   echo $table >>inputs/$datasrc/import_order.txt
   

2. extract flat files from the compressed extract, if applicable
3. translate flat files to a supported format (CSV/TSV):
   • .xls:
     

     (cd inputs/$datasrc/_src/; /Applications/LibreOffice.app/Contents/MacOS/soffice --headless --convert-to csv *.xls)
     

4. place extracted flat file(s) for the table in the table subdir
   • note that if the dataset consists only of flat files and all the flat files are used by a table subdir, the _src/ subdir will end up empty after the flat files have been moved
5. rename files so their names don't contain any spaces
6. if the header is repeated in each segment, standardize it:
   1. check the headers:
      

      inputs/$datasrc/$table/run check_headers
      

   2. if there is a "header mismatched" error, fix the header in the corresponding segment file
   3. repeat until no errors
7. install the staging table and create the map spreadsheet:
   

   inputs/$datasrc/$table/run
   

8. for plots datasources, prevent column collisions upon left-join:
   in inputs/$datasrc/$table/map.csv:
   1. replace text ,*
      with text ,*$table--
      where $table should be replaced with the actual value of that var
9. map the columns in inputs/$datasrc/$table/map.csv to terms in the VegCore data dictionary
10. rename staging table columns to VegCore:
    

    inputs/$datasrc/$table/run
    

11. if any datasource-specific postprocessing is needed:
    1. add postprocess.sql:
       

       "cp" -f inputs/VegBank/plot/postprocess.sql inputs/$datasrc/$table/
       

    2. in postprocess.sql, modify the existing code to fit the datasource. remove any template code that doesn't apply.
    3. run the postprocessing:
       

       inputs/$datasrc/$table/run
       

ensure all pre-import actions have been performed¶

inputs/$datasrc/run # runtime: 1 min ("1m0.511s")

→ see sample call graph for VegBank