Project

General

Profile

4. set up datasource folder

from README.TXT > Datasource setup

  1. add folder for datasource in inputs/$datasrc/:
    make inputs/$datasrc/add
    "cp" -f inputs/.NCBI/{Makefile,run,table.run} inputs/$datasrc/ # add new-style import files
    mkdir inputs/$datasrc/_src/
    
  2. place extract in inputs/$datasrc/_src/

5. map to VegCore

from README.TXT > Datasource setup

1. map metadata

  1. add subdir:
    echo Source >>inputs/$datasrc/import_order.txt
    "cp" -f inputs/.NCBI/Source/{run,data.csv} inputs/$datasrc/Source/ # add new-style import files
    inputs/$datasrc/Source/run # create map spreadsheet
    
  2. fill out metadata in inputs/$datasrc/Source/map.csv

2. map each table

  1. add subdir:
    make inputs/$datasrc/$table/add
    "cp" -f inputs/.NCBI/nodes/run inputs/$datasrc/$table/ # add new-style import files
    echo $table >>inputs/$datasrc/import_order.txt
    
  2. extract flat files from the compressed extract, if applicable
  3. translate flat files to a supported format (CSV/TSV):
    • .xls:
      (cd inputs/$datasrc/_src/; /Applications/LibreOffice.app/Contents/MacOS/soffice --headless --convert-to csv *.xls)
      
  4. place extracted flat file(s) for the table in the table subdir
    • note that if the dataset consists only of flat files and all the flat files are used by a table subdir, the _src/ subdir will end up empty after the flat files have been moved
  5. rename files so their names don't contain any spaces
  6. if the header is repeated in each segment, standardize it:
    1. check the headers:
      inputs/$datasrc/$table/run check_headers
      
    2. if there is a "header mismatched" error, fix the header in the corresponding segment file
    3. repeat until no errors
  7. install the staging table and create the map spreadsheet:
    inputs/$datasrc/$table/run
    
  8. for plots datasources, prevent column collisions upon left-join:
    in inputs/$datasrc/$table/map.csv:
    1. replace text ,*
      with text ,*$table--
      where $table should be replaced with the actual value of that var
  9. map the columns in inputs/$datasrc/$table/map.csv to terms in the VegCore data dictionary
  10. rename staging table columns to VegCore:
    inputs/$datasrc/$table/run
    
  11. if any datasource-specific postprocessing is needed:
    1. add postprocess.sql:
      "cp" -f inputs/VegBank/plot/postprocess.sql inputs/$datasrc/$table/
      
    2. in postprocess.sql, modify the existing code to fit the datasource. remove any template code that doesn't apply.
    3. run the postprocessing:
      inputs/$datasrc/$table/run
      

3. ensure all pre-import actions have been performed

inputs/$datasrc/run # runtime: 1 min ("1m0.511s")

→ see sample call graph for VegBank