VegCSV subfolders¶

• Table of contents
• VegCSV subfolders
  • Move src files into subfolders
  • Rename subfolders with import order
  • Replace Veg+ prefix with map on via maps
  • Move maps into subfolders
  • Move test outputs into subfolders
  • Move log files into subfolders
  • Determine import order dynamically
  • Move src subdir into main dir
  • Rename subfolders to VegCSV names

Move src files into subfolders¶

1. Move files on your local machine:
   

   shopt -s nullglob
   for table in plots organisms stems specimens; do
       for map in inputs/*/src/{,.}*.$table.{csv,tsv,txt,xml}{,.last_cleanup}; do
           dir="$(dirname "$map")/$table" 
           file="$(basename "$map")" 
           test -e "$dir" || { svn mkdir "$dir"; svn propset svn:ignore "*" "$dir"; }
           mv "$map" "$dir/${file/$table./}" 
       done
   done
   

2. Manually svn mv header override files (containing the word header) that were under version control:
   

   svn mv inputs/Madidi/src/+header.plots.csv inputs/Madidi/src/plots/+header.csv
   svn mv inputs/REMIB/src/node.0.header.specimens.csv inputs/REMIB/src/specimens/node.0.header.csv
   svn mv inputs/GBIF/src/GBIF.0.header.specimens.txt inputs/GBIF/src/specimens/GBIF.0.header.txt
   svn mv inputs/SpeciesLink/src/specieslink.0.header.specimens.csv inputs/SpeciesLink/src/specimens/specieslink.0.header.csv
   

3. Add svn:ignores:
   

   for dir in inputs/*/src/{plots,organisms,stems,specimens}/; do svn propset svn:ignore "*" "$dir"; done
   

4. Change input.Makefile to use the new paths
5. make inputs/remake
6. make test
7. Test reinstalling: make inputs/MT/reinstall
8. Check diffs
9. Commit
10. svn up on vegbiendev and nimoy
11. Move files on vegbiendev and nimoy as above

Rename subfolders with import order¶

Do the following for each table of plots, organisms, stems, and specimens.
When multiple command blocks are listed, use the block that corresponds to the current table.

1. shopt -s nullglob
2. Add new subfolders
   Important: Can not use svn mv because this "is equivalent to a 'copy' and 'delete'" (`svn help mv`), which will copy all the large source files
   

   for dir in inputs/*/src/plots/; do svn mkdir "${dir/plots/0.plots}"; done
   

   for dir in inputs/*/src/organisms/; do svn mkdir "${dir/organisms/1.organisms}"; done
   

   for dir in inputs/*/src/stems/; do svn mkdir "${dir/stems/2.stems}"; done
   

3. Manually svn mv header override files (containing the word header) that were under version control:
   

   svn mv inputs/Madidi/src/plots/+header.csv inputs/Madidi/src/0.plots/+header.csv
   

   # organisms: N/A
   

   # stems: N/A
   

4. Move source files
   

   (shopt -u nullglob; for dir in inputs/*/src/plots/; do (set -x; mv "$dir"* "${dir/plots/0.plots}"); done)
   

   (shopt -u nullglob; for dir in inputs/*/src/organisms/; do (set -x; mv "$dir"* "${dir/organisms/1.organisms}"); done)
   

   (shopt -u nullglob; for dir in inputs/*/src/stems/; do (set -x; mv "$dir"* "${dir/stems/2.stems}"); done)
   

5. Add svn:ignores:
   

   for dir in inputs/*/src/0.plots/; do svn propset svn:ignore "*" "$dir"; done
   

   for dir in inputs/*/src/1.organisms/; do svn propset svn:ignore "*" "$dir"; done
   

   for dir in inputs/*/src/2.stems/; do svn propset svn:ignore "*" "$dir"; done
   

6. Rename mapping files: (Warning: This command is not idempotent!)
   

   for file in inputs/*/{maps,test}/{,.}*plots*.{csv{,.last_cleanup},ref}; do (set -x; svn mv "$file" "${file/.plots/.0.plots}"); done
   

   for file in inputs/*/{maps,test}/{,.}*organisms*.{csv{,.last_cleanup},ref}; do (set -x; svn mv "$file" "${file/.organisms/.1.organisms}"); done
   

   for file in inputs/*/{maps,test}/{,.}*stems*.{csv{,.last_cleanup},ref}; do (set -x; svn mv "$file" "${file/.stems/.2.stems}"); done
   

7. Rename installed staging tables:
   

   ALTER TABLE "Madidi".plots RENAME TO "0.plots";
   ALTER TABLE "Madidi"."plots.errors" RENAME TO "0.plots.errors";
   ALTER TABLE "SALVIAS-CSV".plots RENAME TO "0.plots";
   ALTER TABLE "SALVIAS-CSV"."plots.errors" RENAME TO "0.plots.errors";
   ALTER TABLE "SALVIAS".plots RENAME TO "0.plots";
   ALTER TABLE "SALVIAS"."plots.errors" RENAME TO "0.plots.errors";
   

   ALTER TABLE "CVS".organisms RENAME TO "1.organisms";
   ALTER TABLE "CVS"."organisms.errors" RENAME TO "1.organisms.errors";
   ALTER TABLE "Madidi".organisms RENAME TO "1.organisms";
   ALTER TABLE "Madidi"."organisms.errors" RENAME TO "1.organisms.errors";
   ALTER TABLE "SALVIAS-CSV".organisms RENAME TO "1.organisms";
   ALTER TABLE "SALVIAS-CSV"."organisms.errors" RENAME TO "1.organisms.errors";
   ALTER TABLE "SALVIAS".organisms RENAME TO "1.organisms";
   ALTER TABLE "SALVIAS"."organisms.errors" RENAME TO "1.organisms.errors";
   

   ALTER TABLE "SALVIAS".stems RENAME TO "2.stems";
   ALTER TABLE "SALVIAS"."stems.errors" RENAME TO "2.stems.errors";
   

8. Change the input.Makefile $(tablesSort) entry:
   • plots -> 0.plots
   • organisms -> 1.organisms
   • stems -> 2.stems
9. Remove old subfolders
   

   for dir in inputs/*/src/plots/; do svn rm "$dir"; done
   

   for dir in inputs/*/src/organisms/; do svn rm "$dir"; done
   

   for dir in inputs/*/src/stems/; do svn rm "$dir"; done
   

10. Test the inputs that have changed: (Not all of these inputs will change for all tables.)
    

    make inputs/{CVS,Madidi,SALVIAS-CSV,SALVIAS}/test/
    

11. Check diffs
12. Commit
13. On vegbiendev, immediately do the following, using the corresponding steps above:
    1. svn up
    2. Move source files
    3. Rename the installed staging tables
    4. Test the inputs that have changed
    5. Remove old, empty dirs:
       

       rmdir inputs/*/src/organisms/
       rmdir inputs/*/src/plots/
       rmdir inputs/*/src/stems/
       

Replace Veg+ prefix with map on via maps¶

1. Rename the maps using svn mv: (order matters)
   

   for map in inputs/*/maps/{,.}Veg+.*.csv{,.last_cleanup}; do svn mv "$map" "${map/Veg+/map}"; done
   

2. In input.Makefile, change $(via) to map when it applies to something in the maps subdir
3. make inputs/remake
4. make test
5. Check diffs
6. Commit
7. svn up on vegbiendev and nimoy

Move maps into subfolders¶

1. Move files on your local machine:
   

   shopt -s nullglob
   for table in 0.plots 1.organisms 2.stems specimens; do
       for map in inputs/*/maps/{,.}*.$table{,.full}.csv{,.last_cleanup}; do
           dir="$(dirname "$map")/../src/$table" 
           file="$(basename "$map")" 
           (set -x; svn mv "$map" "$dir/${file/$table./}") 
       done
   done
   

2. Manually move disabled CTFS maps
3. Change input.Makefile to use the new paths:
   1. Replace regexp maps/(\.?\w+)\.([^.]+)((?:\.full)?\.csv(?:\.last_cleanup)?) with src/$2/$1$3
   2. Replace string maps/%.csv with src/%/map.csv
   3. Replace string maps/%.full.csv with src/%/map.full.csv
   4. Replace string maps/src.% with src/%/src.csv
   5. Replace string maps/map.% with src/%/map.csv
   6. Replace string maps/.%.last_cleanup with src/%/.map.csv.last_cleanup
   7. Manually change occurrences of maps/%
   8. Check that there are no occurrences of maps/
   9. Change $(inputFiles) and $(srcs) to filter out maps
4. make inputs/remake
5. make test
6. Test reinstalling: make inputs/MT/reinstall
7. Remove maps subdirs:
   

   svn rm inputs/*/maps/
   

8. Check diffs
9. Commit
10. svn up on vegbiendev and nimoy

Move test outputs into subfolders¶

1. Move files on your local machine:
   

   shopt -s nullglob
   for table in 0.plots 1.organisms 2.stems specimens; do
       for map in inputs/*/test/import.$table.xml.ref; do
           dir="$(dirname "$map")/../src/$table" 
           file="$(basename "$map")" 
           file="${file/import/test}" 
           (set -x; svn mv "$map" "$dir/${file/$table./}") 
       done
       for map in inputs/*/test/import.$table{,.staging}.xml; do
           dir="$(dirname "$map")/../src/$table" 
           file="$(basename "$map")" 
           file="${file/import/test}" 
           (set -x; mv "$map" "$dir/${file/$table./}") 
       done
   done
   

2. Change input.Makefile to use the new paths:
   1. Replace regexp test/import\.%(\.staging)?\.xml with src/%/test$1.xml
   2. Manually change occurrences of test/%
   3. Check that there are no occurrences of test/
   4. Change $(srcsOnly) to filter out test outputs
3. Remove test subdirs:
   

   svn rm inputs/*/test/
   

4. make test
5. Check diffs
6. Commit
7. On vegbiendev and nimoy, do the following, using the corresponding steps above:
   1. svn up
   2. Remove old, empty dirs:
      

      rmdir inputs/*/test/
      

8. make inputs/download-logs test=1

Move log files into subfolders¶

1. Move files on your local machine:
   

   shopt -s nullglob
   for file in inputs/*/import/*.log; do (set -x; mv "$file" "${file/%.log/.log.sql}"); done
   (set -x; mv inputs/SALVIAS-CSV/import/install.log.sql inputs/SALVIAS/import/install-organisms.log.sql)
   (set -x; mv inputs/SALVIAS/import/install.log.sql inputs/SALVIAS/import/install-stems.log.sql)
   for file in inputs/*/import/install.log.sql; do (set -x; mv "$file" "${file/install/install-specimens}"); done
   for file in inputs/*/import/{,install-}plots*.log.sql; do (set -x; mv "$file" "${file/plots/0.plots}"); done
   for file in inputs/*/import/{,install-}organisms*.log.sql; do (set -x; mv "$file" "${file/organisms/1.organisms}"); done
   for file in inputs/*/import/{,install-}stems*.log.sql; do (set -x; mv "$file" "${file/stems/2.stems}"); done
   for table in 0.plots 1.organisms 2.stems specimens; do
       for file in inputs/*/import/{,install-}$table*.log.sql; do
           dir="$(dirname "$file")/../src/$table/logs" 
           filename="$(basename "$file")" 
           filename="${filename/install-/install.}" 
           test -e "$dir" || { svn mkdir "$dir"; svn propset svn:ignore "*.log.sql
   *.trace" "$dir"; }
           (set -x; mv "$file" "$dir/${filename/$table./}") 
       done
   done
   

2. Manually move inputs/QMOR/import/steps.by_col.sql and update README.TXT
3. Change inputs/input.Makefile and inputs/Makefile to use the new paths:
   1. Change each occurrence of import/ to src/<table>/logs/
   2. Update $(srcsOnly) to filter out logs
   3. Check that there are no occurrences of import/
4. Remove import subdirs:
   

   ls -d inputs/*/import/
   # Check that dirs are empty
   svn rm inputs/*/import/
   

5. make inputs/QMOR/src/specimens/logs/steps.by_col.sql
6. make inputs/download-logs test=1
7. Check diffs
8. Commit
9. svn up on vegbiendev and nimoy
10. On vegbiendev and nimoy, do the following, using the corresponding steps above:
    1. svn up
    2. Move files
    3. Remove old, empty dirs:
       

       rmdir inputs/*/import/
       

11. make inputs/download-logs test=1

Determine import order dynamically¶

1. Add an "_" prefix to non-source-file-related subfolders of src subdirs:
   

   mv inputs/CTFS/src/CTFS_Files_22Feb/ inputs/CTFS/src/_CTFS_Files_22Feb/
   svn mv inputs/CTFS/src/scripts_to_drop_extra_tables/ inputs/CTFS/src/_scripts_to_drop_extra_tables/
   

   Also do the mv command on vegbiendev and nimoy
2. Commit
3. Change input.Makefile to determine import order from subfolders:
   1. Change $(tablesSort) and $(tables)
   2. Check that there are no occurrences of 0.plots 1.organisms 2.stems specimens
4. make inputs/remake
5. make test
6. Check diffs
7. Commit
8. svn up on vegbiendev and nimoy

Move src subdir into main dir¶

1. Make a clean checkout of the repository, which doesn't contain any unversioned files
2. Move files in the clean checkout:
   

   shopt -s nullglob
   (slash=/; for item in inputs/*/src/*; do svn mv "$item" "${item/src${slash}/}"; done)
   

3. Add svn:ignores:
   

   for dir in inputs/*/; do svn propset svn:ignore "*" "$dir"; done
   

4. Change inputs/input.Makefile, inputs/Makefile, and README.TXT to use the new paths:
   1. Remove each occurrence of src/
   2. Replace $(call addDirWithIgnore,src,'*') with $(call setSvnIgnore,.,'*')
   3. Update $(tables) to filter out src and verify
   4. Check that there are no occurrences of src/
5. Remove src subdirs:
   

   ls -d inputs/*/src/
   # Check that dirs are empty
   svn rm inputs/*/src/
   

6. make inputs/QMOR/specimens/logs/steps.by_col.log.sql
7. make inputs/download-logs test=1
8. Check diffs
9. Commit
10. Immediately do the following on your main local checkout, vegbiendev, and nimoy:
    1. svn up
    2. Move unversioned files:
       

       (slash=/; for item in $(find inputs -path "*/src/*" -and ! -path "*/.*"|xargs echo) inputs/Madidi/src/*.xls; do (set -x; mv "$item" "${item/src${slash}/}"); done)
       (slash=/; for name in specimens 0.plots 1.organisms 2.stems; do for item in inputs/*/$name/$name/*; do (set -x; mv "$item" "${item/$name${slash}/}"); done; done)
       (slash=/; for name in specimens 0.plots 1.organisms 2.stems; do for item in inputs/*/$name/logs/logs/*; do (set -x; mv "$item" "${item/logs${slash}/}"); done; done)
       mv inputs/CTFS/_1.organisms.disabled/_1.organisms.disabled/* inputs/CTFS/_1.organisms.disabled/
       

    3. Remove old, empty dirs:
       

       for name in specimens 0.plots 1.organisms 2.stems; do rm inputs/*/$name/$name/{.DS_Store,.~*}; rmdir inputs/*/$name/$name/ inputs/*/$name/logs/logs/; done
       rmdir inputs/CTFS/_1.organisms.disabled/_1.organisms.disabled inputs/CTFS/_1.organisms.disabled/logs/logs
       rm inputs/*/src/{.DS_Store,.~*}
       rmdir inputs/*/src/
       

11. On your main local checkout:
    1. make inputs/download test=1
    2. make inputs/download-logs test=1
    3. make test

Rename subfolders to VegCSV names¶

1. Make a clean checkout of the repository, which doesn't contain any unversioned files
2. Move files in the clean checkout:
   

   shopt -s nullglob
   for item in inputs/*/specimens/; do svn mv "$item" "${item/specimens/Specimen}"; done
   for item in inputs/*/0.plots/; do svn mv "$item" "${item/0.plots/Plot}"; done
   for item in inputs/*/1.organisms/; do svn mv "$item" "${item/1.organisms/Organism}"; done
   for item in inputs/*/2.stems/; do svn mv "$item" "${item/2.stems/Stem}"; done
   

3. Change names in the following datasources' import_order.txt (only found in datasources with multiple tables):
   Madidi, SALVIAS-CSV, SALVIAS
4. Rename installed staging tables:
   

   ALTER TABLE "ACAD".specimens RENAME TO "Specimen";
   ALTER TABLE "ARIZ".specimens RENAME TO "Specimen";
   ALTER TABLE "GBIF".specimens RENAME TO "Specimen";
   ALTER TABLE "MO".specimens RENAME TO "Specimen";
   ALTER TABLE "MT".specimens RENAME TO "Specimen";
   ALTER TABLE "NCU-NCSC".specimens RENAME TO "Specimen";
   ALTER TABLE "NY".specimens RENAME TO "Specimen";
   ALTER TABLE "QMOR".specimens RENAME TO "Specimen";
   ALTER TABLE "REMIB".specimens RENAME TO "Specimen";
   ALTER TABLE "SpeciesLink".specimens RENAME TO "Specimen";
   ALTER TABLE "U".specimens RENAME TO "Specimen";
   ALTER TABLE "UNCC".specimens RENAME TO "Specimen";
   
   ALTER TABLE "ACAD"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "ARIZ"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "GBIF"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "MO"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "MT"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "NCU-NCSC"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "NY"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "QMOR"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "REMIB"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "SpeciesLink"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "U"."specimens.errors" RENAME TO "Specimen.errors";
   ALTER TABLE "UNCC"."specimens.errors" RENAME TO "Specimen.errors";
   

   ALTER TABLE "Madidi"."0.plots" RENAME TO "Plot";
   ALTER TABLE "Madidi"."0.plots.errors" RENAME TO "Plot.errors";
   ALTER TABLE "SALVIAS-CSV"."0.plots" RENAME TO "Plot";
   ALTER TABLE "SALVIAS-CSV"."0.plots.errors" RENAME TO "Plot.errors";
   ALTER TABLE "SALVIAS"."0.plots" RENAME TO "Plot";
   ALTER TABLE "SALVIAS"."0.plots.errors" RENAME TO "Plot.errors";
   

   ALTER TABLE "CVS"."1.organisms" RENAME TO "Organism";
   ALTER TABLE "CVS"."1.organisms.errors" RENAME TO "Organism.errors";
   ALTER TABLE "Madidi"."1.organisms" RENAME TO "Organism";
   ALTER TABLE "Madidi"."1.organisms.errors" RENAME TO "Organism.errors";
   ALTER TABLE "SALVIAS-CSV"."1.organisms" RENAME TO "Organism";
   ALTER TABLE "SALVIAS-CSV"."1.organisms.errors" RENAME TO "Organism.errors";
   ALTER TABLE "SALVIAS"."1.organisms" RENAME TO "Organism";
   ALTER TABLE "SALVIAS"."1.organisms.errors" RENAME TO "Organism.errors";
   

   ALTER TABLE "SALVIAS"."2.stems" RENAME TO "Stem";
   ALTER TABLE "SALVIAS"."2.stems.errors" RENAME TO "Stem.errors";
   

5. Check diffs
6. Commit
7. Immediately do the following on your main local checkout, vegbiendev, and nimoy, using the corresponding steps above:
   1. svn up
   2. Move unversioned files:
      

      for item in inputs/*/specimens/logs/*; do (set -x; mv "$item" "${item/specimens/Specimen}"); done
      for item in inputs/*/0.plots/logs/*; do (set -x; mv "$item" "${item/0.plots/Plot}"); done
      for item in inputs/*/1.organisms/logs/*; do (set -x; mv "$item" "${item/1.organisms/Organism}"); done
      for item in inputs/*/2.stems/logs/*; do (set -x; mv "$item" "${item/2.stems/Stem}"); done
      
      for item in inputs/*/specimens/*; do (set -x; mv "$item" "${item/specimens/Specimen}"); done
      for item in inputs/*/0.plots/*; do (set -x; mv "$item" "${item/0.plots/Plot}"); done
      for item in inputs/*/1.organisms/*; do (set -x; mv "$item" "${item/1.organisms/Organism}"); done
      for item in inputs/*/2.stems/*; do (set -x; mv "$item" "${item/2.stems/Stem}"); done
      

   3. Remove old, empty dirs:
      

      rmdir inputs/*/specimens/logs/ inputs/*/specimens/
      rmdir inputs/*/0.plots/logs/ inputs/*/0.plots/
      rmdir inputs/*/1.organisms/logs/ inputs/*/1.organisms/
      rmdir inputs/*/2.stems/logs/ inputs/*/2.stems/
      rmdir inputs/*/*/logs/logs
      

   4. Rename the installed staging tables (don't need to do again on your main local checkout)
8. On your main local checkout:
   1. make inputs/download test=1
   2. make inputs/download-logs test=1
   3. make test