fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)
inputs/input.Makefile: add: verify/: also svn:ignore *.log
inputs/input.Makefile: add!: verify/: also svn:ignore *.tsv, *.txt
moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).
copyright scrub: inputs/: removed data provider-owned schema and documentation files, which are not BIEN copyright and should not be part of what is submitted for open-sourcing. these files will remain accessible via the web interface (fs.vegpath.org), but will not be in the repository.
fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error
inputs/*/*/test.xml.ref: updated source.shortname for new datasource name, which now starts out with .new suffix
added inputs/newWorld/iso_code_gadm/VegBIEN.csv, etc., generated by running ./run
added inputs/newWorld/newWorldCountries/map.csv, etc., generated by running ./run
inputs/{.,}*/*.schema.sql: regenerated using the instructions in bin/my2pg. this primarily replaces timestamp with text/*timestamp*/ (to preserve indefinite dates).
inputs/newWorld/: switched to new-style import, which adds the staging table column renaming. these tables are used by the public schema (schemas/vegbien.sql), so the renamings are applied there as well.
bugfix: inputs/newWorld/newWorldCountries/postprocess.sql: use UPDATE statement (followed by VACUUM ANALYZE to remove dead tuples) instead of in-place update (ALTER COLUMN TYPE USING), so that the statement can be run even after the public schema has been installed and its views use the columns. (a view using the columns would normally block an ALTER COLUMN TYPE statement on a referenced column.)
bugfix: inputs/*/*/postprocess.sql: made all operations idempotent, so that postprocess.sql can be run repeatedly (e.g. by new-style import)
added inputs/newWorld/iso_code_gadm/.map.csv.last_cleanup
inputs/*/Source/VegBIEN.csv: regenerated for new-style import, which uses a symlink to mappings/VegCore-VegBIEN.csv instead of a custom mapping using the original column names
bugfix: inputs/*/Source/map.csv: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table. the staging table column renaming is now used by all Source tables.
added inputs/newWorld/iso_code_gadm/header.csv
bugfix: inputs/*/Source/: added missing ./run, which creates the new-style staging tables with the metadata fields as part of the table. this is needed now that these subdirs use installed staging tables instead of metadata-only map.csvs.
inputs/*/: added table.run for use by the table subdirs in new-style import. datasources without table subdirs do not need this.
inputs/*/: added top-level Makefile which includes inputs/input.Makefile, so that make can be run directly on the datasrc dir without needing to specify `--makefile=../input.Makefile` (see input.Makefile $(selfMake))
bugfix: inputs/*/Source/: use installed staging table (with blank-line data.csv) in order to also work with new-style import. this also fixes a benign diff between the by-row and by-col test outputs, where row-based import would not import the Source/ entries because there was not at least one row in the input. note that in order to ensure that all datasources are properly run, you need to check `svn st|sort` against the datasource schema names to see if any are missing.
inputs/*/logs: updated svn:ignore
inputs/*/*/logs: updated svn:ignore
empty inputs/*/import_order.txt: added subdirs in the order they are used by inputs/input.Makefile, by running make on the inputs to auto-populate import_order.txt. import_order.txt is needed by the runscripts to run the right set of subdirs in the right order.
inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)
mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
Added inputs/newWorld/newWorldCountries/_no_import
Added inputs/newWorld/newWorldCountries/ with postprocess.sql that sets isNewWorld to false wherever it's NULL. (The input table only marks New World countries as true, but doesn't mark non-New World countries as false.)
inputs/*/Source/map.csv without mappings: Added referenceType, etc. mappings. This also ensures that the source table entry for the datasource will be created before the herbaria list is imported, causing all top-level datasources to sort at the top of the source table.
input.Makefile: SVN: add: verify: Also ignore *.xlsx
inputs/input.Makefile: SVN: add: verify/: Added *.xls to svn:ignore
input.Makefile: SVN: add: Add a Source table to store datasource metadata. This adds a Source table to all herbaria which are listed in .herbaria, and therefore didn't previously need a Source table to indicate their referenceType and sampleType.
inputs/*/verify/: Updated svn:ignore
inputs/newWorld/geoscrub.schema.~.changes.sql: Reversed order of adding unique constraints and changing types
inputs/newWorld/geoscrub.schema.~.changes.sql: Changed isoCode type to text. Added unique constraint on isoCode.
Added inputs/newWorld/iso_code_gadm/
Added inputs/newWorld/_src/
inputs/newWorld/geoscrub.schema.~.changes.sql: Change countryNameStd type to text to allow merge-joining with place.country in analytical_stem_view
inputs/newWorld/geoscrub.schema.~.changes.sql: ALTER TABLE ... ALTER COLUMN statement: Reformatted to allow adding additional ALTER COLUMN clauses
inputs/newWorld/: Added geoscrub.schema.~.changes.sql to add a unique constraint on countryNameStd and change isNewWorld's type to boolean
Added inputs/newWorld/
