inputs/CTFS/: Added import_order.txt
Added inputs/CTFS/Subplot/
mappings/Veg+-VegCore.csv: Mapped CTFS QuadratID
mappings/VegCore-VegBIEN.csv: Mapped subplotID
mappings/Veg+.terms.csv: Added subplotID
mappings/Veg+-VegCore.csv: Mapped CTFS Quadrat columns
mappings/VegCore-VegBIEN.csv: Mapped subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Removed empty mappings for unmapped DwC terms because these terms are now listed and maintained in mappings/Veg+.terms.csv
mappings/Veg+.terms.csv: Added Brad's descriptive comments for several VegCore terms
mappings/Veg+.terms.csv: Added subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Made organismX, organismY the official VegCore terms and map relativePlotX, relativePlotY to them in mappings/Veg+-VegCore.csv
mappings/Veg+.terms.csv: Added organismX, organismY as clearer alternatives to relativePlotX, relativePlotY
mappings/Veg+.terms.csv: Added CTFS Quadrat columns
Added inputs/CTFS/
input.Makefile: Testing: Only run column-based tests if column-based mode enabled, because these tests are much slower than the row-based tests for small numbers of rows. Note that this involves explicitly turning off column-based mode in the row-based test, to prevent propagation of the by_col env var which both enables these extra tests and sets bin/map to run in column-based mode.
input.Makefile: Testing: Added by-column test, which is compared to the row-based test's accepted output
input.Makefile: Testing: Merged $(runTest) and $(test2Db) because all tests go to the database
input.Makefile: Testing: Moved `$(foreach use_staged,1,...)` from $(test2Db) to $(runTest) because all tests now use the staging tables
input.Makefile: Testing: Merged $(test2Db) and $(testStaged2Db) because all tests now use the staging tables
input.Makefile: Testing: $(runTest): Always use $(map2db) because there are no tests that use other programs (and haven't been in awhile)
input.Makefile: Testing: Run the core test from the staging table, because derived tables only have a staging table and the flat-file test would produce inconsistent results
mappings/Makefile: Fixed bug where rules needed to generate Veg+.self.csv ($(viaSelfMap)) were still using a pattern match that required a table (`.%.`, `.*.`), even though we are no longer using separate maps for separate tables
mappings/Veg+-VegCore.csv: Mapped CTFS Country and Site columns
mappings/Veg+.terms.csv: Added CTFS Country and Site columns
README.TXT: Datasource setup: Adding input data: svn adding the generated map spreadsheets and related files: Added header.csv to the list of files added (for derived tables)
README.TXT: Datasource setup: Adding input data: Documented how to create tables that will be joined together with another table, and how to create tables that are joins of other tables
input.Makefile: Staging tables installation: %/install: Also create header.csv so that there is a CSV header that the map spreadsheets can be autogenerated from
input.Makefile: Staging tables installation: %/install: Add row_num column to derived staging tables so they will have a pkey
sql.py: pkey(): Use pkey_col constant if this column exists, to allow using a row_num column as the pkey even when it is placed at the end of the table (due to being added after the table was created)
input.Makefile: Staging tables installation: %/install: Support alternative generation of a staging table by joining together other staging tables in a create.sql file
input.Makefile: Staging tables installation: %/install: Don't create a row_num column when the table is a joined table because it collides during joins
csv2db: Made input_cmd optional when errors_table_only is on, because the CSV header is not needed to create the errors table
csv2db: Added has_row_num param to disable creating a row_num column
input.Makefile: Existing maps discovery: $(allTables): When prepending unsorted (joined) tables, save them in $(joinedTables) for later use in determining which tables should have a row_num column
README.TXT: Fixed indent
input.Makefile: Staging tables installation: Install all tables, not just those present in import_order.txt. This will later allow staging tables to be derived by joining together other staging tables, which themselves are not imported but still need to be installed.
input.Makefile: Existing maps discovery: $(tables): Prepend unsorted tables (those that are not present in import_order.txt)
input.Makefile: Renamed "...-%" targets to "%/..." so they are more logically associated with a specific subdir
mappings/Veg+.terms.csv: Added Madidi terms that don't exist in other datasources
inputs/Madidi/0.plots/map.csv: Added [Veg+] to root to enable auto-mapping
inputs/import.stats.xls: Updated with stats from latest import
inputs/SALVIAS*/1.organisms/map.csv: Map directly to locationID, plotName instead of parentLocationID, parentPlotName because these terms now map correctly to the parent location when a subplot column exists
mappings/VegCore-VegBIEN.csv: plotName -> /location/authorlocationcode mapping: When subplot is provided, remove this mapping using _if ... _exists instead of _alt so that a NULL subplot value will not cause the parent plot's name to be used for the subplot name
input.Makefile: Testing: $(runTest): Remove outputs of successful tests to reduce clutter
input.Makefile: Testing: %/test.staging.xml: Don't create test.staging.xml at all for non-flat-file inputs, because it is not needed (diff does not run in this case)
mappings/VegCore-VegBIEN.csv: Fixed bug where "if subplot" conditions would evaluate to true only if the subplot was NOT NULL, when they should actually evaluate to true if the datasource specified any subplot column, nullable or not
xml_func.py: simplify(): Removed no longer needed hardcoded _if simplifying code now that there is an _if() simplifying function
db_xml.py: input_col_prefix: Use value of xml_func.var_name_prefix, which is now the place where this value is configured
db_xml.py: Moved input_col_prefix above the put() function that uses it
xml_func.py: Added _if() simplifying function
xml_func.py: Added is_var_name() and is_var()
xml_dom.py: Added NodeEntryIter
xml_func.py: Added _exists()
xml_func.py: simplify(): Added support for custom simplifying functions, which are not hard-coded in simplify()
xml_dom.py: replace_with_text(): Use new bool2str() so that False causes the node to be removed instead of replaced with the empty string
xml_dom.py: Added bool2str()
inputs/SALVIAS*/1.organisms/map.csv: Mapped subplot, Line to new subplot VegCore term
mappings/VegCore-VegBIEN.csv: Mapped subplot, which involved replacing an _if with _alt to both remove plotName as the authorlocationcode and use subplot instead when subplot is specified
mappings/VegCore-VegBIEN.csv: locationID, plotName: Redirect to /location/parent_id/location/* if subplot field is specified
xml_func.py: simplify(): Also remove _if statements with only a condition. This is a required transformation, because such _if statements can't be handled by functions._if() due to there being no argument to provide the anyelement type.
xml_func.py: simplify(): Added pruning optimization that removes empty children. Empty children are created when some mappings don't apply to the current datasource.
xml_func.py: simplify(): Only generate children list if node is a function
xml_func.py: simplify(): Refactored to support processing nodes that are not functions. Changed var names for clarity.
mappings/VegCore-VegBIEN.csv: _simplifyPath() calls: Removed no longer needed `require` arg, and removed no longer needed table suffix from `next` arg
db_xml.py: put(): _simplifyPath() built-in function: Removed `require` param, which is not used by this _simplifyPath() implementation because the database constraints handle this
mappings/Veg+.terms.csv: Added subplot
input.Makefile: SVN: add: Also add empty import_order.txt
lib/common.Makefile: SVN: Added $(addFile)
input.Makefile: SVN: add: Don't automatically add a Specimen subdir, because some plots datasources don't have that table
README.TXT: Datasource setup: Adding input data: Added step to add <table> to inputs/<datasrc>/import_order.txt
README.TXT: Datasource setup: Changed "<name>" to "<datasrc>" to distinguish it more clearly from "<table>", which is also a name
README.TXT: Datasource setup: Adding input data: Changed steps to use new %/add command to add table's subdir
input.Makefile: SVN: Added %/add to add a new table subdir. add: Changed default subdir name to Specimen to match suggested table names at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCSV#Suggested-table-names>. Use new %/add to add it.
README.TXT: Datasource setup: Replaced fixed table names with link to VegCSV suggested table names
input.Makefile: $(srcsOnly): Include only files ending in one of the data extensions: csv tsv txt xml. This allows the data provider to include other documentation files, such as SQL export queries, in the table subdirs.
bin/map: Documented that it is duplicate-column safe (supports multiple columns of the same name)
README.TXT: Datasource setup: Obtaining CSVs: Documented that when exporting relational databases to CSVs, you MUST ensure that embedded quotes are escaped by doubling them, not by preceding them with a "\" as is the default in phpMyAdmin
csvs.py: delims: Added ";", which is phpMyAdmin's default CSV delimiter
sql_io.py: null_strs: Added 'NULL', which is used by phpMyAdmin as the default "Replace NULL with" value for CSV exports
sql_io.py: cleanup_table(): Refactored to use for loop with array constant, so that additional NULL-equivalent strings can easily be added
mappings/roots/: Merged roots for different tables into one mappings/root.sh for Veg+, which handles all tables' mappings to VegBIEN
sql_io.py: put_table(): When ignoring all rows for an iteration, return literal NULL value instead of column of NULLs as an optimization for callers using that iteration's pkeys
mappings/VegCore-VegBIEN.csv: Primary taxondetermination: Removed [role=identifier] because the role of the entity making the determination is unknown. Added [!isoriginal] filter to those mappings to ensure that primary taxondetermination XPaths map to a different taxondetermination than the [isoriginal=true] determination when both are present.
inputs/SALVIAS*/1.organisms/map.csv: Remapped cfaff to identificationQualifier, because it was previously mapped to the same taxondetermination as the Orig* terms but does not have a corresponding Orig prefix to indicate that it should apply to the original determination instead of the primary TNRS one
mappings/Veg+.terms.csv: Removed no longer used computer.* taxonomic terms
mappings/VegCore-VegBIEN.csv: Removed no longer used computer.* taxonomic terms
inputs: Regenerated VegBIEN.csv for several datasources, which had apparently not gotten regenerated when make was run after the taxonRank mapping addition
backups/: svn:ignore: Also ignore .*, which includes temp files generated by rsync
xml_func.py: simplify(): Also consider _name() to be an aggregate function
inputs/SALVIAS*/1.organisms/map.csv: Removed computer.* prefix from primary (TNRS) taxondetermination, so it would map to the main taxondetermination in VegBIEN
mappings/VegCore-VegBIEN.csv: Mapped taxonRank analogously to computer.taxonRank
inputs/SALVIAS*/1.organisms/map.csv: Remapped OrigFamily/OrigGenus/OrigSpecies to new verbatim* taxonomic names. Also remapped cfaff to verbatimIdentificationQualifier, because it was previously mapped to the same taxondetermination as the Orig* terms, but this will later need to be remapped to identificationQualifier (not in this commit because that is a separate change). Note that the switch to the verbatim* taxonomic names removes a concatenated binomial that was part of the previous mappings, which put OrigGenus and OrigSpecies together into one scientificName.
mappings/VegCore-VegBIEN.csv: Mapped verbatimScientificName to taxonoccurrence.authortaxoncode as an alternative to scientificName
mappings/VegCore-VegBIEN.csv: Mapped verbatim* taxonomic terms
mappings/Veg+.terms.csv: Added verbatimIdentificationQualifier
mappings/Veg+.terms.csv: Added verbatimScientificName
schemas/vegbien.sql: taxondetermination: taxondetermination_unique unique index: Added isoriginal so an "original" determination in the same row (as found in SALVIAS) will be seen as distinct from the scrubbed determination, even if they are to the same plant name
