mappings/VegCore-VegBIEN.csv: Changed _merge to _join wherever the duplicate-eliminating functionality of _merge is not needed and a simple concatenation of non-NULL values is sufficient
xml_func.py: Added _join() simplifying function
schemas/functions.sql: Added _join()
mappings/VegCore-VegBIEN.csv: Moved "if subplot" _if statement around /location/parent_id and /location/sourceaccessioncode themselves, so that only one _if cond mapping for subplot is needed. Note that this is only possible because this _if statement uses _exists, allowing it to be fully evaluated by the XML template simplifying mechanism, which supports subtrees as arguments to _if.
mappings/VegCore-VegBIEN.csv: Removed no longer used parentLocationID, parentPlotName (locationID and plotName now automatically map to the correct location). mappings/Veg+-VegCore.csv: Removed no longer used parentPlotID.
xml_func.py: passthru(): Use xml_dom.prune() so that after empty children are removed, the node itself is also removed if it's empty. This enables further pruning of any node that contains the pruned node.
xml_dom.py: Added prune()
xml_func.py: Removed no longer used prune() (use xml_dom.prune_children() instead)
xml_func.py: Use new xml_dom.prune_children()
xml_dom.py: Added prune_empty() and prune_children()
inputs/CTFS/: Moved VegX export subdir to _archive and renamed it to remove ".disabled" suffix and have a VegCSV-like name
inputs/CTFS/: Renamed README.TXT to DFtemp.analysis_query.txt because it relates only to a particular query from Shash, and moved it to the _archive/ subdir
inputs/CTFS/: Moved source files into new _src/ subdir to avoid cluttering up the main dir
Added inputs/CTFS/_src/
inputs/CTFS/: Added non-data files that weren't under version control
inputs/CTFS/: Moved _scripts_to_drop_extra_tables to _archive because they are for a different version of the CTFS database than the extract we received (bci.sql)
inputs/CTFS/: Moved DBv5.txt to _archive because it's for a different version of the CTFS database than the extract we received (bci.sql)
inputs/CTFS/: Moved CTFS_conversion_bci.php to _archive since it's just for the DFtemp (aggregated) mapping
Added inputs/CTFS/_archive
inputs/import.stats.xls: Updated with stats from latest import
Added inputs/CTFS/PlotObservation/
mappings/VegCore-VegBIEN.csv: fieldNumber (authoreventcode): Don't copy to location.authorlocationcode if an actual locationID was specified
xml_func.py: simplify(): Removed no longer needed pass-through optimizations for XML functions, which are now handled by each function's own simplifying function
xml_func.py: Added _name simplifying function
xml_func.py: Added _alt, _merge simplifying functions
xml_func.py: passthru(): First prune the node
xml_func.py: simplify(): Use new passthru()
xml_func.py: Added passthru()
xml_func.py: simplify(): Use new prune()
xml_func.py: Added prune()
mappings/VegCore-VegBIEN.csv: Mapped eventID
mappings/Veg+-VegCore.csv: Mapped CTFS Census terms
mappings/Veg+.terms.csv: Added CTFS Census terms
mappings/VegCore-VegBIEN.csv: Changed plotEventStartDate, plotEventEndDate to startDate, endDate because a date range always applies to the event
mappings/Veg+.terms.csv: Added startDate, endDate
README.TXT: Testing: Mapping process: Added command to include column-based import tests
README.TXT: Datasource setup: Update vegbiendev: Added step to run the tests, to make sure the staging tables were installed properly
inputs/CTFS/Plot/: Added create.sql
inputs/CTFS/: Added import_order.txt
Added inputs/CTFS/Subplot/
mappings/Veg+-VegCore.csv: Mapped CTFS QuadratID
mappings/VegCore-VegBIEN.csv: Mapped subplotID
mappings/Veg+.terms.csv: Added subplotID
mappings/Veg+-VegCore.csv: Mapped CTFS Quadrat columns
mappings/VegCore-VegBIEN.csv: Mapped subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Removed empty mappings for unmapped DwC terms because these terms are now listed and maintained in mappings/Veg+.terms.csv
mappings/Veg+.terms.csv: Added Brad's descriptive comments for several VegCore terms
mappings/Veg+.terms.csv: Added subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Made organismX, organismY the official VegCore terms and map relativePlotX, relativePlotY to them in mappings/Veg+-VegCore.csv
mappings/Veg+.terms.csv: Added organismX, organismY as clearer alternatives to relativePlotX, relativePlotY
mappings/Veg+.terms.csv: Added CTFS Quadrat columns
Added inputs/CTFS/
input.Makefile: Testing: Only run column-based tests if column-based mode enabled, because these tests are much slower than the row-based tests for small numbers of rows. Note that this involves explicitly turning off column-based mode in the row-based test, to prevent propagation of the by_col env var which both enables these extra tests and sets bin/map to run in column-based mode.
input.Makefile: Testing: Added by-column test, which is compared to the row-based test's accepted output
input.Makefile: Testing: Merged $(runTest) and $(test2Db) because all tests go to the database
input.Makefile: Testing: Moved `$(foreach use_staged,1,...)` from $(test2Db) to $(runTest) because all tests now use the staging tables
input.Makefile: Testing: Merged $(test2Db) and $(testStaged2Db) because all tests now use the staging tables
input.Makefile: Testing: $(runTest): Always use $(map2db) because there are no tests that use other programs (and haven't been in awhile)
input.Makefile: Testing: Run the core test from the staging table, because derived tables only have a staging table and the flat-file test would produce inconsistent results
mappings/Makefile: Fixed bug where rules needed to generate Veg+.self.csv ($(viaSelfMap)) were still using a pattern match that required a table (`.%.`, `.*.`), even though we are no longer using separate maps for separate tables
mappings/Veg+-VegCore.csv: Mapped CTFS Country and Site columns
mappings/Veg+.terms.csv: Added CTFS Country and Site columns
README.TXT: Datasource setup: Adding input data: svn adding the generated map spreadsheets and related files: Added header.csv to the list of files added (for derived tables)
README.TXT: Datasource setup: Adding input data: Documented how to create tables that will be joined together with another table, and how to create tables that are joins of other tables
input.Makefile: Staging tables installation: %/install: Also create header.csv so that there is a CSV header that the map spreadsheets can be autogenerated from
input.Makefile: Staging tables installation: %/install: Add row_num column to derived staging tables so they will have a pkey
sql.py: pkey(): Use pkey_col constant if this column exists, to allow using a row_num column as the pkey even when it is placed at the end of the table (due to being added after the table was created)
input.Makefile: Staging tables installation: %/install: Support alternative generation of a staging table by joining together other staging tables in a create.sql file
input.Makefile: Staging tables installation: %/install: Don't create a row_num column when the table is a joined table because it collides during joins
csv2db: Made input_cmd optional when errors_table_only is on, because the CSV header is not needed to create the errors table
csv2db: Added has_row_num param to disable creating a row_num column
input.Makefile: Existing maps discovery: $(allTables): When prepending unsorted (joined) tables, save them in $(joinedTables) for later use in determining which tables should have a row_num column
README.TXT: Fixed indent
input.Makefile: Staging tables installation: Install all tables, not just those present in import_order.txt. This will later allow staging tables to be derived by joining together other staging tables, which themselves are not imported but still need to be installed.
input.Makefile: Existing maps discovery: $(tables): Prepend unsorted tables (those that are not present in import_order.txt)
input.Makefile: Renamed "...-%" targets to "%/..." so they are more logically associated with a specific subdir
mappings/Veg+.terms.csv: Added Madidi terms that don't exist in other datasources
inputs/Madidi/0.plots/map.csv: Added [Veg+] to root to enable auto-mapping
inputs/SALVIAS*/1.organisms/map.csv: Map directly to locationID, plotName instead of parentLocationID, parentPlotName because these terms now map correctly to the parent location when a subplot column exists
mappings/VegCore-VegBIEN.csv: plotName -> /location/authorlocationcode mapping: When subplot is provided, remove this mapping using _if ... _exists instead of _alt so that a NULL subplot value will not cause the parent plot's name to be used for the subplot name
input.Makefile: Testing: $(runTest): Remove outputs of successful tests to reduce clutter
input.Makefile: Testing: %/test.staging.xml: Don't create test.staging.xml at all for non-flat-file inputs, because it is not needed (diff does not run in this case)
mappings/VegCore-VegBIEN.csv: Fixed bug where "if subplot" conditions would evaluate to true only if the subplot was NOT NULL, when they should actually evaluate to true if the datasource specified any subplot column, nullable or not
xml_func.py: simplify(): Removed no longer needed hardcoded _if simplifying code now that there is an _if() simplifying function
db_xml.py: input_col_prefix: Use value of xml_func.var_name_prefix, which is now the place where this value is configured
db_xml.py: Moved input_col_prefix above the put() function that uses it
xml_func.py: Added _if() simplifying function
xml_func.py: Added is_var_name() and is_var()
xml_dom.py: Added NodeEntryIter
xml_func.py: Added _exists()
xml_func.py: simplify(): Added support for custom simplifying functions, which are not hard-coded in simplify()
xml_dom.py: replace_with_text(): Use new bool2str() so that False causes the node to be removed instead of replaced with the empty string
xml_dom.py: Added bool2str()
inputs/SALVIAS*/1.organisms/map.csv: Mapped subplot, Line to new subplot VegCore term
mappings/VegCore-VegBIEN.csv: Mapped subplot, which involved replacing an _if with _alt to both remove plotName as the authorlocationcode and use subplot instead when subplot is specified
mappings/VegCore-VegBIEN.csv: locationID, plotName: Redirect to /location/parent_id/location/* if subplot field is specified
xml_func.py: simplify(): Also remove _if statements with only a condition. This is a required transformation, because such _if statements can't be handled by functions._if() due to there being no argument to provide the anyelement type.
xml_func.py: simplify(): Added pruning optimization that removes empty children. Empty children are created when some mappings don't apply to the current datasource.
xml_func.py: simplify(): Only generate children list if node is a function
