inputs/CTFS/: Added TaxonOccurrence/ and its joined tables
inputs/CTFS/_archive/Organism.VegX/README.TXT: Added calculation of StemObservation rows distribution for each plot, which indicates that the bci plot actually contains 90% of the StemObservation rows. This brings the size inflation of VegX down to ~6x.
inputs/CTFS/_archive/Organism.VegX/: Added README.TXT describing that this VegX export includes only one of 157 CTFS plots. This is important, because it indicates that VegX creates a ~1000x (!) increase in storage size (613.6 MB for bci.sql with 157 plots vs. 3.78 GB for VegX_CTFS_row_*.xml with 1 plot, assuming roughly equal #s of stems per plot).
inputs/CTFS/StemObservation/map.csv: Remapped StemID to authorStemCode since it's only unique within the parent organism (Tree), not a globally unique ID as is required for stemID
mappings/VegCore-VegBIEN.csv: Mapped authorStemCode
mappings/Veg+.terms.csv: Added authorStemCode
mappings/VegCore-VegBIEN.csv: Mapped stemID
inputs/SALVIAS/2.stems/map.csv: Mapped stem_id
README.TXT: Datasource setup: Added steps to install any MySQL export
mappings/Veg+-VegCore.csv: Mapped stem_id
repl: Support treating all patterns as plain text (non-regexp)
mappings/Veg+.terms.csv: Added stem_id
mappings/Veg+.terms.csv: Added stemID
mappings/Veg+-VegCore.csv: Mapped speciesName, subSpeciesName
mappings/Veg+.terms.csv: Added CTFS taxonomic name columns
mappings/Veg+.terms.csv: Removed comments not applicable to the term itself
Inputs with multiple tables: Added explicit import_order.txt files, so that sort orders can later be removed from the subdir names
inputs/CTFS/: Added StemObservation/ and tables it is joined from
mappings/Veg+-VegCore.csv: Mapped stemTag
mappings/Veg+.terms.csv: Added stemTag
mappings/Veg+-VegCore.csv: Mapped DBH
mappings/Veg+.terms.csv: Added DBH
input.Makefile: Maps building: Added comment that you cannot make a subdir separately from the entire datasource dir
inputs/CTFS/Plot/create.sql: Added newline at end of file
inputs/CTFS/: Renamed Site.src to Plot.src to use a VegCSV name for the table
README.TXT: Datasource setup: Adding input data for each table: `make inputs/<datasrc>/<table>/add`: Added note explaining why you need to use this command instead of just creating an empty directory of the desired name
inputs/CTFS/: Added SubplotObservation/
mappings/VegCore-VegBIEN.csv: Redirect eventID, fieldNumber (authoreventcode) to parent locationevent when subplot columns exist
inputs/CTFS/import_order.txt: Added PlotObservation
inputs/CTFS/PlotObservation/: Remade (hadn't been automatically remade because it wasn't part of import_order.txt)
mappings/VegCore-VegBIEN.csv: Also redirect locationID/plotName to parent location if subplotID column was provided
mappings/VegCore-VegBIEN.csv: location.authorlocationcode mappings: Use _first to remove specimens-related alternatives for this field from consideration when plots-related alternatives exist. This avoids unintentionally using specimens-related columns for this field in plots data.
xml_func.py: Added _first() simplifying function
xml_func.py: Added helper functions variadic_args() and map_names()
mappings/VegCore-VegBIEN.csv: location.authorlocationcode mappings: Placed inside "if subplot" _if statement along with sourceaccessioncode to reduce the number of separate _if statements needing a condition mapping
xml_dom.py: NodeEntryIter: Support entries with multiple children
xml_dom.py: replace(): Support a list of new nodes to replace the old node with
xml_dom.py: Moved only_child() near related method has_one_child()
xml_dom.py: only_child(): Raise exception instead of failing assertion. Include invalid node in exception message for easier debugging.
xml_dom.py: Added only_child() and use it where its definition was used
mappings/VegCore-VegBIEN.csv: Changed _merge to _join wherever the duplicate-eliminating functionality of _merge is not needed and a simple concatenation of non-NULL values is sufficient
xml_func.py: Added _join() simplifying function
schemas/functions.sql: Added _join()
mappings/VegCore-VegBIEN.csv: Moved "if subplot" _if statement around /location/parent_id and /location/sourceaccessioncode themselves, so that only one _if cond mapping for subplot is needed. Note that this is only possible because this _if statement uses _exists, allowing it to be fully evaluated by the XML template simplifying mechanism, which supports subtrees as arguments to _if.
mappings/VegCore-VegBIEN.csv: Removed no longer used parentLocationID, parentPlotName (locationID and plotName now automatically map to the correct location). mappings/Veg+-VegCore.csv: Removed no longer used parentPlotID.
xml_func.py: passthru(): Use xml_dom.prune() so that after empty children are removed, the node itself is also removed if it's empty. This enables further pruning of any node that contains the pruned node.
xml_dom.py: Added prune()
xml_func.py: Removed no longer used prune() (use xml_dom.prune_children() instead)
xml_func.py: Use new xml_dom.prune_children()
xml_dom.py: Added prune_empty() and prune_children()
inputs/CTFS/: Moved VegX export subdir to _archive and renamed it to remove ".disabled" suffix and have a VegCSV-like name
inputs/CTFS/: Renamed README.TXT to DFtemp.analysis_query.txt because it relates only to a particular query from Shash, and moved it to the _archive/ subdir
inputs/CTFS/: Moved source files into new _src/ subdir to avoid cluttering up the main dir
Added inputs/CTFS/_src/
inputs/CTFS/: Added non-data files that weren't under version control
inputs/CTFS/: Moved _scripts_to_drop_extra_tables to _archive because they are for a different version of the CTFS database than the extract we received (bci.sql)
inputs/CTFS/: Moved DBv5.txt to _archive because it's for a different version of the CTFS database than the extract we received (bci.sql)
inputs/CTFS/: Moved CTFS_conversion_bci.php to _archive since it's just for the DFtemp (aggregated) mapping
Added inputs/CTFS/_archive
inputs/import.stats.xls: Updated with stats from latest import
Added inputs/CTFS/PlotObservation/
mappings/VegCore-VegBIEN.csv: fieldNumber (authoreventcode): Don't copy to location.authorlocationcode if an actual locationID was specified
xml_func.py: simplify(): Removed no longer needed pass-through optimizations for XML functions, which are now handled by each function's own simplifying function
xml_func.py: Added _name simplifying function
xml_func.py: Added _alt, _merge simplifying functions
xml_func.py: passthru(): First prune the node
xml_func.py: simplify(): Use new passthru()
xml_func.py: Added passthru()
xml_func.py: simplify(): Use new prune()
xml_func.py: Added prune()
mappings/VegCore-VegBIEN.csv: Mapped eventID
mappings/Veg+-VegCore.csv: Mapped CTFS Census terms
mappings/Veg+.terms.csv: Added CTFS Census terms
mappings/VegCore-VegBIEN.csv: Changed plotEventStartDate, plotEventEndDate to startDate, endDate because a date range always applies to the event
mappings/Veg+.terms.csv: Added startDate, endDate
README.TXT: Testing: Mapping process: Added command to include column-based import tests
README.TXT: Datasource setup: Update vegbiendev: Added step to run the tests, to make sure the staging tables were installed properly
inputs/CTFS/Plot/: Added create.sql
inputs/CTFS/: Added import_order.txt
Added inputs/CTFS/Subplot/
mappings/Veg+-VegCore.csv: Mapped CTFS QuadratID
mappings/VegCore-VegBIEN.csv: Mapped subplotID
mappings/Veg+.terms.csv: Added subplotID
mappings/Veg+-VegCore.csv: Mapped CTFS Quadrat columns
mappings/VegCore-VegBIEN.csv: Mapped subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Removed empty mappings for unmapped DwC terms because these terms are now listed and maintained in mappings/Veg+.terms.csv
mappings/Veg+.terms.csv: Added Brad's descriptive comments for several VegCore terms
mappings/Veg+.terms.csv: Added subplotX, subplotY
mappings/VegCore-VegBIEN.csv: Made organismX, organismY the official VegCore terms and map relativePlotX, relativePlotY to them in mappings/Veg+-VegCore.csv
mappings/Veg+.terms.csv: Added organismX, organismY as clearer alternatives to relativePlotX, relativePlotY
mappings/Veg+.terms.csv: Added CTFS Quadrat columns
Added inputs/CTFS/
input.Makefile: Testing: Only run column-based tests if column-based mode enabled, because these tests are much slower than the row-based tests for small numbers of rows. Note that this involves explicitly turning off column-based mode in the row-based test, to prevent propagation of the by_col env var which both enables these extra tests and sets bin/map to run in column-based mode.
input.Makefile: Testing: Added by-column test, which is compared to the row-based test's accepted output
input.Makefile: Testing: Merged $(runTest) and $(test2Db) because all tests go to the database
input.Makefile: Testing: Moved `$(foreach use_staged,1,...)` from $(test2Db) to $(runTest) because all tests now use the staging tables
input.Makefile: Testing: Merged $(test2Db) and $(testStaged2Db) because all tests now use the staging tables
input.Makefile: Testing: $(runTest): Always use $(map2db) because there are no tests that use other programs (and haven't been in awhile)
