Regenerated vegbien.ERD exports
vegbien.ERD.mwb: Added reference and party tables
vegbien.ERD.mwb: Fixed lines
filter_ERD.csv: Remove fkeys to heavily-linked tables (reference, party)
Added to_do/milestones.doc
Renamed milestones.doc to timeline.doc
Added schemas/filter_ERD.csv and use it when generating vegbien.my.sql
vegbien.ERD.mwb: Added cover* to main diagram
vegbien.ERD.mwb: Started adding additional tables "below the fold" on the 2nd page
vegbien.ERD.mwb: Moved legend to top left to make room for more misc tables. Organized legend by location on diagram.
vegbien.ERD.mwb: Added soilobs table
vegbien.ERD.mwb: Added userdefined tables. Fixed lines.
vegbien.ERD.mwb: Changed location color to match VegBank ERD
vegbien.ERD.mwb: Added trait to diagram
vegbien.ERD.mwb: Added plantstatus to diagram. Added margins around diagram.
Added milestones.doc
DwC mappings: Fixed syntax of _date XML funcs to not wrap dates twice in a _date func
xml_func.py: Fixed bug in SyntaxException constructor where the cause was not passed to ExceptionWithCause
xml_dom.py: Override Node.__repr__ and Element.__repr__ to make sure self.toprettyxml() is used in all cases where a Node is converted to a string
util.py: Don't print type name in ConstraintError message because sometimes an entry tuple (e.g. from a dict) is checked, and then the type isn't useful
xml_func.py: Use conv_items(str, ...) in XML funcs that require strings, but don't otherwise use conv_items()
xml_func.py: Use xml_dom.TextEntryOnlyIter to check inputs for XML funcs that only handle strings, not whole elements
xml_dom.py: Added TextEntryOnlyIter to check that values returned by NodeTextEntryIter are strings from text nodes
util.py: Added is_str() and CheckedIter
input.Makefile: Log each import to a new log file named according to the current time
xml_func.py: SyntaxException: Use exc.ExceptionWithCause
exc.py: Added str_() to convert an Exception to a string. Use str_() in print_ex().
xml_func.py: conv_items(): Only trap type conversion errors, not errors in map_items(). A previous bug incorrectly interpreted an error caught by map_items() as a SyntaxException.
xml_func.py: _date func: Fixed bug where setting defaults of a date part and removing zeros needed to happen after elements were converted to int. Refactored to use conv_items().
Makefiles: Added documentation labels to each section
Makefile: Added documentation labels to each section
input.Makefile: Added documentation labels to each section
xml_dom.py: Fixed conversion of minidom.Node to a string to use toprettyxml() again
Added inputs/SALVIAS-CSV/maps/VegBIEN.* maps
input.Makefile: If a test doesn't yet have accepted output, print its current output
mappings/Makefile: Only clean up a core map (intermediate format->VegBIEN) when it's modified. Note that the .last_cleanup files need to be under version control to prevent cleanups upon running any map command in a new checkout.
xml_func.py: Support dates with year/month/day == 0
input.Makefile: Print informative message if test failure is ignored
input.Makefile: Print path to test output so you can tell which directory a test is running in
input.Makefile: Fixed 2-step tests, which were still using $(root)/map when all bin/map calls were supposed to be routed through $(map)
xml_dom.py: Print simple XML elements on one line
Added inputs/SALVIAS-CSV/maps
input.Makefile: Added import support for CSV and XML inputs (no test support yet)
inputs/SALVIAS-CSV: Added src files
inputs: Removed no longer needed svn:ignore
input.Makefile: Coded svn:ignore directly in Makefile
inputs/Makefile: Added %-add target to add new inputs
input.Makefile: Added svn_props rule for updating svn:ignore of dir and subdirs
vegbien.sql: Made aggregateoccurrence:taxonoccurrence relationship 1:1
Added inputs/CTFS/src, with subset of files that will be in version control
mappings/for_review/DwC-VegBIEN.specimens.csv: Updated
inputs/NYBG: Map via DwC instead of VegX
mappings/DwC-VegBIEN.specimens.csv: Provide schema URLs for namespaces (in the column header)
input.Makefile: Deal with core maps (the main *-VegBIEN map) whose roots are other than organisms
bin/map: In the xml_func.process() for XML outputs, pass an on_error that outputs to ex_tracker.track()
xpath.py: set_id(): Deal with XPaths with an
xml_dom.py: replace_with_text(): Accept Node inputs for cases where e.g. an XML func with an error has been replaced with a comment
xml_func.py: _date: Year defaults to 1900
DwC-VegBIEN.specimens.csv: Output DwC as XML instead of CSV
xpath.py: Allow XPath names to be quoted literals, just like values. Handle : embedded in a name (even without quotes), often indicating a namespace.
Removed no longer needed lib/ERD-wiki.csv
Parser.py: In syntax errors, only print <END> at end of string if error was at end of string
bin/map: Print error message rather than assertion failure for CSV output, which isn't supported yet
input.Makefile: Don't abort tester if via test fails, because CSV output is not supported yet
input.Makefile: Deal properly with more than one format that's being mapped via (e.g. DwC and VegX)
Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
mappings/DwC-VegBIEN.organisms.csv: Fixed syntax error in a mapping
mappings/DwC-VegBIEN.organisms.csv: Sorted
mappings/Makefile: Sort DwC-VegBIEN.organisms.csv
inputs/NYBG/maps/VegBIEN.organisms.csv: Regenerated
mappings/DwC-VegBIEN.organisms.csv: Sort on output column
Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
Makefile: Support recursing into subdir when no target is specified (e.g. make mappings/)
union: Merge rows (and column labels) using maps.merge_mappings()
join: Merge the column labels as well
maps.py: Eliminate duplicates when merging values in the same column
join: Moved mappings-specific merge functionality into maps.merge_mappings()
join: Use merge_rows() from new maps.py
Added new library maps.py for map spreadsheet manipulation
join: Merge comments of input map and join map
join: Report which input mappings are missing a mapping in the join map
inputs/NYBG/maps/VegX.organisms.csv: Added note that the primary key has NULL values in some rows
input.Makefile: Preserve as many intermediate files as possible (make likes to delete intermediates if they aren't marked as .PRECIOUS)
sort: Sort empty strings last so that inputs with no mapping go at the end of the map spreadsheet
VegBIEN-VegBank.csv: Updated for recent table renames
join: For input mappings with no match in the join map, include them in the output map with an empty mapping
input.Makefile: Generalized to handle mapping via any format, not just VegX
input.Makefile: Don't print message to accept output for failing 2-step tests, because they use another test's accepted output
input.Makefile: Don't abort tester if only 2-step test fails, as it's often finicky
xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by date.strftime() (e.g. year out of range due to poor Y2K support in some OS implementations of strftime)
xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by datetime.date() (e.g. month out of range)
vegbien.sql: Added project.reference_id to namespace project names by datasource
input.Makefile: Import all tables at once by default
bin/map: Print "Inserted ... new rows into database" message to stdout rather than stderr so it can be stored in the test case output as a validation check
Accepted initial test output for NYBG/test/import.out.ref
