sql.py: Added index_cols() to get cols used by an index (similar to constraint_cols())
vegbien.sql: Fixed duplicate elimination for party to use a UNIQUE index with COALESCE for nullable fields
sql.py: Fixed bug in try_insert() where DuplicateKeyException was passed only cols0 instead of cols array
vegbien.ERD.mwb: Fixed lines
Added get_errors to select just the error messages from `map` output
Added profile_stats to analyze a profiling statistics file
bin/map: Added profile_to option which turns on profiling to the specified file
bin/map: Added "if name == '__main__': main()" idiom so file can be included as well as run. This will be useful for profiling.
dates.py: Fixed strftime() to pad years and days with leading zeros as datetime.strftime() does
dates.py: Work around strftime() bug that can't deal with 2/29 on a leap year
xml_func.py: Added FormatException for SyntaxExceptions generated by strftime() (which are often Python bugs)
Added schemas/vegbank.revised.sql. Initial version has all "character varying" types replaced with text.
vegbien.sql: Replaced all "character varying" types with text, removing the length limits. Note that in PostgreSQL, text and "character varying" are stored the same way internally, so this does not affect performance or indexes.
xml_dom.py: Added documentation labels to each section
xml_dom.py: Fixed bug in NodeTextEntryIter where an entry containing an element instead of a text node would be returned as the whole entry, instead of the value of the entry
bin/map: Added support for starting import at a specific row. Refactored row-processing code with and without a map to use a common process_rows() function (with the previous process_rows() being renamed to map_rows()).
bin/map: Use new util.cast()
util.py: Added cast() to cast a value while passing None through
bin/map: Print row # of rows with errors
sql.py: Fixed error in pkey() where recover was not passed as a named parameter to run_query()
sql.py: Added documentation labels to each section
db_xml.py: Used new sql.py recover functionality
sql.py: Added ability to recover from database errors so you don't get the error "InternalError: current transaction is aborted, commands ignored until end of transaction block"
vegbien.sql: Removed taxonoccurrence.taxoninferencearea because it's duplicated in aggregateoccurrence.inferencearea
bin/map: Highlight the "input row" and "output row:" labels in error messages
xml_func.py: Highlight nodes that were commented out because of errors
exc.py: Print exceptions with the first line highlighted in red
term.py: Added emph() and error()
vegbien.ERD.mwb: Added note, notelink, and revision
vegbien.ERD.mwb: Added embargo to diagram
Regenerated vegbien.ERD exports
vegbien.ERD.mwb: Fixed lines. Added "Core subset" and "Other tables" labels.
xml_func.py: Changed _date func to use new dates.strftime(), which can handle years before 1900
Added dates.py to handle date/time manipulation, such as fixing Python's broken strftime() that can't handle years before 1900
Regenerated mappings/for_review/DwC-VegBIEN.specimens.csv
vegbien.ERD.mwb: Added reference and party tables
filter_ERD.csv: Remove fkeys to heavily-linked tables (reference, party)
Added to_do/milestones.doc
Renamed milestones.doc to timeline.doc
Added schemas/filter_ERD.csv and use it when generating vegbien.my.sql
vegbien.ERD.mwb: Added cover* to main diagram
vegbien.ERD.mwb: Started adding additional tables "below the fold" on the 2nd page
vegbien.ERD.mwb: Moved legend to top left to make room for more misc tables. Organized legend by location on diagram.
vegbien.ERD.mwb: Added soilobs table
vegbien.ERD.mwb: Added userdefined tables. Fixed lines.
vegbien.ERD.mwb: Changed location color to match VegBank ERD
vegbien.ERD.mwb: Added trait to diagram
vegbien.ERD.mwb: Added plantstatus to diagram. Added margins around diagram.
Added milestones.doc
DwC mappings: Fixed syntax of _date XML funcs to not wrap dates twice in a _date func
xml_func.py: Fixed bug in SyntaxException constructor where the cause was not passed to ExceptionWithCause
xml_dom.py: Override Node.__repr__ and Element.__repr__ to make sure self.toprettyxml() is used in all cases where a Node is converted to a string
util.py: Don't print type name in ConstraintError message because sometimes an entry tuple (e.g. from a dict) is checked, and then the type isn't useful
xml_func.py: Use conv_items(str, ...) in XML funcs that require strings, but don't otherwise use conv_items()
xml_func.py: Use xml_dom.TextEntryOnlyIter to check inputs for XML funcs that only handle strings, not whole elements
xml_dom.py: Added TextEntryOnlyIter to check that values returned by NodeTextEntryIter are strings from text nodes
util.py: Added is_str() and CheckedIter
input.Makefile: Log each import to a new log file named according to the current time
xml_func.py: SyntaxException: Use exc.ExceptionWithCause
exc.py: Added str_() to convert an Exception to a string. Use str_() in print_ex().
xml_func.py: conv_items(): Only trap type conversion errors, not errors in map_items(). A previous bug incorrectly interpreted an error caught by map_items() as a SyntaxException.
xml_func.py: _date func: Fixed bug where setting defaults of a date part and removing zeros needed to happen after elements were converted to int. Refactored to use conv_items().
Makefiles: Added documentation labels to each section
Makefile: Added documentation labels to each section
input.Makefile: Added documentation labels to each section
xml_dom.py: Fixed conversion of minidom.Node to a string to use toprettyxml() again
Added inputs/SALVIAS-CSV/maps/VegBIEN.* maps
input.Makefile: If a test doesn't yet have accepted output, print its current output
mappings/Makefile: Only clean up a core map (intermediate format->VegBIEN) when it's modified. Note that the .last_cleanup files need to be under version control to prevent cleanups upon running any map command in a new checkout.
xml_func.py: Support dates with year/month/day == 0
input.Makefile: Print informative message if test failure is ignored
input.Makefile: Print path to test output so you can tell which directory a test is running in
input.Makefile: Fixed 2-step tests, which were still using $(root)/map when all bin/map calls were supposed to be routed through $(map)
xml_dom.py: Print simple XML elements on one line
Added inputs/SALVIAS-CSV/maps
input.Makefile: Added import support for CSV and XML inputs (no test support yet)
inputs/SALVIAS-CSV: Added src files
inputs: Removed no longer needed svn:ignore
input.Makefile: Coded svn:ignore directly in Makefile
inputs/Makefile: Added %-add target to add new inputs
input.Makefile: Added svn_props rule for updating svn:ignore of dir and subdirs
vegbien.sql: Made aggregateoccurrence:taxonoccurrence relationship 1:1
Added inputs/CTFS/src, with subset of files that will be in version control
mappings/for_review/DwC-VegBIEN.specimens.csv: Updated
inputs/NYBG: Map via DwC instead of VegX
mappings/DwC-VegBIEN.specimens.csv: Provide schema URLs for namespaces (in the column header)
input.Makefile: Deal with core maps (the main *-VegBIEN map) whose roots are other than organisms
bin/map: In the xml_func.process() for XML outputs, pass an on_error that outputs to ex_tracker.track()
xpath.py: set_id(): Deal with XPaths with an
xml_dom.py: replace_with_text(): Accept Node inputs for cases where e.g. an XML func with an error has been replaced with a comment
xml_func.py: _date: Year defaults to 1900
DwC-VegBIEN.specimens.csv: Output DwC as XML instead of CSV
xpath.py: Allow XPath names to be quoted literals, just like values. Handle : embedded in a name (even without quotes), often indicating a namespace.
