vegbien.sql: Removed no longer used location.reference_id. Datasource scoping is now done on locationevent instead, so that locations can be shared across datasources that refer to the same plot or point.

VegX-VegBIEN mapping: Map datasource name (/_ignore/inLabel) to new locationevent.datasource instead of location.reference

vegbien.sql: Added locationevent.datasource_id

vegbien.sql: locationevent: Removed VegBank-internal interp_* fields

VegBIEN: Renamed specimenreplicate.reference_id to datasource_id and pointed it to party instead of reference, since party is better optimized for storing names

DwC mappings: Mapped datasource name to specimenreplicate.reference instead of location.reference

DwC mappings: Mapped specimen description via fieldNotes instead of custom bien.specimenDescription field

VegBIEN: Moved specimenreplicate.verbatimcollectorname to taxonoccurrence since it can also apply to aggregateoccurrences. Removed no longer needed taxonoccurrence fields which are now in taxondetermination.

SALVIAS mappings: Mapped habit to growthForm (user-defined field) instead of habit

DwC-VegBIEN mapping: Convert latitude/longitude values of exactly zero to NULL

xml_func.py: Added _nullIf

util.py: Fixed cast() to not cast a subclass to a superclass (which doesn't make sense in a dynamically-typed language). Added none_if().

util.py: Removed locale import since it's no longer used by util

NYBG-DwC mappings: Map Vegetation to habitat (merged with Habitat). DwC-VegBIEN mapping: Removed remaining mappings to plantobservation.

DwC-VegBIEN mapping: Added datasource name to location.reference using /_ignore/inLabel

profiling.py: Support Python before 2.7 by using new dates.total_seconds(). Also use dates.now() to ensure datetimes always have a timezone.

dates.py: Fixed timestamp() to deal with microseconds correctly by adding them after time.mktime()

dates.py: Deal properly with different timezones by using external dateutil package. Added total_seconds() to replace datetime.timedelta.total_seconds() on Python before 2.7.

vegbien.sql: *method tables: Added table comments

VegX-VegBIEN mapping: Reordered 2-step-only mappings that use /_ignore/inLabel so they run at the same time as other mappings that set the field that uses /_ignore/inLabel. This fixes almost all of the failing 2-step tests.

Regenerated vegbien.ERD exports

vegbien.sql: method: Added lengthunits field

vegbien.sql: Changed types of numerical plotmethod fields to double precision

vegbien.sql: method, plotmethod: Added comments to fields

vegbien.ERD.mwb: Adjusted lines

vegbien.sql: Added plotmethod. locationevent points to plotmethod instead of directly to method

vegbien.sql: Point to covermethod from method instead of locationevent

vegbien.sql: Removed no longer needed sizeclass table (whose fields are now in method)

vegbien.sql: Replaced stratumtype, stratummethod with method

vegbien.sql: Attach method to aggregateoccurrence instead of taxonoccurrence

vegbien.sql: Removed methodtrait* tables and added first-class method attributes as first-class fields of method. Removed *method tables from the ERD that will be replaced by method.

vegbien.sql: Removed location.dsgpoly because it is now locationdetermination.footprintgeometry_dwc

VegBIEN mappings: Remap to new locationdetermination fields

VegBIEN: Renamed location.reallatitude,reallongitude to centerlatitude,centerlongitude to reflect that it's now a value calculated from the centroid of the current locationdetermination

vegbien.sql: locationdetermination: Reordered fields

vegbien.sql: locationdetermination.coordsaccuracy: Added comment with units

vegbien.sql: locationdetermination: Added determination status columns from taxondetermination

vegbien.sql: locationdetermination: Added coordinates-related fields

VegX-VegBIEN mapping: Include the datasource name (now provided by map in /_ignore/inLabel) in the appropriate places in both VegX and VegBIEN

bin/map: Removed metadata values feature since the syntax used was causing problems with mappings starting with a ":", and metadata can instead be stored as attributes of the primary key's mapping

xml_dom.py: Fixed bug in parent() where it didn't account for NodeParentIter's first element returned being the current node, not its parent. Refactored parent() to use parentNode directly, and NodeParentIter to use parent(), instead of the other way around.

xml_dom.py: Fixed bug in parent() where incorrect variable name was used

VegX-VegBIEN mapping: Use the input data source's label (e.g. SALVIAS) everywhere a reference is needed

bin/map: Store the input data source's label (e.g. SALVIAS) in the output XML tree for use by references in the mappings

xpath.py: get(): Fixed bug where it would try to create a node named . or .. if . or .. didn't have matching attributes. Now it will just reuse the current or parent node, but create any needed attrs if create is True.

util.py: Added list_eq_is() to compare two lists using is

xpath.py: Don't allow rooted attributes (doesn't make sense), in case someone tries to do elem[/rooted_attr]

bin/map: Moved root.clear() into separate function prep_root() that can be called whenever needed

xpath.py: Added get() support for references (different from pointers) to dynamically set the value of an attribute

util.py: Added list_get()

util.py: Added is_list()

bin/map: Use var doc0_root for quick reference to doc0's root

xpath.py: get(): Go to root when empty element is encountered at the beginning of an XPath. Added allow_rooted parameter to turn off this functionality when XPaths with a leading slash should not be considered rooted.

xpath.py: Don't consider a path starting with "." to be rooted. Do this by not automatically translating an empty path name to ".".

xpath.py: Added is_rooted()

xpath.py: Added elem_is_empty()

xpath.py: Added documentation labels to each section

xpath.py: Added support for getting the parent node when encountering ".."

xml_dom.py: Added parent() to get parent node without recursing past the root node to the document object. Documented that NodeParentIter incorporates this sanity check.

xpath.py: get(): Renamed parent to root to better reflect that it's the starting point for the search. Calling it parent will later be confusing when we want to get the parent node using "..".

xpath.py: Added parser support for attribute values that are references to another part of the XML tree

xml_func.py: Fixed module description comment to reflect that not all XML funcs generate text

xml_func.py: Refactored to add funcs to the module funcs variable as they are defined. Renamed defined functions to the name of the corresponding XML function.

xml_func.py: Added _ignore func to "comment out" an XML subtree

input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported

input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported

input.Makefile: Don't run tests in verbose mode because the run time stats, etc. are not relevant

bin/map: Only print error/run time stats in verbose mode. input.Makefile: Run import in verbose mode so that error/run time stats are still printed.

Moved value to string conversion functions infrom util.py to new module format.py

exc.py, profiling.py: Use util.int2str() to print # iters with thousands separators

util.py: Added int2str()

bin/map: Document that the exit status is the # of errors in the import, up to the maximum exit status

exc.py: Generalize ExTracker to not just print the # of errors at exit. Instead, provide an exit() method that the ExTracker creator can call at exit to set the exit status to the # of errors. This fixes the Python bug where a benign error message was printed if SystemExit was raised in an atexit function.

bin/map: Set ExPercentTracker's iter_text. Start ExPercentTracker after input processing, because errors in command line options should just end the program and don't need to be tracked.

exc.py: ExPercentTracker: Added ability to set custom iter_text, similar to ItersProfiler

bin/map: Use profiling.ItersProfiler. Refactored input row count calculation to have each function aggregate and return the row count, and then display the row count and statistics that depend on it at the end of the program.

Added profiling.py to time operations and provide the user with statistical information

util.py: Added basic to_si() to add SI prefix to value

util.py: Added format_str() to use locale-specific formatting settings, including thousands separator. Use it in to_percent().

bin/map: Use new ExPercentTracker to print error rate (% of # rows) when program exits

exc.py: Added ExPercentTracker to track errors as % of iterations

util.py: Added to_percent()

exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format

exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format

inputs/SALVIAS/maps/VegX.organisms.csv: Mapped OrigSpecies and OrigGenus combined to new plantlevel Binomial

xpath.py: Fixed bug where value of XPath (used for copying to other branches) is retrieved after first XPath element is popped rather than before, which can sometimes leave an empty XPath for value() to run on

mappings/DwC-VegBIEN.specimens.csv: Fixed bien.vegetation mapping to point to commconcept->commname. Fixed bien.substrate mapping to point to locationevent.landscapenarrative.

inputs/NYBG/maps/DwC.specimens.csv: Mapped CoordinatePrecision using _noCV

xml_func.py: Added _noCV func to check that non-ratio-scale data does not contain CV values

mappings/DwC-VegBIEN.specimens.csv: Fixed locality fields mapping to go to location.locationnarrative

input.Makefile: For all input types, including DB, import each table in a separate map invocation

xml_func.py: _range: Treat a None from or to value as an unknown (a la SQL NULL) and return None instead of raising a SyntaxException

xml_dom.py: NodeTextEntryIter: Convert empty entries (including entries containing error comments) to None

xml_dom.py: replace(): Added support for new node that's None (deletes existing node)

xml_func.py: Put SyntaxException's cause on same line as error message so that the whole error is treated as distinct by error_stats

Added errors_filter_before and errors_filter_after to prepare `map` error messages for easy filtering and then restore line breaks

error_stats: Fixed to work on Mac

error_stats: Simplified to use uniq --count option

input.Makefile: Print error message if no input file found (for file input type). This fixes a bug where map would just take input from stdin when no input file redirect or input DB env vars were specified.