input.Makefile: Don't try to recreate an input DB if it already exists
Added UArizona DB input
Renaming UArizona to UArizona-CSV because there is also a DB input in bien2_staging.ariz_raw on nimoy
Added UArizona input
env_password: Fixed bug where exit command would not cause it to exit, because pipefail shell option was not set. Moved automatic exiting of the calling script into env_password itself.
map: Exit if password not set
env_password: cc stderr if it's a log file
env_password: Print all messages to /dev/tty so the user sees them even if stderr is redirected to a log file. Exit if password not already set, because e.g. scripts run in the background will not be able to prompt for it.
input.Makefile: Don't have make import call verify, because the user often runs import as a test and will not want the output cluttered with verification information. Also, the full imports for which this was intended are often run asynchronously, so that the user will not see the output anyway.
input.Makefile: Don't abort on verification errors, which are expected during development
SALVIAS tests: Fixed invalid accepted test outputs due to not running `make empty_db` before running tests when using the no-redo optimization shortcut
SALVIAS mappings: Fixed plot key mappings to map the correct values to subplot and parent plot
vegbien.sql: locationevent: Added unique constraint for subplots based on subplot location
SALVIAS-db VegX mapping: Map subplots correctly the way SALVIAS-CSV does
SALVIAS verification: Updated to schema changes
input.Makefile: Fixed syntax error in verify %.ref target (outdated variable name)
input.Makefile: Halt psql commands on first error
vegbien.sql: Removed location.authorlocationcode because it's now stored in locationevent as an author-specific setting
vegbien.sql: locationevent: Redid unique constraints to avoid applying authorlocationcode-only duplicate elimination to subplots
SALVIAS mappings: Map SiteCode/plot_code to locationevent.authorlocationcode because locationevent is now the place to store author-specific plot information
SALVIAS mappings: Fixed PlotID mapping to go to locationevent.sourceaccessioncode
VegBIEN: Renamed locationevent.authoreventcode to authorlocationcode to reflect that datasources usually use an author-defined code for a plot rather than a plot event
vegbien.sql: locationevent: Redid unique constraints to handle datasources that treat the authoreventcode as an authorlocationcode. Eventually, authoreventcode will be renamed to authorlocationcode.
vegbien.sql: locationevent: Redid unique constraints to work properly for all fully-specified combinations of keys
VegBIEN mappings: Mapped datasource name to new project.datasource. Fixes project duplicate elimination.
vegbien.sql: Renamed project.reference_id to datasource_id and pointed it to party, to match locationevent, etc.
VegBIEN mappings: Mapped current lat/long to centerlat/long as well so location duplicate elimination will work properly
xpath.py: Added support for common subpath after split path's {}
sql.py: put(): When encountering a DuplicateKeyException, use dict_subset_right_join to fill in explicit NULL values for columns which don't have data. This causes the database to use the UNIQUE constraint's index to look up the record, instead of relying on individual column indexes for the columns that did have data, which may or may not be available.
util.py: Added DefaultDict to wrap collections.defaultdict with a simple value passed in the constructor, defaulting to None. Added dict_subset_right_join() to fill in None for subset keys that don't exist.
vegbien.sql: Added method and plotmethod UNIQUE indexes
vegbien.ERD.mwb: Removed embargo table from ERD because its functionality is provided in location.confidentialitystatus,confidentialityreason
Regenerated vegbien.ERD exports
vegbien.sql: Moved locationevent method fields to plotmethod and method. Added comments to method/plotmethod fields, as provided by Michael Lee.
VegX-VegBIEN mapping: Mapped locationevent.methodnarrative to new plotmethod table
VegX-VegBIEN mapping: Map sourceaccessioncode and voucher (catalognumber_dwc) to correct place. SALVIAS mappings: Map SourceVoucher as an alternative to coll_number.
vegbien.sql: Removed VegBank-internal tables (including user account tables) because they do not belong in the ecological database. Any web interface should store user account information, cached queries, etc. in a separate interface-specific database.
VegX-VegBIEN mapping: Mapped stem tags to new stemtag table
vegbien.sql: Renamed planttag to stemtag and made it a child of stemobservation. Removed trait table from ERD because it's not used for the purpose we want to use traits for.
vegbien.sql: Removed no longer used location.reference_id. Datasource scoping is now done on locationevent instead, so that locations can be shared across datasources that refer to the same plot or point.
VegX-VegBIEN mapping: Map datasource name (/_ignore/inLabel) to new locationevent.datasource instead of location.reference
vegbien.sql: Added locationevent.datasource_id
vegbien.sql: locationevent: Removed VegBank-internal interp_* fields
VegBIEN: Renamed specimenreplicate.reference_id to datasource_id and pointed it to party instead of reference, since party is better optimized for storing names
DwC mappings: Mapped datasource name to specimenreplicate.reference instead of location.reference
DwC mappings: Mapped specimen description via fieldNotes instead of custom bien.specimenDescription field
VegBIEN: Moved specimenreplicate.verbatimcollectorname to taxonoccurrence since it can also apply to aggregateoccurrences. Removed no longer needed taxonoccurrence fields which are now in taxondetermination.
SALVIAS mappings: Mapped habit to growthForm (user-defined field) instead of habit
DwC-VegBIEN mapping: Convert latitude/longitude values of exactly zero to NULL
xml_func.py: Added _nullIf
util.py: Fixed cast() to not cast a subclass to a superclass (which doesn't make sense in a dynamically-typed language). Added none_if().
util.py: Removed locale import since it's no longer used by util
NYBG-DwC mappings: Map Vegetation to habitat (merged with Habitat). DwC-VegBIEN mapping: Removed remaining mappings to plantobservation.
DwC-VegBIEN mapping: Added datasource name to location.reference using /_ignore/inLabel
profiling.py: Support Python before 2.7 by using new dates.total_seconds(). Also use dates.now() to ensure datetimes always have a timezone.
dates.py: Fixed timestamp() to deal with microseconds correctly by adding them after time.mktime()
dates.py: Deal properly with different timezones by using external dateutil package. Added total_seconds() to replace datetime.timedelta.total_seconds() on Python before 2.7.
vegbien.sql: *method tables: Added table comments
VegX-VegBIEN mapping: Reordered 2-step-only mappings that use /_ignore/inLabel so they run at the same time as other mappings that set the field that uses /_ignore/inLabel. This fixes almost all of the failing 2-step tests.
vegbien.sql: method: Added lengthunits field
vegbien.sql: Changed types of numerical plotmethod fields to double precision
vegbien.sql: method, plotmethod: Added comments to fields
vegbien.ERD.mwb: Adjusted lines
vegbien.sql: Added plotmethod. locationevent points to plotmethod instead of directly to method
vegbien.sql: Point to covermethod from method instead of locationevent
vegbien.sql: Removed no longer needed sizeclass table (whose fields are now in method)
vegbien.sql: Replaced stratumtype, stratummethod with method
vegbien.sql: Attach method to aggregateoccurrence instead of taxonoccurrence
vegbien.sql: Removed methodtrait* tables and added first-class method attributes as first-class fields of method. Removed *method tables from the ERD that will be replaced by method.
vegbien.sql: Removed location.dsgpoly because it is now locationdetermination.footprintgeometry_dwc
VegBIEN mappings: Remap to new locationdetermination fields
VegBIEN: Renamed location.reallatitude,reallongitude to centerlatitude,centerlongitude to reflect that it's now a value calculated from the centroid of the current locationdetermination
vegbien.sql: locationdetermination: Reordered fields
vegbien.sql: locationdetermination.coordsaccuracy: Added comment with units
vegbien.sql: locationdetermination: Added determination status columns from taxondetermination
vegbien.sql: locationdetermination: Added coordinates-related fields
VegX-VegBIEN mapping: Include the datasource name (now provided by map in /_ignore/inLabel) in the appropriate places in both VegX and VegBIEN
bin/map: Removed metadata values feature since the syntax used was causing problems with mappings starting with a ":", and metadata can instead be stored as attributes of the primary key's mapping
xml_dom.py: Fixed bug in parent() where it didn't account for NodeParentIter's first element returned being the current node, not its parent. Refactored parent() to use parentNode directly, and NodeParentIter to use parent(), instead of the other way around.
xml_dom.py: Fixed bug in parent() where incorrect variable name was used
VegX-VegBIEN mapping: Use the input data source's label (e.g. SALVIAS) everywhere a reference is needed
bin/map: Store the input data source's label (e.g. SALVIAS) in the output XML tree for use by references in the mappings
xpath.py: get(): Fixed bug where it would try to create a node named . or .. if . or .. didn't have matching attributes. Now it will just reuse the current or parent node, but create any needed attrs if create is True.
util.py: Added list_eq_is() to compare two lists using is
xpath.py: Don't allow rooted attributes (doesn't make sense), in case someone tries to do elem[/rooted_attr]
bin/map: Moved root.clear() into separate function prep_root() that can be called whenever needed
xpath.py: Added get() support for references (different from pointers) to dynamically set the value of an attribute
util.py: Added list_get()
util.py: Added is_list()
bin/map: Use var doc0_root for quick reference to doc0's root
xpath.py: get(): Go to root when empty element is encountered at the beginning of an XPath. Added allow_rooted parameter to turn off this functionality when XPaths with a leading slash should not be considered rooted.
xpath.py: Don't consider a path starting with "." to be rooted. Do this by not automatically translating an empty path name to ".".
xpath.py: Added is_rooted()
xpath.py: Added elem_is_empty()
xpath.py: Added documentation labels to each section
xpath.py: Added support for getting the parent node when encountering ".."
xml_dom.py: Added parent() to get parent node without recursing past the root node to the document object. Documented that NodeParentIter incorporates this sanity check.
xpath.py: get(): Renamed parent to root to better reflect that it's the starting point for the search. Calling it parent will later be confusing when we want to get the parent node using "..".
