Generate MySQL DDL schemas/vegbien.my.sql from vegbien.sql

Added repl spreadsheet to map PostgreSQL DDL to MySQL

Ignore src subdirectories in inputs

VegBIEN: Use stem instead of individualplant to store diameter/dbh

vegbien.sql: Removed no longer needed taxondetermination.plantname_id

VegBIEN: Made taxondetermination.plantconcept_id required and plantname_id optional

vegbien.sql: Added specimen.taxonoccurrence_id fkey

vegbien.sql: Removed no longer needed taxondetermination.individualplant_id, specimen_id fields

VegBIEN: Renamed plot to location to reflect that a location can also be used for specimens

vegbien.sql: Added individualplant.stemcount

vegbien.sql: Added accessioncode fields to existing, modified tables

vegbien.sql: Added accessioncode fields to new tables

vegbien.sql: Removed no longer needed individualplant.taxonarea

VegBIEN: Moved taxondetermination.sourceaccessionnumber and museum_id to specimen

vegbien.sql: Split taxondetermination UNIQUE CONSTRAINT into one constraint for each main key (taxonoccurrence_id, specimen_id, individualplant_id) to deal with Postgres treating a row with NULL in any of those columns as distinct. Added CHECK constraint to require one of taxonoccurrence_id, specimen_id, individualplant_id to be NOT NULL.

VegBIEN: Renamed museumaccessionnumber to sourceaccessionnumber

VegBIEN: Renamed authorcode to sourceid

vegbien.sql: Updated sizeclass to use ranges for parameters and remove taxonarea

input.Makefile: Fixed verify by making %.out: % rule use %.make instead

input.Makefile: Moved verifications into one set of verify.* files

input.Makefile: All tests are run in the same transaction, with output to test.out

bin/map: Added support for processing multiple map_paths at once in the same transaction

SALVIAS organisms mapping: Removed redundant PlotCode mapping because the association to plotevent is done with PlotID

bin/map: Started adding support for multiple map_paths, all run in the same db transaction

Filled in SALVIAS stems mappings. make test2's stems test fails because we don't wrap the import transaction around all the tests so that organisms are already in the database when stems are imported.

mappings/Makefile: Corrected stemobservation to stem

Use baseDistance/complexUserDefined/method to distinguish between different dbhs in VegX

VegX mappings: Use baseDistance/value instead of baseDistance so we can later use complexUserDefined to distinguish between different types of dbh

VegX-VegBIEN.organisms.csv: Added initial stems mappings

mappings/Makefile: Updated VegX-VegBIEN.stems.csv input/output roots

ch_root: Deal with roots followed by something other than \b

input.Makefile: Remove *.out in new verify and test subdirs

Moved inputs files into subdirs for maps, verify, and test operations

vegbien.sql: Removed taxondetermination.collector_id because that's now in specimen

VegBIEN: Map to specimen.collector_id instead of taxondetermination

vegbien.sql: Added specimen.collector_id column

vegbien.sql: Changed taxondetermination.stem_id to individualplant_id

vegbien.sql: Removed taxondetermination columns that have been moved to specimen

VegBIEN: Merged specimen.collectionnumber and catalognumber_dwc fields. Removed NOT NULL constraints on several specimen columns because not all specimens have all identifiers. Mapped to new specimen fields.

VegX-VegBIEN mapping: Map collectiondate to specimen table

vegbien.sql: Added specimen.collectionnumber and collectiondate fields

Updated README.TXT to reflect new make targets

vegbien.sql: Added sizeclass columns

VegBIEN: Removed stem* prefix from stem measurements because it's redundant and because these same field names were reused in non-stem tables

VegBIEN: Renamed stemobservation to stem

vegbien.sql: Added comments to renamed tables specifying VegBank equivalent

vegbien.sql: Added comments to new tables

vegbien.sql: Removed r# prefix from foreign key constraints

vegbien.sql: Added plot.namedplace_id for theplots which are identified by a namedplace rather than lat/long or name

vegbien.sql: Made plot.authorcode optional as not all plots have names

vegbien.sql: Added _dwc to specimen columns that are from DwC

vegbien.sql: Added UNIQUE constraint for specimen table. Made catalognumber required so that all specimens can be uniquely identified.

vegbien.sql: Made specimen.individualplant_id optional because specimens are not required to have a corresponding plot event (which all individualplants must have)

VegBIEN-VegBank.csv: Updated to reflect renames

vegbien.sql: Renamed specimen.code to collectioncode to match DwC term

vegbien.sql: Added sizeclass table

vegbien.sql: Renamed collection to specimen to reflect that it's for a single collection specimen

VegBIEN: Renamed collectiveobservation to aggregateoccurrence to reflect VegX usage of aggregate and the new taxonoccurrence table

VegBIEN: Renamed stemobservation.stemcode to authorcode

vegbien.sql: Renamed individualplant.code to authorcode

sql.py: Reordered get() args to match put()

db_xml.py: Use new sql.put() which only eliminates duplicates based on database UNIQUE constraints

sql.py: Added new function put() and changed get() to use it

sql.py: Fixed raising of NullValueException to pass array of columns rather than single column

vegbien.sql: Added UNIQUE constraints to tables used in the mapping so that duplicates are eliminated by the database

sql.py: ExceptionWithColumn now stores multiple columns so that they can all be used in DuplicateKeyExceptions

util.py: Fixed dict_subset() to work correctly when a key does not exist

util.py: Added dict_subset()

sql.py: Use constraint_cols() to determine DuplicateKeyException column in try_insert()

sql.py: Added constraint_cols() to get columns of a constraint for use in determining the columns for a DuplicateKeyException

db_xml.py: Map empty fields to None instead of ignoring them, to ensure that an existing row with a value in that field is not used in the duplicate elimination

bin/map: Use db_xml.put()'s new on_error handler

db_xml.py: put(): Added on_error handler. Reordered arguments according to how often they are likely to be used.

bin/map: Print # input rows processed

VegX-VegBIEN mapping: Merge different sources of plot elevation

vegbien.sql: Added individualplant.code field

xml_func.py: Expand XML funcs in XML func args

xml_dom.py: In NodeTextEntryIter, return non-text children as-is instead of ignoring them so that XML functions can use them

xml_dom.py: Improved readability by separating into sections

db_xml.py: Ignore only fields that are actually empty (rather than just not having element children), as a field containing anything other than a comment should be flagged as invalid input

xml_dom.py: Changed is_empty() to consider nodes with only comments to be empty

bin/map: Add input row to exceptions

db_xml.py: Ignore empty fields. Fixes bug where a field that's not text but has no Element children will be treated as a table, and an error is triggered because the corresponding table does not exist in the database.

xml_dom.py: Added has_elems() to check if node has Element children

bin/map: Use xml_func.process()'s new on_error error handler. Add row to database errors and xml_func.SyntaxExceptions.

xml_func.py: Changed formatting of comment that node with syntax errors is replaced with so that it starts on its own line

xml_func.py: Use on_error exception handler to allow caller to print exceptions instead of raising them

db_xml.py: Add node to any database exceptions generated in put()

exc.py: Added raise_() wrapper for raise statement

Ignore .~* (lock files generated by OpenOffice.org) in inputs

inputs/SALVIAS: Added stems map

bin/map: Handle empty rows for db output

mappings: Use stemParent instead of stemOrganism to identify stems so that it's clear that the stemParent is an individualOrganismObservation, not an individualOrganism

input.Makefile: Include VegX maps not in tablesSort

mappings/Makefile: Generate initial VegX-VegBIEN.stems.csv

mappings/Makefile: Cleaned up in preparation for adding rule to generate VegX-VegBIEN.stems.csv

bin/map: Continue with database insertion even if a row had XML func syntax errors, because these errors have been fixed by removing the offending function node

xml_func.py: Remove function nodes with syntax errors to prevent them from causing DatabaseErrors later

xml_dom.py: Added remove() function to remove a node

bin/map: Track Exceptions with ExTracker