mappings/verify.specimens.sql: Use ORDER BY ... NULLS FIRST to match MySQL
input.Makefile: verify: Time the verification since it can take a long time
specimens verification: Added duplicate catalog numbers test
map: On nimoy, use bien2_staging unless otherwise specified
specimens verification: Added # counties test
specimens verification: Added collection codes and # catalog numbers tests
inputs/SALVIAS/maps/VegX.organisms.csv: Mapped custom Habit values not listed in the SALVIAS data dictionary
strings.py: Added unicode_reader for later use in handling Unicode characters in map spreadsheets
xpath.py: Removed unnecessary copy.deepcopy()'s and instead changed set_value() and set_id() to make copies of any elements they change. This should result in up to a 17% speed increase in the import, because deepcopy() was taking a lot of time. Added documentation to set_value() and set_id() that caller must make a shallow copy of the path to prevent modifications from propagating to other copies of the path. (Previously, a deep copy was needed, but there was no comment specifying this.)
mappings/VegX-VegBIEN.organisms.csv: Removed unneeded lookahead assertions from stemtag mappings. They relied on a bug ("feature"?) in the XPath engine that made the value of the lookahead assertion's path the same as the value of the main path, even though the value is set after the path is parsed.
xml_func.py: _date: For year/month/day dates, require the year (it would not make sense to default to a particular year)
inputs/UArizona: Added test outputs
mappings/DwC1-DwC2.specimens.csv: Fixed to allow datasource to define custom date mappings that don't pass through the default date mapping
input.Makefile: Generate maps/src.join.*.csv, which can be used to determine which DwC fields for a particular dataset do not yet have a join mapping to VegBIEN
Makefile: Fixed subdir remake target to work for nested subdirs as well
inputs/UArizona: Renamed maps/src.csv to maps/src.specimens.csv because there will be one for each input table
inputs/UArizona: Added maps/src.csv with columns from source data
Added autogen mappings/DwC-VegBIEN.specimens.no_empty.csv, which will be used for determining which DwC fields for a particular dataset do not yet have a join mapping to VegBIEN
Added remove_empty to remove empty mappings in a map spreadsheet
join: Don't raise "No join mapping" error for empty mappings because you only want the error for empty mappings for your particular dataset, which requires more information (namely, the subset of the mappings used by your dataset, some of which will not be in the mappings if standard fields have been subtracted out)
join: Fixed bug in "No join mapping" error generation where rows with no existing comments column would cause an IndexError
util.py: Added list_set() and list_setdefault()
inputs/UArizona/maps/DwC.specimens.csv: Merge FieldNotes and Remarks
inputs/UArizona/maps/DwC.specimens.csv: Finished mappings
inputs/UArizona/maps/DwC.specimens.csv: Removed fields already present in DwC mappings
inputs/NYBG-CSV/maps/DwC.specimens.csv: Removed mappings already present in case-insensitive DwC2 mapping
inputs/NYBG/maps/DwC.specimens.csv: Removed mappings already present in case-insensitive DwC2 mapping
mappings/DwC1-DwC2.specimens.csv: Removed fields already present in DwC2.ci-VegBIEN.specimens.csv
Makefiles: Moved remake into main Makefile. Fixed remake to run `make all` in a new make so that cache of existing files is reset. Have main remake run clean and then all instead of forwarding remake to subdirs, so that everything is cleaned before everything is remade.
input.Makefile: maps: maps/$(via).%.full.csv: Fixed bug where $(selfMap) would be ignored if it had not yet been made
mappings/Makefile: Reorganized into DwC and VegX sections
Added autogenerated mappings/DwC2.ci-VegBIEN.specimens.csv. Use it to include DwC2 fields with first letter uppercased in the full DwC mapping, so that datasources that use DwC2 terms with a different case can still use the DwC2 mapping.
inputs/UArizona/maps/DwC.specimens.csv: Mapped CollectedDate to eventDate/_alt/2 even though it's not used because other datasources might copy these mappings and want it already filled in
Added ucase_first to uppercase the first character of columns in a spreadsheet
Added inputs/UArizona/maps/DwC.specimens.csv autogen maps
inputs/UArizona/maps/DwC.specimens.csv: Mapped more fields
mappings/DwC1-DwC2.specimens.csv: Remove date -> date/_alt/2 mappings because they prevent the original DwC2 date field from being mapped to without an extra /_alt/2 appended
xml_func.py: Use new dates.strtotime(). When component date parts specified, year defaults to dates.epoch.year.
dates.py: Added strtotime() to wrap dateutil.parser.parse() with default defaulting to epoch, so that e.g. months with day missing default to day 1 instead of the current day of the month
mappings/DwC1-DwC2.specimens.csv: Map eventDate,dateIdentified using /_alt/2 and year/month/day using /_alt/1 so that inputs with both a date and date parts will select between the two
input.Makefile: Added comment that self map must be made first if it's needed for maps/$(via).%.full.csv
Makefiles: Use .SECONDARY with no prerequisites instead of setting a .PRECIOUS for each intermediate, to simplify turning off automatic deletion of intermediate files
inputs/UArizona: Added initial maps/DwC.specimens.csv
DwC mappings: Map datasource name via institutionID to avoid conflicting with existing institutionCode fields that many DwC data sources have
input.Makefile: Don't profile by default because it appears to slow things down significantly on long imports
Added inputs/UArizona/maps
Makefile: python-Linux: Added python-profiler
specimens verification: Added # binomials test
vegbien.sql: specimenreplicate: Removed specimenreplicate_unique_collectionnumber index because the collectionnumber (NYBG FieldNumber) is not always unique within a collector, even though it should be. Changed specimenreplicate_unique_catalognumber to only operate on rows with no sourceaccessioncode (of which there are 8 in NYBG).
mappings/verify.specimens.sql: # species test: Fixed to join separately on taxondeterminations for genus and species. # genera test: Removed no longer needed join on party.
vegbien.sql: specimenreplicate: Added fki index on taxonoccurrence_id
vegbien.sql: plantname: Added index on rank to speed up specimens verifications, where the query planner insists on joining from plantname to specimenreplicate instead of the other way around (which takes much longer without the index)
mappings/verify.*: Use nested SELECT instead of JOIN on party to get datasource_id, so that party will not be joined on after other joins have already occurred (which slows things down)
vegbien.sql: party: Changed party_unique_name to ignore NULL values and the organizationname (a first(+middle)+last name is considered unique)
vegbien.sql: party: Added party_unique_organizationname constraint
Specimens verification: Added # genera and # species
input.Makefile: verify: Create target dir if it doesn't exist
inputs/NYBG: Added verify/specimens.ref.sql
Added mappings/verify.specimens.sql
Added inputs/NYBG-CSV/verify/
Makefile: Print done message after verify
VegX-VegBIEN mapping: Use new lookup-only element syntax to ensure that stemtag 1 is not created if it doesn't exist when stemtag 2 tries to set its iscurrent status to false. This should fix the 136 "NullValueException: columns: tag" errors in the SALVIAS organisms import.
xpath.py: get(): Added support for lookup-only elements which are not created if they don't exist
xpath.py: parse(): Added support for lookup-only elements which are not created if they don't exist
VegX-VegBIEN mapping: Map stemtags using [] instead of :[] for attrs that are really keys
Regenerated vegbien.ERD exports
VegX-VegBIEN mapping: Handle user-defined field voucherType (SALVIAS DetType) by mapping specimenreplicates for voucherTypes other than direct via voucher
xml_func.py: Added _if and _eq. Added cast() to throw SyntaxException if can't cast and use it in conv_items(). _merge: Check types of input using conv_items(strings.ustr, items).
util.py: Added all_not_none() and bool2str()
strings.py: Added ustr() (like built-in str() but converts to unicode object)
PostgreSQL-MySQL.csv: Fixed bug in removal of casts of default values, which treated NOT NULL as part of the datatype
VegBIEN: soilobs: Added default value for horizon. Adjusted mappings to remove now-unecessary horizon value.
repl: Removed automatic case-insensitivity because Python apparently only supports turning on case-insensitivity via (?i) but not off via (?-i) (as Java does)
VegBIEN: soilobs: Removed soil* prefix from fields
VegX-VegBIEN mapping: Map to new soilobs fields
SALVIAS inputs: Use new _units:[units="%"] on soil fields that are percents. Replace "<..." values with 0.
xml_func.py: Added _units
vegbien.sql: soilobs: Converted user-defined fields to first-class. Labeled appropriate fields as "fraction".
VegBIEN mappings: Changed tableRecord_ID to tablerecord_id to match PostgreSQL field name
DwC2-VegBIEN mapping: Adjusted user-defined mappings
vegbien.sql: userdefined: Made userdefinedname NOT NULL. userdefined, definedvalue: Added unique constraints.
VegX-VegBIEN mapping: Mapped userdefined fields to new first-class fields
xml_func.py: Added _map and _replace
vegbien.ERD.mwb: Fixed lines. Expanded truncated tables where there was room.
vegbien.sql: locationevent: Added temperature and precipitation
vegbien.sql: aggregateoccurrence: Added growthform
vegbien.ERD.mwb: Reversed the locations of soiltaxon and soilobs to give soilobs room to add new fields
vegbien.sql: Removed embargo table and emb_* fields because we're using a central field, location.confidentialitystatus, for embargo information and coordinate fuzzing
vegbien.sql: stemobservation: Added heightfirstbranch
vegbien.sql: stemobservation: Added diameteraccuracy. Reordered fields.
VegBIEN: stemobservation: Renamed diameter to diameterbreastheight to be more accurate
vegbien.ERD.mwb: Expanded tables where there was room
DwC mappings: Fixed user-defined field mappings according to Brad Boyle's changes
vegbien.sql: Changed specimenreplicate_unique_collectionnumber constraint to include verbatimcollectorname because collection number is assigned by collector
VegBIEN: Moved taxonoccurrence.verbatimcollectorname to specimenreplicate and aggregateoccurrence so that it can be used in specimenreplicate duplicate elimination
