VegBIEN: taxonrank enum: Made values lowercase to match case convention in other enums
Regenerated vegbien.ERD exports
vegbien.sql: Renamed plantconceptscope to plantnamescope because it's now attached to plantname
vegbien.sql: Moved parent_id from plantconcept to plantname, since plantnames themselves are unique according to their parent taxons (a species under one genus is not the same as a species under another genus)
vegbien.ERD.mwb: Fixed lines
vegbien.sql: Moved scope_id from plantconcept to plantname, since plantnames themselves are scoped, not just the plantconcepts that use them (e.g. "sp. 1" has different meanings in different scopes, so it should not be shared between scopes). plantname: Added accessioncode.
vegbien.sql: Moved plantconcept parent_id from plantstatus to plantconcept. plantconcept: Removed datasource-specific fields to make it globally unique (one plantconcept for each assigned parent taxon of a plantname, of which there will usually be just one)
vegbien.sql: plantname: Removed datasource-specific fields to make this a globally-unique table (the datasource-specific fields belong in plantconcept)
Added inputs/UArizona/verify
mappings/verify.specimens.sql: Updated for schema changes
vegbien.sql: placerank enum: Added "village"
VegBIEN mappings: lat/long locationdetermination: Removed [!namedplace_id] key so that it's merged into the namedplace locationdetermination
VegBIEN mappings: Changed namedplace mappings to use new nested format for storing place containment relationships
xml_func.py: Added _simplifyPath
xpath.py: Added get_1()
vegbien.sql: namedplace: Removed parent_id from unique constraint because some data might be missing intervening links (e.g. state for a county, country), but the place (e.g. county) should still be attached to the existing place of the same name and rank (which will hopefully already have the correct parent_id link)
vegbien.sql: namedplace: Made rank required
vegbien.sql: namedplace: Removed no longer needed placesystem, which has been replaced by rank closed list
VegBIEN mappings: Map namedplaces using new rank field
vegbien.sql: namedplace: Added rank. Do duplicate elimination using rank and parent_id instead of placesystem
vegbien.sql: placerank: Standardized names to DwC/GML
vegbien.sql: Added placerank enum
vegbien.sql: namedplace: Removed VegBank internal fields and datasource scoping fields (namedplaces are globally unique). Added parent_id to point to containing namedplace.
xml_func.py: Added _dateRangePart with partial implementation (only works on strings with no range)
DwC mappings: Moved date _date filter outside _alt so it would run only on the string that was actually chosen, and not produce date format errors when a pre-parsed year/month/day is already available
xml_func.py: _date: Map date with only empty fields to NULL (occurs when all fields were e.g. 0 and were filtered to NULL by _nullIf)
xml_func.py: _date: Removed mapping year/month/day of 0 to NULL because that is now handled on a case-by-case basis in the mappings
mappings/DwC1-DwC2.specimens.csv: Map year/month/day of 0 to NULL
inputs/SALVIAS/maps/VegX.organisms.csv: Habit: Fixed syntax error in growthForm map
inputs/SALVIAS/maps/VegX.organisms.csv: Habit: Removed input values from growthForm map that Brad said were invalid
xml_func.py: _map: Added option to make map a closed list
mappings/DwC2-VegBIEN.specimens.csv: Fixed waterdepth mappings to use _avg
mappings/verify.specimens.sql: Use ORDER BY ... NULLS FIRST to match MySQL
input.Makefile: verify: Time the verification since it can take a long time
specimens verification: Added duplicate catalog numbers test
map: On nimoy, use bien2_staging unless otherwise specified
specimens verification: Added # counties test
specimens verification: Added collection codes and # catalog numbers tests
inputs/SALVIAS/maps/VegX.organisms.csv: Mapped custom Habit values not listed in the SALVIAS data dictionary
strings.py: Added unicode_reader for later use in handling Unicode characters in map spreadsheets
xpath.py: Removed unnecessary copy.deepcopy()'s and instead changed set_value() and set_id() to make copies of any elements they change. This should result in up to a 17% speed increase in the import, because deepcopy() was taking a lot of time. Added documentation to set_value() and set_id() that caller must make a shallow copy of the path to prevent modifications from propagating to other copies of the path. (Previously, a deep copy was needed, but there was no comment specifying this.)
mappings/VegX-VegBIEN.organisms.csv: Removed unneeded lookahead assertions from stemtag mappings. They relied on a bug ("feature"?) in the XPath engine that made the value of the lookahead assertion's path the same as the value of the main path, even though the value is set after the path is parsed.
xml_func.py: _date: For year/month/day dates, require the year (it would not make sense to default to a particular year)
inputs/UArizona: Added test outputs
mappings/DwC1-DwC2.specimens.csv: Fixed to allow datasource to define custom date mappings that don't pass through the default date mapping
input.Makefile: Generate maps/src.join.*.csv, which can be used to determine which DwC fields for a particular dataset do not yet have a join mapping to VegBIEN
Makefile: Fixed subdir remake target to work for nested subdirs as well
inputs/UArizona: Renamed maps/src.csv to maps/src.specimens.csv because there will be one for each input table
inputs/UArizona: Added maps/src.csv with columns from source data
Added autogen mappings/DwC-VegBIEN.specimens.no_empty.csv, which will be used for determining which DwC fields for a particular dataset do not yet have a join mapping to VegBIEN
Added remove_empty to remove empty mappings in a map spreadsheet
join: Don't raise "No join mapping" error for empty mappings because you only want the error for empty mappings for your particular dataset, which requires more information (namely, the subset of the mappings used by your dataset, some of which will not be in the mappings if standard fields have been subtracted out)
join: Fixed bug in "No join mapping" error generation where rows with no existing comments column would cause an IndexError
util.py: Added list_set() and list_setdefault()
inputs/UArizona/maps/DwC.specimens.csv: Merge FieldNotes and Remarks
inputs/UArizona/maps/DwC.specimens.csv: Finished mappings
inputs/UArizona/maps/DwC.specimens.csv: Removed fields already present in DwC mappings
inputs/NYBG-CSV/maps/DwC.specimens.csv: Removed mappings already present in case-insensitive DwC2 mapping
inputs/NYBG/maps/DwC.specimens.csv: Removed mappings already present in case-insensitive DwC2 mapping
mappings/DwC1-DwC2.specimens.csv: Removed fields already present in DwC2.ci-VegBIEN.specimens.csv
Makefiles: Moved remake into main Makefile. Fixed remake to run `make all` in a new make so that cache of existing files is reset. Have main remake run clean and then all instead of forwarding remake to subdirs, so that everything is cleaned before everything is remade.
input.Makefile: maps: maps/$(via).%.full.csv: Fixed bug where $(selfMap) would be ignored if it had not yet been made
mappings/Makefile: Reorganized into DwC and VegX sections
Added autogenerated mappings/DwC2.ci-VegBIEN.specimens.csv. Use it to include DwC2 fields with first letter uppercased in the full DwC mapping, so that datasources that use DwC2 terms with a different case can still use the DwC2 mapping.
inputs/UArizona/maps/DwC.specimens.csv: Mapped CollectedDate to eventDate/_alt/2 even though it's not used because other datasources might copy these mappings and want it already filled in
Added ucase_first to uppercase the first character of columns in a spreadsheet
Added inputs/UArizona/maps/DwC.specimens.csv autogen maps
inputs/UArizona/maps/DwC.specimens.csv: Mapped more fields
mappings/DwC1-DwC2.specimens.csv: Remove date -> date/_alt/2 mappings because they prevent the original DwC2 date field from being mapped to without an extra /_alt/2 appended
xml_func.py: Use new dates.strtotime(). When component date parts specified, year defaults to dates.epoch.year.
dates.py: Added strtotime() to wrap dateutil.parser.parse() with default defaulting to epoch, so that e.g. months with day missing default to day 1 instead of the current day of the month
mappings/DwC1-DwC2.specimens.csv: Map eventDate,dateIdentified using /_alt/2 and year/month/day using /_alt/1 so that inputs with both a date and date parts will select between the two
input.Makefile: Added comment that self map must be made first if it's needed for maps/$(via).%.full.csv
Makefiles: Use .SECONDARY with no prerequisites instead of setting a .PRECIOUS for each intermediate, to simplify turning off automatic deletion of intermediate files
inputs/UArizona: Added initial maps/DwC.specimens.csv
DwC mappings: Map datasource name via institutionID to avoid conflicting with existing institutionCode fields that many DwC data sources have
input.Makefile: Don't profile by default because it appears to slow things down significantly on long imports
Added inputs/UArizona/maps
Makefile: python-Linux: Added python-profiler
specimens verification: Added # binomials test
vegbien.sql: specimenreplicate: Removed specimenreplicate_unique_collectionnumber index because the collectionnumber (NYBG FieldNumber) is not always unique within a collector, even though it should be. Changed specimenreplicate_unique_catalognumber to only operate on rows with no sourceaccessioncode (of which there are 8 in NYBG).
mappings/verify.specimens.sql: # species test: Fixed to join separately on taxondeterminations for genus and species. # genera test: Removed no longer needed join on party.
vegbien.sql: specimenreplicate: Added fki index on taxonoccurrence_id
vegbien.sql: plantname: Added index on rank to speed up specimens verifications, where the query planner insists on joining from plantname to specimenreplicate instead of the other way around (which takes much longer without the index)
mappings/verify.*: Use nested SELECT instead of JOIN on party to get datasource_id, so that party will not be joined on after other joins have already occurred (which slows things down)
vegbien.sql: party: Changed party_unique_name to ignore NULL values and the organizationname (a first(+middle)+last name is considered unique)
vegbien.sql: party: Added party_unique_organizationname constraint
Specimens verification: Added # genera and # species
input.Makefile: verify: Create target dir if it doesn't exist
inputs/NYBG: Added verify/specimens.ref.sql
Added mappings/verify.specimens.sql
Added inputs/NYBG-CSV/verify/
Makefile: Print done message after verify
VegX-VegBIEN mapping: Use new lookup-only element syntax to ensure that stemtag 1 is not created if it doesn't exist when stemtag 2 tries to set its iscurrent status to false. This should fix the 136 "NullValueException: columns: tag" errors in the SALVIAS organisms import.
xpath.py: get(): Added support for lookup-only elements which are not created if they don't exist
xpath.py: parse(): Added support for lookup-only elements which are not created if they don't exist
VegX-VegBIEN mapping: Map stemtags using [] instead of :[] for attrs that are really keys
