SALVIAS verifications: Fixed to exclude subplots from locations/location events and uniqify locations based on coords
inputs/SALVIAS/verify.sql: Updated for schema changes
Regenerated vegbien.ERD exports
vegbien.ERD.mwb: Re-marked aggregateoccurrence:plantobservation relationship as 1:1 in the ERD. (I think this will need to be manually re-marked whenever either of those tables is updated.)
vegbien.sql: Removed methodgrowthform and growthform, since growthforms can be accommodated by plantconcept in a similar way as higher-order taxonomic ranks
vegbien.sql: methodgrowthform, methodtaxonclass: Removed "included" default value so it's always obvious whether the author intended the classes to be inclusions or exclusions
vegbien.sql: aggregateoccurrence: Removed unneeded fields. Added aggregateoccurrence->coverindex fkey.
vegbien.sql: Added constraint to enforce 1:1 aggregateoccurrence:plantobservation relationship
vegbien.sql: Added plantname unique constraint
bin/map: Use new util.ListDict and util.WrapIter to simplify getting rows by column name instead of index, and to enable a row to be printed with its column names in error messages
util.py: Added WrapIter to wrap an iterator and ListDict to view a list as a dict
bin/map: Use new util.list_flip()
util.py: Added list_flip()
env_password: Fixed to set the environment variable in the calling shell. Do this by cc-ing the tty only on messages before the "Enter password" prompt, because the redirect creates a subshell which causes the env var to only be set within that subshell.
inputs/NYBG-CSV/maps/DwC.specimens.csv: Removed mappings that are already present in mappings/DwC1-DwC2.specimens.csv. This map now contains only the mappings where NYBG-CSV differs from standard DwC1.
inputs/NYBG/maps/DwC.specimens.csv: Removed mappings that are already present in mappings/DwC1-DwC2.specimens.csv. This map now contains only the mappings where NYBG differs from standard DwC1.
Remove accidentally-committed temp file inputs/NYBG/DwC.specimens2.csv
mappings/Makefile: Generate DwC.self.specimens.csv from DwC-VegBIEN.specimens.csv for use in creating full via maps for inputs
input.Makefile: Generate full via maps from input via maps by appending mappings from the via format to itself when available
inputs/NYBG/maps/DwC.specimens.csv: Changed label to "NYBG-DwC" to take advantage of automatic filling in of DwC mappings not specified in the NYBG map
subtract: Support custom column numbers to compare on (instead of just input col). Added ignore option to continue even if input columns don't match.
bin/map: DB inputs: Get all rows in one query (hopefully a significant optimization). Allow maps to contain entries for columns that are not in the DB table.
sql.py: select(): Select all fields if fields == None. Replaced col(cur, idx) with col_names(cur) because an iterator is easier to use than getting by index.
bin/map: Fixed bug in previous implementation of allowing maps for CSV inputs to contain entries for columns that are not in the CSV file
bin/map: Allow maps for CSV inputs to contain entries for columns that are not in the CSV file
Use new sort_map instead of manually specifying the sort order
Added sort_map to sort a map spreadsheet in the standard order
Removed no longer needed join_passthru, because join_union_sort now serves its purpose
Don't generate mappings/for_review/DwC-VegBIEN.specimens.csv because it's a derived map with lots of duplicated mappings for the various DwC versions
mappings/Makefile: Generate DwC-VegBIEN.specimens.csv directly from DwC1-DwC2 and DwC2-VegBIEN mappings by using join_union_sort with header_num=1, rather than via intermediate DwC1-VegBIEN.specimens.csv
union: Added header_num option to select which map's header to use as the output header
Rename join_sort to join_union_sort and have it run union in ignore mode. This will automatically append the joined map when the input map is a derivative of the joined map, such as for NYBG-DwC.
union: Pass through map 0, so that if ignore is set, the input map will still be output. Allow either map's input label to contain the other's input label to enable e.g. appending mappings for an older input version to those for a newer input version.
DwC1-DwC2 mapping: Changed input label to DwC1, which is allowed by the now relaxed label constraints imposed by union
union: Check if two maps can be combined based on whether map 0 column 0 label contains map 1 column 0 label instead of being equal. This allows map 0's input 0 root to contain the datasource name as well as a format that allows it to be combined with a more general map. Added ignore flag to not print an error if column labels don't match.
bin/map: Support optional data format tag in map spreadsheet labels, used by union to check if two maps can be combined
mappings: Added DwC1-DwC2.specimens.csv to core maps so it gets cleaned up
Only generate for_review mappings of core maps and end products
Generate DwC-VegBIEN mapping as union of DwC1 and DwC2 mappings
NYBG DB mapping: Removed IdentifiedDate and CollectedDate mappings because they are generated from the year/month/day
Added mappings/for_review/DwC1-VegBIEN.specimens.csv
Added DwC1-DwC mapping. Generate DwC1-VegBIEN mapping automatically.
vegbien.sql: Renamed _keys unique constraints/unique indexes to _unique to better reflect their purpose
vegbien.sql: Added method.diameterheight to store DBH height
VegBIEN: Moved plantstatus.plantlevel to plantname.rank because the taxonomic rank is a property of the name itself
PostgreSQL-MySQL.csv: Fixed custom types translation to match shorter type names
vegbien.sql: Added plantstatus unique constraint
DwC-VegBIEN mapping: Map datasource name via DwC institutionCode
vegbien.ERD.mwb: Lined up logo and legend with other ERD elements
vegbien.sql: Renamed methodgrowthform.growthformmethod_id to submethod_id. Added methodtaxonclass.submethod_id (similar to methodgrowthform.submethod_id).
vegbien.sql: Added methodgrowthform.growthformmethod_id for specifying a method used by just the growthform
vegbien.ERD.mwb: Rearranged legend to more closely match layout of ERD
vegbien.sql: Reordered plantstatus fields to put the most important fields at the top, which will be visible in the ERD
vegbien.sql: Replaced method.taxonclassincluded,taxonclassexcluded with new many:many methodtaxonclass table. Added methodgrowthform, growthform tables to do the same thing as methodtaxonclass for growth forms.
vegbien.sql: method: Added comment on reference_id
VegBIEN: Moved plotmethod fields to method because they can also apply to strata. Removed no longer used plotmethod table.
input.Makefile: input DB creation: Removed "IF NOT EXISTS" because that check is handled by $(dbExists)
input.Makefile: Don't try to recreate an input DB if it already exists
Added UArizona DB input
Renaming UArizona to UArizona-CSV because there is also a DB input in bien2_staging.ariz_raw on nimoy
Added UArizona input
env_password: Fixed bug where exit command would not cause it to exit, because pipefail shell option was not set. Moved automatic exiting of the calling script into env_password itself.
map: Exit if password not set
env_password: cc stderr if it's a log file
env_password: Print all messages to /dev/tty so the user sees them even if stderr is redirected to a log file. Exit if password not already set, because e.g. scripts run in the background will not be able to prompt for it.
input.Makefile: Don't have make import call verify, because the user often runs import as a test and will not want the output cluttered with verification information. Also, the full imports for which this was intended are often run asynchronously, so that the user will not see the output anyway.
input.Makefile: Don't abort on verification errors, which are expected during development
SALVIAS tests: Fixed invalid accepted test outputs due to not running `make empty_db` before running tests when using the no-redo optimization shortcut
SALVIAS mappings: Fixed plot key mappings to map the correct values to subplot and parent plot
vegbien.sql: locationevent: Added unique constraint for subplots based on subplot location
SALVIAS-db VegX mapping: Map subplots correctly the way SALVIAS-CSV does
SALVIAS verification: Updated to schema changes
input.Makefile: Fixed syntax error in verify %.ref target (outdated variable name)
input.Makefile: Halt psql commands on first error
vegbien.sql: Removed location.authorlocationcode because it's now stored in locationevent as an author-specific setting
vegbien.sql: locationevent: Redid unique constraints to avoid applying authorlocationcode-only duplicate elimination to subplots
SALVIAS mappings: Map SiteCode/plot_code to locationevent.authorlocationcode because locationevent is now the place to store author-specific plot information
SALVIAS mappings: Fixed PlotID mapping to go to locationevent.sourceaccessioncode
VegBIEN: Renamed locationevent.authoreventcode to authorlocationcode to reflect that datasources usually use an author-defined code for a plot rather than a plot event
vegbien.sql: locationevent: Redid unique constraints to handle datasources that treat the authoreventcode as an authorlocationcode. Eventually, authoreventcode will be renamed to authorlocationcode.
vegbien.sql: locationevent: Redid unique constraints to work properly for all fully-specified combinations of keys
VegBIEN mappings: Mapped datasource name to new project.datasource. Fixes project duplicate elimination.
vegbien.sql: Renamed project.reference_id to datasource_id and pointed it to party, to match locationevent, etc.
VegBIEN mappings: Mapped current lat/long to centerlat/long as well so location duplicate elimination will work properly
xpath.py: Added support for common subpath after split path's {}
sql.py: put(): When encountering a DuplicateKeyException, use dict_subset_right_join to fill in explicit NULL values for columns which don't have data. This causes the database to use the UNIQUE constraint's index to look up the record, instead of relying on individual column indexes for the columns that did have data, which may or may not be available.
util.py: Added DefaultDict to wrap collections.defaultdict with a simple value passed in the constructor, defaulting to None. Added dict_subset_right_join() to fill in None for subset keys that don't exist.
vegbien.sql: Added method and plotmethod UNIQUE indexes
vegbien.ERD.mwb: Removed embargo table from ERD because its functionality is provided in location.confidentialitystatus,confidentialityreason
vegbien.sql: Moved locationevent method fields to plotmethod and method. Added comments to method/plotmethod fields, as provided by Michael Lee.
VegX-VegBIEN mapping: Mapped locationevent.methodnarrative to new plotmethod table
VegX-VegBIEN mapping: Map sourceaccessioncode and voucher (catalognumber_dwc) to correct place. SALVIAS mappings: Map SourceVoucher as an alternative to coll_number.
vegbien.sql: Removed VegBank-internal tables (including user account tables) because they do not belong in the ecological database. Any web interface should store user account information, cached queries, etc. in a separate interface-specific database.
VegX-VegBIEN mapping: Mapped stem tags to new stemtag table
vegbien.sql: Renamed planttag to stemtag and made it a child of stemobservation. Removed trait table from ERD because it's not used for the purpose we want to use traits for.
