mappings/VegCore-VegBIEN.csv: Remapped cultivated to location when a TaxonOccurrence is not provided, indicating that the record is a plot
schemas/vegbien.sql: location: Added iscultivated for cases when entire plots rather than individual taxonoccurrences are marked as cultivated
inputs/FIA/: Added FIA_COND table from nimoy.geoscrub and code to generate a unique plot table from it, including the oldgrowth calculated field
Added inputs/FIA/Organism/postprocess.sql to cast PlotCD to a bigint
my2pg: Also remove (#) after bigint
Regenerated vegbien.ERD exports
schemas/vegbien.ERD.mwb: Fixed lines
schemas/vegbien.sql: source: Renamed fulltext to citation because according to the VegBank data dictionary <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=reference&entity=dba_tabledescription&where=where_tablename#fulltext> this is actually the full text of the reference citation, not of the reference itself (it would be unusual to store that in VegBank)
schemas/vegbien.sql: Removed no longer needed sourcejournal, which can be stored in source and pointed to via parent_id instead of sourcejournal_id. sourcejournal.journal maps to source.fulltext, issn to isbn, and abbreviation to shortname.
mappings/VegCore-VegBIEN.csv: Mapped acceptedCounty, county to the matched place
schemas/vegbien.sql: source: Added matched_source_id
sql.py: parse_exception(): function MissingCastException: If 1st param is hstore, only perform the cast on the value param. This fixes a bug in _map() calls whose value is a non-text type, such as SALVIAS.plotMetadata.AccessCode.
sql_io.py: cast(): Use sql_gen.Cast() to generate the cast, in order to take advantage of its support for casts to unknown
sql_gen.py: Cast: Support casts to unknown by casting to text first
schemas/postgresql.conf: Turn on the error log
schemas/pg_hba.conf: Also grant the bien user access to template1, which is accessed by phpPgAdmin
schemas/vegbien.sql: source: Added parent_id for nested sources, e.g. an article in a journal
lib/forwarding.Makefile: $(subdirs): Also exclude .archive/
Renamed inputs/_archive/ to .archive/ so it wouldn't be treated as a datasource
README.TXT: Documentation: Redmine-formatted list of steps for column-based import: Use ACAD instead of QMOR, which was removed
inputs/Makefile: Import logs: $(rsyncLogs): Include log files at any depth in the directory tree rather than just 1-2 levels deep. This adds log files whose containing directories have been moved to _archive/ directories.
Added inputs/_archive/
Removed inputs/QMOR/ because it's an insect collection
schemas/vegbien.sql: projectcontributor: Removed surname, since this information is stored in party_id->party.surname
schemas/vegbien.sql: projectcontributor: Removed cheatrole, since there is already a role field and this field was unused in VegBank
schemas/vegbien.sql: role: Added values from projectcontributor.ROLE_ID <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=projectcontributor&entity=dba_tabledescription&where=where_tablename#ROLE_ID>
schemas/vegbien.sql: sourcecontributor: role: Changed type to role
schemas/vegbien.sql: role enum: Added VegBank data dictionary values from <http://vegbank.org/vegbank/views/dba_fielddescription_detail.jsp?view=detail&wparam=1331&entity=dba_fielddescription¶ms=1331>
schemas/vegbien.sql: sourcecontributor: Renamed position to order for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#position> and disambiguation from other meanings of position which are similar to role
schemas/vegbien.sql: sourcecontributor: Renamed roletype to role for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#roleType>
inputs/.geoscrub/geoscrub_output/map.csv: Mapped to county, acceptedCounty
mappings/VegCore.csv: Added acceptedCounty
schemas/pg_hba.Mac.conf: Changed to match schemas/pg_hba.conf
schemas/pg_hba.conf: Fixed bug where also need an IPv6 bien entry with md5 authentication, because the IPv4 md5 authentication does not apply to "localhost", which is translated to the IPv6 address ::1
schemas/pg_hba.conf: Fixed bug where also need a local bien entry with md5 authentication, because the host-based md5 authentication applies only to literal IP addresses, not "localhost"
Added schemas/pg_hba.Mac.conf
schemas/pg_hba.conf: Restrict all accesses to the server except the bien user accessing vegbien using ident or a password, and the postgres superuser logging in using ident
inputs/.geoscrub/geoscrub_output/map.csv: Mapped countyvalidity to latLongInCounty
schemas/functions.sql: _map(): Fixed bug where entries that map to NULL were incorrectly being treated as if the entry didn't exist. Note that -> returns NULL both if the entry's value is NULL and if the entry doesn't exist, so ? must be used to recheck the presence of the key in the hstore.
mappings/VegCore-VegBIEN.csv: Mapped latLongInCounty
mappings/VegCore.csv: Added latLongInCounty
schemas/vegbien.sql: Added distance_to_county_m. Note that this can also be used to store latLongInCounty by mapping true to 0 and false to -1.
schemas/pg_hba.conf: Changed trust authentication back to ident/md5. Not sure how it got set to trust since I used md5 when enabling remote access to the DB for the bien user.
Added schemas/pg_hba.conf
schemas/vegbien.sql: place: Removed placecode to prevent datasources from creating duplicate entries for the same place, with different placecodes. This was a problem with the original BIEN2 geoscrub dataset, which contained duplicates.
inputs/import.stats.xls: Updated import times
schemas/vegbien.sql: analytical_stem_view: Fixed bug where need to join taxonoccurrence.collector_id to party because it's now an fkey rather than a literal name
schemas/vegbien.sql: analytical_*: Added coordinateUncertaintyInMeters
schemas/vegbien.sql: analytical_stem_view: Join to newWorldCountries on 2-digit ISO code instead of country name, to increase (BIEN2) newWorldCountries and GADM overlap
psql_vegbien: Run with sh because it no longer needs bash support
psql_script_vegbien: Fixed bug where needs to be run with bash instead of sh
Added inputs/newWorld/iso_code_gadm/
Added inputs/newWorld/_src/
inputs/XAL/Specimen/map.csv: darwin:FieldNumber: Removed command to determine that field is unused, because UNUSED is a factual assertion that does not need a reason to be specified each time
inputs/XAL/Specimen/map.csv: Remapped darwin:CoordinatePrecision to UNUSED
inputs/NY/Specimen/map.csv: Remapped CoordinatePrecision to coordinateUncertaintyInMeters, assuming units of m based on the range and precision of values
mappings/VegCore.csv: coordinatePrecision: Added units (degrees) to form coordinatePrecision_deg
mappings/VegCore-VegBIEN.csv: Removed mapping for coordinatePrecision, which is not the same as coordsaccuracy_m. coordinatePrecision is instead "the precision of the coordinates" themselves in degrees (<http://rs.tdwg.org/dwc/terms/#coordinatePrecision>).
schemas/vegbien.sql: coordinates: Changed coordinates.coordsaccuracy_deg units to m
Regenerated inputs/bien_web/observation/test.xml.ref
schemas/vegbien.ERD.mwb: Added projectcontributor, locationeventcontributor to ERD
schemas/vegbien.sql: higher_plant_group_nodes: Added root->NULL mapping to store all the families that don't match any higher plant group
schemas/vegbien.sql: higher_plant_group_nodes: Allow NULL values for higher_plant_group, to allow mapping all remaining families to NULL in family_higher_plant_group
psql_vegbien: Fixed bug where did not display command prompt when run from command line, by moving automatic setting of search_path to psql_script_vegbien. psql_script_vegbien is now used instead of psql_vegbien wherever the search_path needs to be set, so removing this functionality from psql_vegbien is not a problem.
input.Makefile: BIEN commands: $(psqlAsBien): Use psql_script_vegbien, which automatically adds the $(psqlOpts), instead of psql_vegbien
schemas/functions.sql: _map(): Support any entry having the value '*' (not just the '*' entry), which passes through that value. Support an entry having the value '!', which raises an exception.
inputs/SALVIAS/plotMetadata_/map.csv: AccessCode: Removed _map entry for 4, which does not apply to plots
schemas/vegbien.sql: locationevent: Added accesslevel
inputs/SALVIAS/plotMetadata_/map.csv: Mapped AccessCode to dcterms:accessRights with appropriate _map filter
Added inputs/.geoscrub/geoscrub_cleaned_unique/_no_import to disable geoscrub_cleaned_unique, since the new geoscrub_output supersedes it
Added inputs/.geoscrub/geoscrub_output/
Added inputs/.geoscrub/_src/README.TXT
Regenerated inputs/bien_web/observation/VegBIEN.csv
Added inputs/.geoscrub/_src/ to store Jim's geoscrub CSV
schemas/functions.sql: _map(): Changed error message for an unmapped value to "Value not in map" rather than "Invalid map value", because an unmapped value is not necessarily explicitly invalid
inputs/VegBank/plot_/map.csv: confidentialitystatus filter: Merged mappings for 0 with other public-equivalent fields. Note that fuzzed plots are still public, because the private columns have been removed.
inputs/VegBank/plot_/map.csv: Mapped confidentialitystatus to dcterms:accessRights with an appropriate _map filter
mappings/VegCore-VegBIEN.csv: Mapped dcterms:accessRights
schemas/functions.sql: _map(): Raise data_exception if value not in map and no default provided (not the same as a NULL default value)
mappings/VegCore-VegBIEN.csv: verbatimGrowthForm: Removed _map filter, which applied only to SALVIAS and has now been moved to the applicable SALVIAS tables
inputs/SALVIAS*/plotObservations/map.csv: Remapped Habit to growthForm with _map filter applied
sql_io.py: put_table(): Special handling for functions with hstore params: Fixed bug where need to unwrap literal values of mapping, which might be sql_gen.Literal objects
sql_gen.py: Added get_value()
dicts.py: join(): Added support for unhashable types, which are passed through. This adds support for SQL literal values which are dicts (hstores).
xml_func.py: Removed no longer used _map(), which has been replaced by a corresponding DB function
schemas/functions.sql: Added _map(), which uses the new hstore functionality. This expands _map() functionality to column-based import.
root Makefile: VegBIEN DB: DB and bien user: mk_db: hstore extension: Fixed bug where need to use `CREATE EXTENSION hstore SCHEMA pg_catalog` instead of createlang, because hstore must be explicitly created in pg_catalog or else it will be created in the public schema instead, causing it to get deleted every time the public schema is reinstalled and cascading the delete to everything (including in other schemas) that uses hstore
sql_io.py: put_table(): Added special handling for functions with hstore params. Note that although _map() doesn't exist yet as a DB function, this code must be in place before _map() is created to avoid param type mismatch errors.
root Makefile: PostgreSQL: postgres-Linux: Changed plpython to plpython3 in order to install plpython3u
schemas/py_functions.sql: _date(): Removed features that require dateutil, which is not available under plpython3u. This includes removing the now-unused date string parameter.
mappings/VegCore-VegBIEN.csv: Removed _date/date, because _date using a string date argument is no longer supported under plpython3u (dateutil is missing). Note that PostgreSQL's own date parsing is sufficient for most dates, so this use of _date is not strictly necessary and removing it will improve import times.
schemas/py_functions.sql: Replaced xrange() with range() for plpython3u
root Makefile: Python: python-Linux: Also install python3, needed by plpython3u
schemas/py_functions.sql: Updated except clause syntax for PostgreSQL 9.1.6
