Added schemas/vegbien.ERD.poster.pdf
Added schemas/vegbien.ERD.poster.core.print_options.png
Added schemas/vegbien.ERD.poster.core.pdf
schemas/pg_hba.Mac.conf: Fixed bug where needed ident entry for postgres superuser
Added config/bien_read_password
README.TXT: Schema changes: Added instructions to calculate the minimum VegBIEN poster size (to make the text as least as big as on the VegBank ERD poster), which is 35x54in portrait
schemas/vegbien.sql: analytical_stem_view: cultivated: Use location.iscultivated when taxonoccurrence.iscultivated is not available
Added inputs/FIA/FIA_COND_unique/, which contains the oldgrowth flag
mappings/VegCore-VegBIEN.csv: Mapped oldGrowth
schemas/functions.sql: Added _not()
mappings/VegCore.csv: Added oldGrowth
mappings/VegCore-VegBIEN.csv: Remapped cultivated to location when a TaxonOccurrence is not provided, indicating that the record is a plot
schemas/vegbien.sql: location: Added iscultivated for cases when entire plots rather than individual taxonoccurrences are marked as cultivated
inputs/FIA/: Added FIA_COND table from nimoy.geoscrub and code to generate a unique plot table from it, including the oldgrowth calculated field
Added inputs/FIA/Organism/postprocess.sql to cast PlotCD to a bigint
my2pg: Also remove (#) after bigint
Regenerated vegbien.ERD exports
schemas/vegbien.ERD.mwb: Fixed lines
schemas/vegbien.sql: source: Renamed fulltext to citation because according to the VegBank data dictionary <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=reference&entity=dba_tabledescription&where=where_tablename#fulltext> this is actually the full text of the reference citation, not of the reference itself (it would be unusual to store that in VegBank)
schemas/vegbien.sql: Removed no longer needed sourcejournal, which can be stored in source and pointed to via parent_id instead of sourcejournal_id. sourcejournal.journal maps to source.fulltext, issn to isbn, and abbreviation to shortname.
mappings/VegCore-VegBIEN.csv: Mapped acceptedCounty, county to the matched place
schemas/vegbien.sql: source: Added matched_source_id
sql.py: parse_exception(): function MissingCastException: If 1st param is hstore, only perform the cast on the value param. This fixes a bug in _map() calls whose value is a non-text type, such as SALVIAS.plotMetadata.AccessCode.
sql_io.py: cast(): Use sql_gen.Cast() to generate the cast, in order to take advantage of its support for casts to unknown
sql_gen.py: Cast: Support casts to unknown by casting to text first
schemas/postgresql.conf: Turn on the error log
schemas/pg_hba.conf: Also grant the bien user access to template1, which is accessed by phpPgAdmin
schemas/vegbien.sql: source: Added parent_id for nested sources, e.g. an article in a journal
lib/forwarding.Makefile: $(subdirs): Also exclude .archive/
Renamed inputs/_archive/ to .archive/ so it wouldn't be treated as a datasource
README.TXT: Documentation: Redmine-formatted list of steps for column-based import: Use ACAD instead of QMOR, which was removed
inputs/Makefile: Import logs: $(rsyncLogs): Include log files at any depth in the directory tree rather than just 1-2 levels deep. This adds log files whose containing directories have been moved to _archive/ directories.
Added inputs/_archive/
Removed inputs/QMOR/ because it's an insect collection
schemas/vegbien.sql: projectcontributor: Removed surname, since this information is stored in party_id->party.surname
schemas/vegbien.sql: projectcontributor: Removed cheatrole, since there is already a role field and this field was unused in VegBank
schemas/vegbien.sql: role: Added values from projectcontributor.ROLE_ID <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=projectcontributor&entity=dba_tabledescription&where=where_tablename#ROLE_ID>
schemas/vegbien.sql: sourcecontributor: role: Changed type to role
schemas/vegbien.sql: role enum: Added VegBank data dictionary values from <http://vegbank.org/vegbank/views/dba_fielddescription_detail.jsp?view=detail&wparam=1331&entity=dba_fielddescription¶ms=1331>
schemas/vegbien.sql: sourcecontributor: Renamed position to order for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#position> and disambiguation from other meanings of position which are similar to role
schemas/vegbien.sql: sourcecontributor: Renamed roletype to role for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#roleType>
inputs/.geoscrub/geoscrub_output/map.csv: Mapped to county, acceptedCounty
mappings/VegCore.csv: Added acceptedCounty
schemas/pg_hba.Mac.conf: Changed to match schemas/pg_hba.conf
schemas/pg_hba.conf: Fixed bug where also need an IPv6 bien entry with md5 authentication, because the IPv4 md5 authentication does not apply to "localhost", which is translated to the IPv6 address ::1
schemas/pg_hba.conf: Fixed bug where also need a local bien entry with md5 authentication, because the host-based md5 authentication applies only to literal IP addresses, not "localhost"
Added schemas/pg_hba.Mac.conf
schemas/pg_hba.conf: Restrict all accesses to the server except the bien user accessing vegbien using ident or a password, and the postgres superuser logging in using ident
inputs/.geoscrub/geoscrub_output/map.csv: Mapped countyvalidity to latLongInCounty
schemas/functions.sql: _map(): Fixed bug where entries that map to NULL were incorrectly being treated as if the entry didn't exist. Note that -> returns NULL both if the entry's value is NULL and if the entry doesn't exist, so ? must be used to recheck the presence of the key in the hstore.
mappings/VegCore-VegBIEN.csv: Mapped latLongInCounty
mappings/VegCore.csv: Added latLongInCounty
schemas/vegbien.sql: Added distance_to_county_m. Note that this can also be used to store latLongInCounty by mapping true to 0 and false to -1.
schemas/pg_hba.conf: Changed trust authentication back to ident/md5. Not sure how it got set to trust since I used md5 when enabling remote access to the DB for the bien user.
Added schemas/pg_hba.conf
schemas/vegbien.sql: place: Removed placecode to prevent datasources from creating duplicate entries for the same place, with different placecodes. This was a problem with the original BIEN2 geoscrub dataset, which contained duplicates.
inputs/import.stats.xls: Updated import times
schemas/vegbien.sql: analytical_stem_view: Fixed bug where need to join taxonoccurrence.collector_id to party because it's now an fkey rather than a literal name
schemas/vegbien.sql: analytical_*: Added coordinateUncertaintyInMeters
schemas/vegbien.sql: analytical_stem_view: Join to newWorldCountries on 2-digit ISO code instead of country name, to increase (BIEN2) newWorldCountries and GADM overlap
psql_vegbien: Run with sh because it no longer needs bash support
psql_script_vegbien: Fixed bug where needs to be run with bash instead of sh
Added inputs/newWorld/iso_code_gadm/
Added inputs/newWorld/_src/
inputs/XAL/Specimen/map.csv: darwin:FieldNumber: Removed command to determine that field is unused, because UNUSED is a factual assertion that does not need a reason to be specified each time
inputs/XAL/Specimen/map.csv: Remapped darwin:CoordinatePrecision to UNUSED
inputs/NY/Specimen/map.csv: Remapped CoordinatePrecision to coordinateUncertaintyInMeters, assuming units of m based on the range and precision of values
mappings/VegCore.csv: coordinatePrecision: Added units (degrees) to form coordinatePrecision_deg
mappings/VegCore-VegBIEN.csv: Removed mapping for coordinatePrecision, which is not the same as coordsaccuracy_m. coordinatePrecision is instead "the precision of the coordinates" themselves in degrees (<http://rs.tdwg.org/dwc/terms/#coordinatePrecision>).
schemas/vegbien.sql: coordinates: Changed coordinates.coordsaccuracy_deg units to m
Regenerated inputs/bien_web/observation/test.xml.ref
schemas/vegbien.ERD.mwb: Added projectcontributor, locationeventcontributor to ERD
schemas/vegbien.sql: higher_plant_group_nodes: Added root->NULL mapping to store all the families that don't match any higher plant group
schemas/vegbien.sql: higher_plant_group_nodes: Allow NULL values for higher_plant_group, to allow mapping all remaining families to NULL in family_higher_plant_group
psql_vegbien: Fixed bug where did not display command prompt when run from command line, by moving automatic setting of search_path to psql_script_vegbien. psql_script_vegbien is now used instead of psql_vegbien wherever the search_path needs to be set, so removing this functionality from psql_vegbien is not a problem.
input.Makefile: BIEN commands: $(psqlAsBien): Use psql_script_vegbien, which automatically adds the $(psqlOpts), instead of psql_vegbien
schemas/functions.sql: _map(): Support any entry having the value '*' (not just the '*' entry), which passes through that value. Support an entry having the value '!', which raises an exception.
inputs/SALVIAS/plotMetadata_/map.csv: AccessCode: Removed _map entry for 4, which does not apply to plots
schemas/vegbien.sql: locationevent: Added accesslevel
inputs/SALVIAS/plotMetadata_/map.csv: Mapped AccessCode to dcterms:accessRights with appropriate _map filter
Added inputs/.geoscrub/geoscrub_cleaned_unique/_no_import to disable geoscrub_cleaned_unique, since the new geoscrub_output supersedes it
Added inputs/.geoscrub/geoscrub_output/
Added inputs/.geoscrub/_src/README.TXT
Regenerated inputs/bien_web/observation/VegBIEN.csv
Added inputs/.geoscrub/_src/ to store Jim's geoscrub CSV
schemas/functions.sql: _map(): Changed error message for an unmapped value to "Value not in map" rather than "Invalid map value", because an unmapped value is not necessarily explicitly invalid
inputs/VegBank/plot_/map.csv: confidentialitystatus filter: Merged mappings for 0 with other public-equivalent fields. Note that fuzzed plots are still public, because the private columns have been removed.
inputs/VegBank/plot_/map.csv: Mapped confidentialitystatus to dcterms:accessRights with an appropriate _map filter
mappings/VegCore-VegBIEN.csv: Mapped dcterms:accessRights
schemas/functions.sql: _map(): Raise data_exception if value not in map and no default provided (not the same as a NULL default value)
mappings/VegCore-VegBIEN.csv: verbatimGrowthForm: Removed _map filter, which applied only to SALVIAS and has now been moved to the applicable SALVIAS tables
inputs/SALVIAS*/plotObservations/map.csv: Remapped Habit to growthForm with _map filter applied
sql_io.py: put_table(): Special handling for functions with hstore params: Fixed bug where need to unwrap literal values of mapping, which might be sql_gen.Literal objects
