inputs/FIA/import_order.txt: Fixed bug where FIA_COND_unique needed to be explicitly included in import_order.txt now that we're using import_order.txt to import the Source metadata table before the data tables
inputs/import.stats.xls: Updated import times
root Makefile: PostgreSQL: $(postgresReload-Linux): Try chmoding both as your user and as the bien user
input.Makefile: Testing: $(runTest): Ignore failed diffs when the test is compared to another test's output (e.g. in by_col mode)
bin/map: in_is_db: If table does not exist, set table to None so that db_xml.put_table() doesn't try to access it. This fixes a bug in metadata-only map spreadsheets under column-based import.
db_xml.py: put_table(): Support None in_table by calling put() directly
Removed no longer used geoscrub.*.sql. Use geoscrub_output instead.
Removed no longer used geoscrub_cleaned_unique. Use geoscrub_output instead.
Removed no longer used geoscrub_cultivated. Use analytical_stem_view.cultivated instead.
schemas/vegbien.sql: analytical_stem_view: cultivated: Removed BIEN2's geoscrub_cultivated, which has now been replaced by the primary corresponding scripts (and never had particularly many matches to the locations in any case)
schemas/vegbien.sql: analytical_stem_view: cultivated: Use OR instead of _or() to combine cultivated_family_locations.country IS NOT NULL with the other values, because this field's false value should not be used in place of NULL if all the other values are NULL, as it would be with _or(). (cultivated_family_locations.country IS NOT NULL can indicate presence, but not absence, of cultivated status.)
schemas/functions.sql, vegbien.sql: _and(), _or(): Added comment comparing the function and the corresponding logical operator
schemas/vegbien.sql: public: Added _or(), for use by analytical_stem_view
schemas/vegbien.sql: analytical_stem_view: cultivated: Also set if family/country combination found in cultivated_family_locations
schemas/vegbien.sql: cultivated_family_locations: Added data from nimoy:/home/boyle/bien2/geoscrub/cultivated/cult_by_taxon/flag_by_taxa.inc
schemas/vegbien.sql: Added cultivated_family_locations to store locations where various taxon families are considered cultivated
mappings/VegCore-VegBIEN.csv: Mapped locality description fields to location.iscultivated using _locationnarrative_is_cultivated()
xml_func.py: Simplifying functions: Added passthru entries for _and, _or
schemas/vegbien.sql: Added _locationnarrative_is_cultivated()
lib/PostgreSQL-MySQL.csv: Change text to varchar(255) because text columns can't be used in indexes in MySQL
lib/PostgreSQL-MySQL.csv: Resaved in Excel, which removed unnecessary quotes around fields
schemas/vegbien.sql: analytical_aggregate: Added identifiedBy, which is no longer a scoping field (which would prevent scientificNameWithMorphospecies from being unique) now that there is only one taxondetermination for each taxonoccurrence
schemas/vegbien.sql: analytical_stem_view: dateCollected: For plots data, use the locationevent obsstartdate instead of the collectiondate in order to group taxonoccurrences/stems from the same locationevent together
schemas/vegbien.sql: analytical_* pkeys: Added dateCollected because the records are actually unique within the location*event*, not the location
schemas/vegbien.sql: analytical_stem_view: Exclude records with no collectiondate or obsstartdate, which is required to uniquely identify a record
analytical_stem_view: dateCollected: Use locationevent.obsstartdate when aggregateoccurrence.collectiondate is not provided
schemas/vegbien.sql: analytical_stem_view: Include only the current taxondetermination for each taxonoccurrence, to avoid cross-joining taxondeterminations with stems and thus multiplying the number of rows for datasources that have multiple taxondeterminations per taxonoccurrence
schemas/vegbien.sql: taxondetermination: Added AFTER trigger to set the current taxondetermination for the taxonoccurrence
lib/PostgreSQL-MySQL.csv: Statements ending in ";": When matching any character, use .*? (with the (?s) flag) instead of [^;]* in order to allow embedded ; to be matched. This fixes a bug where a CREATE VIEW statement was not removed because it contained an embedded ; .
schemas/vegbien.sql: taxondetermination: Added unique index to ensure that there is only one current determination for each taxonoccurrence
lib/PostgreSQL-MySQL.csv: Remove indexes with WHERE clauses
schemas/vegbien.sql: analytical_aggregate: Added primary key on institutionCode, plotName, scientificNameWithMorphospecies, recordNumber. Note that this makes these fields NOT NULL, which should not be a problem because there are inner joins instead of LEFT JOINs on most of the tables which provide them, and LEFT JOINed tables have their identifying fields combined to create a NOT NULL value.
schemas/vegbien.sql: analytical_stem_view: recordNumber: Combine identifying fields in taxonoccurrence, plantobservation, and stemobservation to ensure that this field is unique within the plot and not NULL
lib/PostgreSQL-MySQL.csv: Only match a statement-terminating ; when it's at the end of a line
schemas/vegbien.sql: analytical_aggregate: Added primary key on institutionCode, plotName, scientificNameWithMorphospecies. Note that this makes these fields NOT NULL, which should not be a problem because there are inner joins instead of LEFT JOINs on the tables which provide them.
db_xml.py: put(): _setDefault(): Delay the evaluation of each col_default's value until the col_default is actually retrieved. This fixes a bug in the source table mappings where the explicit source entry was being created after the col_default source entry, causing the initial entry, which did not have the additional fields populated, to be used instead.
dicts.py: Added WrapDict, a dict that runs a function on each value retrieved
db_xml.py: put(): _setDefault(): Fixed bug where need to copy col_defaults before calling update() on it, to avoid modifying the input value (which may be reused by the caller, expecting it to be unmodified)
db_xml.py: put(): col_defaults param: Fixed bug where need to use None as default value, because col_defaults will be modified by put() and the {} default value is a global instance
mappings/VegCore-VegBIEN.csv: source table mappings: Set shortname to env var $source when it's not explicitly specified, because shortname is a required field of source
db_xml.py: put(): Pass through the values of nodes which are text nodes
db_xml.py: put(): put_(): Support setDefault() values which are text nodes, by passing text strings through when put() is run on all col_defaults entries
db_xml.py: put(): _setDefault(): Support setting multiple col_defaults at once by using the param names themselves as the column names
dicts.py: DictProxy: Implemented delitem()
bin/map: update_in_label(): Removed hardcoded source_id col_default, which is now set in mappings/VegCore-VegBIEN.csv's output root
mappings/VegCore-VegBIEN.csv: Set the source_id col_default to the datasource name using the new _setDefault() built-in function and _env()
db_xml.py: put(): Added _setDefault() built-in function, which adds an entry to col_defaults
xml_func.py: _env(): Fixed bug where need to retrieve actual string value of name param using xml_dom.NodeTextEntryIter instead of NodeEntryIter
xml_func.py: _env(): Fixed bug where need to use xml_dom.replace_with_text() instead of xml_dom.replace() because replace() requires a DOM node
bin/map: update_in_label(): Set $source env var to the in_label (datasource name), to make it available to _env()
xml_func.py: Simplifying functions: Added _env()
Added inputs/VegBank/Source/, containing referenceType metadata
Added inputs/SpeciesLink/Source/, containing referenceType metadata
Added inputs/SALVIAS*/Source/, containing referenceType metadata
Added inputs/REMIB/Source/, containing referenceType metadata
Added inputs/GBIF/Source/, containing referenceType metadata
Added inputs/TEAM/Source/, containing referenceType metadata
Placed inputs/TEAM/_src/Vegetation-Tree-and-Liana-Metadata-1.5.pdf under version control
inputs/FIA/import_order.txt: Added Source, which needs to come before Organism
Added inputs/Madidi/Source/, containing referenceType metadata
Added inputs/FIA/Source/, containing referenceType metadata
Added inputs/CVS/Source/, containing referenceType metadata
Added inputs/CTFS/Source/, containing referenceType metadata
bin/map: Support map spreadsheets containing only metadata mappings (with no corresponding staging table), by falling back to an empty table when the named table does not exist
mappings/VegCore-VegBIEN.csv: institutionCode: Also map to the sourcename's matched source, which identifies whether the source is a herbarium
schemas/vegbien.sql: source: Made shortname NOT NULL to ensure that all datasources have a globally-unique short name
import_all: Added import of inputs/.herbaria/ before the main import
Added inputs/.herbaria/
input.Makefile: SVN: add: Also run %/add on all data subdirs
input.Makefile: Existing maps discovery: Moved tables discovery to its own section, above SVN so it can be used by SVN
mappings/VegCore.csv: referenceType: Fixed sort order
mappings/VegCore-VegBIEN.csv: Mapped referenceType
mappings/VegCore.csv: Added referenceType
mappings/VegCore-VegBIEN.csv: institutionCode: Remap to source.shortname when specimen information is not provided, as is the case for geoscrub.herbaria on nimoy
inputs/bien_web/observation/map.csv: Mapped observationID->occurrenceID
README.TXT: Datasource setup: Add input data for each table present in the datasource: Added step to run `make inputs/<datasrc>/<table>/install` if the table is in a .sql export
README.TXT: Datasource setup: MySQL inputs: Added step to install the export, which needs to happen before mapping individual tables
README.TXT: Datasource setup: Add input data for each table present in the datasource: Replaced "CSV" with "CSV" because there can be multiple CSV part files for one table
README.TXT: Datasource setup: Add input data for each table present in the datasource: Don't add a CSV or create.sql file for tables that are in a .sql export
README.TXT: Schema changes: Sync ERD with vegbien.sql schema: Changed instructions to just select tables with arrows next to them rather than all tables, because each table that's updated will have its lines reset and the number of lines that need to be fixed should be minimized
README.TXT: Datasource setup: Accept the test cases: `make inputs/<datasrc>/test by_col=1`: Clarified that errors could indicate bugs in the VegBIEN unique constraints
README.TXT: Data import: To remake analytical DB: Added explicit public schema setting since the analytical DB is often manually remade after the public schema has been renamed. Removed warnings that certain commands must be run after running make_analytical_db, because the "remake analytical DB" instructions no longer require this.
README.TXT: Datasource setup: MySQL inputs: Added steps to export the database to a PostgreSQL-compatible .sql file, which can be directly used by the install process without the need to export each table as CSV
README.TXT: Datasource setup: Choosing a table name: Documented that for .sql exports, you must use the name of the table in the DB export, not a suggested or custom name
input.Makefile: Staging tables installation: $(dbExports): Also include the files that would be generated by running _MySQL/*.make and creating the corresponding PostgreSQL translations
input.Makefile: Staging tables installation: Moved .sql export downloading and translation to separate Input data retrieval section
Added lib/MySQL.{data,schema}.sql.make templates to use in datasources' _MySQL/ dirs
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Changed to use Brad's formula, which concatenates genus and specific_epithet/morphospecies, and uses family if just the family is present, rather than using the full taxonomic name
mappings/VegCore-VegBIEN.csv: Concatenated taxonlabel: Don't prepend family if the taxonName/scientificName itself is the family, so that the family is not duplicated in the concatenated taxonomic name
schemas/functions.sql: _nullIf(): Removed NOT NULL constraint on null param, to support use a (nullable) column rather than a literal as the null-equivalent value
xml_func.py: Simplifying functions: Added _nullIf(), to remove calls with no null value
xml_dom.py: Added prune_parent()
schemas/functions.sql: Added _or()
schemas/functions.sql: Added _merge_words()
schemas/vegbien.sql: analytical_*: Renamed geosourceValid to geovalid. (It had gotten renamed in the reference -> source rename.)
mappings/VegCore.csv: Renamed georeferenceValid to geovalid
inputs/import.stats.xls: Updated import times. This now includes the Canadensys plants-related datasources HIBG, JBM, QFA, TRT, TRTE, UBC, VASCAN, and WIN.
