sql_io.py: put_table(): is_function: Fixed bug where need to add the pkeys table's test pkey constraint after the data is added rather than when the empty table is created, to avoid adding a pkey constraint that will later be violated by data which returns multiple output rows for an input row (such as calls to _split())

sql_io.py: put_table(): insert_into_pkeys(): Allow callers to override run_query_into()'s add_pkey_ param in case the initial version of the pkeys table should not yet have the test pkey constraint (e.g. because data is added after the table is created)

README.TXT: Data import: Checking for errors: Search for "Command exited with non-zero status" to find errors, which is faster than checking that each input's log ends in "Encountered 0 error(s)"

inputs/import.stats.xls: Updated import times

README.TXT: Data import: import_all: Corrected text of note about time until control is returned to the shell

README.TXT: Data import: Moved download of logs to right after the import is done, because this is a quick step that doesn't depend on the backup- and export-creation steps

mappings/VegCore-VegBIEN.csv: institutionCode: Removed mapping to sourcename.matched_source_id, which is now autopopulated. Split any list of institutionCodes apart using new _split().

schemas/vegbien.sql: sourcename: Added sourcename_set_matched_source_id() trigger

schemas/functions.sql: Added _split()

schemas/vegbien.sql: sourcelist_unique: Removed COALESCE around name because it's NOT NULL

schemas/vegbien.sql: Allow multiple institutionCodes for each specimenreplicate by linking new sourcelist table many-to-many to source via sourcename (which is now a linking table)

schemas/vegbien.sql: sourcename: Removed system, which has been replaced by source_id as the scoping field

schemas/vegbien.sql: party: Added sourceaccessioncode and uniquify on it instead when provided. vegbien.ERD.mwb: Rearranged party-related tables to allow the tables to be fully expanded.

schemas/vegbien.sql: Renamed sampletype to observationtype to match the VegCore term

Added inputs/SALVIAS/salvias_users.~.clean_up.sql

inputs/SALVIAS/: Added salvias_users tables

my2pg: Translate blob to bytea

my2pg: Also remove UNIQUE and FULLTEXT inline indexes

mappings/VegCore.csv: UNUSED: Comments: Added Redmine formatting

mappings/VegCore.csv: OMIT: Changed "is omitted" to "should be omitted", because the mappings specify suggestions rather than requirements as to how a field should be used

mappings/VegCore.csv: Removed no longer used subInstitutionCode. Use datasource, institutionCode instead.

schemas/vegbien.sql: analytical_*: Renamed subInstitutionCode to institutionCode because this is the institution storing the specimen, as defined by DwC

schemas/vegbien.sql: analytical_*: Renamed institutionCode to datasource because this is actually the top-level datasource providing the record, not the institution storing the specimen

mappings/VegCore.csv: Added datasource

schemas/vegbien.sql: Renamed sampletype to observationtype to match the VegCore term

mappings/VegCore.csv: referenceType: Added closed list values

mappings/VegCore.csv: observationMeasure: Re-sourced to SALVIAS:observation_type, since SALVIAS comes before VegBIEN in the source precedence

mappings/VegCore.csv: Renamed sampleType to observationType to match the SALVIAS term it's derived from

inputs/SALVIAS-CSV/Plot/map.csv: Mapped observation_type

mappings/VegCore.csv: Added individualCount_*cm_or_more used by analytical_aggregate

mappings/VegCore.csv: subplotX/Y: Added definition

mappings/VegCore.csv: subplot*: Sources: Put SALVIAS:subplot last, because the specific field is closer in meaning to the term than the category

mappings/VegCore.csv: project*Date: Added definition

mappings/VegCore.csv: parentPlotName: Added definition

mappings/VegCore.csv: parent*: Sources: Put VegBank:PARENT_ID last, because the specific field is closer in meaning to the term than the category

mappings/VegCore.csv: locationName: Added definition

mappings/VegCore.csv: eventDate/startDate/endDate: Added definition

mappings/VegCore.csv: locationName: Sources: Put VegX:plotName first because it is closest in meaning to the term

mappings/VegCore.csv: recordedBy*: Added definition

mappings/VegCore.csv: recordedBy.middleName: Added source to DwC:recordedBy

inputs/CVS/: Joined together stemCount and stemLocation tables to create stemLocation_, in order to include the stem size class's measurements in each tagged stem's stemobservation (in addition to in the stemobservation for the aggregateoccurrence as a whole)

inputs/CVS/: Joined together taxonImportance and stemCount tables to create stemCount_, because stemCount actually stores stem abundance by size, rather than grouping stems by organism (http://vegbankdev.nceas.ucsb.edu/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename), and is thus an AggregateOccurrence-related table along with taxonImportance

inputs/CVS/: Joined together taxonImportance and stemCount tables to create stemCount_, because stemCount actually stores stem abundance by size, rather than grouping stems by organism (http://vegbankdev.nceas.ucsb.edu/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename), and is thus an AggregateOccurrence-related table along with taxonImportance

inputs/CVS/taxonObservation_/map.csv: Fixed bug where need to indicate that data is plots data to prevent the specimenreplicate ID from being forwarded to the location ID

inputs/VegBank/taxonobservation_/map.csv: Fixed bug where need to indicate that data is plots data to prevent the specimenreplicate ID from being forwarded to the location ID

mappings/VegCore-VegBIEN.csv: Don't forward specimenreplicate IDs to location for plots data (where the specimenreplicate IDs apply only to the specimen)

xml_func.py: Simplifying functions: Added _eq()

xml_func.py: Added is_scalar()

xml_func.py: process(): row-based mode: preserving complex funcs: Fixed bug where functions with no params would crash reduce() because it requires at least one value when no initial value is specified

Added scalar.py

Renamed inputs/NCU-NCSC/ to NCU because this is the primary herbarium contained in the data

inputs/VegBank/: Joined together stemcount and stemlocation tables to create stemlocation_, in order to include the stem size class's measurements in each tagged stem's stemobservation (in addition to in the stemobservation for the aggregateoccurrence as a whole)

inputs/VegBank/: Joined together stemcount and stemlocation tables to create stemlocation_, in order to include the stem size class's measurements in each tagged stem's stemobservation (in addition to in the stemobservation for the aggregateoccurrence as a whole)

inputs/VegBank/stemlocation/map.csv: Also mapped stemlocation_id to individualID to create one plantobservation for each stemobservation

inputs/VegBank/stemlocation/map.csv: Remapped stemcount_id to aggregateOccurrenceID to match stemcount_id's mapping in stemcount_

inputs/VegBank/: Joined together taxonimportance and stemcount tables to create stemcount_, because stemcount actually stores stem abundance by size, rather than grouping stems by organism (http://vegbankdev.nceas.ucsb.edu/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename)

Added inputs/VegBank/_archive

input.Makefile: Testing: Added `%/test: %/test.xml` to allow testing just a subdir

input.Makefile: General targets: Added `%/: %/map.csv` to allow remaking just a subdirectory

inputs/CVS/: Refreshed data with new export from Bob

inputs/CVS/cvs-archive-2012-12-04.schema.sql: Fixed types using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Tools#MS-Access-database-MDB>

bin/map: Removed column names simplification, which was causing columns with the same alphanumeric characters but different punctuation to be simplified to the same name. Name simplification is now performed by the mapping mechanism itself, and can be overridden in the mappings.

Regenerated inputs/VegBank/new_terms.csv

Added inputs/NCU/_src/NCU_specimens_public_2012-12-10.zip.url

inputs/NCU/: Refreshed data with new export from Bob

Renamed inputs/NCU-NCSC/ to NCU because this is the primary herbarium contained in the data

Renamed inputs/NCU-NCSC/ to NCU because this is the primary herbarium contained in the data

Added inputs/NCU-NCSC/_archive

input.Makefile: SVN: add: Also add _archive/ subdir

publish_analytical_db: Time the import of the data

export_analytical_db: Also create a .md5 for the export

export_analytical_db: Run commands in the root svn dir

mappings/VegCore.csv: soil composition terms: Removed ppm units from the definition, since units are actually fraction or percent

README.TXT: Data import: Moved On local machine steps after On nimoy steps, because the On nimoy steps are more important

mappings/VegCore.csv: Comments: Added quotes around quotations from other sources

mappings/VegCore.csv: Definitions: Added quotes around quotations from other sources

Added backups/fix_perms

backups/Makefile: Synchronization: %/download: Also download any .md5 file for the file

README.TXT: Data import: On nimoy: Added instructions to verify the export's MD5 sum

README.TXT: Data import: On nimoy: Replaced step to manually upload the analytical_aggregate export with the command to download it from jupiter

README.TXT: Data import: On nimoy: Removed step to rename any existing analytical_aggregate table, since the import is now done directly into the versioned table

mappings/VegCore.csv: VegX terms without definitions in VegX: Added definitions from non-VegX sources, etc.

README.TXT: Data import: Added instructions to verify the backups' MD5 sums on jupiter

README.TXT: Data import: Removed step to copy backups to jupiter, because this now done by `make backups/upload`

schemas/vegbien.sql: sync_*_to_view(): Also add `GRANT SELECT TO bien_read` on the view used to generate the table, in case the permission was lost when the view was modified

schemas/vegbien.sql: sync_*_to_view(): Added `GRANT SELECT TO bien_read`

schemas/vegbien.sql: analytical_*: Added back bien_read's SELECT permissions, which had gotten removed when the tables were re-synced to their views

schemas/vegbien.my.sql: Regenerated with expanded repl word matching

repl: :-prefixing of words to form vars: Fixed bug where : must be matched as a lookbehind assertion, not a capturing group, because the provided regexp itself or its replacement may reference capturing groups, which it expects to be numbered starting with 1

inputs/import.stats.xls: Updated import times

Regenerated inputs/NY/Specimen/new_terms.csv

inputs/JBM/Specimen/test.xml.ref: Updated inserted row count, which had gotten changed when a test was run on a non-empty database

mappings/VegCore.csv: height_ft: Added source to VegBank:stemHeight, which includes a description of the term

mappings/VegCore.csv: height_m: Added source to VegBank:stemHeight, which includes a description of the term

mappings/VegCore.csv: projectName: Added definition from VegX schema

mappings/VegCore.csv: project*Date: Re-sourced to VegBank:project.*Date, since VegX does not have an equivalent term

mappings/VegCore.csv: VegX terms: Added definitions from VegX schema, where provided

mappings/VegCore.csv: projectName: Added source to VegX:project.title

mappings/Makefile: .VegCore.csv.last_cleanup, .Veg+-VegCore.csv.last_cleanup: Also replace Veg+ terms in sources list, which are references to VegCore terms that have since been renamed

repl: text mode: Also match "vars" with the term prefixed by ":". Consider .- to be word characters. Only match a word when preceeded by whitespace or CSV field start characters.