Added inputs/CVS/Source/, containing referenceType metadata
Added inputs/CTFS/Source/, containing referenceType metadata
bin/map: Support map spreadsheets containing only metadata mappings (with no corresponding staging table), by falling back to an empty table when the named table does not exist
mappings/VegCore-VegBIEN.csv: institutionCode: Also map to the sourcename's matched source, which identifies whether the source is a herbarium
schemas/vegbien.sql: source: Made shortname NOT NULL to ensure that all datasources have a globally-unique short name
import_all: Added import of inputs/.herbaria/ before the main import
Added inputs/.herbaria/
input.Makefile: SVN: add: Also run %/add on all data subdirs
input.Makefile: Existing maps discovery: Moved tables discovery to its own section, above SVN so it can be used by SVN
mappings/VegCore.csv: referenceType: Fixed sort order
mappings/VegCore-VegBIEN.csv: Mapped referenceType
mappings/VegCore.csv: Added referenceType
mappings/VegCore-VegBIEN.csv: institutionCode: Remap to source.shortname when specimen information is not provided, as is the case for geoscrub.herbaria on nimoy
inputs/bien_web/observation/map.csv: Mapped observationID->occurrenceID
README.TXT: Datasource setup: Add input data for each table present in the datasource: Added step to run `make inputs/<datasrc>/<table>/install` if the table is in a .sql export
README.TXT: Datasource setup: MySQL inputs: Added step to install the export, which needs to happen before mapping individual tables
README.TXT: Datasource setup: Add input data for each table present in the datasource: Replaced "CSV" with "CSV" because there can be multiple CSV part files for one table
README.TXT: Datasource setup: Add input data for each table present in the datasource: Don't add a CSV or create.sql file for tables that are in a .sql export
README.TXT: Schema changes: Sync ERD with vegbien.sql schema: Changed instructions to just select tables with arrows next to them rather than all tables, because each table that's updated will have its lines reset and the number of lines that need to be fixed should be minimized
README.TXT: Datasource setup: Accept the test cases: `make inputs/<datasrc>/test by_col=1`: Clarified that errors could indicate bugs in the VegBIEN unique constraints
README.TXT: Data import: To remake analytical DB: Added explicit public schema setting since the analytical DB is often manually remade after the public schema has been renamed. Removed warnings that certain commands must be run after running make_analytical_db, because the "remake analytical DB" instructions no longer require this.
README.TXT: Datasource setup: MySQL inputs: Added steps to export the database to a PostgreSQL-compatible .sql file, which can be directly used by the install process without the need to export each table as CSV
README.TXT: Datasource setup: Choosing a table name: Documented that for .sql exports, you must use the name of the table in the DB export, not a suggested or custom name
input.Makefile: Staging tables installation: $(dbExports): Also include the files that would be generated by running _MySQL/*.make and creating the corresponding PostgreSQL translations
input.Makefile: Staging tables installation: Moved .sql export downloading and translation to separate Input data retrieval section
Added lib/MySQL.{data,schema}.sql.make templates to use in datasources' _MySQL/ dirs
inputs/import.stats.xls: Updated import times
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Changed to use Brad's formula, which concatenates genus and specific_epithet/morphospecies, and uses family if just the family is present, rather than using the full taxonomic name
mappings/VegCore-VegBIEN.csv: Concatenated taxonlabel: Don't prepend family if the taxonName/scientificName itself is the family, so that the family is not duplicated in the concatenated taxonomic name
schemas/functions.sql: _nullIf(): Removed NOT NULL constraint on null param, to support use a (nullable) column rather than a literal as the null-equivalent value
xml_func.py: Simplifying functions: Added _nullIf(), to remove calls with no null value
xml_dom.py: Added prune_parent()
schemas/functions.sql: Added _or()
schemas/functions.sql: Added _merge_words()
schemas/vegbien.sql: analytical_*: Renamed geosourceValid to geovalid. (It had gotten renamed in the reference -> source rename.)
mappings/VegCore.csv: Renamed georeferenceValid to geovalid
inputs/import.stats.xls: Updated import times. This now includes the Canadensys plants-related datasources HIBG, JBM, QFA, TRT, TRTE, UBC, VASCAN, and WIN.
Added inputs/HIBG/
Added inputs/JBM/
Added inputs/VASCAN/
Added inputs/WIN/
Added inputs/UBC/
Added inputs/TRTE/Specimen/
Added inputs/QFA/
Added inputs/TRT/
schemas/vegbien.sql: Allow bien_read to SELECT from all tables in the public schema
schemas/vegbien.sql: Allow bien_read to SELECT from analytical_aggregate, analytical_stem
lib/PostgreSQL-MySQL.csv: Removed GRANT/REVOKE because SCHEMA GRANTs are not supported in MySQL
pg_dump_vegbien: non-$owners mode: Removed --no-privileges in order to include GRANTs to other users
root Makefile: PostgreSQL: $(postgresReload-Linux): Making schemas/*.conf world-readable: Fixed bug where need to do this as the bien user, which owns the files
root Makefile: PostgreSQL: $(postgresReload-*): Make schemas/*.conf world-readable so it's readable by the postgres user, which the .conf installation is run as
root Makefile: PostgreSQL: $(postgresReload-*): Also install pg_hba.conf
root Makefile: PostgreSQL: Added postgres_reload to reload postgresql.conf and restart the DB
root Makefile: PostgreSQL: postgres-*: Factored postgresql.conf installation out in to $(postgresReload-*)
schemas/: Synced pg_hba.conf and pg_hba.Mac.conf's bien entries, which adds phpPgAdmin support (template1 access) on the Mac and bien_read access on Linux
root Makefile: VegBIEN DB: DB and users: Also create bien_read user for read-only access to the DB
schemas/pg_hba.Mac.conf: Allow access to the bien group rather than just the bien user, which will include bien_read
schemas/pg_hba.Mac.conf: Fixed bug where also need to allow password-based logins from the same machine, in order to work with pgAdmin
schemas/vegbien.ERD.poster.pdf: Updated to 33x51in poster size and 0.25in margins
README.TXT: Schema changes: Creating a poster of the ERD: Added section with the State St FedEx Kinkos' rates for posters ($10.25/sq ft laminated)
README.TXT: Schema changes: Creating a poster of the ERD: Changed "Measure the fractional height of the text onscreen" to "Determine the poster size"
Added schemas/vegbien.ERD.poster.pdf
Added schemas/vegbien.ERD.poster.core.print_options.png
Added schemas/vegbien.ERD.poster.core.pdf
schemas/pg_hba.Mac.conf: Fixed bug where needed ident entry for postgres superuser
Added config/bien_read_password
README.TXT: Schema changes: Added instructions to calculate the minimum VegBIEN poster size (to make the text as least as big as on the VegBank ERD poster), which is 35x54in portrait
schemas/vegbien.sql: analytical_stem_view: cultivated: Use location.iscultivated when taxonoccurrence.iscultivated is not available
Added inputs/FIA/FIA_COND_unique/, which contains the oldgrowth flag
mappings/VegCore-VegBIEN.csv: Mapped oldGrowth
schemas/functions.sql: Added _not()
mappings/VegCore.csv: Added oldGrowth
mappings/VegCore-VegBIEN.csv: Remapped cultivated to location when a TaxonOccurrence is not provided, indicating that the record is a plot
schemas/vegbien.sql: location: Added iscultivated for cases when entire plots rather than individual taxonoccurrences are marked as cultivated
inputs/FIA/: Added FIA_COND table from nimoy.geoscrub and code to generate a unique plot table from it, including the oldgrowth calculated field
Added inputs/FIA/Organism/postprocess.sql to cast PlotCD to a bigint
my2pg: Also remove (#) after bigint
Regenerated vegbien.ERD exports
schemas/vegbien.ERD.mwb: Fixed lines
schemas/vegbien.sql: source: Renamed fulltext to citation because according to the VegBank data dictionary <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=reference&entity=dba_tabledescription&where=where_tablename#fulltext> this is actually the full text of the reference citation, not of the reference itself (it would be unusual to store that in VegBank)
schemas/vegbien.sql: Removed no longer needed sourcejournal, which can be stored in source and pointed to via parent_id instead of sourcejournal_id. sourcejournal.journal maps to source.fulltext, issn to isbn, and abbreviation to shortname.
mappings/VegCore-VegBIEN.csv: Mapped acceptedCounty, county to the matched place
schemas/vegbien.sql: source: Added matched_source_id
sql.py: parse_exception(): function MissingCastException: If 1st param is hstore, only perform the cast on the value param. This fixes a bug in _map() calls whose value is a non-text type, such as SALVIAS.plotMetadata.AccessCode.
sql_io.py: cast(): Use sql_gen.Cast() to generate the cast, in order to take advantage of its support for casts to unknown
sql_gen.py: Cast: Support casts to unknown by casting to text first
schemas/postgresql.conf: Turn on the error log
schemas/pg_hba.conf: Also grant the bien user access to template1, which is accessed by phpPgAdmin
schemas/vegbien.sql: source: Added parent_id for nested sources, e.g. an article in a journal
lib/forwarding.Makefile: $(subdirs): Also exclude .archive/
Renamed inputs/_archive/ to .archive/ so it wouldn't be treated as a datasource
README.TXT: Documentation: Redmine-formatted list of steps for column-based import: Use ACAD instead of QMOR, which was removed
inputs/Makefile: Import logs: $(rsyncLogs): Include log files at any depth in the directory tree rather than just 1-2 levels deep. This adds log files whose containing directories have been moved to _archive/ directories.
Added inputs/_archive/
Removed inputs/QMOR/ because it's an insect collection
