schemas/vegbien.sql: sync_*_to_view(): Added `GRANT SELECT TO bien_read`
schemas/vegbien.sql: analytical_*: Added back bien_read's SELECT permissions, which had gotten removed when the tables were re-synced to their views
schemas/vegbien.my.sql: Regenerated with expanded repl word matching
repl: :-prefixing of words to form vars: Fixed bug where : must be matched as a lookbehind assertion, not a capturing group, because the provided regexp itself or its replacement may reference capturing groups, which it expects to be numbered starting with 1
inputs/import.stats.xls: Updated import times
Regenerated inputs/NY/Specimen/new_terms.csv
inputs/JBM/Specimen/test.xml.ref: Updated inserted row count, which had gotten changed when a test was run on a non-empty database
mappings/VegCore.csv: height_ft: Added source to VegBank:stemHeight, which includes a description of the term
mappings/VegCore.csv: height_m: Added source to VegBank:stemHeight, which includes a description of the term
mappings/VegCore.csv: projectName: Added definition from VegX schema
mappings/VegCore.csv: project*Date: Re-sourced to VegBank:project.*Date, since VegX does not have an equivalent term
mappings/VegCore.csv: VegX terms: Added definitions from VegX schema, where provided
mappings/VegCore.csv: projectName: Added source to VegX:project.title
mappings/Makefile: .VegCore.csv.last_cleanup, .Veg+-VegCore.csv.last_cleanup: Also replace Veg+ terms in sources list, which are references to VegCore terms that have since been renamed
repl: text mode: Also match "vars" with the term prefixed by ":". Consider .- to be word characters. Only match a word when preceeded by whitespace or CSV field start characters.
repl: column mode: Removed parsing and checking of column name, which prevents using repl for general-purpose regexp/word replacement
mappings/VegCore.csv: Definition: Moved closed list values to new Values column
mappings/VegCore.csv: Added Values column to store closed list values
mappings/VegCore.csv: geovalidation terms: Removed source to DwC:georeferenceVerificationStatus, because that is for georeferencing, not geovalidation
mappings/VegCore.csv, Veg+-VegCore.csv: obs*Date: Re-sourced to VegX:obs*Date
mappings/VegCore.csv: projectID: Re-sourced to plotObservation.projectID
dict2redmine: RedmineTableWriter: Fixed bug where need to escape embedded | , using new redmine_table_esc()
dict2redmine: Added redmine_table_esc()
dict2redmine: Added redmine_esc()
mappings/VegCore.csv: TCS terms: Added TCS comments from <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegBIEN_taxonomic_schema#TCS>
dict2redmine: redmine_add_links(): Include the [] in the link text, to avoid the need for redmine_pad(), etc.
dict2redmine: redmine_add_links(): Make the link bold so it stands out as a link
dict2redmine: redmine_add_links(): Use new redmine_pad()
dict2redmine: Added redmine_pad()
dict2redmine: redmine_add_links(): Use redmine_url() to create the internal link
dict2redmine: redmine_url(): Support internal links
dict2redmine: redmine_add_links(): Fixed bug where need to explicitly specify the source name as the link text
dict2redmine: RedmineDictWriter: Link citations to entry in sources list
mappings/VegCore.csv: Restored name of latLongDomainValid term, which had gotten replaced with coordinatePrecision
mappings/VegCore.csv: startDate, endDate: Changed comment to "a date range usually applies to the event"
mappings/VegCore.csv: Added Examples column to store data in TCS Examples column at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegBIEN_taxonomic_schema#TCS>
mappings/VegCore.csv: non-phylogenetic taxonomic terms: Added definitions from TCS schema
mappings/VegCore.csv: *forma, *variety: Fixed sources, which had been swapped between the two sets of terms
mappings/VegCore.csv: Special values: Moved comments to Comments column
dict2redmine: Fixed bug where all header fields need to be preserved because columns are now filtered out instead of removed in each row
dict2redmine: Put the definition before and outside of the fields table
mappings/VegCore.csv: Moved Definition values that are actually comments into separate Comments column
dict2redmine: RedmineDictWriter: Omit empty columns from the fields table
dict2redmine: Generate an outline instead of a table so each term will be indexed in the page's table of contents
schemas/vegbien.sql: coordinates: coordinates_unique: Removed md5() around verbatimcoordinates because functions within unique indexes (other than the standard COALESCE) are not yet supported by the import algorithm
exc.py: e_msg(): Emit a warning instead of an AssertionError if e.args0 isn't a string, to assist in debugging malformed exceptions
mappings/VegCore.csv: sampleType: Re-sourced to bien_web.observationType
schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use the taxonomicname in accepted_taxonlabel instead of accepted_taxonverbatim, because taxonverbatim only contains fields provided by the data provider (in this case, TNRS), but TNRS does not provide the taxonomic name (taxon name+author), only the taxon name and author components separately
schemas/vegbien.sql: coordinates: coordinates_unique: Use md5() on verbatimcoordinates so that it doesn't cause the index row size to be exceeded. This should fix a bug in the HIBG import where long verbatimcoordinates values were causing the error 'OperationalError: index row size 2784 exceeds maximum 2712 for index "coordinates_unique"'.
backups/Makefile: Synchronization: Replaced download target, which downloads all backups, with %/download, which downloads just a specific backup, because you would generally only want to extract a single backup from the archive for reinstallation
backups/Makefile: Synchronization: Sync with jupiter instead of vegbiendev. This requires running `make backups/upload` on vegbiendev to archive the files, instead of `make backups/download` to download them to your local machine.
inputs/.geoscrub/geoscrub_output/map.csv: Removed no longer accurate comment that county is not yet used by VegBIEN
inputs/.geoscrub/geoscrub_output/map.csv: *validity: Remapped 2 ("Point is <=5km from putative GADM polygon, but still outside it") to true instead of false, because 5km is close enough to the polygon that the mismatch could result from shapefile simplifying, boundary changes, or other factors that don't affect geovalidity
inputs/.geoscrub/geoscrub_output/map.csv: *validity: Remapped 0 ("Complete name provided, but couldn't be scrubbed to GADM") to NULL instead of false, because the absence of a name match does not mean the coordinates are invalid
inputs/.{NCBI,TNRS}/import_order.txt: Added Source
input.Makefile: SVN: add: Add a Source table to store datasource metadata. This adds a Source table to all herbaria which are listed in .herbaria, and therefore didn't previously need a Source table to indicate their referenceType and sampleType.
inputs/input.Makefile: SVN: add: verify/: Added *.xls to svn:ignore
inputs/.geoscrub/geoscrub_output/postprocess.sql: Added index on decimallatitude, decimallongitude
Added inputs/.geoscrub/geoscrub_output/postprocess.sql, which adds NOT NULL constraints on decimallatitude, decimallongitude
schemas/vegbien.sql: analytical_*: Changed type of boolean columns to integer so that they will be exported as 1/0 instead of t/f by export_analytical_db. This will enable MySQL's LOAD DATA INFILE to import the values correctly.
backups/Makefile: Checksums: %.md5/test: Only use md5sum's -v option on Mac, because it's not supported on Linux (there, verbose mode is the default)
mappings/VegCore.csv: cultivated* source: Added picklist value to URL
README.TXT: Data import: On nimoy: Creating analytical_aggregate table: publish_analytical_db: Rewrapped line
README.TXT: Data import: On nimoy: Creating analytical_aggregate table: Changed name to analytical_aggregate_r<revision> to allow storing different versions simultaneously
publish_analytical_db: Require caller to specify the name of the table to load data into. This allows appending a revision to analytical_aggregate, or publishing a table other than analytical_aggregate.
backups/Makefile: SQL: Full DB: vegbien.%.backup: Also generate MD5 sum
README.TXT: Data import: Delete previous imports based on the full DB backup file
backups/Makefile: Support removing public schema versions based on the version of a full DB backup
mappings/VegCore.csv, Veg+-VegCore.csv: Removed the additional dict namespace for the SALVIAS sources. This removes the extra "dict:" namespace on the generate Redmine source term names.
mappings/VegCore.csv, Veg+-VegCore.csv: Added TNRS provider namespace, inserting it before BIEN in the sort order
mappings/VegCore.csv: Changed + to _ in URL fragments
mappings/VegCore.csv, Veg+-VegCore.csv: Removed the additional BIEN namespace for the BIEN sources, and use just BIEN2 and VegBIEN as the sub-namespaces. This removes the extra "BIEN:" namespace on the generate Redmine source term names.
mappings/VegCore.csv, Veg+-VegCore.csv: Removed the "terms" text in the current DwC terms' provider, and leave just the sort order. This removes the extra "terms:" namespace on the generate Redmine source term names.
dict2redmine: url_term(): Remove empty URL comments
dict2redmine: url_comment_text(): Interpret a URL comment containing just a number as a sort order without text
dict2redmine: url_term(): Prefix any provider in the URL to the term name, to create a namespace. Each hierarchical component of the provider is stored in a URL comment.
dict2redmine: Added url_comment_re
dict2redmine: Added url_comment_text()
dict2redmine: Call simplify_url() just on the first source so that source2redmine_url() can use the raw URL (to extract comments, etc.)
dict2redmine: Removed no longer used explicit Definition column #
dict2redmine: Use the input spreadsheet's column names and order, and pass through columns other than the term and sources columns
mappingsf/VegCore.csv, Veg+-VegCore.csv: Renamed Comments to Definition to match Redmine table
mappings/VegCore.csv, Veg+-VegCore.csv: Reversed order of Comments, Sources columns to match Redmine table order
dict2redmine: Store term_str in a var before using it, like sources_str
dict2redmine: Added Definition column
dict2redmine: Take term and sources col #s as args instead of hardcoding them by column name or position
dict2redmine: url_term(): Also match any namespace that's part of the term
dict2redmine: Sources: Use source2redmine_url() to extract the term from each source URL
dict2redmine: source2redmine_url(): Support empty URLs
dict2redmine: url_term(): Fixed bug where need to use match.group() instead of match.groups()
mappings/Makefile: Create VegCore.redmine from VegCore.csv
Added dict2redmine
mappings/VegCore.csv, Veg+-VegCore.csv: Renamed Source column to Sources because it can contain multiple sources
mappings/VegCore.csv, Veg+-VegCore.csv: Source: DwC terms: Scoped sort order by category, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore_refactoring#Scope-DwC-sort-order-by-category>
mappings/VegCore.csv, Veg+-VegCore.csv: Source: VegX terms: Split combined field group/field sort order into separate sort orders for field and field group
