Added inputs/ARIZ/ARIZ_DiGIR_21012010.csv.tar.gz.url
Added inputs/UNCC/Specimen/UNCC.csv.url, UNCC.csv.md5
Added inputs/XAL/_src/digir.xml.gz.md5
Added inputs/UNCC/_src/ with UNCC.csv.zip.md5
Added inputs/SpeciesLink/_src
README.TXT: Datasource setup: MySQL inputs: .sql exports: Use new mysql_bien to connect to the MySQL DB created for the datasource
Added mysql_bien, which runs a MySQL command on the local MySQL server
Added inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump.md5 (md5sum of the expanded file)
root Makefile: MySQL: mysql-Linux: Also install phpMyAdmin
root Makefile: MySQL: mysql-Linux: Split apt-get dependencies into separate commands, like for other apt-get commands, to avoid having one failed dependency prevent the following dependencies from being installed
root Makefile: MySQL: *mysql_users: Also add bien_read user
root Makefile: MySQL: Renamed *mysql_user to *mysql_users because there can be multiple users
inputs/: Added .md5 files for all .zip, .gz
Added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.url
lib/common.Makefile: rsync: $(rsync*): Use --no-group because the file group is different depending on the machine
input.Makefile: SVN: $(_svnFilesGlob): Also add .md5 files. This allows svn to track where unversioned files should be in the directory tree.
input.Makefile: SVN: $(_svnFilesGlob): .url, .pdf, and README.TXT in the top-level dir: Fixed bug where had extra / after brace expr
input.Makefile: SVN: $(_svnFilesGlob): Also add .url, .pdf, and README.TXT in the top-level dir
input.Makefile: SVN: $(_svnFilesGlob): Add .url, .pdf, and README.TXT files in all subdirs, not just _src
lib/common.Makefile: remote server: Use jupiter instead of vegbiendev, to ensure that all files get uploaded there rather than only to vegbiendev. This involves adding an extra database import step to download the uploaded files from jupiter onto vegbiendev.
inputs/FIA/_src/Makefile: all: Extract zip files before running tables target, because it requires the created dirs
schemas/vegbien.ERD.mwb: Fixed table sizes
Removed no longer used fix_permissions. Use root fix_perms instead.
Added root fix_perms
Moved Checksums from backups/Makefile to lib/common.Makefile so all dirs (including inputs/) can use md5sum testing
lib/common.Makefile: $(remote): Made remote basepath configurable in $(remote_basepath)
lib/common.Makefile: Renamed $(src_server) to $(remote_host) and $(src_user) to $(remote_user) for clarity
inputs/GBIF/: Added refresh metadata
Added inputs/HVAA/
Added inputs/ARIZ/_archive
inputs/ARIZ/: Removed previous data now that it has been refreshed
inputs/ARIZ/: Mapped refresh
Added inputs/ARIZ/import_order.txt
Added inputs/NY/_archive/
inputs/NY/: Removed tables from previous extract
inputs/NY/: Mapped refresh
inputs/*/*/VegBIEN.csv: Regenerated from mappings/VegCore-VegBIEN.csv
Added inputs/NY/import_order.txt
inputs/ARIZ/: Added SQL export for refresh
my2pg.data: Translate indefinite (zero) months which have a definite day. This is unusual, but does appear in some data such as the ARIZ DB.
my2pg.data: Translate indefinite dates (dates with 0 as the month or day)
my2pg: Use my2pg.data to perform data-only replacements, instead of duplicating them in both my2pg and my2pg.data
my2pg: named UNIQUE KEYs: Comment out the name because PostgreSQL requires it to be globally unique, but MySQL only requires it to be unique within the table
my2pg: Translate UNIQUE KEYs instead of removing them
my2pg*: Removed KEYs: Comment out the definition rather than removing it
my2pg*: Remove FOREIGN KEYs because MySQL does not dump tables in dependency order, which prevents PostgreSQL from creating tables whose fkeys refer to a later table
my2pg*: Replacing invalid table elements to remove them: Use a dummy CHECK constraint instead of a boolean field to avoid adding fields to the table. The elements can't always simply be removed because sed can't remove the trailing comma of the previous element, and removing the following comma doesn't work for the last element in the table.
my2pg*: Replace '0000-00-00 00:00:00' with '-infinity'
my2pg: Replace datetime with timestamp
my2pg: Remove COLLATE field attribute
lib/MySQL.*.sql.make: Documented that $server user/host are for ssh, not the DB
lib/MySQL.*.sql.make: Documented that $server can also contain a username (which will be used by ssh)
my2pg_export: Use the --quick option to facilitate exporting large tables (it avoids retrieving all rows before outputting any of them)
README.TXT: Datasource setup: Added instructions for MS Access databases
README.TXT: Datasource setup: MySQL inputs: Added instruction to skip the Add input data for each table section
inputs/NY/: Added SQL export for refresh
mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.
inputs/CVS/observation_/map.csv: baseSaturation: Resolved ambiguous term
mappings/Makefile: VegCore.vocab.csv: Ignore leading ? when sorting so that ambiguous terms sort alphabetically with other terms. This prevents terms from moving from their previous location when they become ambiguous.
Added sort_ci to sort a spreadsheet, ignoring leading punctuation
mappings/VegCore.vocab.csv: Changed line endings to \r\n in preparation for having a Python script run on it (which changes the line endings)
mappings/Makefile: VegCore.vocab.csv: Added back ambiguous terms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are ambiguous or unambiguous terms
mappings/Makefile: VegCore.vocab.csv: Added back synonyms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are synonyms or primary terms. This also prevents VegCore.vocab.csv from losing entries when terms are renamed, which made it difficult to verify that no terms were lost when refactoring.
inputs/MO/Specimen/postprocess.sql: Remove frameshifted rows by detecting InstitutionCodes without any letters
inputs/ARIZ/Specimen/map.csv: CollectorNumber/FieldNumber: Use /_first to map these identical fields to the same location
inputs/ARIZ/Specimen/map.csv: Fixed bug where the column names for InstitutionCode and CollectionCode were reversed in the source data
inputs/*/Specimen/map.csv for Canadensys sources: Remapped institutionID to UNUSED
mappings/VegCore.htm: Regenerated from wiki. The original*, accepted*, and verbatim* Taxon fields have now been moved to separate OriginalTaxonDetermination, AcceptedTaxonDetermination, and TaxonVerbatim tables.
mappings/VegCore.htm: Regenerated from wiki
README.TXT: Maintenance: VegCore data dictionary: Replaced VegCore.*.csv with VegCore.htm because now that VegCore.*.csv are sorted alphabetically, they generally don't change when VegCore.htm changes
mappings/VegCore.*.csv: Regenerated from wiki. A plain text label is now used for Replace with, which fixes a bug where the PRIVATE permalink pointed to its Replace with in realLatitude instead of its definition.
redmine_synonyms: Support plain text labels other than Alternative, such as Replace with
mappings/VegCore.*.csv: Regenerated from wiki. Alternatives now contain the "Alternative" label as plain text rather than as an image title, thus avoiding an HTML anchor conflict with the definition and allowing ambiguous terms to be placed before their alternatives as well as after.
README.TXT: Maintenance: VegCore data dictionary: Updated VegCore.csv filename to VegCore.*.csv
redmine_synonyms: Support alternatives which contain the "Alternative" label as plain text rather than as an image title. This is done to include the "Alternative" label in the HTML anchor and thus prevent the anchor from conflicting with the actual definition of the alternative (which would otherwise have the same anchor text). This allows ambiguous terms to be placed before their alternatives as well as after, because there won't be anchor conflicts that need to be resolved with careful ordering.
mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch
inputs/bien_web/observation/test.xml.ref: Regenerated
inputs/import.stats.xls: Updated import times using the import_times bugfix for times longer than a day
import_times: times(): Fixed bug where need to match whitespace in times, in order to match times with days
inputs/*/Specimen/map.csv: Remapped ID to datasourceRecordID
mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID
inputs/import.stats.xls: Updated import times
inputs/FIA/_src/_README.TXT: Documented that the refresh is missing some PLT_CN values present in the original version
inputs/FIA/import_order.txt: Reverted back to using FIA_COND_unique instead of COND_unique because the PLT_CN IDs in the refresh don't match the PLT_CN IDs in the original version, making COND_unique and Organism incompatible
inputs/FIA/import_order.txt: Removed FIA_COND_unique, which is superseded by COND_unique
inputs/FIA/import_order.txt: Fixed bug where need to import COND_unique before Organism because the plot entries need to be created before they can be linked to by organisms
redmine_synonyms: sed pattern: Match <h# directly at the beginning of the line rather than after ^.*, which greatly speeds up the pattern matching because the first character is a literal character. (If <h# were not located at the left margin, the ^.* would unfortunately still be needed because the beginning of the line needs to be matched in order to be removed by the replacement operation.)
mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.
mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
redmine_synonyms: Don't require ambiguous terms to start with ?, because the ambiguous term for an alternative can be identified simply by choosing the last term that didn't have a type label (previously, this would have been the last term that wasn't h3 or h4)
redmine_synonyms: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).
mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.
mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match
Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity
mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary
