Activity
From 02/04/2013 to 03/05/2013
03/05/2013
- 11:07 PM Revision 7848: *.sql: concat_ws() calls: Fixed bug where need to surround it with NULLIF(..., '') because concat_ws() returns '' instead of the required NULL when supplied with only NULL args
- 10:22 PM Revision 7847: schemas/vegbien.sql: analytical_stem_view: concat_ws() calls: Fixed bug where arguments were still enclosed in ARRAY[], because the name had been changed from concat_delim() but PostgreSQL removed the VARIADIC keyword without also removing the ARRAY[] wrapper
- 10:11 PM Revision 7846: schemas/vegbien.sql: Removed no longer needed unscrubbed_taxondetermination_view. Use taxonlabel joined to ScrubbedTaxon instead.
- 10:05 PM Revision 7845: inputs/.TNRS/schema.sql: ScrubbedTaxon: Include only matches with a match score of at least 0.8 (score_ok())
- 09:59 PM Revision 7844: inputs/.TNRS/schema.sql: Added back score_ok() and tnrs.tnrs_score_ok index
- 09:53 PM Revision 7843: *.sql: Replaced concat_delim() with concat_ws(), which is built-in as of PostgreSQL 9.1 (http://stackoverflow.com/questions/1943433/postgresql-concat-ws-like-function)
- 09:08 PM Revision 7842: inputs/U/Specimen/map.csv: Populated specimenHolderInstitutions
- 09:07 PM Revision 7841: mappings/VegCore.htm: Regenerated from wiki. Renamed specimenHolders to specimenHolderInstitutions to make it obvious that this is a list of institutions, such as would be in institutionCode in a DwC export.
- 09:06 PM Revision 7840: inputs/SpeciesLink/Specimen/map.csv: Fixed embedded line endings
- 09:05 PM Task #577 (Rejected): use views instead of map spreadsheets to store the datasource mappings
- * this allows the contents of the view to be appended directly to the flat Occurrence table, with the column names ma...
- 08:28 PM Revision 7839: inputs/NY/Ecatalog_all/map.csv: PlantFungDescription->occurrenceRemarks: Removed "plant fungus" label because this field actually contains the specimenDescription
- 08:06 PM Revision 7838: inputs/.TNRS/import_order.txt: Removed public.unscrubbed_taxondetermination_view
- 08:01 PM Revision 7837: inputs/.TNRS/: Moved no longer needed public.unscrubbed_taxondetermination_view/ to _archive/
- 07:55 PM Task #565 (Resolved): partition the TaxonDetermination table by row into the different types of determinations
- See TNRS.AcceptedTaxon, MatchedTaxon, ScrubbedTaxon
- 07:52 PM Revision 7836: root Makefile: Datasources: scrub: Removed no longer needed scrub-remake, which has been replaced by direct use of TNRS.ScrubbedTaxon
- 07:46 PM Revision 7835: schemas/vegbien.sql: analytical_stem_view: taxondetermination: Fixed bug where need to include only the is_datasource_current taxondetermination
- 07:43 PM Revision 7834: schemas/vegbien.sql: analytical_stem_view: Use new TNRS.ScrubbedTaxon directly instead of the scrubbed taxondeterminations added to the normalized DB
- 07:20 PM Revision 7833: inputs/.TNRS/schema.sql: MatchedTaxon: Renamed matchedTaxonomicStatus to preferred name taxonomicStatus
- 07:19 PM Revision 7832: mappings/VegCore.htm: Regenerated from wiki. Taxon: Moved scrubbing-related terms to MatchedTaxon.
- 07:16 PM Revision 7831: inputs/.TNRS/schema.sql: Added matched* prefix where needed
- 07:02 PM Revision 7830: inputs/.TNRS/schema.sql: MatchedTaxon: Added missing Name_submitted column
- 06:59 PM Revision 7829: inputs/.TNRS/schema.sql: MatchedTaxon: Name_matched: Fixed mapping to matchedTaxonName
- 06:56 PM Revision 7828: inputs/.TNRS/schema.sql: MatchedTaxon: Fixed bug where can't map Source to source because it refers just to the source of the matched name, rather than of the record (which would be TNRS itself)
- 06:55 PM Revision 7827: inputs/.TNRS/schema.sql: tnrs_populate_derived_fields(): Use new concat_delim()
- 06:43 PM Revision 7826: schemas/functions.sql: _join*(): Use new concat_delim()
- 06:41 PM Revision 7825: schemas/functions.sql: Added concat_delim() (also in public, but this copy is for use by the functions schema)
- 06:37 PM Revision 7824: inputs/.TNRS/schema.sql: Removed no longer used tnrs+accepted. Use ScrubbedTaxon instead.
- 06:22 PM Revision 7823: inputs/.TNRS/schema.sql: Added MatchedTaxon, AcceptedTaxon, ScrubbedTaxon views, which rename the columns to VegCore names
- 05:17 PM Revision 7822: Removed inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump*, which has been replaced by GBIFPortalDB-2013-02-20.dump
- 05:14 PM Revision 7821: mappings/VegCore.htm: Regenerated from wiki. Renamed MatchedTaxonDetermination/AcceptedTaxonDetermination to MatchedTaxon/AcceptedTaxon because they relate to a taxon *name* rather than to an application of that name to a TaxonOccurrence
- 02:43 PM Revision 7820: schemas/functions.sql: functions._map(): Made function STRICT because it always returns NULL if either argument is NULL
- 02:39 PM Revision 7819: inputs/UNCC/Specimen/map.csv: campus: Mapped to cultivated, interpreting non-NULL values as true
- 02:29 PM Revision 7818: inputs/UNCC/Specimen/map.csv: campus: Removed mapping to occurrenceRemarks because this is just a code, not a description. Added definition from Bob's validation comments.
- 02:19 PM Revision 7817: inputs/UNCC/Specimen/map.csv: accession: Removed comment that this is actually the catalogNumber, because they are now synonyms
- 01:56 PM Revision 7816: mappings/VegCore.htm: Regenerated from wiki. Split specimenIndexer into specimenStampingInstitution and specimenDonor (which are often the same).
- 12:25 PM Revision 7815: mappings/VegCore-VegBIEN.csv: Removed specimenIndexer->institutionCode mappings because the institutionCode should refer only to the specimenHolder
- 12:18 PM Revision 7814: Added inputs/GBIF/_src/GBIFPortalDB-2013-02-20.dump.gz (a new download they posted to the FTP site)
- 12:13 PM Revision 7813: schemas/vegbien.sql: analytical_specimen: Removed growthForm, threatened_bien at Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#UNCC > Brad Boyle's comments > growthForm, threatened_bien)
- 11:24 AM Revision 7812: inputs/UNCC/Specimen/map.csv: Removed mappings to reproductiveCondition because we don't know what the single-letter codes mean
- 11:16 AM Revision 7811: inputs/UNCC/Specimen/map.csv: Fixed embedded line endings
- 11:05 AM Revision 7810: schemas/vegbien.sql: _locationnarrative_is_cultivated(): Fixed bug where need to use NULL instead of false (and bool_or() instead of OR) for a non-match, because the absence of a match doesn't mean it's not cultivated
- 05:09 AM Task #576 (New): merge the datasources' mappings into the VegCore data dictionary
- * each mapping defines a synonym of a VegCore term
- 05:07 AM Task #468 (Resolved): rename/delete and recreate public schema in one transaction
- 05:04 AM Task #340 (Resolved): put together definitions of abstract BIEN terms
- See [[VegCore data dictionary]]
- 03:02 AM Revision 7809: mappings/VegCore.htm: Regenerated from wiki. Editing: Changed ambiguous terms templates to list the name of the alternative itself in the alternatives list, instead of the ambiguous term + a source suffix. This avoids the need to choose a source suffix for each alternative of an ambiguous term, as the ambiguous term itself (without the suffix) can be listed as the synonym. Coordinates: Disambiguated verbatimCoordinateSystem.
- 02:30 AM Revision 7808: mappings/Makefile: VegCore.thesaurus.csv: `uniq`ify the map after applying initial translations, to support mappings that appear twice (once as the result of an alternative's synonym entry and again as the result of an ambiguous term's alternatives listing)
03/04/2013
03/01/2013
- 08:17 PM Revision 7806: mappings/VegCore.htm: Regenerated from wiki. Added specimenDescription.
- 08:00 PM Revision 7805: mappings/VegCore.htm: Regenerated from wiki. Split apart identificationQualifier and taxonFit. Split apart taxonConfidence and identificationVerificationStatus. Moved identificationQualifier to new QualifiedTaxonName table.
- 07:48 PM Revision 7804: README.TXT: Full database import: To view progress: Use tail -100 to show more of the log files, and not tail -f to avoid blocking when the user just wants to check the status
- 04:56 PM Revision 7803: inputs/BRIT/specimen_flat/map.csv: Only map Notes_Plant to occurrenceRemarks if there is a separate Locality_Description, in order to filter out Notes_Plant values that concatenate everything together (often including a herbarium name that would mess up the cultivated flag parsing)
- 04:49 PM Revision 7802: my2pg.data: Replace \0 (with the backslash) with 0 because \0 isn't allowed in UTF-8
- 03:53 PM Revision 7801: root Makefile: VegBIEN DB: mk_db: Added command to import arbitrary DB users from a config file
02/28/2013
- 03:31 PM Revision 7800: inputs/import.stats.xls: Updated import times
- 03:29 PM Revision 7799: inputs/TEX/Specimen2/map.csv: _join() PLANT DATA and MISC NOTES to form occurrenceRemarks instead of _alt()ing them
- 03:27 PM Revision 7798: inputs/TEX/Specimen2/test.xml.ref: Accepted test
- 02:58 PM Revision 7797: inputs/BRIT/_MySQL/MySQL.*.sql.make: Include all tables, not just specimen_flat
- 02:57 PM Revision 7796: my2pg: Replace tinytext, mediumtext with text
- 11:48 AM Revision 7795: Added inputs/GBIF/_src/_README.TXT with Brad's GBIF filtering rules
- 11:30 AM Revision 7794: inputs/FIA/TREE/postprocess.sql: Use :table instead of hardcoding the table name
- 11:26 AM Revision 7793: inputs/FIA/*/postprocess.sql: UNIQUE constraint: Added UNITCD because although it may not be needed to scope PLOT, it is present in all the core tables and thus must be part of the joins' USING columns
- 10:58 AM Revision 7792: inputs/FIA/*/postprocess.sql for core tables: Added unique constraint on the datasource's primary key fields
- 02:38 AM Revision 7791: Added inputs/CTFS/AggregateObservation/ from BIEN2 data
- 02:16 AM Revision 7790: inputs/CTFS/: Switched global _no_import to table-specific _no_imports to allow adding new tables that *are* imported
02/27/2013
- 09:54 PM Revision 7789: Added schemas/TAPIR/SpeciesLink.tapir_request.txt symlink
- 09:53 PM Revision 7788: Added inputs/SpeciesLink/_src/tapir_request.txt
- 09:52 PM Revision 7787: Added schemas/TAPIR/
- 09:48 PM Revision 7786: Added planning/use_cases/rare_species.png
- 09:47 PM Revision 7785: Added planning/goals/BIEN deliverables.png
- 09:31 PM Revision 7784: Added planning/resources/iPlant_Overview_BIEN_2011.pdf.url
- 09:29 PM Revision 7783: planning/goals/Enquist_BIEN_Overview_2011.pptx: Replaced with .url file. Note that the Plone site <http://groups.nceas.ucsb.edu/bien/meetings> is not searchable by document filename.
- 09:26 PM Revision 7782: Added planning/workflow/Boyle_BIEN_data_quality.pdf.url
- 09:21 PM Revision 7781: Added planning/goals/BIEN_Science_2011.pptx.url, BIEN-Tech-Oct2011.ppt.url, Enquist_BIEN_Overview_2011.pptx
- 09:00 PM Revision 7780: Added schemas/VegX/run.py
- 08:53 PM Revision 7779: Added schemas/VegBIEN/growth_forms/Barbara_Dobrin.parsing.scripts/
- 08:35 PM Revision 7778: Added schemas/VegX/Veg-X_paper_in_press_Applied_Vegetation_Science.pdf.url
- 08:25 PM Revision 7777: Added schemas/VegX/JVS-VegX.pdf.url
- 08:22 PM Revision 7776: Added schemas/VegBank/VegBank_Data_Model.with_Collections_intro.ppt
- 08:13 PM Revision 7775: Added planning/goals/BIEN_White_Paper.pdf.url
- 08:11 PM Revision 7774: Added planning/meetings/2010-12-08.working_group/BIEN3-plan.docx
- 08:04 PM Revision 7773: Added schemas/VegBIEN/planning/taxonomy/On_the_Use_of_Taxonomic_Concepts_in_Support_of_Biodiversity_Research_and_Taxonomy.pdf.url
- 07:59 PM Revision 7772: Added schemas/BIEN2/ERD/
- 07:54 PM Revision 7771: Added schemas/BIEN2/bien2_core_schema.png
- 07:54 PM Revision 7770: Added schemas/BIEN2/mappings/BIEN2.viewFullOccurrence-DwC.xls
- 07:51 PM Revision 7769: Added schemas/VegBank/mappings/VegBank.observation-VegX.plotObservation.xls
- 07:49 PM Revision 7768: schemas/vegbien.ERD.mwb: Fixed table positions due to plantobservation field addition
- 07:35 PM Revision 7767: Added schemas/DiGIR/gbif-occurrencestore.digir.client.url
- 07:33 PM Revision 7766: Added schemas/DiGIR/gbif-crawler.digir.*.url
- 07:27 PM Revision 7765: inputs/.geoscrub/import_order.txt: Added Source
- 07:09 PM Revision 7764: Added schemas/VegX/
- 07:00 PM Revision 7763: Added schemas/VegBIEN/taxonomy/family_higher_plant_group.sql
- 06:57 PM Revision 7762: Added planning/use_cases/Trait_BIEN_data_use_case.docx
- 06:56 PM Revision 7761: Added planning/workflow/2010.working_group.white_board/
- 06:54 PM Revision 7760: Added planning/milestones/_archive/BIEN3 Milestones and Development Timeline_MLN2.*
- 06:51 PM Revision 7759: Added planning/workflow/BIEN-modArch-Dec2010 NS-SBD 1.4.ppt
- 06:50 PM Revision 7758: Added planning/milestones/_archive/BIEN3_milestones.2011-12-13.doc
- 06:48 PM Revision 7757: Added planning/milestones/BIEN Milestone – feedback.docx
- 06:46 PM Revision 7756: Added schemas/VegBIEN/planning/taxonomy/Nomenclature_excerpt.ppt
- 06:44 PM Revision 7755: Added planning/use_cases/
- 06:42 PM Revision 7754: Added planning/meetings/2012-11-26~30.working_group/
- 06:41 PM Revision 7753: Added schemas/BIEN2/Guide_to_the_BIEN2_database.pdf
- 06:39 PM Revision 7752: Added planning/meetings/2011-10-24~28.working_group/
- 06:36 PM Revision 7751: Added schemas/VegBIEN/planning/BIEN_OctWG-Whtbrd*.jpg
- 06:32 PM Revision 7750: Added schemas/BIEN2/
- 06:29 PM Revision 7749: Added schemas/VegBIEN/
- 06:20 PM Revision 7748: Moved schemas/vegbank.* to schemas/VegBank/
- 06:15 PM Revision 7747: Added planning/milestones/BIEN3 - Progress since the 2011 working group.docx
- 06:13 PM Revision 7746: planning/meetings/: Added Martha's conference call notes
- 06:06 PM Revision 7745: Moved planning/milestones.doc to planning/milestones/
- 06:04 PM Revision 7744: Moved planning/timeline.* to timeline/
- 06:01 PM Revision 7743: Renamed to_do/ to planning/
- 05:53 PM Revision 7742: Added schemas/VegCore/
- 05:53 PM Revision 7741: Added schemas/DwC/
- 05:47 PM Task #575 (New): put all wiki attachments in svn
- Wiki attachments are not version controlled, so it's better to link to the web URL of a file in the repository:
* ... - 05:35 PM Revision 7740: Added schemas/BIEN_logo.png from the bien.nceas.ucsb.edu website
- 05:29 PM Revision 7739: Renamed schemas/BIEN_logo.png to BIEN_logo.small.png
- 05:10 PM Revision 7738: Added inputs/SALVIAS/_archive/salvias_bien_19nov2009.tar.gz.url, .md5
- 04:57 PM Revision 7737: Added inputs/REMIB/_archive/remib_raw.0.header.specimens.txt
- 04:53 PM Revision 7736: Added inputs/CTFS/_src/ctfs_extract.tab.url, .md5
- 04:48 PM Revision 7735: Added inputs/NVS/_src/NVS_VegX.XSLT, _src/VegX/
- 04:43 PM Revision 7734: Added inputs/NVS/_src/_README.TXT
- 04:41 PM Revision 7733: Added inputs/TurboVeg/_src/DBASEDIC_rkp2011*.xlsx
- 04:40 PM Revision 7732: Added inputs/TurboVeg/_src/_README.TXT
- 03:45 PM Revision 7731: inputs/import.stats.xls: Updated import times
- 03:36 PM Revision 7730: README.TXT: Full database import: Don't run backups/fix_perms with sudo because owners are not rsynced to jupiter, so fixing them first is unnecessary. (It's not necessary to change the file owner to bien at any other time, either, because commands are not run as the bien user. It's in fact better to leave the file owner as yourself, to allow rsync to set the modification times.)
- 03:24 PM Revision 7729: README.TXT: Full database import: Added steps to set $version in all vegbiendev terminals
- 03:00 PM Revision 7728: lib/common.Makefile: rsync: $(rsync): Also exclude .lk* lock files
- 02:57 PM Revision 7727: README.TXT: Full database import: Updated log file sync steps for syncing via jupiter
- 02:56 PM Revision 7726: inputs/BRIT/specimen_flat/test.xml.ref: Updated inserted row count
02/26/2013
- 08:27 PM Revision 7725: schemas/py_functions.sql: Added _date(year integer, month integer, day integer)
- 08:08 PM Revision 7724: inputs/BRIT/specimen_flat/postprocess.sql: Added primary key
- 07:40 PM Revision 7723: schemas/functions.sql: Added _dms_to_dd() that takes text arguments and casts them to the appropriate type
- 07:11 PM Revision 7722: inputs/HVAA/Specimen/map.csv: Remapped fieldNotes to UNUSED. This also fixes a bug where fieldNotes was colliding with other occurrenceRemarks inputs because it was missing a filter.
- 06:46 PM Revision 7721: schemas/vegbien.sql: analytical_specimen/analytical_plot: Added occurrenceRemarks
- 06:32 PM Revision 7720: inputs/REMIB/Specimen/map.csv: Mapping NULL-equivalent lat/long to NULL: Fixed bug where need special *=* mapping to pass through values not in the map
- 06:30 PM Revision 7719: inputs/REMIB/Specimen/map.csv: Map NULL-equivalent text fields (country, state, etc.) to NULL
- 06:23 PM Revision 7718: inputs/REMIB/Specimen/map.csv: Map NULL-equivalent lat/long to NULL
- 05:59 PM Revision 7717: inputs/SpeciesLink/Specimen/map.csv: dwc_dwcore_Remarks: Remapped to occurrenceRemarks (it had been incorrectly automapped to taxonRemarks). Combined occurrenceRemarks inputs using /_alt.
- 05:47 PM Revision 7716: mappings/VegCore.htm: Regenerated from wiki. Remapped organismNotes to be a synonym of occurrenceRemarks, since notes on an organism are more generally notes on an occurrence.
- 05:28 PM Revision 7715: inputs/HVAA/Specimen/map.csv: Remapped dynamicProperties to occurrenceRemarks because it contains occurrenceRemarks data
- 05:23 PM Revision 7714: schemas/vegbien.sql: analytical_stem_view: Added occurrenceRemarks
- 05:19 PM Revision 7713: mappings/VegCore-VegBIEN.csv: Mapped occurrenceRemarks
- 04:53 PM Revision 7712: mappings/VegCore-VegBIEN.csv: Only use one of collection and collectionURL instead of concatenating them when both are provided (as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>)
- 04:53 PM Revision 7711: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:52 PM Revision 7710: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:44 PM Revision 7709: inputs/FIA/_src/_README.TXT: Added e-mail from Brad Boyle on 2013-2-8
- 04:42 PM Revision 7708: inputs/FIA/_src/: Added attachments from Brad's e-mail on 2013-2-8
- 04:37 PM Revision 7707: xml_func.py: Removed no longer used _range(). Use SQL functions._range() instead.
- 04:34 PM Revision 7706: schemas/functions.sql: Added _range()
- 04:30 PM Revision 7705: xml_func.py: Removed no longer used _avg(). Use SQL functions._avg() instead.
- 04:29 PM Revision 7704: schemas/functions.sql: Added _avg()
- 04:27 PM Revision 7703: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:15 PM Revision 7702: schemas/functions.sql: _dms_to_dd(text): Corrected concatenated form to interpret last 5 digits as MMmmm = MM.mmm instead of MMSSS, per Tom Wendt's explanation that this form is "degrees/minutes/decimalminutes"
- 04:05 PM Revision 7701: inputs/MO/Specimen/postprocess.sql: Use :table instead of hardcoding "MO"."Specimen"
- 04:04 PM Revision 7700: inputs/MO/Specimen/postprocess.sql: Also remove frameshifted rows where InstitutionCode is NULL
- 03:44 PM Revision 7699: inputs/UNCC/Specimen/map.csv: herbarium->specimenIndexer: Replace UNCCD (UNCC database?) with UNCC
- 03:27 PM Revision 7698: schemas/functions.sql: functions._dms_to_dd(): Added support for DMS values without separators
- 03:09 PM Revision 7697: README.TXT: Datasource setup: Update vegbiendev: Removed command to run the tests on vegbiendev because vegbiendev no longer has an empty public schema for running tests on (the public schema instead contains the latest published import)
- 03:04 PM Revision 7696: Mapped inputs/BRIT/
- 02:36 PM Revision 7695: my2pg: Remove (0) after char
- 02:35 PM Revision 7694: my2pg: Remove any (#) after bytea
- 02:33 PM Revision 7693: my2pg: Translate binary to bytea
- 02:30 PM Revision 7692: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Set $server to vegbiendev instead so the export command will work from any machine (though it will be faster from vegbiendev itself)
- 02:28 PM Revision 7691: inputs/BRIT/_MySQL/MySQL.*.sql.make: Fixed bug where needs to be set to just localhost and *--user* to bien instead
- 02:27 PM Revision 7690: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Fixed bug where $server needs to be set to just localhost and *--user* to bien instead.
- 02:25 PM Revision 7689: lib/MySQL.*.sql.make: Added --user option to demonstrate how to set the DB user vs. the ssh user
- 02:19 PM Revision 7688: inputs/BRIT/_MySQL/MySQL.*.sql.make: Fixed bug where $server needs to be set to bien@localhost
- 02:18 PM Revision 7687: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Specified that $server needs to be set to bien@localhost. Install the staging tables: Specified that for a MySQL .sql export, this must be done on vegbiendev.
- 02:15 PM Revision 7686: inputs/BRIT/: Added _MySQL/MySQL.*.sql.make
- 02:08 PM Revision 7685: README.TXT: Datasource setup: MySQL inputs: .sql exports: Run mysql_bien in the background
- 02:03 PM Revision 7684: README.TXT: Datasource setup: MySQL inputs: .sql exports: Added steps to sync the export to vegbiendev. Specified that the database for the MySQL export should be created on vegbiendev.
- 01:58 PM Revision 7683: README.TXT: Datasource setup: Update vegbiendev: Updated steps for syncing via jupiter
- 01:56 PM Revision 7682: Added inputs/BRIT/ (not yet translated to PostgreSQL)
- 01:35 PM Revision 7681: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_DMS to coordinates.latitude_deg using new _dms_to_dd(text)
- 01:34 PM Revision 7680: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_DMS to coordinates.latitude_deg using new _dms_to_dd(text)
- 01:29 PM Revision 7679: schemas/functions.sql: Added _dms_to_dd(text)
- 01:28 PM Revision 7678: schemas/functions.sql: _dms_to_dd(): Fixed bug where need to use 1 as the multiplier when dir is NULL
- 01:19 PM Revision 7677: schemas/functions.sql: _dms_to_dd(): Fixed bug where need to apply the sign multiplier *after* the components are added, because it applies to the DMS as a whole rather than just to the degree component
- 12:58 PM Revision 7676: README.TXT: Full database import: Before running `make inputs/upload`, added step to do ./fix_perms on local machine so that files are not uploaded to jupiter with incorrect permissions
- 12:57 PM Revision 7675: README.TXT: Full database import: Before running `make inputs/upload`, added step to do svn up on jupiter so that unversioned files are not uploaded before their versioned equivalents are checked out from svn (requiring each file to be manually svn-added before the svn up can proceed)
- 12:48 PM Task #574 (New): mechanism to manually override the CSV dialect, encoding, etc.
- 12:37 PM Task #573 (New): automap the analytical DB columns
- * This will ensure the analytical DB schema stays up to date with VegCore
h3. Method
In @schemas/vegbien.sql@:
... - 12:30 PM Revision 7674: schemas/functions.sql: _dms_to_dd(): Added dir param to specify compass dir suffix, in order to calculate the sign correctly
- 11:05 AM Revision 7673: Moved compass_dir from schemas/vegbien.sql to schemas/functions.sql so it can be used by _dms_to_dd()
- 10:56 AM Revision 7672: Moved _dms_to_dd() from schemas/vegbien.sql to schemas/functions.sql so it can use functions schema functions
- 10:51 AM Revision 7671: schemas/vegbien.sql: Added compass_dir enum
- 10:42 AM Revision 7670: inputs/TEX/Specimen/map.csv: Remapped LAT/LONG_DEGREES,MINUTES,SECONDS to new latitude/longitude_deg,min,sec, allowing the DMS coordinates to be translated
- 10:40 AM Revision 7669: inputs/REMIB/Specimen/map.csv: Remapped lat/long_deg,min,sec to new latitude/longitude_deg,min,sec, allowing the DMS coordinates to be translated
- 10:39 AM Revision 7668: mappings/VegCore-VegBIEN.csv: latitude/longitude_deg,min,sec: Also mapped to the geoscrub coordinates entry
- 10:34 AM Revision 7667: mappings/VegCore-VegBIEN.csv: latitude/longitude_sec: Fixed name, which had been incorrectly automapped to verbatim*
- 10:31 AM Revision 7666: mappings/VegCore.htm: Regenerated from wiki. latitude/longitude: Moved *_deg,min,sec components after unit alternatives so that unit alternatives following them wouldn't incorrectly be interpreted as alternatives for the *components* instead of for the overall, unitless term.
- 10:22 AM Revision 7665: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_deg,min,sec
- 10:15 AM Revision 7664: mappings/VegCore.htm: Regenerated from wiki. Merged coordinateUncertainty and coordinatePrecision, since datasources often use these interchangeably (the only difference is the DwC-recommended units).
- 10:08 AM Revision 7663: mappings/VegCore.htm: Regenerated from wiki. Source: Added authors. Renamed creator to firstPublisher. Moved dataOwner to Source as list. Coordinates: latitude/longitude, coordinatePrecision: Added units disambiguation. latitude/longitude_DMS: Added components. Global IDs: Changed "globally unique" tag to "global ID" and hyperlinked it to <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs> (which now includes detailed examples of the various types of global IDs).
- 09:58 AM Revision 7662: inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_CoordinatePrecision: Remapped to coordinateUncertaintyInMeters, assuming units based on the range and precision of values
- 09:53 AM Revision 7661: inputs/HVAA/Specimen/map.csv: Remapped coordinatePrecision to UNUSED
- 09:50 AM Revision 7660: inputs/ARIZ/omoccurrences/map.csv: Remapped coordinatePrecision to UNUSED
- 09:05 AM Revision 7659: schemas/vegbien.sql: Added _dms_to_dd()
- 08:49 AM Revision 7658: inputs/bien2_traits/TraitObservation/map.csv: Remapped Source to sourceType
- 08:43 AM Revision 7657: Added inputs/TEX/
- 08:42 AM Revision 7656: units.py: convert(): Support ranges
- 08:35 AM Revision 7655: Moved parse_range() from xml_func.py to units.py
- 08:02 AM Revision 7654: units.py: conversions: Added ft->m conversion
- 07:12 AM Revision 7653: src_map: Reopen stdin (/dev/stdin) in universal newlines mode to support files with \r as the line ending
- 06:51 AM Revision 7652: csv2db: Open input stream in universal newlines mode to support files with \r as the line ending
02/22/2013
- 11:15 PM Revision 7651: mappings/VegCore.htm: Regenerated from wiki. Moved Forming IDs section to separate Global IDs wiki page (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs).
- 11:09 PM Revision 7650: input.Makefile: $(exts): Added .tab
- 10:29 PM Revision 7649: mappings/VegCore.htm: Regenerated from wiki. Moved preferred ID order to Forming IDs section as examples list. Made examples globally unique by prefixing the outer ID that scopes the value.
- 09:29 PM Revision 7648: mappings/VegCore.htm: Regenerated from wiki. Forming IDs: Added instructions for making a local ID global. Added note that these globally unique IDs are not the same as UUIDs. Forming preferred IDs: Reworded as a set of instructions rather than a definition. Clarified that this becomes the database primary key.
- 08:47 PM Revision 7647: mappings/VegCore.htm: Regenerated from wiki. Added section on forming preferred IDs, which are the ID which is most directly associated with the identified object.
- 08:17 PM Revision 7646: mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
- 08:00 PM Revision 7645: inputs/NY/Ecatalog_all/test.xml.ref: Updated inserted row count
02/21/2013
- 05:13 PM Revision 7644: schemas/vegbien.sql: analytical_stem_view: scientificName_verbatim: Use the taxonname (which omits the family and author) instead when available, at Brad's request
- 04:02 PM Revision 7643: schemas/vegbien.sql: Disambiguated taxonlabel.taxonomicname, taxonverbatim.taxonomicname, and taxonverbatim.taxonname
- 03:14 PM Revision 7642: schemas/vegbien.sql: taxon_trait_view: Use taxonverbatim.taxonname (the binomial) in preference to taxonverbatim.taxonomicname (the binomial+author) or taxonlabel.taxonomicname (the family+binomial+author), at Mark's request
- 02:05 PM Revision 7641: to_do/timeline.2013.xls: Obtain and load any additional new data: Extended beyond April because it's ongoing
- 02:02 PM Revision 7640: README.TXT: Full database import: On nimoy: Moved analytical_aggregate table creation right after analytical_aggregate..csv download because it can happen while the file is downloaded
- 11:13 AM Revision 7639: to_do/timeline.2013.xls: Reload existing data in need of refresh: Added additional cell because this is still in progress
- 11:11 AM Revision 7638: to_do/timeline.2013.xls: Placed check mark in completed cells. Multi-cell tasks that are partially complete have check marks in part of their cells to indicate this.
02/20/2013
- 05:35 PM Revision 7637: Added inputs/FIA/REF_*/
- 05:33 PM Revision 7636: inputs/FIA/_src/Makefile: Table bundling: Also unpack FIADB_REFERENCE/
- 05:13 PM Revision 7635: Added inputs/FIA/_src/FIADB_REFERENCE.ZIP.md5, .url
- 04:11 PM Revision 7634: schemas/vegbien.sql: provider_count_view: Group data by dataprovider and dataset, rather than by each separately, to satisfy the GBIF attribution requirements as described by Brad
- 02:00 PM Revision 7633: schemas/vegbien.sql: analytical_plot/specimen: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
- 01:54 PM Revision 7632: schemas/vegbien.sql: analytical_stem_view: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
- 01:53 PM Revision 7631: schemas/vegbien.sql: sync_analytical_stem_to_view(): Also drop and re-create new dependent views analytical_plot, analytical_specimen
- 01:20 PM Revision 7630: inputs/TEAM/V*/map.csv: Mapped plotArea_ha based on "1ha Plot Number" field name
- 01:12 PM Revision 7629: inputs/TEAM/V*/map.csv: Id: Remapped to individualObservationID
- 01:10 PM Revision 7628: mappings/VegCore.htm: Regenerated from wiki. verbatimScientificName, verbatimTaxonName have been merged into concatenatedScientificName, and ID has been disambiguated.
- 01:09 PM Revision 7627: inputs/*/*/map.csv: Automapped taxonConfidence
- 12:28 PM Revision 7626: schemas/vegbien.sql: analytical_stem_view: plotArea_ha: Fixed bug where need to use parent plot's area when subplot's area is not provided
- 12:18 PM Revision 7625: inputs/SALVIAS/plotMetadata_/create.sql: Added primary key
- 11:42 AM Revision 7624: inputs/NY/NY_DATA_20130212.~.utils.sql: pkey: Made table an enum instead of a text field so that Ecatalog_LatLong could be sorted before Ecatalog
- 11:32 AM Revision 7623: inputs/NY/Ecatalog_all/create.sql: primary key: Combine table, key into one value so that the primary key has a single column. This is necessary because multi-column primary keys are not supported by the import algorithm.
- 11:28 AM Revision 7622: inputs/NY/NY_DATA_20130212.~.utils.sql: Added pkey record type for use by Ecatalog_all's generated pkey
- 10:47 AM Revision 7621: inputs/NY/Ecatalog_all/create.sql: Add table column which scopes the key field. Add primary key to allow sorting by it. inputs/NY/Ecatalog_all/map.csv: Documented that UniqueNYInternalRecordNumber is sometimes duplicated.
- 09:20 AM Revision 7620: make_analytical_db: Wait for tnrs.make to acquire the tnrs.make lock if being run right after import
- 09:13 AM Revision 7619: schemas/vegbien.sql: analytical_stem_view: taxonName_matched: Fixed bug where need to use the matched_taxonverbatim instead of the datasource_taxonverbatim
- 08:58 AM Revision 7618: Moved wait on tnrs.make lock from import_all to make_analytical_db, so that running make_analytical_db for a one-time import also waits on the lock
- 08:56 AM Revision 7617: schemas/vegbien.sql: taxondetermination: taxondetermination_unique: Added determinationtype so that when the matched and accepted determinations are the same, they still both get created rather than the second one being removed due to the unique constraint
- 07:59 AM Revision 7616: schemas/vegbien.sql: analytical_specimen: Removed speciesBinomialWithMorphospecies because it doesn't apply to specimens
- 07:58 AM Revision 7615: schemas/vegbien.sql: Added analytical_plot view
- 07:55 AM Revision 7614: schemas/vegbien.sql: Added analytical_specimen view
- 07:44 AM Revision 7613: schemas/vegbien.sql: analytical_stem_view: Moved recordedBy, recordNumber before dateCollected as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
- 07:40 AM Revision 7612: schemas/vegbien.ERD.mwb: Synced with schema
- 07:38 AM Revision 7611: schemas/vegbien.sql: Added reproductiveCondition
- 07:33 AM Revision 7610: mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition
- 07:28 AM Revision 7609: schemas/vegbien.sql: plantobservation: Added reproductivecondition
- 05:33 AM Revision 7608: mappings/VegCore.htm: Regenerated from wiki. matched*Fit_fraction has been renamed to matched*Confidence_fraction.
- 05:32 AM Revision 7607: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Updated for new mappings/VegCore.htm
- 05:10 AM Revision 7606: inputs/bien_web/observation/map.csv: Re-automapped taxonMorphospecies
- 05:08 AM Revision 7605: mappings/VegCore.htm: Regenerated from wiki. Data owner terms and taxon synonyms have been added, and morphospecies has been disambiguated.
- 04:51 AM Revision 7604: schemas/vegbien.sql: analytical_stem_view: Moved identifiedBy, dateIdentified, identificationRemarks right after the *_verbatim terms that they relate to, as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
- 02:25 AM Revision 7603: schemas/vegbien.sql: analytical_stem_view: Use new concat_delim() instead of array_to_string() surrounded by NULLIF(..., '')
- 02:19 AM Revision 7602: schemas/vegbien.sql: Added concat_delim()
- 01:43 AM Revision 7601: schemas/vegbien.sql: analytical_stem_view: Removed deprecated taxonNameWithMorphospecies now that we have speciesBinomialWithMorphospecies
- 01:17 AM Revision 7600: schemas/vegbien.sql: analytical_stem_view: speciesBinomial: Added morphospecies suffix to create speciesBinomialWithMorphospecies
- 12:15 AM Revision 7599: inputs/import.stats.xls: Updated import times
- 12:06 AM Revision 7598: README.TXT: Full database import: Check that unscrubbed_taxondetermination_view returns no rows: Documented that this takes 90 s with LIMIT 1
02/19/2013
- 11:16 PM Revision 7597: schemas/vegbien.sql: _taxon_family_require_std(): Also allow non-aceae families accepted by TNRS
- 09:38 PM Revision 7596: Added inputs/SALVIAS/_archive/salvias_plots.*.sql.zip.md5
- 09:35 PM Revision 7595: Added inputs/VegBank/_archive/vegbank_for_bien.tar.gz.url
- 09:29 PM Revision 7594: Added inputs/U/UtrechtHerbarium.csv.tar.gz.url
- 09:28 PM Revision 7593: Added inputs/TEAM/_archive/ci-team_extract.tar.gz.url
- 09:27 PM Revision 7592: Added inputs/SpeciesLink/_archive/specieslink*.txt.gz.url
- 09:22 PM Revision 7591: Added inputs/REMIB/_archive/remib_raw.csv.tar.gz.url
- 09:19 PM Revision 7590: Added inputs/NY/NYSpecimenDataAmericas.csv.tar.gz.url
- 09:17 PM Revision 7589: Added inputs/NCU/_archive/NCU-NCSC_2010-02-12.csv.tar.gz.url
- 09:14 PM Revision 7588: Added inputs/MO/mo_digirexport.tar.gz.url
- 09:13 PM Revision 7587: Added inputs/Madidi/_archive/2010-1-2/madidi_plots_original_12jan2010.zip.url
- 09:11 PM Revision 7586: Added inputs/GBIF/gbif_extract.tar.gz.url
- 09:10 PM Revision 7585: Added inputs/FIA/fia_extract.tar.gz.url
- 09:08 PM Revision 7584: Added inputs/CVS/_archive/CVS-allTaxonOccurrences_2010-01-12.txt.tar.gz.url
- 09:04 PM Revision 7583: Added inputs/ARIZ/ARIZ_DiGIR_21012010.csv.tar.gz.url
- 08:55 PM Revision 7582: Added inputs/UNCC/Specimen/UNCC.csv.url, UNCC.csv.md5
- 08:45 PM Revision 7581: Added inputs/XAL/_src/digir.xml.gz.md5
- 08:39 PM Revision 7580: Added inputs/UNCC/_src/ with UNCC.csv.zip.md5
- 08:23 PM Revision 7579: Added inputs/SpeciesLink/_src
02/16/2013
- 08:24 AM Revision 7578: README.TXT: Datasource setup: MySQL inputs: .sql exports: Use new mysql_bien to connect to the MySQL DB created for the datasource
- 08:22 AM Revision 7577: Added mysql_bien, which runs a MySQL command on the local MySQL server
- 08:06 AM Revision 7576: Added inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump.md5 (md5sum of the expanded file)
- 08:02 AM Revision 7575: root Makefile: MySQL: mysql-Linux: Also install phpMyAdmin
- 08:01 AM Revision 7574: root Makefile: MySQL: mysql-Linux: Split apt-get dependencies into separate commands, like for other apt-get commands, to avoid having one failed dependency prevent the following dependencies from being installed
- 07:57 AM Revision 7573: root Makefile: MySQL: *mysql_users: Also add bien_read user
- 07:49 AM Revision 7572: root Makefile: MySQL: Renamed *mysql_user to *mysql_users because there can be multiple users
- 06:51 AM Revision 7571: inputs/: Added .md5 files for all .zip, .gz
- 06:47 AM Revision 7570: Added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.url
- 06:39 AM Revision 7569: lib/common.Makefile: rsync: $(rsync*): Use --no-group because the file group is different depending on the machine
- 06:10 AM Revision 7568: input.Makefile: SVN: $(_svnFilesGlob): Also add .md5 files. This allows svn to track where unversioned files should be in the directory tree.
- 06:07 AM Revision 7567: input.Makefile: SVN: $(_svnFilesGlob): .url, .pdf, and README.TXT in the top-level dir: Fixed bug where had extra / after brace expr
- 06:00 AM Revision 7566: input.Makefile: SVN: $(_svnFilesGlob): Also add .url, .pdf, and README.TXT in the top-level dir
- 05:53 AM Revision 7565: input.Makefile: SVN: $(_svnFilesGlob): Add .url, .pdf, and README.TXT files in all subdirs, not just _src
- 05:25 AM Revision 7564: lib/common.Makefile: remote server: Use jupiter instead of vegbiendev, to ensure that all files get uploaded there rather than only to vegbiendev. This involves adding an extra database import step to download the uploaded files from jupiter onto vegbiendev.
- 02:50 AM Revision 7563: inputs/FIA/_src/Makefile: all: Extract zip files before running tables target, because it requires the created dirs
- 02:40 AM Revision 7562: schemas/vegbien.ERD.mwb: Fixed table sizes
- 01:17 AM Revision 7561: Removed no longer used fix_permissions. Use root fix_perms instead.
- 01:16 AM Revision 7560: Added root fix_perms
02/15/2013
- 11:58 PM Revision 7559: Moved Checksums from backups/Makefile to lib/common.Makefile so all dirs (including inputs/) can use md5sum testing
- 11:08 PM Revision 7558: lib/common.Makefile: $(remote): Made remote basepath configurable in $(remote_basepath)
- 11:04 PM Revision 7557: lib/common.Makefile: Renamed $(src_server) to $(remote_host) and $(src_user) to $(remote_user) for clarity
- 10:16 PM Revision 7556: inputs/GBIF/: Added refresh metadata
02/14/2013
- 11:49 AM Revision 7555: Added inputs/HVAA/
- 11:14 AM Revision 7554: Added inputs/ARIZ/_archive
- 11:13 AM Revision 7553: inputs/ARIZ/: Removed previous data now that it has been refreshed
- 11:08 AM Revision 7552: inputs/ARIZ/: Mapped refresh
- 11:04 AM Task #566 (New): automatically adjust staging tables for easier mapping
- * remove empty columns
* mark columns with data in every row as NOT NULL - 09:48 AM Revision 7551: Added inputs/ARIZ/import_order.txt
- 09:22 AM Revision 7550: Added inputs/NY/_archive/
- 09:20 AM Revision 7549: inputs/NY/: Removed tables from previous extract
- 08:59 AM Revision 7548: inputs/NY/: Mapped refresh
- 08:58 AM Revision 7547: inputs/*/*/VegBIEN.csv: Regenerated from mappings/VegCore-VegBIEN.csv
- 08:52 AM Revision 7546: Added inputs/NY/import_order.txt
- 07:20 AM Task #386 (Resolved): load Canadensys data
- 07:19 AM Task #470 (Resolved): source terms from old versions of DwC to the DwC history page
- 07:18 AM Task #472 (Rejected): replace accessioncodes with datasource_id+sourceaccessioncode
- Accessioncode fields have been removed instead. Globally unique ID fields (#561) will eventually serve the purpose th...
- 07:06 AM Task #565 (Resolved): partition the TaxonDetermination table by row into the different types of determinations
- * this makes it easy to horizontally join the different types of determinations for a row
** it avoids the need for ... - 06:37 AM Task #564 (New): make all VegBIEN column names globally unique
- * this enables creating a table to contain the results of a join, without needing to resolve column name collisions
... - 06:33 AM Task #563 (New): refactor VegBIEN to use VegCore terms
- * add key VegCore tables such as Occurrence and Record
* note that many VegCore tables have an inheritance relations... - 05:17 AM Task #562 (New): flatten the mappings
- * put the destination table at the beginning of the mapping, rather than nesting it within a hierarchy of tables it h...
- 04:47 AM Task #561 (New): make VegBIEN ID fields plain-text instead of numeric
- h3. Rationale
* this makes it possible to append data from multiple sources without having pkey collisions
h3. ... - 04:12 AM Task #560 (New): move VegCore data dictionary to a phpPgAdmin-accessible database
- * term details go in column comments, with Redmine formatting translated to HTML
* each synonym becomes a parameter ... - 02:51 AM Revision 7545: inputs/ARIZ/: Added SQL export for refresh
- 02:33 AM Revision 7544: my2pg.data: Translate indefinite (zero) months which have a definite day. This is unusual, but does appear in some data such as the ARIZ DB.
- 02:28 AM Revision 7543: my2pg.data: Translate indefinite dates (dates with 0 as the month or day)
- 02:23 AM Revision 7542: my2pg: Use my2pg.data to perform data-only replacements, instead of duplicating them in both my2pg and my2pg.data
- 02:01 AM Revision 7541: my2pg: named UNIQUE KEYs: Comment out the name because PostgreSQL requires it to be globally unique, but MySQL only requires it to be unique within the table
- 01:53 AM Revision 7540: my2pg: Translate UNIQUE KEYs instead of removing them
- 01:49 AM Revision 7539: my2pg*: Removed KEYs: Comment out the definition rather than removing it
- 01:45 AM Revision 7538: my2pg*: Remove FOREIGN KEYs because MySQL does not dump tables in dependency order, which prevents PostgreSQL from creating tables whose fkeys refer to a later table
- 01:33 AM Revision 7537: my2pg*: Replacing invalid table elements to remove them: Use a dummy CHECK constraint instead of a boolean field to avoid adding fields to the table. The elements can't always simply be removed because sed can't remove the trailing comma of the previous element, and removing the following comma doesn't work for the last element in the table.
- 12:11 AM Revision 7536: my2pg*: Replace '0000-00-00 00:00:00' with '-infinity'
- 12:04 AM Revision 7535: my2pg: Replace datetime with timestamp
02/13/2013
- 11:59 PM Revision 7534: my2pg: Remove COLLATE field attribute
- 11:56 PM Revision 7533: lib/MySQL.*.sql.make: Documented that $server user/host are for ssh, not the DB
- 11:55 PM Revision 7532: lib/MySQL.*.sql.make: Documented that $server can also contain a username (which will be used by ssh)
- 11:51 PM Revision 7531: my2pg_export: Use the --quick option to facilitate exporting large tables (it avoids retrieving all rows before outputting any of them)
- 11:00 PM Revision 7530: README.TXT: Datasource setup: Added instructions for MS Access databases
- 10:43 PM Revision 7529: README.TXT: Datasource setup: MySQL inputs: Added instruction to skip the Add input data for each table section
- 10:40 PM Revision 7528: inputs/NY/: Added SQL export for refresh
02/12/2013
- 01:08 PM Revision 7527: mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.
- 12:15 PM Revision 7526: inputs/CVS/observation_/map.csv: baseSaturation: Resolved ambiguous term
- 12:09 PM Revision 7525: mappings/Makefile: VegCore.vocab.csv: Ignore leading ? when sorting so that ambiguous terms sort alphabetically with other terms. This prevents terms from moving from their previous location when they become ambiguous.
- 12:07 PM Revision 7524: Added sort_ci to sort a spreadsheet, ignoring leading punctuation
- 12:05 PM Revision 7523: mappings/VegCore.vocab.csv: Changed line endings to \r\n in preparation for having a Python script run on it (which changes the line endings)
- 11:47 AM Revision 7522: mappings/Makefile: VegCore.vocab.csv: Added back ambiguous terms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are ambiguous or unambiguous terms
- 11:44 AM Revision 7521: mappings/Makefile: VegCore.vocab.csv: Added back synonyms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are synonyms or primary terms. This also prevents VegCore.vocab.csv from losing entries when terms are renamed, which made it difficult to verify that no terms were lost when refactoring.
- 05:50 AM Revision 7520: inputs/MO/Specimen/postprocess.sql: Remove frameshifted rows by detecting InstitutionCodes without any letters
- 04:59 AM Revision 7519: inputs/ARIZ/Specimen/map.csv: CollectorNumber/FieldNumber: Use /_first to map these identical fields to the same location
- 04:54 AM Revision 7518: inputs/ARIZ/Specimen/map.csv: Fixed bug where the column names for InstitutionCode and CollectionCode were reversed in the source data
- 04:14 AM Revision 7517: inputs/*/Specimen/map.csv for Canadensys sources: Remapped institutionID to UNUSED
02/09/2013
- 07:45 AM Revision 7516: mappings/VegCore.htm: Regenerated from wiki. The original*, accepted*, and verbatim* Taxon fields have now been moved to separate OriginalTaxonDetermination, AcceptedTaxonDetermination, and TaxonVerbatim tables.
- 06:52 AM Revision 7515: mappings/VegCore.htm: Regenerated from wiki
- 06:34 AM Revision 7514: mappings/VegCore.htm: Regenerated from wiki
- 04:08 AM Revision 7513: README.TXT: Maintenance: VegCore data dictionary: Replaced VegCore.*.csv with VegCore.htm because now that VegCore.*.csv are sorted alphabetically, they generally don't change when VegCore.htm changes
- 04:04 AM Revision 7512: mappings/VegCore.*.csv: Regenerated from wiki. A plain text label is now used for Replace with, which fixes a bug where the PRIVATE permalink pointed to its Replace with in realLatitude instead of its definition.
- 03:55 AM Revision 7511: redmine_synonyms: Support plain text labels other than Alternative, such as Replace with
- 03:13 AM Revision 7510: mappings/VegCore.*.csv: Regenerated from wiki. Alternatives now contain the "Alternative" label as plain text rather than as an image title, thus avoiding an HTML anchor conflict with the definition and allowing ambiguous terms to be placed before their alternatives as well as after.
- 03:11 AM Revision 7509: README.TXT: Maintenance: VegCore data dictionary: Updated VegCore.csv filename to VegCore.*.csv
- 02:57 AM Revision 7508: redmine_synonyms: Support alternatives which contain the "Alternative" label as plain text rather than as an image title. This is done to include the "Alternative" label in the HTML anchor and thus prevent the anchor from conflicting with the actual definition of the alternative (which would otherwise have the same anchor text). This allows ambiguous terms to be placed before their alternatives as well as after, because there won't be anchor conflicts that need to be resolved with careful ordering.
- 02:48 AM Revision 7507: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
- 02:47 AM Revision 7506: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
- 02:47 AM Revision 7505: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
02/07/2013
- 01:57 PM Revision 7504: mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch
- 01:57 PM Revision 7503: inputs/bien_web/observation/test.xml.ref: Regenerated
- 01:48 PM Revision 7502: inputs/import.stats.xls: Updated import times using the import_times bugfix for times longer than a day
- 01:45 PM Revision 7501: import_times: times(): Fixed bug where need to match whitespace in times, in order to match times with days
- 12:00 PM Revision 7500: inputs/*/Specimen/map.csv: Remapped ID to datasourceRecordID
- 11:55 AM Revision 7499: mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID
- 11:51 AM Revision 7498: inputs/import.stats.xls: Updated import times
- 08:38 AM Revision 7497: inputs/FIA/_src/_README.TXT: Documented that the refresh is missing some PLT_CN values present in the original version
- 08:33 AM Revision 7496: inputs/FIA/import_order.txt: Reverted back to using FIA_COND_unique instead of COND_unique because the PLT_CN IDs in the refresh don't match the PLT_CN IDs in the original version, making COND_unique and Organism incompatible
- 08:27 AM Revision 7495: inputs/FIA/import_order.txt: Removed FIA_COND_unique, which is superseded by COND_unique
- 08:26 AM Revision 7494: inputs/FIA/import_order.txt: Fixed bug where need to import COND_unique before Organism because the plot entries need to be created before they can be linked to by organisms
- 07:25 AM Revision 7493: redmine_synonyms: sed pattern: Match <h# directly at the beginning of the line rather than after ^.*, which greatly speeds up the pattern matching because the first character is a literal character. (If <h# were not located at the left margin, the ^.* would unfortunately still be needed because the beginning of the line needs to be matched in order to be removed by the replacement operation.)
- 07:22 AM Revision 7492: mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.
- 07:16 AM Revision 7491: mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
- 07:05 AM Revision 7490: redmine_synonyms: Don't require ambiguous terms to start with ?, because the ambiguous term for an alternative can be identified simply by choosing the last term that didn't have a type label (previously, this would have been the last term that wasn't h3 or h4)
- 07:01 AM Revision 7489: redmine_synonyms: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
- 06:26 AM Revision 7488: mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).
- 06:21 AM Revision 7487: mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.
- 06:00 AM Revision 7486: mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match
- 02:06 AM Revision 7485: Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity
- 02:03 AM Revision 7484: mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary
- 01:57 AM Revision 7483: mappings/Makefile: Replaced occurrences of VegCore.csv with $(vocab) and Veg+-VegCore.csv with $(dict)
02/06/2013
- 07:34 PM Revision 7482: README.TXT: Maintenance: VegCore data dictionary: When moving terms, check that no terms were lost: Updated steps now that VegCore.csv and Veg+-VegCore.csv are sorted by name, so that a comparison of added/deleted counts is not necessary and a simple `svn di` can be used
- 07:33 PM Revision 7481: mappings/Makefile: Veg+-VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause Veg+-VegCore.csv to change. This makes it much easier to identify synonyms and ambiguous terms that were accidentally deleted during a data dictionary refactoring. (Note that these are no longer included in VegCore.csv, so this is required in addition to sorting VegCore.csv by name.)
- 07:26 PM Revision 7480: mappings/Makefile: VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause VegCore.csv to change. This makes it much easier to identify terms that were accidentally deleted during a data dictionary refactoring.
02/05/2013
- 06:19 PM Revision 7479: mappings/VegCore.csv: Regenerated from wiki. This adds cf_aff.
- 06:18 PM Revision 7478: mappings/Makefile: VegCore.csv: Filter out namespaces by matching only terms whose header links within the data dictionary
- 06:08 PM Revision 7477: mappings/VegCore.csv: Regenerated from wiki. This causes TNRS's Annotations (cf/aff) to be mapped into VegBIEN.
- 06:05 PM Revision 7476: mappings/VegCore-VegBIEN.csv: matched*Fit_fraction: Remapped to taxonconfidence instead of taxonfit
- 05:56 PM Revision 7475: mappings/Makefile: VegCore.csv: Fixed bug where need to remove duplicates, which are no longer supported by canon, by removing alternatives of ambiguous terms when these occur separately from their definitions
- 05:29 PM Revision 7474: mappings/Makefile: VegCore.csv: Removed synonyms and ambiguous terms, since the canonicalization of them is handled by Veg+-VegCore.csv. This also reduces the time it takes canon to build the in-memory Python dict of replacements, which scales to all inputs and should speed up the build/test command.
- 05:22 PM Revision 7473: mappings/Makefile: VegCore.csv: Removed synonyms, since the canonicalization of them is handled by Veg+-VegCore.csv
- 05:10 PM Revision 7472: mappings/Makefile: VegCore.csv: Match terms by header # instead of matching all anchors, in order to include the leading ? before an ambiguous term
- 04:42 PM Revision 7471: mappings/Makefile: Veg+-VegCore.csv: Generate dynamically from VegCore.htm, which allows the VegCore thesaurus to be automatically kept up to date. More importantly, it allows terms in all map spreadsheets to be updated simultaneously when a term is renamed (e.g. by replacing a term with one of its synonyms).
- 04:40 PM Revision 7470: mappings/VegX-VegCore.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv. Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping several fields.
- 04:32 PM Revision 7469: mappings/VegCore-VegBIEN.csv, inputs/*/*/map.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv, which reflects the current state of the data dictionary. (Permanently switching to the new Veg+-VegCore.csv will be a separate change.) Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping and inferring units on several fields.
- 04:27 PM Revision 7468: mappings/VegCore-VegBIEN.csv: Mapped basalDiameter_in
- 04:15 PM Revision 7467: mappings/VegCore-VegBIEN.csv: Mapped diameterBreastHeightGentry_cm, basalDiameter_cm, precipitation_mm
- 04:14 PM Revision 7466: schemas/vegbien.sql: Added _mm_to_m()
- 03:56 PM Revision 7465: mappings/Makefile: Veg+-VegCore.csv: Fixed bugs where also need to filter out ambiguous tables, but shouldn't filter out acronyms (which are regular fields)
- 03:40 PM Revision 7464: mappings/VegCore-VegBIEN.csv: locationID->location.sourceaccessioncode: Removed restriction that this mapping can't occur if geovalidation information is present. The locationID is no longer mapped to the place.sourceaccessioncode, so this filter is not necessary.
- 03:38 PM Revision 7463: mappings/VegCore.csv: Regenerated from wiki
- 03:19 PM Revision 7462: mappings/Makefile: Veg+-VegCore.csv: Fixed bug where need to filter out table names to avoid applying table replacements to fields which have the same name as a table
- 03:03 PM Revision 7461: inputs/Madidi/map.csv: Fixed bug where needed to remove duplicate input names, now that translate doesn't allow them
- 01:59 PM Revision 7460: mappings/Makefile: VegX-VegCore.csv: Sort by the input column instead of the output column to keep the sort order stable across VegCore term renames
- 01:46 PM Revision 7459: mappings/Makefile: Veg+-VegCore.csv: Before running collapse_multimap, canonicalize alternatives of ambiguous terms using unambiguous mappings. This ensures that the alternatives lists contain only canonical VegCore terms rather than synonyms.
- 01:43 PM Revision 7458: mappings/VegCore.csv: Regenerated from wiki. All synonyms are now hyperlinked, allowing them to be matched by redmine_synonyms.
- 01:31 PM Revision 7457: mappings/Veg+-VegCore.csv: Removed Sources, Definition columns because source information is now in the VegCore data dictionary
- 01:25 PM Revision 7456: mappings/VegCore.csv: Regenerated from wiki. Ambiguous terms newly available to redmine_synonyms due to the bugfix now have multiple alternatives.
- 01:25 PM Revision 7455: redmine_synonyms: Ambiguous terms: Fixed bug where need to use header # instead of term name to determine whether a term is an alternative, because some alternatives (e.g. verbatimElevation) don't follow the units-suffix naming convention.
- 12:58 PM Revision 7454: mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation
- 12:50 PM Revision 7453: mappings/Makefile: Veg+-VegCore.csv: translate: Fixed bug where need to run on $@ instead of $<
- 12:49 PM Revision 7452: mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation
- 12:22 PM Revision 7451: mappings/VegCore.csv: Regenerated from wiki. All mappings/Veg+-VegCore.csv terms are now added as synonyms or separate terms.
- 10:26 AM Revision 7450: mappings/VegCore.csv: Regenerated from wiki. Most ambiguous terms are now split into alternatives, and most mappings/Veg+-VegCore.csv terms are now added as synonyms.
- 06:12 AM Revision 7449: canon: Raise an error if two input terms map to the same simplified string
- 04:34 AM Revision 7448: translate: Changed dictionary to thesaurus, since the map used actually has synonyms rather than definitions
- 04:31 AM Revision 7447: mappings/Makefile: Veg+-VegCore.csv: Translate the thesaurus's output terms using itself in order to map a synonym of an ambiguous term directly to its alternatives list rather than only to the ambiguous term itself
- 04:26 AM Revision 7446: mappings/Makefile: Veg+-VegCore.csv: Run collapse_multimap on the generated map so that all alternatives are included, rather than just the first alternative, when translate maps an ambiguous term
- 04:25 AM Revision 7445: redmine_synonyms: Fixed bug where need to output a CSV rather than TSV to be usable by other programs that use map spreadsheets
- 04:23 AM Revision 7444: Added collapse_multimap, which collapses multimap entries in a spreadsheet dictionary
- 03:45 AM Revision 7443: mappings/Veg+-VegCore.csv: Separate alternatives of ambiguous terms with , instead of ", " for easier machine-parsability
- 03:31 AM Revision 7442: redmine_synonyms: Added support for ambiguous terms, which unlike the synonyms format nests the term (the alternative) under the synonym (the ambiguous term) rather than the synonym under the term. Note that ambiguous terms must also be prefixed with ? to differentiate them from composites (e.g. recordedBy_givenName), which use the same _-based naming convention.
- 03:08 AM Revision 7441: mappings/VegCore.csv: Regenerated from wiki
- 02:49 AM Revision 7440: mappings/VegCore.csv: Regenerated from wiki
- 02:22 AM Revision 7439: schemas/vegbien.sql: analytical_stem_view: Renamed scientificNameWithMorphospecies to taxonNameWithMorphospecies because it does not contain the scientific name author, as required by DwC scientificName <http://rs.tdwg.org/dwc/terms/#scientificName>
- 01:56 AM Revision 7438: mappings/Makefile: VegCore.tables.csv: Exclude ambiguous table names, which should not be part of the tables summary (as neither are table synonyms)
- 01:51 AM Revision 7437: input.Makefile: $(translate?): Merged with $(translate), which is not used independently
- 01:50 AM Revision 7436: input.Makefile: Use new translate_ci instead of translate
- 01:47 AM Revision 7435: mappings/Makefile: Use new translate_ci instead of translate
- 01:39 AM Revision 7434: Added translate_ci
02/04/2013
Also available in: Atom