Project

General

Profile

Statistics
| Revision:

# Date Author Comment
7871 03/07/2013 03:19 PM Aaron Marcuse-Kubitza

Added planning/goals/BIEN3_derived_data_products.docx

7870 03/06/2013 11:00 PM Aaron Marcuse-Kubitza

inputs/ARIZ/omoccurrences/map.csv: cultivationStatus: Map to NULL if it's false, because cultivated can only be true or NULL

7869 03/06/2013 10:01 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Merged taxonName_verbatim and scientificNameAuthorship_verbatim into scientificName_verbatim

7868 03/06/2013 09:29 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_plot: Added specimens fields at the end, for use by plots data that also tracks specimens (e.g. Madidi, SALVIAS)

7867 03/06/2013 09:21 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: In PostgreSQL: Removed tests related to unscrubbed_taxondetermination_view and TNRS taxondeterminations, which are no longer needed because TNRS is joined directly to analytical_stem_view

7866 03/06/2013 08:36 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Renamed specimenDonor to specimenDonorInstitution to clarify that this is an institution, not a collector.

7865 03/06/2013 08:30 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: Mapped flower, fruit -> reproductiveCondition using translation is from e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet on 2010-12-2~3

7864 03/06/2013 08:17 PM Aaron Marcuse-Kubitza

make_analytical_db: Support the $log flag, which toggles output to the terminal or a log file

7863 03/06/2013 07:50 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times. The core import time has dropped by more than half (!) to ~12 hours, now that the TNRS scrubbing is added using a simple LEFT JOIN, instead of being pushed through the normalized schema. Not since October has the import been this fast!

7862 03/06/2013 07:27 PM Aaron Marcuse-Kubitza

import_times: When there are no matching logs, exit with status 0 instead of 1 because this is a normal condition

7861 03/06/2013 07:26 PM Aaron Marcuse-Kubitza

import_times: Fixed bug where sed would wait for user input from stdin when there were no matching logs

7860 03/06/2013 07:22 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: record the import times in inputs/import.stats.xls: If the row counts do not match up: Added step to get the datasource names from the log file footers

7859 03/06/2013 07:12 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: cultivated: assume anything non-NULL to be cultivated, so that we err on the side of excluding possibly-cultivated specimens

7858 03/06/2013 06:49 PM Aaron Marcuse-Kubitza

inputs/UNCC/_src/_README.TXT: Added e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet about the meanings of the Leaf, Flower, Fruit, and Root codes

7857 03/06/2013 05:38 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: Added steps to manually back up the DB. (These steps are specific to the import process, and include filters to exclude previous imports. There are also general backup steps in the Backups > Full DB section.)

7856 03/06/2013 02:56 PM Aaron Marcuse-Kubitza

inputs/ACAD/Specimen/map.csv: institutionCode: Replace "Acadia University" with the IH code ACAD

7855 03/06/2013 02:18 PM Aaron Marcuse-Kubitza

inputs/.TNRS/data.sql: Re-ran TNRS on the test_taxonomic_names so that the sample data would contain all the accepted names under the current acceptedScientificName formula, and would not produce any entries for unscrubbed names in tnrs_input_name

7854 03/06/2013 01:59 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Don't exclude taxonlabels with a matched_label_id, because matched_label_id is no longer used to indicate that a name has a scrubbed name (this was used when all of TNRS was imported at the beginning of the import)

7853 03/06/2013 01:56 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Renamed to the VegCore table name AcceptedTaxon

7852 03/06/2013 01:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Use new MatchedTaxon view, which use VegCore column names where possible

7851 03/06/2013 01:46 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Use new MatchedTaxon, AcceptedTaxon views, which use VegCore column names where possible

7850 03/06/2013 01:22 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: Added instructions to run TNRS and view the TNRS log

7849 03/06/2013 10:21 AM Aaron Marcuse-Kubitza

README.TXT: Full database import: Removed no longer necessary command to scrub unscrubbed taxondeterminations

7848 03/05/2013 11:07 PM Aaron Marcuse-Kubitza

*.sql: concat_ws() calls: Fixed bug where need to surround it with NULLIF because concat_ws() returns '' instead of the required NULL when supplied with only NULL args

7847 03/05/2013 10:22 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: concat_ws() calls: Fixed bug where arguments were still enclosed in ARRAY[], because the name had been changed from concat_delim() but PostgreSQL removed the VARIADIC keyword without also removing the ARRAY[] wrapper

7846 03/05/2013 10:11 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Removed no longer needed unscrubbed_taxondetermination_view. Use taxonlabel joined to ScrubbedTaxon instead.

7845 03/05/2013 10:05 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: ScrubbedTaxon: Include only matches with a match score of at least 0.8 (score_ok())

7844 03/05/2013 09:59 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Added back score_ok() and tnrs.tnrs_score_ok index

7843 03/05/2013 09:53 PM Aaron Marcuse-Kubitza

*.sql: Replaced concat_delim() with concat_ws(), which is built-in as of PostgreSQL 9.1 (http://stackoverflow.com/questions/1943433/postgresql-concat-ws-like-function)

7842 03/05/2013 09:08 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: Populated specimenHolderInstitutions

7841 03/05/2013 09:07 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Renamed specimenHolders to specimenHolderInstitutions to make it obvious that this is a list of institutions, such as would be in institutionCode in a DwC export.

7840 03/05/2013 09:06 PM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: Fixed embedded line endings

7839 03/05/2013 08:28 PM Aaron Marcuse-Kubitza

inputs/NY/Ecatalog_all/map.csv: PlantFungDescription->occurrenceRemarks: Removed "plant fungus" label because this field actually contains the specimenDescription

7838 03/05/2013 08:06 PM Aaron Marcuse-Kubitza

inputs/.TNRS/import_order.txt: Removed public.unscrubbed_taxondetermination_view

7837 03/05/2013 08:01 PM Aaron Marcuse-Kubitza

inputs/.TNRS/: Moved no longer needed public.unscrubbed_taxondetermination_view/ to _archive/

7836 03/05/2013 07:52 PM Aaron Marcuse-Kubitza

root Makefile: Datasources: scrub: Removed no longer needed scrub-remake, which has been replaced by direct use of TNRS.ScrubbedTaxon

7835 03/05/2013 07:46 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: taxondetermination: Fixed bug where need to include only the is_datasource_current taxondetermination

7834 03/05/2013 07:43 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Use new TNRS.ScrubbedTaxon directly instead of the scrubbed taxondeterminations added to the normalized DB

7833 03/05/2013 07:20 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: Renamed matchedTaxonomicStatus to preferred name taxonomicStatus

7832 03/05/2013 07:19 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Taxon: Moved scrubbing-related terms to MatchedTaxon.

7831 03/05/2013 07:16 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Added matched* prefix where needed

7830 03/05/2013 07:02 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: Added missing Name_submitted column

7829 03/05/2013 06:59 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: Name_matched: Fixed mapping to matchedTaxonName

7828 03/05/2013 06:56 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: Fixed bug where can't map Source to source because it refers just to the source of the matched name, rather than of the record (which would be TNRS itself)

7827 03/05/2013 06:55 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_derived_fields(): Use new concat_delim()

7826 03/05/2013 06:43 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _join*(): Use new concat_delim()

7825 03/05/2013 06:41 PM Aaron Marcuse-Kubitza

schemas/functions.sql: Added concat_delim() (also in public, but this copy is for use by the functions schema)

7824 03/05/2013 06:37 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Removed no longer used tnrs+accepted. Use ScrubbedTaxon instead.

7823 03/05/2013 06:22 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: Added MatchedTaxon, AcceptedTaxon, ScrubbedTaxon views, which rename the columns to VegCore names

7822 03/05/2013 05:17 PM Aaron Marcuse-Kubitza

Removed inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump*, which has been replaced by GBIFPortalDB-2013-02-20.dump

7821 03/05/2013 05:14 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Renamed MatchedTaxonDetermination/AcceptedTaxonDetermination to MatchedTaxon/AcceptedTaxon because they relate to a taxon name rather than to an application of that name to a TaxonOccurrence

7820 03/05/2013 02:43 PM Aaron Marcuse-Kubitza

schemas/functions.sql: functions._map(): Made function STRICT because it always returns NULL if either argument is NULL

7819 03/05/2013 02:39 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: campus: Mapped to cultivated, interpreting non-NULL values as true

7818 03/05/2013 02:29 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: campus: Removed mapping to occurrenceRemarks because this is just a code, not a description. Added definition from Bob's validation comments.

7817 03/05/2013 02:19 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: accession: Removed comment that this is actually the catalogNumber, because they are now synonyms

7816 03/05/2013 01:56 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Split specimenIndexer into specimenStampingInstitution and specimenDonor (which are often the same).

7815 03/05/2013 12:25 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed specimenIndexer->institutionCode mappings because the institutionCode should refer only to the specimenHolder

7814 03/05/2013 12:18 PM Aaron Marcuse-Kubitza

Added inputs/GBIF/_src/GBIFPortalDB-2013-02-20.dump.gz (a new download they posted to the FTP site)

7813 03/05/2013 12:13 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_specimen: Removed growthForm, threatened_bien at Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#UNCC > Brad Boyle's comments > growthForm, threatened_bien)

7812 03/05/2013 11:24 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: Removed mappings to reproductiveCondition because we don't know what the single-letter codes mean

7811 03/05/2013 11:16 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: Fixed embedded line endings

7810 03/05/2013 11:05 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: _locationnarrative_is_cultivated(): Fixed bug where need to use NULL instead of false (and bool_or() instead of OR) for a non-match, because the absence of a match doesn't mean it's not cultivated

7809 03/05/2013 03:02 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Editing: Changed ambiguous terms templates to list the name of the alternative itself in the alternatives list, instead of the ambiguous term + a source suffix. This avoids the need to choose a source suffix for each alternative of an ambiguous term, as the ambiguous term itself (without the suffix) can be listed as the synonym. Coordinates: Disambiguated verbatimCoordinateSystem.

7808 03/05/2013 02:30 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.thesaurus.csv: `uniq`ify the map after applying initial translations, to support mappings that appear twice (once as the result of an alternative's synonym entry and again as the result of an ambiguous term's alternatives listing)

7807 03/04/2013 11:57 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Added temperature_C, precipitation_m for possible use in John Wiens' plant climatic niche evolution project ("how fast is the rate of climatic niche evolution among species compared to projected changes in climate over the next 100 years? How do these rates compare for temperature and precipitation variables?")

7806 03/01/2013 08:17 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Added specimenDescription.

7805 03/01/2013 08:00 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Split apart identificationQualifier and taxonFit. Split apart taxonConfidence and identificationVerificationStatus. Moved identificationQualifier to new QualifiedTaxonName table.

7804 03/01/2013 07:48 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: To view progress: Use tail -100 to show more of the log files, and not tail -f to avoid blocking when the user just wants to check the status

7803 03/01/2013 04:56 PM Aaron Marcuse-Kubitza

inputs/BRIT/specimen_flat/map.csv: Only map Notes_Plant to occurrenceRemarks if there is a separate Locality_Description, in order to filter out Notes_Plant values that concatenate everything together (often including a herbarium name that would mess up the cultivated flag parsing)

7802 03/01/2013 04:49 PM Aaron Marcuse-Kubitza

my2pg.data: Replace \0 (with the backslash) with 0 because \0 isn't allowed in UTF-8

7801 03/01/2013 03:53 PM Aaron Marcuse-Kubitza

root Makefile: VegBIEN DB: mk_db: Added command to import arbitrary DB users from a config file

7800 02/28/2013 03:31 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

7799 02/28/2013 03:29 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen2/map.csv: _join() PLANT DATA and MISC NOTES to form occurrenceRemarks instead of _alt()ing them

7798 02/28/2013 03:27 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen2/test.xml.ref: Accepted test

7797 02/28/2013 02:58 PM Aaron Marcuse-Kubitza

inputs/BRIT/_MySQL/MySQL.*.sql.make: Include all tables, not just specimen_flat

7796 02/28/2013 02:57 PM Aaron Marcuse-Kubitza

my2pg: Replace tinytext, mediumtext with text

7795 02/28/2013 11:48 AM Aaron Marcuse-Kubitza

Added inputs/GBIF/_src/_README.TXT with Brad's GBIF filtering rules

7794 02/28/2013 11:30 AM Aaron Marcuse-Kubitza

inputs/FIA/TREE/postprocess.sql: Use :table instead of hardcoding the table name

7793 02/28/2013 11:26 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: UNIQUE constraint: Added UNITCD because although it may not be needed to scope PLOT, it is present in all the core tables and thus must be part of the joins' USING columns

7792 02/28/2013 10:58 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql for core tables: Added unique constraint on the datasource's primary key fields

7791 02/28/2013 02:38 AM Aaron Marcuse-Kubitza

Added inputs/CTFS/AggregateObservation/ from BIEN2 data

7790 02/28/2013 02:16 AM Aaron Marcuse-Kubitza

inputs/CTFS/: Switched global _no_import to table-specific _no_imports to allow adding new tables that are imported

7789 02/27/2013 09:54 PM Aaron Marcuse-Kubitza

Added schemas/TAPIR/SpeciesLink.tapir_request.txt symlink

7788 02/27/2013 09:53 PM Aaron Marcuse-Kubitza

Added inputs/SpeciesLink/_src/tapir_request.txt

7787 02/27/2013 09:52 PM Aaron Marcuse-Kubitza

Added schemas/TAPIR/

7786 02/27/2013 09:48 PM Aaron Marcuse-Kubitza

Added planning/use_cases/rare_species.png

7785 02/27/2013 09:47 PM Aaron Marcuse-Kubitza

Added planning/goals/BIEN deliverables.png

7784 02/27/2013 09:31 PM Aaron Marcuse-Kubitza

Added planning/resources/iPlant_Overview_BIEN_2011.pdf.url

7783 02/27/2013 09:29 PM Aaron Marcuse-Kubitza

planning/goals/Enquist_BIEN_Overview_2011.pptx: Replaced with .url file. Note that the Plone site <http://groups.nceas.ucsb.edu/bien/meetings> is not searchable by document filename.

7782 02/27/2013 09:26 PM Aaron Marcuse-Kubitza

Added planning/workflow/Boyle_BIEN_data_quality.pdf.url

7781 02/27/2013 09:21 PM Aaron Marcuse-Kubitza

Added planning/goals/BIEN_Science_2011.pptx.url, BIEN-Tech-Oct2011.ppt.url, Enquist_BIEN_Overview_2011.pptx

7780 02/27/2013 09:00 PM Aaron Marcuse-Kubitza

Added schemas/VegX/run.py

7779 02/27/2013 08:53 PM Aaron Marcuse-Kubitza

Added schemas/VegBIEN/growth_forms/Barbara_Dobrin.parsing.scripts/

7778 02/27/2013 08:35 PM Aaron Marcuse-Kubitza

Added schemas/VegX/Veg-X_paper_in_press_Applied_Vegetation_Science.pdf.url

7777 02/27/2013 08:25 PM Aaron Marcuse-Kubitza

Added schemas/VegX/JVS-VegX.pdf.url

7776 02/27/2013 08:22 PM Aaron Marcuse-Kubitza

Added schemas/VegBank/VegBank_Data_Model.with_Collections_intro.ppt

7775 02/27/2013 08:13 PM Aaron Marcuse-Kubitza

Added planning/goals/BIEN_White_Paper.pdf.url

7774 02/27/2013 08:11 PM Aaron Marcuse-Kubitza

Added planning/meetings/2010-12-08.working_group/BIEN3-plan.docx

7773 02/27/2013 08:04 PM Aaron Marcuse-Kubitza

Added schemas/VegBIEN/planning/taxonomy/On_the_Use_of_Taxonomic_Concepts_in_Support_of_Biodiversity_Research_and_Taxonomy.pdf.url

7772 02/27/2013 07:59 PM Aaron Marcuse-Kubitza

Added schemas/BIEN2/ERD/