Project

General

Profile

Statistics
| Revision:

# Date Author Comment
6855 12/14/2012 09:29 AM Aaron Marcuse-Kubitza

inputs/*/Source/map.csv without mappings: Added referenceType, etc. mappings. This also ensures that the source table entry for the datasource will be created before the herbaria list is imported, causing all top-level datasources to sort at the top of the source table.

6854 12/14/2012 09:02 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Granted the public_ user read-only access to the contents of the source table

6853 12/14/2012 08:53 AM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: $(editPhppgadmin): Ignore errors if patch has already been applied

6852 12/14/2012 08:52 AM Aaron Marcuse-Kubitza

lib/phpPgAdmin.config.inc.php.diff: Remove context so segment matching would depend only on the $conf['extra_login_security'] line itself

6851 12/14/2012 08:29 AM Aaron Marcuse-Kubitza

mappings/Makefile: Added VegCore.tables.redmine, which contains the Redmine-formatted list of VegCore tables to paste into <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Tables>

6850 12/14/2012 08:18 AM Aaron Marcuse-Kubitza

mappings/: Removed no longer used VegCore.redmine. VegCore.csv is now generated from the Redmine page instead of the other way around.

6849 12/14/2012 08:12 AM Aaron Marcuse-Kubitza

mappings/Makefile: Added VegCore.tables.csv, which contains all the tables in the VegCore data dictionary

6848 12/14/2012 06:59 AM Aaron Marcuse-Kubitza

README.TXT: Data import: backups/fix_perms: Run using sudo to also change permissions on files owned by the bien user, and to change the owner of files owned by you to the bien user

6847 12/14/2012 06:45 AM Aaron Marcuse-Kubitza

Regenerated mappings/VegCore.csv, which adds categories

6846 12/14/2012 05:47 AM Aaron Marcuse-Kubitza

README.TXT: Maintenance: Added instructions to regenerate mappings/VegCore.csv whenever the VegCore data dictionary page is changed

6845 12/14/2012 05:41 AM Aaron Marcuse-Kubitza

mappings/Makefile: Generate VegCore.csv from the VegCore data dictionary page by extracting all HTML anchors (in Redmine, each section heading, and therefore each VegCore term, gets its own anchor)

6844 12/14/2012 05:34 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Changed line endings to \n to match what sed generates from the VegCore data dictionary page

6843 12/14/2012 05:31 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Removed informational columns, because this information is now maintained on the VegCore data dictionary page at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore>

6842 12/14/2012 05:11 AM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv.csv: Removed hypothetical terms which are not in use by any VegBIEN datasource

6841 12/14/2012 05:00 AM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: habit: Remapped to growthForm, which replaces verbatimGrowthForm

6840 12/14/2012 04:59 AM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv.csv: Removed hypothetical terms which are not in use by any VegBIEN datasource

6839 12/14/2012 04:50 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: BIEN2 terms: Added sub-namespaces (bien_web, geoscrub, etc.) to source URLs

6838 12/14/2012 04:15 AM Aaron Marcuse-Kubitza

dict2redmine: redmine_add_links(): Hyperlink just the source name, not also the () around it

6837 12/14/2012 03:54 AM Aaron Marcuse-Kubitza

dict2redmine: RedmineDictWriter: Use h2 instead of h3 for the term name so that the term will be normal-sized instead of smaller in the Redmine table of contents

6836 12/14/2012 03:52 AM Aaron Marcuse-Kubitza

dict2redmine: Renamed redmine_url() to redmine_link() because it generates links, not URLs

6835 12/14/2012 03:49 AM Aaron Marcuse-Kubitza

dict2redmine: redmine_add_links(): Put citations in () instead of [] to avoid conflicting with the Redmine syntax for internal links ( ... )

6834 12/14/2012 03:18 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Terms: Removed namespace prefixes (dcterms:), because VegCore terms are globally unique within VegCore and there should not be multiple versions of the same VegCore term with different namespaces. Provenance is instead indicated in the Sources column, which contains not just a namespace but a full URL to each source term.

6833 12/14/2012 03:00 AM Aaron Marcuse-Kubitza

dict2redmine: Hyperlink each term to its anchor in the data dictionary, rather than to its first source, which is not necessarily the definitive definition of the term. This also allows clicking the term to get its permalink in the address bar, rather than having to click the small, light gray paragraph mark next to the term name that Redmine provides.

6832 12/14/2012 02:57 AM Aaron Marcuse-Kubitza

dict2redmine: redmine_add_links(): Fixed bug where need to avoid matching internal links ( ... ) as citations ( [...] )

6831 12/14/2012 02:46 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Term names: Changed special characters to _ because Redmine doesn't support special characters in HTML anchors (it removes everything except letters, numbers, _, and -)

6830 12/14/2012 02:42 AM Aaron Marcuse-Kubitza

mappings/Makefile: .Veg+-VegCore.csv.last_cleanup: Also canon the output (VegCore) column to the VegCore.csv vocabulary. ? prefixes are not a problem because there are always at least two alternatives listed for these terms, so canon will not modify the output field.

6829 12/14/2012 01:49 AM Aaron Marcuse-Kubitza

psql_script_vegbien: Run psql_vegbien with `nice -n +5` to prevent CPU-intensive operations from slowing down the shell/UI

6828 12/14/2012 01:46 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

6827 12/14/2012 01:37 AM Aaron Marcuse-Kubitza

Regenerated inputs/CVS/taxonObservation_/new_terms.csv. Note that it includes mappings to terms which are not in mappings/VegCore-VegBIEN.csv, which are prefixed with *.

6826 12/14/2012 01:34 AM Aaron Marcuse-Kubitza

input.Makefile: Maps validation: %/new_terms.csv: Undid incorrect change of column to filter terms out of. This actually needs to be the input column, even though unmapped_terms.csv is generated from the output column, because it's possible to have a mapping to a term which is not in mappings/VegCore-VegBIEN.csv, and such a term would show up in unmapped_terms.csv but should not be filtered out of new_terms.csv.

6825 12/14/2012 01:17 AM Aaron Marcuse-Kubitza

lib/phpPgAdmin.login.php.diff: public_ user's password message: Print as its own message instead of appending it to $msg. Print it before any error message so it always appears at the top of the page.

6824 12/14/2012 12:51 AM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: phpPgAdmin: Edit config file to allow passwordless logins. Edit login page to fill in public_ as the default username and add a message to leave the password blank for that user.

6823 12/12/2012 10:45 PM Aaron Marcuse-Kubitza

root Makefile: $(postgresReload-*): Ignore `mv -n` errors, which generally indicate that the existing *.conf was already renamed to *.conf.old

6822 12/12/2012 10:36 PM Aaron Marcuse-Kubitza

Makefile mk_db, schemas/pg_hba*.conf: Added passwordless public_ user with access to just the database schema. Note that in PostgreSQL, only users with explicit GRANT permissions on a table can read data in that table, but all DB users with a login can view all table schemas.

6821 12/12/2012 10:26 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: system updates that affect PostgreSQL: Added that this applies to both Linux and Mac OS X

6820 12/12/2012 10:26 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: system updates that affect PostgreSQL: list of things that could break if PostgreSQL is not restarted: Added that you may not be able to access the database as the postgres superuser

6819 12/12/2012 10:24 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: system updates that affect PostgreSQL: list of things that could break if PostgreSQL is not restarted: Added that you may not be able to access the database as the postgres superuser

6818 12/12/2012 09:40 PM Aaron Marcuse-Kubitza

backups/fix_perms: Removed world read permissions from backups dir. Note that this will require superuser permissions to view archived backups on jupiter, because the bien group is not set up with the same members as on vegbiendev. (On jupiter, it contains only stri,regetz,donoghue,naiamh.)

6817 12/12/2012 08:55 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: Mapped plantname, plantNameWithAuthority

6816 12/12/2012 08:47 PM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.utils.sql: plantconcept_plantnames(): Use CVS's taxonLevel values, which are different from the VegBank plantLevel values that the original version of this function used

6815 12/12/2012 08:25 PM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.utils.sql: plantconcept_*(): Use plantConcept.lowestParentConcept_ID,taxonLevel instead of plantStatus.plantParent_ID,plantLevel to find the plantConcept's ancestors, because CVS does not use plantStatus except in very few cases and instead puts the parent link directly in plantConcept

6814 12/12/2012 08:09 PM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.utils.sql: plantconcept_plantnames(): Made function STABLE instead of VOLATILE because it does not modify any tables

6813 12/12/2012 08:08 PM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.utils.sql: plantconcept_plantnames(): Made function STABLE instead of VOLATILE because it does not modify any tables

6812 12/12/2012 06:57 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Removed no longer used verbatimGrowthForm. Use growthForm instead.

6811 12/12/2012 06:56 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed no longer used verbatimGrowthForm. Map to growthForm instead and translate growth form values to VegBIEN's growthform enum.

6810 12/12/2012 06:54 PM Aaron Marcuse-Kubitza

inputs/Madidi/Organism/map.csv: Habit: Mapped growth form values

6809 12/12/2012 06:39 PM Aaron Marcuse-Kubitza

inputs/Madidi/Organism/map.csv: Remapped Habit from verbatimGrowthForm to growthForm, which points to the same place

6808 12/12/2012 06:27 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: Use denorm_* denormalized taxonomic ranks in place of the normalized ranks when both are provided

6807 12/12/2012 06:25 PM Aaron Marcuse-Kubitza

input.Makefile: Maps validation: %/new_terms.csv: Fixed bug where need to filter unmapped_terms.csv's terms out of the output column, not the input column, because that's what the unmapped terms are generated from. Usually these columns are the same for unmapped terms, but sometimes an output term is changed from the original column's name but still doesn't match a VegCore term in mappings/VegCore-VegBIEN.csv.

6806 12/12/2012 06:08 PM Aaron Marcuse-Kubitza

input.Makefile: SVN: add: Added comment with instructions to update all inputs with these settings, using `make inputs/add`

6805 12/12/2012 06:07 PM Aaron Marcuse-Kubitza

input.Makefile: SVN: add: verify: Also ignore *.xlsx

6804 12/12/2012 06:00 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Creating enough disk space: Added instructions for removing archived backups to free up space

6803 12/12/2012 05:15 PM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: Fixed bug where taxonLevel, not taxonRank, needs to be mapped to taxonRank, because CVS's taxonRank is actually a number, while taxonLevel contains the corresponding text string

6802 12/12/2012 05:12 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Before import, added step to make sure there is at least 100GB of disk space

6801 12/12/2012 04:41 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): is_function: Fixed bug where need to add the pkeys table's test pkey constraint after the data is added rather than when the empty table is created, to avoid adding a pkey constraint that will later be violated by data which returns multiple output rows for an input row (such as calls to _split())

6800 12/12/2012 04:36 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): insert_into_pkeys(): Allow callers to override run_query_into()'s add_pkey_ param in case the initial version of the pkeys table should not yet have the test pkey constraint (e.g. because data is added after the table is created)

6799 12/12/2012 04:24 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Checking for errors: Search for "Command exited with non-zero status" to find errors, which is faster than checking that each input's log ends in "Encountered 0 error(s)"

6798 12/12/2012 04:13 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

6797 12/12/2012 03:50 PM Aaron Marcuse-Kubitza

README.TXT: Data import: import_all: Corrected text of note about time until control is returned to the shell

6796 12/12/2012 03:42 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Moved download of logs to right after the import is done, because this is a quick step that doesn't depend on the backup- and export-creation steps

6795 12/11/2012 11:41 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: institutionCode: Removed mapping to sourcename.matched_source_id, which is now autopopulated. Split any list of institutionCodes apart using new _split().

6794 12/11/2012 11:28 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcename: Added sourcename_set_matched_source_id() trigger

6793 12/11/2012 11:22 AM Aaron Marcuse-Kubitza

schemas/functions.sql: Added _split()

6792 12/11/2012 11:13 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcelist_unique: Removed COALESCE around name because it's NOT NULL

6791 12/11/2012 11:11 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Allow multiple institutionCodes for each specimenreplicate by linking new sourcelist table many-to-many to source via sourcename (which is now a linking table)

6790 12/11/2012 10:50 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcename: Removed system, which has been replaced by source_id as the scoping field

6789 12/11/2012 10:42 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: party: Added sourceaccessioncode and uniquify on it instead when provided. vegbien.ERD.mwb: Rearranged party-related tables to allow the tables to be fully expanded.

6788 12/11/2012 10:40 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Renamed sampletype to observationtype to match the VegCore term

6787 12/11/2012 10:09 AM Aaron Marcuse-Kubitza

Added inputs/SALVIAS/salvias_users.~.clean_up.sql

6786 12/11/2012 10:01 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/: Added salvias_users tables

6785 12/11/2012 10:00 AM Aaron Marcuse-Kubitza

my2pg: Translate blob to bytea

6784 12/11/2012 09:55 AM Aaron Marcuse-Kubitza

my2pg: Also remove UNIQUE and FULLTEXT inline indexes

6783 12/11/2012 08:55 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: UNUSED: Comments: Added Redmine formatting

6782 12/11/2012 08:55 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: OMIT: Changed "is omitted" to "should be omitted", because the mappings specify suggestions rather than requirements as to how a field should be used

6781 12/11/2012 08:51 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Removed no longer used subInstitutionCode. Use datasource, institutionCode instead.

6780 12/11/2012 08:51 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_*: Renamed subInstitutionCode to institutionCode because this is the institution storing the specimen, as defined by DwC

6779 12/11/2012 08:45 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_*: Renamed institutionCode to datasource because this is actually the top-level datasource providing the record, not the institution storing the specimen

6778 12/11/2012 08:38 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added datasource

6777 12/11/2012 08:32 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Renamed sampletype to observationtype to match the VegCore term

6776 12/11/2012 08:16 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: referenceType: Added closed list values

6775 12/11/2012 08:08 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: observationMeasure: Re-sourced to SALVIAS:observation_type, since SALVIAS comes before VegBIEN in the source precedence

6774 12/11/2012 08:01 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Renamed sampleType to observationType to match the SALVIAS term it's derived from

6773 12/11/2012 07:57 AM Aaron Marcuse-Kubitza

inputs/SALVIAS-CSV/Plot/map.csv: Mapped observation_type

6772 12/11/2012 07:37 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added individualCount_*cm_or_more used by analytical_aggregate

6771 12/11/2012 07:10 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: subplotX/Y: Added definition

6770 12/11/2012 07:07 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: subplot*: Sources: Put SALVIAS:subplot last, because the specific field is closer in meaning to the term than the category

6769 12/11/2012 07:02 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: project*Date: Added definition

6768 12/11/2012 06:58 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: parentPlotName: Added definition

6767 12/11/2012 06:56 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: parent*: Sources: Put VegBank:PARENT_ID last, because the specific field is closer in meaning to the term than the category

6766 12/11/2012 06:40 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: locationName: Added definition

6765 12/11/2012 06:33 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: eventDate/startDate/endDate: Added definition

6764 12/11/2012 06:27 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: locationName: Sources: Put VegX:plotName first because it is closest in meaning to the term

6763 12/11/2012 06:22 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: recordedBy*: Added definition

6762 12/11/2012 06:15 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: recordedBy.middleName: Added source to DwC:recordedBy

6761 12/11/2012 06:01 AM Aaron Marcuse-Kubitza

inputs/CVS/: Joined together stemCount and stemLocation tables to create stemLocation_, in order to include the stem size class's measurements in each tagged stem's stemobservation (in addition to in the stemobservation for the aggregateoccurrence as a whole)

6760 12/11/2012 05:32 AM Aaron Marcuse-Kubitza

inputs/CVS/: Joined together taxonImportance and stemCount tables to create stemCount_, because stemCount actually stores stem abundance by size, rather than grouping stems by organism (http://vegbankdev.nceas.ucsb.edu/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename), and is thus an AggregateOccurrence-related table along with taxonImportance

6759 12/11/2012 05:30 AM Aaron Marcuse-Kubitza

inputs/CVS/: Joined together taxonImportance and stemCount tables to create stemCount_, because stemCount actually stores stem abundance by size, rather than grouping stems by organism (http://vegbankdev.nceas.ucsb.edu/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=stemcount&entity=dba_tabledescription&where=where_tablename), and is thus an AggregateOccurrence-related table along with taxonImportance

6758 12/11/2012 04:41 AM Aaron Marcuse-Kubitza

inputs/CVS/taxonObservation_/map.csv: Fixed bug where need to indicate that data is plots data to prevent the specimenreplicate ID from being forwarded to the location ID

6757 12/11/2012 04:38 AM Aaron Marcuse-Kubitza

inputs/VegBank/taxonobservation_/map.csv: Fixed bug where need to indicate that data is plots data to prevent the specimenreplicate ID from being forwarded to the location ID

6756 12/11/2012 04:37 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Don't forward specimenreplicate IDs to location for plots data (where the specimenreplicate IDs apply only to the specimen)