Project

General

Profile

Statistics
| Revision:
  • svn:ignore: .~*

# Date Author Comment
12367 02/23/2014 12:13 PM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: _traits_08_taxonname_trait_and_value_for_first_5000_records: renamed to _traits_08_taxonname_trait_and_value because this actually includes all the records, not just the first 5000. this uses the new public_validations.rename_query_view() to rename all associated tables and views, including handling truncated names.

12286 02/17/2014 01:58 PM Aaron Marcuse-Kubitza

bugfix: inputs/bien2_traits/validations.sql: _traits_01_count_records: changed column names to match public_validations._traits_01_count_records

12246 02/16/2014 04:22 PM Aaron Marcuse-Kubitza

bugfix: inputs/bien2_traits/validations.sql: use a wrapper function for util.ifnull() so that the views don't get dropped when the util schema is reinstalled

12224 02/14/2014 03:09 PM Aaron Marcuse-Kubitza

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

12221 02/14/2014 12:20 PM Aaron Marcuse-Kubitza

added inputs/bien2_traits/validations.sql, from validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql

12220 02/14/2014 12:20 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(svnFilesGlob): added validations.sql

12213 02/14/2014 11:00 AM Aaron Marcuse-Kubitza

added inputs/bien2_traits/validations.sql.run

12158 02/13/2014 07:26 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

12157 02/13/2014 06:53 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

12112 02/07/2014 05:37 AM Aaron Marcuse-Kubitza

fix: inputs/VegBIEN/Redmine/wiki/.htaccess: redirect to new main page when accessed without trailing /

12060 02/06/2014 01:14 PM Aaron Marcuse-Kubitza

inputs/bien2_traits/TraitObservation/postprocess.sql: remove rows with no taxon name, which are invalid, and which helps simplify the aggregating validations queries

12053 02/06/2014 03:05 AM Aaron Marcuse-Kubitza

fix: inputs/VegBIEN/Redmine/svn/.htaccess: updated repository URL to point to trunk/

12040 02/04/2014 10:42 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/verify/plots.out.sql: fixed ' quoting syntax to use '' instead of \' to escape '

12039 02/04/2014 10:32 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: verify/%.out: use a *.sql file in the verify/ directory itself to generate *.out, so that each datasource can have its own set of output queries. for datasources that should share the same set of queries, they can instead be symlinked to the same file.

12038 02/04/2014 10:01 AM Aaron Marcuse-Kubitza

fix: inputs/CVS/project/: added _no_import since this should not also be imported separately from taxon_observation.**

12030 02/02/2014 11:18 PM Aaron Marcuse-Kubitza

added inputs/XAL/Specimen/_no_import, since this is a demo-only datasource and there isn't a staging table for it

12029 02/02/2014 11:10 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/county_centroids/test.xml.ref, inputs/.NCBI/{names.src,nodes.src}/test.xml.ref: accepted test outputs (generated now that these tables are in import_order.txt)

12028 02/02/2014 10:31 PM Aaron Marcuse-Kubitza

inputs/FIA/taxon_observation.**/header.csv: updated for new REF_RESEARCH_STATION.country metadata value col

12018 02/02/2014 12:49 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: add!: verify/: also svn:ignore *.tsv, *.txt

12005 01/23/2014 01:44 AM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

12004 01/23/2014 01:43 AM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

12003 01/23/2014 01:35 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: remove projects that do not have "There are no specific use conditions attached to this dataset"

12002 01/23/2014 01:22 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/salvias_plots.~.clean_up.sql: Remove private data that should not be publicly visible: also need to remove metadata-only plots

12001 01/23/2014 01:11 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/plotMetadata_/map.csv: things mapped to project_participant: remapped to event__participant because these actually relate to the event, not the project, even though they seem like project-related fields

11999 01/23/2014 12:54 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/plotMetadata_/map.csv, inputs/Madidi/LocationObservation/map.csv: things mapped to communityID: remapped to communityName, which is what's used in analytical_stem (communityID is for numeric IDs)

11997 01/22/2014 11:01 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/create.sql, map.csv: expanded plot_administrator:party_code_party_ and mapped plot_administrator_name to a 2nd project_participant

11996 01/22/2014 10:59 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: project_participant: use [!...] negative lookahead assertion so that multiple project_participant columns will properly map to separate projectcontributor rows

11994 01/22/2014 09:16 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/map.csv: mapped PrimOwnerID_name->project_participant

11993 01/22/2014 09:07 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/create.sql: added join to PrimOwnerID:party_code_party_

11992 01/22/2014 01:06 PM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/import_order.txt: added party_code_party_

11991 01/22/2014 12:50 PM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/party_code_party_/create.sql: need to remove duplicate entries in party_code_party

11988 01/22/2014 11:10 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/party_code_party_/map.csv: mapped fullname->event_participant_name for use by other tables

11987 01/22/2014 10:34 AM Aaron Marcuse-Kubitza

mapped inputs/SALVIAS/party_code_party_/

11983 01/20/2014 10:12 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/_MySQL/salvias_plots.*.sql: refreshed. this adds the party and party_code_party tables Brad provided for mapping the plot contributors.

11982 01/20/2014 10:10 PM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/salvias_plots.~.clean_up.sql: Delete rows that do not satisfy foreign key constraints: also need to do this for plotObservations, since the refreshed data contains dangling rows for that as well

11981 01/20/2014 10:08 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/run_: documented *.sql install runtime (3 min), as separate from the full `datasrc_make reinstall` runtime (3.5 min)

11980 01/20/2014 10:07 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/run_: refresh(): `datasrc_make reinstall`: updated runtime. documented that runtimes are from starscream.

11979 01/20/2014 08:09 PM Aaron Marcuse-Kubitza

added inputs/SALVIAS/run_, which includes a refresh() target

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11965 01/16/2014 01:22 AM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: scrubbed_family: Name_matched_accepted_family was missing from the TNRS results at one point, so we are now using Family_matched as a workaround to populate this. the workaround is for accepted names only, as no opinion names do not have an Accepted_name_family to prepend to the scrubbed name to parse.

11964 01/16/2014 01:19 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: reexported from live DB, which changes the element order

11961 01/15/2014 10:18 AM Aaron Marcuse-Kubitza

inputs/VegBank/import_order.txt: added projectcontributor_

11960 01/15/2014 10:11 AM Aaron Marcuse-Kubitza

inputs/VegBank/projectcontributor_/map.csv, postprocess.sql: added project_participant

11957 01/15/2014 09:41 AM Aaron Marcuse-Kubitza

added inputs/VegBank/projectcontributor_/

11956 01/15/2014 09:29 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: projectcontributor.surname: prepend table name to avoid join collisions

11955 01/15/2014 09:23 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql, inputs/CVS/cvs.~.clean_up.sql: Prevent "column name specified more than once" errors when tables are joined: put tables in alphabetical order for consistency

11943 01/14/2014 08:34 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

11942 01/14/2014 08:31 PM Aaron Marcuse-Kubitza

inputs/datasource_release_status.xlsx: renamed to `publishable datasources.xlsx` to match the spreadsheet title

11934 12/20/2013 04:41 PM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql, map.csv: added new project columns

11933 12/20/2013 04:31 PM Aaron Marcuse-Kubitza

inputs/VegBank/taxon_observation.**/postprocess.sql: added the project table

11932 12/20/2013 04:25 PM Aaron Marcuse-Kubitza

mapped inputs/VegBank/project/, which includes the projectName for attribution

11931 12/20/2013 02:56 PM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/create.sql, map.csv: added new project columns

11930 12/20/2013 02:44 PM Aaron Marcuse-Kubitza

inputs/CVS/taxon_observation.**/postprocess.sql: added the project table

11929 12/20/2013 02:42 PM Aaron Marcuse-Kubitza

inputs/CVS/project/map.csv: mapped stopDate->projectEndDate

11928 12/20/2013 02:35 PM Aaron Marcuse-Kubitza

mapped inputs/CVS/project/, which includes the projectName for attribution

11927 12/20/2013 01:25 AM Aaron Marcuse-Kubitza

inputs/VegBIEN/Redmine/svn/.htaccess: updated to use much faster direct repository URL rather than Redmine web interface, now that the repository itself is publicly accessible in addition to the Redmine view of it

11924 12/20/2013 12:28 AM Aaron Marcuse-Kubitza

fix: inputs/TEX/Specimen*/map.csv, postprocess.sql: habitat: also placed in occurrenceRemarks so that this field gets parsed for growth form information, as requested by Brad (wiki.vegpath.org/TEX_validation#2013-2-26)

11923 12/19/2013 11:49 PM Aaron Marcuse-Kubitza

fix: inputs/TEX/Specimen*/map.csv: mapped constant values for specimenHolderInstitutions, country. these have to be added with `rm=1 ./inputs/TEX/Specimen.../run postprocess`.

11922 12/19/2013 11:42 PM Aaron Marcuse-Kubitza

bugfix: inputs/TEX/Specimen2/map.csv: mapped BARCODE to accessionNumber so that we have a unique ID for each row

11920 12/17/2013 08:06 AM Aaron Marcuse-Kubitza

inputs/datasource_release_status.xlsx: updated

11917 12/16/2013 07:05 PM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/create.sql: added Mike Lee's additional plots used to validate confidentiality-related fields (wiki.vegpath.org/CVS_validation#plots-to-include)

11916 12/16/2013 06:00 PM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/^taxon_observation.**.sample/create.sql: include taxonName in the subset of columns that's imported for the validation, because it is _alt-ed with scientificName for forming the TNRS input name. this is unique to CVS, which is why it was not part of the validation subset copied from the VegBank subset.

11912 12/16/2013 01:43 PM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: granted bien_read SELECT access to derived views as well as the core tnrs table

11911 12/15/2013 05:30 PM Aaron Marcuse-Kubitza

updated inputs/datasource_release_status.xlsx

11910 12/15/2013 05:27 PM Aaron Marcuse-Kubitza

added inputs/datasource_release_status.xlsx, export of Google spreadsheet at https://docs.google.com/spreadsheet/ccc?key=0ArZXrTAXd-TYdDRRb2RxYi11TWZrQVh5bVdKOURCeFE

11905 12/11/2013 10:54 PM Aaron Marcuse-Kubitza

fix: inputs/CVS/^taxon_observation.**.sample/: added _no_import because this table duplicates part of what's imported from taxon_observation.**

11904 12/11/2013 10:42 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/plot/: added _no_import because this table is left-joined and should not be imported separately

11903 12/11/2013 10:40 PM Aaron Marcuse-Kubitza

bugfix: inputs/{.NCBI,CTFS}/*.src/: added _no_import because these tables are left-joined and should not be imported separately

11902 12/11/2013 09:56 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: removed table names from datasources where only one table is imported

11901 12/11/2013 09:52 PM Aaron Marcuse-Kubitza

fix: inputs/import.stats.xls: removed deleted tables from current import

11900 12/11/2013 09:51 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

11888 12/10/2013 06:35 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/map.csv: row_num: remapped to plain *row_num, like the other datasources that have this field

11887 12/10/2013 06:31 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: rerun time: noted that this is only fast after manual vacuuming of the table (to remove the deleted rows from the index). autovacuum apparently does not run, although it should.

11881 12/09/2013 07:24 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: reran test, which added yearCollected/monthCollected/dayCollected

11880 12/09/2013 07:23 PM Aaron Marcuse-Kubitza

inputs/CVS/plantConcept_/create.sql: documented runtime (3 min)

11879 12/09/2013 06:59 PM Aaron Marcuse-Kubitza

inputs/CTFS/*.src/: added test.xml.ref

11878 12/09/2013 06:58 PM Aaron Marcuse-Kubitza

inputs/CTFS/*.src/: added VegBIEN.csv

11877 12/09/2013 06:56 PM Aaron Marcuse-Kubitza

bugfix: inputs/CTFS/TaxonOccurrence*/map.csv: things mapped to taxonObservationID: remapped to taxonOccurrenceID since taxonObservationID is not mapped to anything in VegBIEN (denormalized VegCore doesn't distinguish between taxon occurrences and taxon observations of them)

11876 12/09/2013 05:46 PM Aaron Marcuse-Kubitza

bugfix: inputs/ARIZ/~.clean_up.sql: prevent "column already exists" errors when there is an input column of the same name as an output column

11873 12/09/2013 04:16 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/import_order.txt: added county_centroids so that it would be installed by new-style import

11871 12/09/2013 03:37 PM Aaron Marcuse-Kubitza

inputs/FIA/TREE/run: documented import() runtime (1.5 h), which includes table cleanup runtime (1 h)

11869 12/09/2013 02:43 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: updated import() runtime (same), documented table cleanup runtime (1.5 h)

11868 12/09/2013 02:38 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: CREATE INDEX ... specimenHolderInstitutions: documented runtime (45 min)

11867 12/09/2013 02:28 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: documented runtime (3.5 min)

11865 12/06/2013 07:46 AM Aaron Marcuse-Kubitza

bugfix: inputs/CTFS/import_order.txt: added *.src so that these would be installed under new-style import as well. this means that their columns will now be automapped, requiring the names to be renamed to VegCore names in */create.sql. note that VegCore taxonOccurrenceID has been renamed to taxonObservationID since this was last run.

11864 12/06/2013 06:56 AM Aaron Marcuse-Kubitza

inputs/.geoscrub/run: documented import() runtime (20 min)

11863 12/06/2013 06:12 AM Aaron Marcuse-Kubitza

bugfix: inputs/.NCBI/import_order.txt: added nodes.src, names.src so that these would be installed under new-style import as well. this means that their columns will now be automapped, requiring the names to be renamed to VegCore names in nodes/create.sql.

11849 12/06/2013 02:44 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: install: for new-style datasources, use the associated runscript instead (the old-style install target will not do everything that's needed for a new-style datasource)

11847 12/06/2013 12:51 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: install: for new-style datasources, use the associated runscript instead (the old-style install target will not do everything that's needed for a new-style datasource)

11843 12/05/2013 11:38 PM Aaron Marcuse-Kubitza

inputs/FIA/COND/postprocess.sql: filtering formula: documented that this was created by Brad, and provided the URL to it on nimoy

11842 12/05/2013 12:27 PM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.clean_up.sql: remove plot.realLatitude/realLongitude, since this is private data that should not be publicly visible

11841 12/05/2013 12:19 PM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.clean_up.sql: remove plot.realLatitude/realLongitude, since this is private data that should not be publicly visible

11820 12/04/2013 04:57 PM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/create.sql: uncommented identifiedBy since this is now part of taxonObservation_

11819 12/04/2013 04:08 PM Aaron Marcuse-Kubitza

fix: inputs/CVS/observation_community/create.sql: communityName: populate from commConcept.commName instead, because commInterpretation.commname is not always populated. this requires left-joining to commConcept.

11818 12/04/2013 03:58 PM Aaron Marcuse-Kubitza

inputs/CVS/observation_community/map.csv: updated output column names to new input column names, to avoid later output column collisions

11817 12/04/2013 03:42 PM Aaron Marcuse-Kubitza

inputs/CVS/observation_community/header.csv, map.csv: updated input column names for cvs.~.clean_up.sql renamings

11815 12/04/2013 04:19 AM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.clean_up.sql: commClass, commConcept fields: prepend table name to avoid inter-table collisions upon join

11814 12/04/2013 03:43 AM Aaron Marcuse-Kubitza

added inputs/CVS/observation_community/, as for VegBank

11813 12/04/2013 03:32 AM Aaron Marcuse-Kubitza

inputs/CVS/cvs.~.clean_up.sql: commClass.dba_src_ID: prepend table name to avoid inter-table collisions upon join

11812 12/03/2013 04:32 PM Aaron Marcuse-Kubitza

added inputs/CVS/observationContributor_/, which adds the people collecting the plot