Project

General

Profile

Statistics
| Revision:
  • svn:ignore: .~*

# Date Author Comment
12785 03/20/2014 08:27 PM Aaron Marcuse-Kubitza

inputs/MO/Specimen/test.xml.ref: updated, which adds dateCollected mappings

12784 03/20/2014 08:20 PM Aaron Marcuse-Kubitza

inputs/WIN/Specimen/test.xml.ref: updated to map.csv, which has eventDate->dateCollected

12783 03/20/2014 08:13 PM Aaron Marcuse-Kubitza

inputs/VegBank/plantconcept_/create.sql: updated runtime (25 min, ~same)

12779 03/20/2014 07:58 PM Aaron Marcuse-Kubitza

*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`

12776 03/20/2014 07:47 PM Aaron Marcuse-Kubitza

inputs/VegBank/plot/postprocess.sql: remove institutions that we have direct data for: CVS: updated runtime (same)

12758 03/18/2014 05:47 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/plot/postprocess.sql: use CVS.plot_ instead because that has the renamed staging table columns, and is compatible with auto-renaming of the SQL script columns

12757 03/18/2014 05:41 PM Aaron Marcuse-Kubitza

inputs/CVS/plot_/postprocess.sql: add unique constraint on locationName (analogous to the unique constraint in plot), for use by inputs/VegBank/plot/postprocess.sql in removing inter-datasource duplication

12753 03/18/2014 05:10 PM Aaron Marcuse-Kubitza

inputs/VegBank/taxon_observation.**/test.xml.ref: updated inserted row count

12752 03/18/2014 05:34 AM Aaron Marcuse-Kubitza

inputs/run: postprocess(): documented runtime (30 min)

12751 03/18/2014 05:16 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/postprocess.sql: don't perform replacements using map.csv, because map.csv is not idempotent. this functionality was only there to facilitate switching to new-style import, which is now largely done. (the remaining datasources NVS, SALVIAS, TEAM contain only 1 postprocess.sql: inputs/SALVIAS/projects/postprocess.sql (`st inputs/{NVS,SALVIAS,TEAM}/*/postprocess.sql`).)

12747 03/18/2014 04:33 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: %/postprocess.sql: always run this, not just if the associated map spreadsheets change, to avoid needing to `touch` them to cause %/postprocess.sql to run

12745 03/18/2014 04:24 AM Aaron Marcuse-Kubitza

fix: inputs/*/*/postprocess.sql: un-doubled *

12744 03/18/2014 04:06 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/postprocess.sql: also need to apply renames from mappings/VegCore.thesaurus.csv, as these have been applied to map.csv

12714 03/14/2014 07:35 PM Aaron Marcuse-Kubitza

added inputs/run, which runs all the inputs' runscripts using the new auto-forwarding

12703 03/14/2014 05:25 PM Aaron Marcuse-Kubitza

removed unused inputs/table.run. inputs/*/table.run include lib/runscripts/table.run directly.

12679 03/13/2014 05:03 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/validations.sql: implemented _plots_19_count_of_censuses_per_plot_in_each_project

12638 03/11/2014 09:56 PM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/validations.sql: plots_07_list_of_plots_with_counts_of_individuals_per_species: renamed to _plots_07_list_of_plots*which_use*_... because this query is not intended to include the actual counts, just to say which plots have them (the correct "which use" wording is also used in queries #8, 9)

12635 03/07/2014 10:49 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql, inputs/SALVIAS/validations.sql: added _plots_06a_list_of_stems, for use in figuring out the diff in _plots_06_list_of_plots_with_stem_measurements

12605 03/06/2014 08:52 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/validations.sql: _plots_18_list_of_subplots_codes_for_each_plot_for_each_project: changed columns to match output query

12603 03/06/2014 08:29 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/validations.sql: _plots_15_pct_cover_of_each_verb_taxon_in_each_plot_in_each_pro: changed types to match output query

12602 03/06/2014 08:14 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/validations.sql: _plots_15_pct_cover_of_each_verb_taxon_in_each_plot_in_each_pro: changed summarizing column from mean_cover->totalpercentcover to match output query

12601 03/06/2014 08:12 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/validations.sql: _plots_10a_aggregate_observation_individual_counts: changed individual_id type to match output query

12596 03/06/2014 12:07 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql, inputs/SALVIAS/validations.sql: added _plots_10a_aggregate_observation_individual_counts, for use in debugging diffs in _plots_10_count_of_individuals_per_plot_in_each_proj

12538 02/27/2014 07:56 PM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/validations.sql: renamed SiteCode to plot_code to match output queries

12526 02/27/2014 06:58 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/validations.sql: use plot_code instead of plotcode for easier readability

12516 02/27/2014 01:27 PM Aaron Marcuse-Kubitza

bugfix: *.sql: public.source_by_shortname(): need to wrap it in a nested SELECT because Postgres incorrectly does not constant-fold (inline) it, leading to a slowdown when it is therefore run many times. this is done using the steps at wiki.vegpath.org/Postgres_queries#wrap-function-call-in-nested-SELECT .

12508 02/26/2014 11:58 PM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/validations.sql: plotMetadata.SiteCode: need to match types with the output query column

12417 02/24/2014 10:51 PM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/validations.sql: _plots_02_list_of_project_names: altered column aliases to match output query

12407 02/24/2014 08:58 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/validations.sql: added Brad's comments from validation/aggregating/plots/SALVIAS/bien3_validations_salvias_db_original.VegCore.sql

12406 02/24/2014 08:53 AM Aaron Marcuse-Kubitza

added inputs/SALVIAS/validations*.sql

12367 02/23/2014 12:13 PM Aaron Marcuse-Kubitza

fix: schemas/vegbien.sql: _traits_08_taxonname_trait_and_value_for_first_5000_records: renamed to _traits_08_taxonname_trait_and_value because this actually includes all the records, not just the first 5000. this uses the new public_validations.rename_query_view() to rename all associated tables and views, including handling truncated names.

12286 02/17/2014 01:58 PM Aaron Marcuse-Kubitza

bugfix: inputs/bien2_traits/validations.sql: _traits_01_count_records: changed column names to match public_validations._traits_01_count_records

12246 02/16/2014 04:22 PM Aaron Marcuse-Kubitza

bugfix: inputs/bien2_traits/validations.sql: use a wrapper function for util.ifnull() so that the views don't get dropped when the util schema is reinstalled

12224 02/14/2014 03:09 PM Aaron Marcuse-Kubitza

validation/aggregating/*/*.sql, schemas/vegbien.sql, lib/runscripts/validations.pg.sql.run, inputs/bien2_traits/validations.sql: added _ to beginning of each view name so the validation views would sort at the top in the datasource's tables list. this will also make the validation result sets easily distinguishable from the data tables.

12221 02/14/2014 12:20 PM Aaron Marcuse-Kubitza

added inputs/bien2_traits/validations.sql, from validation/aggregating/traits/BIEN2_traits/bien3_validations_traits_original_mysql.VegCore.sql

12220 02/14/2014 12:20 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(svnFilesGlob): added validations.sql

12213 02/14/2014 11:00 AM Aaron Marcuse-Kubitza

added inputs/bien2_traits/validations.sql.run

12158 02/13/2014 07:26 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

12157 02/13/2014 06:53 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

12112 02/07/2014 05:37 AM Aaron Marcuse-Kubitza

fix: inputs/VegBIEN/Redmine/wiki/.htaccess: redirect to new main page when accessed without trailing /

12060 02/06/2014 01:14 PM Aaron Marcuse-Kubitza

inputs/bien2_traits/TraitObservation/postprocess.sql: remove rows with no taxon name, which are invalid, and which helps simplify the aggregating validations queries

12053 02/06/2014 03:05 AM Aaron Marcuse-Kubitza

fix: inputs/VegBIEN/Redmine/svn/.htaccess: updated repository URL to point to trunk/

12040 02/04/2014 10:42 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/verify/plots.out.sql: fixed ' quoting syntax to use '' instead of \' to escape '

12039 02/04/2014 10:32 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: verify/%.out: use a *.sql file in the verify/ directory itself to generate *.out, so that each datasource can have its own set of output queries. for datasources that should share the same set of queries, they can instead be symlinked to the same file.

12038 02/04/2014 10:01 AM Aaron Marcuse-Kubitza

fix: inputs/CVS/project/: added _no_import since this should not also be imported separately from taxon_observation.**

12030 02/02/2014 11:18 PM Aaron Marcuse-Kubitza

added inputs/XAL/Specimen/_no_import, since this is a demo-only datasource and there isn't a staging table for it

12029 02/02/2014 11:10 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/county_centroids/test.xml.ref, inputs/.NCBI/{names.src,nodes.src}/test.xml.ref: accepted test outputs (generated now that these tables are in import_order.txt)

12028 02/02/2014 10:31 PM Aaron Marcuse-Kubitza

inputs/FIA/taxon_observation.**/header.csv: updated for new REF_RESEARCH_STATION.country metadata value col

12018 02/02/2014 12:49 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: add!: verify/: also svn:ignore *.tsv, *.txt

12005 01/23/2014 01:44 AM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

12004 01/23/2014 01:43 AM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

12003 01/23/2014 01:35 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: remove projects that do not have "There are no specific use conditions attached to this dataset"

12002 01/23/2014 01:22 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/salvias_plots.~.clean_up.sql: Remove private data that should not be publicly visible: also need to remove metadata-only plots

12001 01/23/2014 01:11 AM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/plotMetadata_/map.csv: things mapped to project_participant: remapped to event__participant because these actually relate to the event, not the project, even though they seem like project-related fields

11999 01/23/2014 12:54 AM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/plotMetadata_/map.csv, inputs/Madidi/LocationObservation/map.csv: things mapped to communityID: remapped to communityName, which is what's used in analytical_stem (communityID is for numeric IDs)

11997 01/22/2014 11:01 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/create.sql, map.csv: expanded plot_administrator:party_code_party_ and mapped plot_administrator_name to a 2nd project_participant

11996 01/22/2014 10:59 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: project_participant: use [!...] negative lookahead assertion so that multiple project_participant columns will properly map to separate projectcontributor rows

11994 01/22/2014 09:16 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/map.csv: mapped PrimOwnerID_name->project_participant

11993 01/22/2014 09:07 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/create.sql: added join to PrimOwnerID:party_code_party_

11992 01/22/2014 01:06 PM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/import_order.txt: added party_code_party_

11991 01/22/2014 12:50 PM Aaron Marcuse-Kubitza

bugfix: inputs/SALVIAS/party_code_party_/create.sql: need to remove duplicate entries in party_code_party

11988 01/22/2014 11:10 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/party_code_party_/map.csv: mapped fullname->event_participant_name for use by other tables

11987 01/22/2014 10:34 AM Aaron Marcuse-Kubitza

mapped inputs/SALVIAS/party_code_party_/

11983 01/20/2014 10:12 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/_MySQL/salvias_plots.*.sql: refreshed. this adds the party and party_code_party tables Brad provided for mapping the plot contributors.

11982 01/20/2014 10:10 PM Aaron Marcuse-Kubitza

fix: inputs/SALVIAS/salvias_plots.~.clean_up.sql: Delete rows that do not satisfy foreign key constraints: also need to do this for plotObservations, since the refreshed data contains dangling rows for that as well

11981 01/20/2014 10:08 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/run_: documented *.sql install runtime (3 min), as separate from the full `datasrc_make reinstall` runtime (3.5 min)

11980 01/20/2014 10:07 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/run_: refresh(): `datasrc_make reinstall`: updated runtime. documented that runtimes are from starscream.

11979 01/20/2014 08:09 PM Aaron Marcuse-Kubitza

added inputs/SALVIAS/run_, which includes a refresh() target

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11965 01/16/2014 01:22 AM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: scrubbed_family: Name_matched_accepted_family was missing from the TNRS results at one point, so we are now using Family_matched as a workaround to populate this. the workaround is for accepted names only, as no opinion names do not have an Accepted_name_family to prepend to the scrubbed name to parse.

11964 01/16/2014 01:19 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: reexported from live DB, which changes the element order

11961 01/15/2014 10:18 AM Aaron Marcuse-Kubitza

inputs/VegBank/import_order.txt: added projectcontributor_

11960 01/15/2014 10:11 AM Aaron Marcuse-Kubitza

inputs/VegBank/projectcontributor_/map.csv, postprocess.sql: added project_participant

11957 01/15/2014 09:41 AM Aaron Marcuse-Kubitza

added inputs/VegBank/projectcontributor_/

11956 01/15/2014 09:29 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql: projectcontributor.surname: prepend table name to avoid join collisions

11955 01/15/2014 09:23 AM Aaron Marcuse-Kubitza

inputs/VegBank/vegbank.~.clean_up.sql, inputs/CVS/cvs.~.clean_up.sql: Prevent "column name specified more than once" errors when tables are joined: put tables in alphabetical order for consistency

11943 01/14/2014 08:34 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

11942 01/14/2014 08:31 PM Aaron Marcuse-Kubitza

inputs/datasource_release_status.xlsx: renamed to `publishable datasources.xlsx` to match the spreadsheet title

11934 12/20/2013 04:41 PM Aaron Marcuse-Kubitza

inputs/VegBank/^taxon_observation.**.sample/create.sql, map.csv: added new project columns

11933 12/20/2013 04:31 PM Aaron Marcuse-Kubitza

inputs/VegBank/taxon_observation.**/postprocess.sql: added the project table

11932 12/20/2013 04:25 PM Aaron Marcuse-Kubitza

mapped inputs/VegBank/project/, which includes the projectName for attribution

11931 12/20/2013 02:56 PM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/create.sql, map.csv: added new project columns

11930 12/20/2013 02:44 PM Aaron Marcuse-Kubitza

inputs/CVS/taxon_observation.**/postprocess.sql: added the project table

11929 12/20/2013 02:42 PM Aaron Marcuse-Kubitza

inputs/CVS/project/map.csv: mapped stopDate->projectEndDate

11928 12/20/2013 02:35 PM Aaron Marcuse-Kubitza

mapped inputs/CVS/project/, which includes the projectName for attribution

11927 12/20/2013 01:25 AM Aaron Marcuse-Kubitza

inputs/VegBIEN/Redmine/svn/.htaccess: updated to use much faster direct repository URL rather than Redmine web interface, now that the repository itself is publicly accessible in addition to the Redmine view of it

11924 12/20/2013 12:28 AM Aaron Marcuse-Kubitza

fix: inputs/TEX/Specimen*/map.csv, postprocess.sql: habitat: also placed in occurrenceRemarks so that this field gets parsed for growth form information, as requested by Brad (wiki.vegpath.org/TEX_validation#2013-2-26)

11923 12/19/2013 11:49 PM Aaron Marcuse-Kubitza

fix: inputs/TEX/Specimen*/map.csv: mapped constant values for specimenHolderInstitutions, country. these have to be added with `rm=1 ./inputs/TEX/Specimen.../run postprocess`.

11922 12/19/2013 11:42 PM Aaron Marcuse-Kubitza

bugfix: inputs/TEX/Specimen2/map.csv: mapped BARCODE to accessionNumber so that we have a unique ID for each row

11920 12/17/2013 08:06 AM Aaron Marcuse-Kubitza

inputs/datasource_release_status.xlsx: updated

11917 12/16/2013 07:05 PM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/create.sql: added Mike Lee's additional plots used to validate confidentiality-related fields (wiki.vegpath.org/CVS_validation#plots-to-include)

11916 12/16/2013 06:00 PM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/^taxon_observation.**.sample/create.sql: include taxonName in the subset of columns that's imported for the validation, because it is _alt-ed with scientificName for forming the TNRS input name. this is unique to CVS, which is why it was not part of the validation subset copied from the VegBank subset.

11912 12/16/2013 01:43 PM Aaron Marcuse-Kubitza

bugfix: inputs/.TNRS/schema.sql: granted bien_read SELECT access to derived views as well as the core tnrs table

11911 12/15/2013 05:30 PM Aaron Marcuse-Kubitza

updated inputs/datasource_release_status.xlsx

11910 12/15/2013 05:27 PM Aaron Marcuse-Kubitza

added inputs/datasource_release_status.xlsx, export of Google spreadsheet at https://docs.google.com/spreadsheet/ccc?key=0ArZXrTAXd-TYdDRRb2RxYi11TWZrQVh5bVdKOURCeFE

11905 12/11/2013 10:54 PM Aaron Marcuse-Kubitza

fix: inputs/CVS/^taxon_observation.**.sample/: added _no_import because this table duplicates part of what's imported from taxon_observation.**

11904 12/11/2013 10:42 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/plot/: added _no_import because this table is left-joined and should not be imported separately

11903 12/11/2013 10:40 PM Aaron Marcuse-Kubitza

bugfix: inputs/{.NCBI,CTFS}/*.src/: added _no_import because these tables are left-joined and should not be imported separately

11902 12/11/2013 09:56 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: removed table names from datasources where only one table is imported

11901 12/11/2013 09:52 PM Aaron Marcuse-Kubitza

fix: inputs/import.stats.xls: removed deleted tables from current import