Project

General

Profile

Statistics
| Revision:

# Date Author Comment
10668 08/10/2013 11:55 PM Aaron Marcuse-Kubitza

inputs/VegBIEN/db/.htaccess: allow access to schemas other than the public schema by requiring a schema before the table. currently, this does not break any existing URLs.

10667 08/09/2013 01:57 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: rebalanced between the months. Geoscrubbing re-run: extended to September since it's not clear when Jim will have time for this.

10666 08/09/2013 01:53 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: split Data provider validations (spot-checking) into separate tasks for VegBIEN and VegCore

10665 08/09/2013 01:49 PM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for July progress

10664 08/09/2013 01:25 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: geovalidation: added step at end to georeference all places to corrected_geoplace

10663 08/09/2013 12:51 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: moved all required fields before optional fields

10662 08/09/2013 12:39 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

10661 08/09/2013 12:23 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: individual: identifying_place: documented that for specimens, this is the place the specimen was collected at. within_place: documented that this is not specified for specimens since their coordinates are usually not precise enough to identify an individual. specimen: documented that when there are multiple specimen replicates (copies) of a specimen, each gets its own specimen_observation pointing to the same specimen.

10660 08/09/2013 12:12 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: individual: added identifying_place since some individuals may be identified by their coordinates rather than a tag

10659 08/09/2013 12:06 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added back aggregate_observation as a superclass of taxon_absence, taxon_presence. have only aggregate_observation, not taxon_observation, inherit from taxa_sampling_event because taxa_sampling_event is designed for multiple or aggregate samplings, but taxon_observation can be for just one occurrence. taxa_sampling_event: added required within_place since taxa must be sampled within a particular place. individual.place: renamed to within_place to clarify that each individual does not necessarily need its own place instance. individual_observation: added place_observed_at to serve the role that individual.place previously did (the place of an individual may change between observations).

10658 08/09/2013 10:41 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: renamed to individual_count for clarity. added count field. renamed traits to aggregating_traits since these are used in uniquely identifying the individual_count. taxon_presence: added cover_percent.

10657 08/09/2013 10:17 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: taxon_presence: documented that this is actually equivalent to VegX's aggregateOrganismObservation ("An observation applying to all occurrences of an organism"), in spite of VegCore having its own aggregate_observation table with a different purpose

10656 08/09/2013 10:12 AM Aaron Marcuse-Kubitza

schemas/VegX/.htaccess, TCS/.htaccess: prepend / to subpath so that it becomes an absolute XPath. the subpath is now usually provided as a dotpath translated to a dirpath, which would not have a leading / .

10655 08/09/2013 12:08 AM Aaron Marcuse-Kubitza

fix: schemas/VegCore/VegCore.ERD.mwb.run: comment about why the Mac version does not export the PDF correctly: clarified that it is the single-page PDF where the lines take very long to load, and the printed PDF where the colors are wrong in Adobe Reader

10654 08/09/2013 12:02 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb.run: documented that to edit VegCore.ERD.mwb, you must use the Linux version of MySQL Workbench. this is because the Mac version does not export the PDF correctly: the colors are wrong in Adobe Reader and the lines take very long to load. unfortunately, you also cannot switch back and forth between the two versions, because they render the lines with a different spacing, which makes the tables different sizes and misaligns the connector lines. it is especially unfortunate that the Mac version messes up the PDFs, because it uses a smaller line spacing and therefore smaller tables which allows the tables to be spaced farther apart for better readability.

10653 08/08/2013 11:49 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: aggregate_observation- and stratum-related tables: documented VegBank equivalents

10652 08/08/2013 10:20 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements

10651 08/08/2013 10:16 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements

10650 08/08/2013 10:14 PM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make(): use new create_db=1 option to mysqldump_local so that the dumpfile contains the necessary schema-qualifiers to be syncable with the ERD (i.e. to allow editing the ERD by changing the dumpfile)

10649 08/08/2013 07:15 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysqldump(): added create_db=1 flag to print the CREATE DATABASE statement

10648 08/08/2013 07:15 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: pf(): echo func decl to stderr instead of stdout

10647 08/08/2013 07:10 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_vars(): documented that it only prints vars that are defined

10646 08/08/2013 07:10 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: local_inv: also echo_vars the new var

10645 08/08/2013 06:17 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

10644 08/08/2013 06:16 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10643 08/08/2013 06:06 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10642 08/08/2013 06:04 PM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: updated name of sampling_event to taxa_sampling_event

10641 08/08/2013 06:03 PM Aaron Marcuse-Kubitza

schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved place_observation-related tables before sampling_event because plots-related tables should come before occurrence-related tables

10640 08/08/2013 06:01 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map

10639 08/08/2013 05:34 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: soil_observation: allow multiple soil_observations per place_observation (e.g. at different spots), as VegBank does

10638 08/08/2013 05:27 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: method: spaced more evenly from other tables

10637 08/08/2013 05:21 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: fixed lines

10636 08/08/2013 05:20 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: *.footprint_geom_WKT: renamed to boundary_WKT for clarity

10635 08/08/2013 05:14 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added reobservable_presence, which allows a vouchered taxon_presence (e.g. a set of aggregate_observations) to have taxon redeterminations

10634 08/08/2013 04:44 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: changed relationship to taxon_presence to many:1 instead of 1:1 inheritance, because there can be multiple aggregate_observations (e.g. size classes) for each aggregated taxon. added optional size_class field to indicate the aggregating group.

10633 08/08/2013 04:34 PM Aaron Marcuse-Kubitza

bugfix: schemas/VegCore/VegCore.ERD.mwb: method: made parent optional because top-level methods will not have a parent. rearranged to make more room for place_observation, soil_observation.

10632 08/08/2013 04:18 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: sampling_event: renamed to taxa_sampling_event for clarity. (fkey order has reversed.)

10631 08/08/2013 04:12 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: place_observation: don't inherit from sampling_event because sampling_event refers specifically to sampling taxa, and does not have anything to do with environmental measurements on the place itself. size_class: moved to occurrence half since it relates specifically to aggregate_observations, rather than plot strata.

10630 08/08/2013 03:14 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: stratum: added layer, size_class subclasses (with name now only required for layers)

10629 08/08/2013 02:51 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: stratum_event: added fkey to parent sampling_event. stratum: made space for subclasses.

10628 08/08/2013 02:22 PM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: added stratum_event. (the fkey order also got reversed.)

10627 08/08/2013 01:26 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/observation_/create.sql: ensure only one row per observation by selecting the first soilobs for each observation

10626 08/08/2013 12:09 PM Aaron Marcuse-Kubitza

bugfix: inputs/VegBank/stemlocation_/create.sql: include 1 row for each stemlocation rather than for each stemcount, because stemcounts are instead imported as part of stemcount_ (which also includes taxonimportance)

10625 08/07/2013 04:51 PM Aaron Marcuse-Kubitza

inputs/TEX/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource

10624 08/07/2013 04:44 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen2/: translated new multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10623 08/07/2013 04:41 PM Aaron Marcuse-Kubitza

bugfix: inputs/TEX/Specimen2/map.csv: DETERMINER FIRST NAME/LAST NAME: combined using _join_words() instead of the default _alt()

10622 08/07/2013 04:20 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen2/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10621 08/07/2013 04:13 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen2/map.csv: MISC NOTES: remapped to locationRemarks since these primarily relate to the coords. PLANT DATA: remapped to occurrenceRemarks since this is the specimenDescription.

10620 08/07/2013 03:15 PM Aaron Marcuse-Kubitza

inputs/TEX/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10619 08/07/2013 10:22 AM Aaron Marcuse-Kubitza

inputs/UNCC/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource

10618 08/07/2013 09:50 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns. reproductiveCondition: use the suggested values at DwC.vegpath.org?reproductiveCondition instead of just joining together the values in flower and fruit.

10617 08/07/2013 09:19 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: usdaRank, infrarank: documented that they should actually be mapped to infraspecificRank*. it is better to map them to taxonRank for now because infraspecificRank is now mapped to VegBIEN.

10616 08/07/2013 09:07 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).

10615 08/07/2013 09:04 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: infrarank: remapped to EQUIV (of usdaRank). usdaRank, infrarank: documented the VegCore term that each would map to.

10614 08/07/2013 08:49 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/postprocess.sql: infrarank: remove mismapped values "L." (belong in authors, and are already there)

10613 08/07/2013 08:30 AM Aaron Marcuse-Kubitza

schemas/util.sql: added ?*>= operator (is populated more often than) from wiki.vegpath.org/Postgres_queries#is-populated-more-often-than

10612 08/07/2013 08:12 AM Aaron Marcuse-Kubitza

bugfix: web/.htaccess: parse dotpath in query string: make dotpath replacement start with ./ instead of / (which would incorrectly create an absolute URL path). this bug only appears when there is a trailing /, because this causes the matched text to be empty instead of a filesystem path with leading /.

10611 08/07/2013 08:04 AM Aaron Marcuse-Kubitza

bugfix: web/.htaccess: parse dotpath in query string: redirect so REQUEST_URI is populated from REQUEST_FILENAME: need to use - instead of $0 as the redirect destination so that REQUEST_FILENAME is used instead of the empty REQUEST_URI (i.e. the redirect vars are left untouched)

10610 08/07/2013 07:45 AM Aaron Marcuse-Kubitza

bugfix: web/.htaccess: parse dotpath in query string: need to add trailing / if dotpath so that dest path is URL path, not filesystem path (when trailing / missing, matched path is filesystem path)

10609 08/07/2013 07:44 AM Aaron Marcuse-Kubitza

backups/TNRS.backup.md5: updated for last backup

10608 08/07/2013 06:33 AM Aaron Marcuse-Kubitza

schemas/util.sql: hstore(text[], anyelement): replaced with just hstore(text[], text) so that bare literals (of type unknown) are auto-cast to text instead of causing a "could not determine polymorphic type because input has type "unknown"" error

10607 08/07/2013 05:56 AM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: loanto, inorout: removed mapping to occurrenceRemarks because this information does not relate to the occurrence. it is also rarely populated.

10606 08/06/2013 05:07 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: added backup MD5 test time for last import

10605 08/06/2013 05:03 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: added backup upload time for last import

10604 08/06/2013 04:55 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10603 08/06/2013 04:53 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: min disk space: updated import schema size for last import

10602 08/06/2013 04:30 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: added backup times from last import

10601 08/06/2013 01:35 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

10600 08/06/2013 01:16 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run

10599 08/06/2013 12:32 AM Aaron Marcuse-Kubitza

bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the beginning of the line only

10598 08/06/2013 12:29 AM Aaron Marcuse-Kubitza

bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list

10597 08/04/2013 06:07 PM Aaron Marcuse-Kubitza

inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource

10596 08/04/2013 06:06 PM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)

10595 08/04/2013 05:58 PM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: trim(): need to remove all added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state

10594 08/04/2013 05:55 PM Aaron Marcuse-Kubitza

schemas/util.sql: added added_cols()

10593 08/04/2013 04:06 PM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name

10592 08/04/2013 01:29 AM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10591 08/04/2013 01:23 AM Aaron Marcuse-Kubitza

inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.

10590 08/04/2013 01:10 AM Aaron Marcuse-Kubitza

inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count

10589 08/04/2013 12:59 AM Aaron Marcuse-Kubitza

bin/after_import: usage: documented that it requires $version

10588 08/04/2013 12:58 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background

10587 08/04/2013 12:57 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import

10586 08/03/2013 09:14 PM Aaron Marcuse-Kubitza

bin/import_all: use new bin/after_import

10585 08/03/2013 09:13 PM Aaron Marcuse-Kubitza

added bin/after_import, which performs post-normalized-import actions separately from bin/import_all

10584 08/03/2013 03:34 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1

10583 08/03/2013 03:32 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: added steps to restart an aborted import for a specific table

10582 08/03/2013 03:28 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.

10581 08/03/2013 03:11 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table

10580 08/03/2013 12:25 AM Aaron Marcuse-Kubitza

bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately

10579 08/03/2013 12:24 AM Aaron Marcuse-Kubitza

bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.

10578 08/03/2013 12:03 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)

10577 08/03/2013 12:00 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all

10576 08/02/2013 11:55 PM Aaron Marcuse-Kubitza

bin/import_all: don't set $dump_opts until running the backup command that uses it, so that the user can run this backup command separately just by copying the line out of the script (without worrying about env vars that need to be set, other than $version which is visible outside the script)

10575 08/01/2013 05:02 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10574 08/01/2013 05:00 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10573 08/01/2013 04:56 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: Gazetteer/Newgazett, Majorarea: documented that these are the closest equivalent for Guyana/Suriname

10572 08/01/2013 04:55 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: Majorarea: mapped to stateProvince, which is the closest equivalent for Guyana/Suriname

10571 08/01/2013 04:54 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: Gazetteer/Newgazett: remapped to county, which is the closest equivalent for Guyana/Suriname

10570 08/01/2013 04:37 PM Aaron Marcuse-Kubitza

inputs/U/Specimen/map.csv: Gazetteer, Newgazett: combine them with _alt() instead of _join() because only one of Gazetteer, Newgazett is ever populated

10569 08/01/2013 04:04 PM Aaron Marcuse-Kubitza

inputs/NCU/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource