Activity
From 07/08/2013 to 08/06/2013
08/06/2013
- 05:07 PM Revision 10606: inputs/import.stats.xls: added backup MD5 test time for last import
- 05:03 PM Revision 10605: inputs/import.stats.xls: added backup upload time for last import
- 04:55 PM Revision 10604: inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:53 PM Revision 10603: /README.TXT: Full database import: min disk space: updated import schema size for last import
- 04:30 PM Revision 10602: inputs/import.stats.xls: added backup times from last import
- 01:35 AM Revision 10601: inputs/import.stats.xls: Updated import times
- 01:16 AM Revision 10600: /README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run
- 12:32 AM Revision 10599: bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the *beginning* of the line only
- 12:29 AM Revision 10598: bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list
08/04/2013
- 06:07 PM Revision 10597: inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 06:06 PM Revision 10596: bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)
- 05:58 PM Revision 10595: bugfix: schemas/util.sql: trim(): need to remove *all* added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state
- 05:55 PM Revision 10594: schemas/util.sql: added added_cols()
- 04:06 PM Revision 10593: bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name
- 01:29 AM Revision 10592: inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:23 AM Revision 10591: inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.
- 01:10 AM Revision 10590: inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count
- 12:59 AM Revision 10589: bin/after_import: usage: documented that it requires $version
- 12:58 AM Revision 10588: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background
- 12:57 AM Revision 10587: /README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import
08/03/2013
- 09:14 PM Revision 10586: bin/import_all: use new bin/after_import
- 09:13 PM Revision 10585: added bin/after_import, which performs post-normalized-import actions separately from bin/import_all
- 03:34 PM Revision 10584: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1
- 03:32 PM Revision 10583: /README.TXT: Full database import: added steps to restart an aborted import for a specific table
- 03:28 PM Revision 10582: inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.
- 03:11 PM Revision 10581: inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table
- 12:25 AM Revision 10580: bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately
- 12:24 AM Revision 10579: bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.
- 12:03 AM Revision 10578: bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)
- 12:00 AM Revision 10577: /README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all
08/02/2013
08/01/2013
- 05:02 PM Revision 10575: inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 05:00 PM Revision 10574: inputs/U/Specimen/: translated some multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:56 PM Revision 10573: inputs/U/Specimen/map.csv: Gazetteer/Newgazett, Majorarea: documented that these are the closest equivalent for Guyana/Suriname
- 04:55 PM Revision 10572: inputs/U/Specimen/map.csv: Majorarea: mapped to stateProvince, which is the closest equivalent for Guyana/Suriname
- 04:54 PM Revision 10571: inputs/U/Specimen/map.csv: Gazetteer/Newgazett: remapped to county, which is the closest equivalent for Guyana/Suriname
- 04:37 PM Revision 10570: inputs/U/Specimen/map.csv: Gazetteer, Newgazett: combine them with _alt() instead of _join() because only one of Gazetteer, Newgazett is ever populated
- 04:04 PM Revision 10569: inputs/NCU/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 04:02 PM Revision 10568: bugfix: placed inputs/NCU/Specimen/postprocess.sql under version control
- 03:57 PM Revision 10567: bugfix: inputs/NCU/Specimen/map.csv: CatalogSeriesPrefix: enclosed comment in "
- 03:43 PM Revision 10566: inputs/NCU/Specimen/map.csv: OwnerInstitution: remapped to specimenOwner rather than specimenHolderInstitutions. OwnerInstitution, CatalogSeriesPrefix: documented the VegCore SQL dotpath (http://wiki.vegpath.org/SQL_dotpaths) that would be used to refer to the field. this specifies the destination field at a much finer level of detail than the one-size-fits-all denormalized name.
- 02:56 PM Revision 10565: mappings/VegCore-VegBIEN.csv: mapped municipality
- 02:46 PM Revision 10564: inputs/NCU/Specimen/map.csv: CityLocality: remapped to municipality because this is a placename, not a verbatim locality description
- 02:25 PM Revision 10563: inputs/NCU/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 02:07 PM Revision 10562: inputs/NY/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 02:02 PM Revision 10561: inputs/$dest/$subdir/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:28 PM Revision 10560: inputs/NY/Ecatalog_all/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:11 PM Revision 10559: inputs/bien_web/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 11:37 AM Revision 10558: inputs/UBC/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 11:07 AM Revision 10557: /README.TXT: Full database import: don't exit the screen until after getting $version, which is defined within it
- 09:49 AM Revision 10556: planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Attribution and conditions of use up because it's high priority. deferred Importing to normalized VegCore until after October due to decision to use VegBIEN for the October DB.
- 03:51 AM Revision 10555: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:46 AM Revision 10554: schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: refactored to separate geoscrubbing (matching a named place) from prepping for geovalidation (uniquifying the lat/long and parent place). note that only bare lat/longs without official placenames will be geovalidated.
- 02:36 AM Revision 10553: schemas/VegCore/VegCore.ERD.mwb: geovalidation: inherit from georeferencing, since applying a corrected (or confirmed) lat/long is a form of georeferencing
- 02:29 AM Revision 10552: bugfix: schemas/VegCore/VegCore.ERD.mwb: scrubbed_geoplace: made parent_geoplace nullable to allow a scrubbed root node which has no parent. added scrubbed_name to emphasize that any place.name applied in this table is scrubbed.
- 12:47 AM Revision 10551: inputs/CTFS/StemObservation/unmapped_terms.csv: regenerated
- 12:43 AM Revision 10550: inputs/UNCC/Specimen/new_terms.csv: regenerated
- 12:22 AM Revision 10549: /README.TXT: Full database import: make test by_col=1: documented that if you encounter errors, they are most likely related to the PostgreSQL error parsing in /lib/sql.py parse_exception()
07/31/2013
- 11:58 PM Revision 10548: bugfix: lib/sql.py: parse_exception(): MissingCastException from DoesNotExistException for function: handle overloaded functions where none of the overloads supports the given arg types (so assume text). this may have become a bug from system upgrades?
- 11:35 PM Revision 10547: added inputs/newWorld/iso_code_gadm/VegBIEN.csv, etc., generated by running ./run
- 11:34 PM Revision 10546: added inputs/newWorld/newWorldCountries/map.csv, etc., generated by running ./run
- 10:54 PM Revision 10545: inputs/NY/Ecatalog_all/map.csv: PlantFungDescription: documented that PlantFung confusingly refers to the plant/fungus the specimen came from, rather than to a fungus growing on the plant
- 10:29 PM Revision 10544: schemas/VegCore/VegCore.ERD.mwb: fixed lines, including changing inheritance connectors to 1:1/optional on child table
- 09:21 PM Revision 10543: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that bin/redmine_synonyms can be used as a template
- 09:14 PM Revision 10542: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: reset(): noted that the dimensions precede the matched name
- 09:13 PM Revision 10541: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: added reset() stub, with instructions of how to autogenerate this using dimensions in ../document.mwb.xml
- 08:55 PM Revision 10540: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 08:28 PM Revision 10539: bugfix: /Makefile: postgres-Linux: phppgadmin.conf: updated `ln -s` to /etc/apache2/conf-available/ for current name of /etc/apache2/conf.d/phppgadmin.conf, which is now just phppgadmin
- 07:52 PM Revision 10538: added planning/workflow/derived_columns/range_measurements/low-high_vs_midpoint-uncertainty.VegBank.png diagram, generated from a screenshot of the VegBank Data Dictionary (VegBank.vegpath.org/plot#Elevation)
- 05:21 PM Revision 10537: schemas/VegCore/VegCore.ERD.mwb: party category: made it pink to match VegBank (http://vegpath.org/exchange_schemas/VegBank/vegbank.ERD.pdf), swapping colors with source. (note that source isn't red like in VegBank because black text doesn't show up well against it.)
- 05:06 PM Revision 10536: bugfix: schemas/VegCore/VegCore.ERD.pdf: regenerated from schemas/VegCore/VegCore.ERD.mwb, without any tables moused-over. apparently, mousing over a table while saving the *PDF* also saves the highlighting on the fields, not just when saving the PNG.
- 04:56 PM Revision 10535: inputs/HVAA/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 04:52 PM Revision 10534: inputs/HVAA/Specimen/map.csv: fieldNumber: remapped to UNUSED
- 04:32 PM Revision 10533: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:31 PM Revision 10532: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:25 PM Revision 10531: inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 04:04 PM Revision 10530: inputs/SpeciesLink/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 03:59 PM Revision 10529: inputs/SpeciesLink/Specimen/map.csv: renamed DUPLICATE#of:... output columns to be <= 63 chars long, in order to be valid PostgreSQL columns without collisions
- 03:37 PM Revision 10528: inputs/SpeciesLink/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 03:12 AM Revision 10527: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 03:10 AM Revision 10526: schemas/VegCore/VegCore.ERD.mwb: rel_place: renamed to subplace to clarify that this is any place contained within another place, not just places *with a position* relative to their parent place
- 03:05 AM Revision 10525: schemas/VegCore/VegCore.ERD.mwb: individual.place: don't require this to be a rel_place, because some individuals will be associated with standalone specimens, which don't have a parent place
- 02:55 AM Revision 10524: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 02:49 AM Revision 10523: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
- 02:45 AM Revision 10522: schemas/VegCore/VegCore.ERD.mwb: require rel_place to have a parent place. merge plot_element into subplot because subplot is the only place the parent_plot constraint is imposed (individuals can now be located in any place, not just a plot).
- 02:38 AM Revision 10521: schemas/VegCore/VegCore.ERD.mwb: individual: require it to have a place (which may be a rel_place within another place). do not require the place to be a plot_element, because individuals can be located in places other than uniformly-shaped plots.
- 02:23 AM Revision 10520: schemas/VegCore/VegCore.ERD.mwb: regenerated exports. this now includes updating the image map for moved and new tables.
- 02:11 AM Revision 10519: schemas/VegCore/VegCore.ERD.mwb: added plot_element to require that the associated parent place of a rel_place be a *plot*. added subplot to require that a subplot be a plot_element (i.e. have a parent_plot) and to show where to put subplot data.
- 01:28 AM Revision 10518: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: ERD.pdf: fixed URL since the map is now in a subdir
- 01:22 AM Revision 10517: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: added URLs, which now autoredirect to the appropriate DB table on vegbiendev
- 01:21 AM Revision 10516: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: cleanup(): trim growing whitespace: remove multiple spaces at a time, in case the user saved and reopened VegCore.ERD.png.map.htm but didn't run this script in between
- 01:01 AM Revision 10515: added schemas/VegCore/ERD/.htaccess to forward unknown subdirs to vegbiendev MySQL as tables
- 01:00 AM Revision 10514: added web/servers/vegbiendev/db/my/, which forwards to MySQL instead of PostgreSQL
- 01:00 AM Revision 10513: web/servers/vegbiendev/db/: moved PostgreSQL engine to separate pg/ subdir to allow for other engines
- 12:23 AM Revision 10512: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run: #$AUTHOR: trim growing whitespace which Gimp repeatedly adds on each save
- 12:21 AM Revision 10511: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL
- 12:19 AM Revision 10510: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: VegCore: fixed URL
- 12:10 AM Revision 10509: schemas/VegCore/ERD/VegCore.ERD.png.map.htm, index.htm: re-ran VegCore.ERD.png.map.htm.run (it needs to be run every time VegCore.ERD.png.map.htm is edited)
- 12:02 AM Revision 10508: added schemas/VegCore/ERD/VegCore.ERD.png.map.htm.run, which cleans up and formats Gimp's image map for publishing; along with derived file index.htm
07/30/2013
- 11:59 PM Revision 10507: added schemas/VegCore/ERD/VegCore.ERD.png.map.htm image map for VegCore.ERD.png. note that the tables are sorted, and this sort order supersedes the data dictionary sort order (which is somewhat similar). the table URLs have not been added yet.
- 10:07 PM Revision 10506: lib/sh/util.sh: $sed_cmd: added usage
- 09:46 PM Revision 10505: schemas/VegCore/VegCore.ERD.mwb: GNRS & geovalidation steps: 2. GNRS: split into substeps 2a. unique lat/long, 2b. names, 2c. geovalidatable place for clarity. don't refer to the scrubbed_geoplace as the GADM shape, because only the parent_geoplace has the shape (the scrubbed_geoplace just has the scrubbed names).
- 09:10 PM Revision 10504: added schemas/VegCore/VegCore.ERD.letter_size.pdf. this must be generated in Linux rather than Mac, because the Mac PDF printer messes up the colors in the PDF (missing color profile?).
- 08:34 PM Revision 10503: schemas/VegCore/VegCore.ERD.mwb: added labels for GNRS and geovalidation steps, analogous to the TNRS taxonomic scrubbing steps labels
- 07:25 PM Revision 10502: schemas/VegCore/VegCore.ERD.mwb: regenerated exports. VegCore.ERD.png now gets the sRGB color profile attached in Gimp so that the colors don't look washed out on some LCD screens.
- 07:13 PM Revision 10501: schemas/VegCore/VegCore.ERD.mwb: georeferencing: added hstore extender
- 07:08 PM Revision 10500: schemas/VegCore/VegCore.ERD.mwb: individual: made it a plot element by optionally attaching a plot position (a rel_place whose parent is the containing plot)
- 06:54 PM Revision 10499: schemas/VegCore/VegCore.ERD.mwb: plot: replaced bounding_box_rect with length_m, width_m, since the bounding box was intended to store plot dimensions (along the plot azimuth) rather than an actual bounding box aligned to the compass directions. added azimuth_deg_N, which is used to resolve plot element x/y coordinates to absolute geocoordinates while taking into account the rotation of the plot.
- 06:42 PM Revision 10498: schemas/VegCore/VegCore.ERD.mwb: tables with parent hierarchies: made parent optional, since the root(s) of the hierarchy will not have an entry for this, and any unique constraints that include this column should be ignored (which they will be if the value is NULL instead of a self-pointer)
- 06:37 PM Revision 10497: schemas/VegCore/VegCore.ERD.mwb: subplot: renamed to rel_place and inherit from place directly, in order to store other plot elements that are relative to their containing plot
- 06:26 PM Revision 10496: schemas/VegCore/VegCore.ERD.mwb: taxon_path: converted to an auxiliary table of taxon_name instead of a subclass of it (like geopath for the place table). this causes distinct taxon_paths to be stored only once, instead of repeatedly for each taxon_name.
- 06:16 PM Revision 10495: schemas/VegCore/VegCore.ERD.mwb: place hierarchy: reorganized to store scrubbed geoplaces in a containment hierarchy instead of a denormalized geopath. this allows each source-specific place to be GNRS-scrubbed to a GADM place, and then have its coordinates geovalidated to see if it is within the matched GADM place. this uses the georeferencing table to store the matched GADM *place* (scrubbed_geoplace) for each input *place*, instead of geopath_scrub to store the matched GADM geo*path* for each input geo*path*. (this avoids the need to scrub every combination of place ranks, because just the name of each place is scrubbed relative to its parent place.) geopath instead becomes an auxiliary table to store the place table's verbatim ranks, for easy access and storage.
- 04:51 PM Revision 10494: inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_BoundingBox: remapped to UNUSED
- 03:22 PM Revision 10493: schemas/VegCore/VegCore.ERD.mwb: place: renamed to local_place to distinguish it from geoplace, which is *not* a subclass of place (it is a separate, global table, while local_place is source-specific). note that renames sometimes need to be done manually on vegbiendev, to avoid triggering a MySQL bug that blocks the new table from being created and requires the entire database to be recreated to clear the error.
- 03:02 PM Revision 10492: schemas/VegCore/VegCore.ERD.mwb: stem, stem_observation: made associated individual/individual_observation optional, because some stems (e.g. in VegBank) are not grouped together into individuals. note that a stem is still considered to BE-AN individual, but it is a type of individual which may be grouped under another, plant-level individual.
- 02:47 PM Revision 10491: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 02:45 PM Revision 10490: schemas/VegCore/VegCore.ERD.mwb: specimen_observation: added description (vegcore.vegpath.org?specimenDescription). taxon_presence: added occurrence_status (vegcore.vegpath.org?occurrenceStatus). stem_observation, aggregate_observation: made room for them to expand with additional first-class fields.
- 02:22 PM Revision 10489: schemas/VegCore/VegCore.ERD.mwb: taxon_presence, taxon_absence: inherit from taxon_determination rather than taxon_observation, so that the taxon_determination's taxon can be used as the identifying taxon (i.e. the authorPlantName, VegCore.vegpath.org?authorPlantName)
- 01:41 PM Revision 10488: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: inherit from taxon_observation again because now that redeterminations can only occur on reobservable things, it makes sense to only allow one taxon_determination per observation event. this means that each redetermination on a specimen would get its own taxon_observation (where any additional attributes noted in the reobservation could also be included).
- 01:31 PM Revision 10487: schemas/VegCore/VegCore.ERD.mwb: taxon_occurrence: renamed to reobservable to emphasize that this is only for things on which taxon redeterminations can be made, such as individuals and specimens (including voucher specimens). a redetermination on an aggregate_observation would instead be made on its voucher specimen, which is the only reobservable part of it.
- 01:07 PM Revision 10486: schemas/VegCore/VegCore.ERD.mwb: moved taxon_observation subclasses closer to taxon_observation so that it would be clear they were observation-related rather than occurrence-related (e.g. there is no concept of "repeat-sampling" of an aggregate_observation, because each sampling it is the collector's opinion that the plants correspond to a particular taxon)
- 02:00 AM Revision 10485: bugfix: schemas/VegCore/VegCore.ERD.png: switched back to attaching the sRGB color profile directly, because actually, the native->sRGB translation happens in the monitor driver itself (and can be adjusted in System Preferences > Displays > Color), rather than in the specific application. this means that the hex color values color-matched in MySQL Workbench *were* actually sRGB (translated *by the OS* to monitor-native for display), and that the sRGB profile merely needed to be *explicitly indicated* for other monitors that are *not* close to sRGB (and thus need the translation). the closeness of the 27-inch iMac screen to sRGB can be verified by selecting sRGB in System Preferences > Displays > Color, and noting that the desktop background does not change from when the default "iMac" setting is selected.
- 01:38 AM Revision 10484: bugfix: schemas/VegCore/VegCore.ERD.png: convert to sRGB color profile after attaching the *native* monitor profile instead of attaching it directly. this allows the hex colors that were color-matched in MySQL Workbench (which presumably uses raw monitor RGB) to be translated to the universal sRGB space, where they can then be localized to a different monitor's local color space. note that this does not visibly change the image on the 27-inch iMac screen from what was produced via the previous, incorrect method (attaching the sRGB profile without conversion from native), which would imply that the iMac's screen is very close to the sRGB color space already. if this is the case, it is instead older LCDs that have off-white color spaces that need translation from sRGB.
07/29/2013
07/27/2013
- 12:13 PM Revision 10482: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 12:09 PM Revision 10481: schemas/VegCore/VegCore.ERD.mwb: added separate geo category (turquoise) to visually distinguish the broader geoplace tables from the more specific plot tables. (note that georeferencing is actually a plot table despite geo- in its name, because it assigns a geoplace *to a plot*.)
- 11:51 AM Revision 10480: schemas/VegCore/VegCore.ERD.mwb: georeferencing: added georeferenced_by
- 11:46 AM Revision 10479: schemas/VegCore/VegCore.ERD.mwb: added georeferencing table for georeference* DwC fields. this can be used to link a place to a georeferenced geoplace other than (or in addition to) the original geoplace.
- 11:14 AM Revision 10478: schemas/VegCore/VegCore.ERD.mwb: added geopath_scrub for GNRS results (separate from point-in-polygon validation)
- 10:51 AM Revision 10477: schemas/VegCore/VegCore.ERD.mwb: place: factored optional geocoords, geopath out into separate geoplace table (with both nullable), which validatable_geoplace (renamed from geoplace, with both NOT NULL) extends
- 10:25 AM Revision 10476: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: made scrubbed_geoplace optional because not all geoplaces will scrub to a valid geoplace
- 10:22 AM Revision 10475: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: need to inherit from record now that this is source-specific
- 10:17 AM Revision 10474: bugfix: schemas/VegCore/VegCore.ERD.mwb: geovalidation: HAVE-AN input geoplace rather than BEING-ONE, to allow multiple geovalidations for a geoplace by different sources
- 10:05 AM Revision 10473: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: changed IS-A relationship with taxon_observation to HAS-A so that a separate taxon_observation doesn't need to be created for each taxon_determination (even though each taxon_determination event is theoretically a reobservation of the specimen, etc.). instead, inherit from sampling_event to include the necessary event-related fields.
- 09:45 AM Revision 10472: bugfix: schemas/VegCore/VegCore.ERD.mwb: geopath: made country NOT NULL so that every geoplace (for input to geovalidation) has something on the geopath side. geocoords: made latitude_deg/longitude_deg NOT NULL so that every geoplace (for input to geovalidation) has something on the geocoords side. added geocoords_unique constraint since this is a global table with one entry for each lat/long.
- 09:30 AM Revision 10471: schemas/VegCore/VegCore.ERD.mwb: place: added coords hstore extender, for verbatim coordinates, etc.
- 09:14 AM Revision 10470: schemas/VegCore/VegCore.ERD.mwb: coordinates: abbreviated to coords (unambiguous abbreviation)
- 08:59 AM Revision 10469: schemas/VegCore/VegCore.ERD.mwb: replaced parsed_taxon_assertion with taxon_scrub, which HAS-A parsed taxon_assertion rather than BEING-A parsed_taxon_assertion. (multiple TNRS results may parse to the same thing.)
- 08:08 AM Revision 10468: schemas/VegCore/VegCore.ERD.mwb: geovalidatable_place: renamed to geoplace, since this uniquification is useful independently of geovalidation. note that the MySQL upgrade on vegbiendev has now reordered the fkeys again, this time in forwards order.
- 08:02 AM Revision 10467: planning/timeline/timeline.2013.xls: updated for July progress
- 07:43 AM Revision 10466: schemas/VegCore/VegCore.ERD.mwb: place tables that are absolute within Earth rather than relative to a parent place: prefixed geo- to table name for clarity
- 07:23 AM Revision 10465: schemas/VegCore/VegCore.ERD.mwb: plot, subplot: added hstore extenders (dimensions, coordinates)
- 07:17 AM Revision 10464: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance connectors to be 1:1, optional on subclass
- 07:11 AM Revision 10463: schemas/VegCore/VegCore.ERD.mwb: plot: added shape. bounding_box: changed units to rect, since this just needs a width/height (the x/y coord is the lat/long).
- 07:05 AM Revision 10462: schemas/VegCore/VegCore.ERD.mwb: plot: added footprint_geom_WKT. bounding_box: added units (WKT).
- 06:51 AM Revision 10461: schemas/VegCore/VegCore.ERD.mwb: back-synced from staging copy on vegbiendev to flush out sync changes that it kept trying to re-make
- 06:47 AM Revision 10460: schemas/VegCore/VegCore.ERD.mwb: event: moved method to separate sampling_event subclass
- 06:28 AM Revision 10459: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 06:25 AM Revision 10458: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: inherit from taxon_presence, since this is a type of taxon_presence and it avoids duplicating the taxon_concept field
- 06:11 AM Revision 10457: schemas/VegCore/VegCore.ERD.mwb: added taxon_absence, to avoid including absence observations in the same table as presence observations (which needlessly complicates queries). note that the fkey order now gets set back to forwards whenever a table is changed.
- 06:07 AM Revision 10456: schemas/VegCore/VegCore.ERD.mwb: re-saved. the fkey order is now apparently reversed for recently-changed tables.
07/26/2013
- 11:07 PM Revision 10455: schemas/VegCore/VegCore.ERD.mwb: collector, identified_by: allow multiple parties for these fields, using the new party_list array table
- 10:44 PM Revision 10454: schemas/VegCore/VegCore.ERD.mwb: party arrays: use new party_list array table instead of adding a separate many:many table for each table that uses a party array. this also allows using the party_list ID in a unique constraint, because it is now a first-class field.
- 10:06 PM Revision 10453: schemas/VegCore/VegCore.ERD.mwb: party: added party_list array table
- 09:45 PM Revision 10452: schemas/VegCore/VegCore.ERD.mwb: party: added optional fkey to organization
- 09:32 PM Revision 10451: schemas/VegCore/VegCore.ERD.mwb: geovalidation: renamed lat_long_in_ranks to lat_long_in_place_ranks for clarity
- 09:12 PM Revision 10450: schemas/VegCore/VegCore.ERD.mwb: individual: added tag_history hstore to store custom identity attributes
- 08:39 PM Revision 10449: schemas/VegCore/VegCore.ERD.mwb: taxon_string: documented that to get the parsed_taxon_assertion (TNRS result) for a taxon_string, you would join using the SQL dotpath taxon_string.string<-taxon_assertion(string)::parsed_taxon_assertion[source='TNRS.version'] (see wiki.vegpath.org/SQL_dotpaths). important how-to comments such as this one are now included in the version-controlled MySQL schema file itself, not just the .mwb file and the staging copy on vegbiendev.
- 08:16 PM Revision 10448: bin/my2pg: use s!...!...! when either the regexp or the replacement contains / , to avoid unnecessary \-s
- 08:09 PM Revision 10447: bin/my2pg: commenting out table options: added explanatory comment, because it is not obvious from the regexp what this does
- 08:06 PM Revision 10446: lib/sh/db.sh: mysqldump(): don't use --compatible=postgresql when the table structure is being exported, because this removes the table options (which include the COMMENT attribute). --compatible=postgresql remains on in data-only mode because embedded ` in data cannot easily be distinguished from ` around column names, so ANSI_QUOTES is needed to do the translation to " (and data sections do not contain table options). note that all --compatible modes that offer ANSI_QUOTES unfortunately exclude the table options, and there is no way to run a SQL query to set the SQL mode before beginning the dump, so ANSI_QUOTES translation must be handled by my2pg instead.
- 06:35 PM Revision 10445: bin/my2pg: comment out table options (http://dev.mysql.com/doc/refman/5.5/en/server-sql-mode.html#sqlmode_no_table_options) instead of removing them, because they include table COMMENTs, which contain important metadata such as table definitions. (note that table COMMENTs use a slightly different syntax than column COMMENTs, so the table COMMENTs will not be commented out twice.)
- 06:19 PM Revision 10444: bin/my2pg: comment out COMMENTs instead of removing them so that they will be included in the PostgreSQL translation. COMMENTs contain important metadata about columns, such as definitions and the meanings of integer flag values.
- 05:58 PM Revision 10443: inputs/{.,}*/*.schema.sql: regenerated using the instructions in bin/my2pg. this primarily replaces timestamp with text/*timestamp*/ (to preserve indefinite dates).
- 05:56 PM Revision 10442: bin/my2pg: added instructions for regenerating *.schema.sql whenever this script is changed
- 05:22 PM Revision 10441: bin/my2pg: COMMENT: also match COMMENTs with embedded ', because there will only be one COMMENT per line, so the contents of the COMMENT can just extend to the last ' on the line
- 05:16 PM Revision 10440: bugfix: lib/sh/util.sh: $sed_cmd: make output unbuffered, so that running e.g. bin/my2pg at the command line produces output as each line is read
- 04:29 PM Revision 10439: bin/my2pg: replace MySQL ` quotes with " quotes to support exports that were generated without ANSI_QUOTES mode. (this replacement only applies to schema exports, not data.) ANSI_QUOTES is only available with mysqldump --compatible modes that also include NO_TABLE_OPTIONS, which omits important table options such as comments. in particular, these comments are part of schemas/VegCore/VegCore.ERD.mwb but were not being included in VegCore.my.sql.
- 01:41 PM Revision 10438: schemas/VegCore/VegCore.ERD.mwb: taxon_string: removed parsed_taxon_assertion field, since there may be more than one parsing (TNRS result) for a given taxon_string. the parsing relationship can better be represented by adding a parsed_taxon_assertion whose taxon_assertion.string points to the parsed taxon_string. getting the parsed_taxon_assertion for a taxon_string now requires joining on parsed_taxon_assertion using a backwards instead of forwards fkey, and filtering the corresponding assertions to include only the ones for TNRS (of the desired TNRS version). documented that taxon_assertion.string was previously the concatenated matched name, but is now the TNRS input name. the concatenated matched name is still in parsed_taxon_assertion.matched_taxon_concept->:taxon_name.unique_name.
- 01:22 PM Revision 10437: schemas/VegCore/VegCore.my.sql: regenerated from .mwb schema, which apparently reverses the order of the fkeys (possibly a Linux MySQL bug?)
- 12:26 PM Revision 10436: inputs/SpeciesLink/Specimen/map.csv: remapped Darwin Core synonyms to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic _alts to perform to resolve filter-less collisions. the svn diff was verified by replacing DUPLICATE#of:dwc_terms_<term>#... with <term>, removing the comment, and checking that this removes the diff (except where VegCore has renamed a DwC term).
- 12:17 PM Revision 10435: bugfix: inputs/SpeciesLink/Specimen/map.csv: *scientificName: remapped to scientificName instead of taxonName to match the DwC term's name (this is the same dwc_terms_scientificName mismapping that was fixed in r10434)
- 11:56 AM Revision 10434: bugfix: inputs/SpeciesLink/Specimen/map.csv: dwc_terms_scientificName: remapped to scientificName instead of taxonName to match that DwC term name, as well as the mappings of other *scientificName terms
- 11:06 AM Revision 10433: inputs/SpeciesLink/Specimen/map.csv: marked dwc_geospatial_VerbatimLatitude,Longitude as exact duplicates of dwc_terms_*
- 10:52 AM Revision 10432: inputs/SpeciesLink/Specimen/map.csv: remapped identical _alt-ed fields to DUPLICATE. this avoids the need to translate these to postprocessing derived columns for new-style import, and also speeds up column-based import because there are less automatic _alts to perform to resolve filter-less collisions.
- 10:06 AM Revision 10431: bugfix: inputs/SpeciesLink/Specimen/map.csv: *CollectorNumber: moved these to the same _alt group as recordNumber, because they are actually duplicates
- 09:43 AM Revision 10430: correction: inputs/SpeciesLink/Specimen/map.csv: *FieldNumber: fixed incorrect comment that these fields are identical to recordNumber, when instead they have the same *meaning* but not the same values. instead, values are stored under *either* of the two terms. the previous conclusion had been based on an incorrect query, which used != instead of the NULL-sensitive IS NOT DISTINCT FROM.
07/25/2013
- 08:14 PM Revision 10429: planning/timeline/timeline.2013.xls: Adding derived columns: extended to overlap with all subtasks
- 08:12 PM Revision 10428: planning/timeline/timeline.2013.xls: Geoscrubbing: split into separate re-run and automated pipeline tasks
- 08:09 PM Revision 10427: planning/timeline/timeline.2013.xls: moved Data provider validations before Adding derived columns because ensuring that the source data is in the database is more important than the derived data, which can always be added later
- 08:00 PM Revision 10426: planning/timeline/timeline.2013.xls: Data provider validations: added dot in July because some amount of datasource-level validation happens when mappings issues are discovered during the refactoring
- 07:34 PM Revision 10425: bugfix: inputs/*/*/map.csv for specimen tables: remapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself rather than to any parent event (specimens don't have a parent event)
- 07:34 PM Revision 10424: inputs/*/*/map.csv for IndividualObservation tables: also mapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself in addition to any parent event which it may be a part of
- 06:27 PM Revision 10423: bugfix: inputs/XAL/Specimen/, NY/Ecatalog_all/: *JulianDay: remapped to dayOfYear instead of day (the day of the month)
- 05:08 PM Revision 10422: inputs/SpeciesLink/Specimen/map.csv: remapped *dayOfYear-related terms to UNUSED
- 04:53 PM Revision 10421: bugfix: inputs/SpeciesLink/Specimen/map.csv: remapped conceptual_darwin_2003_1_0_JulianDay, dwc_dwcore_DayOfYear to dayOfYear instead of day (the day of the month)
- 04:43 PM Revision 10420: mappings/VegCore.htm: regenerated from wiki. added dayOfYear (=julianDay), which is different from startDayOfYear/endDayOfYear.
- 01:59 PM Revision 10419: inputs/CTFS/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 01:50 PM Revision 10418: inputs/CTFS/StemObservation/: translated collisions (missing filters) to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 10:57 AM Revision 10417: planning/timeline/timeline.2013.xls: rebalanced tasks across the remaining months, taking into account priority changes made in the conference call (e.g. that we should not be handling people's individual data requests (Brad, wiki.vegpath.org/2013-07-25_conference_call#Decisions-made))
- 10:50 AM Revision 10416: planning/timeline/timeline.2013.xls: updated with additional tasks added in conference call: translate source-specific derived columns to plain SQL, flatten the datasources, automated geoscrubbing pipeline
- 08:43 AM Revision 10415: planning/goals/BIEN_3_derived_data_products_NormalizedDB_only.docx: removed BIEN species-level phylogeny, which Brad says is out of scope for the BIEN DB
- 08:24 AM Revision 10414: removed planning/workflow/bien3_architecture.odp because the current version is now in bien3_architecture.pptx
- 08:13 AM Revision 10413: added planning/workflow/validation/TNRS_results.ppt symlink to inputs/test_taxonomic_names/_scrub/TNRS_results.ppt
- 08:10 AM Revision 10412: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: highlighted the sample row and related rows
- 08:04 AM Revision 10411: inputs/test_taxonomic_names/_scrub/TNRS_results.xls: moved arrows to TNRS_results.ppt so they can be changed more easily
- 07:51 AM Revision 10410: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: TNRS.tnrs: added diagram labels for the various names and steps
- 07:32 AM Revision 10409: inputs/test_taxonomic_names/_scrub/TNRS_results.xls: use "Poa annua var. eriolepis"->"Poaceae Poa annua L." as the synonym example instead of "Poa annua fo. lanuginosa"->"Poaceae Poa annua var. annua" because the input name is simpler and it's closer to the beginning of the list
- 07:20 AM Revision 10408: inputs/test_taxonomic_names/_scrub/run: exports/make(): tnrs.csv: include Name_matched instead of Genus_matched+Specific_epithet_matched because this also contains lower ranks, which are used in the TNRS synonymizing
- 07:06 AM Revision 10407: inputs/test_taxonomic_names/_scrub/TNRS_results.ppt: added annotations explaining the import steps
- 06:36 AM Revision 10406: added inputs/test_taxonomic_names/_scrub/TNRS_results.ppt, containing the *.png screenshots with tables labeled
- 06:35 AM Revision 10405: added inputs/test_taxonomic_names/_scrub/*.png, screenshots of the TNRS_results.xls tabs (LibreOffice does not preserve the formatting when pasting a spreadsheet to a PowerPoint as a table, and the table editing options are limited)
- 06:31 AM Revision 10404: added inputs/test_taxonomic_names/_scrub/TNRS_results.xls with formatted versions of the *.csv tables
07/24/2013
- 05:15 PM Revision 10403: inputs/test_taxonomic_names/_scrub/run: exports/make(): subset the columns to include only the most important to demo how the data is represented
- 05:13 PM Revision 10402: lib/sh/db.sh: mk_select(): support passing $cols as array instead of SQL string, which is easier to enter in a shell script (less quotes, \ , etc.)
- 05:12 PM Revision 10401: lib/sh/db.sh: added cols2list()
- 05:10 PM Revision 10400: lib/sh/util.sh: added is_array()
- 04:38 PM Revision 10399: inputs/test_taxonomic_names/_scrub/run: exports/make(): allow specifying an explicit columns list for each table using cols=... (initially set to all columns)
- 04:09 PM Revision 10398: added inputs/test_taxonomic_names/_scrub/*.csv exports
- 04:09 PM Revision 10397: added inputs/test_taxonomic_names/_scrub/run, which exports the test_scrub-populated tables to CSV
- 04:08 PM Revision 10396: lib/sh/db_make.sh: added pg_export_table_to_dir(), pg_export_tables_to_dir(). unlike db.sh pg_export_table_to_dir_no_header(), these functions are make-aware and will not clobber an existing file.
- 03:15 PM Revision 10395: reran inputs/test_taxonomic_names/test_scrub, which generates the public.test_taxonomic_names sample schema
- 01:50 PM Revision 10394: inputs/CTFS/Plot/map.csv: DescriptionOfSite: remapped to locationRemarks, not locality
- 01:38 PM Revision 10393: inputs/CTFS/AggregateObservation/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 01:24 PM Revision 10392: schemas/vegbien.sql: geoscrub_input_new: updated for VegCore-renamed geoscrub_output column names
- 01:09 PM Revision 10391: schemas/util.sql: added ?>= operator with is_more_complete_than() function
- 12:44 PM Revision 10390: inputs/.geoscrub/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 12:15 PM Revision 10389: inputs/.geoscrub/geoscrub_output/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 11:18 AM Revision 10388: schemas/util.sql: SQL-language IMMUTABLE functions marked STRICT: removed STRICT to enable dynamic inlining, which speeds up the function up to 7x. STRICT was not removed where the function was particularly complex and the STRICT optimization would likely be more significant than inlining.
- 11:07 AM Revision 10387: bugfix: inputs/BRIT/specimen_flat/postprocess.sql: diameterBreastHeight_cm, height_m: use newly NULL-mapped versions of columns instead of the *_verbatim columns
- 11:04 AM Revision 10386: inputs/BRIT/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource
- 10:49 AM Revision 10385: inputs/BRIT/specimen_flat/: translated multi-column filters with _join() to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 10:43 AM Revision 10384: inputs/BRIT/specimen_flat/map.csv: Habitat_Summary: remapped to UNUSED
- 10:16 AM Revision 10383: inputs/BRIT/specimen_flat/postprocess.sql: diameterBreastHeight_cm, height_m: updated runtimes
- 10:15 AM Revision 10382: inputs/BRIT/specimen_flat/: DBH_*, Height_*: mapped NULL-equivalent values, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 09:27 AM Revision 10381: inputs/.../: translated multi-column filters with _avg() to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
- 08:18 AM Revision 10380: inputs/BRIT/specimen_flat/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "translate single-column filters to postprocessing derived columns"
07/20/2013
- 05:25 AM Revision 10379: /README.TXT: Maintenance: added instructions for what to do if http://vegbiendev.nceas.ucsb.edu/phppgadmin/ goes down (sometimes displaying a Not found error)
- 05:21 AM Revision 10378: schemas/util.sql: schema comment: added note that IMMUTABLE SQL-language functions should never be declared STRICT, because this prevents them from being inlined. inlining can create a significant speed improvement (7x+), by avoiding function calls and enabling additional constant folding.
- 05:09 AM Revision 10377: inputs/REMIB/Specimen/postprocess.sql: map_nulls() derived cols: documented total runtime (7.5 min on vegbiendev)
- 05:07 AM Revision 10376: inputs/REMIB/Specimen/postprocess.sql: map_nulls() derived cols: updated runtimes for map_nulls() inlining, which created a speed improvement of *7x* for the numeric columns and *2.5x* for the text columns (292563.362->41929.772 ms and 83640.424->35690.797 ms, respectively). note that the map_nulls__coord__*() calls could be optimized further by combining the successive map_nulls() calls into one, with the hstores merged.
- 04:37 AM Revision 10375: schemas/util.sql: map_nulls(): documented that inputs/REMIB/Specimen/postprocess.sql > country also shows that inlining is now happening properly. note that the speed improvement due to inlining is not as much, %-wise, when the values util._map() is run on are long strings instead of the short strings used in the initial profiling. this is because a greater % of the time is spent in system functions such as hstore->text, which are not affected by the inlining because they are run either way.
- 04:18 AM Revision 10374: schemas/util.sql: map_nulls(): use new nulls_map(). proper inlining (i.e. same runtime before and after change) has been verified with the following profiling query:
- SELECT util.map_nulls(array[1, 2, 3]::text[], v) FROM unnest(array_fill(1, array[100000])) f (v)
- 04:05 AM Revision 10373: schemas/util.sql: added nulls_map(), for use with _map()
- 03:39 AM Revision 10372: lib/runscripts/table.run: postprocess(): added remake action that calls trim_table()
- 03:37 AM Revision 10371: lib/runscripts/table.run: added trim_table(), which calls util.trim(regclass, regclass)
- 03:23 AM Revision 10370: lib/runscripts/table.run: map_table(): added remake action that calls reset_col_names()
- 03:21 AM Revision 10369: lib/runscripts/table.run: added reset_col_names(), which calls util.reset_col_names()
- 03:19 AM Revision 10368: bugfix: lib/runscripts/table.run: map_table(): moved $map_table to global var so it can be used by other functions
- 03:09 AM Revision 10367: bugfix: lib/runscripts/table.run: postprocess(): don't propagate $remake to remake_VegBIEN_mappings(), since this will cause map.csv to be remade, which is not related to the postprocessing.
- 03:08 AM Revision 10366: lib/runscripts/table.run: map_table(): util.set_col_names_with_metadata(): removed unnecessary cast to regclass, which is performed implicitly. this used to be needed when the polymorphic util.rename_cols() was used instead.
- 02:57 AM Revision 10365: schemas/util.sql: added trim(), which trims a table to include only original columns, as defined by a map table
- 02:53 AM Revision 10364: schemas/util.sql: added derived_cols(), which gets table_'s derived columns (all the columns not in the names table)
- 02:29 AM Revision 10363: schemas/util.sql: added eval2set()
- 02:14 AM Revision 10362: schemas/util.sql: added drop_column()
- 01:27 AM Revision 10361: inputs/REMIB/Specimen/postprocess.sql: map_nulls__*(): turned off STRICT to allow dynamic inlining, which speeds up the mk_derived_col() statements by *5x* (342799.823 ms -> 71533.252 ms (6 min -> 1 min) for latitude_sec)
07/19/2013
- 07:23 PM Revision 10360: inputs/REMIB/Specimen/postprocess.sql: runtimes: updated for vegbiendev, *before* dynamic inlining. the times are about twice as fast as on starscream, so vegbiendev is faster at whatever is the limiting speed factor (probably not CPU, based on other benchmarks).
- 07:05 PM Revision 10359: schemas/util.sql: map_nulls(): documented that due to dynamic inlining, this is just as fast as util._map() which it wraps. dynamic inlining now brings altogether a *40x* speed improvement to map_nulls() (4000 ms -> 100 ms), and would likely bring a comparable improvement for other functions that are run repeatedly and call other user-defined functions.
- 06:35 PM Revision 10358: bugfix: schemas/util.sql: map_nulls(): updated to use hstore(text[], anyelement), which has replaced hstore(anyarray, anyelement)
- 06:30 PM Revision 10357: schemas/util.sql: removed hstore(anyarray, anyelement), which did not support dynamic inlining, to avoid confusion over which hstore() function to use. use new hstore(text[], anyelement) instead (with explicit cast on the keys array if needed).
- 06:23 PM Revision 10356: schemas/util.sql: added hstore(text[], anyelement), which dynamically inlines properly, unlike hstore(anyarray, anyelement). this can be selected by explicitly casting the keys array to text[], which now provides a 6x speed improvement (380 ms -> 60 ms) for map_nulls().
- 05:31 PM Revision 10355: schemas/util.sql: fix_array(): turned off STRICT to allow dynamic inlining, which speeds up util.map_nulls() by 3x (1500 ms -> 500 ms)
- 05:15 PM Revision 10354: schemas/util.sql: array_length(anyarray), array_length(anyarray, dimension integer): turned off STRICT to allow dynamic inlining, which speeds up util.map_nulls(). this requires adding a `CASE WHEN $1 IS NULL THEN NULL` statement to array_length(anyarray, dimension integer) to replace the functionality provided by STRICT.
- 04:41 PM Revision 10353: schemas/util.sql: map_nulls(): turned off STRICT to allow dynamic inlining, which causes a 2x speed improvement[1]. (see r10352 for an explanation of dynamic inlining.) note that turning off STRICT disables NULL-skipping (avoiding running a function when all its params are NULL), so it should only be used when the NULL-skipping optimization is needed less than dynamic inlining.
- [1] the profiling query
SELECT util.map_nulls(array[v, 2, 3], v) FROM unnest(array_fill(1, array[10000])) f (v)
has a... - 04:23 PM Revision 10352: schemas/util.sql: inlinable IMMUTABLE functions: avoid using config params (e.g. `SET search_path TO util`) because these prevent dynamic inlining (i.e. inlining of a function call with *variable* instead of constant arguments, by substituting the arguments into the function's body). dynamic inlining can speed up function evaluation significantly, because a (slow) call to a user-defined SQL function is avoided.
- 04:15 PM Revision 10351: schemas/vegbien.my.sql: updated for new bin/repl text mode matching, which also affects non-regexps. this causes the replacement of a few more occurrences of PostgreSQL-only one-word typenames with their MySQL equivalents.
- 02:26 PM Revision 10350: inputs/REMIB/Specimen/postprocess.sql: runtimes: documented the machine the times are from
- 01:52 PM Revision 10349: inputs/REMIB/: switched to new-style import, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "run the following for each datasource"
- 11:40 AM Revision 10348: bugfix: bin/repl: text mode: repurpose this to match SQL identifiers, for use by inputs/input.Makefile %/postprocess.sql. %/postprocess.sql is the only place currently using this mode, so this will not affect other scripts.
- 10:51 AM Revision 10347: bugfix: inputs/input.Makefile: %/postprocess.sql: need to run bin/repl in text mode (text=1) so that values to match are treated as literal strings rather than regular expressions. this difference is important for column names with spaces or special characters.
- 10:24 AM Revision 10346: bugfix: inputs/Madidi/LocationObservation/map.csv: resolved Notes, Notes 2 -> locationRemarks collision by _alt()ing them together. note that _alt() is fine because only one of these is ever populated.
- 09:54 AM Revision 10345: bugfix: schemas/util.sql: set_col_names(): need to generate error if destination column already exists (rather than suppressing it with try_create()), because this indicates a collision
- 09:30 AM Revision 10344: bugfix: inputs/Madidi/IndividualObservation/map.csv: removed derived column FieldFamilyFullName#originalFamily, which should not be in the map table because it can contain only columns that are initially in the table before running postprocess.sql
- 09:23 AM Revision 10343: schemas/util.sql: map table: added unique constraint on the to column as well, because the destination names also need to be distinct in order to be a valid set of column names
- 09:14 AM Revision 10342: schemas/util.sql: map table: changed pkey to a unique constraint so pgAdmin would sort the entries in table order (matching the order they are in the staging table) instead of alphabetized by the pkey
- 08:56 AM Revision 10341: bugfix: inputs/REMIB/Specimen/map.csv: state: changed output column name to stateProvince_verbatim to match the renaming in postprocess.sql
- 08:40 AM Revision 10340: inputs/REMIB/Specimen/postprocess.sql: remove frameshifted rows: removed out-of-date rerun time, which applied to doing all the deletes in the same statement (however, the current rerun time is approximately the same). note that index scans are not actually used (as the previous comment incorrectly stated) because the conditions for this filter are prefix-less regexps.
- 08:32 AM Revision 10339: inputs/REMIB/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "translate single-column filters to postprocessing derived columns". null-mapping filters now use wrappers around new util.map_nulls(). note that the verbatim columns input to the filters need to be renamed to avoid name collisions with their filtered columns, which must be VegCore terms for new-style import.
- 07:53 AM Revision 10338: inputs/REMIB/Specimen/postprocess.sql: remove frameshifted rows: also filter out non-numbers for long_sec, lat_min, lat_sec
- 07:18 AM Revision 10337: inputs/REMIB/Specimen/postprocess.sql: remove frameshifted rows: remove rows where long_min is not a number
- 07:15 AM Revision 10336: inputs/REMIB/Specimen/postprocess.sql: change E'' to regular '' to avoid the need to double \ (instead ' would be doubled). E'' used to be necessary in previous versions of PostgreSQL to avoid a warning about escape string syntax.
- 07:09 AM Revision 10335: inputs/REMIB/Specimen/postprocess.sql: remove frameshifted rows: removed unnecessary () around `DELETE FROM :table WHERE long_deg ...`
- 07:03 AM Revision 10334: inputs/REMIB/Specimen/postprocess.sql: removed coll_year, country, long_deg indexes because the frameshift filter conditions on these columns do not use index scans (because their regexp patterns do not contain a fixed prefix). eventually, some regexp patterns may be able to be modified to use prefixes.
- 07:01 AM Revision 10333: bugfix: inputs/REMIB/Specimen/postprocess.sql: remove frameshifted rows: can't OR together conditions to determine rows to delete, because if any condition is NULL instead of true/false, this will NULL out the entire WHERE condition and prevent any other true conditions from causing a deletion. the best way to fix this is to use a separate DELETE statement for each condition, so that NULLs only impact that particular condition's DELETE. unlike using a modified, NULL-insensitive OR, which would prevent the use of index scans, this allows indexes to be used for conditions that support them.
- 06:05 AM Revision 10332: inputs/REMIB/Specimen/postprocess.sql: removed duplicate CREATE INDEX for the acronym column
- 05:59 AM Revision 10331: bugfix: inputs/REMIB/Specimen/postprocess.sql: switched back to the input column names, since the renaming to *_verbatim is part of a later step
- 05:26 AM Revision 10330: inputs/REMIB/Specimen/create.sql: moved filtering out of frameshifted rows to postprocess.sql, where it can happen in an idempotent DELETE. this allows filters to remove additional rows to easily be added on top of the existing filters, without needing to remake Specimen (which takes a long time, because of the many stage I derived columns that get added). the logical inversion inherent in the DELETE condition has been factored through rather than wrapped in NOT (...), because removal of frameshifted rows is more accurately specified as the detection of specific patterns that indicate frameshifting rather than the validation of all fields.
- 03:13 AM Revision 10329: bugfix: schemas/util.sql: not_empty(anyarray): array_length() now refers to different functions, with different semantics, depending on whether util is in the search_path. this necessitates explicitly selecting util.array_length() and switching to its semantics (ARRAY[] -> 0 instead of NULL)
- 03:02 AM Revision 10328: schemas/util.sql: map_nulls(): support all datatypes, not just text
- 02:55 AM Revision 10327: schemas/util.sql: added hstore(keys anyarray, value anyelement) and => (anyarray, anyelement) operator to support other element types for hstore
07/18/2013
- 06:43 PM Revision 10326: inputs/REMIB/Specimen/create.sql: also remove frameshifted rows with invalid long_deg values
- 04:31 PM Revision 10325: schemas/util.sql: added map_nulls(), a common use case of _map()
- 04:29 PM Revision 10324: bugfix: schemas/util.sql: hstore(keys text[], value text): use new fix_array() so that an empty keys array is made 1-dimensional to match up with the array generated by array_fill()
- 04:26 PM Revision 10323: schemas/util.sql: added fix_array(), which ensures that the array will always have proper non-NULL dimensions
- 04:21 PM Revision 10322: schemas/util.sql: added empty_array(), for constructing proper empty 1-dimensional arrays whose dimensions are not NULL ( {}::text[] does not do this)
- 03:36 PM Revision 10321: bugfix: schemas/util.sql: array_length(anyarray): need to call util.array_length() instead of just array_length() (which uses pg_catalog.array_length()) so that empty arrays will be returned as 0 instead of NULL. note that for some reason, adding `SET search_path=util` to the function does not have the same effect.
- 02:48 PM Revision 10320: inputs/ACAD/Specimen/postprocess.sql, inputs/ARIZ/omoccurrences/postprocess.sql: removed unnecessary "" around keys/values. "" are required in hstore input syntax in approximately the same places as they are in XPaths (around values containing spaces or special characters).
- 01:46 PM Revision 10319: inputs/ACAD/Specimen/map.csv, inputs/ARIZ/omoccurrences/map.csv: removed derived columns, which cause an error when trying to rename a table that does not yet have the derived columns added. this error will not be noticed locally if the derived columns were added before switching to new-style import, but will be noticed on vegbiendev.
- 01:17 PM Revision 10318: inputs/ARIZ/: switched to new-style import, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "run the following for each datasource"
- 01:03 PM Revision 10317: added inputs/ARIZ/omoccurrences/postprocess.sql
- 12:32 PM Revision 10316: inputs/ARIZ/omoccurrences/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "translate single-column filters to postprocessing derived columns"
- 12:25 PM Revision 10315: bugfix: schemas/util.sql: _map(map hstore, value anyelement): need to cast result to unknown to support types that don't have a cast directly from text
- 12:06 PM Revision 10314: schemas/util.sql: added _map(map hstore, value anyelement) to seamlessly map types other than text (by casting back and forth between text and the value type)
- 11:44 AM Revision 10313: inputs/ACAD/: switched to new-style import, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "run the following for each datasource"
- 11:38 AM Revision 10312: inputs/input.Makefile: added %/postprocess.sql to replace input column names with the corresponding output column names when switching to new-style import (this target must be manually run, but does simplify the process of renaming the postprocess.sql input columns)
- 11:02 AM Revision 10311: planning/timeline/timeline.2013.xls: moved Individual datasource refresh under Importing to normalized VegCore instead of Switching to new-style import because it is actually related to the refactor-in-place method used to import to VegCore
- 10:38 AM Revision 10310: inputs/ACAD/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Switching_to_new-style_import#stage-I-source-specific > "translate single-column filters to postprocessing derived columns"
- 09:47 AM Revision 10309: bugfix: schemas/util.sql: rename_cols(): run additional `SELECT NULL::void` query after the main for-loop query so that PostgreSQL does not try to fold away the execution of util.try_create() just because multiple rows are not returned by the function. the result set of the first query will still be discarded, but will be fully evaluated. (this has nothing to do with VOLATILE vs. IMMUTABLE; util.try_create() is already declared VOLATILE and would normally not be folded.) rename_cols() is used to rename derived columns, which are not part of the map.csv and cannot be positionally renamed.
- 08:38 AM Revision 10308: schemas/util.sql: added text[] => text operator, analogous to text => text for multiple keys (uses hstore(keys text[], value text))
- 08:29 AM Revision 10307: schemas/util.sql: added hstore(keys text[], value text), which can be used to avoid repeating the same value for each key. there are many /_map filters which use the XPath syntax for doing this, which now need to use an equivalent SQL syntax to avoid duplicating the value many times.
- 08:23 AM Revision 10306: web/links/index.htm: updated to Firefox bookmarks. added link to Brian Enquist's fractals video on PBS NOVA.
- 07:35 AM Revision 10305: schemas/util.sql: added array_fill(anyelement, integer), which doesn't require lengths for multiple dimensions
- 07:31 AM Revision 10304: schemas/util.sql: added array_length(anyarray, dimension integer) wrapper, which returns 0 instead of NULL for empty arrays
- 07:23 AM Revision 10303: schemas/util.sql: added array_length(anyarray), which does not require a second dimension argument
- 12:04 AM Revision 10302: lib/sql_io.py: put_table(): documented that PostgreSQL 9.1+ now provides a way to implement insert/on duplicate select just once for each table (instead of dynamically for each insert) using the new INSTEAD OF triggers (http://www.postgresql.org/docs/9.1/static/plpgsql-trigger.html). INSTEAD OF triggers were not used when put_table() was developed, because it was necessary to support PostgreSQL 9.0, which was installed on the Mac and not easily upgradeable. it was eventually upgraded to add PostGIS, which required a complete reinstall of the DB from the staging tables, with the associated staging table reload bugs, as well as complete removal of the old Postgres version.
07/17/2013
- 11:40 AM Revision 10301: inputs/Madidi/: switched to new-style import
- 10:17 AM Revision 10300: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 10:16 AM Revision 10299: schemas/VegCore/VegCore.ERD.mwb: categories: fixed position of place to match where it now is in the diagram. lined up boxes so that there is a visible line between the place- and occurrence-related categories.
- 09:27 AM Revision 10298: inputs/Madidi/IndividualObservation/map.csv: translated 1:many mappings ( FieldFamilyFullName->{family,originalFamily} ) to derived columns (in postprocess.sql) to work with new-style import, which must have a 1:1 relationship between input and output columns
- 09:05 AM Revision 10297: schemas/util.sql: added reset_col_names(), the counterpart to set_col_names(). note that this alters the map table, so it will need to be repopulated after running this function.
- 09:01 AM Revision 10296: schemas/util.sql: mk_derived_col(): support using this function to overwrite an existing column (i.e. as a general-purpose function to perform in-place update with ALTER COLUMN TYPE USING)
- 07:40 AM Revision 10295: lib/sh/db.sh: psql(): display stack traces and DETAIL sections of error messages at verbosity 2+, to help debugging (previously they were always turned off). in particular, the DETAIL section of a "duplicate key value violates unique constraint" error is useful because it contains the duplicated key.
- 04:54 AM Revision 10294: inputs/.TNRS/: switched to new-style import. because this does not have data subdirs (data comes from the TNRS client), this is just a matter of adding ./run.
- 04:53 AM Revision 10293: inputs/.TNRS/Source/: switched to new-style import. this had been missed when all the Source/ subdirs were batch-switched to new-style import.
- 04:24 AM Revision 10292: inputs/*/*/map.csv: replaced /_first filter with mapping to DUPLICATE special term (VegCore.vegpath.org?DUPLICATE). this removes collisions that don't need a postprocessing formula to combine the columns.
- 04:02 AM Revision 10291: inputs/Madidi/map.csv: removed filters on columns from before the refresh (which are not in active use), so that they don't show up in a search for map.csvs with filters (indicating collisions)
- 03:15 AM Revision 10290: inputs/Madidi/IndividualObservation/map.csv: SeniorCollector: don't prepend it to the CollectorString because the CollectorString already contains it. this may be a change between the BIEN2 and refreshed Madidi data (which uses a significantly different schema).
- 02:37 AM Revision 10289: mappings/VegCore.htm: regenerated from wiki. Special terms: added instructions for adding a distinguishing suffix to each special term in the format special_term#suffix. this is needed for new-style import to make the resulting column name unique within the staging table.
- 02:35 AM Revision 10288: mappings/VegCore-VegBIEN.csv: mapped DUPLICATE to nothing so that it would not be treated as an unmapped term
- 02:33 AM Revision 10287: mappings/VegCore.htm: regenerated from wiki. Special terms: added DUPLICATE.
- 01:56 AM Revision 10286: /README.TXT: Maintenance: regenerate mappings/VegCore.csv: commit command: use single quotes ' instead of double quotes " to avoid needing to \-escape every special char (single quotes ' still need to be escaped)
- 01:51 AM Revision 10285: mappings/VegCore.htm: regenerated from wiki. moved UNUSED, PRIVATE underneath OMIT as subterms.
07/14/2013
- 06:02 AM Revision 10284: mappings/VegCore.htm: Regenerated from wiki
- 05:52 AM Revision 10283: bugfix: bin/*: spell out [:alnum:] as [a-zA-Z0-9] because Python unfortunately doesn't support character classes
- 05:18 AM Revision 10282: web/links/index.htm: updated to Firefox bookmarks. moved Linux, Mac into Unix folder. added instructions to remove old Linux kernels, which fill up the /boot partition. added instructions to force sed to use raw binary mode instead of UTF-8 when UTF-8 is set in the environment. added methods of implementing DB disk space quotas in Postgres. added comparison on my Mac's CPU (2.66 GHz Intel Core i5) with vegbiendev's (2.44 GHz AMD Phenom X4). my Mac's seems to be much faster, so it might make sense to check that the Thor CPUs are faster than the Vis Lab computers' CPUs the next time it gets upgraded. (these diffs can be seen in WinMerge with Moved block detection on. see /README.TXT > WinMerge setup for details.)
- 04:43 AM Revision 10281: inputs/bien_web/observation/VegBIEN.csv: regenerated now that *_index dummy columns have been removed
- 03:26 AM Revision 10280: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtimes. it now takes 25 min instead of 16 min to regenerate the derived cols.
- 03:07 AM Revision 10279: inputs/IRMNG/_README.TXT: added note that when refreshing this datasource, remember to regenerate the TNRS derived cols using the instructions in inputs/.TNRS/schema.sql > tnrs_populate_fields()
- 02:44 AM Revision 10278: bin/*: replaced confusing regexp constructs involving \W inside [] with the much clearer explicit character class [:alnum:] . this avoids adding or subtracting from an inverted class in order to reach a subset of the corresponding positive class, because the subset can just be named explicitly instead.
- 02:38 AM Revision 10277: bugfix: bin/repl: doesn't make sense to use other chars in a [^\W_] regexp, because they will have no effect since \w doesn't include the other chars to begin with. this is a result of confusion with the ^ and \W double negative.
- 02:14 AM Revision 10276: lib/runscripts/table.run: postprocess(): propagate the $remake flag to remake_VegBIEN_mappings using self_make, so that a remake=1 on postprocess will cause map.csv to be regenerated as it would for a remake=1 directly on remake_VegBIEN_mappings
- 02:10 AM Revision 10275: bugfix: postprocess(): moved $can_test flag from import() to this function because it is used here
- 02:08 AM Revision 10274: lib/runscripts/table.run: import(): moved postprocessing commands to separate postprocess() function that can be invoked on an already-imported staging table to avoid running the load_data() target. this is especially useful when running the postprocessing on a working copy without the unversioned data files, for datasources whose load_data() target would otherwise try to download the files because they don't already exist.
- 02:01 AM Revision 10273: lib/runscripts/table.run: postprocess(): renamed to custom_postprocess() since this runs only the datasource's custom postprocessing commands, not all the postprocessing commands including map_table, mk_derived
- 01:52 AM Revision 10272: lib/runscripts/util.run: added , function, which treats each of the command-line args as commands the way make does (instead of as args to the same command, the way runscripts do)
- 01:39 AM Revision 10271: lib/sh/util.sh: moved runscript-related commands to lib/runscripts/util.run because these only apply to runscripts
- 01:26 AM Revision 10270: bugfix: inputs/*/*/map.csv (e.g. inputs/GBIF/raw_occurrence_record_plants/map.csv): remapped author to scientificNameAuthorship rather than authors, which it had gotten incorrectly automapped to. note that the VegCore term authors has now been renamed to data_authors to avoid ambiguity, but incorrect automappings resulting from it had not yet been fixed.
- 12:54 AM Revision 10269: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: updated herbaria.ih column names for staging table column renaming
- 12:33 AM Revision 10268: bugfix: inputs/GBIF/table.run: need to include lib/runscripts/mysql.table.run instead of table.run (table.run was accidentally substituted when inputs/.NCBI/table.run was copied to all new-style datasources
07/13/2013
07/12/2013
07/11/2013
- 04:08 PM Revision 10265: planning/workflow/bien3_architecture.odp: added wiki page notes (wiki.vegpath.org/2013-06-20_conference_call, wiki.vegpath.org/2013-06-27_conference_call) in the slide notes
- 03:41 PM Revision 10264: planning/workflow/bien3_architecture.odp: added responses to the red-highlighted questions (from e-mails to the list) in the slide notes
- 02:54 PM Revision 10263: planning/timeline/timeline.2013.xls: fixed formatting: removed internal cell borders in spacer lines
- 02:48 PM Revision 10262: added planning/workflow/bien3_architecture.odp with changes from Skype call with Martha, which include wiki page notes (wiki.vegpath.org/2013-06-20_conference_call) about the refactor-in-place method in the Notes area
- 02:44 PM Revision 10261: planning/timeline/timeline.2013.xls: updated with changes from Skype call with Martha
- 12:53 PM Revision 10260: inputs/*/ which do not contain any explicit collisions (wiki.vegpath.org/2013-06-27_conference_call#To-do-for-Aaron > #3.2 > the following datasources ...): switched to new-style import, which adds the staging table column renaming
- 12:41 PM Revision 10259: inputs/newWorld/: switched to new-style import, which adds the staging table column renaming. these tables are used by the public schema (schemas/vegbien.sql), so the renamings are applied there as well.
- 12:26 PM Revision 10258: inputs/bien_web/bien_web.schema.sql: regenerated using bin/my2pg, to remove the *_index dummy columns so they don't create lots of OMIT#... staging table columns
- 12:09 PM Revision 10257: inputs/*/*/map.csv: added distinguishing #... suffix (e.g. UNUSED#institutionID) to the special terms OMIT, PRIVATE, UNUSED (VegCore.vegpath.org#Special-terms) to avoid creating a collision in the staging table renaming
- 11:56 AM Revision 10256: bugfix: inputs/input.Makefile: Staging tables installation: $(allInstalls): don't filter out Source table, because it is now an installed table rather than just a mapping
- 11:33 AM Revision 10255: bin/filter_out_ci, lib/maps.py: simplify(): also remove distinguishing #... suffix from terms (e.g. UNUSED#institutionID), to support mapping multiple columns to the special terms OMIT, PRIVATE, UNUSED (VegCore.vegpath.org#Special-terms), *without* creating a collision in the staging table renaming. note that this change must *not* be made to bin/canon, because this would cause suffixed terms to be autorenamed to their *un*suffixed VegCore versions.
- 05:54 AM Revision 10254: backups/Makefile: $(restore): added --verbose to display pg_restore's incremental progress
- 05:34 AM Revision 10253: bugfix: inputs/newWorld/newWorldCountries/postprocess.sql: use UPDATE statement (followed by VACUUM ANALYZE to remove dead tuples) instead of in-place update (ALTER COLUMN TYPE USING), so that the statement can be run even after the public schema has been installed and its views use the columns. (a view using the columns would normally block an ALTER COLUMN TYPE statement on a referenced column.)
- 03:56 AM Revision 10252: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): when remaking, do not remake header.csv, because it should keep the original CSV columns rather than being reset to whatever the current staging table columns happen to be. to force-regenerate this, instead delete it first and then run remake_VegBIEN_mappings(). remake mode will now just regenerate map.csv from header.csv, in case map.csv's columns are incomplete or out of order.
- 03:55 AM Revision 10251: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): when remaking, do not remake header.csv, because it should keep the original CSV columns rather than being reset to whatever the current staging table columns happen to be. to force-regenerate this, instead delete it first and then run remake_VegBIEN_mappings(). remake mode will now just regenerate map.csv from header.csv, in case map.csv's columns are incomplete or out of order.
- 03:50 AM Revision 10250: bugfix: lib/runscripts/table.run: map_table(): do not rename view columns, since their column names come from their (column-renamed) joined tables rather than from a map.csv. header.csv, map.csv for views will generally become out-of-date whenever the joined tables change, so it is better not to generate them at all.
- 03:48 AM Revision 10249: lib/runscripts/table.run: added $is_view
- 03:27 AM Revision 10248: lib/runscripts/table.run: added $postprocess_sql to store postprocess.sql path, and use it in postprocess()
- 02:20 AM Revision 10247: bugfix: lib/sh/local.sh: prevent automated tests when the public schema contains the live DB, so the user doesn't have to explicitly specify can_test= when running the import on vegbiendev
- 02:19 AM Revision 10246: bugfix: lib/runscripts/table.run: import(): allow automated tests (remake_VegBIEN_mappings) to be disabled by setting can_test= if the public schema shouldn't be modified (e.g. if it's the live DB)
- 12:55 AM Revision 10245: bugfix: inputs/*/*/postprocess.sql: made all operations idempotent, so that postprocess.sql can be run repeatedly (e.g. by new-style import)
- 12:03 AM Revision 10244: schemas/util.sql: create_if_not_exists(): also suppress "multiple primary keys are not allowed" error
07/10/2013
- 10:10 PM Revision 10243: added inputs/newWorld/iso_code_gadm/.map.csv.last_cleanup
- 10:07 PM Revision 10242: inputs/*/Source/VegBIEN.csv: regenerated for new-style import, which uses a symlink to mappings/VegCore-VegBIEN.csv instead of a custom mapping using the original column names
- 09:53 PM Revision 10241: inputs/input.Makefile: Staging tables installation: %/install: run %/map_table at end to rename the staging table columns for new-style datasources
- 09:52 PM Revision 10240: inputs/input.Makefile: Staging tables installation: added %/map_table to run the new-style import staging table renaming
- 08:37 PM Revision 10239: inputs/bien2_traits/TraitObservation/map.csv: removed no longer needed mappings of dummy columns to OMIT, which were creating an unnecessary collision of staging table column names
- 08:30 PM Revision 10238: inputs/bien2_traits/bien2_staging.schema.sql: regenerated from MySQL version so that dummy columns (which used to be generated by bin/my2pg) will be replaced with dummy CHECK constraints instead. this avoids needing to map several dummy columns all to OMIT, which was creating an unnecessary collision of staging table column names.
- 08:20 PM Revision 10237: bin/my2pg*: keep MySQL indefinite dates as text strings instead of translating them (to the first of the month or year) to fit into a PostgreSQL timestamp. this allows the application to decide how to handle these values, which otherwise have no corresponding value in PostgreSQL. this requires changing the date/time related types to text instead of leaving them as-is, so that they can store the custom MySQL strings.
- 07:36 PM Revision 10236: planning/timeline/timeline.2013.xls: Geoscrubbing: made it a subtask of Adding derived columns. moved it to July so that it can be run for Naia's new project.
- 07:00 PM Revision 10235: planning/timeline/timeline.2013.xls: reordered tasks approximately in priority order (which corresponds to the month(s) in which they are scheduled). indented subtasks under their parent tasks.
- 06:51 PM Revision 10234: planning/timeline/timeline.2013.xls: crossed out completed rows and moved them to the bottom
- 06:46 PM Revision 10233: planning/timeline/timeline.2013.xls: use different-style checkmark because LibreOffice doesn't display the font of the previous one correctly anymore (it may already have been displayed incorrectly on other people's computers)
- 06:14 PM Revision 10232: planning/timeline/timeline.2013.xls: Reload existing data in need of refresh: added Oct because Rick Condit is supposed to provide us with a CTFS refresh that we would be allowed to use (he wouldn't let us use the 2011-4-1 full-DB export)
- 06:11 PM Revision 10231: planning/timeline/timeline.2013.xls: continuous tasks: populated past months
- 06:09 PM Revision 10230: planning/timeline/timeline.2013.xls: added Sep, Oct months and moved tasks into them. moved continuous tasks to separate section at bottom to avoid confusion with discrete tasks.
- 05:33 PM Revision 10229: planning/timeline/timeline.2013.xls: use bullet points (•) instead of background shading to indicate future tasks. this allows cells to easily be cleared by pressing Backspace, rather than having to copy a white-background cell on top of the cell.
- 05:26 PM Revision 10228: planning/timeline/timeline.2013.xls: use 3-letter months to make room for more months
- 05:23 PM Revision 10227: planning/timeline/timeline.2013.xls: added missing tasks: switching to new-style import, importing to normalized VegCore, adding derived columns
- 05:16 PM Revision 10226: planning/timeline/timeline.2013.xls: removed alterate-row color highlighting because it makes it difficult to reorder rows or insert new rows in the middle
- 04:51 PM Revision 10225: bin/my2pg: use util.sh $top_dir instead of setting $selfDir
- 04:50 PM Revision 10224: bin/my2pg*: use the util.sh sed wrapper, which fixes the LANG=*.UTF-8 "illegal byte sequence" errors on invalid UTF-8
- 04:33 PM Revision 10223: /Makefile: mysql-Linux: also install mysql-workbench, for use in modifying the VegCore ERD. (note that it has to be modified on Linux, because the Linux and Mac versions of MySQL Workbench position the lines differently.)
- 04:10 PM Revision 10222: /README.TXT: Maintenance: to backup files not in Time Machine: removed VirtualBox VMs because they are now in Time Machine, and do not need to be backed up separately
- 04:08 PM Revision 10221: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: added steps to upload just the VirtualBox VMs
- 04:02 PM Revision 10220: bugfix: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: added overwrite=1 so that old snapshots, etc. are also deleted
- 04:01 PM Revision 10219: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: use better bin/sync_upload instead of put
- 03:59 PM Revision 10218: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: removed no longer needed inplace=1, because the VirtualBox VMs now all use a snapshot covering the full disk, so that the full disk is not altered (removing the need to optimize backing up a large file) and just the diff files need to be backed up each time
- 03:41 PM Revision 10217: bugfix: lib/sh/util.sh: sed: must use alias instead of function because function causes segfault in redir() subshell when used with make.sh make() filter (may be bug in bash?). this involves translating `unset LANG` to `env LANG=` (`env -u` to unset a var isn't supported on Mac, but fortunately sed treats LANG="" the same as unset LANG).
- 03:06 PM Revision 10216: archived planning/goals/BIEN3_derived_data_products.docx and replaced with symlink to new BIEN_3_derived_data_products_NormalizedDB_only.docx
- 02:59 PM Revision 10215: added planning/goals/BIEN_3_derived_data_products_NormalizedDB_only.docx from Brad's e-mail
- 02:42 PM Revision 10214: bugfix: lib/sh/util.sh: sed: unset LANG to avoid "illegal byte sequence" errors on invalid UTF-8 for LANG=*.UTF-8. these occur e.g. with MySQL data that is in Latin-1.
- 02:36 PM Revision 10213: lib/sh/util.sh: sed: use function instead of alias so that env can be set up before calling sed
- 02:15 PM Revision 10212: planning/workflow/bien3_architecture.pptx: updated to Martha's revised version from 2013-7-3
- 04:13 AM Revision 10211: lib/runscripts/table.run: map_table(): run map_table repeatedly until no more renames are made: added command to do this
- 03:53 AM Revision 10210: lib/runscripts/table.run: map_table(): documented that collisions may prevent all renames from being made at once. if this is the case, map_table must be run repeatedly until no more renames are made. collisions may result if the staging table gets messed up (e.g. due to missing input columns in map.csv).
- 02:32 AM Revision 10209: inputs/*/*/map.csv for CSV tables with a row_num column: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table
- 02:27 AM Revision 10208: bugfix: inputs/*/Source/map.csv: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table. the staging table column renaming is now used by all Source tables.
- 02:18 AM Revision 10207: bugfix: populated empty inputs/IUCN/European_Red_List_Plants/header.csv
- 02:17 AM Revision 10206: inputs/CTFS/*/map.csv: added *.src.row_num from joined tables so that the map.csv input columns would match the staging table. this is needed for the staging table column renaming, which is positional rather than name-based to work with any existing column name.
- 01:50 AM Revision 10205: bugfix: inputs/input.Makefile: map.csv and derived files: use $(tables) instead of $(importTables) when making them so that the mappings of those tables are still kept up-to-date even though they are marked _no_import (and not imported into the main DB)
- 01:46 AM Revision 10204: inputs/CTFS/*/test.xml.ref: regenerated. these got out of date because even though these tables are included in import_order.txt, they are marked as _no_import, which prevents map.csvs and derived files from being kept up-to-date.
- 01:24 AM Revision 10203: bugfix: inputs/CTFS/*/VegBIEN.csv: regenerated from map.csv. they may have gotten out of date because they are marked as _no_import, even though they *are* in import_order.txt.
07/09/2013
- 05:33 PM Revision 10202: bugfix: added missing inputs/MO/Specimen/header.csv
- 05:32 PM Revision 10201: bugfix: added missing inputs/QFA/Specimen/header.csv
- 05:26 PM Revision 10200: bugfix: inputs/TEX/Specimen/header.csv: generated from staging table (was empty previously)
- 04:44 PM Revision 10199: bugfix: inputs/*/Source/map.csv: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table. the staging table column renaming is now used by all Source tables.
- 04:42 PM Revision 10198: added inputs/newWorld/iso_code_gadm/header.csv
- 04:31 PM Revision 10197: added inputs/analytical_db/table.run
- 02:59 PM Revision 10196: bugfix: inputs/VASCAN/Taxon/map.csv: added missing row_num column added by bin/csv2db
- 02:50 PM Revision 10195: lib/sql_io.py: cleanup_table(): added assertion that the table exists, so that if it doesn't, the error will occur as part of an assertion rather than as part of the util.table_nulls_mapped__get() call, which might confusingly lead users to believe that this is a bug in util.table_nulls_mapped__get() when in fact the problem is that the table is not installed
- 02:30 AM Revision 10194: fix: inputs/import.stats.xls: removed spurious diff comment on total time, which only applied to the previous import
- 02:28 AM Revision 10193: inputs/import.stats.xls: reformatted times longer than one day as a # of days instead of hours, for clarity. the days format is chosen automatically when the # hours exceeds one day.
- 01:04 AM Revision 10192: bugfix: inputs/*/Source/: added missing ./run, which creates the new-style staging tables with the metadata fields as part of the table. this is needed now that these subdirs use installed staging tables instead of metadata-only map.csvs.
- 12:56 AM Revision 10191: bin/map: removed no longer used support for map.csv input column prefixes (expand out the prefixes instead). this used to be used by SpeciesLink to use just one mapping for a single term with multiple DwC namespaces, but was replaced with an explicit, ordered rather than implicit, unordered /_alt-ing together of the terms.
07/08/2013
Also available in: Atom