bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off

bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd

bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell

bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window

bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)

bugfix: bin/import_all: need to always use log files for background processes

fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.

fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted

bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources

bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded

fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.

fix: bin/import_all: updated for new metadata datasource names (see issue #940)

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)

fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)

fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)

bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done

config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure

added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import

inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.

fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates

validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column

schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on each function call

schemas/util.sql: added map_words()

schemas/util.sql: added date_from_spanish()

schemas/util.sql: added replace_words()

schemas/util.sql: added words(text)

schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these

schemas/util.sql: added spanish_date_words()

schemas/util.sql: added `replacement` type

bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()

schemas/util.sql: added mk_set_comments(regclass)

schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions

schemas/util.sql: added mk_set_col_comments(regclass)

schemas/util.sql: added cols(regclass)

schemas/util.sql: added mk_set_comment(col_ref)

added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets

fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.

bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import

inputs/FIA/taxon_observation.**/test.xml.ref: updated

inputs/FIA/TREE/test.xml.ref: updated

inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location

fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate

fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)

schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)

schemas/util.sql: added mk_set_comment(col col_ref, comment text)

schemas/util.sql: added sql(col_ref)

schemas/util.sql: added mk_set_comment(on_ text, comment text)

schemas/util.sql: added comment(col_ref)

schemas/util.sql: added col_num(col_ref)

removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run

bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run

schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this

exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)

bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn

schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.

fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row

bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled

bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used

bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times

bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn

schemas/public_.sql: added in_iucn_red_list()

schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view

bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names

schemas/public_.sql: added iucn_red_list

schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred

schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized

schemas/public_.sql: added iucn_red_list

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added collection_type (using source__observation_type())

schemas/public_.sql: added source__observation_type()

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added accepted_subspecies, and removed accepted_taxon_name_with_author which had been substituting for it

schemas/public_.sql: added subspecies() wrapper

schemas/util.sql: added subspecies()

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only coordinates in South America

schemas/util.sql: added south_america(), in_south_america()

fix: schemas/util.sql: OPERATOR ~(geocoord, postgis.geometry): renamed to because this is not an approximate comparison for geometry

fix: schemas/util.sql: contained_within_approx(geocoord, postgis.geometry): renamed to contained_within__no_dateline(__) because this is not an approximate comparison for geometry

schemas/util.sql: lat_long_in_new_world(): renamed to just in_new_world() because the lat/long is implied by the param type

schemas/util.sql: lat_long_in_new_world(): take a geocoord param instead of separate lat/long params

backups/TNRS.backup.md5: updated

schemas/util.sql: added contained_within_approx(geocoord, geometry) and corresponding OPERATOR ~@(geocoord, geometry)

schemas/util.sql: added OPERATOR ~@(geocoord, geography)

schemas/util.sql: lat_long_in_new_world(): use new contained_within_approx(geocoord, geography)

schemas/util.sql: added contained_within_approx(geocoord, postgis.geography), which enables specifying just `(lat, long)` without the ::util.geocoord type specifier

schemas/util.sql: OPERATOR (postgis.geography, postgis.geography): renamed to ~ because it's approximate

schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box

schemas/util.sql: added contained_within__no_dateline(geometry, geometry) and corresponding operator @

schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box

schemas/util.sql: geometry(geocoord): documented that it is not possible to create a cast for this, as a bug in pg_dump prevents the cast from being exported, even when no export filters are applied

schemas/util.sql: point(geocoord): renamed to geometry(geocoord) since this is now a cast

schemas/util.sql: point(): return geometry instead of geography to support using points with geometry arithmetic

schemas/util.sql: point(): take a single util.geocoord param instead of separate lat/long

schemas/util.sql: added geocoord type

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only georeferenced occurrences (lat/long NOT NULL)

schemas/util.sql: bounding_box(): use bounding_box__no_dateline() to construct the postgis.geometry object

schemas/util.sql: added bounding_box__no_dateline(), which is more accurate than util.bounding_box() (latitude lines will be straight), but geocoordinate wraparound is not supported