config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (25 min)

config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to create snapshot before running pg_restore

config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to configure the VM before booting

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions for selecting the initial snapshot and booting the VM

config/VirtualBox_VMs/vegbiendev/README.TXT: perform long-running operations inside `screen`

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to use a database backup: only run mk_db, not db, to avoid restoring schemas that are part of the backup

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: schemas must be in dependency order

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: don't restore analytical_db schema because we don't use it and it depends on public, which we're not restoring

web/links/index.htm: updated to Firefox bookmarks: WebEx: added steps to join a test meeting. pg_restore: documented that you need to create the schema before you restore into it, or else you will get a cryptic error message that it's using pg_catalog instead. added Homebrew links, including instructions for setting psycopg2 up with it. added PostGIS links. username URLs: documented how to enable these in Internet Explorer. TWiki: documented what to do when moving the TWiki install dir.

bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to create the schema before you restore into it, to avoid cryptic error messages

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: need to run `make db` to restore users, which are not part of the backup

config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (35 min)

config/VirtualBox_VMs/vegbiendev/README.TXT: turning off of non-ARIZ inputs: removed metadata datasources because these have been renamed to start with ".", and so aren't included in */

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: pg_restore: need to use for loop because pg_restore doesn't support multiple --schema options (it just uses the last one)

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to restore the necessary schemas from the DB

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: rsync: documented runtime (13 min)

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to fetch the database backup via rsync

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: moved steps to enable the user to access VirtualBox shared folders to separate section because VirtualBox shared folders currently don't work, likely because of a bug in VirtualBox Guest Additions when installed on something other than a clean Linux install

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added steps to enable the user to access VirtualBox shared folders

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added section for database backup approach, as alternative to clean database approach

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: use ".." instead of - to indicate a range, to avoid this being confused for a minus sign

bugfix: /Makefile: postgres_restart-Darwin: need to have the user run the command outside of make because it doesn't appear to do anything when run inside make (this is the same problem as on Linux)

/Makefile: $(pg_ctl-Darwin): use simpler `brew services` instead of `launchctl __/homebrew.mxcl.postgresql.plist`

bugfix: bin/stop_imports: also need to include `bin/after_import`

inputs/.TNRS/Source/map.csv: datasetURL: updated to indicate that this uses the TNRS dev server (the actual URL is private)

fix: inputs/VASCAN/Source/map.csv: observationGranularity should not be specimen because this datasource contains only taxa

bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off

bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd

bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell

bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window

bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)

bugfix: bin/import_all: need to always use log files for background processes

fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.

fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted

bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources

bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded

fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.

fix: bin/import_all: updated for new metadata datasource names (see issue #940)

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)

fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)

fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)

bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.

config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done

config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure

added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import

inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.

fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates

validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column

schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on each function call

schemas/util.sql: added map_words()

schemas/util.sql: added date_from_spanish()

schemas/util.sql: added replace_words()

schemas/util.sql: added words(text)

schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these

schemas/util.sql: added spanish_date_words()

schemas/util.sql: added `replacement` type

bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()

schemas/util.sql: added mk_set_comments(regclass)

schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions

schemas/util.sql: added mk_set_col_comments(regclass)

schemas/util.sql: added cols(regclass)

schemas/util.sql: added mk_set_comment(col_ref)

added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets

fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.

bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import

inputs/FIA/taxon_observation.**/test.xml.ref: updated

inputs/FIA/TREE/test.xml.ref: updated

inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location

bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location

fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate

fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)

schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)

schemas/util.sql: added mk_set_comment(col col_ref, comment text)

schemas/util.sql: added sql(col_ref)

schemas/util.sql: added mk_set_comment(on_ text, comment text)

schemas/util.sql: added comment(col_ref)

schemas/util.sql: added col_num(col_ref)

removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run

bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run

schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this

exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)

bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn

schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.

fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row

bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled

bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used

bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times

bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times

schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn

schemas/public_.sql: added in_iucn_red_list()

schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view

bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names

schemas/public_.sql: added iucn_red_list

schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred

schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized