Activity
From 06/16/2014 to 07/15/2014
07/15/2014
- 11:14 PM Revision 14079: bugfix: /README.TXT: logs glob: don't unintentionally match . or ..
- 10:43 PM Revision 14078: inputs/VegBank/stemlocation_/test.xml.ref: updated
- 10:41 PM Revision 14077: /README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)
- 10:07 PM Revision 14076: inputs/TEX/Specimen2/test.xml.ref: updated
- 09:35 PM Revision 14075: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: updated
- 05:50 PM Task #887 (Rejected): fix disk space leak that fills the disk and crashes the import
- 05:48 PM Task #902 (Resolved): fix bug that causes joining on the wrong columns in the import
- 05:47 PM Task #905 (Resolved): narrow down the cause of the import bug (incorrect join columns and disk space leak)
- bug fixed in r14074
- 05:44 PM Revision 14074: bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add *new* entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.
- 04:49 PM Revision 14073: bin/import_all: delete_logs(): documented that `trap EXIT` doesn't run until *shell* exit
- 04:48 PM Revision 14072: bin/import_all: delete_logs(): print when this happens, so it can be verified that it's happening properly
- 04:32 PM Revision 14071: bugfix: bin/import_all: need to run delete_logs manually because `trap EXIT` doesn't run until bg cmds done
- 04:28 PM Revision 14070: bin/import_all: delete_logs: moved testing of whether to delete logs to delete_logs() so that delete_logs() can be run regardless of the $delete_logs setting
- 03:58 PM Revision 14069: bugfix: bin/import_all: delete_logs(): also need to match log filenames when n=""
- 03:11 PM Revision 14068: bugfix: /Makefile: postgres_restart-*: argument to echo always needs quotes
- 03:05 PM Revision 14067: bugfix: /Makefile: $(pg_ctl-*): tell user that if it doesn't work, need to run `rm $(macPostgresDir)/postmaster.pid` and retry
- 02:57 PM Revision 14066: /Makefile: $(pg_ctl-Darwin): documented the pg_ctl version of this command in case `brew services` doesn't work
- 02:39 PM Revision 14065: bugfix: bin/with_all: isset(): need to use `>&- 2>&-` because &> does not work with - as the dest
- 01:19 PM Revision 14064: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: determine if the revision works: use the in# of the query instead of waiting an arbitrary time, because this is more accurate in determining whether the import is successful
- 12:52 PM Revision 14063: config/VirtualBox_VMs/vegbiendev/README.TXT: regressions that occurred during the revision range: named these "concurrent regressions"
- 12:45 PM Revision 14062: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to update system time so wait times can be calculated correctly
- 12:22 PM Revision 14061: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented that the patches are specifically for regressions that occurred during the revision range, not all bugs that were fixed in that range
- 12:14 PM Revision 14060: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: waiting for query: increased time to min. 10 min because some revisions take longer
- 12:09 PM Revision 14059: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: added step to type y and press ENTER
- 12:07 PM Revision 14058: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to determine if the revision works, and then stop the import
- 11:27 AM Revision 14057: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12994
- 11:09 AM Revision 14056: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use for loop to avoid needing to repeat the command
- 11:05 AM Revision 14055: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12760
- 10:54 AM Revision 14054: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: --fuzz apparently needs to be 3 even though we have only 2 lines of context
- 10:52 AM Revision 14053: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use --fuzz to allow patching even when surrounding lines have changed
- 10:44 AM Revision 14052: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: sorted by revision #
- 10:43 AM Revision 14051: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12865
- 10:33 AM Revision 14050: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: patch: need to use --force to avoid prompting user
- 10:16 AM Revision 14049: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn revert`: need to use `--depth infinity` when reverting more than a file
- 10:14 AM Revision 14048: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: use `patch` instead of `svn merge` because `svn merge` sometimes inexplicably doens't apply patches
- 09:43 AM Revision 14047: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: `svn revert` all changes instead of specific files, to support bugfixes to other files
- 09:42 AM Revision 14046: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: use `svn merge` instead of `sed --in-place` because this can be used for more complex changes
- 09:41 AM Revision 14045: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: documented revision
- 09:20 AM Revision 14044: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to apply r12872 bugfix to lib/sh/util.sh because it occurred after r12025
- 09:10 AM Revision 14043: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make` commands: use `yes|` to avoid needing to confirm prompts
- 09:08 AM Revision 14042: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make mk_db ...`: use `yes|` to avoid needing to confirm prompts
- 08:56 AM Revision 14041: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: to test a different revision: added step that if any of the commands after `svn up` are broken, try a different revision
- 08:46 AM Revision 14040: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make db`: need to run `schemas/reinstall` manually because older versions of the codebase just used `schemas/install`
07/14/2014
- 11:30 PM Revision 14039: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented the range from within the revision should be chosen
- 09:19 PM Revision 14038: fix: inputs/.IRMNG/*/: added _no_import because this datasource is only used as a lookup table and does not need to be imported/scrubbed
- 09:18 PM Revision 14037: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn up`: documented runtime (1 min)
- 08:24 PM Revision 14036: bugfix: added nodes/_no_import, because the NCBI import doesn't work due to a Postgres 9.3 bug (#859). _no_import used to not be needed because we didn't import the hidden sources.
- 08:05 PM Revision 14035: fix: bin/with_all: removed debug statements
- 05:19 PM Revision 14034: bugfix: bin/with_all: testing if @inputs is set: `"${inputs+isset}"` syntax doesn't work for empty arrays, so need to use `declare -p` instead
- 04:46 PM Revision 14033: bugfix: to set up the VM: to use a database backup: also need to `make rm_initial_public` because initdb creates a non-empty DB
- 04:42 PM Revision 14032: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to test a different revision
- 04:40 PM Revision 14031: /Makefile: db: use reinstall instead of install to allow running this on an existing DB
- 02:49 PM Revision 14030: config/VirtualBox_VMs/vegbiendev/README.TXT: `sudo apt-get dist-upgrade`: documented runtime (12 min)
- 02:47 PM Revision 14029: config/VirtualBox_VMs/vegbiendev/README.TXT: updated VM snapshot names
- 02:44 PM Revision 14028: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:34 PM Revision 14027: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:29 PM Revision 14026: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to install system updates
- 01:23 PM Revision 14025: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: documented runtime (50 min)
- 01:07 PM Revision 14024: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to run the import
- 12:24 PM Revision 14023: config/VirtualBox_VMs/vegbiendev/README.TXT: to set up the VM: added steps to move vegbiendev.vdi to the internal hard drive
- 12:19 PM Revision 14022: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to start the VM
- 12:16 PM Revision 14021: config/VirtualBox_VMs/vegbiendev/README.TXT: moved "set up the VM" steps to separate section
- 12:06 PM Revision 14020: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added steps to create a booted snapshot
- 12:01 PM Revision 14019: config/VirtualBox_VMs/vegbiendev/README.TXT: moved creation of pg_restore snapshot to be part of just the database backup section
- 11:52 AM Revision 14018: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: documented runtime (15 min)
- 11:51 AM Revision 14017: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: time the operation
- 11:32 AM Revision 14016: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added name for VM snapshot after this step
- 11:31 AM Revision 14015: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to run as DB superuser to restore py_util
- 11:24 AM Revision 14014: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (25 min)
- 11:24 AM Revision 14013: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to create snapshot before running pg_restore
- 10:55 AM Revision 14012: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to configure the VM before booting
- 10:53 AM Revision 14011: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions for selecting the initial snapshot and booting the VM
- 10:36 AM Revision 14010: config/VirtualBox_VMs/vegbiendev/README.TXT: perform long-running operations inside `screen`
- 10:35 AM Revision 14009: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to use a database backup: only run mk_db, not db, to avoid restoring schemas that are part of the backup
- 09:33 AM Revision 14008: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: schemas must be in dependency order
- 09:21 AM Revision 14007: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: don't restore analytical_db schema because we don't use it and it depends on public, which we're not restoring
- 09:16 AM Revision 14006: web/links/index.htm: updated to Firefox bookmarks: WebEx: added steps to join a test meeting. pg_restore: documented that you need to *create* the schema before you restore into it, or else you will get a cryptic error message that it's using pg_catalog instead. added Homebrew links, including instructions for setting psycopg2 up with it. added PostGIS links. username URLs: documented how to enable these in Internet Explorer. TWiki: documented what to do when moving the TWiki install dir.
- 09:06 AM Revision 14005: bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag
- 08:36 AM Revision 14004: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to *create* the schema before you restore into it, to avoid cryptic error messages
- 08:33 AM Revision 14003: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
- 08:18 AM Revision 14002: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
07/13/2014
- 11:50 PM Revision 14001: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: need to run `make db` to restore users, which are not part of the backup
- 11:40 PM Revision 14000: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (35 min)
07/11/2014
- 04:54 PM Revision 13999: config/VirtualBox_VMs/vegbiendev/README.TXT: turning off of non-ARIZ inputs: removed metadata datasources because these have been renamed to start with ".", and so aren't included in */
- 04:48 PM Revision 13998: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: pg_restore: need to use for loop because pg_restore doesn't support multiple --schema options (it just uses the last one)
- 04:35 PM Revision 13997: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to restore the necessary schemas from the DB
- 04:32 PM Revision 13996: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: rsync: documented runtime (13 min)
- 04:11 PM Revision 13995: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to fetch the database backup via rsync
- 04:04 PM Revision 13994: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: moved steps to enable the user to access VirtualBox shared folders to separate section because VirtualBox shared folders currently don't work, likely because of a bug in VirtualBox Guest Additions when installed on something other than a clean Linux install
- 03:54 PM Revision 13993: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added steps to enable the user to access VirtualBox shared folders
- 03:06 PM Revision 13992: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added section for database backup approach, as alternative to clean database approach
- 10:32 AM Revision 13991: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: use ".." instead of - to indicate a range, to avoid this being confused for a minus sign
- 10:26 AM Revision 13990: bugfix: /Makefile: postgres_restart-Darwin: need to have the user run the command outside of make because it doesn't appear to do anything when run inside make (this is the same problem as on Linux)
- 10:25 AM Revision 13989: /Makefile: $(pg_ctl-Darwin): use simpler `brew services` instead of `launchctl __/homebrew.mxcl.postgresql.plist`
- 10:03 AM Revision 13988: bugfix: bin/stop_imports: also need to include `bin/after_import`
- 09:56 AM Revision 13987: inputs/.TNRS/Source/map.csv: datasetURL: updated to indicate that this uses the TNRS dev server (the actual URL is private)
- 09:46 AM Revision 13986: fix: inputs/VASCAN/Source/map.csv: observationGranularity should not be specimen because this datasource contains only taxa
- 09:13 AM Revision 13985: bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off
- 08:45 AM Revision 13984: bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd
- 08:41 AM Revision 13983: bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell
- 08:38 AM Revision 13982: bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window
- 08:32 AM Revision 13981: bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)
- 08:25 AM Revision 13980: bugfix: bin/import_all: need to always use log files for background processes
- 08:12 AM Revision 13979: fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.
- 08:06 AM Revision 13978: fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted
- 07:40 AM Revision 13977: bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources
- 07:39 AM Revision 13976: bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded
- 07:34 AM Revision 13975: fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.
07/10/2014
- 07:25 PM Revision 13974: fix: bin/import_all: updated for new metadata datasource names (see issue #940)
- 07:25 PM Revision 13973: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)
- 06:41 PM Task #940 (Resolved): resequence datasources in dependency order
- 04:22 PM Task #940 (Resolved): resequence datasources in dependency order
- * this enables bootstrapping the DB from a VM backup
steps
# put all [[VegBIEN_contents#datasources|metadata da... - 06:32 PM Revision 13972: fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)
- 06:04 PM Revision 13971: fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)
- 04:59 PM Revision 13970: bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.
- 03:52 PM Revision 13969: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done
- 03:37 PM Revision 13968: config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)
- 03:33 PM Revision 13967: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure
- 03:32 PM Revision 13966: added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import
- 12:17 PM Revision 13965: inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.
- 10:57 AM Revision 13964: fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates
- 09:45 AM Revision 13963: validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column
- 08:41 AM Revision 13962: schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on *each* function call
- 08:39 AM Revision 13961: schemas/util.sql: added map_words()
- 08:16 AM Revision 13960: schemas/util.sql: added date_from_spanish()
- 08:13 AM Revision 13959: schemas/util.sql: added replace_words()
- 08:09 AM Revision 13958: schemas/util.sql: added words(text)
- 08:02 AM Revision 13957: schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these
- 07:20 AM Revision 13956: schemas/util.sql: added spanish_date_words()
- 07:19 AM Revision 13955: schemas/util.sql: added `replacement` type
- 04:09 AM Revision 13954: bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()
- 04:07 AM Revision 13953: schemas/util.sql: added mk_set_comments(regclass)
- 04:06 AM Revision 13952: schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions
- 04:01 AM Revision 13951: schemas/util.sql: added mk_set_col_comments(regclass)
- 03:58 AM Revision 13950: schemas/util.sql: added cols(regclass)
- 03:56 AM Revision 13949: schemas/util.sql: added mk_set_comment(col_ref)
- 03:18 AM Revision 13948: added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets
07/09/2014
- 02:50 PM Revision 13947: fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.
- 02:45 PM Revision 13946: bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import
- 02:34 PM Revision 13945: inputs/FIA/taxon_observation.**/test.xml.ref: updated
- 02:34 PM Revision 13944: inputs/FIA/TREE/test.xml.ref: updated
- 02:29 PM Revision 13943: inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated
- 02:13 PM Revision 13942: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location
- 02:12 PM Revision 13941: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location
- 12:14 PM Revision 13940: fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate
- 12:05 PM Revision 13939: fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)
- 09:40 AM Revision 13938: schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)
- 09:38 AM Revision 13937: schemas/util.sql: added mk_set_comment(col col_ref, comment text)
- 09:37 AM Revision 13936: schemas/util.sql: added sql(col_ref)
- 09:35 AM Revision 13935: schemas/util.sql: added mk_set_comment(on_ text, comment text)
- 07:53 AM Revision 13934: schemas/util.sql: added comment(col_ref)
- 07:50 AM Revision 13933: schemas/util.sql: added col_num(col_ref)
- 07:03 AM Revision 13932: removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run
- 07:03 AM Revision 13931: bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run
- 06:22 AM Revision 13930: schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this
07/08/2014
- 04:17 PM Revision 13929: exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)
- 03:23 PM Revision 13928: bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn
07/07/2014
- 07:33 AM Revision 13927: schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.
- 07:26 AM Revision 13926: fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row
- 07:04 AM Revision 13925: bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled
- 06:55 AM Revision 13924: bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used
- 06:51 AM Revision 13923: bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:48 AM Revision 13922: bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:43 AM Revision 13921: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn
- 06:37 AM Revision 13920: schemas/public_.sql: added in_iucn_red_list()
- 06:36 AM Revision 13919: schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view
- 06:22 AM Revision 13918: bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names
- 06:17 AM Revision 13917: schemas/public_.sql: added iucn_red_list
- 06:14 AM Revision 13916: schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred
- 06:02 AM Revision 13915: schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized
- 06:01 AM Revision 13914: schemas/public_.sql: added iucn_red_list
- 05:44 AM Revision 13913: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added collection_type (using source__observation_type())
- 05:36 AM Revision 13912: schemas/public_.sql: added source__observation_type()
- 05:04 AM Revision 13911: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added accepted_subspecies, and removed accepted_taxon_name_with_author which had been substituting for it
- 04:50 AM Revision 13910: schemas/public_.sql: added subspecies() wrapper
- 04:38 AM Revision 13909: schemas/util.sql: added subspecies()
- 04:10 AM Revision 13908: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only coordinates in South America
- 03:55 AM Revision 13907: schemas/util.sql: added south_america(), in_south_america()
- 03:37 AM Revision 13906: fix: schemas/util.sql: OPERATOR ~@(geocoord, postgis.geometry): renamed to @ because this is not an approximate comparison for geometry
- 03:35 AM Revision 13905: fix: schemas/util.sql: contained_within_approx(geocoord, postgis.geometry): renamed to contained_within__no_dateline(__) because this is not an approximate comparison for geometry
- 03:30 AM Revision 13904: schemas/util.sql: lat_long_in_new_world(): renamed to just in_new_world() because the lat/long is implied by the param type
- 03:08 AM Revision 13903: schemas/util.sql: lat_long_in_new_world(): take a geocoord param instead of separate lat/long params
- 03:00 AM Revision 13902: backups/TNRS.backup.md5: updated
07/06/2014
- 11:08 PM Revision 13901: schemas/util.sql: added contained_within_approx(geocoord, geometry) and corresponding OPERATOR ~@(geocoord, geometry)
- 10:39 PM Revision 13900: schemas/util.sql: added OPERATOR ~@(geocoord, geography)
- 10:23 PM Revision 13899: schemas/util.sql: lat_long_in_new_world(): use new contained_within_approx(geocoord, geography)
- 10:17 PM Revision 13898: schemas/util.sql: added contained_within_approx(geocoord, postgis.geography), which enables specifying just `(lat, long)` without the ::util.geocoord type specifier
- 04:04 PM Revision 13897: schemas/util.sql: OPERATOR @(postgis.geography, postgis.geography): renamed to ~@ because it's approximate
- 03:52 PM Revision 13896: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 03:50 PM Revision 13895: schemas/util.sql: added contained_within__no_dateline(geometry, geometry) and corresponding operator @
- 02:51 PM Revision 13894: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 05:49 AM Revision 13893: schemas/util.sql: geometry(geocoord): documented that it is not possible to create a cast for this, as a bug in pg_dump prevents the cast from being exported, even when no export filters are applied
07/05/2014
- 12:40 PM Revision 13892: schemas/util.sql: point(geocoord): renamed to geometry(geocoord) since this is now a cast
- 12:03 PM Revision 13891: schemas/util.sql: point(): return geometry instead of geography to support using points with geometry arithmetic
- 11:24 AM Revision 13890: schemas/util.sql: point(): take a single util.geocoord param instead of separate lat/long
- 10:45 AM Revision 13889: schemas/util.sql: added geocoord type
07/04/2014
- 08:29 PM Revision 13888: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only *georeferenced* occurrences (lat/long NOT NULL)
- 08:25 PM Revision 13887: schemas/util.sql: bounding_box(): use bounding_box__no_dateline() to construct the postgis.geometry object
- 08:23 PM Revision 13886: schemas/util.sql: added bounding_box__no_dateline(), which is more accurate than util.bounding_box() (latitude lines will be straight), but geocoordinate wraparound is not supported
- 08:14 PM Revision 13885: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added functional traits that we have 1st-class columns for (dbh_cm, height_m)
- 12:29 PM Revision 13884: added exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run
- 11:12 AM Revision 13883: inputs/publishable datasources.xlsx: updated
- 08:45 AM Revision 13882: inputs/publishable datasources.xlsx: updated
07/03/2014
- 08:13 PM Revision 13881: inputs/.TNRS/schema.sql: matchedFamily: just use Name_matched_accepted_family, because TNRS has now been reloaded so that the names that were missing this have it populated
- 09:59 AM Revision 13880: planning/meetings/BIEN conference call availability.xlsx: updated
- 03:47 AM Task #917: TNRS: Instructions for new version with TPL
- a single "-" creates strikethrough formatting, so you have to use "--" instead
- 12:34 AM Revision 13879: inputs/.TNRS/schema.sql: taxon_match: taxon_match__valid_match: replaced with taxon_best_match__valid_match, which also applies taxon_best_match's filters, since taxon_match is now accessed through taxon_best_match
- 12:17 AM Revision 13878: fix: inputs/.TNRS/schema.sql: MatchedTaxon: use taxon_best_match instead of taxon_match because this should provide only one match per taxon
07/02/2014
- 02:17 AM Revision 13877: bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter
- 02:00 AM Revision 13876: /README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi
06/27/2014
- 03:15 PM Task #917: TNRS: Instructions for new version with TPL
- test update now that watcher notifications have been fixed
06/26/2014
- 06:30 PM Revision 13875: inputs/.TNRS/schema.sql: added taxon_best_match view
- 06:12 PM Revision 13874: inputs/.TNRS/schema.sql: taxon_match: added taxon_match__one_selected_match unique index
- 05:06 PM Revision 13873: backups/TNRS.backup.md5: updated
- 09:49 AM Revision 13872: planning/meetings/BIEN conference call availability.xlsx: updated
- 09:25 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications again after e-mail address fix...
- 09:23 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications...
- 05:41 AM Revision 13871: inputs/.TNRS/schema.sql: taxon_match__fill(): split into separate DECLARE blocks for each field for clarity
- 04:58 AM Revision 13870: inputs/.TNRS/data.sql: refreshed
- 04:57 AM Revision 13869: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:33 AM Revision 13868: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:28 AM Revision 13867: inputs/.TNRS/data.sql: refreshed
- 04:11 AM Revision 13866: inputs/.TNRS/schema.sql: taxon_match: insert names via taxon_match_input auto-updatable view instead of directly into taxon_match, to allow the taxon_match columns to be renamed while still supporting inserts using the TNRS column names
- 02:43 AM Revision 13865: fix: schemas/Makefile: vegbien.sql: also need to update inputs/.TNRS/data.sql, since its contents change along with this
- 02:41 AM Revision 13864: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:39 AM Revision 13863: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:38 AM Revision 13862: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:14 AM Revision 13861: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
06/25/2014
- 07:54 PM Revision 13860: lib/tnrs.py: dirty: documented that this actually used to be on in the web app (see r9910, 2013-6-18), but does not appear to be needed (the source_sorting bug alluded to in r9910 is not fixed by enabling the dirty setting)
- 07:48 PM Task #917: TNRS: Instructions for new version with TPL
- multiple @Selected@ names bug fixed in r13855
- 07:46 PM Revision 13859: lib/tnrs.py: requests: also debug-print request URL
- 07:44 PM Revision 13858: lib/tnrs.py: Download: include the same debug info as do_request()
- 07:41 PM Revision 13857: lib/tnrs.py: do_request(): also debug-print request headers
- 07:39 PM Revision 13856: lib/tnrs.py: download_request_template: dirty: documented why this must be off
- 07:36 PM Revision 13855: bugfix: lib/tnrs.py: download_request_template: fixed bug where multiple names were being marked as Selected, because dirty was incorrectly set to true unlike in the web app
- 07:30 PM Revision 13854: added derived/TNRS/web_app/protocol/
- 04:28 PM Revision 13853: inputs/.TNRS/data.sql: refreshed
- 04:25 PM Revision 13852: bugfix: inputs/.TNRS/schema.sql: taxon_name_is_safe(): need to use `NOT (_ = ANY(_)) instead of `_ != ANY(_)`, because the != operator is applied to each element
- 03:47 PM Revision 13851: inputs/.TNRS/schema.sql: tnrs: renamed to tnrs_match to distinguish it from other TNRS-related tables
- 03:33 PM Revision 13850: inputs/.TNRS/schema.sql: tnrs: renamed to tnrs_match to distinguish it from other TNRS-related tables
- 02:38 PM Revision 13849: fix: lib/phpPgAdmin.login.php.diff: use relative file path rather than the path the file was at when the patch was created
- 02:34 PM Revision 13848: /Makefile, lib/phpPgAdmin.login.php.diff: public_ user: added auto-filled password so that users would not be confused as to what to type in the password field
- 02:15 PM Revision 13847: added schemas/VegCore/phpMyAdmin/libraries/plugins/auth/AuthenticationCookie.class.php.diff
- 01:19 PM Task #938 (New): Enquist's taxon validation
- From Brian Enquist:
I can start by checking the accuracy of the fields, and data for a random subset of plots. To ... - 10:39 AM Revision 13846: inputs/.TNRS/schema.sql: `taxon_scrub.scrubbed_unique_taxon_name.*`: added to-modify instructions
- 10:36 AM Revision 13845: inputs/.TNRS/schema.sql: *_modify(): merged these into the "to modify" instructions in the corresponding views, because there is no need to create a separate *_modify() function for every view now that their definitions are all the same
- 10:21 AM Revision 13844: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
- 05:15 AM Revision 13843: schemas/public_.sql: analytical_stem_view and related views: updated COMMENTs from data dictionary spreadsheet, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#copy-data-dictionary-definitions-to-database
- 04:11 AM Revision 13842: schemas/public_.sql: ran analytical_stem_view_modify(), which transfers the column COMMENTs from analytical_stem_view to analytical_stem
- 04:04 AM Revision 13841: bugfix: schemas/util.sql: view_def_to_orig(): need to handle cases when list of cols from the same table is not an expanded * expression
- 03:59 AM Revision 13840: schemas/util.sql: added view_is_subset(view_def text)
- 03:58 AM Revision 13839: schemas/util.sql: added view_is_automatically_updatable(view_def text)
- 03:23 AM Revision 13838: bugfix: schemas/util.sql: show_create_view(): use the overridden version of pg_get_viewdef(), which supports expanded * expressions. this was possibly being used already whenever util happened to be in the search_path.
06/24/2014
- 05:03 PM Revision 13837: bugfix: schemas/public_.sql: analytical_stem_view_modify(): updated to new analytical_stem_view column names
- 05:01 PM Revision 13836: fix: schemas/public_.sql: analytical_stem_view derived and related views: applied data dictionary renamings, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#apply-data-dictionary-renamings-to-database but with the current columns of analytical_stem as the left-hand column
- 04:36 PM Revision 13835: bugfix: schemas/public_.sql: analytical_stem_view_modify(): updated to new analytical_stem_view column names
- 04:34 PM Revision 13834: schemas/public_.sql: analytical_stem_view: applied data dictionary renamings, using the steps at wiki.vegpath.org/VegBIEN_schema_refactoring#apply-data-dictionary-renamings-to-database
- 03:27 PM Revision 13833: lib/tnrs.py: source_sorting (Constrain by Source): documented the different behavior for this in each match mode (all-matches and best-match)
06/20/2014
06/19/2014
- 04:13 PM Revision 13831: bugfix: web/.phpPgAdmin/.htaccess: : handlers: use `chain` to only run them when the RewriteRule they apply to is run
- 04:11 PM Revision 13830: web/.phpPgAdmin/.htaccess: schema : handler: removed `env=subject:schema` because that has already been set
- 04:08 PM Revision 13829: web/.phpPgAdmin/.htaccess: tables: added : suffix handler
- 04:06 PM Revision 13828: bugfix: web/.phpPgAdmin/.htaccess: view : match: moved to right after `view=` because it only applies to views, not functions
- 03:58 PM Revision 13827: web/.phpPgAdmin/.htaccess: schema: page: indicate by prefixing with : to avoid needing to explicitly list all possible pages
- 03:54 PM Revision 13826: web/.phpPgAdmin/.htaccess: table: action: indicate by prefixing with : to avoid needing to explicitly list all possible actions
- 11:13 AM Revision 13825: exports/2014-6-12.Jeff_Ott.climatic_range_determinants.csv.run: updated export_() runtime (35 min) and rowcount (23 million)
- 11:00 AM Task #916: Taxon name validation: VegBank
- most bugs fixed, just needs TNRS bug workaround
- 02:33 AM Revision 13824: fix: /README.TXT: Full database import: disk space: removed instructions to rerun the import if the disk space gets used up, because this is actually a bug and is *not* generally "fixed just by rerunning the import"
- 02:27 AM Revision 13823: schemas/util.sql: lat_long_in_new_world(): use function rather than operator+search_path to allow inlining, which enables util.new_world() to only be evaluated once
- 02:24 AM Revision 13822: bugfix: schemas/util.sql: operator @(postgis.geography, postgis.geography): must use wrapper function because st_coveredby() needs postgis to be in the search_path
- 01:57 AM Revision 13821: fix: lib/runscripts/extract.run: export_(): explicitly prevent files from becoming web-accessible, to protect against an incorrect umask in the calling process
- 01:45 AM Revision 13820: fix: schemas/util.sql: point(): hide benign "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" notices
06/18/2014
- 08:56 PM Revision 13819: schemas/public_.sql: 2014-6-12.Jeff_Ott.climatic_range_determinants: also include New World occurrences by coordinates, using new lat_long_in_new_world(). this modification (as requested by Jeff) will help reduce the false negatives filtered out by including only data with placenames.
- 08:52 PM Revision 13818: schemas/public_.sql: added lat_long_in_new_world() wrapper around util.lat_long_in_new_world()
- 08:49 PM Revision 13817: schemas/util.sql: lat_long_in_new_world(): documented that this includes false positives above and below the New World bounding box, as described in util.bounding_box()
- 08:44 PM Revision 13816: schemas/util.sql: bounding_box(): documented that the geography type stores all edges as arcs of great circles, resulting in the latitude lines bulging outward from the true bounding box. this will create false positives above and below the bounding box.
- 08:38 PM Revision 13815: schemas/util.sql: added lat_long_in_new_world()
- 08:35 PM Revision 13814: schemas/util.sql: added operator @(postgis.geography, postgis.geography). can't use && for this because it only compares 2D bounding boxes (which are geometry objects that do not support geocoordinate wraparound).
- 08:06 PM Revision 13813: schemas/util.sql: added point()
- 08:00 PM Revision 13812: schemas/util.sql: new_world(): removed no longer needed cast to postgis.geography
- 07:59 PM Revision 13811: bugfix: schemas/util.sql: bounding_box(): must use postgis.geography (instead of postgis.geometry) because that handles geocoordinate wraparound correctly
- 07:52 PM Revision 13810: bugfix: schemas/util.sql: bounding_box(): need to explicitly set SRID to make sure the correct value is used
- 07:39 PM Revision 13809: bugfix: schemas/util.sql: bounding_box(): use st_makeenvelope() instead of st_makebox2d() because st_makebox2d() doesn't support geocoordinate wraparound (it is not SRID-aware)
- 07:32 PM Revision 13808: schemas/util.sql: new_world(): removed no longer needed cast to postgis.geometry
- 07:31 PM Revision 13807: schemas/util.sql: bounding_box(): return postgis.geometry instead of postgis.box2d because box2d is not directly used in postgis functions
- 07:12 PM Revision 13806: bugfix: /Makefile: mk_db: plpython3u: use CREATE EXTENSION instead of createlang so that the plpython3u extension is created in addition to the plpython3u language (as it is on Linux)
- 04:43 PM Revision 13805: schemas/util.sql: added new_world()
- 04:30 PM Revision 13804: bugfix: schemas/util.sql: bounding_box(): use util.range instead of numrange to support wraparound ranges for geocoordinates
- 04:26 PM Revision 13803: schemas/util.sql: range(numeric, numeric): use util.range instead of numrange to support wraparound ranges (which use a modulus system such as geocoordinates)
- 04:20 PM Revision 13802: schemas/util.sql: added `range` type (and cast to numrange), which allows wraparound ranges such as for geocoordinates
- 03:56 PM Revision 13801: schemas/util.sql: bounding_box(): documented usage
- 03:53 PM Revision 13800: schemas/public_.sql, inputs/.TNRS/schema.sql: upgraded to Postgres 9.3.4 format, which removes trailing " "
- 03:52 PM Revision 13799: schemas/util.sql: added bounding_box()
- 03:41 PM Revision 13798: bugfix: /Makefile: postgis schema: also need to make it accessible to non-superusers
- 03:12 PM Revision 13797: /Makefile, /README.TXT: PostgreSQL: Mac OS X: switched to using homebrew, which supports PostGIS
- 01:53 PM Task #917: TNRS: Instructions for new version with TPL
- all the taxon names have now been rescrubbed
- 01:40 PM Revision 13796: backups/TNRS.backup.md5: updated
06/17/2014
- 05:25 PM Task #935 (Resolved): TNRS crashes when scrubbing names with two infraspecific epithets
- iPlant fixed this
- 04:48 PM Revision 13795: schemas/util.sql: range(numeric, numeric): use simpler numrange() constructor instead of text input syntax
- 04:39 PM Revision 13794: schemas/util.sql: operator functions: renamed to plain-text names to follow the Postgres convention of making the function name be the human-readable name of the operator
- 04:26 PM Revision 13793: fix: bin/in_place: usage: removed duplicate copy of [preserve_mtime=1]
- 04:22 PM Revision 13792: /Makefile: removed no longer used nonApacheOnPort80 code
- 04:13 PM Revision 13791: bugfix: /Makefile: phppgadmin-Linux: don't change the Apache port if something "else" is running on port 80, because this feature was only needed when tomcat was running on that port, but is now buggy
06/16/2014
- 06:50 PM Revision 13790: schemas/util.sql: added ~ operator for ranges
- 04:35 PM Revision 13789: /Makefile: mk_db: added PostGIS so it can be used by our functions
- 04:34 PM Revision 13788: bugfix: /Makefile: postgres-Linux: postgis: updated apt-get target names for Ubuntu 14.04
- 03:52 PM Revision 13787: **/LICENSE.TXT: use lowercase when there is a single set of *__*
- 03:50 PM Revision 13786: /LICENSE.TXT: refers to: removed trunk/ because of course it also applies to the branches/ and tags/ subdirs
- 02:13 PM Revision 13785: schemas/public_.sql: 2014-6-12.Jeff_Ott.climatic_range_determinants: allow geovalid_bien = NULL to enable filtering New World status by coordinates rather than country
- 02:03 PM Revision 13784: schemas/public_.sql: 2014-6-12.Jeff_Ott.climatic_range_determinants: added geovalid_bien
Also available in: Atom