Project

General

Profile

Activity

From 08/04/2014 to 09/02/2014

08/29/2014

03:55 PM Revision 14633: added inputs/CVS/verify/Review of CVS data in BIEN3.docx
Aaron Marcuse-Kubitza
12:40 AM Revision 14632: backups/*retention_policy*: added explanations
Aaron Marcuse-Kubitza
12:39 AM Revision 14631: backups/*retention_policy*: on jupiter: backups further back: removed "if disk space permits" because this is already labeled "optionally"
Aaron Marcuse-Kubitza
12:38 AM Revision 14630: backups/*retention_policy*: changed to require retaining *.backup of the last 2 successful imports on all machines
Aaron Marcuse-Kubitza
12:25 AM Revision 14629: backups/*retention_policy*: allow keeping *.backup of the last 2 successful imports on all machines, not just jupiter
Aaron Marcuse-Kubitza
12:17 AM Revision 14628: **: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE** since one might not see the **SAVE** file in it
Aaron Marcuse-Kubitza
12:13 AM Revision 14627: **: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE** since one might not see the **SAVE** file in it
Aaron Marcuse-Kubitza
12:09 AM Revision 14626: **/"**DO_NOT_DELETE**": renamed to shorter **SAVE**
Aaron Marcuse-Kubitza
12:04 AM Revision 14625: added backups/*retention_policies*/ with retention policy files for each partition
Aaron Marcuse-Kubitza

08/28/2014

11:58 PM Revision 14624: backups/README.TXT: renamed to *retention_policy* to match the naming convention of the retention policy files in the various partitions
Aaron Marcuse-Kubitza
11:42 PM Revision 14623: /README.TXT: to back up the local machine's hard drive: also exclude *-files indicating the (differing) retention statuses of the partitions involved
Aaron Marcuse-Kubitza
08:13 PM Revision 14622: lib/tnrs.py single_tnrs_request(), bin/tnrs_client: use_tnrs_export: default to False because this mode uses incorrect selected matches (vegpath.org/issues/943), and the JSON mode that fixes this is now available
Aaron Marcuse-Kubitza
08:05 PM Revision 14621: bin/tnrs_db: tnrs.tnrs_request() call: explicitly set use_tnrs_export=True so that this continues to work if the default value is changed
Aaron Marcuse-Kubitza
07:57 PM Revision 14620: bugfix: lib/csvs.py: JsonReader: need to pass col_order to row_dict_to_list_reader
Aaron Marcuse-Kubitza
07:43 PM Revision 14619: config/VirtualBox_VMs/vegbiendev/README.TXT: ~/Documents/BIEN/vegbiendev.2014-2-2_1-07-32PT.+VirtualBox_changes/: renamed to vegbiendev.2014-2-2_1-07-32PT.VirtualBox/ to make clear that this is the VirtualBox version of vegbiendev
Aaron Marcuse-Kubitza
07:12 PM Revision 14618: bugfix: lib/tnrs.py: JSON output: need to stringify arrays so they match what is output in TSV-export mode
Aaron Marcuse-Kubitza
07:10 PM Revision 14617: lib/csvs.py: JsonReader: added support for values that are arrays
Aaron Marcuse-Kubitza
07:05 PM Revision 14616: lib/csvs.py: MultiFilter: inherit from WrapReader instead of Filter to avoid needing to define a no-op filter_() function
Aaron Marcuse-Kubitza
06:49 PM Revision 14615: bugfix: lib/csvs.py: row_dict_to_list_reader: need to override next() directly instead of just using Filter, because Filter doesn't support returning multiple rows for one input row (in this case, prepending a header row). this caused the 1st data row to be missing.
Aaron Marcuse-Kubitza
06:47 PM Revision 14614: lib/csvs.py: Filter: inherit from WrapReader, which separates out the CSV-reader API code
Aaron Marcuse-Kubitza
06:43 PM Revision 14613: lib/csvs.py: added WrapReader
Aaron Marcuse-Kubitza
06:43 PM Revision 14612: lib/csvs.py: added Reader
Aaron Marcuse-Kubitza
06:00 PM Revision 14611: schemas/public_.sql: views that use view_full_occurrence_individual_view: use the view_full_occurrence_individual table instead, now that this is materialized.
Aaron Marcuse-Kubitza
05:58 PM Revision 14610: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza
08:57 AM Revision 14609: /README.TXT: to back up the local machine's hard drive: renamed backup partition to BIEN3 to make clear what the backup drive contains
Aaron Marcuse-Kubitza
08:54 AM Revision 14608: fix: /README.TXT: to back up the local machine's hard drive: updated location of `screen` for added commands
Aaron Marcuse-Kubitza
08:53 AM Revision 14607: /README.TXT: added trailing / on dirs to make clear that they're dirs
Aaron Marcuse-Kubitza
08:40 AM Revision 14606: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to configure the VM to support VirtualBox
Aaron Marcuse-Kubitza
08:22 AM Revision 14605: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the contents of the VM, with the VirtualBox changes added
Aaron Marcuse-Kubitza
07:47 AM Revision 14604: config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the original contents of the backup from the VM: added steps to restore the correct VM snapshot
Aaron Marcuse-Kubitza
07:40 AM Revision 14603: config/VirtualBox_VMs/vegbiendev/README.TXT: also generate list of all the files whose permissions were changed since the backup, but which are extracted with their changed permissions instead of their original ones in the backup
Aaron Marcuse-Kubitza
07:05 AM Revision 14602: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the original contents of the backup from the VM
Aaron Marcuse-Kubitza
05:47 AM Revision 14601: fix: /README.TXT: to back up vegbiendev: also back up /home/aaronmk/bien/ (instead of just symlinking to the local copy), since this can be done space-efficiently with hardlinks. this ensures that the vegbiendev backup will not be modified when the local copy of bien/ is.
Aaron Marcuse-Kubitza
03:10 AM Revision 14600: lib/csvs.py: JsonReader: factored out row-dict-to-list into new row_dict_to_list_reader so that JSON-specific preprocessing is kept separate from the row format translation
Aaron Marcuse-Kubitza

08/27/2014

03:17 PM Revision 14599: lib/csvs.py: added MultiFilter, which enables applying multiple filters by nesting
Aaron Marcuse-Kubitza

08/26/2014

07:57 PM Revision 14598: lib/tnrs.py: single_tnrs_request(): JSON mode: implemented output of JSON data
Aaron Marcuse-Kubitza
07:53 PM Revision 14597: lib/tnrs.py: single_tnrs_request(): factored out wrapping in TnrsOutputStream, since this is done for both modes
Aaron Marcuse-Kubitza
07:47 PM Revision 14596: fix: lib/tnrs.py: JSON mode: TSV export columns: need to translate these to JSON column names before they can be used with the JSON data
Aaron Marcuse-Kubitza
07:44 PM Revision 14595: lib/csvs.py: added JsonReader, which reads parsed JSON data as row tuples
Aaron Marcuse-Kubitza
07:43 PM Revision 14594: lib/csvs.py: added row_dict_to_list(), which translates a CSV dict-based row to a list-based one
Aaron Marcuse-Kubitza
07:43 PM Revision 14593: lib/csvs.py: RowNumFilter: added support for filtering the header row as well
Aaron Marcuse-Kubitza
07:42 PM Revision 14592: lib/csvs.py: ColInsertFilter: added support for filtering the header row as well
Aaron Marcuse-Kubitza
05:12 PM Revision 14591: lib/csvs.py: InputRewriter: documented that this is also a stream (in addition to inheriting from StreamFilter)
Aaron Marcuse-Kubitza
05:11 PM Revision 14590: bugfix: lib/csvs.py: InputRewriter: accept a reader, as would be expected, instead of a custom stream whose lines are tuples
Aaron Marcuse-Kubitza
05:08 PM Revision 14589: fix: lib/sql_io.py: append_csv(): use new csvs.ProgressInputFilter instead of streams.ProgressInputStream(csvs.StreamFilter(__)), so that the input to csvs.InputRewriter is a reader, not a stream. this avoids the need for csvs.InputRewriter to accept a stream whose lines are tuples, instead of the expected reader.
Aaron Marcuse-Kubitza
05:02 PM Revision 14588: bugfix: inputs/input.Makefile: %/install: $(exportHeader) must come before postprocess because postprocess renames columns
Aaron Marcuse-Kubitza
04:50 PM Revision 14587: exports/: svn:ignore: added *.gz
Aaron Marcuse-Kubitza
04:49 PM Revision 14586: lib/csvs.py: added ProgressInputFilter, analogous to streams.ProgressInputStream
Aaron Marcuse-Kubitza
04:46 PM Revision 14585: lib/sql_io.py: added commented-out debug statement used to troubleshoot copy_expert() errors
Aaron Marcuse-Kubitza
04:45 PM Revision 14584: lib/dicts.py: added pair_keys(), pair_values()
Aaron Marcuse-Kubitza
04:15 PM Revision 14583: bugfix: lib/streams.py: CaptureStream: end_idx must also be > start_idx
Aaron Marcuse-Kubitza
04:07 PM Revision 14582: bugfix: inputs/input.Makefile: $(import_install_): need `set -o pipefail` to enable errexit
Aaron Marcuse-Kubitza
03:47 AM Revision 14581: /README.TXT: to backup files not in Time Machine: don't need to review diff because command is unidirectional
Aaron Marcuse-Kubitza
02:59 AM Revision 14580: fix: /README.TXT: to back up the local machine's hard drive: "repeat until only minimal changes" should refer to the first sync command
Aaron Marcuse-Kubitza
02:52 AM Revision 14579: inputs/.geoscrub/geoscrub_output/run: documented postprocess() rm=1 runtime (6 min)
Aaron Marcuse-Kubitza

08/25/2014

10:17 PM Revision 14578: lib/tnrs.py: single_tnrs_request(): use_tnrs_export=False: need to obtain export columns
Aaron Marcuse-Kubitza
10:16 PM Revision 14577: lib/csvs.py: added header(stream)
Aaron Marcuse-Kubitza
10:16 PM Revision 14576: fix: lib/tnrs.py: single_tnrs_request(): need to `assert name_ct >= 1`, because with no names, TNRS hangs indefinitely
Aaron Marcuse-Kubitza
09:13 PM Revision 14575: bin/tnrs_client: added env var to configure use_tnrs_export
Aaron Marcuse-Kubitza
08:18 PM Revision 14574: /README.TXT: to back up vegbiendev: use inplace=1 to speed stopping and resuming transfer
Aaron Marcuse-Kubitza
07:54 PM Revision 14573: fix: /README.TXT: to back up the local machine's hard drive: removed --extended-attributes (after initial sync) because rsync apparently has to visit every file for this
Aaron Marcuse-Kubitza
07:35 PM Revision 14572: fix: /README.TXT: to back up the local machine's hard drive: also need --extended-attributes
Aaron Marcuse-Kubitza
07:34 PM Revision 14571: /README.TXT: to back up the local machine's hard drive: removed --delete-before now that that partition has been expanded
Aaron Marcuse-Kubitza
07:16 PM Revision 14570: fix: /README.TXT: to back up vegbiendev: exclude /var/lib/mysql.bak,postgresql.bak because the local machine doesn't need 2 copies of this information
Aaron Marcuse-Kubitza
07:05 PM Revision 14569: /README.TXT: to back up vegbiendev: removed no longer needed exclude of Dropbox subdir backup
Aaron Marcuse-Kubitza
06:58 PM Revision 14568: fix: /README.TXT: to back up vegbiendev: also need to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine" because /home/aaronmk/bien is not synced here
Aaron Marcuse-Kubitza
06:52 PM Revision 14567: bugfix: /README.TXT: to back up vegbiendev: need `overwrite=1`
Aaron Marcuse-Kubitza
06:47 PM Revision 14566: /README.TXT: to back up vegbiendev: removed no longer needed exclude of Dropbox subdir backup
Aaron Marcuse-Kubitza
06:46 PM Revision 14565: /README.TXT: to back up the version history: don't also need this on vegbiendev because it's already on jupiter and the local machine
Aaron Marcuse-Kubitza
06:43 PM Revision 14564: bugfix: /README.TXT: to back up vegbiendev: need to include Postgres config files
Aaron Marcuse-Kubitza
06:24 PM Revision 14563: /README.TXT: to back up the local machine's hard drive: don't back up temp files: added /.fseventsd/
Aaron Marcuse-Kubitza
05:54 PM Revision 14562: fix: /README.TXT: to back up the local machine's hard drive: initial runtime: use range instead because some of the later runtime might have been from the same files
Aaron Marcuse-Kubitza
05:52 PM Revision 14561: /README.TXT: to back up the local machine's hard drive: updated initial runtime to include additional transferred files (17 h)
Aaron Marcuse-Kubitza
05:36 PM Revision 14560: fix: /README.TXT: to back up the local machine's hard drive: need to use --delete-before because the backup partition is near capacity
Aaron Marcuse-Kubitza
05:34 PM Revision 14559: /README.TXT: to back up the local machine's hard drive: don't back up temp files such as /private/var/vm/*
Aaron Marcuse-Kubitza
05:30 PM Revision 14558: fix: /README.TXT: to back up the local machine's hard drive: back up most Dropbox/Postgres files before stopping processes, to minimize downtime
Aaron Marcuse-Kubitza

08/21/2014

07:35 PM Revision 14557: bugfix: /README.TXT: to back up the local machine's hard drive: can't use ~ with --exclude
Aaron Marcuse-Kubitza
07:31 PM Revision 14556: fix: inputs/.geoscrub/geoscrub_output/postprocess.sql: map_geovalidity(): unscrubbable names should actually be geo*in*valid, not geovalid=NULL, according to Brad
Aaron Marcuse-Kubitza
07:24 PM Revision 14555: /README.TXT: to back up the local machine's hard drive: back up the non-Dropbox, non-Postgres files separately to minimize the Dropbox and Postgres downtime
Aaron Marcuse-Kubitza
06:03 PM Revision 14554: /README.TXT: to back up the vegbiendev databases: don't need to review diff for these as it's always unidirectional
Aaron Marcuse-Kubitza
05:55 PM Revision 14553: /README.TXT: added instructions to back up vegbiendev
Aaron Marcuse-Kubitza
05:12 PM Revision 14552: fix: /README.TXT: to back up the local machine's hard drive: also need to repeat backup command until only minimal changes
Aaron Marcuse-Kubitza
05:11 PM Revision 14551: /README.TXT: to back up the local machine's hard drive: added step to stop Postgres
Aaron Marcuse-Kubitza
05:10 PM Revision 14550: bugfix: /README.TXT: to back up the local machine's hard drive: also need to stop Dropbox
Aaron Marcuse-Kubitza
05:06 PM Revision 14549: /README.TXT: to back up the local machine's settings: added step to remove .DS_Store
Aaron Marcuse-Kubitza
04:47 PM Revision 14548: fix: /README.TXT: to back up the local machine's settings: Dropbox: shoudl not run with `del=`, because the backup should be an exact replica
Aaron Marcuse-Kubitza
04:25 PM Revision 14547: backups/TNRS.*: removed no longer needed old TNRS backups, which are part of the respective full-database backups in any case
Aaron Marcuse-Kubitza
02:57 PM Revision 14546: added config/phpMyAdmin/ symlink to schemas/VegCore/phpMyAdmin/
Aaron Marcuse-Kubitza
12:40 PM Revision 14545: bugfix: lib/sh/archives.sh: compress(): don't include dir prefix in zip archive
Aaron Marcuse-Kubitza
12:40 PM Revision 14544: lib/sh/util.sh: cd(): use echo_run instead of a manual echo_cmd call
Aaron Marcuse-Kubitza
12:35 PM Revision 14543: fix: lib/sh/util.sh: cd(): indent after running cd rather than before
Aaron Marcuse-Kubitza
12:32 PM Revision 14542: lib/sh/util.sh: cd(): support rebasing path vars for the new dir
Aaron Marcuse-Kubitza
11:51 AM Revision 14541: bugfix: lib/sh/archives.sh: compress(): need to use zip's path syntax to avoid the file in the archive being named "-"
Aaron Marcuse-Kubitza
08:56 AM Revision 14540: lib/tnrs.py: added option to avoid using TNRS's TSV export feature, which currently returns incorrect selected matches (vegpath.org/issues/943). this has been implemented up through the GWT/JSON decoding.
Aaron Marcuse-Kubitza
08:50 AM Revision 14539: lib/tnrs.py: added gwt_decode()
Aaron Marcuse-Kubitza
08:49 AM Revision 14538: lib/strings.py: added unesc_quotes() and helper functions
Aaron Marcuse-Kubitza
08:49 AM Revision 14537: lib/strings.py: added json_decode()
Aaron Marcuse-Kubitza
08:38 AM Revision 14536: /README.TXT: To re-run geoscrubbing: updated runtimes
Aaron Marcuse-Kubitza
08:25 AM Revision 14535: exports/*_GBIF.csv.run: documented compress_() runtime (20 min-1 h)
Aaron Marcuse-Kubitza

08/20/2014

11:12 PM Revision 14534: lib/runscripts/extract.run: export_(): also compress created file
Aaron Marcuse-Kubitza
11:11 PM Revision 14533: lib/sh/archives.sh: added compress(), expand(), which handle compression of individual files
Aaron Marcuse-Kubitza
04:26 PM Revision 14532: bugfix: inputs/input.Makefile: sql/install: ";" for commands inside $(if) blocks need to be inside the $(if) block, too, because otherwise there will be dangling ";" without a statement (bash does not support empty statements containing just ";")
Aaron Marcuse-Kubitza
11:29 AM Revision 14531: /README.TXT: Full database import: converted database commands to command-line commands to make them easier to run
Aaron Marcuse-Kubitza
11:28 AM Revision 14530: /README.TXT: Full database import: converted database commands to command-line commands to make them easier to run
Aaron Marcuse-Kubitza
10:55 AM Revision 14529: web/links/index.htm: updated to Firefox bookmarks: added instructions for how to enable automatic restart on power loss for the UPS (which isn't accessible in the GUI)
Aaron Marcuse-Kubitza

08/19/2014

05:13 PM Revision 14528: fix: schemas/util.sql: contained_within_approx(point geocoord, region postgis.geography): use util.geography() instead of implicit cast to suppress "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" NOTICEs
Aaron Marcuse-Kubitza
05:10 PM Revision 14527: schemas/util.sql: added geography(util.geocoord), which suppresses "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" NOTICEs
Aaron Marcuse-Kubitza
04:48 PM Revision 14526: exports/native_status_resolver.csv.run: updated export_() runtime (5 min, now that we're using the narrower New World criterion)
Aaron Marcuse-Kubitza
04:39 PM Revision 14525: fix: schemas/public_.sql: native_status_resolver: don't include rows with New World coordinates that don't also have New World country names, since the NSR only uses the country name
Aaron Marcuse-Kubitza
04:26 PM Revision 14524: schemas/public_.sql: native_status_resolver: removed rows with is_geovalid NULL, at Brad's request. note that this removes valid rows with standardized country names.
Aaron Marcuse-Kubitza
04:01 PM Revision 14523: exports/native_status_resolver.csv.run: updated export_() runtime (30 min)
Aaron Marcuse-Kubitza
03:28 PM Revision 14522: fix: schemas/public_.sql: native_status_resolver: added country IS NOT NULL filter requested by Brad
Aaron Marcuse-Kubitza
02:35 PM Revision 14521: fix: schemas/public_.sql: native_status_resolver: remove the id because this prevents SELECT DISTINCT from having the desired effect. instead, the results will be joined back using the other columns.
Aaron Marcuse-Kubitza
02:25 PM Revision 14520: exports/native_status_resolver.csv.run: upload_(): documented runtime (2.5 min)
Aaron Marcuse-Kubitza
02:18 PM Revision 14519: bugfix: exports/native_status_resolver.csv.run: upload_(): $live must be exported
Aaron Marcuse-Kubitza
02:15 PM Revision 14518: exports/native_status_resolver.csv.run: upload_(): use `live=1` instead for consistency with other invocations of put
Aaron Marcuse-Kubitza
02:14 PM Revision 14517: fix: exports/native_status_resolver.csv.run: upload_(): need `l=1`
Aaron Marcuse-Kubitza
02:12 PM Revision 14516: exports/native_status_resolver.csv.run: documented export_() runtime (45 min)
Aaron Marcuse-Kubitza
01:48 PM Revision 14515: exports/native_status_resolver.csv.run: added upload_() to get the file onto nimoy
Aaron Marcuse-Kubitza
01:25 PM Revision 14514: added exports/native_status_resolver.csv.run
Aaron Marcuse-Kubitza
01:23 PM Revision 14513: schemas/public_.sql: added native_status_resolver view, requested by Brad (wiki.vegpath.org/Data_requests)
Aaron Marcuse-Kubitza
11:47 AM Revision 14512: inputs/publishable datasources.xlsx: udpated
Aaron Marcuse-Kubitza
08:37 AM Revision 14511: lib/tnrs.py: documentation about output of the retrieve step: added that this is also unusable because the array does not contain all the columns and contains no column names
Aaron Marcuse-Kubitza
07:54 AM Revision 14510: removed no longer used web/BIEN3/Redmine/main/. use Redmine/!__ instead.
Aaron Marcuse-Kubitza
07:52 AM Revision 14509: web/BIEN3/Redmine/issues/.htaccess: perform .. redirects using new ! prefix
Aaron Marcuse-Kubitza
07:51 AM Revision 14508: web/BIEN3/Redmine/.htaccess: enable redirects that avoid using a subdir's .htaccess
Aaron Marcuse-Kubitza
07:46 AM Revision 14507: web/BIEN3/Redmine/wiki/.htaccess: removed no longer needed ignore_fs, since the .htaccess does not have RewriteRules that would need this in a RewriteCond
Aaron Marcuse-Kubitza
07:42 AM Revision 14506: web/BIEN3/Redmine/issues/.htaccess: main issues page: added default filter conditions
Aaron Marcuse-Kubitza
07:41 AM Revision 14505: bugfix: web/BIEN3/Redmine/issues/.htaccess: need to redirect to separate URL for individual issues, because they are not located under the issues/ subdir in Redmine
Aaron Marcuse-Kubitza
06:35 AM Revision 14504: added web/.issues symlink and dest dir (needed because Apache does not support dangling symlinks)
Aaron Marcuse-Kubitza
06:35 AM Revision 14503: web/BIEN3/Redmine/wiki/.htaccess: documented that this dir is needed because Apache does not support dangling symlinks
Aaron Marcuse-Kubitza
06:32 AM Revision 14502: bugfix: web/.htaccess: need to expand top-level symlinks to avoid RewriteBase issues
Aaron Marcuse-Kubitza
06:29 AM Revision 14501: web/main.conf: added RewriteMap for readlink
Aaron Marcuse-Kubitza
06:28 AM Revision 14500: added web/readlink
Aaron Marcuse-Kubitza
05:52 AM Revision 14499: web/links/index.htm: updated to Firefox bookmarks: updated favicons
Aaron Marcuse-Kubitza
05:47 AM Revision 14498: web/links/index.htm: updated to Firefox bookmarks: updated favicons
Aaron Marcuse-Kubitza

08/18/2014

04:52 PM Revision 14497: web/BIEN3/Redmine/wiki/.htaccess: just use this dir as symlink dest, since the dir name is the same as the URL path within Redmine
Aaron Marcuse-Kubitza
04:50 PM Revision 14496: web/.htaccess: don't rewrite existing files/dirs: allow forcing rewrite of existing things with %{ENV:ignore_fs}
Aaron Marcuse-Kubitza
03:41 PM Revision 14495: web/BIEN3/Redmine/svn-web/.htaccess: use Redmine/ instead of main/ subdir
Aaron Marcuse-Kubitza
03:37 PM Revision 14494: web/BIEN3/Redmine/.htaccess: point this to the Redmine root instead of to the wiki, to avoid the need to append /main
Aaron Marcuse-Kubitza
03:21 PM Revision 14493: backups/vegbien.r14089.backup.md5: updated
Aaron Marcuse-Kubitza
01:58 PM Revision 14492: inputs/.TNRS/schema.sql: taxon_match: added taxon_scrub_best_match_jerry_lu index to facilitate finding names affected by the match-picking bug (#943)
Aaron Marcuse-Kubitza

08/17/2014

12:04 PM Revision 14491: backups/TNRS.backup.md5: updated
Aaron Marcuse-Kubitza
07:49 AM Revision 14490: fix: inputs/HVAA/Specimen/postprocess.sql, map.csv: monthCollected/dayCollected: fix indefinite dates (which aren't supported by Postgres), as decided by Bob (https://docs.google.com/spreadsheets/d/1PI8n0CRttN7ttsXs5qfh5OFFzSoAfJj0gSbylgX6vj4/edit#gid=0)
Aaron Marcuse-Kubitza
07:23 AM Revision 14489: schemas/util.sql: added date_part_fix()
Aaron Marcuse-Kubitza

08/16/2014

05:55 PM Revision 14488: schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's and Martha's request (e-mails from Martha on 2014-8-12 at 17:37PT, and from Brian M on 2014-8-13 at 16:21PT). note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
05:51 PM Revision 14487: schemas/public_.sql: added view_full_occurrence_individual_*GBIF, analytical_stem_*GBIF requested by Mark. note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
05:37 PM Revision 14486: schemas/VegBIEN/data_dictionary/VegBIEN data dictionary.xlsx: updated
Aaron Marcuse-Kubitza
05:15 PM Revision 14485: schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's request (e-mail from Martha on 2014-8-12 at 17:37PT). note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
05:11 PM Revision 14484: schemas/public_.sql: view_full_occurrence_individual, analytical_stem: renamed columns to shortened names at Brian M's request (e-mail from Martha on 2014-8-12 at 17:37PT). note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
02:58 PM Revision 14483: schemas/public_.sql: analytical_stem_view: renamed to analytical_stem since this replaces the materialized analytical_stem table. note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
02:35 PM Revision 14482: schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's request (e-mail from Martha on 2014-8-12 at 17:37PT). note that this change has already been made on vegbiendev.
Aaron Marcuse-Kubitza
01:20 PM Revision 14481: schemas/public_.sql: viewFullOccurrence_individual: re-ran viewFullOccurrence_individual_view_modify() to udpate this
Aaron Marcuse-Kubitza
01:15 PM Revision 14480: bugfix: schemas/util.sql: copy_struct(): also need to util.copy_comment() because CREATE TABLE __ INCLUDING ALL is missing this
Aaron Marcuse-Kubitza
01:11 PM Revision 14479: schemas/util.sql: added copy_comment(regclass, regclass)
Aaron Marcuse-Kubitza
12:29 PM Revision 14478: schemas/public_.sql: viewFullOccurrence_individual_view: CREATE INDEX runtime: documented 2-column runtime (~2 h)
Aaron Marcuse-Kubitza
12:06 PM Revision 14477: exports/analytical_stem_*GBIF.csv.run: documented runtime (40 min-1.2 h)
Aaron Marcuse-Kubitza
09:36 AM Revision 14476: exports/view_full_occurrence_individual_*GBIF.csv.run: documented runtime (20-30 min)
Aaron Marcuse-Kubitza
08:37 AM Revision 14475: added exports/analytical_stem_GBIF.csv.run, analytical_stem_non_GBIF.csv.run
Aaron Marcuse-Kubitza

08/15/2014

07:15 PM Revision 14474: added exports/view_full_occurrence_individual_GBIF.csv.run, view_full_occurrence_individual_non_GBIF.csv.run
Aaron Marcuse-Kubitza
04:13 PM Revision 14473: schemas/public_.sql: viewFullOccurrence_individual_view: added "after updating this" instructions
Aaron Marcuse-Kubitza
04:01 PM Revision 14472: schemas/public_.sql: viewFullOccurrence_individual_view: documented CREATE INDEX runtime (10 min - 1.5 h depending on the datatype and % populated)
Aaron Marcuse-Kubitza
03:32 PM Revision 14471: web/links/index.htm: updated to Firefox bookmarks: updated to use frenzy instead of starscream
Aaron Marcuse-Kubitza

08/14/2014

03:25 PM Revision 14470: fix: lib/tnrs.py: retrieval_request_template: source_sorting (Constrain by Source): corrected explanation to reflect that the behavior is actually the same in both modes, since only one match is ever marked as selected, and that match should always come first
Aaron Marcuse-Kubitza
10:50 AM Revision 14469: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza

08/11/2014

07:48 AM Revision 14468: inputs/.TNRS/schema.sql: taxon_match: added taxon_scrub_by_name index
Aaron Marcuse-Kubitza
07:46 AM Revision 14467: inputs/.TNRS/schema.sql: taxon_match: added taxon_scrub_by_family index
Aaron Marcuse-Kubitza
07:10 AM Revision 14466: inputs/.TNRS/schema.sql: taxon_match: added taxon_scrub_by_species_binomial index
Aaron Marcuse-Kubitza
05:28 AM Revision 14465: bugfix: schemas/public_.sql: *_view_modify(): need to set the search_path so that tables are created in the same schema as the function, rather than whichever schema happens to be at the beginning of the search_path
Aaron Marcuse-Kubitza
05:19 AM Revision 14464: bugfix: schemas/public_.sql: analytical_stem_view_modify(): updated to support being used directly instead of via a materialized table
Aaron Marcuse-Kubitza
05:13 AM Revision 14463: schemas/public_.sql: removed no longer used analytical_stem. use analytical_stem_view or viewFullOccurrence_individual instead.
Aaron Marcuse-Kubitza
05:08 AM Revision 14462: fix: schemas/public_.sql: viewfulloccurrence_individual_view_modify(): fixed capitalization in function name (needs "" to preserve case)
Aaron Marcuse-Kubitza
05:03 AM Revision 14461: schemas/public_.sql: other derived views: use viewFullOccurrence_individual instead of analytical_stem because analytical_stem is no longer materialized (viewFullOccurrence_individual_view is materialized instead)
Aaron Marcuse-Kubitza
05:01 AM Revision 14460: schemas/public_.sql: validation views: use analytical_stem_view instead of analytical_stem because analytical_stem is no longer materialized (viewFullOccurrence_individual_view is materialized instead)
Aaron Marcuse-Kubitza
04:57 AM Revision 14459: schemas/public_.sql: derived views: use analytical_stem_view instead of analytical_stem because analytical_stem is no longer materialized (viewFullOccurrence_individual_view is materialized instead)
Aaron Marcuse-Kubitza
04:46 AM Revision 14458: bugfix: schemas/util.sql: expansion_min_cols(): increased to properly handle all public-schema views (the largest is currently 2014-6-12.Jeff_Ott.climatic_range_determinants)
Aaron Marcuse-Kubitza
04:35 AM Revision 14457: schemas/util.sql: view_def_to_orig(): use new expansion_min_cols() for easier configuration
Aaron Marcuse-Kubitza
04:33 AM Revision 14456: schemas/util.sql: added expansion_min_cols() (the minimum # of cols from the same table to be treated as a * expression)
Aaron Marcuse-Kubitza
04:05 AM Revision 14455: fix: schemas/public_.sql: analytical_stem_view: regenerated columns list for viewFullOccurrence_individual (instead of viewFullOccurrence_individual_view)
Aaron Marcuse-Kubitza
03:40 AM Revision 14454: bugfix: mappings/VegCore-VegBIEN.csv: prefixed taxonomic ranks: use _concat_nullify() so that the prefix is only added if the epithet is non-NULL
Aaron Marcuse-Kubitza
03:29 AM Revision 14453: fix: schemas/util.sql: col_re(): support column names with " in them
Aaron Marcuse-Kubitza
02:59 AM Revision 14452: schemas/util.sql: added _concat_nullify(), which uses || instead of concat()
Aaron Marcuse-Kubitza
02:16 AM Revision 14451: bugfix: inputs/FIA/REF_RESEARCH_STATION/map.csv: mapped country, which is not provided in the FIA data
Aaron Marcuse-Kubitza
01:46 AM Revision 14450: schemas/public_.sql: removed no longer used threatened_taxonlabel_view. use iucn_red_list_view instead.
Aaron Marcuse-Kubitza
01:35 AM Revision 14449: schemas/public_.sql: viewFullOccurrence_individual_view: documented materialize time (22 h)
Aaron Marcuse-Kubitza

08/10/2014

05:53 AM Revision 14448: inputs/.TNRS/schema.sql: taxon_match: removed no longer used scrubbed_unique_taxon_name. the scrubbed name ranks are now generated from the other TNRS columns instead.
Aaron Marcuse-Kubitza
05:28 AM Revision 14447: /README.TXT: Full database import: added steps to re-run geoscrubbing
Aaron Marcuse-Kubitza
05:22 AM Revision 14446: exports/geoscrub_input.csv.run: export_(): updated runtime (30 s)
Aaron Marcuse-Kubitza
03:57 AM Revision 14445: fix: exports/viewFullOccurrence_*.csv.run: need to limit # rows to 5000 as requested by Bob
Aaron Marcuse-Kubitza
03:42 AM Revision 14444: schemas/public_.sql: viewFullOccurrence_*: documented materialize time
Aaron Marcuse-Kubitza

08/09/2014

10:23 PM Revision 14443: bin/make_analytical_db: materialize viewFullOccurrence_individual_view instead of analytical_stem_view because analytical_stem_view is now generatable via a simple join onto viewFullOccurrence_individual_view. this avoids running into potential disk space constraints when materializing and backing up both tables (~50 GB/table * 2 tables * 2 copies (incl. the backup) = 200 GB, which is very close to the available disk space).
Aaron Marcuse-Kubitza
10:17 PM Revision 14442: schemas/public_.sql: analytical_stem_view: use new materialized viewFullOccurrence_individual
Aaron Marcuse-Kubitza
10:08 PM Revision 14441: schemas/public_.sql: added viewFullOccurrence_individual by running viewFullOccurrence_individual_view_modify()
Aaron Marcuse-Kubitza
10:05 PM Revision 14440: schemas/public_.sql: added viewFullOccurrence_individual_view_modify(), analogous to analytical_stem_view_modify()
Aaron Marcuse-Kubitza
09:52 PM Revision 14439: bin/make_analytical_db: removed extra () around psql_verbose_vegbien
Aaron Marcuse-Kubitza
09:51 PM Revision 14438: bin/make_analytical_db: removed no longer used mk_table()
Aaron Marcuse-Kubitza
09:49 PM Revision 14437: bin/make_analytical_db: use more up-to-date *_view_modify() functions instead of mk_table()
Aaron Marcuse-Kubitza
08:46 PM Revision 14436: inputs/.TNRS/schema.sql: removed no longer used view ValidMatchedTaxon. use taxon_scrub instead.
Aaron Marcuse-Kubitza
08:44 PM Revision 14435: schemas/public_.sql: iucn_red_list_view: use taxon_scrub instead of ValidMatchedTaxon since they are equivalent
Aaron Marcuse-Kubitza
08:42 PM Revision 14434: inputs/.TNRS/schema.sql: taxon_scrub: use taxon_best_match directly, to avoid the need for a separate ValidMatchedTaxon view
Aaron Marcuse-Kubitza
08:25 PM Revision 14433: fix: inputs/.TNRS/schema.sql: taxon_scrub: merged synonymous columns
Aaron Marcuse-Kubitza
08:11 PM Revision 14432: schemas/vegbien.sql: taxon_scrub: documented steps to merge synonymous columns
Aaron Marcuse-Kubitza
07:45 PM Revision 14431: inputs/.TNRS/schema.sql: removed no longer used view MatchedTaxon. use taxon_best_match instead.
Aaron Marcuse-Kubitza
07:43 PM Revision 14430: inputs/.TNRS/schema.sql: ValidMatchedTaxon: use taxon_best_match now that it's equivalent to MatchedTaxon
Aaron Marcuse-Kubitza
07:38 PM Revision 14429: bugfix: schemas/public_.sql: tnrs_input_name: use taxon_match instead of taxon_best_match because there is no index on taxon_match that includes just the filters used by taxon_best_match
Aaron Marcuse-Kubitza
07:37 PM Revision 14428: schemas/public_.sql: tnrs_input_name: use taxon_best_match now that it's equivalent to MatchedTaxon
Aaron Marcuse-Kubitza
07:20 PM Revision 14427: backups/TNRS.backup.md5: updated
Aaron Marcuse-Kubitza
07:19 PM Revision 14426: fix: inputs/.TNRS/schema.sql: MatchedTaxon: merged synonymous columns
Aaron Marcuse-Kubitza
07:02 PM Revision 14425: fix: schemas/util.sql: view_is_subset_or_renaming(): views with CASE statements are also not subsets/renamings
Aaron Marcuse-Kubitza
06:52 PM Revision 14424: inputs/.TNRS/schema.sql: removed no longer used taxon_scrub.scrubbed_unique_taxon_name.* . use taxon_scrub instead.
Aaron Marcuse-Kubitza
06:50 PM Revision 14423: inputs/.TNRS/schema.sql: taxon_scrub: use taxon_match derived columns instead of the incorrect values in taxon_scrub.scrubbed_unique_taxon_name.* (which does not work with the multi-match strategy)
Aaron Marcuse-Kubitza
05:54 AM Revision 14422: inputs/.TNRS/schema.sql: MatchedTaxon: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.
Aaron Marcuse-Kubitza
05:27 AM Revision 14421: inputs/.TNRS/schema.sql: taxon_scrub: use derived columns from taxon_match. this also incorporates the fixes in the new derived columns.
Aaron Marcuse-Kubitza

08/08/2014

07:14 PM Revision 14420: added exports/viewFullOccurrence_{CVS,VegBank,NCU}.csv.run
Aaron Marcuse-Kubitza
07:13 PM Revision 14419: schemas/public_.sql: added viewFullOccurrence_{CVS,VegBank,NCU} for Bob
Aaron Marcuse-Kubitza
02:32 PM Revision 14418: bugfix: schemas/util.sql: added GRANT USAGE for bien_read, public_ so that util schema functions can be called by other users
Aaron Marcuse-Kubitza

08/05/2014

05:31 PM Revision 14417: inputs/.TNRS/schema.sql: taxon_match: to port derived column changes to vegbiendev: derived_cols_export() code: documented runtime (6 h)
Aaron Marcuse-Kubitza

08/04/2014

06:25 AM Revision 14416: bugfix: inputs/.TNRS/schema.sql: removed no longer used derived column __accepted_infraspecific_label, which had a buggy formula that broke derived_cols_populate()
Aaron Marcuse-Kubitza
06:18 AM Revision 14415: bugfix: schemas/util.sql: drop_derived_col(): also need to run util.derived_cols_trigger_update()
Aaron Marcuse-Kubitza
06:17 AM Revision 14414: bugfix: inputs/.TNRS/schema.sql: removed no longer used derived column __accepted_infraspecific_label, which had a buggy formula that broke derived_cols_populate()
Aaron Marcuse-Kubitza
06:10 AM Revision 14413: fix: inputs/.TNRS/schema.sql: taxon_match: to remove a column: updated instructions
Aaron Marcuse-Kubitza
05:09 AM Revision 14412: bugfix: lib/sh/util.sh: str2varname(): need to lowercase str because on case-insensitive filesystems, paths sometimes canonicalize to a different capitalization than the original
Aaron Marcuse-Kubitza
05:00 AM Revision 14411: lib/sh/util.sh: added lowercase()
Aaron Marcuse-Kubitza
 

Also available in: Atom