Activity
From 09/04/2014 to 10/03/2014
10/03/2014
- 05:56 PM Revision 14801: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtLevel: decreased to 5% to allow working longer on UPS power
- 05:53 PM Revision 14800: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtMinutesLeft: decreased to 3 min to allow working longer on UPS power
- 05:36 PM Revision 14799: bugfix: inputs/CVS/plot_/postprocess.sql: locality: site_name should come before directions_to_place because it is at a higher level of granularity
- 05:08 PM Revision 14798: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/run`: should just need to reinstall the geoscrub_output table
- 05:06 PM Revision 14797: schemas/public_.sql: geoscrub_input_view_modify(): add unique constraint on the same columns as geoscrub.geoscrub_output, as a sanity check to assist future debugging
- 05:04 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- 03:23 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- h3. error...
- 04:57 PM Revision 14796: bugfix: schemas/postgresql.conf: need to set extra_float_digits = 3 (the maximum) to enable verbatim export. this should fix issue #955.
- 03:21 PM Task #883 (Rejected): have import scripts regularly check disk space and pause processes if getting close to limit
- our scripts do not trigger this Postgres bug unless they are running buggy queries
- 03:20 PM Task #904 (Resolved): add MySQL public user to allow accessing the normalized VegCore data dictionary
- 12:37 PM Revision 14795: added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7.docx from Bob
- 08:18 AM Revision 14794: added config/dev_machine/com.apple.PowerManagement.plist from /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist , so that this file can be easily restored when the system spuriously resets settings in it
10/02/2014
10/01/2014
- 09:51 PM Revision 14792: fix: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/geoscrub_output/geoscrub.csv.run`: use a subshell to avoid closing screen on errexit
- 09:39 PM Revision 14791: fix: /README.TXT: Single datasource refresh: also back up DB after this
- 09:38 PM Revision 14790: /README.TXT: datasource removal: documented bien2_traits runtime
- 09:35 PM Revision 14789: added inputs/bien2_traits/_no_import since bien2_traits has been replaced by Cyrille_traits
- 09:32 PM Revision 14788: /README.TXT: added datasource removal instructions
- 09:22 PM Revision 14787: fix: /README.TXT: Single datasource refresh: place updated extract: generalized datasrc name
09/30/2014
- 07:45 AM Revision 14786: added inputs/Cyrille_traits/
- 07:36 AM Revision 14785: lib/sql_io.py: null_strs: added N/A and NA (this will remove a common abbr for North America, but we don't use the continent, so this is OK)
- 05:04 AM Revision 14784: mappings/VegCore.htm: regenerated from wiki. made verbatimLocality a synonym of locality since they are used to store the same data.
- 03:25 AM Revision 14783: bugfix: schemas/util.sql: col_rename(): also need to run util.name_truncate() on the names used in the "don't try to rename to itself" comparison
- 01:54 AM Revision 14782: fix: schemas/util.sql: col_rename(): don't try to rename a column to itself (this will cause an error)
- 01:22 AM Revision 14781: schemas/util.sql: set_col_names(): use util.col_rename() so that all renames uses this function
- 12:54 AM Revision 14780: schemas/util.sql: added col_rename()
- 12:02 AM Revision 14779: schemas/util.sql: added quote_ident() wrapper
- 12:01 AM Revision 14778: schemas/util.sql: added name_truncate()
09/26/2014
- 10:57 PM Revision 14777: fix: inputs/input.Makefile: $(nonHeaderSrcs): updated to exclude new header.txt
- 10:21 PM Revision 14776: lib/runscripts/table.run: added check_headers()
- 10:21 PM Revision 14775: bugfix: lib/runscripts/table.run: @srcs: missing [@]
- 10:08 PM Revision 14774: lib/runscripts/table.run: added header() and use it in header.txt()
- 09:50 PM Revision 14773: lib/runscripts/table.run: 1st_src(): use a variable for this instead, to avoid needing to run this function each time it's used, and to make @srcs available
- 09:22 PM Revision 14772: lib/sh/util.sh: wildcard.(): clarified that it only removes . .. when at then beginning of the list
- 07:10 PM Revision 14771: fix: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated revision
- 06:45 PM Revision 14770: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated (2 days)
- 06:32 PM Revision 14769: web/links/index.htm: updated to Firefox bookmarks: added bash shellshock links. LibreOffice: added steps to convert files from the command line.
- 02:31 AM Revision 14768: lib/runscripts/table.run: added header.txt()
- 02:31 AM Revision 14767: lib/runscripts/table.run: added 1st_src()
- 02:30 AM Revision 14766: inputs/input.Makefile: added %/list_srcs
- 02:30 AM Revision 14765: lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir
- 02:29 AM Revision 14764: lib/runscripts/subdir.run: added $datasrc_dir
- 02:29 AM Revision 14763: lib/sh/make.sh: make(): added support for $output_data mode which uses --silent
- 12:36 AM Revision 14762: fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting
09/25/2014
- 04:28 PM Revision 14761: fix: inputs/input.Makefile: need to escape $ in commands, including inside comments
- 03:28 PM Revision 14760: bugfix: lib/common.Makefile: $(wildcard/): need final pass with $(wildcard) to support inputs without wildcard chars
- 03:15 PM Revision 14759: bugfix: inputs/input.Makefile: `$(call add*,$(svnFiles))` must be invoked externally to clear the $(wildcard) cache before expanding $(svnFiles)
- 02:48 AM Revision 14758: /README.TXT: Installation: added instructions for inputs/download/live prompts
- 02:44 AM Revision 14757: /README.TXT: Installation: added instructions for MySQL installation prompts
- 02:18 AM Revision 14756: fix: /README.TXT: Installation: "Are you sure you want to continue connecting" prompt comes before "aaronmk@jupiter's password" prompt
- 02:13 AM Revision 14755: /Makefile: phppgadmin-Linux: removed no longer needed $(editApacheConfForPhppgadmin), as phpPgAdmin no longer uses this file (/etc/phppgadmin/apache.conf)
- 02:12 AM Revision 14754: /Makefile: phppgadmin-Linux: removed no longer needed $(editPhppgadminApacheConf)
- 01:58 AM Revision 14753: /Makefile: misc-Linux: added htop
- 01:58 AM Revision 14752: /Makefile: misc-Linux: added jedit
- 01:50 AM Revision 14751: /Makefile: postgres-Linux: added pgadmin3
- 01:45 AM Revision 14750: bugfix: /Makefile: $(postgresReload-*) calls: do not prepend $(MAKE) or - because these are multi-command functions
- 01:36 AM Revision 14749: bugfix: /Makefile: $(bien*Password): need to use = instead of := because these files are not available until config/download/live has run
09/24/2014
- 11:53 PM Revision 14748: fix: /README.TXT: Installation: added instructions for "aaronmk@jupiter's password" prompt
- 11:49 PM Revision 14747: /README.TXT: Installation: added step to open a terminal window
- 11:48 PM Revision 14746: fix: /README.TXT: Installation: added instructions for "Are you sure you want to continue connecting" prompt
- 11:34 PM Revision 14745: inputs/VegBank/run*.log: updated. this adds the function call context in addition to the function location.
- 11:28 PM Revision 14744: bugfix: /Makefile: $(uutils_install): need to exit after printing instructions instead of continuing
- 11:13 PM Revision 14743: fix: /README.TXT: Installation: need instructions for "reload PATH" prompt
- 11:10 PM Revision 14742: fix: /README.TXT: Installation: `apt-get install subversion`: need --yes
- 10:58 PM Revision 14741: bugfix: /Makefile: bin/install: uutils: need to reopen terminal window before can proceed with other installation steps
- 09:04 PM Revision 14740: /Makefile: python-Linux: removed no longer used installation of pp (Parallel Python). we use Python's multiprocessing package instead.
- 07:57 PM Revision 14739: bugfix: /Makefile: install: must be config/download/live, not config/download
- 07:52 PM Revision 14738: README.TXT: bin/sync_upload calls as root: don't need to set $USER because this is now set to the correct value in bin/sync_upload's scripts
- 07:50 PM Revision 14737: bugfix: lib/sh/local.sh: $sync_remote_url: jupiter user is always aaronmk, not the local user
- 07:41 PM Revision 14736: *{.sh,run}: use standard **WARNING** syntax for warning labels
- 07:32 PM Revision 14735: bugfix: /Makefile: bin/install: also need to install uutils
- 07:26 PM Revision 14734: *{.sh,run}: use standard **WARNING** syntax for warning labels
- 07:25 PM Revision 14733: fix: lib/sh/util.sh: verbosity_compat(): documented that this should not be run until right before executing an external command, so that it doesn't mess up the logging mechanism
- 07:23 PM Revision 14732: bugfix: bin/make: don't run verbosity_compat until right before executing the external command, so that it doesn't mess up the logging mechanism. this is run automatically by command(), so there is no need to do anything here. note that logging bugs like these can now be troubleshooted much more easily with pst() to narrow down which functions could be causing the problem.
- 06:55 PM Revision 14731: bugfix: lib/sh/util.sh: functions called by pst(): commented out/removed logging calls that would have caused infinite recursion when pst()'ing a logging function
- 06:38 PM Revision 14730: lib/sh/util.sh: echo_func(): get call context before wrapper(s), which is more useful for debugging. this uses skip_stack_frames()'s lookahead=1 mode.
- 06:37 PM Revision 14729: fix: lib/sh/util.sh: skip_stack_frames() callers: updated for new skip_stack_frames rather than get_stack_frame behavior
- 06:32 PM Revision 14728: lib/sh/util.sh: skip_stack_frames(): added lookahead support, which looks at entry *after* current to deterine whether to skip current. this is useful for skipping wrappers, by looking at the *calling* function's name.
- 06:30 PM Revision 14727: fix: lib/sh/util.sh: skip_stack_frames(): for new skip_stack_frames rather than get_stack_frame behavior, stack frames must be skipped in the caller to preserve the stack frame pointer
- 06:26 PM Revision 14726: lib/sh/util.sh: get_stack_frame_after(): renamed to skip_stack_frames() for clarity
- 06:25 PM Revision 14725: lib/sh/util.sh: added skip_stack_frame_in_caller, unskip_stack_frame_in_caller
- 06:24 PM Revision 14724: lib/sh/util.sh: added prev_stack_frame
- 07:04 AM Revision 14723: lib/sh/util.sh: echo_func(): use new format_stack_frame, which adds call context information to what was provided by func_loc
- 07:02 AM Revision 14722: fix: lib/sh/util.sh: format_stack_frame(): need to hide canon_rel_path() info using log+
- 07:01 AM Revision 14721: lib/sh/util.sh: added get_stack_frame_after()
- 06:59 AM Revision 14720: lib/sh/util.sh: added matches()
- 06:58 AM Revision 14719: lib/sh/util.sh: next_stack_frame: documented usage
- 04:24 AM Revision 14718: fix: lib/runscripts/util.run: runscript template: all(): moved example commands to target(), where they would more likely be located
- 02:35 AM Revision 14717: lib/sh/util.sh: format_stack_frame(): support including args
- 12:08 AM Revision 14716: lib/sh/util.sh: debugging: added pst() (print_stack_trace)
- 12:05 AM Revision 14715: lib/sh/util.sh: added stack_trace(), print_stack_trace()
- 12:04 AM Revision 14714: lib/sh/util.sh: added format_stack_frame()
- 12:03 AM Revision 14713: lib/sh/util.sh: added get_stack_frame() and helpers
09/23/2014
- 10:29 PM Revision 14712: lib/sh/util.sh: terminal: moved before errors so it can be used by it
- 10:25 PM Revision 14711: lib/sh/util.sh: errors, debugging: moved after datatype sections so their functions can use these
09/22/2014
- 05:45 PM Revision 14710: bugfix: lib/sh/util.sh: canon_rel_path() stub: proper no-op requires passing through original path
- 05:42 PM Revision 14709: lib/sh/util.sh: canon_rel_path(): fall back to original path if can't resolve, instead of errexiting
- 05:39 PM Revision 14708: bugfix: lib/sh/util.sh: canon_rel_path(): don't re-localize $path because this clears it
- 05:36 PM Revision 14707: lib/sh/util.sh: canon_rel_path(): import $1 to $path before function body, so that the function body can be moved to a nested function
- 03:48 PM Revision 14706: lib/sh/util.sh: added canon_rel_path() stub for use by debugging functions
- 03:39 PM Revision 14705: bugfix: /Makefile: install: run bin/install first since config/download depends on it
- 03:36 PM Revision 14704: lib/sh/util.sh: moved func_loc() to before debugging section so it can be used by debugging functions
09/20/2014
09/19/2014
- 03:51 PM Revision 14702: bugfix: /Makefile: install: also need to run config/download to get the necessary passwords
- 03:43 PM Revision 14701: config/Makefile: added download target
- 03:11 PM Revision 14700: /Makefile: config: renamed to bin/install because config is the name of a directory
- 02:53 PM Revision 14699: /README.TXT: Installation: `make install`: added instructions for what to do at each prompt
- 02:25 PM Revision 14698: /README.TXT: Installation: Check out svn: need to install svn first as this does not come preinstalled on Ubuntu
09/18/2014
09/17/2014
- 09:37 AM Bug #953 (New): CVS and VegBank data
- Martha,
Here is a rehash of issues with CVS. Several of these are likely to apply to Vegbank as well. I view th...
09/16/2014
- 05:35 PM Revision 14696: /README.TXT: to back up the local machine's hard drive: just do a single pass, to avoid the numerous different steps
- 05:25 PM Revision 14695: backups/*retention_policy*: on jupiter: changed to avoid retaining backups further back, as these consume significant disk space on jupiter and are not useful for anything
- 05:12 PM Task #952 (New): refresh GBIF so it can be published
- see [[conditions of use#GBIF|GBIF conditions of use]] > new portal
- 04:25 PM Revision 14694: fix: lib/sh/local.sh: $sync_remote_url: need $USER so user can be overridden when running as root
- 04:07 PM Revision 14693: fix: /README.TXT: to back up the local machine's settings: to jupiter: need `sudo -E` for new Documents/BIEN/vegbiendev*/
- 03:47 PM Revision 14692: /README.TXT: backups of ~/.dropbox/: just pause and resume Dropbox instead of quitting and restarting it
- 03:40 PM Revision 14691: /README.TXT: to back up the local machine's settings: to Dropbox: exclude gmvault-db/ to save space in Dropbox
- 03:36 PM Revision 14690: /README.TXT: to back up the local machine's settings: don't need to exclude ~/Library/Thunderbird/Profiles/9oo8rcyn.default/global-messages-db.sqlite because the disk that was corrupting this file has been replaced (#907)
- 03:33 PM Revision 14689: /README.TXT: to back up e-mails: don't need to also sync aaronmk@nceas.ucsb.edu because these e-mails are also in aaronmk.nceas@gmail.com (auto-forwarded)
- 03:16 PM Revision 14688: /README.TXT: Notes on system stability: removed no longer applicable warning about system upgrades, which is now incorrect because the disk space overrun bug (#887) was found *not* to have been caused by a system upgrade
- 02:57 PM Revision 14687: web/links/index.htm: updated to Firefox bookmarks: Roundtable: added links for Roundtable on "obstacles faced by researchers who reuse, share and manage data, and strategies for overcoming them"
- 02:46 PM Revision 14686: lib/Firefox_bookmarks.reformat.csv: label page's self-description as such: also support quotations enclosed in '
09/10/2014
- 11:07 PM Revision 14685: fix: exports/native_status_resolver.csv.run: added `users_by_name=1` to preserve the file group
- 11:00 PM Revision 14684: added inputs/VegBank/run.call_graph.log
- 05:45 PM Task #907 (Resolved): troubleshoot rsync verification errors
- this problem no longer occurs after the Time Machine restore, so it was most likely a disk corruption issue with the ...
- 05:03 PM Revision 14683: inputs/VegBank/run.log: updated for echo_vars() changes. the PG* vars, which contain important information, will now not need to be filtered out.
- 04:48 PM Revision 14682: lib/sh/util.sh: echo_vars(): merge repeated flags so there aren't flags in between the vars (which is also not valid declare syntax)
- 04:15 PM Revision 14681: lib/sh/db.sh: pg_cmd(): log vars on same line to avoid clutter
- 04:10 PM Task #887: fix disk space leak that fills the disk and crashes the import
- not fixing this because we fixed the bug in our code that was triggering this Postgres bug
- 03:58 PM Task #476 (New): develop map spreadsheet -> header override file translation utility
- multiple output locations for the same input column are now handled by source-specific derived columns, so this is no...
- 03:56 PM Task #483 (Resolved): rename staging table columns according to map.csv
- this was later implemented in a way that avoids needing to reinstall the staging tables
- 03:49 PM Task #951 (Resolved): test that `make install` is able to fully re-create vegbiendev
- h3. steps
# update vegbiendev:... - 03:17 PM Revision 14680: lib/sh/util.sh: echo_vars(): put all the vars on the same line so they don't clutter up the call graph generated at the default verbosity
- 01:56 PM Revision 14679: web/links/index.htm: updated to Firefox bookmarks: Mac: added link for VNC client. extended attributes: added link for chflags.
- 12:52 PM Revision 14678: added planning/workflow/staging_tables_installation_for_SQL_datasource.odg.src.log
- 12:51 PM Revision 14677: added inputs/VegBank/run.log
- 12:49 PM Revision 14676: fix: inputs/input.Makefile: $(svnFilesGlob): *.log should be in both the subdirs and the main dir
- 12:48 PM Revision 14675: inputs/input.Makefile: $(svnFilesGlob): *.log
09/09/2014
09/08/2014
- 04:09 PM Revision 14673: schemas/VegBIEN/data_dictionary/VegBIEN data dictionary.xlsx: updated
- 04:01 PM Revision 14672: bugfix: schemas/public_.sql: view_full_occurrence_individual_view and related views: synced to data dictionary spreadsheet, which adds back the links to the definitions (which used to be part of the column name itself)
- 03:50 PM Revision 14671: fix: schemas/public_.sql: analytical_plot, analytical_specimen: updated column names to be the same as analytical_stem, which these are a subset of
09/05/2014
- 10:51 PM Revision 14670: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: avoid extraneous diffs when rsyncing: clarified the machines that the command should be run on
- 10:47 PM Revision 14669: web/links/index.htm: updated to Firefox bookmarks: removed broken favicons
- 10:45 PM Revision 14668: web/links/index.htm: updated to Firefox bookmarks: updated favicons
- 10:43 PM Revision 14667: fix: /README.TXT: to backup files not in Time Machine: need to use 2 TB external hard drive instead of Time Machine drive because Time Machine drive does not have ~/Documents/BIEN/ in a location where it can be hardlinked against
- 10:02 PM Revision 14666: web/links/index.htm: updated to Firefox bookmarks: categorized uncategorized bookmarks
- 10:00 PM Revision 14665: web/links/index.htm: updated to Firefox bookmarks: updated favicons
- 09:55 PM Revision 14664: web/links/index.htm: updated to Firefox bookmarks: local machine phpPgAdmin: removed this so the Mac won't get woken up on network access whenever someone opens the links page, which attempts to load the favicon from the local machine. the previous solution of manually deleting the favicon (r13406) doesn't work because the favicon will just get re-added whenever this bookmark is visited.
- 09:37 PM Revision 14663: web/links/index.htm: updated to Firefox bookmarks: find: added instructions for searching by <>, not just =
- 08:48 PM Revision 14662: /README.TXT: Datasource setup: For MS Access databases: added that one should use the settings in the associated .ini file where available
- 08:46 PM Revision 14661: /README.TXT: Datasource setup: For MS Access databases: program link: added page subsections
- 07:44 PM Revision 14660: /README.TXT: to backup files not in Time Machine: note that Time Machine dereferences hard links: added commands documenting that this is the case
- 05:12 PM Revision 14659: fix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: added mkdir for Postgres
- 05:11 PM Revision 14658: bugfix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: mkdir: need sudo
- 05:07 PM Revision 14657: /README.TXT: to backup files not in Time Machine: moved to root/ subdir to group the multiple top-level dirs together
- 04:53 PM Revision 14656: /README.TXT: to backup files not in Time Machine: added the vegbiendev archival backups, which cannot be backed up by Time Machine because it dereferences hard links
- 04:51 PM Bug #950 (Resolved): fix view_full_occurrence_individual_view rows with is_geovalid NULL
- h3. test case
eg. this happens for CVS rows:
|datasource|country|state_province|county|latitude|longitude|is_ge... - 04:12 PM Revision 14655: /README.TXT: to backup files not in Time Machine: documented why Postgres cannot be backed up by Time Machine
09/04/2014
- 11:52 AM Revision 14654: /README.TXT: Single datasource refresh: added steps to place the updated extract and extracted flat file(s)
- 11:48 AM Revision 14653: /README.TXT: Single datasource refresh: connect to vegbiendev first, even though steps before it have their own step to do this
- 11:47 AM Revision 14652: /README.TXT: Single datasource refresh: reimport_scrub: added step to view progress
- 11:45 AM Revision 14651: /README.TXT: Single datasource refresh: moved to top since these steps are performed more often
- 10:23 AM Revision 14650: added planning/workflow/BIEN data workflow-2_bje.png export
- 10:05 AM Revision 14649: planning/meetings/BIEN conference call availability.xlsx: updated
- 10:04 AM Bug #948 (Resolved): fix duplicated rows in view_full_occurrence_individual
- 10:03 AM Revision 14648: planning/workflow/BIEN data workflow-2_bje.pptx: updated with Martha's changes and changes during conference call
- 08:10 AM Revision 14647: bugfix: web/BIEN3/Redmine/.htaccess: subpath redirect: also redirect dirs, so that empty-subdir main-page redirects (eg. wiki.vegpath.org) work properly
- 07:55 AM Revision 14646: bugfix: web/BIEN3/Redmine/.htaccess: main page should continue to redirect to wiki, not Redmine project page
- 07:44 AM Revision 14645: schemas/public_.sql: *_view: re-ran *_view_modify(), which use the new non-blocking rematerialize_view()
- 07:41 AM Revision 14644: schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's and Martha's request (e-mails from Martha on 2014-8-12 at 17:37PT, and from Brian M on 2014-8-13 at 16:21PT). note that this change has already been made on vegbiendev.
- 07:21 AM Revision 14643: schemas/public_.sql: view_full_occurrence_individual: re-ran view_full_occurrence_individual_view_modify(), which uses the new non-blocking rematerialize_view()
- 07:20 AM Revision 14642: schemas/util.sql: rematerialize_view(): made it non-blocking, so that it would allow full access to the original materialized table during the operation
- 07:11 AM Revision 14641: schemas/util.sql: added identifier_replace()
- 07:08 AM Revision 14640: schemas/util.sql: added relation_replace()
- 01:50 AM Revision 14639: /README.TXT: Single datasource import: renamed to Single datasource refresh since it works on existing datasources
- 01:45 AM Revision 14638: /README.TXT: Single datasource import: also need to reload staging tables
- 01:42 AM Revision 14637: /README.TXT: Single datasource import: added steps to re-run geoscrubbing and back up the vegbiendev database
Also available in: Atom