Project

General

Profile

Statistics
| Revision:
  • svn:ignore: extern

# Date Author Comment
14678 09/10/2014 12:52 PM Aaron Marcuse-Kubitza

added planning/workflow/staging_tables_installation_for_SQL_datasource.odg.src.log

14677 09/10/2014 12:51 PM Aaron Marcuse-Kubitza

added inputs/VegBank/run.log

14676 09/10/2014 12:49 PM Aaron Marcuse-Kubitza

fix: inputs/input.Makefile: $(svnFilesGlob): *.log should be in both the subdirs and the main dir

14675 09/10/2014 12:48 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(svnFilesGlob): *.log

14674 09/09/2014 06:43 PM Aaron Marcuse-Kubitza

inputs/Makefile: install: install an empty VegBIEN schema instead of all the datasources, at Mark's request. this enables loading just a single datasource.

14673 09/08/2014 04:09 PM Aaron Marcuse-Kubitza

schemas/VegBIEN/data_dictionary/VegBIEN data dictionary.xlsx: updated

14672 09/08/2014 04:01 PM Aaron Marcuse-Kubitza

bugfix: schemas/public_.sql: view_full_occurrence_individual_view and related views: synced to data dictionary spreadsheet, which adds back the links to the definitions (which used to be part of the column name itself)

14671 09/08/2014 03:50 PM Aaron Marcuse-Kubitza

fix: schemas/public_.sql: analytical_plot, analytical_specimen: updated column names to be the same as analytical_stem, which these are a subset of

14670 09/05/2014 10:51 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: avoid extraneous diffs when rsyncing: clarified the machines that the command should be run on

14669 09/05/2014 10:47 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: removed broken favicons

14668 09/05/2014 10:45 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: updated favicons

14667 09/05/2014 10:43 PM Aaron Marcuse-Kubitza

fix: /README.TXT: to backup files not in Time Machine: need to use 2 TB external hard drive instead of Time Machine drive because Time Machine drive does not have ~/Documents/BIEN/ in a location where it can be hardlinked against

14666 09/05/2014 10:02 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: categorized uncategorized bookmarks

14665 09/05/2014 10:00 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: updated favicons

14664 09/05/2014 09:55 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: local machine phpPgAdmin: removed this so the Mac won't get woken up on network access whenever someone opens the links page, which attempts to load the favicon from the local machine. the previous solution of manually deleting the favicon (r13406) doesn't work because the favicon will just get re-added whenever this bookmark is visited.

14663 09/05/2014 09:37 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks: find: added instructions for searching by <>, not just =

14662 09/05/2014 08:48 PM Aaron Marcuse-Kubitza

/README.TXT: Datasource setup: For MS Access databases: added that one should use the settings in the associated .ini file where available

14661 09/05/2014 08:46 PM Aaron Marcuse-Kubitza

/README.TXT: Datasource setup: For MS Access databases: program link: added page subsections

14660 09/05/2014 07:44 PM Aaron Marcuse-Kubitza

/README.TXT: to backup files not in Time Machine: note that Time Machine dereferences hard links: added commands documenting that this is the case

14659 09/05/2014 05:12 PM Aaron Marcuse-Kubitza

fix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: added mkdir for Postgres

14658 09/05/2014 05:11 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: mkdir: need sudo

14657 09/05/2014 05:07 PM Aaron Marcuse-Kubitza

/README.TXT: to backup files not in Time Machine: moved to root/ subdir to group the multiple top-level dirs together

14656 09/05/2014 04:53 PM Aaron Marcuse-Kubitza

/README.TXT: to backup files not in Time Machine: added the vegbiendev archival backups, which cannot be backed up by Time Machine because it dereferences hard links

14655 09/05/2014 04:12 PM Aaron Marcuse-Kubitza

/README.TXT: to backup files not in Time Machine: documented why Postgres cannot be backed up by Time Machine

14654 09/04/2014 11:52 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource refresh: added steps to place the updated extract and extracted flat file(s)

14653 09/04/2014 11:48 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource refresh: connect to vegbiendev first, even though steps before it have their own step to do this

14652 09/04/2014 11:47 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource refresh: reimport_scrub: added step to view progress

14651 09/04/2014 11:45 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource refresh: moved to top since these steps are performed more often

14650 09/04/2014 10:23 AM Aaron Marcuse-Kubitza

added planning/workflow/BIEN data workflow-2_bje.png export

14649 09/04/2014 10:05 AM Aaron Marcuse-Kubitza

planning/meetings/BIEN conference call availability.xlsx: updated

14648 09/04/2014 10:03 AM Aaron Marcuse-Kubitza

planning/workflow/BIEN data workflow-2_bje.pptx: updated with Martha's changes and changes during conference call

14647 09/04/2014 08:10 AM Aaron Marcuse-Kubitza

bugfix: web/BIEN3/Redmine/.htaccess: subpath redirect: also redirect dirs, so that empty-subdir main-page redirects (eg. wiki.vegpath.org) work properly

14646 09/04/2014 07:55 AM Aaron Marcuse-Kubitza

bugfix: web/BIEN3/Redmine/.htaccess: main page should continue to redirect to wiki, not Redmine project page

14645 09/04/2014 07:44 AM Aaron Marcuse-Kubitza

schemas/public_.sql: *_view: re-ran *_view_modify(), which use the new non-blocking rematerialize_view()

14644 09/04/2014 07:41 AM Aaron Marcuse-Kubitza

schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's and Martha's request (e-mails from Martha on 2014-8-12 at 17:37PT, and from Brian M on 2014-8-13 at 16:21PT). note that this change has already been made on vegbiendev.

14643 09/04/2014 07:21 AM Aaron Marcuse-Kubitza

schemas/public_.sql: view_full_occurrence_individual: re-ran view_full_occurrence_individual_view_modify(), which uses the new non-blocking rematerialize_view()

14642 09/04/2014 07:20 AM Aaron Marcuse-Kubitza

schemas/util.sql: rematerialize_view(): made it non-blocking, so that it would allow full access to the original materialized table during the operation

14641 09/04/2014 07:11 AM Aaron Marcuse-Kubitza

schemas/util.sql: added identifier_replace()

14640 09/04/2014 07:08 AM Aaron Marcuse-Kubitza

schemas/util.sql: added relation_replace()

14639 09/04/2014 01:50 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource import: renamed to Single datasource refresh since it works on existing datasources

14638 09/04/2014 01:45 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource import: also need to reload staging tables

14637 09/04/2014 01:42 AM Aaron Marcuse-Kubitza

/README.TXT: Single datasource import: added steps to re-run geoscrubbing and back up the vegbiendev database

14636 09/03/2014 02:43 AM Aaron Marcuse-Kubitza

planning/workflow/BIEN data workflow-2_bje.pptx: fixed text alignment

14635 09/03/2014 02:35 AM Aaron Marcuse-Kubitza

planning/workflow/BIEN data workflow-2_bje.pptx: answered questions asked in the diagram

14634 09/03/2014 02:19 AM Aaron Marcuse-Kubitza

added planning/workflow/BIEN data workflow-2_bje.pptx from Martha/Brian E (in Asana)

14633 08/29/2014 03:55 PM Aaron Marcuse-Kubitza

added inputs/CVS/verify/Review of CVS data in BIEN3.docx

14632 08/29/2014 12:40 AM Aaron Marcuse-Kubitza

backups/*retention_policy*: added explanations

14631 08/29/2014 12:39 AM Aaron Marcuse-Kubitza

backups/*retention_policy*: on jupiter: backups further back: removed "if disk space permits" because this is already labeled "optionally"

14630 08/29/2014 12:38 AM Aaron Marcuse-Kubitza

backups/*retention_policy*: changed to require retaining *.backup of the last 2 successful imports on all machines

14629 08/29/2014 12:25 AM Aaron Marcuse-Kubitza

backups/*retention_policy*: allow keeping *.backup of the last 2 successful imports on all machines, not just jupiter

14628 08/29/2014 12:17 AM Aaron Marcuse-Kubitza

: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE since one might not see the SAVE file in it

14627 08/29/2014 12:13 AM Aaron Marcuse-Kubitza

: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE since one might not see the SAVE file in it

14626 08/29/2014 12:09 AM Aaron Marcuse-Kubitza

/"**DO_NOT_DELETE": renamed to shorter SAVE

14625 08/29/2014 12:04 AM Aaron Marcuse-Kubitza

added backups/*retention_policies*/ with retention policy files for each partition

14624 08/28/2014 11:58 PM Aaron Marcuse-Kubitza

backups/README.TXT: renamed to retention_policy to match the naming convention of the retention policy files in the various partitions

14623 08/28/2014 11:42 PM Aaron Marcuse-Kubitza

/README.TXT: to back up the local machine's hard drive: also exclude *-files indicating the (differing) retention statuses of the partitions involved

14622 08/28/2014 08:13 PM Aaron Marcuse-Kubitza

lib/tnrs.py single_tnrs_request(), bin/tnrs_client: use_tnrs_export: default to False because this mode uses incorrect selected matches (vegpath.org/issues/943), and the JSON mode that fixes this is now available

14621 08/28/2014 08:05 PM Aaron Marcuse-Kubitza

bin/tnrs_db: tnrs.tnrs_request() call: explicitly set use_tnrs_export=True so that this continues to work if the default value is changed

14620 08/28/2014 07:57 PM Aaron Marcuse-Kubitza

bugfix: lib/csvs.py: JsonReader: need to pass col_order to row_dict_to_list_reader

14619 08/28/2014 07:43 PM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: ~/Documents/BIEN/vegbiendev.2014-2-2_1-07-32PT.+VirtualBox_changes/: renamed to vegbiendev.2014-2-2_1-07-32PT.VirtualBox/ to make clear that this is the VirtualBox version of vegbiendev

14618 08/28/2014 07:12 PM Aaron Marcuse-Kubitza

bugfix: lib/tnrs.py: JSON output: need to stringify arrays so they match what is output in TSV-export mode

14617 08/28/2014 07:10 PM Aaron Marcuse-Kubitza

lib/csvs.py: JsonReader: added support for values that are arrays

14616 08/28/2014 07:05 PM Aaron Marcuse-Kubitza

lib/csvs.py: MultiFilter: inherit from WrapReader instead of Filter to avoid needing to define a no-op filter_() function

14615 08/28/2014 06:49 PM Aaron Marcuse-Kubitza

bugfix: lib/csvs.py: row_dict_to_list_reader: need to override next() directly instead of just using Filter, because Filter doesn't support returning multiple rows for one input row (in this case, prepending a header row). this caused the 1st data row to be missing.

14614 08/28/2014 06:47 PM Aaron Marcuse-Kubitza

lib/csvs.py: Filter: inherit from WrapReader, which separates out the CSV-reader API code

14613 08/28/2014 06:43 PM Aaron Marcuse-Kubitza

lib/csvs.py: added WrapReader

14612 08/28/2014 06:43 PM Aaron Marcuse-Kubitza

lib/csvs.py: added Reader

14611 08/28/2014 06:00 PM Aaron Marcuse-Kubitza

schemas/public_.sql: views that use view_full_occurrence_individual_view: use the view_full_occurrence_individual table instead, now that this is materialized.

14610 08/28/2014 05:58 PM Aaron Marcuse-Kubitza

planning/meetings/BIEN conference call availability.xlsx: updated

14609 08/28/2014 08:57 AM Aaron Marcuse-Kubitza

/README.TXT: to back up the local machine's hard drive: renamed backup partition to BIEN3 to make clear what the backup drive contains

14608 08/28/2014 08:54 AM Aaron Marcuse-Kubitza

fix: /README.TXT: to back up the local machine's hard drive: updated location of `screen` for added commands

14607 08/28/2014 08:53 AM Aaron Marcuse-Kubitza

/README.TXT: added trailing / on dirs to make clear that they're dirs

14606 08/28/2014 08:40 AM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to configure the VM to support VirtualBox

14605 08/28/2014 08:22 AM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the contents of the VM, with the VirtualBox changes added

14604 08/28/2014 07:47 AM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the original contents of the backup from the VM: added steps to restore the correct VM snapshot

14603 08/28/2014 07:40 AM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: also generate list of all the files whose permissions were changed since the backup, but which are extracted with their changed permissions instead of their original ones in the backup

14602 08/28/2014 07:05 AM Aaron Marcuse-Kubitza

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the original contents of the backup from the VM

14601 08/28/2014 05:47 AM Aaron Marcuse-Kubitza

fix: /README.TXT: to back up vegbiendev: also back up /home/aaronmk/bien/ (instead of just symlinking to the local copy), since this can be done space-efficiently with hardlinks. this ensures that the vegbiendev backup will not be modified when the local copy of bien/ is.

14600 08/28/2014 03:10 AM Aaron Marcuse-Kubitza

lib/csvs.py: JsonReader: factored out row-dict-to-list into new row_dict_to_list_reader so that JSON-specific preprocessing is kept separate from the row format translation

14599 08/27/2014 03:17 PM Aaron Marcuse-Kubitza

lib/csvs.py: added MultiFilter, which enables applying multiple filters by nesting

14598 08/26/2014 07:57 PM Aaron Marcuse-Kubitza

lib/tnrs.py: single_tnrs_request(): JSON mode: implemented output of JSON data

14597 08/26/2014 07:53 PM Aaron Marcuse-Kubitza

lib/tnrs.py: single_tnrs_request(): factored out wrapping in TnrsOutputStream, since this is done for both modes

14596 08/26/2014 07:47 PM Aaron Marcuse-Kubitza

fix: lib/tnrs.py: JSON mode: TSV export columns: need to translate these to JSON column names before they can be used with the JSON data

14595 08/26/2014 07:44 PM Aaron Marcuse-Kubitza

lib/csvs.py: added JsonReader, which reads parsed JSON data as row tuples

14594 08/26/2014 07:43 PM Aaron Marcuse-Kubitza

lib/csvs.py: added row_dict_to_list(), which translates a CSV dict-based row to a list-based one

14593 08/26/2014 07:43 PM Aaron Marcuse-Kubitza

lib/csvs.py: RowNumFilter: added support for filtering the header row as well

14592 08/26/2014 07:42 PM Aaron Marcuse-Kubitza

lib/csvs.py: ColInsertFilter: added support for filtering the header row as well

14591 08/26/2014 05:12 PM Aaron Marcuse-Kubitza

lib/csvs.py: InputRewriter: documented that this is also a stream (in addition to inheriting from StreamFilter)

14590 08/26/2014 05:11 PM Aaron Marcuse-Kubitza

bugfix: lib/csvs.py: InputRewriter: accept a reader, as would be expected, instead of a custom stream whose lines are tuples

14589 08/26/2014 05:08 PM Aaron Marcuse-Kubitza

fix: lib/sql_io.py: append_csv(): use new csvs.ProgressInputFilter instead of streams.ProgressInputStream(csvs.StreamFilter(__)), so that the input to csvs.InputRewriter is a reader, not a stream. this avoids the need for csvs.InputRewriter to accept a stream whose lines are tuples, instead of the expected reader.

14588 08/26/2014 05:02 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/install: $(exportHeader) must come before postprocess because postprocess renames columns

14587 08/26/2014 04:50 PM Aaron Marcuse-Kubitza

exports/: svn:ignore: added *.gz

14586 08/26/2014 04:49 PM Aaron Marcuse-Kubitza

lib/csvs.py: added ProgressInputFilter, analogous to streams.ProgressInputStream

14585 08/26/2014 04:46 PM Aaron Marcuse-Kubitza

lib/sql_io.py: added commented-out debug statement used to troubleshoot copy_expert() errors

14584 08/26/2014 04:45 PM Aaron Marcuse-Kubitza

lib/dicts.py: added pair_keys(), pair_values()

14583 08/26/2014 04:15 PM Aaron Marcuse-Kubitza

bugfix: lib/streams.py: CaptureStream: end_idx must also be > start_idx

14582 08/26/2014 04:07 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: $(import_install_): need `set -o pipefail` to enable errexit

14581 08/26/2014 03:47 AM Aaron Marcuse-Kubitza

/README.TXT: to backup files not in Time Machine: don't need to review diff because command is unidirectional

14580 08/26/2014 02:59 AM Aaron Marcuse-Kubitza

fix: /README.TXT: to back up the local machine's hard drive: "repeat until only minimal changes" should refer to the first sync command

14579 08/26/2014 02:52 AM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/run: documented postprocess() rm=1 runtime (6 min)