Project

General

Profile

Statistics
| Revision:
  • svn:ignore: extern

# Date Author Comment
14887 10/25/2014 07:32 PM Aaron Marcuse-Kubitza

/README.TXT: Full database import: should also snapshot Postgres before starting import

14886 10/25/2014 06:52 PM Aaron Marcuse-Kubitza

bugfix: inputs/import.stats.xls: need to exclude postprocessing from the ms/row and Change formulas, also for the "<2014-2-2" tab

14885 10/25/2014 06:51 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

14884 10/25/2014 06:42 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: updated import times

14883 10/25/2014 06:08 PM Aaron Marcuse-Kubitza

/README.TXT: record the import times: documented how to update the Postprocessing times

14882 10/25/2014 06:07 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: globs that list all the datasources: this was incorrectly excluding U, so fixed to match bin/with_all $hidden_srcs

14881 10/25/2014 05:29 PM Aaron Marcuse-Kubitza

fix: inputs/VegBank/project/: added _no_import because this is imported as part of taxon_observation.**

14880 10/25/2014 05:08 PM Aaron Marcuse-Kubitza

bugfix: inputs/import.stats.xls: need to exclude postprocessing from the ms/row and Change formulas. removed deleted rows that don't apply to the most recent imports. updated runtime formulas to match bin/import_all.

14879 10/25/2014 04:41 PM Aaron Marcuse-Kubitza

bugfix: inputs/import.stats.xls: restored missing formatting for multi-day times. prepped tabs to have new import stats data added.

14878 10/25/2014 04:18 PM Aaron Marcuse-Kubitza

fix: /README.TXT: record the import times: need to first click the "current" tab

14877 10/25/2014 03:19 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes and revision #s

14876 10/24/2014 06:33 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: To re-run geoscrubbing: don't close subshell because it closes automatically (this would instead exit `screen`)

14875 10/24/2014 06:31 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes

14874 10/24/2014 05:38 PM Aaron Marcuse-Kubitza

_license/VegBank_schema/applies_to.txt: added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx

14873 10/24/2014 05:36 PM Aaron Marcuse-Kubitza

added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx from Bob

14872 10/24/2014 04:10 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes

14871 10/24/2014 03:58 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: To re-run geoscrubbing: "press Enter" and `unset TMOUT` steps are reversed

14870 10/24/2014 03:50 PM Aaron Marcuse-Kubitza

bugfix: schemas/public_.sql: geoscrub_input_new: need to join on the ARRAY[] of each column, to compare NULLs properly

14869 10/24/2014 03:32 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: become_sudo don't continue running function after self_sudo finishes

14868 10/24/2014 01:29 PM Aaron Marcuse-Kubitza

/README.TXT: datasource removal: linked to runtimes on the wiki

14867 10/24/2014 12:33 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/map.csv: need to save it if errors occur in unmapped_terms.csv, new_terms.csv

14866 10/24/2014 12:05 PM Aaron Marcuse-Kubitza

fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill

14865 10/24/2014 12:04 PM Aaron Marcuse-Kubitza

fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill

14864 10/24/2014 11:05 AM Aaron Marcuse-Kubitza

added backups/old_backups.txt

14863 10/24/2014 10:49 AM Aaron Marcuse-Kubitza

added backups/TNRS.2014-6-2.backup.md5

14862 10/23/2014 05:37 PM Aaron Marcuse-Kubitza

_license/non-open-source/applies_to.txt: clarified that the things in this list are not embargoed, the issue is just that they are not formally open-source

14861 10/23/2014 03:43 PM Aaron Marcuse-Kubitza

schemas/public_.sql: view_full_occurrence_individual_view: join to taxonoccurrence: can use a USING join for this because plot.** only exports one locationevent_id column

14860 10/20/2014 03:33 PM Aaron Marcuse-Kubitza

inputs/Cyrille_traits/trait_observation/postprocess.sql: accessRights index: added vegbiendev runtime

14859 10/20/2014 03:29 PM Aaron Marcuse-Kubitza

fix: inputs/Cyrille_traits/trait_observation/postprocess.sql: need to remove private data, as clarified by Cyrille

14858 10/20/2014 03:13 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated: Cyrille traits: updated with Cyrille's clarifications on publishability

14857 10/19/2014 03:24 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/sync.sh, db.sh: *_snapshot() commands that run multiple sudo commands: use become_sudo so that the program won't hang at later sudo prompts. for pg_snapshot() with pg_start_backup(), this also prevents the backup from staying open at the end waiting on a sudo prompt, causing excess WAL logs to accrue.

14856 10/19/2014 03:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: become_sudo: documented that this is for funcs that run multiple sudo commands

14855 10/19/2014 03:15 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: $sudo_cmd_wrapper: use esc() so this does not need to be embedded with [@] array notation

14854 10/19/2014 03:09 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $wrap_fn: this always needs to contain `is_runscript=1` because $is_runscript is not (and cannot be) exported

14853 10/19/2014 03:06 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: sudo(): when don't need sudo: do need `command` to have effects of invoking external command

14852 10/19/2014 03:00 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_snapshot(): first need to run pg_stop_backup() in case a backup was not stopped cleanly

14851 10/19/2014 03:00 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_stop_backup(): ignore benign "a backup is not in progress" error

14850 10/19/2014 01:19 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: sudo(): only run sudo if needed

14849 10/19/2014 01:10 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: sudo(): factored PATH-preserving code out into separate wrapper var that can also be used separately, eg. with `sudo -u`

14848 10/19/2014 01:03 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: sudo(): moved addition of -E (which should always happen) to separate, overridden sudo() function

14847 10/19/2014 12:51 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved -E to is_callable section because it just applies to that

14846 10/19/2014 11:35 AM Aaron Marcuse-Kubitza

fix: inputs/input.Makefile: $(svnFilesGlob): only svn:ignore *.log in the top-level dir

14845 10/18/2014 07:32 PM Aaron Marcuse-Kubitza

added inputs/XAL/Specimen/data.csv (empty template)

14844 10/18/2014 05:03 PM Aaron Marcuse-Kubitza

fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip

14843 10/18/2014 05:03 PM Aaron Marcuse-Kubitza

fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip

14842 10/18/2014 04:24 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added become_sudo alias

14841 10/18/2014 04:21 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $is_root

14840 10/17/2014 02:52 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated: replaced BIEN2 traits with Cyrille traits

14839 10/17/2014 01:02 PM Aaron Marcuse-Kubitza

exports/view_full_occurrence_CVS.csv.run: documented runtime (2 min for 5000 rows)

14838 10/16/2014 01:51 PM Aaron Marcuse-Kubitza

fix: inputs/publishable datasources.xlsx: updated: conditions of use: Canadensys sources: these should actually be marked as no restrictions, in contrast to Brian E's earlier annotations, because they are public domain: the citation is requested, but not legally required

14837 10/16/2014 01:46 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated

14836 10/16/2014 01:40 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated: conditions of use: indicated which datasources have no restrictions

14835 10/16/2014 10:55 AM Aaron Marcuse-Kubitza

web/BIEN3/Redmine/issues/.htaccess: dest: decoded chars that don't need to be escaped, for clarity

14834 10/16/2014 10:51 AM Aaron Marcuse-Kubitza

bugfix: web/BIEN3/Redmine/issues/.htaccess: need to store dest in separate var because % is treated specially in the replacement str

14833 10/16/2014 10:49 AM Aaron Marcuse-Kubitza

inputs/CVS/^taxon_observation.**.sample/test.xml.ref: updated

14832 10/16/2014 10:49 AM Aaron Marcuse-Kubitza

fix: inputs/CVS/plantConcept_/header.csv: regenerated after fixing the postprocess/cleanup ordering bug (r14827), which had caused header.csv to be incorrectly generated after renames in postprocess.sql were applied

14831 10/16/2014 10:33 AM Aaron Marcuse-Kubitza

web/BIEN3/Redmine/issues/.htaccess: main page: sort by priority

14830 10/16/2014 10:14 AM Aaron Marcuse-Kubitza

planning/meetings/BIEN conference call availability.xlsx: updated

14829 10/15/2014 05:14 PM Aaron Marcuse-Kubitza

planning/meetings/BIEN conference call availability.xlsx: updated

14828 10/15/2014 04:08 PM Aaron Marcuse-Kubitza

planning/meetings/BIEN conference call availability.xlsx: updated: added and populated fall 2014 tab

14827 10/14/2014 02:50 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: postprocess must be run after cleanup rather than before because it depends on the cleanup having been performed.

this bug was not previously detected because this is only a problem when refreshing a datasource to data in the same format: this would attempt to run an existing postprocess.sql, out of order, instead of starting with no postprocess.sql as we usually do....

14826 10/14/2014 12:36 PM Aaron Marcuse-Kubitza

lib/sql_io.py: cleanup_table(): trim(): documented that this also converts character varying fields to text

14825 10/14/2014 11:05 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: $(dbExports): also need to put data.sql before clean_up.sql, etc. previously, this ordering had to be done by naming clean_up.sql, etc so they would sort after data.sql alphabetically, but it can be confusing to have to remember to do this. this fixes a bug in the CVS refresh where cvs.~.clean_up.sql was being run before data.sql, causing some private columns to have been deleted before the data was imported into the tables, creating a column mismatch error.

14824 10/14/2014 10:34 AM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: re-create public-schema views that were cascadingly deleted: listed which views these currently are

14823 10/14/2014 10:26 AM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes

14822 10/14/2014 10:09 AM Aaron Marcuse-Kubitza

bugfix: lib/sql_io.py: null_strs_str_default: removed "NA" because this is the abbr for a Spanish province (Navarra). this fixes the 2nd bug of #955, geovalidation duplicated rows.

14821 10/14/2014 10:00 AM Aaron Marcuse-Kubitza

lib/sql_io.py: cleanup_table(): debug-print null_strs

14820 10/14/2014 09:56 AM Aaron Marcuse-Kubitza

lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)

14819 10/14/2014 09:54 AM Aaron Marcuse-Kubitza

inputs/Cyrille_traits/Makefile: set custom $(null_strs) which handles "NA"

14818 10/14/2014 09:49 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: pass make var $(null_strs) to invoked commands so it can be used by lib/sql_io.py

14817 10/14/2014 09:39 AM Aaron Marcuse-Kubitza

fix: *Makefile: changed line endings to \n so that `patch` can work with pasted input. use `svn di --extensions --ignore-eol-style` to verify no diff.

14816 10/14/2014 08:35 AM Aaron Marcuse-Kubitza

lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)

14815 10/13/2014 07:40 AM Aaron Marcuse-Kubitza

bugfix: web/links/index.htm: updated to Firefox bookmarks: Mac: automatic restart on power loss: also need to disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss. Bookmarks Toolbar: reordered links to put those most relevant to VegBIEN users first.

14814 10/13/2014 07:25 AM Aaron Marcuse-Kubitza

bugfix: config/dev_machine/com.apple.PowerManagement.plist: disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss

14813 10/13/2014 07:17 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Datasource setup: For MS Access databases: also need to comment out `CREATE DATABASE` because pg_dump_limit is only applied to data.sql (due to requirements of the TNRS datasource)

14812 10/13/2014 06:31 AM Aaron Marcuse-Kubitza

added inputs/CVS/_src/cvs-eep-archive-2014-10-07-correctedCVSData.{data,schema}.sql.ini

14811 10/13/2014 06:31 AM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/_src/{data,schema}.sql.ini: sourcefilename: this needs to be on the VM's own HD to avoid crashing MSAccess to PostgreSQL. destinationdatabase: added this back since it is fine to leave this blank.

14810 10/13/2014 06:23 AM Aaron Marcuse-Kubitza

added inputs/CVS/_src/cvs-eep-archive-2013-10-22-VegBIEN.{data,schema}.sql.ini

14809 10/13/2014 06:17 AM Aaron Marcuse-Kubitza

added inputs/CVS/_src/{data,schema}.sql.ini

14808 10/13/2014 05:25 AM Aaron Marcuse-Kubitza

added backups/vegbien.r14673.backup.md5

14807 10/13/2014 04:45 AM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes

14806 10/12/2014 03:58 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated: use white text on dark backgrounds for better visibility, and to create more visual contrast for the unredistributable indicators

14805 10/12/2014 03:52 PM Aaron Marcuse-Kubitza

inputs/publishable datasources.xlsx: updated: consolidated Brian E's new columns into a single "conditions?" column. fix: "what is needed to publish it": renamed to "conditions of use/remaining tasks". "conditions of use": changed color scheme to match "publishable?" columns.

14804 10/09/2014 12:34 AM Aaron Marcuse-Kubitza

added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7-Revised.docx from Bob

14803 10/07/2014 12:01 AM Aaron Marcuse-Kubitza

bugfix: lib/tnrs.py: encode_map: also need to encode + because TNRS removes it from the morphospecies (vegpath.org/wiki/CVS_validation#Bobs-revised-document > issue #4)

14802 10/06/2014 11:35 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: also need to `unset TMOUT` because TMOUT causes shell to exit even with background processes

14801 10/03/2014 05:56 PM Aaron Marcuse-Kubitza

config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtLevel: decreased to 5% to allow working longer on UPS power

14800 10/03/2014 05:53 PM Aaron Marcuse-Kubitza

config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtMinutesLeft: decreased to 3 min to allow working longer on UPS power

14799 10/03/2014 05:36 PM Aaron Marcuse-Kubitza

bugfix: inputs/CVS/plot_/postprocess.sql: locality: site_name should come before directions_to_place because it is at a higher level of granularity

14798 10/03/2014 05:08 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/run`: should just need to reinstall the geoscrub_output table

14797 10/03/2014 05:06 PM Aaron Marcuse-Kubitza

schemas/public_.sql: geoscrub_input_view_modify(): add unique constraint on the same columns as geoscrub.geoscrub_output, as a sanity check to assist future debugging

14796 10/03/2014 04:57 PM Aaron Marcuse-Kubitza

bugfix: schemas/postgresql.conf: need to set extra_float_digits = 3 (the maximum) to enable verbatim export. this should fix issue #955.

14795 10/03/2014 12:37 PM Aaron Marcuse-Kubitza

added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7.docx from Bob

14794 10/03/2014 08:18 AM Aaron Marcuse-Kubitza

added config/dev_machine/com.apple.PowerManagement.plist from /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist , so that this file can be easily restored when the system spuriously resets settings in it

14793 10/02/2014 09:42 PM Aaron Marcuse-Kubitza

/README.TXT: To re-run geoscrubbing: updated runtimes. note that these are faster on the new hardware.

14792 10/01/2014 09:51 PM Aaron Marcuse-Kubitza

fix: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/geoscrub_output/geoscrub.csv.run`: use a subshell to avoid closing screen on errexit

14791 10/01/2014 09:39 PM Aaron Marcuse-Kubitza

fix: /README.TXT: Single datasource refresh: also back up DB after this

14790 10/01/2014 09:38 PM Aaron Marcuse-Kubitza

/README.TXT: datasource removal: documented bien2_traits runtime

14789 10/01/2014 09:35 PM Aaron Marcuse-Kubitza

added inputs/bien2_traits/_no_import since bien2_traits has been replaced by Cyrille_traits

14788 10/01/2014 09:32 PM Aaron Marcuse-Kubitza

/README.TXT: added datasource removal instructions