Project

General

Profile

  • svn:executable: *

# Date Author Comment
11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

10268 07/14/2013 12:33 AM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: need to include lib/runscripts/mysql.table.run instead of table.run (table.run was accidentally substituted when inputs/.NCBI/table.run was copied to all new-style datasources

10179 07/06/2013 05:39 PM Aaron Marcuse-Kubitza

inputs/*/: added table.run for use by the table subdirs in new-style import. datasources without table subdirs do not need this.

9820 06/10/2013 11:48 PM Aaron Marcuse-Kubitza

added lib/runscripts/mysql.table.run (general to all MySQL datasources) and use it in inputs/GBIF/table.run

9818 06/09/2013 09:21 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): don't run table.tsv.gz/upload in test mode, to avoid clobbering the backup of a full table.tsv with a partial, testing table.tsv

9816 06/09/2013 09:08 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): don't use inplace mode because it leaves a newer mtime when aborted, causing rsync to think that the partial upload is actually newer than the source. note that rsync's --partial-dir mode is just as capable of resuming an aborted upload (it will just use a file in .rsync-tmp instead). inplace mode is primarily designed for fixed-offset files which don't change much between edits, but this is not true for exports (or the gzips of them), which will change the file offsets of most data if even one row or column is added or removed.

9815 06/09/2013 09:01 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): run table.tsv.gz/upload here instead of in table.tsv/make() because it should not run until table.tsv.gz is finished being made, which is not the case in table.tsv/make() because table.tsv.gz/make is run in the background

9814 06/09/2013 08:59 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv.gz/upload(): moved before table.tsv.gz/make() so it can be used by it

9813 06/09/2013 08:39 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): need overwrite=1 because the mtime of an aborted inplace upload is newer

9812 06/09/2013 08:32 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv*/upload(): renamed to table.tsv.gz/upload() to upload only table.tsv.gz, not table.tsv, in order to save bandwidth

9807 06/09/2013 07:00 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv*/upload(): need to run put in live mode (live=1)

9803 06/09/2013 06:30 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): run table.tsv*/upload when the file make is done so that the file is backed up to jupiter

9802 06/09/2013 06:29 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: added table.tsv*/upload()

9749 06/06/2013 05:33 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made

9748 06/06/2013 05:27 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: added table.tsv.gz/make()

9746 06/06/2013 05:04 AM Aaron Marcuse-Kubitza

lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode

9743 06/06/2013 02:36 AM Aaron Marcuse-Kubitza

*{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism

9709 06/05/2013 03:11 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking

9704 06/04/2013 06:35 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.

9699 06/01/2013 09:32 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)

9681 06/01/2013 05:17 AM Aaron Marcuse-Kubitza

bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped

9675 06/01/2013 03:30 AM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv/make(): force table.tsv.md5 to be remade (using remake=1) because the table.tsv contents will have changed

9469 05/20/2013 07:27 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: include make.sh so runscripts based on it can use make-related utils

9391 05/15/2013 11:27 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: moved table.tsv.md5/make() and invocation of it to inputs/GBIF/table.run because it's general to all tables (which would all use table.tsv for this datasource). use $target_filename in calling table.tsv.md5/make from table.tsv/make.

9348 05/15/2013 08:22 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run table.tsv/make() and functions used by it: added usage comments for cmd line usage, caller usage, and declaring function usage

9154 05/11/2013 12:06 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export

9074 05/10/2013 12:08 PM Aaron Marcuse-Kubitza

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

9053 05/10/2013 09:45 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly

9039 05/10/2013 07:39 AM Aaron Marcuse-Kubitza

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

9016 05/10/2013 05:01 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: split make utils out into separate make.sh

8991 05/10/2013 01:00 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error

8990 05/10/2013 12:59 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists

8965 05/09/2013 05:36 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): use extern so the MySQL_export command is echoed

8964 05/09/2013 05:24 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): don't make the target if it already exists

8963 05/09/2013 05:16 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make (): create the file directly rather than using inline_make, so that the command to create the file can use shell functions such as mysql

8962 05/09/2013 05:07 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv/make (): need to use $(target) (target's path relative to curdir) instead of just the filename, now that make does not change the directory

8961 05/09/2013 05:01 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: moved table.tsv creation to separate table.tsv/make () function

8950 05/09/2013 02:21 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run

8881 05/08/2013 08:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: changed echo_func to an alias that includes the "$", so that callers don't need to specify the "$" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.

8774 05/02/2013 04:47 AM Aaron Marcuse-Kubitza

moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/

8705 04/25/2013 04:49 PM Aaron Marcuse-Kubitza

*run: added include guards

8652 04/23/2013 09:58 PM Aaron Marcuse-Kubitza

lib/util.run: inline_make (): pass Makefile commands using /dev/fd/3 (i.e. 3<<'EOF' ...) instead of stdin, to allow stdin to be used by commands that prompt the user

8463 04/13/2013 02:28 AM Aaron Marcuse-Kubitza

lib/util.run: echo_func (): use the caller's FUNCNAME via the $FUNCNAME[] array instead of requiring them to pass it in the function args as `echo_func "$FUNCNAME" "$@"`

8302 04/02/2013 03:13 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: import(): Updated for lib/table.run template changes

8291 04/02/2013 02:21 PM Aaron Marcuse-Kubitza

*run: Use -e option to bash on the #! line instead of separate `set -o errexit` line so that there is no issue with the `set -o errexit` line getting separated from the #! line (errexit is required for the scripts to work properly)

8289 04/02/2013 02:07 PM Aaron Marcuse-Kubitza

lib/util.run: Run run_cmd at shell exit (using trap) instead of requiring every runscript to have `run_cmd ` at the end of it

8286 04/02/2013 01:47 PM Aaron Marcuse-Kubitza

Added inputs/GBIF/table.run