Activity
From 04/03/2013 to 05/02/2013
05/02/2013
- 11:07 PM Revision 8808: /run: export_ (): pass "$@" to invoked functions since it may contain psql connection or config info
- 11:06 PM Revision 8807: lib/util.sh: pg_* (): pass "$@" to invoked functions since it may contain psql connection or config info
- 10:59 PM Revision 8806: bugfix: lib/util.sh: pg_header (): use limit=0 to prevent COPY TO from returning all rows in the table (not an issue for empty tables!)
- 10:53 PM Revision 8805: lib/util.sh: pg_copy_to (): support sources besides whole tables (such as custom queries and tables with columns). support limiting the # of rows retrieved when using whole tables.
- 10:01 PM Revision 8804: lib/runscripts/import.run: added export_ () command, which is run after import in all ()
- 09:57 PM Revision 8803: bin/psql_verbose_vegbien: also echo queries issued by backslash commands (--echo-hidden)
- 09:34 PM Revision 8802: /run: make config vars such as $schema overridable by the environment, to allow pointing the export script at a different public schema version
- 08:53 PM Revision 8801: inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files
- 08:50 PM Revision 8800: added inputs/HIBG/Specimen/header.csv
- 08:50 PM Revision 8799: added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5
- 08:40 PM Revision 8798: exports/: svn:ignore *.csv
- 08:39 PM Revision 8797: added /run, whose export_ command exports DB tables (currently the geoscrub_input table) to CSV. column information is included in the filename when the export format is specified not to contain a header row.
- 08:35 PM Revision 8796: lib/runscripts/local.run: moved log_sql (), pg_* () to lib/util.sh because these do not depend on local external scripts in bin/
- 08:27 PM Revision 8795: lib/runscripts/local.run: added pg_copy_to (), pg_header (), pg_export_table_no_header (), pg_export_table_to_dir_no_header ()
- 08:26 PM Revision 8794: lib/runscripts/local.run: added mk_schema_esc, mk_table_esc aliases as shortcuts for `local *_esc; mk_esc_name *`
- 08:09 PM Revision 8793: lib/util.sh: self_not_included (): added support for loading newly-defined aliases for use in functions in the same file, and documentation for how to do this
- 02:49 PM Revision 8792: lib/runscripts/local.run: psql (): output only the query results to stdout (using --output), and redirect everything else to stderr
- 02:26 PM Revision 8791: added /exports/
- 02:24 PM Revision 8790: lib/runscripts/local.run: added log_sql (). mysql (): don't echo SQL commands when not in verbose mode. this is needed to support commands that use the query result in a $() expression, and should not have echoed commands cluttering up stdout. (unfortunately, the SQL commands are echoed to stdout rather than stderr.)
- 01:52 PM Revision 8789: schemas/vegbien.sql: range_modeling_input: use analytical_stem instead of analytical_stem_view because it takes a long time (~2 min) to get a row from the left-joined view, due to the large number of joins and large number of rows in the joined tables
- 01:41 PM Revision 8788: web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
- 01:28 PM Revision 8787: web/links/index.htm: author comments at beginning of bookmark desc: start with a line containing only dashes (-) rather than a blank line, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
- 01:24 PM Revision 8786: lib/Firefox_bookmarks.reformat.csv: label page's description: support starting a description with a line containing only dashes (-) to indicate author comments. this is better than a newline, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
- 01:16 PM Revision 8785: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: switch back to using <a href=...> instead of onclick, but put the <a> tag *around* the <H3> rather than inside it so that Firefox can correctly parse out the folder name when importing the bookmarks
- 12:52 PM Revision 8784: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: use `onclick="javascript:document.location = '#...'"` instead of <a href=...> to allow clicking the header to go to the anchor, so that the <a> tag doesn't prevent Firefox from parsing the folder name
- 12:42 PM Revision 8783: bugfix: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: moved <a name=.../> outside the <H3> so that Firefox can correctly parse out the folder name when importing the bookmarks
- 12:27 PM Revision 8782: web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
- 12:19 PM Revision 8781: lib/Firefox_bookmarks.reformat.csv: indexing: add hyperlinked HTML anchors for folders
- 11:50 AM Revision 8780: web/links/index.htm: updated to Firefox bookmarks
- 08:30 AM Revision 8779: schemas/vegbien.sql: added range_modeling_input view
- 08:21 AM Revision 8778: schemas/vegbien.sql: analytical_stem_view: order by datasource name so that the rows have a predictable rather than random ordering
- 06:14 AM Revision 8777: bugfix: lib/util.sh: moved mysql (), psql () to local.run because they use local commands that are only available after local.run sets the PATH. retain a mysql () function in util.sh that adds the --verbose option to mysql.
- 05:44 AM Revision 8776: lib/util.sh: removed "$bin_dir"/ before commands because the bin dir is now in the PATH
- 05:13 AM Revision 8775: lib/runscripts/table.run: moved general DB commands to lib/util.sh
- 04:47 AM Revision 8774: moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/
- 04:19 AM Revision 8773: bugfix: lib/util.sh: canon_rel_path (): use $(pwd -P) instead of $PWD because symlinks are resolved by realpath (readlink -f), so they also need to be resolved in the current dir, but $PWD does not contain the symlinks-resolved version of the current dir
- 03:48 AM Revision 8772: inputs/GBIF/MySQL_export: use lib/util.sh instead of util.run now that the non-runscript-specific functions have been separated out into it. this ensures that MySQL_export is not unintentionally treated as a runscript (which functions differently from a normal shell script).
- 02:46 AM Revision 8771: /.htaccess: use canonical URL without symlinks
- 02:44 AM Revision 8770: added /.htaccess to create the right self-referential URL since / is outside the document root (linked from /vegbiendev/fs/)
- 02:14 AM Revision 8769: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: moved labels to the *start* of the arrow they label, so that the user can more easily follow the arrows from step to step by looking for an outward-pointing arrow with a label
- 01:48 AM Revision 8768: added inputs/GBIF/_MySQL/MySQL.data.sql.gz.md5
- 12:46 AM Revision 8767: bugfix: lib/common.Makefile: Compression: %.gz <-> %: only run command if target does not exist, to avoid overwriting the target when the compressed or uncompressed version is newer than it. the difference in mtimes can arise when one file is created after the other, and should not cause the opposite operation to be performed to try to make the other file up-to-date (leading to an infinite back-and-forth of creating one file from the other).
04/30/2013
- 11:13 PM Revision 8766: web/links/index.htm: updated to Firefox bookmarks. the VegPath link now includes the new favicon.
- 11:12 PM Revision 8765: added web/favicon.ico generated from schemas/BIEN_logo.png
- 11:01 PM Revision 8764: web/main.conf: added SSL-protected site for use with password-protected areas of the site, so that the user's password is not transmitted in plaintext. (because the SSL certificate is self-signed, it will unfortunately display the Confirm Security Exception message in Firefox.) note that digest authentication cannot be used to avoid plaintext passwords, because it requires knowing the user's original password to generate the digest, but we have only the MD5 sums in /etc/shadow.
- 09:26 PM Revision 8763: web/links/index.htm: updated to Firefox bookmarks
- 08:08 PM Revision 8762: web/links/index.htm: put all bookmarks into one subfolder named BIEN links, to facilitate importing them all into one folder on another machine. this also helps when upgrading to a newer version of the bookmarks, because the previous version must be deleted and this now requires deleting only a single folder.
- 08:02 PM Revision 8761: lib/Firefox_bookmarks.reformat.csv: use <H3 style="font-size: 200%;"> instead of <H1> to make the first folder's name the page title, so that Firefox can reimport the first folder as a bookmarks folder
- 07:42 PM Revision 8760: bugfix: lib/Firefox_bookmarks.reformat.csv: make first folder name the page title: added back replacement string
- 07:40 PM Revision 8759: web/links/index.htm.run: clean_up (): removed command to add page title because this is now done by lib/Firefox_bookmarks.reformat.csv
- 07:40 PM Revision 8758: lib/Firefox_bookmarks.reformat.csv: use page title as window title
- 07:31 PM Revision 8757: lib/Firefox_bookmarks.reformat.csv: remove previous page title
- 07:29 PM Revision 8756: lib/Firefox_bookmarks.reformat.csv: make first folder name the page title
- 07:21 PM Revision 8755: lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar inside first folder instead of before it. this allows storing all bookmarks in one subfolder so that when the web page is imported into another computer's bookmarks, the imported bookmarks are all together in one folder.
- 07:16 PM Revision 8754: bugfix: lib/Firefox_bookmarks.reformat.csv: refactored regexps to account for Python applying (?s) to the entire regexp, even if it appears inside a () subgroup. the Python behavior is in contrast to the standard regexp behavior used by Java, etc. where (?s) applies only to the subgroup it appears in.
- 06:19 PM Revision 8753: web/links/index.htm: updated to Firefox bookmarks. resources: put unsorted links into subfolders.
- 06:11 PM Revision 8752: bugfix: lib/Firefox_bookmarks.reformat.csv: add <style>: format bookmark description as plaintext: use `white-space: pre-wrap` instead of pre in order to wrap long lines
- 06:07 PM Revision 8751: bugfix: lib/Firefox_bookmarks.reformat.csv: favicon images: made replacements idempotent so that the favicons would not be deleted the next time the web/links/index.htm.run is run (due to a newline being added before ICON= )
- 05:52 PM Revision 8750: lib/Firefox_bookmarks.reformat.csv: favicon images: display placeholder favicon image for every link to ensure uniform alignment
- 05:45 PM Revision 8749: added web/links/blank.gif symlink to web/blank.gif for shorter blank.gif URLs
- 05:43 PM Revision 8748: added web/blank.gif
- 05:14 PM Revision 8747: lib/Firefox_bookmarks.reformat.csv: display ICON attributes as separate images. note that this uses the "data:" embedded image contents provided in the bookmarks file, so the images load instantly along with the page instead of needing to be separately retrieved from each server.
- 04:45 PM Revision 8746: lib/Firefox_bookmarks.reformat.csv: instead of removing ICON attributes with favicon contents, put them on their own line so the rest of the line is easier to see
- 04:35 PM Revision 8745: lib/Firefox_bookmarks.reformat.csv: divided replacements into sections
- 04:27 PM Revision 8744: lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar to beginning, remove its desc, and rename it to Quick links
- 04:13 PM Revision 8743: web/links/index.htm: removed extra divider after Recent Tags, Recently Bookmarked because section dividers interfere with regexp matching and the Bookmarks Toolbar is already visually separate from them
- 04:09 PM Revision 8742: web/links/index.htm: moved Recent Tags, Recently Bookmarked special folders to end because they are Firefox-specific and not part of the BIEN links (they also do not work in a browser because they are Firefox-specific query bookmarks instead of actual folders)
- 03:32 PM Revision 8741: web/links/index.htm.run: clean_up (): rename the page title from "Bookmarks" to "BIEN links"
- 03:18 PM Revision 8740: bin/repl: skip header instead of parsing it, since its columns are not used. this also allows using just a blank line as the header when the column names are not important.
- 03:05 PM Revision 8739: web/links/index.htm: use initial blank line instead of previous "-" workaround to indicate author comment at start of bookmark desc
- 03:04 PM Revision 8738: web/links/index.htm: use initial blank line to indicate author comment at start of bookmark desc
- 02:59 PM Revision 8737: lib/Firefox_bookmarks.reformat.csv: label page's description: don't label a bookmark desc starting with a blank line (indicating an author-added comment) as a page's description
- 02:51 PM Revision 8736: lib/Firefox_bookmarks.reformat.csv: label page's description: don't label an author-added quotation (which starts with " ) as the page's description
- 02:33 PM Revision 8735: bin/in_place: set the permissions of the new file to the original file when `chmod --reference` is available. this ensures that when a file is marked web-accessible, it stays web-accessible when you run an in_place command on it.
- 02:22 PM Revision 8734: web/links/index.htm: BitTorrent Sync: security warning about data leak: added link to reported bug at http://forum.bittorrent.com/topic/18612-security-hole-in-linux-webui-makes-all-your-files-publicly-accessible/ . updated workaround instructions.
- 01:20 PM Revision 8733: web/links/index.htm: updated to Firefox bookmarks. BitTorrent Sync: added security warning that running btsync on Linux will by default create a *publicly-accessible, unprotected* WebUI, allowing anyone on the web to create a sync folder to view and edit any of your files. to avoid this, always run btsync with a config file which sets a password on the WebUI. if you have already run btsync, run `killall btsync` to turn off the WebUI.
- 12:59 PM Revision 8732: web/links/index.htm: updated to Firefox bookmarks. changes include links about BitTorrent Sync.
- 12:51 PM Revision 8731: lib/Firefox_bookmarks.reformat.csv: label <meta> description: renamed label text to "page's description" to be understandable to non-programmers
- 12:49 PM Revision 8730: lib/Firefox_bookmarks.reformat.csv: label <meta> description: remove any existing description label, to avoid duplicating the description label when the text of the description label changes
- 12:37 PM Revision 8729: lib/Firefox_bookmarks.reformat.csv: label <meta> description as such to avoid confusing it with comments added by the bookmark author, who may not agree with it. the <meta> description is in the 1st paragraph(s), when provided.
- 12:13 PM Revision 8728: lib/Firefox_bookmarks.reformat.csv: remove LAST_MODIFIED attributes, which seem to change whenever the favicon is updated and clutter up the svn diff
- 12:05 PM Revision 8727: lib/Firefox_bookmarks.reformat.csv: remove ICON attributes with favicon contents, which clutter up the svn diffs
- 11:52 AM Revision 8726: added web/links/index.htm.run with clean_up action to run whenever the index.htm bookmarks file is changed (`web/links/index.htm.run clean_up`)
- 11:51 AM Revision 8725: added lib/Firefox_bookmarks.reformat.csv
04/25/2013
- 09:58 PM Revision 8724: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: exclude only backup files in the Documents/BIEN/svn/backups/ dir (sync the other files)
- 09:53 PM Revision 8723: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: don't sync the group, which can be bien on vegbiendev but not on jupiter (where aaronmk is not a member of the bien group)
- 09:51 PM Revision 8722: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: include *.md5
- 09:08 PM Revision 8721: README.TXT: Maintenance: removed step to backup the Ubuntu VM, because this is now done as part of the "backup files not in Time Machine" step
- 09:07 PM Revision 8720: README.TXT: Maintenance: added steps to backup files not in Time Machine
- 08:57 PM Revision 8719: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: back up (almost) all files in the home directory instead of just Library/ (to both jupiter and Dropbox)
- 06:45 PM Revision 8718: README.TXT: put commands: removed unnecessary /** after dirs to exclude
- 06:26 PM Revision 8717: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: added steps to also backup the settings to Dropbox (in addition to jupiter)
- 06:13 PM Revision 8716: lib/util.sh: added realpath () and use it where `readlink -f` is used
- 06:05 PM Revision 8715: lib/util.run: doc comment: changed "run scripts" to "runscripts"
- 06:04 PM Revision 8714: lib/util.run: factored general utils (non-runscript-specific) out into new util.sh
- 05:55 PM Revision 8713: lib/util.run: renamed $verbose to $verbosity because it's an integer level, not a boolean. continue to support specifying the verbosity in $verbose.
- 05:52 PM Revision 8712: lib/util.run: support $verbose="", which is treated as 0
- 05:50 PM Revision 8711: lib/util.run: only automatically echo env vars when they are set if verbosity is >= 2
- 05:49 PM Revision 8710: lib/util.run: added $verbose var to control logging. this is normally set to 0, but defaults to 2 for a runscript.
- 05:41 PM Revision 8709: lib/util.run: moved `shopt -s expand_aliases` inside the include guard so callers can override the setting without it being reset if util.run is included again
- 05:39 PM Revision 8708: lib/local.run: add $bin_dir to the PATH
- 05:15 PM Revision 8707: bin/in_place: don't run if no command to run, because that would erase the file
- 05:10 PM Revision 8706: bin/in_place: use -e (exit on error) option to sh instead of adding `|| exit` after every command
- 04:49 PM Revision 8705: *run: added include guards
- 04:30 PM Revision 8704: bugfix: lib/util.run: added include guard to prevent issues in "diamond includes", where a runscript includes two runscripts and *each* includes util.run, causing the second include of util.run to reset any methods overridden in the first included runscript. this prevents util.run from being included twice in the first place.
- 04:25 PM Revision 8703: lib/util.run: added include_guard_var (), self_not_included () for use by path-based include guards
- 03:40 PM Revision 8702: lib/util.run: echo_stdin (): print a separator before and after the echoed output to help distinguish it from other verbose output
- 03:37 PM Revision 8701: lib/table.run: use new local.run instead of defining $bin_dir, etc. itself
- 03:36 PM Revision 8700: lib/util.run: echo_export (): run the export builtin using `builtin` to avoid recursively calling any export alias or function defined by the user (e.g. `alias export=echo_export`. this causes util.run to be idempotent, such that it can be included multiple times without causing problems due to the redefinition of functions, vars, etc.
- 03:04 PM Revision 8699: added lib/local.run, which links to locally-available resources such as $bin_dir
- 02:50 PM Revision 8698: lib/table.run: added mysql (), mysql_ANSI ()
- 02:41 PM Revision 8697: added web/links/index.htm containing Firefox bookmarks exported to HTML. CSS style information has been added to fix default HTML tag formatting, improve readability, and handle the Firefox bookmarks format. <dd> tags have been manually closed where HTML would not auto-close them properly. note that these changes must be reapplied each time the bookmarks file is reexported (`svn di` will show the changes to make).
- 04:54 AM Revision 8696: lib/table.run: run mk_esc_name for $schema, $table
04/24/2013
- 11:18 PM Revision 8695: lib/table.run: added esc_name (), mk_esc_name ()
- 11:16 PM Revision 8694: lib/util.run: added set_var ()
- 10:58 PM Revision 8693: bugfix: lib/util.run: run_args_cmd (): `set --`: need to use eval so that the output of reverse is interpreted as escaped args rather than split using $IFS
- 10:40 PM Revision 8692: bugfix: lib/util.run: run_args_cmd (): BASH_ARGV is stored in reverse order, so it needs to be reversed to be usable as $@
- 10:38 PM Revision 8691: lib/util.run: added reverse ()
- 02:51 PM Revision 8690: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 02:49 PM Revision 8689: schemas/VegCore/VegCore.ERD.mwb: source: renamed contact -> owner because smaller sources may not have an admin/contact person, but would have a copyright holder or other entity with control over the source's distribution
- 02:45 PM Revision 8688: schemas/VegCore/VegCore.ERD.mwb: added table comments from VegCore data dictionary (VegCore.vegpath.org)
- 01:08 PM Revision 8687: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 01:07 PM Revision 8686: schemas/VegCore/VegCore.ERD.mwb: added hstore expansion columns where needed
- 12:54 PM Revision 8685: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 12:34 PM Revision 8684: schemas/VegCore/VegCore.ERD.mwb: specimen: added unique constraint for each set of identifying columns
- 12:25 PM Revision 8683: schemas/VegCore/VegCore.ERD.mwb: collection: added back primary key
- 12:24 PM Revision 8682: schemas/VegCore/VegCore.ERD.mwb: specimen: made owner an fkey to collection instead of organization to correspond with orig_collection and current_collection (and to allow auto-filling it from current_collection in a trigger)
- 12:18 PM Revision 8681: schemas/VegCore/VegCore.ERD.mwb: specimen: reordered columns to group together each method of identifying a specimen
- 12:14 PM Revision 8680: schemas/VegCore/VegCore.ERD.mwb: collection: don't inherit from source, because a collection is a scoping entity, not a dataset. a collection *may* also be used as a dataset, but need not be for the purposes of scoping a specimen's accession_number.
- 12:07 PM Revision 8679: schemas/VegCore/VegCore.ERD.mwb: specimen: removed redundant stamping_institution and donor_institution, because these are now in orig_collection.institution. (it is assumed that stamping_institution and donor_institution are the same, because the first institution to acquire and store a specimen (the donor) is the one that stamps it with an accession_number.)
- 12:01 PM Revision 8678: schemas/VegCore/VegCore.ERD.mwb: specimen: split collection into orig,current_collection because the collection that scoped the accession_number (the orig_collection) is sometimes different than the collection where the specimen is now held (the current_collection, or specimen_holder_institutions), e.g. when the specimen is donated
- 11:49 AM Revision 8677: schemas/VegCore/VegCore.ERD.mwb: collection: added institution, name to identify and scope the collection
- 11:39 AM Revision 8676: schemas/VegCore/VegCore.ERD.mwb: changed taxon_name fkeys to taxon_concept where appropriate, and renamed the referencing columns to *_taxon_concept to clearly indicate the referenced table. taxon_concept fkeys are appropriate whenever a taxon is defined or associated with an observed entity, in which case the assertion is not just to a taxon name but to a defined taxon concept (albeit sometimes an unofficial taxon concept scoped by the collector or identifier).
- 11:08 AM Revision 8675: schemas/VegCore/VegCore.ERD.mwb: specimen: added code_in_individual column
- 10:40 AM Revision 8674: schemas/VegCore/VegCore.ERD.mwb: specimen: added fkey to individual for the individual the specimen was taken from
- 10:30 AM Revision 8673: schemas/VegCore/VegCore.ERD.mwb: individual/stem_observation: made individual/stem nullable because there is not always enough identifying information (stem tag, etc.) to actually associate an identifiable individual with an observation
- 10:27 AM Revision 8672: schemas/VegCore/VegCore.ERD.mwb: moved code from individual to individual_observation because the code is something applied at observation time, rather than something that can be used to track individuals *between observations*
- 10:17 AM Revision 8671: schemas/VegCore/VegCore.ERD.mwb: moved collection_event from taxon_occurrence to specimen because only specimens are actually collected, while other taxon_occurrences (e.g. individuals) must be uniquely identified by some means that are shared between re-observations (e.g. the stem tag)
- 10:03 AM Revision 8670: schemas/VegCore/VegCore.ERD.mwb: moved collector, collector_number from taxon_occurrence to taxon_observation because these are actually attributes that are available or assigned at observation time, rather than being intrinsic to the taxon_occurrence (which can be either an individual or a specimen)
- 09:34 AM Revision 8669: schemas/VegCore/VegCore.ERD.mwb: method: colored it forest green like stratum because these are both tables in the same category (method), and forest green is the color VegBank uses for methods. (the fact that VegBank gives stratum the method color indicates that we are on the right track when we are treating stratum as a type of method that's essentially a height class.)
- 09:27 AM Revision 8668: schemas/VegCore/VegCore.ERD.mwb: method: added parent fkey which can be used to autoinherit attributes from another method, in the same way that event autoinherits a method from its parent
- 09:16 AM Revision 8667: schemas/VegCore/VegCore.ERD.mwb: replaced < event label for diagram half with < place, because the occurrence half actually also includes events (observations), but does not include places
- 09:10 AM Revision 8666: schemas/VegCore/VegCore.ERD.mwb: removed extra empty page used for moving objects around
- 09:08 AM Revision 8665: schemas/VegCore/VegCore.ERD.mwb: swapped position of place and source categories so that the place tables, which users are apt to query often, are at the top, and the attribution information, which is likely to be used less often, is at the bottom. this also moves the plot table closer to taxon_occurrence, which should eventually have an fkey to it.
- 08:19 AM Revision 8664: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 08:05 AM Revision 8663: schemas/VegCore/VegCore.ERD.mwb: record: added source_id_scope and added it to the record_unique unique index
- 08:01 AM Revision 8662: schemas/VegCore/VegCore.ERD.mwb: Default*ColumnType: updated to VARBINARY type used in key columns
04/23/2013
- 11:19 PM Revision 8661: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: repositioned arrow labels so they point in the direction of the fkey lines
- 11:06 PM Revision 8660: schemas/VegCore/VegCore.ERD.mwb.run: use new zip/unzip_newer
- 11:06 PM Revision 8659: lib/util.run: added zip/unzip_newer aliases
- 11:05 PM Revision 8658: lib/util.run: added zip/unzip echo_run aliases
- 10:57 PM Revision 8657: schemas/VegCore/VegCore.ERD.mwb.run: export_ (): always overwrite existing files. this option is safe, because -u only extracts files with a newer mtime than their copies on disk.
- 10:55 PM Revision 8656: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps labels: converted tables to notes, which now have bold formatting so that empty tables aren't needed for formatting purposes
- 10:25 PM Revision 8655: schemas/VegCore/VegCore.ERD.mwb.run: import/export_ (): use zip/unzip's -u flag instead of `make` to auto-sync the zip file contents. the -u flag has the added benefit of preserving the contained file's mtime, at the expense of an mtime lookup in the zip archive's table of contents.
- 10:17 PM Revision 8654: added schemas/VegCore/VegCore.ERD.mwb.run, which syncs VegCore.ERD.mwb's zip file contents
- 09:58 PM Revision 8653: bugfix: lib/util.run: inline_make (): put echo_func at the top of the function
- 09:58 PM Revision 8652: lib/util.run: inline_make (): pass Makefile commands using /dev/fd/3 (i.e. 3<<'EOF' ...) instead of stdin, to allow stdin to be used by commands that prompt the user
- 09:44 PM Revision 8651: lib/util.run: inline_make (): use <() to pass Makefile commands as a named descriptor rather than using stdin, to allow stdin to be used by commands that prompt the user
- 09:27 PM Revision 8650: bugfix: lib/util.run: echo_vars (): remove any value to set (=*) from the vars, since this is often used with export
- 09:14 PM Revision 8649: bugfix: lib/util.run: echo_export (): set the vars before echoing them
- 08:55 PM Revision 8648: lib/util.run: run_args_cmd (): use new canon_rel_path() to resolve .. in the script path, so it's easier to see where the script is actually located
- 08:52 PM Revision 8647: lib/util.run: echo_func (): use new canon_rel_path() to resolve .. in the script path, so it's easier to see where the script is actually located
- 08:51 PM Revision 8646: lib/util.run: added canon_rel_path ()
- 08:40 PM Revision 8645: lib/util.run: inline_make (): provide $self as make var
- 08:35 PM Revision 8644: lib/util.run: inline_make (): provide self_dir to make as env var instead of as make var, to avoid specifying every make var both as a local var and as a make var in the make command
- 08:31 PM Revision 8643: lib/util.run: export alias: added comment explaining what it does
- 08:30 PM Revision 8642: lib/util.run: aliased export to echo_export so all env vars are automatically echoed when they are set
- 08:29 PM Revision 8641: lib/util.run: added echo_vars (), echo_export ()
- 08:17 PM Revision 8640: lib/util.run: inline_make (): added local $self and use it to create $self_dir
- 08:15 PM Revision 8639: lib/util.run: inline_make (): set $self_dir as a local var before passing it to make, to avoid clutter in the command that invokes make
- 07:54 PM Revision 8638: added schemas/VegCore/document.mwb.xml, so changes to the diagram can be tracked line-by-line. keeping this up-to-date requires running `unzip VegCore.ERD.mwb document.mwb.xml` whenever VegCore.ERD.mwb is changed.
- 07:53 PM Revision 8637: schemas/VegCore/VegCore.ERD.mwb: target MySQL version: updated to 5.5.3
- 07:49 PM Revision 8636: schemas/VegCore/VegCore.ERD.mwb: NoteFigure:TextFont (labels): made it bold to stand out better
- 07:45 PM Revision 8635: schemas/VegCore/VegCore.ERD.mwb: Layer:TitleFont (legends): increased size and made it bold to stand out better
- 07:31 PM Revision 8634: schemas/VegCore/VegCore.ERD.mwb: made stratum extend method, so that any observation event can have an associated stratum. removed no longer needed plot_stratum, since strata are now represented as sampling methods (i.e. using height classes) rather than a variant of subplots.
- 07:21 PM Revision 8633: schemas/VegCore/VegCore.ERD.mwb: VegCore DB: changed collation to utf8_bin to match table collation
- 07:15 PM Revision 8632: schemas/VegCore/VegCore.ERD.mwb: saved settings for publishing schema to vegbiendev phpMyAdmin (vegbiendev.nceas.ucsb.edu/phpmyadmin/index.php?db=VegCore). this adds `DROP TABLE IF EXISTS` before each CREATE TABLE statement, renames the default DB to VegCore, and adds a USE statement to select the VegCore DB.
- 06:29 PM Revision 8631: schemas/VegCore/connections.xml: vegbiendev.nceas.ucsb.edu: set schema (DB) to VegCore
- 06:20 PM Revision 8630: added schemas/VegCore/connections.xml, which should be installed as the connections.xml settings file
- 06:19 PM Revision 8629: schemas/VegCore/VegCore.ERD.mwb: imported database connections from the connections.xml settings file into document.mwb.xml in the .mwb zip file. these connections are not used directly by MySQL Workbench, but instead need to be copied into the connections.xml settings file.
- 05:17 PM Revision 8628: schemas/VegCore/VegCore.ERD.mwb: use Database > Forward Engineer (Ctrl+G) instead of File > Export > Forward Engineer SQL CREATE Script (Ctrl+Shift+G), so that comments will use the max comment length for the *server's* MySQL version rather than MySQL Workbench's internal MySQL version, which has a max comment length of only 60 chars for tables. this commit stores the default export settings for Forward Engineer, which are saved each time it's run.
- 04:31 PM Revision 8627: bugfix: root Makefile: $(postgresReload-Linux): fixed the "shmat(id=...) failed: Cannot allocate memory" error by telling the user to restart PostgreSQL manually from the command line
- 04:29 PM Revision 8626: root Makefile: $(postgresReload-Linux): don't restart Postgres twice, because this does not fix the "shmat(id=...) failed: Cannot allocate memory" error. this error is instead produced whenever PostgreSQL is restarted from *within a Makefile*, but not when it's run directly from the command line or /bin/sh.
- 03:20 PM Revision 8625: root Makefile: $(postgresReload-Linux): restart Postgres twice, because the first time after the postgresql.conf has been udpated, it always produces a "shmat(id=...) failed: Cannot allocate memory" error
- 03:10 PM Revision 8624: schemas/VegCore/VegCore.ERD.mwb: re-synced with VegCore.my.sql: step 2: `DEFAULT NULL` added on all nullable columns
- 03:10 PM Revision 8623: schemas/VegCore/VegCore.ERD.mwb: re-synced with VegCore.my.sql: step 1: table order changed
- 02:54 PM Revision 8622: schemas/VegCore/VegCore.ERD.mwb: relationship: renamed the two fk_relationship_record1 fkeys so their names wouldn't collide
- 02:52 PM Revision 8621: schemas/VegCore/VegCore.ERD.mwb: changed encoding back to utf8/utf8_bin now that latin1 isn't needed to make the key columns satisfy the index length limits. note that the connection encoding must be utf8 on import so that the correct byte sequence is stored in the VARBINARY columns.
- 02:11 PM Revision 8620: schemas/VegCore/VegCore.ERD.mwb: changed VARCHAR BINARY columns to VARBINARY because VARCHAR BINARY is actually only an alias for the *_bin collation of the table's encoding, but still uses multibyte characters if the encoding does. note that VARBINARY will still be displayed as text (in the connection encoding), rather than hex values.
- 02:02 PM Revision 8619: schemas/VegCore/VegCore.ERD.mwb: changed the type of TEXT columns used in indexes to VARCHAR(767) BINARY to satisfy prefix length limits ("a prefix can be up to [...] 767 bytes for InnoDB tables" <http://dev.mysql.com/doc/refman/5.5/en/create-index.html>). the BINARY flag was added to allow storing Unicode values (from UTF-8), which are not representable in the latin1 encoding.
- 01:15 PM Revision 8618: schemas/VegCore/VegCore.ERD.mwb: changed encoding to latin1 to fit more characters in length-limited indexes (indexes treat utf8 as having *3* bytes/character (http://stackoverflow.com/questions/1814532/1071-specified-key-was-too-long-max-key-length-is-767-bytes#comment-18031137), which significantly reduces the max length of an indexed column)
- 01:08 PM Revision 8617: schemas/VegCore/VegCore.ERD.mwb: changed collation to utf8_bin to avoid case-*in*sensitive comparison of case-sensitive strings. (utf8_bin is also much faster than encoding-aware collations because it uses the raw bytes.)
- 12:47 PM Revision 8616: schemas/VegCore/VegCore.ERD.mwb: set DefaultPkColumnType to TEXT to match the type of current pkey columns
- 12:16 PM Revision 8615: schemas/VegCore/VegCore.ERD.mwb: set all tables to use utf8/utf8_unicode_ci rather than latin1/latin1_swedish_ci because our DB is in UTF-8. utf8_unicode_ci is better than utf8_general_ci because it supports international characters better (http://forums.mysql.com/read.php?103,187048,188748)
- 12:10 PM Revision 8614: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps labels: removed fkey columns to the next step since these are not needed and were causing problems with round-trip sync to the .my.sql file
- 11:42 AM Revision 8613: bugfix: schemas/VegCore/VegCore.ERD.mwb: fixed foreign keys without a referenced columns list
- 11:36 AM Revision 8612: schemas/VegCore/VegCore.ERD.mwb: upgraded for MySQL Workbench 5.2.47 (MySQL 5.5), which requires indexes to have different names than fkeys
- 10:25 AM Revision 8611: schemas/postgresql.conf: shared_buffers: reduced again for Ubuntu VM
- 10:22 AM Revision 8610: bugfix: schemas/postgresql.conf: shared_buffers: removed invalid Unicode character U+FEFF
- 02:40 AM Revision 8609: schemas/postgresql.conf: shared_buffers: reduced to work in Ubuntu VM
- 12:58 AM Revision 8608: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: add * after analytical_aggregate.*.csv to match *.md5 files as well
04/19/2013
- 07:33 PM Revision 8607: inputs/VegBIEN/TWiki/templates/login.tmpl: increased font size further using <span> style. changed text to fit on two lines with the new size.
- 07:26 PM Revision 8606: inputs/VegBIEN/TWiki/templates/login.tmpl: added large text with instructions to register (the TWiki instructions are small and easily missed)
- 07:09 PM Revision 8605: added inputs/VegBIEN/TWiki/templates/login.tmpl
- 05:14 PM Revision 8604: inputs/VegBIEN/TWiki/twiki.conf: main TWiki dir: changed redirect to go directly to the Main TWiki web, so that we don't need to worry about creating different "webs" (subsites) for top-level wiki pages
- 05:03 PM Revision 8603: inputs/VegBIEN/TWiki/twiki.conf: main TWiki dir: redirect shortened wiki page URLs of the form vegpath.org/twiki/Path to do/view/ for use with VegPath
- 04:49 PM Revision 8602: added web/.twiki symlink to datasources/VegBIEN/TWiki/
- 04:48 PM Revision 8601: added inputs/VegBIEN/TWiki/twiki.conf
- 12:58 PM Revision 8600: schemas/VegCore/VegCore.ERD.mwb: coordinates: added latitude_deg, longitude_deg. geovalidation: added lat_long_in_ranks hstore.
- 12:32 PM Revision 8599: schemas/VegCore/VegCore.ERD.mwb: realigned event/occurrence divider. moved method to avoid overlapping the halves.
- 12:29 PM Revision 8598: schemas/VegCore/VegCore.ERD.mwb: specimen: replaced specimen_holder_institutions with specimen_holder_institution many:many table. added fkeys for the various institution fields. collection: removed name because this is already a required field in source. event_participant: removed *_id suffixes from fields.
- 11:55 AM Revision 8597: schemas/VegCore/VegCore.ERD.mwb: replaced party_list with event_participant many:many table, since the nature of each type of party_list (ordered, not, etc.) may differ between tables that HAVE-A party_list, and because it is not possible to create an fkey to a column that is not unique in the referenced table (as is the case for party_list.list_id)
- 11:41 AM Revision 8596: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 11:28 AM Revision 8595: schemas/VegCore/VegCore.ERD.mwb: made first_publisher a field of source (fkey to party) instead of a subclass of it, because a source can be both an aggregator for some datasets and a first publisher for others and which one it is depends on the dataset
- 11:06 AM Revision 8594: schemas/VegCore/VegCore.ERD.mwb: added organization
- 11:01 AM Revision 8593: schemas/VegCore/VegCore.ERD.mwb: added party_list
- 10:29 AM Revision 8592: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 10:09 AM Revision 8591: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 09:50 AM Revision 8590: schemas/VegCore/VegCore.ERD.mwb: taxon_presence, aggregate_observation: added required taxon field to identify (distinguish) the observation itself. this is expected to be the same as the first taxon_determination applied to the observation.
- 09:21 AM Revision 8589: schemas/VegCore/VegCore.ERD.mwb: collection: renamed code to name because the collection code is often the full name of the collection, rather than just an abbreviation. made name required.
- 09:17 AM Revision 8588: schemas/VegCore/VegCore.ERD.mwb: source: added name
04/18/2013
- 04:54 PM Revision 8587: schemas/VegCore/VegCore.ERD.mwb: set page size to fit the ERD contents. this cannot be done through the GUI, so instead you need to use the following steps:
- $ unzip ERD.mwb document.mwb.xml
open document.mwb.xml and search for "this ERD"
units are mm
$ zip -u ERD.mwb docume...
04/17/2013
- 05:28 PM Revision 8586: schemas/VegCore/VegCore.ERD.mwb: fixed split connector
- 05:25 PM Revision 8585: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance lines
- 05:22 PM Revision 8584: schemas/VegCore/VegCore.ERD.mwb: source: added contact field (fkey to party)
- 05:17 PM Revision 8583: schemas/VegCore/VegCore.ERD.mwb: party: inherit from record instead of source, because a party is defined by and scoped by a datasource but isn't a datasource itself (however, a named datasource can have an associated admin party)
- 05:12 PM Revision 8582: schemas/VegCore/VegCore.ERD.mwb: moved tables to make more room between specimen and taxon tables
- 04:45 PM Revision 8581: schemas/VegCore/VegCore.ERD.mwb: split taxon_occurrence into taxon_occurrence and taxon_observation, because a taxon_*occurrence* is actually the *thing* on which a taxon_*observation* is made, not the observation itself. rearranged tables significantly to put observations next to the things they observe, and to make the ERD less wide so it will fit on a small screen without as much scaling (allowing a bigger font size when the ERD is shrunk to fit the screen).
- 02:32 PM Revision 8580: schemas/VegCore/VegCore.ERD.mwb: added primary columns. changed VARCHAR(45) columns to TEXT and also set default type in ERD so future columns won't need to be changed.
- 01:04 PM Revision 8579: schemas/VegCore/VegCore.ERD.mwb: added primary columns, including catch-all hstores for custom columns. made stem inherit from individual and stem_observation from individual_observation because any stem can have individual-level traits and identifying info.
- 11:38 AM Revision 8578: schemas/VegCore/VegCore.ERD.mwb: replaced trait table with traits hstore on taxon_occurrence, since the trait table is basically just a way of allowing the user to add arbitrary measurement columns to the taxon_occurrence. units can be indicated by storing a record with the quantity and units, converted to a string, and parsing it back out as a record upon retrieval.
- 11:29 AM Revision 8577: schemas/VegCore/VegCore.ERD.mwb: renamed taxon_observation to taxon_presence for clarity and to distinguish it better from taxon_occurrence
- 11:25 AM Revision 8576: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: added identified_date and unique index on identifying columns
- 11:15 AM Revision 8575: schemas/VegCore/VegCore.ERD.mwb: taxon_determination: added identified_by. rearranged taxon tables to visually differentiate between inheritance relationships (1:1) and directional HAS-A arrows
- 10:38 AM Revision 8574: schemas/VegCore/VegCore.ERD.mwb: taxon tables: added catch-all hstores for custom columns
- 10:17 AM Revision 8573: schemas/VegCore/VegCore.ERD.mwb: split taxon_concept into taxon_name (containing the components of a name) and taxon_concept (containing the taxon's formal specification and relationships to other taxon_concepts). this avoids requiring a taxon_concept entry for names that are not true taxon concepts. taxon_name: added taxon name/author-related components. TNRS taxonomic scrubbing steps: added dummy id column to avoid sync problems with empty tables.
- 08:38 AM Revision 8572: schemas/VegCore/VegCore.ERD.mwb: taxon tables: added nonidentifying columns
- 06:55 AM Revision 8571: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 06:51 AM Revision 8570: schemas/VegCore/VegCore.ERD.mwb: categories: updated box sizes
- 06:45 AM Revision 8569: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 06:40 AM Revision 8568: schemas/VegCore/VegCore.ERD.mwb: validatable_place: added unique index on identifying columns
- 06:38 AM Revision 8567: schemas/VegCore/VegCore.ERD.mwb: made place_path and coordinates globally-scoped rather than per-datasource, so that there is only one geovalidation row for each combination of them rather than one per datasource place. added globally-scoped validatable_place table that stores combinations of place_path and coordinates, and made geovalidation extend it (since geovalidation provides optional fields and is 1:1 with it).
- 06:06 AM Revision 8566: schemas/VegCore/VegCore.ERD.mwb: made geovalidation extend place_path instead of being pointed to by it, because it provides a set of optional fields a (geovalidated) place_path may have
- 05:33 AM Revision 8565: schemas/VegCore/VegCore.ERD.mwb: made place_path extend place instead of place having a place_path, because place_path now provides an optional set of fields that a place may provide, rather than being a completely separate entity
- 05:20 AM Revision 8564: schemas/VegCore/VegCore.ERD.mwb: renamed named_place to place_path since place now serves the purpose that named_place used to (i.e. representing geographical areas)
- 05:13 AM Revision 8563: schemas/VegCore/VegCore.ERD.mwb: removed invisible U+FEFF Unicode character
- 05:10 AM Revision 8562: schemas/VegCore/VegCore.ERD.mwb: renamed location to place because any place can be a location, not just a plot or point (e.g. Barro Colorado Island is a CTFS "plot" but definitely much larger than the typical plot)
- 05:03 AM Revision 8561: schemas/VegCore/VegCore.ERD.mwb: labeled TNRS taxonomic scrubbing steps
- 04:42 AM Revision 8560: schemas/VegCore/VegCore.ERD.mwb: labeled TNRS taxonomic scrubbing steps
- 03:52 AM Revision 8559: schemas/VegCore/VegCore.ERD.mwb: geovalidation: put named_place before coordinates so they would be in path order (one named_place can have many coordinates in it)
- 03:43 AM Revision 8558: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
- 03:39 AM Revision 8557: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 03:30 AM Revision 8556: schemas/VegCore/VegCore.ERD.mwb: taxon_assertion: added fkey to taxon_concept/taxon_path to store any parsed taxonomic fields provided by the data provider
- 03:04 AM Revision 8555: schemas/VegCore/VegCore.ERD.mwb: linked taxon_concept directly to taxon_string rather than to child table taxon_name, because it creates unnecessary confusion to have two tables storing taxonomic strings. a taxon_string is anything that can be used as input to TNRS, and may or may not include a cf_aff.
- 02:57 AM Revision 8554: schemas/VegCore/VegCore.ERD.mwb: link each taxon_string to a parsed_taxon_assertion. removed verbatim_taxon_assertion because that function is now performed by taxon_string.
- 02:48 AM Revision 8553: schemas/VegCore/VegCore.ERD.mwb: added taxon_name, which is used by taxon_concept
- 02:37 AM Revision 8552: schemas/VegCore/VegCore.ERD.mwb: added taxon_string, which is used by taxon_assertion
- 02:15 AM Revision 8551: schemas/VegCore/VegCore.ERD.mwb: added unique indexes on identifying columns where possible. unique indexes differ from primary keys in that they are generally on multiple columns, while pkeys are on a text field which is the concatenation of the identifying columns.
- 01:49 AM Revision 8550: schemas/VegCore/VegCore.ERD.mwb: hierarchical tables: made parent pointer NOT NULL so that it can be an identifying column. this requires the root(s) of the hierarchy to be indicated with a self-referential parent pointer rather than a NULL value. this also allows materializing parent elements to arbitrary levels without worrying about hitting the root, because the root's ancestors will simlpy be itself for as many levels as are needed (i.e. materializing several levels up the tree is guaranteed to produce a non-NULL entry for all levels).
- 01:36 AM Revision 8549: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 01:35 AM Revision 8548: schemas/VegCore/VegCore.ERD.mwb: taxon_assertion: removed fkey to taxon_concept because only parsed_taxon_assertions have a corresponding taxon_concept
- 01:31 AM Revision 8547: schemas/VegCore/VegCore.ERD.mwb: fixed table positions
- 01:25 AM Revision 8546: schemas/VegCore/VegCore.ERD.mwb: lined up tables
- 01:24 AM Revision 8545: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 01:21 AM Revision 8544: schemas/VegCore/VegCore.ERD.mwb: parsed_taxon_assertion.matched_taxon: changed it to be an fkey to taxon_path, because the parsing must also populate the denormalized path fields in addition to defining a taxon_concept with the concatenated scientific name
- 01:18 AM Revision 8543: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance lines
- 01:17 AM Revision 8542: schemas/VegCore/VegCore.ERD.mwb: added taxon_path
- 01:10 AM Revision 8541: schemas/VegCore/VegCore.ERD.mwb: geological_context: made it the same color as community because it is a type of community for the geological aspects of the location
- 01:01 AM Revision 8540: schemas/VegCore/VegCore.ERD.mwb: renamed fkeys to match renamed tables
- 12:34 AM Revision 8539: schemas/VegCore/VegCore.ERD.mwb: merged synonym_taxon, accepted_taxon into taxon_concept because *any* taxon_concept can have an accepted taxon which indicates the canonical name for that taxon, not just TNRS match results
- 12:15 AM Revision 8538: schemas/VegCore/VegCore.ERD.mwb: merged taxon_concept and taxon, because every taxon is according to someone (whether that's Weakley, Tropicos, or the person identifying a taxon_occurrence). note that the according_to is usually different from the source (which is the datasource that owns the record), but if there is no specific according_to, the according_to is the datasource itself.
04/16/2013
- 11:00 PM Revision 8537: schemas/VegCore/VegCore.ERD.mwb: lightened table colors so dark text is easier to see against them. used colors from VegBank ERD (http://vegpath.org/fs/schemas/VegBank/vegbank.ERD.pdf) where possible, to have a familiar look-and-feel for those who have seen the VegBank ERD.
- 10:38 PM Revision 8536: schemas/VegCore/VegCore.ERD.mwb: categories: added stratum
- 10:29 PM Revision 8535: schemas/VegCore/VegCore.ERD.mwb: categories: put categories on the same halves of the legend as they are in the ERD
- 10:23 PM Revision 8534: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 10:23 PM Revision 8533: schemas/VegCore/VegCore.ERD.mwb: added stratum, plot_stratum
- 06:14 PM Revision 8532: schemas/VegCore/VegCore.ERD.mwb: added plot subclass of location
- 05:49 PM Revision 8531: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance lines
- 05:45 PM Revision 8530: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 05:34 PM Revision 8529: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 05:31 PM Revision 8528: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 05:28 PM Revision 8527: schemas/VegCore/VegCore.ERD.mwb: removed redundant "DEFAULT NULL" clauses on nullable fields
- 05:24 PM Revision 8526: schemas/VegCore/VegCore.ERD.mwb: renamed taxon_occurrence.*_taxon_determination to taxon_occurrence.*_determination because the column name does not need to be globally unique, and can therefore be shorter while still being unambiguous within the table
- 04:34 PM Revision 8525: schemas/VegCore/VegCore.ERD.mwb: fixed lines
- 04:33 PM Revision 8524: schemas/VegCore/VegCore.ERD.mwb: changed all fkeys to CASCADE to support cascading deletes. this allows datasources to be reloaded, because the corresponding source row can be deleted, and the delete will cascade to every record that's part of that datasource.
- 04:25 PM Revision 8523: schemas/VegCore/VegCore.ERD.mwb: changed globally unique column names to locally unique column names, now that the SQL dotpath algorithm (http://vegpath.org/wiki/SQL_dotpaths#Algorithm) can be used to generate globally unique column names automatically according to the fkeys. in particular, this means that the pkeys have all been renamed to "id".
- 02:30 PM Revision 8522: schemas/VegCore/VegCore.my.sql: removed default_schema qualifier from names because all objects are in the same schema
- 02:29 PM Revision 8521: schemas/VegCore/VegCore.my.sql: regenerated from ERD
- 09:25 AM Revision 8520: bugfix: schemas/VegCore/VegCore.ERD.mwb: fixed pkey names to match table names
- 09:18 AM Revision 8519: schemas/VegCore/VegCore.ERD.mwb: renamed matched_taxon table to synonym_taxon because it can be used to store any taxon which has a corresponding accepted_taxon, not just the match results from TNRS
- 09:05 AM Revision 8518: schemas/VegCore/VegCore.ERD.mwb: renamed qualified_taxon tables to *taxon_assertion because they are not just a taxon name plus an optional cf/aff qualifier, but an actual operator/value pair that is the right-hand side of a comparison with a taxon or taxon_occurrence
- 08:43 AM Revision 8517: bugfix: schemas/VegCore/VegCore.ERD.mwb: qualified_taxon subclasses: changed HAS-A arrows to point in the other direction, because the many:1 relationship in fact narrows towards the accepted_taxon. the arrows used to be in the opposite direction because of the way the TNRS results are attached incrementally.
- 08:28 AM Revision 8516: README.TXT: Maintenance: to backup the Ubuntu VM: removed invisible U+FEFF Unicode character
- 08:26 AM Revision 8515: README.TXT: added steps to backup the Ubuntu VM
- 07:48 AM Revision 8514: bugfix: schemas/.htaccess: corrected $innermost_dir
- 07:46 AM Revision 8513: added .htaccess files for inputs/, schemas/ because these reside outside the DOCUMENT_ROOT, and therefore need an explicit RewriteBase directive so that RewriteRules in their subdirs can construct self-referential URLs. this ensures that the input URLs to RewriteRules are always URL paths instead of filesystem paths, which tend to mess up the redirects. (mod_rewrite is capable of doing filesystem-only redirects, but the associated RewriteRules require special care to ensure they match and reconstruct the filesystem path properly.) the presence of filesystem paths in RewriteRule inputs had been causing numerous bugs, especially in rules that use %{REQUEST_URI}, and the addition of the RewriteBases will hopefully avoid these types of bugs in the future. note that *every* path that resides outside the DOCUMENT_ROOT needs an explicit RewriteBase in its top-level .htaccess.
- 07:14 AM Revision 8512: schemas/VegCore/VegCore.ERD.mwb: ensured that inheritance relationships are drawn as 1:1
- 07:00 AM Revision 8511: web/**/.htaccess: appended query strings for external redirects: just use ? rather than ?& when an external redirect will be perfomed, because no dotpath parsing will be performed on the query string first (which would necessitate the & if the new query string looks like a dotpath)
04/14/2013
- 09:30 PM Revision 8510: schemas/VegCore/VegCore.ERD.png: Regenerated again because some tables were highlighted when it was exported, causing the highlighting to appear in the PNG
- 09:27 PM Revision 8509: schemas/VegCore/VegCore.ERD.mwb: Regenerated exports
- 09:26 PM Revision 8508: schemas/VegCore/VegCore.ERD.mwb: shortened embedded ERD URL using new abbreviations
- 09:19 PM Revision 8507: web/**/.htaccess: RewriteRules: ensured flags are in alphabetical order
- 09:10 PM Revision 8506: removed no longer needed web/dotpath.php. use web/.htaccess's parse-dotpath RewriteRule instead.
- 09:07 PM Revision 8505: web/.htaccess: parse dotpath in query string: moved parsing from dotpath.php to RewriteRules. with the eventual addition of config env vars, this makes it possible for subdirs to configure their own dotpath parsing, rather than having *all* levels of the dotpath translated to a URL path. (for inner levels, it may el.g. be more appropriate to parse the dotpath as query params or pass it along to an external URL.) note that it takes only two lines of mod_rewrite (plus a workaround RewriteRule) to replace over a dozen lines of PHP code.
- 08:54 PM Revision 8504: web/.htaccess: renamed REQUEST_URI_no_/ to REQUEST_URI_no_extra_/ to clarify that only an excess / is removed
- 08:52 PM Revision 8503: web/.htaccess: moved set REQUEST_URI_no_/ under translate subdomain to path because it needs to be set right before being used to be accurate
- 08:44 PM Revision 8502: web/.htaccess: setting REQUEST_URI_no_/: use one RewriteCond instead of two since the regexps can be combined
- 02:50 PM Revision 8501: web/.htaccess: parse dotpath in query string: removed dotpath auto-detection code because this is now handled by a separate RewriteRule
- 02:49 PM Revision 8500: web/.htaccess: auto-detect dotpath in query string (having a . before any = ) and mark it using a leading . . this is done as a separate step so that the dotpath is already explicitly marked as such when it's parsed.
- 02:01 PM Revision 8499: web/dotpath.php: just match the QUERY_STRING directly, excluding &, instead of splitting apart the QUERY_STRING and then matching the dotpath portion. this allows the resulting regexp to be used directly in a .htaccess file.
- 01:08 PM Revision 8498: web/.htaccess: prepend dir name if it's part of the filename: support MultiViews filenames without the extension, by using a lookahead request -F instead of a regular -f
- 01:02 PM Revision 8497: schemas/VegCore/.htaccess: added RewriteBase to support relative redirects. this enables auto-prepending dir name if it's part of the filename, e.g. VegCore/ERD.pdf -> VegCore/VegCore.ERD.pdf).
- 12:59 PM Revision 8496: web/.htaccess: prepend dir name if it's part of the filename (this works only if the dir supports relative redirects and sets %{ENV:innermost_dir})
- 11:44 AM Revision 8495: schemas/VegCore/VegCore.ERD.mwb: made matched_taxon and accepted_taxon inherit from taxon instead of qualified_taxon because they don't contain the cf_aff (cf_aff.vegpath.org; aka identificationQualifier) that makes a taxon qualified
- 11:16 AM Revision 8494: schemas/VegCore/VegCore.ERD.mwb: removed occurrence table, because it's really a synonym for taxon_occurrence ("specimen, observation, etc." <http://rs.tdwg.org/dwc/terms/#Occurrence>)
- 10:48 AM Revision 8493: schemas/VegCore/VegCore.ERD.mwb: added labels for the left and right halves (event and occurrence). rearranged tables to help separate event- and occurrence-related tables.
- 10:17 AM Revision 8492: schemas/VegCore/VegCore.ERD.mwb: rearranged to leave more space for location_observation, and to put all event-related fields on the left and occurrence-related fields on the left
- 09:32 AM Revision 8491: web/.htaccess: parse dotpath in query string: heuristically detect query strings that don't contain dotpaths (such as mod_autoindex sort conditions), rather than always interpreting the part before the first & as the dotpath. now, the part before the first & will be interpreted as a dotpath if it has a . before any = . (query param names containing . are rare, so this usually works correctly.) note that you can always prepend . to force a dotpath, or & to force a query param.
- 09:06 AM Revision 8490: web/dotpath.php: remove any . prefix which forces interpretation as a dotpath
- 07:38 AM Revision 8489: schemas/VegCore/VegCore.ERD.mwb: taxon_concept: renamed source to according_to
- 07:06 AM Revision 8488: schemas/VegCore/VegCore.ERD.mwb: removed no longer used images from zip archive using `zip -d`. this needs to be done every time an image is deleted from the diagram to remove it from the archive (and reclaim the unused space, since `zip -d` will also shrink the archive). note that you can't just expand and re-zip the files, because there is a header of some sort that indicates to MySQL Workbench that it's a valid .mwb file.
- 06:27 AM Revision 8487: bugfix: schemas/VegCore/VegCore.ERD.mwb: also fixed inheritance relationships to the record table to be 1:1 (by search-and-replacing and then updating document.mwb.xml in the .mwb zip archive using `zip -u`). added separate legend for relationships and connector styles, which now uses example tables instead of screenshot images to ensure accurate sizing at all ERD zoom levels.
- 05:17 AM Revision 8486: bugfix: schemas/VegCore/VegCore.ERD.mwb: fixed inheritance relationships to be 1:1 with optional participation by the derived table, rather than the incorrect 1:many relationshipwhich MySQL Workbench defaults to. (in OOP, each derived instance has its *own* instance of the base class, because it is a separate object.)
- 05:07 AM Revision 8485: web/**/.htaccess: When there is *no* external redirect for the directory without the trailing /, instead auto-add the trailing / to ensure mod_autoindex relative links work properly. This must be done manually because the top-level .htaccess will not auto-add the trailing / if the dir has an .htaccess file (to allow the custom external redirects). There are actually only two .htaccess files that this applies to (web/.phpPgAdmin/.htaccess, .phpMyAdmin/.htaccess), indicating that dirs which have any kind of virtual paths (redirects) almost always have a "root" redirect when there is no specific subpath. This root redirect is intended to link to the datasource's *own* index of its available resources, e.g. the list of tables for a database source.
- 04:48 AM Revision 8484: removed web/aggregators/.htaccess because it was only there to auto-add the trailing /, but this is now done by web/.htaccess
- 04:29 AM Revision 8483: bugfix: web/.htaccess: auto-add trailing /: %{LA-F:REQUEST_FILENAME} lookaheads don't work, because they do not show the externally redirected URL, so we need to use an alternate mechanism instead, which just assumes that any dir containing an .htaccess file may contain a redirect and ignores just those dirs (subdirs are not ignored and will receive a trailing / as expected). To create the trailing / effect when you have a .htaccess *without* an external redirect, you need to manually tell it to correct the path for just that dir without the / (i.e. match ^/ , because a trailing / causes mod_rewrite to use an absolute filesystem path instead).
- 04:21 AM Revision 8482: web/.htaccess: moved auto-add trailing / to after don't rewrite existing dirs, so that the path will automatically not be rewritten if it contains a trailing / and is a dir
- 04:18 AM Revision 8481: web/.htaccess: don't rewrite existing paths: only prevent existing dirs *with trailing /* from being rewritten, because dirs without trailing / may be redirected
- 01:31 AM Revision 8480: removed web/**/.htaccess whose only purpose was to auto-add the trailing / (which is now handled by the top-level .htaccess)
- 01:30 AM Revision 8479: web/**/.htaccess: removed no longer needed auto-add of trailing / , which is now implemented for all dirs by the top-level .htaccess
- 01:27 AM Revision 8478: web/.htaccess: auto-add trailing / if not external redirect: added test if `%{REQUEST_FILENAME} -d` in addition to checking %{LA-F:REQUEST_FILENAME}, because there are some cases (e.g. subdirs that are symlinked outside the DOCUMENT_ROOT) where LA-F:REQUEST_FILENAME for some reason appends /home to the REQUEST_FILENAME, but the current REQUEST_FILENAME is fortunately already a dir and can be checked directly
- 01:07 AM Revision 8477: web/.htaccess: auto-add trailing / if not external redirect: removed no longer applicable comment that dir without trailing / is provided as an absolute filesystem path, because this is only true in inner .htaccess files for their top-level dirs
- 01:01 AM Revision 8476: bugfix: web/.htaccess: auto-add trailing / if not external redirect: match any path that does not end in / (and satisfies the -d lookahead), rather than only absolute filesystem paths, which are only applicable in inner .htaccess files (and even then only for the outermost dir they control) but not in the top-level .htaccess file. Note that the inner .htaccess files can still use ^/ to match *their own* dir, just not their *subdirs*, without a trailing / .
- 12:25 AM Revision 8475: bugfix: web/.htaccess: auto-add trailing / if not external redirect: replace with "%{REQUEST_URI}/" instead of the empty string (which gets auto-translated to have a trailing /), because the empty string only works in the top-level dir of the .htaccess file (it redirects to the RewriteBase, which ends in /). The entire REQUEST_URI is needed to do the auto-add in dirs which do not have their own auto-add trailing / rule. Also, using REQUEST_URI avoids the need to (sometimes) set RewriteBase for use by *relative* URLs in replacement strings, because this is an absolute URL instead. Setting RewriteBase manually is best avoided, because it hardcodes the URL path to the .htaccess file, which may change.
- 12:17 AM Revision 8474: bugfix: web/.htaccess: auto-add trailing / if not external redirect: use "%{LA-F:REQUEST_FILENAME} -d" instead of "$0 -F" so that the actual value of REQUEST_FILENAME tested, and $0 can also match strings which are not file paths. This is because only those no-trailing-/-paths *which did not match the RewriteBase* are provided to mod_rewrite as absolute filepaths, but RewriteBase is not an issue in the outermost .htaccess (because the RewriteBase is / , which is always matched). This means the method of testing for ^/ to match a no-trailing-/-path *will only work in inner .htaccess files* (where it is currently used extensively).
04/13/2013
- 05:47 PM Revision 8473: web/.htaccess: bugfix: moved auto-add trailing / if not external redirect to before don't rewrite existing paths because the trailing / also needs to be added on paths which *are* existing paths, but are just missing the trailing /
- 08:56 AM Task #565: partition the TaxonDetermination table by row into the different types of determinations
- changed vertical/horizontal to row/column for clarity
- 06:37 AM Revision 8472: web/.htaccess: auto-add trailing / if not external redirect (limited form of DirectorySlash). this allows URLs without trailing /, for dirs which have a special external redirect to the datasource's homepage when there is no trailing /, but still auto-adds the trailing / for all other dirs.
- 06:17 AM Revision 8471: README.TXT: Debugging: mod_rewrite: cached redirects issue: changed to limit the cache clearing to when a *DirectorySlash*-related redirect is changed, since only those redirects are permanent, not the RewriteRule redirects (unless you explicitly set them as permanent)
- 06:03 AM Revision 8470: README.TXT: Debugging: mod_rewrite: added tip that whenever you change a RewriteRule, you *must* clear your browser's cache to ensure that a cached redirect is not used instead (temporary redirects are not supposed to be cached, but sometimes Firefox caches them anyway)
- 04:41 AM Revision 8469: README.TXT: umask: Removed -S because this causes the new umask to be echoed, and is not needed even when setting a symbolic umask
- 04:18 AM Revision 8468: README.TXT: Datasource setup:: added step to use umask to prevent files from becoming web-accessible
- 04:16 AM Revision 8467: README.TXT: Testing: what to put in your .profile: added `umask -S ug=rwx,o=` to prevent files from becoming web-accessible
- 04:05 AM Revision 8466: README.TXT: fixed leading whitespace
- 02:46 AM Revision 8465: lib/util.run: run_cmd (): renamed to run_args_cmd to clarify that this runs only the command line args command, not any command. added doc comment to clarify this.
- 02:41 AM Revision 8464: lib/util.run: EXIT trap: use on_exit() function which calls run_cmd instead of having run_cmd be the handler, so that users can override this function to perform other commands (or no commands) after the script is read
- 02:28 AM Revision 8463: lib/util.run: echo_func (): use the caller's FUNCNAME via the $FUNCNAME[] array instead of requiring them to pass it in the function args as `echo_func "$FUNCNAME" "$@"`
- 02:11 AM Revision 8462: README.TXT: Maintenance: added steps to synchronize a Mac's settings with my testing machine's
- 02:10 AM Revision 8461: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: moved swap=1 right before src= dest=, which it applies to. aligned whitespace of the various commands.
- 01:49 AM Revision 8460: README.TXT: Maintenance: steps to synchronize vegbiendev, jupiter, and your local machine: added steps to install dependencies
- 01:44 AM Revision 8459: README.TXT: Maintenance: Added steps to synchronize vegbiendev, jupiter, and your local machine when changes are made on vegbiendev
- 01:34 AM Revision 8458: README.TXT: converted spaces to tabs for easier readability in a whitespace-highlighting text editor, and so that pressing backspace deletes an indent rather than just a character (creating a misaligned indent)
04/11/2013
- 10:29 PM Revision 8457: Moved validation/ under planning/workflow/
- 10:28 PM Revision 8456: Moved planning/use_cases/ under planning/goals/
- 10:26 PM Revision 8455: Moved planning/milestones/ under planning/timeline/
- 10:23 PM Revision 8454: Added inputs/VegBIEN/planning shortcut to fs/planning
- 10:16 PM Revision 8453: web/.fs: symlink via datasources/VegBIEN/fs instead
- 10:14 PM Revision 8452: Added inputs/VegBIEN/fs symlink to /servers/vegbiendev/fs
- 05:56 PM Revision 8451: Added web/.gbif shortcut to datasources/GBIF
- 04:52 PM Revision 8450: web/**/.htaccess: Use [qsappend] flag with all RewriteRules to ensure that the original query string is never lost
- 04:30 PM Revision 8449: inputs/VegBank/.htaccess: Added RewriteCond to check that the REQUEST_FILENAME is not an existing dir, which is apparently needed to prevent the verify/ subdir from being interpreted as a redirect
- 02:45 PM Revision 8448: web/**/.htaccess: Removed preprocessing commands that need to run for every dir, because these are now inherited from the outermost .htaccess using InheritBefore. This allows the inner .htaccess files to focus just on the fallback rewrites, without needing lots of boilerplate code (except for the obligatory RewriteEngine/RewriteOptions/SetEnvIf preamble).
- 02:34 PM Revision 8447: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex, by deleting the /index filename. This ensures that any fallback redirect is not invoked, and instead the action corresponding to a trailing / (usually the display of a directory listing) is performed.
- 02:07 PM Revision 8446: web/**/.htaccess: Use InheritBefore RewriteOption so that the outermost dir's rules are always run first. This is necessary so that all the HTTP_HOST and dotpath handlers, etc. are applied before any fallback handlers, *without* needing to copy and paste each rule into *every* .htaccess file. (Note that the InheritBefore RewriteOption is only available in Apache 2.4.) The outermost .htaccess file, however, must *not* use InheritBefore because this would inherit rules from the global Apache config, which normally should not be run and which cause an infinite redirect loop.
- 01:56 PM Revision 8445: web/**/.htaccess: fallback rules which should not be inherited by subdirs: Filter using a RewriteCond on %{ENV:innermost_dir} so that the rule is only run if the innermost dir was this dir. (This is needed when using InheritBefore.)
- 01:36 PM Revision 8444: web/**/.htaccess: Added SetEnvIf directives to set the $innermost_dir env var, which is used by InheritBefore RewriteRule inheritance. You can easily populate the current dir name in jEdit by right-clicking the buffer tab and selecting Insert File Path (which requires the BufferTabs plugin), and then picking out the dir right before the .htaccess. If multiple dirs in the dir path have the same name (e.g. a/b/a/), add a distinguishing prefix to the inner dirs (this is rare).
04/10/2013
- 11:17 PM Revision 8443: web/.htaccess: Support rules being inherited by subdirs using InheritBefore, by setting an env var $innermost_dir that stores the name of the innermost subdir to contain an .htaccess file. This env var can then be tested for in a RewriteCond in rules that should only apply locally. This works because SetEnvIf processes directives from the outer dir to the inner dir, so that the inner dir's setting comes last and is the final value of $innermost_dir. Note that you must use SetEnvIf rather than SetEnv because "The internal environment variables set by [SetEnv] are set after most early request processing directives are run, such as access control and URI-to-filename mapping. If the environment variable you're setting is meant as input into this early phase of processing *such as the RewriteRule directive*, you should instead set the environment variable with SetEnvIf." [emphasis added] (http://httpd.apache.org/docs/current/mod/mod_setenvif.html#setenvif)
- 10:20 PM Revision 8442: web/.htaccess: don't rewrite existing paths: Changed match string to what is used in subdirs, so that if this rule is inherited, what is considered an existing path (i.e. a file or subdir) is restricted appropriately
- 10:18 PM Revision 8441: web/**/.htaccess: no-replacement RewriteRules: Removed the standard [discardpath,noescape] flags because they add unnecessary clutter and are not needed for this type of rule. Note that these flags are still required on regular RewriteRules which have a replacement string.
- 10:15 PM Revision 8440: web/**/.htaccess: no-replacement RewriteRules: Use just ^ as the match string because there is no need to actually match characters. This also makes it easy to visually tell what is a no-replacement RewriteRule vs. a RewriteRule that matches the entire path.
- 10:11 PM Revision 8439: web/.htaccess: Renamed $REQUEST_URI_NO_SLASH to $REQUEST_URI_no_/ to take advantage of (some) special chars being allowed in env var names
- 09:42 PM Revision 8438: Added web/people/
- 09:08 PM Revision 8437: web/.htaccess: fallback redirect: Use new VegCore/ instead of specifying the full wiki URL
- 09:07 PM Revision 8436: schemas/VegCore/.htaccess: Fixed bug where need to issue explicit redirect because path contains a fragment (#...), which must be sent to the browser
- 09:03 PM Revision 8435: Added schemas/VegCore/.htaccess
- 08:57 PM Revision 8434: Added web/.vegcore shortcut to exchange_schemas/VegCore
- 08:34 PM Revision 8433: Added root .htaccess (accessed via web/servers/vegbiendev/fs or one of its shortcuts)
- 08:28 PM Revision 8432: web/.htaccess: translate subdomain to path: Use new REQUEST_URI_NO_SLASH to avoid adding an automatic trailing / when a subdir was reached as a subdomain without a path (e.g. VegBank.vegpath.org), which should be equivalent to going to the subdir's homepage (vegpath.org/VegBank) rather than to its directory listing (vegpath.org/VegBank/ ; note the trailing /)
- 08:25 PM Revision 8431: web/.htaccess: set REQUEST_URI_NO_SLASH for use by translating subdomain to path
- 08:19 PM Revision 8430: Added web/.fs symlink to .vegbiendev/fs
- 01:53 PM Revision 8429: web/svn-web, wiki: Made them hidden so they wouldn't appear in the dir listing (which should not contain shortcuts to inner paths, as these would create significant clutter). web/svn unfortunately can't be made hidden because there is already a .svn dir.
- 01:47 PM Revision 8428: Added web/.nimoy, .vegbiendev shortcuts
- 01:44 PM Revision 8427: web/DwC, VegX: Replaced with versions in exchange_schemas/
- 01:39 PM Revision 8426: Moved web/TCS/ to schemas/
- 01:36 PM Revision 8425: Added web/exchange_schemas/ symlink
- 01:35 PM Revision 8424: Removed web/exchange_schemas/ in preparation for making it a symlink
- 01:34 PM Revision 8423: Added schemas/.htaccess
- 01:33 PM Revision 8422: Added web/exchange_schemas/
- 01:28 PM Revision 8421: Merged web/DwC,VegX with the corresponding dir in schemas/
- 01:19 PM Revision 8420: Moved web/BIEN2, IH into aggregators/
- 01:18 PM Revision 8419: Moved web/BIEN2, IH into aggregators/
- 01:09 PM Revision 8418: Added web/aggregators, primary_databases with symlinks to the applicable datasources
- 01:04 PM Revision 8417: web/.tnrs: Updated to point to datasources/.TNRS/
- 01:01 PM Revision 8416: Removed web/TNRS/ in preparation for replacing it with a symlink
- 12:59 PM Revision 8415: Added inputs/.TNRS/.htaccess
- 12:13 PM Revision 8414: web/.redmine/: Updated for new VegBIEN/ location
- 12:12 PM Revision 8413: web/.htaccess: Use new %{REQUEST_SCHEME} to construct self-refential URLs
- 10:14 AM Revision 8412: web/VegBIEN/: Moved to datasources/
- 10:07 AM Revision 8411: Added inputs/.htaccess, which parses dotpaths and adds trailing / when datasources/ subdir is listed. (DirectorySlash cannot be used for this because it would affect all subdirs, too, which must not have DirectorySlash on. Turning DirectorySlash off in each subdir would not work, because the / is added *as soon as* a valid filesystem path is formed, even if the inner dir would not have added the /.)
- 09:48 AM Revision 8410: web/**/.htaccess: internal redirects: Use ?&$0 instead of just ?$0 to prevent the query string from being reinterpreted as a dotpath by the destination dir
- 09:45 AM Revision 8409: web/VegBIEN/Redmine/*/.htaccess: Use relative URL paths. This requires adding a RewriteBase directive because these dirs can be reached by symlinks.
- 09:40 AM Revision 8408: inputs/SALVIAS/.htaccess: Removed test RedirectMatch directive
- 09:38 AM Revision 8407: web/**/.htaccess: RewriteOptions: Redirect dir paths without the trailing / to the datasource's homepage, and let dir paths with the trailing / point to the directory index. This is the opposite of the way it was before, and is more intuitive because the trailing / indicates a directory listing, while the trailing / would generally be omitted when linking to the datasource itself.
- 07:49 AM Revision 8406: web/**/.htaccess: handle DirectoryIndex subrequests, which append "index" to the dir
- 06:15 AM Revision 8405: web/**/.htaccess: RewriteOptions: Removed inherit because if the destination is not found in the current dir, a 404 should be returned instead of trying to invoke the parent dir's fallback handler (which applies only to paths in the parent dir). Ideally, we would use InheritBefore to get the subdomain preprocessing, etc., but the parent dir's fallback handler would still need to be turned off somehow when the rule was being run by a subdir.
- 05:59 AM Revision 8404: web/.htaccess: Only forward specific terms (subpaths) to VegCore, not also the dir index itself
- 05:15 AM Revision 8403: web/**/.htaccess: RewriteOptions: Added AllowNoSlash (available in Apache 2.4) so dir paths without the trailing / can be parsed by mod_rewrite
- 05:12 AM Revision 8402: web/.htaccess: Turned off DirectorySlash so that the dir without the trailing / can have a separate meaning (e.g. as the home page of a datasource, rather than a listing of the short URLs available for it)
- 03:33 AM Revision 8401: web/main.conf: <Directory .>: Added `Require all granted`, which is needed by Apache 2.4 to prevent 403 Forbidden errors. This change breaks compatibility with older versions of Apache, but is unfortunately required by 2.4.
- 03:29 AM Revision 8400: web/**/.htaccess: parse dotpath in the query string: Use just dotpath instead of dotpath.php because the extension is added automatically by MultiViews
- 03:28 AM Revision 8399: web/**/.htaccess: parse dotpath in the query string: Remove any index.* suffix, which is added by MultiViews in Apache 2.4
- 03:28 AM Revision 8398: web/main.conf: Updated the path of the <Directory> directive for the DocumentRoot
- 03:21 AM Revision 8397: web/.htaccess: Set Options +Indexes because this isn't the default in Apache 2.4
- 01:46 AM Revision 8396: web/.htaccess: DirectoryIndex: Use index instead of hardcoding index.php because MultiViews will now add the appropriate extension automatically
- 01:44 AM Revision 8395: web/.htaccess: Turn on MultiViews to allow auto-adding of the file extension
- 12:15 AM Revision 8394: inputs/**/.htaccess: Use a direct filesystem path rather than going through /svn (the Redmine web interface) because this loads much faster, and avoids needing to manually specify svn-web instead of svn for files that should be displayed in the browser instead of downloaded. It also adds support for files not in svn. Note that the use of a relative RewriteRule replacement requires RewriteBase when the directory is not under the document root, in order to construct the correct self-referential URL (http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritebase).
04/09/2013
- 10:59 PM Revision 8393: inputs/**/.htaccess: Removed no longer needed trailing / in paths
- 10:55 PM Revision 8392: inputs/CTFS/*/.htaccess: Use new direct shortcuts to /Redmine subtrees
- 10:47 PM Revision 8391: Added wiki shortcut to Redmine/wiki
- 10:43 PM Revision 8390: web/.htaccess: use separate lowercase version when available: Support paths without a trailing /
- 10:41 PM Revision 8389: web/.htaccess: use separate lowercase version when available: Fixed bug where need to use [last] flag, in order to rewrite in the new subdir instead of applying the rest of this dir's rules (which would forward to VegCore)
- 10:11 PM Revision 8388: web/: datasources indexed in VegPath: Link to the corresponding inputs/ dir instead, which now stores the data and the short URLs in one place
- 10:10 PM Revision 8387: web/.htaccess: Use %{REQUEST_URI} ("The path component of the requested URI" <http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritecond>) instead of /$0, in order to preserve whether or not the path had a trailing slash. This also ensures that the rule functions correctly if it is invoked in a subdir context, which may occur for the "remove www subdomain" rule.
- 10:01 PM Revision 8386: web/.htaccess: Removed "don't redirect subdir paths" because this is now handled by "don't rewrite existing paths" as described in r8375
- 09:48 PM Revision 8385: web/: datasources indexed in VegPath: Added .htaccess files to corresponding inputs/ dir so the short URLs can also be accessed via VegBIEN's own dir for the datasource
- 09:36 PM Revision 8384: inputs/input.Makefile: Removed .PRECIOUS from %/header.csv, %/map.csv so that these scripts are deleted on error. This is useful for the runscripts and for non-data dirs whose header.csv cannot be made.
- 09:33 PM Revision 8383: inputs/input.Makefile: SVN: Only run %/add on subdirs with visible (non-hidden) files. Subdirs with only hidden files (e.g. .htaccess) are assumed to be non-data dirs.
- 09:23 PM Revision 8382: inputs/input.Makefile: Staging tables installation: %/install: $(logInstall): Only use log file if log dir exists, to support non-data dirs
- 09:13 PM Revision 8381: inputs/input.Makefile: Staging tables installation: %/install: ignore errors in $(exportHeader) and $(cleanup) if the table does not exist (i.e. a non-data dir)
- 09:11 PM Revision 8380: inputs/input.Makefile: Staging tables installation: %/install: Don't run $(import_install_) for empty dirs because there is no data to import
- 04:37 PM Revision 8379: lib/sql.py: parse_exception(): typed_name_re: Fixed bug where need to require the "" around the table/column name, because otherwise, the regexp will try to match as few characters as possible, causing it to match only the first letter of the name
- 04:13 PM Revision 8378: web/**/.htaccess: Removed "handle DirectoryIndex" rule, which does not appear to be needed with the new dotpath format
- 04:10 PM Revision 8377: web/VegBank/.htaccess: RewriteRule: Fixed bug where need to match tables by themselves as well as followed by columns
- 03:59 PM Revision 8376: web/**/.htaccess: Added "don't rewrite existing paths" and "handle DirectoryIndex" rules to all .htaccess files, not just those for dirs that contain subdirs, to facilitate linking directly to files in the filesystem without first needing to retrofit the applicable .htaccess file.
- 03:42 PM Revision 8375: web/**/.htaccess: Restrict redirection only of fully-existing paths, rather than of any path whose *subdir* exists, in order to avoid having to hardcode the absolute path to the directory containing the .htaccess file. This more restrictive approach still works because if the subdir exists and contains an .htaccess file, this (outer) .htaccess file's RewriteRules will be ignored completely until the inner .htaccess file's rules have been run, so that any subdir redirects will happen first. This "hole-punch" effect causes an existing filesystem path to always take priority over a virtual path (i.e. a redirect) defined in an outer .htaccess file.
- 02:57 PM Revision 8374: Added web/datasources/ symlink to inputs/
- 02:49 PM Revision 8373: web/servers/vegbiendev/fs: Updated symlink for new web/ dir nesting level
- 02:41 PM Revision 8372: Deleted no longer used www/
- 02:36 PM Revision 8371: Renamed www/ back to web/
- 02:34 PM Revision 8370: Removed www/logs/ because it was actually successfully moved into main/ (but some of its files weren't)
- 02:33 PM Revision 8369: Added www/logs/
- 02:24 PM Revision 8368: web/: Moved auxiliary files into the main/ subdir in preparation for having just the web/ dir. Renamed web/ to www/ so it can be replaced with web/main/.
- 02:09 PM Revision 8367: web/main/servers/vegbiendev/: Split it into db and fs forks, with db being the default. The fs fork links directly to /home/bien/svn on vegbiendev, which makes world-readable files directly web-accessible. (Permissions on /home/bien/svn and subdirs have been checked to ensure that private files are not world-readable.)
- 01:33 PM Revision 8366: Added web/main/IH/db/
- 01:26 PM Revision 8365: web/main/index.php: Updated fragment redirect for new dotpath format (using ? instead of a relative path)
- 01:20 PM Revision 8364: web/main/index.php: Updated path templates for new dotpath format (using ? instead of /)
- 01:06 PM Revision 8363: web/main/**/.htaccess: Support dotpaths in the query string instead of in the path, so that non-dotpath paths don't need to be suffixed with / to prevent their filenames from being interpreted as dotpaths. Putting dotpaths in the query string still requires only one character between the host and the path, but it's ? instead of / . ? is in many ways more natural, because the dotpath is a non-filesystem string to be parsed rather than something that's already a filesystem path. This change also avoids the need to strip trailing /s in many RewriteRules, because the dotpath mechanism is no longer appending them.
- 01:00 PM Revision 8362: Added web/main/dotpath.php, which parses any dotpath in the query string
- 12:58 PM Revision 8361: web/main/util.php: Added coalesce()
- 10:18 AM Revision 8360: web/main/svn*: Fixed symlinks to use .redmine instead of Redmine
- 09:37 AM Revision 8359: web/main/IH/.htaccess: Fixed whitespace
- 09:36 AM Revision 8358: web/main/IH/.htaccess: RewriteRule: Fixed bug where need to \-escape % because it's a special character in the replacement string (it references a regexp group matched by the last RewriteCond)
- 09:28 AM Revision 8357: Removed web/main/Redmine symlink so that it isn't listed in the directory listing as a visible (non-hidden) file. (There is already a .redmine symlink which matches all case variations via web/main/.htaccess logic.)
- 09:22 AM Revision 8356: Added web/main/svn* symlinks to VegBIEN/Redmine/svn*
- 09:18 AM Revision 8355: web/main/svn*/: Removed in preparation for replacing with symlinks, which will work now that absolute paths are used where needed
- 09:15 AM Revision 8354: web/main/**/.htaccess: internal redirects using relative paths with .. : Use absolute paths instead so that when the directory is reached through a symlink, the redirect will still work. Note that relative paths without .. do not need to be absolute paths because the subtree structure is the same (just the parent dirs are different).
- 08:01 AM Revision 8353: schemas/VegCore/VegCore.ERD.mwb: Regenerated exports
- 07:26 AM Revision 8352: schemas/VegCore/VegCore.ERD.mwb legend images: resized to medium-sized squares because apparently MySQL Workbench can't handle rectangular or very large images (it distorts them or resets them to their original size when the diagram is reloaded)
- 06:08 AM Revision 8351: schemas/VegCore/VegCore.ERD.mwb: Legend: Added IS-A, HAS-A, and "inherits from record" entries
- 06:08 AM Revision 8350: Added lib/MySQL_Workbench/connector.png, solid_line.png, dotted_line.png
- 05:00 AM Revision 8349: schemas/VegCore/VegCore.ERD.mwb: Fixed lines
- 04:59 AM Revision 8348: schemas/VegCore/VegCore.ERD.mwb: Renamed placename to named_place to match the table name
- 04:55 AM Revision 8347: schemas/VegCore/VegCore.ERD.mwb: Renamed measurement to trait because this is the more commonly used name for the entity
- 04:47 AM Revision 8346: schemas/VegCore/VegCore.ERD.mwb: Added VegCore logo
- 04:46 AM Revision 8345: Added schemas/VegCore/VegCore.logo.svg
- 04:09 AM Revision 8344: schemas/VegCore/VegCore.ERD.mwb: taxon: Added recursive parent fkey for (optionally) storing taxon hierarchies
- 03:42 AM Revision 8343: schemas/VegCore/VegCore.ERD.mwb: Repositioned taxon_determination
- 03:39 AM Revision 8342: schemas/VegCore/VegCore.ERD.mwb: Moved taxon next to qualified_taxon instead of above it, because inheritance (IS-A) is shown vertically while HAS-A is shown horizontally
- 03:25 AM Revision 8341: schemas/VegCore/VegCore.ERD.mwb: Populated legend
04/05/2013
- 12:23 AM Revision 8340: schemas/VegCore/VegCore.ERD.mwb: Added exports
- 12:23 AM Revision 8339: schemas/VegCore/VegCore.ERD.mwb: Fixed lines and settings for the Linux MySQL Workbench
04/04/2013
- 10:01 PM Revision 8338: schemas/VegCore/VegCore.ERD.mwb: Added table colors
- 10:01 PM Revision 8337: Removed backup file schemas/VegCore/VegCore.ERD.mwb.bak
- 09:48 PM Revision 8336: Added schemas/VegCore/VegCore.ERD.mwb, VegCore.my.sql with first VegCore ERD and MySQL schema. All tables are in the ERD, but contain only pkey and fkey columns.
- 09:52 AM Revision 8335: lib/sql.py: mk_select(): using subset function: Turn off enable_sort (within the transaction) to avoid unwanted slow sorts. This change (along with the subset functions themselves) should significantly reduce the long FIA.occurrence_all table subset time (~8 hours altogether) and with it the total import time, which had more than doubled as a result of the FIA refresh. Note that this issue would have been even more pronounced for larger datasets, such as the GBIF refresh, which would have taken ~2.5 days longer (400 million rows * ~30% are plants * (FIA: ~8 hours/16.7 million rows) * 1 day/24 hours).
- 09:30 AM Revision 8334: lib/sql.py: mk_select(): Use subset function when it's available for fast querying at large OFFSET values
- 09:29 AM Revision 8333: lib/sql.py: Added has_subset_func()
- 08:48 AM Revision 8332: inputs/FIA/occurrence_all/import: Run mk_subset_by_row_num_func() to make the subset functions available for fast querying at large OFFSET values
- 08:43 AM Revision 8331: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to add 1 to the 0-based offset_ to get the 1-based row_num (which is usually a serial column)
- 08:38 AM Revision 8330: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to subtract 1 from the end row_num because BETWEEN limits are inclusive of the bounds
- 08:33 AM Revision 8329: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where also need to COALESCE() offset_ to 0 when it's added to the limit_
- 08:20 AM Revision 8328: schemas/util.sql: mk_subset_by_row_num_func(): subset function which turns off enable_sort: Fixed bug where need to pass ($2, $3) instead of ($1, $2) to the regular subset function
- 08:14 AM Revision 8327: inputs/FIA/occurrence_all/import: Added occurrence_all-row_num column for use with mk_subset_by_row_num_func()
- 08:12 AM Revision 8326: schemas/util.sql: mk_subset_by_row_num_func(): Also create subset function which turns off enable_sort. This is used for limit values greater than ~100,000 to avoid unwanted slow sorts. The regular subset function is still needed to work with EXPLAIN, so that it produces expanded output instead of just a function scan.
- 07:27 AM Revision 8325: schemas/util.sql: Added mk_subset_by_row_num_func()
- 07:10 AM Revision 8324: schemas/util.sql: Added type_qual_name()
- 06:33 AM Revision 8323: schemas/util.sql: force_update_view(): Fixed bug where also need to drop view for "cannot change name of view column" errors
- 05:24 AM Revision 8322: inputs/FIA/occurrence_all/import: Use new force_update_view(), which only drops the view if its columns have changed and otherwise just uses CREATE OR REPLACE VIEW, rather than always first running DROP VIEW IF EXISTS
- 05:20 AM Revision 8321: schemas/util.sql: Added force_update_view()
- 04:23 AM Revision 8320: bin/make_analytical_db: Commented out export_analytical_db because we are not yet using the analytical DB in MySQL, and it doesn't make sense to generate a large, unused CSV export each time
- 04:19 AM Revision 8319: bin/export_analytical_db: Replaced analytical_aggregate with analytical_stem
- 03:53 AM Revision 8318: inputs/FIA/occurrence_all/: Updated header.csv for new column order
- 03:40 AM Revision 8317: inputs/FIA/occurrence_all/import: Use directional joins (LEFT/RIGHT JOIN) instead of inner joins to ensure that the PostgreSQL query planner always joins starting with the TREE table. Note that the directional joins are now needed for a different reason than when they were initially added, which had been to avoid slow sorts. The sorts (at least for LIMIT-only queries) went away when small tables such as COUNTY and REF_UNIT were added to the joins.
- 01:16 AM Revision 8316: inputs/FIA/*/map.csv: Changed newlines between table and field name to - because the newlines mess up the flow of queries and also break pgAdmin's display of EXPLAIN output. The - was chosen because it's a non-whitespace character that linewraps in browsers, phpPgAdmin, and Google spreadsheets (although unfortunately not in pgAdmin). It is better than space because you can set a text editor to treat it as a word character, allowing the entire column name (<table>-<field>) to be selected by double-clicking it.
04/03/2013
- 09:55 PM Revision 8315: Added planning/workflow/normalized_vs_denormalized/denormalized.generic_standardizations.png (a slide from Brad's bien3_architecture_denormalized.pptx PowerPoint), which shows the staging table preprocessing particularly well
- 09:45 PM Revision 8314: README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added instructions for what to do if the rightmost imports start getting truncated due to the 255-column limit in spreadsheets. (This will occur in 8 imports.)
- 09:32 PM Revision 8313: inputs/import.stats.xls: Removed the previous imports from the current tab because they are also in the 2012-6~9 tab, and should not be in two places
- 09:28 PM Revision 8312: inputs/import.stats.xls: Updated import times. MO and FIA have been refreshed.
Also available in: Atom