Project

General

Profile

Activity

From 03/15/2013 to 04/13/2013

04/13/2013

05:47 PM Revision 8473: web/.htaccess: bugfix: moved auto-add trailing / if not external redirect to before don't rewrite existing paths because the trailing / also needs to be added on paths which *are* existing paths, but are just missing the trailing /
Aaron Marcuse-Kubitza
06:37 AM Revision 8472: web/.htaccess: auto-add trailing / if not external redirect (limited form of DirectorySlash). this allows URLs without trailing /, for dirs which have a special external redirect to the datasource's homepage when there is no trailing /, but still auto-adds the trailing / for all other dirs.
Aaron Marcuse-Kubitza
06:17 AM Revision 8471: README.TXT: Debugging: mod_rewrite: cached redirects issue: changed to limit the cache clearing to when a *DirectorySlash*-related redirect is changed, since only those redirects are permanent, not the RewriteRule redirects (unless you explicitly set them as permanent)
Aaron Marcuse-Kubitza
06:03 AM Revision 8470: README.TXT: Debugging: mod_rewrite: added tip that whenever you change a RewriteRule, you *must* clear your browser's cache to ensure that a cached redirect is not used instead (temporary redirects are not supposed to be cached, but sometimes Firefox caches them anyway)
Aaron Marcuse-Kubitza
04:41 AM Revision 8469: README.TXT: umask: Removed -S because this causes the new umask to be echoed, and is not needed even when setting a symbolic umask
Aaron Marcuse-Kubitza
04:18 AM Revision 8468: README.TXT: Datasource setup:: added step to use umask to prevent files from becoming web-accessible
Aaron Marcuse-Kubitza
04:16 AM Revision 8467: README.TXT: Testing: what to put in your .profile: added `umask -S ug=rwx,o=` to prevent files from becoming web-accessible
Aaron Marcuse-Kubitza
04:05 AM Revision 8466: README.TXT: fixed leading whitespace
Aaron Marcuse-Kubitza
02:46 AM Revision 8465: lib/util.run: run_cmd (): renamed to run_args_cmd to clarify that this runs only the command line args command, not any command. added doc comment to clarify this.
Aaron Marcuse-Kubitza
02:41 AM Revision 8464: lib/util.run: EXIT trap: use on_exit() function which calls run_cmd instead of having run_cmd be the handler, so that users can override this function to perform other commands (or no commands) after the script is read
Aaron Marcuse-Kubitza
02:28 AM Revision 8463: lib/util.run: echo_func (): use the caller's FUNCNAME via the $FUNCNAME[] array instead of requiring them to pass it in the function args as `echo_func "$FUNCNAME" "$@"`
Aaron Marcuse-Kubitza
02:11 AM Revision 8462: README.TXT: Maintenance: added steps to synchronize a Mac's settings with my testing machine's
Aaron Marcuse-Kubitza
02:10 AM Revision 8461: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: moved swap=1 right before src= dest=, which it applies to. aligned whitespace of the various commands.
Aaron Marcuse-Kubitza
01:49 AM Revision 8460: README.TXT: Maintenance: steps to synchronize vegbiendev, jupiter, and your local machine: added steps to install dependencies
Aaron Marcuse-Kubitza
01:44 AM Revision 8459: README.TXT: Maintenance: Added steps to synchronize vegbiendev, jupiter, and your local machine when changes are made on vegbiendev
Aaron Marcuse-Kubitza
01:34 AM Revision 8458: README.TXT: converted spaces to tabs for easier readability in a whitespace-highlighting text editor, and so that pressing backspace deletes an indent rather than just a character (creating a misaligned indent)
Aaron Marcuse-Kubitza

04/11/2013

10:29 PM Revision 8457: Moved validation/ under planning/workflow/
Aaron Marcuse-Kubitza
10:28 PM Revision 8456: Moved planning/use_cases/ under planning/goals/
Aaron Marcuse-Kubitza
10:26 PM Revision 8455: Moved planning/milestones/ under planning/timeline/
Aaron Marcuse-Kubitza
10:23 PM Revision 8454: Added inputs/VegBIEN/planning shortcut to fs/planning
Aaron Marcuse-Kubitza
10:16 PM Revision 8453: web/.fs: symlink via datasources/VegBIEN/fs instead
Aaron Marcuse-Kubitza
10:14 PM Revision 8452: Added inputs/VegBIEN/fs symlink to /servers/vegbiendev/fs
Aaron Marcuse-Kubitza
05:56 PM Revision 8451: Added web/.gbif shortcut to datasources/GBIF
Aaron Marcuse-Kubitza
04:52 PM Revision 8450: web/**/.htaccess: Use [qsappend] flag with all RewriteRules to ensure that the original query string is never lost
Aaron Marcuse-Kubitza
04:30 PM Revision 8449: inputs/VegBank/.htaccess: Added RewriteCond to check that the REQUEST_FILENAME is not an existing dir, which is apparently needed to prevent the verify/ subdir from being interpreted as a redirect
Aaron Marcuse-Kubitza
02:45 PM Revision 8448: web/**/.htaccess: Removed preprocessing commands that need to run for every dir, because these are now inherited from the outermost .htaccess using InheritBefore. This allows the inner .htaccess files to focus just on the fallback rewrites, without needing lots of boilerplate code (except for the obligatory RewriteEngine/RewriteOptions/SetEnvIf preamble).
Aaron Marcuse-Kubitza
02:34 PM Revision 8447: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex, by deleting the /index filename. This ensures that any fallback redirect is not invoked, and instead the action corresponding to a trailing / (usually the display of a directory listing) is performed.
Aaron Marcuse-Kubitza
02:07 PM Revision 8446: web/**/.htaccess: Use InheritBefore RewriteOption so that the outermost dir's rules are always run first. This is necessary so that all the HTTP_HOST and dotpath handlers, etc. are applied before any fallback handlers, *without* needing to copy and paste each rule into *every* .htaccess file. (Note that the InheritBefore RewriteOption is only available in Apache 2.4.) The outermost .htaccess file, however, must *not* use InheritBefore because this would inherit rules from the global Apache config, which normally should not be run and which cause an infinite redirect loop.
Aaron Marcuse-Kubitza
01:56 PM Revision 8445: web/**/.htaccess: fallback rules which should not be inherited by subdirs: Filter using a RewriteCond on %{ENV:innermost_dir} so that the rule is only run if the innermost dir was this dir. (This is needed when using InheritBefore.)
Aaron Marcuse-Kubitza
01:36 PM Revision 8444: web/**/.htaccess: Added SetEnvIf directives to set the $innermost_dir env var, which is used by InheritBefore RewriteRule inheritance. You can easily populate the current dir name in jEdit by right-clicking the buffer tab and selecting Insert File Path (which requires the BufferTabs plugin), and then picking out the dir right before the .htaccess. If multiple dirs in the dir path have the same name (e.g. a/b/a/), add a distinguishing prefix to the inner dirs (this is rare).
Aaron Marcuse-Kubitza

04/10/2013

11:17 PM Revision 8443: web/.htaccess: Support rules being inherited by subdirs using InheritBefore, by setting an env var $innermost_dir that stores the name of the innermost subdir to contain an .htaccess file. This env var can then be tested for in a RewriteCond in rules that should only apply locally. This works because SetEnvIf processes directives from the outer dir to the inner dir, so that the inner dir's setting comes last and is the final value of $innermost_dir. Note that you must use SetEnvIf rather than SetEnv because "The internal environment variables set by [SetEnv] are set after most early request processing directives are run, such as access control and URI-to-filename mapping. If the environment variable you're setting is meant as input into this early phase of processing *such as the RewriteRule directive*, you should instead set the environment variable with SetEnvIf." [emphasis added] (http://httpd.apache.org/docs/current/mod/mod_setenvif.html#setenvif)
Aaron Marcuse-Kubitza
10:20 PM Revision 8442: web/.htaccess: don't rewrite existing paths: Changed match string to what is used in subdirs, so that if this rule is inherited, what is considered an existing path (i.e. a file or subdir) is restricted appropriately
Aaron Marcuse-Kubitza
10:18 PM Revision 8441: web/**/.htaccess: no-replacement RewriteRules: Removed the standard [discardpath,noescape] flags because they add unnecessary clutter and are not needed for this type of rule. Note that these flags are still required on regular RewriteRules which have a replacement string.
Aaron Marcuse-Kubitza
10:15 PM Revision 8440: web/**/.htaccess: no-replacement RewriteRules: Use just ^ as the match string because there is no need to actually match characters. This also makes it easy to visually tell what is a no-replacement RewriteRule vs. a RewriteRule that matches the entire path.
Aaron Marcuse-Kubitza
10:11 PM Revision 8439: web/.htaccess: Renamed $REQUEST_URI_NO_SLASH to $REQUEST_URI_no_/ to take advantage of (some) special chars being allowed in env var names
Aaron Marcuse-Kubitza
09:42 PM Revision 8438: Added web/people/
Aaron Marcuse-Kubitza
09:08 PM Revision 8437: web/.htaccess: fallback redirect: Use new VegCore/ instead of specifying the full wiki URL
Aaron Marcuse-Kubitza
09:07 PM Revision 8436: schemas/VegCore/.htaccess: Fixed bug where need to issue explicit redirect because path contains a fragment (#...), which must be sent to the browser
Aaron Marcuse-Kubitza
09:03 PM Revision 8435: Added schemas/VegCore/.htaccess
Aaron Marcuse-Kubitza
08:57 PM Revision 8434: Added web/.vegcore shortcut to exchange_schemas/VegCore
Aaron Marcuse-Kubitza
08:34 PM Revision 8433: Added root .htaccess (accessed via web/servers/vegbiendev/fs or one of its shortcuts)
Aaron Marcuse-Kubitza
08:28 PM Revision 8432: web/.htaccess: translate subdomain to path: Use new REQUEST_URI_NO_SLASH to avoid adding an automatic trailing / when a subdir was reached as a subdomain without a path (e.g. VegBank.vegpath.org), which should be equivalent to going to the subdir's homepage (vegpath.org/VegBank) rather than to its directory listing (vegpath.org/VegBank/ ; note the trailing /)
Aaron Marcuse-Kubitza
08:25 PM Revision 8431: web/.htaccess: set REQUEST_URI_NO_SLASH for use by translating subdomain to path
Aaron Marcuse-Kubitza
08:19 PM Revision 8430: Added web/.fs symlink to .vegbiendev/fs
Aaron Marcuse-Kubitza
01:53 PM Revision 8429: web/svn-web, wiki: Made them hidden so they wouldn't appear in the dir listing (which should not contain shortcuts to inner paths, as these would create significant clutter). web/svn unfortunately can't be made hidden because there is already a .svn dir.
Aaron Marcuse-Kubitza
01:47 PM Revision 8428: Added web/.nimoy, .vegbiendev shortcuts
Aaron Marcuse-Kubitza
01:44 PM Revision 8427: web/DwC, VegX: Replaced with versions in exchange_schemas/
Aaron Marcuse-Kubitza
01:39 PM Revision 8426: Moved web/TCS/ to schemas/
Aaron Marcuse-Kubitza
01:36 PM Revision 8425: Added web/exchange_schemas/ symlink
Aaron Marcuse-Kubitza
01:35 PM Revision 8424: Removed web/exchange_schemas/ in preparation for making it a symlink
Aaron Marcuse-Kubitza
01:34 PM Revision 8423: Added schemas/.htaccess
Aaron Marcuse-Kubitza
01:33 PM Revision 8422: Added web/exchange_schemas/
Aaron Marcuse-Kubitza
01:28 PM Revision 8421: Merged web/DwC,VegX with the corresponding dir in schemas/
Aaron Marcuse-Kubitza
01:19 PM Revision 8420: Moved web/BIEN2, IH into aggregators/
Aaron Marcuse-Kubitza
01:18 PM Revision 8419: Moved web/BIEN2, IH into aggregators/
Aaron Marcuse-Kubitza
01:09 PM Revision 8418: Added web/aggregators, primary_databases with symlinks to the applicable datasources
Aaron Marcuse-Kubitza
01:04 PM Revision 8417: web/.tnrs: Updated to point to datasources/.TNRS/
Aaron Marcuse-Kubitza
01:01 PM Revision 8416: Removed web/TNRS/ in preparation for replacing it with a symlink
Aaron Marcuse-Kubitza
12:59 PM Revision 8415: Added inputs/.TNRS/.htaccess
Aaron Marcuse-Kubitza
12:13 PM Revision 8414: web/.redmine/: Updated for new VegBIEN/ location
Aaron Marcuse-Kubitza
12:12 PM Revision 8413: web/.htaccess: Use new %{REQUEST_SCHEME} to construct self-refential URLs
Aaron Marcuse-Kubitza
10:14 AM Revision 8412: web/VegBIEN/: Moved to datasources/
Aaron Marcuse-Kubitza
10:07 AM Revision 8411: Added inputs/.htaccess, which parses dotpaths and adds trailing / when datasources/ subdir is listed. (DirectorySlash cannot be used for this because it would affect all subdirs, too, which must not have DirectorySlash on. Turning DirectorySlash off in each subdir would not work, because the / is added *as soon as* a valid filesystem path is formed, even if the inner dir would not have added the /.)
Aaron Marcuse-Kubitza
09:48 AM Revision 8410: web/**/.htaccess: internal redirects: Use ?&$0 instead of just ?$0 to prevent the query string from being reinterpreted as a dotpath by the destination dir
Aaron Marcuse-Kubitza
09:45 AM Revision 8409: web/VegBIEN/Redmine/*/.htaccess: Use relative URL paths. This requires adding a RewriteBase directive because these dirs can be reached by symlinks.
Aaron Marcuse-Kubitza
09:40 AM Revision 8408: inputs/SALVIAS/.htaccess: Removed test RedirectMatch directive
Aaron Marcuse-Kubitza
09:38 AM Revision 8407: web/**/.htaccess: RewriteOptions: Redirect dir paths without the trailing / to the datasource's homepage, and let dir paths with the trailing / point to the directory index. This is the opposite of the way it was before, and is more intuitive because the trailing / indicates a directory listing, while the trailing / would generally be omitted when linking to the datasource itself.
Aaron Marcuse-Kubitza
07:49 AM Revision 8406: web/**/.htaccess: handle DirectoryIndex subrequests, which append "index" to the dir
Aaron Marcuse-Kubitza
06:15 AM Revision 8405: web/**/.htaccess: RewriteOptions: Removed inherit because if the destination is not found in the current dir, a 404 should be returned instead of trying to invoke the parent dir's fallback handler (which applies only to paths in the parent dir). Ideally, we would use InheritBefore to get the subdomain preprocessing, etc., but the parent dir's fallback handler would still need to be turned off somehow when the rule was being run by a subdir.
Aaron Marcuse-Kubitza
05:59 AM Revision 8404: web/.htaccess: Only forward specific terms (subpaths) to VegCore, not also the dir index itself
Aaron Marcuse-Kubitza
05:15 AM Revision 8403: web/**/.htaccess: RewriteOptions: Added AllowNoSlash (available in Apache 2.4) so dir paths without the trailing / can be parsed by mod_rewrite
Aaron Marcuse-Kubitza
05:12 AM Revision 8402: web/.htaccess: Turned off DirectorySlash so that the dir without the trailing / can have a separate meaning (e.g. as the home page of a datasource, rather than a listing of the short URLs available for it)
Aaron Marcuse-Kubitza
03:33 AM Revision 8401: web/main.conf: <Directory .>: Added `Require all granted`, which is needed by Apache 2.4 to prevent 403 Forbidden errors. This change breaks compatibility with older versions of Apache, but is unfortunately required by 2.4.
Aaron Marcuse-Kubitza
03:29 AM Revision 8400: web/**/.htaccess: parse dotpath in the query string: Use just dotpath instead of dotpath.php because the extension is added automatically by MultiViews
Aaron Marcuse-Kubitza
03:28 AM Revision 8399: web/**/.htaccess: parse dotpath in the query string: Remove any index.* suffix, which is added by MultiViews in Apache 2.4
Aaron Marcuse-Kubitza
03:28 AM Revision 8398: web/main.conf: Updated the path of the <Directory> directive for the DocumentRoot
Aaron Marcuse-Kubitza
03:21 AM Revision 8397: web/.htaccess: Set Options +Indexes because this isn't the default in Apache 2.4
Aaron Marcuse-Kubitza
01:46 AM Revision 8396: web/.htaccess: DirectoryIndex: Use index instead of hardcoding index.php because MultiViews will now add the appropriate extension automatically
Aaron Marcuse-Kubitza
01:44 AM Revision 8395: web/.htaccess: Turn on MultiViews to allow auto-adding of the file extension
Aaron Marcuse-Kubitza
12:15 AM Revision 8394: inputs/**/.htaccess: Use a direct filesystem path rather than going through /svn (the Redmine web interface) because this loads much faster, and avoids needing to manually specify svn-web instead of svn for files that should be displayed in the browser instead of downloaded. It also adds support for files not in svn. Note that the use of a relative RewriteRule replacement requires RewriteBase when the directory is not under the document root, in order to construct the correct self-referential URL (http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritebase).
Aaron Marcuse-Kubitza

04/09/2013

10:59 PM Revision 8393: inputs/**/.htaccess: Removed no longer needed trailing / in paths
Aaron Marcuse-Kubitza
10:55 PM Revision 8392: inputs/CTFS/*/.htaccess: Use new direct shortcuts to /Redmine subtrees
Aaron Marcuse-Kubitza
10:47 PM Revision 8391: Added wiki shortcut to Redmine/wiki
Aaron Marcuse-Kubitza
10:43 PM Revision 8390: web/.htaccess: use separate lowercase version when available: Support paths without a trailing /
Aaron Marcuse-Kubitza
10:41 PM Revision 8389: web/.htaccess: use separate lowercase version when available: Fixed bug where need to use [last] flag, in order to rewrite in the new subdir instead of applying the rest of this dir's rules (which would forward to VegCore)
Aaron Marcuse-Kubitza
10:11 PM Revision 8388: web/: datasources indexed in VegPath: Link to the corresponding inputs/ dir instead, which now stores the data and the short URLs in one place
Aaron Marcuse-Kubitza
10:10 PM Revision 8387: web/.htaccess: Use %{REQUEST_URI} ("The path component of the requested URI" <http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritecond>) instead of /$0, in order to preserve whether or not the path had a trailing slash. This also ensures that the rule functions correctly if it is invoked in a subdir context, which may occur for the "remove www subdomain" rule.
Aaron Marcuse-Kubitza
10:01 PM Revision 8386: web/.htaccess: Removed "don't redirect subdir paths" because this is now handled by "don't rewrite existing paths" as described in r8375
Aaron Marcuse-Kubitza
09:48 PM Revision 8385: web/: datasources indexed in VegPath: Added .htaccess files to corresponding inputs/ dir so the short URLs can also be accessed via VegBIEN's own dir for the datasource
Aaron Marcuse-Kubitza
09:36 PM Revision 8384: inputs/input.Makefile: Removed .PRECIOUS from %/header.csv, %/map.csv so that these scripts are deleted on error. This is useful for the runscripts and for non-data dirs whose header.csv cannot be made.
Aaron Marcuse-Kubitza
09:33 PM Revision 8383: inputs/input.Makefile: SVN: Only run %/add on subdirs with visible (non-hidden) files. Subdirs with only hidden files (e.g. .htaccess) are assumed to be non-data dirs.
Aaron Marcuse-Kubitza
09:23 PM Revision 8382: inputs/input.Makefile: Staging tables installation: %/install: $(logInstall): Only use log file if log dir exists, to support non-data dirs
Aaron Marcuse-Kubitza
09:13 PM Revision 8381: inputs/input.Makefile: Staging tables installation: %/install: ignore errors in $(exportHeader) and $(cleanup) if the table does not exist (i.e. a non-data dir)
Aaron Marcuse-Kubitza
09:11 PM Revision 8380: inputs/input.Makefile: Staging tables installation: %/install: Don't run $(import_install_) for empty dirs because there is no data to import
Aaron Marcuse-Kubitza
04:37 PM Revision 8379: lib/sql.py: parse_exception(): typed_name_re: Fixed bug where need to require the "" around the table/column name, because otherwise, the regexp will try to match as few characters as possible, causing it to match only the first letter of the name
Aaron Marcuse-Kubitza
04:13 PM Revision 8378: web/**/.htaccess: Removed "handle DirectoryIndex" rule, which does not appear to be needed with the new dotpath format
Aaron Marcuse-Kubitza
04:10 PM Revision 8377: web/VegBank/.htaccess: RewriteRule: Fixed bug where need to match tables by themselves as well as followed by columns
Aaron Marcuse-Kubitza
03:59 PM Revision 8376: web/**/.htaccess: Added "don't rewrite existing paths" and "handle DirectoryIndex" rules to all .htaccess files, not just those for dirs that contain subdirs, to facilitate linking directly to files in the filesystem without first needing to retrofit the applicable .htaccess file.
Aaron Marcuse-Kubitza
03:42 PM Revision 8375: web/**/.htaccess: Restrict redirection only of fully-existing paths, rather than of any path whose *subdir* exists, in order to avoid having to hardcode the absolute path to the directory containing the .htaccess file. This more restrictive approach still works because if the subdir exists and contains an .htaccess file, this (outer) .htaccess file's RewriteRules will be ignored completely until the inner .htaccess file's rules have been run, so that any subdir redirects will happen first. This "hole-punch" effect causes an existing filesystem path to always take priority over a virtual path (i.e. a redirect) defined in an outer .htaccess file.
Aaron Marcuse-Kubitza
02:57 PM Revision 8374: Added web/datasources/ symlink to inputs/
Aaron Marcuse-Kubitza
02:49 PM Revision 8373: web/servers/vegbiendev/fs: Updated symlink for new web/ dir nesting level
Aaron Marcuse-Kubitza
02:41 PM Revision 8372: Deleted no longer used www/
Aaron Marcuse-Kubitza
02:36 PM Revision 8371: Renamed www/ back to web/
Aaron Marcuse-Kubitza
02:34 PM Revision 8370: Removed www/logs/ because it was actually successfully moved into main/ (but some of its files weren't)
Aaron Marcuse-Kubitza
02:33 PM Revision 8369: Added www/logs/
Aaron Marcuse-Kubitza
02:24 PM Revision 8368: web/: Moved auxiliary files into the main/ subdir in preparation for having just the web/ dir. Renamed web/ to www/ so it can be replaced with web/main/.
Aaron Marcuse-Kubitza
02:09 PM Revision 8367: web/main/servers/vegbiendev/: Split it into db and fs forks, with db being the default. The fs fork links directly to /home/bien/svn on vegbiendev, which makes world-readable files directly web-accessible. (Permissions on /home/bien/svn and subdirs have been checked to ensure that private files are not world-readable.)
Aaron Marcuse-Kubitza
01:33 PM Revision 8366: Added web/main/IH/db/
Aaron Marcuse-Kubitza
01:26 PM Revision 8365: web/main/index.php: Updated fragment redirect for new dotpath format (using ? instead of a relative path)
Aaron Marcuse-Kubitza
01:20 PM Revision 8364: web/main/index.php: Updated path templates for new dotpath format (using ? instead of /)
Aaron Marcuse-Kubitza
01:06 PM Revision 8363: web/main/**/.htaccess: Support dotpaths in the query string instead of in the path, so that non-dotpath paths don't need to be suffixed with / to prevent their filenames from being interpreted as dotpaths. Putting dotpaths in the query string still requires only one character between the host and the path, but it's ? instead of / . ? is in many ways more natural, because the dotpath is a non-filesystem string to be parsed rather than something that's already a filesystem path. This change also avoids the need to strip trailing /s in many RewriteRules, because the dotpath mechanism is no longer appending them.
Aaron Marcuse-Kubitza
01:00 PM Revision 8362: Added web/main/dotpath.php, which parses any dotpath in the query string
Aaron Marcuse-Kubitza
12:58 PM Revision 8361: web/main/util.php: Added coalesce()
Aaron Marcuse-Kubitza
10:18 AM Revision 8360: web/main/svn*: Fixed symlinks to use .redmine instead of Redmine
Aaron Marcuse-Kubitza
09:37 AM Revision 8359: web/main/IH/.htaccess: Fixed whitespace
Aaron Marcuse-Kubitza
09:36 AM Revision 8358: web/main/IH/.htaccess: RewriteRule: Fixed bug where need to \-escape % because it's a special character in the replacement string (it references a regexp group matched by the last RewriteCond)
Aaron Marcuse-Kubitza
09:28 AM Revision 8357: Removed web/main/Redmine symlink so that it isn't listed in the directory listing as a visible (non-hidden) file. (There is already a .redmine symlink which matches all case variations via web/main/.htaccess logic.)
Aaron Marcuse-Kubitza
09:22 AM Revision 8356: Added web/main/svn* symlinks to VegBIEN/Redmine/svn*
Aaron Marcuse-Kubitza
09:18 AM Revision 8355: web/main/svn*/: Removed in preparation for replacing with symlinks, which will work now that absolute paths are used where needed
Aaron Marcuse-Kubitza
09:15 AM Revision 8354: web/main/**/.htaccess: internal redirects using relative paths with .. : Use absolute paths instead so that when the directory is reached through a symlink, the redirect will still work. Note that relative paths without .. do not need to be absolute paths because the subtree structure is the same (just the parent dirs are different).
Aaron Marcuse-Kubitza
08:01 AM Revision 8353: schemas/VegCore/VegCore.ERD.mwb: Regenerated exports
Aaron Marcuse-Kubitza
07:26 AM Revision 8352: schemas/VegCore/VegCore.ERD.mwb legend images: resized to medium-sized squares because apparently MySQL Workbench can't handle rectangular or very large images (it distorts them or resets them to their original size when the diagram is reloaded)
Aaron Marcuse-Kubitza
06:08 AM Revision 8351: schemas/VegCore/VegCore.ERD.mwb: Legend: Added IS-A, HAS-A, and "inherits from record" entries
Aaron Marcuse-Kubitza
06:08 AM Revision 8350: Added lib/MySQL_Workbench/connector.png, solid_line.png, dotted_line.png
Aaron Marcuse-Kubitza
05:00 AM Revision 8349: schemas/VegCore/VegCore.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
04:59 AM Revision 8348: schemas/VegCore/VegCore.ERD.mwb: Renamed placename to named_place to match the table name
Aaron Marcuse-Kubitza
04:55 AM Revision 8347: schemas/VegCore/VegCore.ERD.mwb: Renamed measurement to trait because this is the more commonly used name for the entity
Aaron Marcuse-Kubitza
04:47 AM Revision 8346: schemas/VegCore/VegCore.ERD.mwb: Added VegCore logo
Aaron Marcuse-Kubitza
04:46 AM Revision 8345: Added schemas/VegCore/VegCore.logo.svg
Aaron Marcuse-Kubitza
04:09 AM Revision 8344: schemas/VegCore/VegCore.ERD.mwb: taxon: Added recursive parent fkey for (optionally) storing taxon hierarchies
Aaron Marcuse-Kubitza
03:42 AM Revision 8343: schemas/VegCore/VegCore.ERD.mwb: Repositioned taxon_determination
Aaron Marcuse-Kubitza
03:39 AM Revision 8342: schemas/VegCore/VegCore.ERD.mwb: Moved taxon next to qualified_taxon instead of above it, because inheritance (IS-A) is shown vertically while HAS-A is shown horizontally
Aaron Marcuse-Kubitza
03:25 AM Revision 8341: schemas/VegCore/VegCore.ERD.mwb: Populated legend
Aaron Marcuse-Kubitza

04/05/2013

12:23 AM Revision 8340: schemas/VegCore/VegCore.ERD.mwb: Added exports
Aaron Marcuse-Kubitza
12:23 AM Revision 8339: schemas/VegCore/VegCore.ERD.mwb: Fixed lines and settings for the Linux MySQL Workbench
Aaron Marcuse-Kubitza

04/04/2013

10:01 PM Revision 8338: schemas/VegCore/VegCore.ERD.mwb: Added table colors
Aaron Marcuse-Kubitza
10:01 PM Revision 8337: Removed backup file schemas/VegCore/VegCore.ERD.mwb.bak
Aaron Marcuse-Kubitza
09:48 PM Revision 8336: Added schemas/VegCore/VegCore.ERD.mwb, VegCore.my.sql with first VegCore ERD and MySQL schema. All tables are in the ERD, but contain only pkey and fkey columns.
Aaron Marcuse-Kubitza
09:52 AM Revision 8335: lib/sql.py: mk_select(): using subset function: Turn off enable_sort (within the transaction) to avoid unwanted slow sorts. This change (along with the subset functions themselves) should significantly reduce the long FIA.occurrence_all table subset time (~8 hours altogether) and with it the total import time, which had more than doubled as a result of the FIA refresh. Note that this issue would have been even more pronounced for larger datasets, such as the GBIF refresh, which would have taken ~2.5 days longer (400 million rows * ~30% are plants * (FIA: ~8 hours/16.7 million rows) * 1 day/24 hours).
Aaron Marcuse-Kubitza
09:30 AM Revision 8334: lib/sql.py: mk_select(): Use subset function when it's available for fast querying at large OFFSET values
Aaron Marcuse-Kubitza
09:29 AM Revision 8333: lib/sql.py: Added has_subset_func()
Aaron Marcuse-Kubitza
08:48 AM Revision 8332: inputs/FIA/occurrence_all/import: Run mk_subset_by_row_num_func() to make the subset functions available for fast querying at large OFFSET values
Aaron Marcuse-Kubitza
08:43 AM Revision 8331: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to add 1 to the 0-based offset_ to get the 1-based row_num (which is usually a serial column)
Aaron Marcuse-Kubitza
08:38 AM Revision 8330: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to subtract 1 from the end row_num because BETWEEN limits are inclusive of the bounds
Aaron Marcuse-Kubitza
08:33 AM Revision 8329: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where also need to COALESCE() offset_ to 0 when it's added to the limit_
Aaron Marcuse-Kubitza
08:20 AM Revision 8328: schemas/util.sql: mk_subset_by_row_num_func(): subset function which turns off enable_sort: Fixed bug where need to pass ($2, $3) instead of ($1, $2) to the regular subset function
Aaron Marcuse-Kubitza
08:14 AM Revision 8327: inputs/FIA/occurrence_all/import: Added occurrence_all-row_num column for use with mk_subset_by_row_num_func()
Aaron Marcuse-Kubitza
08:12 AM Revision 8326: schemas/util.sql: mk_subset_by_row_num_func(): Also create subset function which turns off enable_sort. This is used for limit values greater than ~100,000 to avoid unwanted slow sorts. The regular subset function is still needed to work with EXPLAIN, so that it produces expanded output instead of just a function scan.
Aaron Marcuse-Kubitza
07:27 AM Revision 8325: schemas/util.sql: Added mk_subset_by_row_num_func()
Aaron Marcuse-Kubitza
07:10 AM Revision 8324: schemas/util.sql: Added type_qual_name()
Aaron Marcuse-Kubitza
06:33 AM Revision 8323: schemas/util.sql: force_update_view(): Fixed bug where also need to drop view for "cannot change name of view column" errors
Aaron Marcuse-Kubitza
05:24 AM Revision 8322: inputs/FIA/occurrence_all/import: Use new force_update_view(), which only drops the view if its columns have changed and otherwise just uses CREATE OR REPLACE VIEW, rather than always first running DROP VIEW IF EXISTS
Aaron Marcuse-Kubitza
05:20 AM Revision 8321: schemas/util.sql: Added force_update_view()
Aaron Marcuse-Kubitza
04:23 AM Revision 8320: bin/make_analytical_db: Commented out export_analytical_db because we are not yet using the analytical DB in MySQL, and it doesn't make sense to generate a large, unused CSV export each time
Aaron Marcuse-Kubitza
04:19 AM Revision 8319: bin/export_analytical_db: Replaced analytical_aggregate with analytical_stem
Aaron Marcuse-Kubitza
03:53 AM Revision 8318: inputs/FIA/occurrence_all/: Updated header.csv for new column order
Aaron Marcuse-Kubitza
03:40 AM Revision 8317: inputs/FIA/occurrence_all/import: Use directional joins (LEFT/RIGHT JOIN) instead of inner joins to ensure that the PostgreSQL query planner always joins starting with the TREE table. Note that the directional joins are now needed for a different reason than when they were initially added, which had been to avoid slow sorts. The sorts (at least for LIMIT-only queries) went away when small tables such as COUNTY and REF_UNIT were added to the joins.
Aaron Marcuse-Kubitza
01:16 AM Revision 8316: inputs/FIA/*/map.csv: Changed newlines between table and field name to - because the newlines mess up the flow of queries and also break pgAdmin's display of EXPLAIN output. The - was chosen because it's a non-whitespace character that linewraps in browsers, phpPgAdmin, and Google spreadsheets (although unfortunately not in pgAdmin). It is better than space because you can set a text editor to treat it as a word character, allowing the entire column name (<table>-<field>) to be selected by double-clicking it.
Aaron Marcuse-Kubitza

04/03/2013

09:55 PM Revision 8315: Added planning/workflow/normalized_vs_denormalized/denormalized.generic_standardizations.png (a slide from Brad's bien3_architecture_denormalized.pptx PowerPoint), which shows the staging table preprocessing particularly well
Aaron Marcuse-Kubitza
09:45 PM Revision 8314: README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added instructions for what to do if the rightmost imports start getting truncated due to the 255-column limit in spreadsheets. (This will occur in 8 imports.)
Aaron Marcuse-Kubitza
09:32 PM Revision 8313: inputs/import.stats.xls: Removed the previous imports from the current tab because they are also in the 2012-6~9 tab, and should not be in two places
Aaron Marcuse-Kubitza
09:28 PM Revision 8312: inputs/import.stats.xls: Updated import times. MO and FIA have been refreshed.
Aaron Marcuse-Kubitza

04/02/2013

04:17 PM Revision 8311: Removed no longer needed inputs/GBIF/import. Use ./run instead.
Aaron Marcuse-Kubitza
04:17 PM Revision 8310: Removed no longer needed inputs/GBIF/_MySQL/import. Use ./run instead.
Aaron Marcuse-Kubitza
04:16 PM Revision 8309: inputs/GBIF/_MySQL/run: import: Run make directly instead of via ./import
Aaron Marcuse-Kubitza
04:15 PM Revision 8308: inputs/GBIF/_MySQL/run: Use new import.run, which defines all()
Aaron Marcuse-Kubitza
04:06 PM Revision 8307: Added planning/workflow/normalized_vs_denormalized/bien3_architecture_(de)normalized.pptx
Aaron Marcuse-Kubitza
03:57 PM Revision 8306: Added planning/workflow/normalized_vs_denormalized/BIEN-modArch-Dec2010 NS-SBD 1.4.ppt.url
Aaron Marcuse-Kubitza
03:50 PM Revision 8305: planning/workflow/: Moved normalized vs. denormalized files to separate normalized_vs_denormalized/ subfolder
Aaron Marcuse-Kubitza
03:21 PM Revision 8304: Regenerated inputs/ACAD/Specimen/logs/steps.by_col.log.sql
Aaron Marcuse-Kubitza
03:15 PM Revision 8303: inputs/GBIF/raw_occurrence_record/run: Override MySQL_export() so $filter can be customized
Aaron Marcuse-Kubitza
03:13 PM Revision 8302: inputs/GBIF/table.run: import(): Updated for lib/table.run template changes
Aaron Marcuse-Kubitza
03:09 PM Revision 8301: lib/table.run: template: import(): Also pass "$@" to superclass method
Aaron Marcuse-Kubitza
03:08 PM Revision 8300: lib/table.run: template: Use "$FUNCNAME" instead of hardcoding import
Aaron Marcuse-Kubitza
03:02 PM Revision 8299: Added inputs/GBIF/MySQL_export, used by ./table.run
Aaron Marcuse-Kubitza
02:57 PM Revision 8298: lib/util.run: echo_func: Fixed bug where need to use BASH_LINENO[0] for the line #s to match up with the files. For some reason the required array indexes for BASH_SOURCE (1) and BASH_LINENO (0) differ by one.
Aaron Marcuse-Kubitza
02:51 PM Revision 8297: inputs/GBIF/run: Use new import.run, which defines all()
Aaron Marcuse-Kubitza
02:51 PM Revision 8296: lib/table.run: Use new import.run, which defines all()
Aaron Marcuse-Kubitza
02:49 PM Revision 8295: Added lib/import.run
Aaron Marcuse-Kubitza
02:48 PM Revision 8294: lib/util.run: echo_func: Include the line # of the function to make it easier to find where the code being run is
Aaron Marcuse-Kubitza
02:32 PM Revision 8293: lib/table.run: Added all (default target)
Aaron Marcuse-Kubitza
02:26 PM Revision 8292: lib/util.run: run_cmd: If bash exited with an error, don't run the "$@" command. This test is necessary because `trap run_cmd EXIT` will run run_cmd as the result of *any* exit from the shell, including an error.
Aaron Marcuse-Kubitza
02:21 PM Revision 8291: *run: Use -e option to bash on the #! line instead of separate `set -o errexit` line so that there is no issue with the `set -o errexit` line getting separated from the #! line (errexit is required for the scripts to work properly)
Aaron Marcuse-Kubitza
02:09 PM Revision 8290: lib/util.run: run_cmd: When no command specified, default to running the `all` command, just like make
Aaron Marcuse-Kubitza
02:07 PM Revision 8289: lib/util.run: Run run_cmd at shell exit (using trap) instead of requiring every runscript to have `run_cmd ` at the end of it
Aaron Marcuse-Kubitza
01:49 PM Revision 8288: Added inputs/GBIF/run
Aaron Marcuse-Kubitza
01:48 PM Revision 8287: Added inputs/GBIF/raw_occurrence_record/run
Aaron Marcuse-Kubitza
01:47 PM Revision 8286: Added inputs/GBIF/table.run
Aaron Marcuse-Kubitza
01:45 PM Revision 8285: Added inputs/GBIF/_MySQL/run
Aaron Marcuse-Kubitza
01:42 PM Revision 8284: lib/util.run: fwd: Check that $subdirs is defined. Added $subdirs to usage.
Aaron Marcuse-Kubitza
01:39 PM Revision 8283: lib/util.run: fwd: Added usage
Aaron Marcuse-Kubitza
01:32 PM Revision 8282: lib/table.run: Switched from echo_run to echo_func
Aaron Marcuse-Kubitza
01:16 PM Revision 8281: lib/util.run: run_cmd: Echo the command being run, including the top-level run script. This is in addition to the echoing of the command in the function itself (using echo_func), which provides *both* the runscript that was run *and* the file where the invoked command was actually located (which may be different due to includes).
Aaron Marcuse-Kubitza
01:12 PM Revision 8280: lib/util.run: Echo the command at the beginning of each function using new echo_func, instead of having to type echo_run before every call to a function. Note that because echo_func uses BASH_SOURCE, the path to the file containing the function will be included in the debug message, which greatly facilitates locating which file a command is in.
Aaron Marcuse-Kubitza
01:08 PM Revision 8279: lib/util.run: Added echo_func
Aaron Marcuse-Kubitza
12:50 PM Revision 8278: lib/util.run: Added echo_cmd and use it in echo_run
Aaron Marcuse-Kubitza
12:46 PM Revision 8277: lib/util.run: echo_cmd(): Renamed to echo_run for clarity, because it also runs the command
Aaron Marcuse-Kubitza
12:39 PM Revision 8276: lib/util.run: Added inline_make()
Aaron Marcuse-Kubitza
12:39 PM Revision 8275: lib/util.run: Added echo_stdin()
Aaron Marcuse-Kubitza
12:30 PM Revision 8274: bin/my2pg_export: Put --password first because it's an authentication-related option
Aaron Marcuse-Kubitza
10:52 AM Revision 8273: Added lib/table.run, which includes the commands in import.sh but uses run scripts to allow running commands other than just import. (For example, map_table or postprocess can be run separately. Uninstall-related commands which would not belong in an import script can also be added, because import is only one of many commands a run script can offer.)
Aaron Marcuse-Kubitza
10:35 AM Revision 8272: Added lib/util.run with general functions and template for run scripts (a bash-based replacement for make). Unlike make, run scripts support full bash functionality including multiline commands. The run script template also includes syntax for various kinds of relative includes in bash.
Aaron Marcuse-Kubitza
12:03 AM Revision 8271: lib/common.Makefile: Added $(require_var)
Aaron Marcuse-Kubitza

04/01/2013

10:42 PM Revision 8270: bin/publish_analytical_db: Fixed bug where need to remove `ESCAPED BY '"'` because this would causing " followed by an escape sequence char to be interpreted specially (e.g. "n -> \n). MySQL automatically takes care of quote doubling when you specify `FIELDS OPTIONALLY ENCLOSED BY`.
Aaron Marcuse-Kubitza
10:13 PM Revision 8269: lib/common.Makefile: Compression: Added `%:: %.gz`, `%.gz: %`
Aaron Marcuse-Kubitza
08:07 PM Revision 8268: planning/workflow/import_process_comparison.odg: Moved "staging tables" under the method labels to reduce empty space
Aaron Marcuse-Kubitza
07:52 PM Revision 8267: planning/workflow/import_process_comparison.odg: Removed margins so the labels would align with the page margin on the Import process wiki page <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Import_process>
Aaron Marcuse-Kubitza
07:32 PM Revision 8266: Added planning/workflow/import_process_comparison.odg and .png export
Aaron Marcuse-Kubitza
06:12 PM Revision 8265: lib/db_xml.py: put_table(): Fixed bug where command to advance start to fetch next set was unintentionally deleted when removing the is_view check
Aaron Marcuse-Kubitza
06:11 PM Revision 8264: inputs/UNCC/Specimen/new_terms.csv: Updated for updated VegCore vocab
Aaron Marcuse-Kubitza
03:53 PM Revision 8263: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5: Regenerated after appending agent table to GBIFPortalDB-2013-02-20.data.sql
Aaron Marcuse-Kubitza
03:51 PM Revision 8262: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.gz.md5
Aaron Marcuse-Kubitza

03/28/2013

08:16 PM Revision 8261: Added inputs/GBIF/raw_occurrence_record/ from refresh
Aaron Marcuse-Kubitza
08:07 PM Revision 8260: inputs/GBIF/MySQL.schema.sql: Regenerated with inline enum type translated to CHECK constraint
Aaron Marcuse-Kubitza
08:07 PM Revision 8259: bin/my2pg: Translate inline enum type to CHECK constraint
Aaron Marcuse-Kubitza
07:43 PM Revision 8258: Added inputs/GBIF/**/MySQL.schema.sql
Aaron Marcuse-Kubitza
07:42 PM Revision 8257: Added inputs/GBIF/_MySQL/MySQL.*.sql.make
Aaron Marcuse-Kubitza
07:36 PM Revision 8256: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
Aaron Marcuse-Kubitza
07:29 PM Revision 8255: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
Aaron Marcuse-Kubitza
07:22 PM Revision 8254: inputs/input.Makefile: SVN: add: Removed Source/map.csv prerequisite because it is not related to adding unversioned files in the dir. It was originally a prerequisite in order to auto-create it when the datasource dir is first created, but the map.csv recipe does not currently create metadata-only map.csvs. In the future, metadata-only map.csvs will be replaced with constant columns added to the applicable tables.
Aaron Marcuse-Kubitza
07:19 PM Revision 8253: Added inputs/FIA/_archive
Aaron Marcuse-Kubitza
07:19 PM Revision 8252: inputs/input.Makefile: %/map.csv: Fixed bug where can only make header.csv if map.csv does not exist, because some subdirs are metadata-only and don't have a corresponding DB table
Aaron Marcuse-Kubitza
07:02 PM Revision 8251: README.TXT: Datasource setup: Install the staging tables: For a MySQL .sql export: Documented which password to use at each of the two password prompts my2pg_export will give you. You could also embed the value of the 2nd prompt in the _MySQL/*.make file using `--password="$(cat path/to/config/bien_password)"`.
Aaron Marcuse-Kubitza
06:56 PM Revision 8250: README.TXT: Datasource setup: Install the staging tables: Removed requirement that `make inputs/<datasrc>/reinstall quiet=1 &` be run on vegbiendev for MySQL .sql exports, because the hostname is now set to vegbiendev instead of localhost
Aaron Marcuse-Kubitza
06:38 PM Revision 8249: inputs/input.Makefile: sql/install: Use psql_script_vegbien instead of $(psqlNoSearchPath) (which uses psql_verbose_vegbien) because the insert statement for each data row should not be echoed
Aaron Marcuse-Kubitza
06:14 PM Revision 8248: inputs/FIA/occurrence_all/import: Run remake_VegBIEN_mappings at end to keep mappings to next stage of import process up to date
Aaron Marcuse-Kubitza
06:14 PM Revision 8247: inputs/FIA/occurrence_all/: Accepted new test output
Aaron Marcuse-Kubitza
06:13 PM Revision 8246: lib/import.sh: remake_VegBIEN_mappings(): Also remake VegBIEN.csv and test.xml.ref use `make test`
Aaron Marcuse-Kubitza
06:11 PM Revision 8245: lib/import.sh: Added remake_VegBIEN_mappings()
Aaron Marcuse-Kubitza
06:10 PM Revision 8244: inputs/input.Makefile: %/map.csv: make $*/header.csv first in case it doesn't exist (e.g. if it has been deleted so that it will be remade)
Aaron Marcuse-Kubitza
06:07 PM Revision 8243: inputs/FIA/occurrence_all/map.csv: Regenerated using new input table mappings
Aaron Marcuse-Kubitza
05:47 PM Revision 8242: lib/import.sh: Added make() and use it instead of the full make command
Aaron Marcuse-Kubitza
05:23 PM Revision 8241: inputs/input.Makefile: postprocess: Use %/postprocess instead of %/postprocess.sql/run so $*/import is also run
Aaron Marcuse-Kubitza
05:21 PM Revision 8240: inputs/FIA/: Ran inputs/FIA/import. This maps to VegCore's commonName.
Aaron Marcuse-Kubitza
05:19 PM Revision 8239: inputs/input.Makefile: %/postprocess: Also run the $*/import script, if it exists. Note that this is not the same as the %/import make target.
Aaron Marcuse-Kubitza
05:12 PM Revision 8238: inputs/input.Makefile: %/postprocess.sql/run: Factored out into separate %/postprocess command, which can eventually also perform other actions
Aaron Marcuse-Kubitza
04:59 PM Revision 8237: inputs/FIA/PLOT/map.csv: ELEV: Remapped to elevation_ft, assuming units based on the actual elevation of the region for a sample plot record
Aaron Marcuse-Kubitza
04:27 PM Revision 8236: inputs/VegBank/taxonobservation_/map.csv: Mapped int_currplantcommon to vernacularName
Aaron Marcuse-Kubitza
04:25 PM Revision 8235: mappings/VegCore.htm: Renamed salvias_plots table plotMetadata to PlotMetadata because of SALVIAS refresh on nimoy
Aaron Marcuse-Kubitza
04:18 PM Revision 8234: mappings/VegCore.htm: Regenerated from wiki. Added flower, fruit, commonName.
Aaron Marcuse-Kubitza
03:37 PM Revision 8233: mappings/Makefile: $(vocab); bin/redmine_synonyms: Support crossed out (deprecated) terms
Aaron Marcuse-Kubitza
03:24 PM Revision 8232: README.TXT: Maintenance: VegCore data dictionary: Added steps to update the data dictionary's Tables section if necessary
Aaron Marcuse-Kubitza
02:14 PM Revision 8231: inputs/GBIF/_MySQL/Makefile: %.data.sql: Added agent table
Aaron Marcuse-Kubitza
01:18 PM Revision 8230: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5
Aaron Marcuse-Kubitza
01:11 PM Revision 8229: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.sql
Aaron Marcuse-Kubitza
11:02 AM Revision 8228: Added web/main/svn*/, now using .htaccess to forward to Redmine/*
Aaron Marcuse-Kubitza
10:55 AM Revision 8227: Removed web/main/svn, svn-web symlinks because they need to be .htaccess-es in order for the relative mod_rewrite commands to work correctly
Aaron Marcuse-Kubitza
10:50 AM Revision 8226: Added web/main/svn, svn-web symlinks to Redmine/* for shorter URLs
Aaron Marcuse-Kubitza
10:49 AM Revision 8225: Added web/main/Redmine/svn-web/
Aaron Marcuse-Kubitza
08:28 AM Revision 8224: inputs/GBIF/: Added scripts for subsetting refresh
Aaron Marcuse-Kubitza
12:24 AM Revision 8223: lib/sql.py: table_order_by(): Documented that it returns None if table is a view, because table_cluster_on() would return None. This is necessary for inputs/FIA/occurrence_all/ sorting to work correctly, because specifying a manual sort order would prevent the query planner from just using fast nested loop joins and instead cause it to perform a slow sort. (This appears to be a bug in the query planner, because when the column list specified matches the joined-on indexes, there should be no need for post-nested loop re-sorting.)
Aaron Marcuse-Kubitza
12:20 AM Revision 8222: inputs/FIA/occurrence_all/test.xml.ref: Updated inserted row count for new row sort order
Aaron Marcuse-Kubitza
12:19 AM Revision 8221: lib/db_xml.py: put_table(): Fixed bug where also need to advance start to fetch next set when table is a view, because the views that are now being used with the import (inputs/FIA/occurrence_all/) are static rather than dynamic and do not return different rows after the previous set of rows has been imported
Aaron Marcuse-Kubitza

03/27/2013

11:43 PM Revision 8220: inputs/FIA/occurrence_all/import: Removed no longer applicable comment that directional joins are needed for PostgreSQL query planner to avoid slow sorts
Aaron Marcuse-Kubitza
11:40 PM Revision 8219: inputs/FIA/TREE/import: Reclustered table by TREE.parent path index, to facilitate path-order joins
Aaron Marcuse-Kubitza
11:39 PM Revision 8218: inputs/FIA/occurrence_all/import: Changed all RIGHT JOINs to inner joins so that tables would be joined in path order (i.e. general->specific). This optimizes the incremental joins so that the small tables are joined to each other before being joined to the large tables, rather than each row of the large tables being looked up in the small tables. This effect may not be noticeable for small LIMIT values, but would become apparent for large LIMIT values, such as the 1-million-row partitions used by db_xml.put_table() for column-based import. Note that inner joins used to cause the query planner to produce incorrect results containing slow sorts, but now this appears to no longer be an issue, perhaps because the result is not sorted by the TREE.ID index (which is not in the same order as the path indexes *.unique, *.parent).
Aaron Marcuse-Kubitza
10:46 PM Revision 8217: inputs/FIA/occurrence_all/import: Removed trailing whitespace
Aaron Marcuse-Kubitza
10:30 PM Revision 8216: Removed unused inputs/FIA/COND_unique/. Use COND instead.
Aaron Marcuse-Kubitza
09:52 PM Revision 8215: inputs/FIA/import: Use `set -o errexit` instead of putting ` || exit` after each command
Aaron Marcuse-Kubitza
09:52 PM Revision 8214: lib/import.sh: map_table(): Removed unneeded () around psql. This also fixes a bug where an error exit status from psql would not have aborted the script because `set -o errexit` does not apply to commands enclosed in (). For () you need to use ` || exit` instead (or ` || return` inside a function).
Aaron Marcuse-Kubitza
09:42 PM Revision 8213: lib/import.sh: Use `set -o errexit` so any command that exits with an error aborts the script. Note that a command's exit status can still be ignored using ` || true`. Removed no longer needed ` || return` in functions.
Aaron Marcuse-Kubitza
09:40 PM Revision 8212: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
Aaron Marcuse-Kubitza
09:33 PM Revision 8211: lib/import.sh: functions: abort if a command encounters an error
Aaron Marcuse-Kubitza
09:17 PM Revision 8210: schemas/VegCore/mk_derived: Added cultivated from oldGrowth
Aaron Marcuse-Kubitza
09:16 PM Revision 8209: schemas/util.sql: Added try_mk_derived_col()
Aaron Marcuse-Kubitza
08:35 PM Revision 8208: inputs/FIA/*/import: Run mk_derived after postprocessing commands
Aaron Marcuse-Kubitza
08:28 PM Revision 8207: inputs/FIA/import_order.txt: Added occurrence_all/
Aaron Marcuse-Kubitza
08:23 PM Revision 8206: mappings/VegCore-VegBIEN.csv: subplotID,subplot -> location.sourceaccessioncode: Fixed bug where need /_first to handle the case where both subplotID and subplot are provided
Aaron Marcuse-Kubitza
08:15 PM Revision 8205: Added inputs/FIA/map.csv, which maps shared columns to VegCore
Aaron Marcuse-Kubitza
08:12 PM Revision 8204: inputs/FIA/FIA_COND_unique/test.xml.ref: Updated now that PLOT, CONDID have been mapped
Aaron Marcuse-Kubitza
08:12 PM Revision 8203: inputs/FIA/*/map.csv for pre-refresh tables: Added back * before unmapped column names
Aaron Marcuse-Kubitza
08:03 PM Revision 8202: lib/csvs.py: stream_info(): Fixed bug where headers with multiline columns were not supported because only the first line (not the first multiline row) is sniffed for the dialect
Aaron Marcuse-Kubitza
06:56 PM Revision 8201: inputs/input.Makefile: %/header.csv: Fixed bug where newlines inside column names were incorrectly formatted by psql's table header formatting, by using COPY TO STDOUT instead
Aaron Marcuse-Kubitza
05:28 PM Revision 8200: schemas/util.sql: Added do_optionally_ignore()
Aaron Marcuse-Kubitza
04:28 PM Revision 8199: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
Aaron Marcuse-Kubitza
04:12 PM Revision 8198: lib/import.sh: Added mk_derived(). Added mk_derived to usage template.
Aaron Marcuse-Kubitza
04:11 PM Revision 8197: Added schemas/VegCore/mk_derived, which will be run in the import scripts
Aaron Marcuse-Kubitza
04:09 PM Revision 8196: lib/import.sh: psql(): Set psql vars :schema, :table, :table_str for use by the psql commands
Aaron Marcuse-Kubitza
03:22 PM Revision 8195: lib/import.sh: Export $schema, $table so they are available to programs invoked within an import script, which should not reset these vars if they include import.sh
Aaron Marcuse-Kubitza
03:20 PM Revision 8194: lib/import.sh: Only set $table, $schema if they don't already exist
Aaron Marcuse-Kubitza
03:11 PM Revision 8193: lib/import.sh: Added $root_dir and use it in $bin_dir
Aaron Marcuse-Kubitza
03:11 PM Revision 8192: inputs/FIA/*/import: Use new mk_*_col()
Aaron Marcuse-Kubitza
02:50 PM Revision 8191: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
Aaron Marcuse-Kubitza
02:43 PM Revision 8190: schemas/util.sql: Added mk_const_col()
Aaron Marcuse-Kubitza
02:37 PM Revision 8189: schemas/util.sql: Added type_qual()
Aaron Marcuse-Kubitza
02:34 PM Revision 8188: schemas/util.sql: mk_derived_col(): Added "idempotent" comment
Aaron Marcuse-Kubitza
02:23 PM Revision 8187: schemas/util.sql: Added mk_derived_col()
Aaron Marcuse-Kubitza
02:22 PM Revision 8186: inputs/FIA/COND/import: oldGrowth: Updated expr column names
Aaron Marcuse-Kubitza
01:49 PM Revision 8185: schemas/util.sql: Added typeof(text, regtype)
Aaron Marcuse-Kubitza
12:54 PM Revision 8184: inputs/FIA/*/import: Removed util. before function names because util is in the search_path
Aaron Marcuse-Kubitza
12:43 PM Revision 8183: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
Aaron Marcuse-Kubitza

03/25/2013

11:19 PM Revision 8182: schemas/functions.sql: Added existing_cols()
Aaron Marcuse-Kubitza
11:12 PM Revision 8181: schemas/functions.sql: col_type(): Fixed bug where a NULL col name crashed the undefined_column throw, because MESSAGE can't be NULL and the NULL name was nulling out the entire message
Aaron Marcuse-Kubitza
11:08 PM Revision 8180: schemas/functions.sql: Added col_exists()
Aaron Marcuse-Kubitza
10:31 PM Revision 8179: inputs/FIA/COND/map.csv: Mapped SLOPE, ASPECT
Aaron Marcuse-Kubitza
10:23 PM Revision 8178: web/main/.htaccess: remove linewraps (of the form table.path.vg/_-term) used to create a newline for Google spreadsheets
Aaron Marcuse-Kubitza
09:45 PM Revision 8177: inputs/FIA/*/map.csv: Replaced . between table and column name with newline, so that table viewers like pgAdmin will display both the table and column name at the left edge of the header cell, rather than displaying only the table name because the column name doesn't fit. This fixes the problem of seeing a bunch of columns whose names all start with a table name, and not knowing what each of them is. It also preserves the ability to see at a glance which table a column is in, which helps in navigating wide tables. Removed * before unmapped terms, because whether a term is mapped is generally obvious from the table name itself.
Aaron Marcuse-Kubitza
09:01 PM Revision 8176: inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)
Aaron Marcuse-Kubitza
08:45 PM Revision 8175: Added bin/fix_line_endings to standardize \r\n line endings to \n
Aaron Marcuse-Kubitza
08:12 PM Revision 8174: inputs/FIA/COND/import: Renamed COND.oldgrowth to VegCore name oldGrowth
Aaron Marcuse-Kubitza
07:52 PM Revision 8173: inputs/FIA/*/map.csv: Ensured that joined columns are globally unique, so they don't map to an ambiguous VegCore term in the future
Aaron Marcuse-Kubitza
07:38 PM Revision 8172: inputs/FIA/*/map.csv: Mapped terms to VegCore
Aaron Marcuse-Kubitza
07:22 PM Revision 8171: schemas/functions.sql: col_type(): Include column name in error message
Aaron Marcuse-Kubitza
06:57 PM Revision 8170: inputs/FIA/*/import: Updated column names to match map.csv
Aaron Marcuse-Kubitza
06:47 PM Revision 8169: schemas/functions.sql: col_type(): Raise undefined_column exception if column does not exist, instead of silently returning NULL
Aaron Marcuse-Kubitza
06:34 PM Revision 8168: inputs/FIA/import: Abort if any invoked script encounters an error
Aaron Marcuse-Kubitza
05:44 PM Revision 8167: planning/timeline/timeline.2013.xls: Updated for current progress
Aaron Marcuse-Kubitza
04:55 PM Revision 8166: inputs/FIA/*/map.csv: Removed no longer needed leading . from joined fields (globally-unique terms), because functions.to_global_col_names() is not used anymore
Aaron Marcuse-Kubitza
04:46 PM Revision 8165: Added inputs/FIA/occurrence_all/, which combines all the core tables in a denormalized view. Note that it is not necessary to materialize this view into a (large) denormalized table, because the unique indexes and left/right joins allow the rows to be denormalized on the fly.
Aaron Marcuse-Kubitza
04:36 PM Revision 8164: inputs/FIA/*/import: Use map_table to set column names based on the contents of map.csv, instead of using functions.to_global_col_names() and functions.rename_if_exists(). Added map.csv for all tables.
Aaron Marcuse-Kubitza
03:19 PM Revision 8163: inputs/FIA/: Changed postprocess.sql scripts to import scripts that can be run directly. Added top-level inputs/FIA/import to run all of them together.
Aaron Marcuse-Kubitza
03:05 PM Revision 8162: inputs/FIA/COND/postprocess.sql: Removed trailing whitespace
Aaron Marcuse-Kubitza
02:25 PM Revision 8161: Added lib/import.sh, for use by new, simpler import scripts used by FIA. Note that for now, input.Makefile is still used to create map.csv.
Aaron Marcuse-Kubitza

03/22/2013

11:13 PM Revision 8160: inputs/input.Makefile: Moved postprocess.sql from $(exportHeader) to %/install because that is not part of the $(exportHeader) functionality. Added %/header.csv and use it in $(exportHeader).
Aaron Marcuse-Kubitza
11:05 PM Revision 8159: inputs/input.Makefile: $(catSrcs): Fixed bug where need to use $(nonHeaderSrcs) instead of $(srcs) to exclude header.csv
Aaron Marcuse-Kubitza
08:07 PM Revision 8158: schemas/functions.sql: map: Added additional columns that are present in the standard map spreadsheet format (filter, notes). These columns are necessary to make COPY FROM work, because it requires the # of columns to be the same in the input data and the output table.
Aaron Marcuse-Kubitza
07:39 PM Revision 8157: inputs/input.Makefile: Moved $(cleanup) from $(exportHeader) to %/install because this is not part of exportHeader's functionality
Aaron Marcuse-Kubitza
07:29 PM Revision 8156: inputs/input.Makefile: $(mkSrcMap): Use header.csv instead of the header of the CSVs, so that the column list in the map spreadsheet matches the actual DB table
Aaron Marcuse-Kubitza
07:18 PM Revision 8155: inputs/input.Makefile: %.sql/run: Change to the directory the file is located in, so that includes (\i) are relative to the file, rather than relative to whatever happens to be the current directory
Aaron Marcuse-Kubitza
07:15 PM Revision 8154: inputs/input.Makefile: %/install: Always generate a header.csv, even for CSV inputs with their own header. This will include the *actual* column names in the staging table, which may differ from their names in the CSVs (e.g. the addition of row_num). Note that header.csv is not included in the CSVs list itself, and will not override the header or dialect in them.
Aaron Marcuse-Kubitza
06:09 PM Revision 8153: schemas/functions.sql: Added set_col_names()
Aaron Marcuse-Kubitza
06:08 PM Revision 8152: schemas/functions.sql: rename_if_exists(): Also ignore duplicate_column exceptions, which are generated when a column is renamed to itself (as well as when two columns are renamed to the same place)
Aaron Marcuse-Kubitza
06:02 PM Revision 8151: schemas/functions.sql: Added col_names(regclass), which unlike col_names(regtype) returns names in the order they are in the table
Aaron Marcuse-Kubitza
04:56 PM Revision 8150: schemas/functions.sql: Added map_values()
Aaron Marcuse-Kubitza
02:50 PM Revision 8149: schemas/functions.sql: map_get(): Fixed bug where can't use STRICT in EXECUTE INTO because there will sometimes be no match, causing a "query returned no rows" error
Aaron Marcuse-Kubitza
02:33 PM Revision 8148: schemas/functions.sql: rename_cols(): Support any renames type with an -> operator
Aaron Marcuse-Kubitza
02:27 PM Revision 8147: schemas/functions.sql: Added operator ->(regclass, text)
Aaron Marcuse-Kubitza
01:49 PM Revision 8146: schemas/functions.sql: Added map_get()
Aaron Marcuse-Kubitza
01:38 PM Revision 8145: schemas/functions.sql: table2hstore(): Made it STABLE instead of IMMUTABLE because the input table is not constant
Aaron Marcuse-Kubitza
01:36 PM Revision 8144: schemas/functions.sql: Added table2hstore()
Aaron Marcuse-Kubitza
01:34 PM Revision 8143: schemas/functions.sql: Added reset_map_table()
Aaron Marcuse-Kubitza
12:51 PM Revision 8142: schemas/functions.sql: Added truncate()
Aaron Marcuse-Kubitza
12:28 PM Revision 8141: schemas/functions.sql: mk_map_table(): Use the sql language instead of plpgsql because EXECUTE is not used directly, so plpgsql is not actually needed
Aaron Marcuse-Kubitza
12:25 PM Revision 8140: schemas/functions.sql: mk_map_table(): Store map table schema in separate `map` table and extend it using LIKE, for easier maintainability of the map schema
Aaron Marcuse-Kubitza

03/21/2013

08:27 AM Revision 8139: schemas/functions.sql: Added mk_map_table()
Aaron Marcuse-Kubitza
07:54 AM Revision 8138: schemas/functions.sql: ensure_prefix(): Made it IMMUTABLE instead of STABLE
Aaron Marcuse-Kubitza
07:37 AM Revision 8137: schemas/functions.sql: Added rename_cols()
Aaron Marcuse-Kubitza
06:01 AM Revision 8136: inputs/FIA/*/postprocess.sql: Avoid using :table, :table_str so that the commands in the script can also be run by pasting them into pgAdmin
Aaron Marcuse-Kubitza
02:13 AM Revision 8135: README.TXT: Full database import: Manual steps to run TNRS/remake analytical DB: Added `export version=<version>` to ensure that the import is run into the correct schema. Since these instructions are for running commands separately from the rest of the import, it's important to first ensure that the import environment is set up properly.
Aaron Marcuse-Kubitza
02:08 AM Revision 8134: schemas/vegbien.ERD.mwb: Added taxon_trait to ERD
Aaron Marcuse-Kubitza
02:04 AM Revision 8133: schemas/vegbien.ERD.mwb: Regenerated exports
Aaron Marcuse-Kubitza
01:58 AM Revision 8132: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
Aaron Marcuse-Kubitza
01:53 AM Revision 8131: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
Aaron Marcuse-Kubitza
01:27 AM Revision 8130: inputs/FIA/*/postprocess.sql: Added index on *.CN (autogen IDs)
Aaron Marcuse-Kubitza
01:25 AM Revision 8129: README.TXT: Full database import: Added steps to use `screen` to allow recovering from a closed terminal window
Aaron Marcuse-Kubitza
01:08 AM Revision 8128: inputs/FIA/TREE/postprocess.sql: TREE.unique index: Renamed to TREE.ID because this is on an autogenerated pkey rather than on domain values (for which a set of unique columns has not yet been found and may not exist)
Aaron Marcuse-Kubitza
01:03 AM Revision 8127: inputs/FIA/REF_SPECIES/postprocess.sql: Matched SPECIES_SYMBOL to .SYMBOL. Added .SYMBOL_TYPE for use in joining to REF_PLANT_DICTIONARY.
Aaron Marcuse-Kubitza
12:41 AM Revision 8126: Added inputs/FIA/REF_UNIT/postprocess.sql
Aaron Marcuse-Kubitza
12:36 AM Revision 8125: Added inputs/FIA/REF_RESEARCH_STATION/postprocess.sql
Aaron Marcuse-Kubitza
12:17 AM Revision 8124: Added inputs/FIA/COUNTY/postprocess.sql
Aaron Marcuse-Kubitza
12:02 AM Revision 8123: Added inputs/FIA/REF_PLANT_DICTIONARY/postprocess.sql
Aaron Marcuse-Kubitza

03/20/2013

05:16 PM Revision 8122: inputs/FIA/COND/postprocess.sql: Matched COND.HABTYPCD1, COND.HABTYPCD1_PUB_CD to REF_HABTYP_DESCRIPTION
Aaron Marcuse-Kubitza
05:07 PM Revision 8121: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to run postprocess.sql before exporting the header, because *it* can change the column names
Aaron Marcuse-Kubitza
05:02 PM Revision 8120: inputs/input.Makefile: Staging tables installation: $(exportHeader): export the header before running $(cleanup), because the header is not affected by the data cleanup operations and thus can be generated right away, to allow mapping while the cleanup operations run
Aaron Marcuse-Kubitza
04:30 PM Revision 8119: inputs/FIA/REF_HABTYP_DESCRIPTION/postprocess.sql: Prepare columns for joining with COND
Aaron Marcuse-Kubitza
03:23 PM Revision 8118: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to use psql_script_vegbien instead of the psql_verbose_vegbien used by $(psqlAsBien), to avoid echoing commands as part of the exported header
Aaron Marcuse-Kubitza
03:10 PM Revision 8117: Added planning/workflow/(de)normalized_import.mappings.png
Aaron Marcuse-Kubitza
03:04 PM Revision 8116: Added planning/workflow/denormalized_import.png, normalized_import.png
Aaron Marcuse-Kubitza
10:37 AM Revision 8115: web/main/IH/: Added lowercase alias
Aaron Marcuse-Kubitza
10:32 AM Revision 8114: Added web/main/IH/
Aaron Marcuse-Kubitza
10:12 AM Revision 8113: inputs/input.Makefile: Staging tables installation: Added postprocess target, which runs all the postprocess.sql files
Aaron Marcuse-Kubitza
09:34 AM Revision 8112: inputs/FIA/REF_SPECIES/postprocess.sql: Cast ID column to integer
Aaron Marcuse-Kubitza
08:52 AM Revision 8111: inputs/FIA/*/postprocess.sql: Cluster tables by their *.unique index for faster joins
Aaron Marcuse-Kubitza
08:51 AM Revision 8110: inputs/FIA/*/postprocess.sql: Cast ID columns to integer using new functions.set_col_types()
Aaron Marcuse-Kubitza
08:49 AM Revision 8109: bin/psql_verbose_vegbien: Run with client_min_messages = NOTICE to display notices for debugging. This is supposed to be the default, but apparently isn't.
Aaron Marcuse-Kubitza
08:47 AM Revision 8108: inputs/input.Makefile: BIEN commands: $(psqlAsBien): Use psql_verbose_vegbien instead of psql_script_vegbien so that timings and notices are displayed, which is useful for profiling and debugging
Aaron Marcuse-Kubitza
08:32 AM Revision 8107: schemas/functions.sql: Added col_cast and set_col_types()
Aaron Marcuse-Kubitza
07:45 AM Revision 8106: schemas/functions.sql: Added col_ref, col_type()
Aaron Marcuse-Kubitza
06:51 AM Revision 8105: schemas/functions.sql: Added cluster_once()
Aaron Marcuse-Kubitza
06:36 AM Revision 8104: schemas/functions.sql: Added cluster_index()
Aaron Marcuse-Kubitza
05:55 AM Revision 8103: schemas/functions.sql: create_if_not_exists(): Also handle duplicate_column exceptions
Aaron Marcuse-Kubitza
05:54 AM Revision 8102: schemas/functions.sql: Added rename_if_exists()
Aaron Marcuse-Kubitza
05:48 AM Revision 8101: inputs/FIA/COND/postprocess.sql: Renamed oldgrowth to COND.oldgrowth so it wouldn't be renamed by to_global_col_names()
Aaron Marcuse-Kubitza
04:28 AM Revision 8100: inputs/FIA/COND/postprocess.sql: Added oldgrowth column as part of the postprocessing instead of as part of the view that left joins the core tables together. This avoids needing to regenerate the oldgrowth field whenever the view is queried or materialized.
Aaron Marcuse-Kubitza
04:01 AM Revision 8099: inputs/FIA/TREE/postprocess.sql: Added index on columns that join to parent tables
Aaron Marcuse-Kubitza
03:00 AM Revision 8098: inputs/FIA/*/postprocess.sql: Removed table prefix from globally-unique columns that should be joined on
Aaron Marcuse-Kubitza
02:25 AM Revision 8097: schemas/functions.sql: Marked STRICT functions as such
Aaron Marcuse-Kubitza
02:22 AM Revision 8096: schemas/functions.sql: col_global_names(): Treat any column name that contains . as already being globally unique, and don't prepend the table name. This allows renaming the table columns after running col_global_names(), without causing the table name to be re-prepended the next time col_global_names() is run.
Aaron Marcuse-Kubitza
02:09 AM Revision 8095: schemas/functions.sql: Added contains()
Aaron Marcuse-Kubitza
02:07 AM Revision 8094: schemas/functions.sql: Added create_if_not_exists()
Aaron Marcuse-Kubitza
01:28 AM Revision 8093: inputs/FIA/*/postprocess.sql: Use functions.to_global_col_names() to ensure that all column names are globally unique. This makes it easy to join the tables together without worrying about column name collisions.
Aaron Marcuse-Kubitza
01:15 AM Revision 8092: inputs/FIA/*/postprocess.sql: Use new functions.create_if_not_exists() to allow re-running postprocess.sql idempotently
Aaron Marcuse-Kubitza

03/19/2013

11:48 PM Revision 8091: inputs/input.Makefile: Staging tables installation: %/install: Use new %.sql/run to run postprocess.sql
Aaron Marcuse-Kubitza
11:47 PM Revision 8090: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
Aaron Marcuse-Kubitza
11:47 PM Revision 8089: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
Aaron Marcuse-Kubitza
10:43 PM Revision 8088: schemas/functions.sql: Added to_global_col_names()
Aaron Marcuse-Kubitza
10:22 PM Revision 8087: schemas/functions.sql: col_global_names(): Use new functions.ensure_prefix() to only add the table name prefix if it doesn't already exist. This makes the function idempotent.
Aaron Marcuse-Kubitza
10:19 PM Revision 8086: schemas/functions.sql: Added ensure_prefix()
Aaron Marcuse-Kubitza
10:17 PM Revision 8085: schemas/functions.sql: Added has_prefix()
Aaron Marcuse-Kubitza
10:09 PM Revision 8084: schemas/functions.sql: Added col_global_names()
Aaron Marcuse-Kubitza
09:59 PM Revision 8083: schemas/functions.sql: Added name(regtype)
Aaron Marcuse-Kubitza
09:43 PM Revision 8082: schemas/functions.sql: Added col_names()
Aaron Marcuse-Kubitza
09:27 PM Revision 8081: root Makefile: Installation: Fixed bug where need to run schemas/public/install separately because schemas/install installs only the util schemas
Aaron Marcuse-Kubitza
09:26 PM Revision 8080: root Makefile: Installation: install util schemas (temp functions py_functions) before inputs, so that inputs can use util functions in their postprocess.sql or create.sql scripts. (However, they must not use util functions in views or index functions, because these would be cascadingly deleted whenever the util schemas are reinstalled before an import.)
Aaron Marcuse-Kubitza
08:07 PM Revision 8079: README.TXT: Single datasource import: Added by_col=1 to all commands
Aaron Marcuse-Kubitza
02:28 AM Revision 8078: mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field
Aaron Marcuse-Kubitza
02:05 AM Revision 8077: lib/sql_io.py: mk_errors_table(): Create a unique index on the MD5 of the value and error instead of on the values directly, because some strings are too long to index (e.g. row 2537268 of MO.Specimen causes an error "index row size 3032 exceeds maximum 2712 for index [...] Values larger than 1/3 of a buffer page cannot be indexed")
Aaron Marcuse-Kubitza
12:49 AM Revision 8076: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza

03/16/2013

02:16 PM Revision 8075: bin/map: No mappings warning: Added explanation that this could also be due to no column name matches, and hint to check if you are importing the correct input table
Aaron Marcuse-Kubitza
01:45 PM Revision 8074: inputs/MO/: Renamed Specimen.2/ -> now available Specimen/
Aaron Marcuse-Kubitza
01:42 PM Revision 8073: inputs/MO/: Removed old import in Specimen/
Aaron Marcuse-Kubitza
01:33 PM Revision 8072: Refreshed MO
Aaron Marcuse-Kubitza
12:44 PM Revision 8071: csvs.py: TsvReader.next(): Fixed bug where empty line needs to be separately returned as [], because csv.reader would interpret it as EOF since the line ending has already been removed
Aaron Marcuse-Kubitza
12:25 PM Revision 8070: csvs.py: sniff(): TSVs: Turn off quoting because TSVs use \-escapes instead of quotes to escape delimeters, newlines, etc.
Aaron Marcuse-Kubitza
11:49 AM Revision 8069: csvs.py: InputRewriter.readline(): Surround function in a try block that prints all exceptions, so that debugging information is available if an error occurs when this stream is used as input for psycopg's copy_expert() (COPY FROM)
Aaron Marcuse-Kubitza
06:56 AM Revision 8068: Populated inputs/MO/import_order.txt
Aaron Marcuse-Kubitza
06:46 AM Revision 8067: Refreshed SALVIAS
Aaron Marcuse-Kubitza
06:33 AM Revision 8066: Added web/main/CTFS/
Aaron Marcuse-Kubitza
06:21 AM Revision 8065: inputs/SALVIAS/: Regenerated salvias_*.schema.sql from the MySQL version, to take advantage of my2pg improvements. The placeholder *_index columns which take the place of MySQL's inline index definitions have now been replaced by no-op CHECK constraints, so that there are no longer lots of dummy *_index columns in the map spreadsheets.
Aaron Marcuse-Kubitza
05:52 AM Revision 8064: Added web/main/Redmine/ alias to VegBIEN/Redmine/
Aaron Marcuse-Kubitza
05:51 AM Revision 8063: Added web/main/VegBIEN/Redmine/
Aaron Marcuse-Kubitza
05:48 AM Revision 8062: web/main/VegBIEN/.htaccess: Forward to new db/ subdir
Aaron Marcuse-Kubitza
05:47 AM Revision 8061: Added web/main/VegBIEN/db/
Aaron Marcuse-Kubitza
05:45 AM Revision 8060: web/main/**/.htaccess: Removed RewriteCond -l tests because one of the -d or -f tests will always also pass, making the -l test unnecessary
Aaron Marcuse-Kubitza
05:38 AM Revision 8059: web/main.conf: Added tolower RewriteMap
Aaron Marcuse-Kubitza
05:19 AM Revision 8058: web/main/.htaccess: use separate lowercase version when available: Also support input strings in mixed case which is not the default capitalization, in addition to all-lowercase strings
Aaron Marcuse-Kubitza
05:18 AM Revision 8057: web/main/.htaccess: use separate lowercase version when available: Generate the new dirname with a separate RewriteCond so its value can be used both in the -d test and in the replacement string, rather than separately for each
Aaron Marcuse-Kubitza
05:03 AM Revision 8056: web/main/.htaccess: translate dotpaths: Allow an unescaped . at the beginning of a filename, because this will never be a . separator. This adds support for hidden files in dir paths, which now won't be interpreted as dotpaths. However, regular files with extensions still need to have the filename escaped because it will otherwise be interpreted as a dotpath.
Aaron Marcuse-Kubitza
04:53 AM Revision 8055: web/main/.htaccess: Set Options +FollowSymLinks. It should be on by default ("All options except for MultiViews. This is the default setting." <http://httpd.apache.org/docs/2.2/mod/core.html#options>), but this makes sure it will always be enabled.
Aaron Marcuse-Kubitza
03:32 AM Revision 8054: web/main/.htaccess: Name the lowercased versions of dirs with a leading . (to make them hidden) instead of a trailing _ , to avoid having each dir listed twice in a row in the dir index
Aaron Marcuse-Kubitza
03:21 AM Revision 8053: Added web/main/TNRS/
Aaron Marcuse-Kubitza
03:15 AM Revision 8052: Added web/main/VegBank/
Aaron Marcuse-Kubitza
02:57 AM Revision 8051: Added web/main/BIEN2/
Aaron Marcuse-Kubitza
02:56 AM Revision 8050: web/main/index.php: Replaced - with . in namespaces to conform to new dotpath naming convention, which allows nesting of namespaces
Aaron Marcuse-Kubitza
02:35 AM Revision 8049: web/main/SALVIAS/.htaccess: Forward to new dd/ subdir
Aaron Marcuse-Kubitza
02:34 AM Revision 8048: Added web/main/SALVIAS/dd/
Aaron Marcuse-Kubitza
02:28 AM Revision 8047: web/main/DwC/.htaccess: Forward to new terms/ subdir
Aaron Marcuse-Kubitza
02:27 AM Revision 8046: Added web/main/DwC/terms/
Aaron Marcuse-Kubitza
01:51 AM Revision 8045: web/main/**/.htaccess: don't redirect subdir paths: Fixed bug where can only match non-empty string, because otherwise the rule would match this directory, which should still have its redirects processed
Aaron Marcuse-Kubitza
01:43 AM Revision 8044: web/main/index.php: Added back smaller spacing between the table columns
Aaron Marcuse-Kubitza
01:40 AM Revision 8043: web/main/main.css: blockquote: Removed right margin so there isn't a big space between the table columns in index.php, which results from nesting right-padded blockquotes inside one another
Aaron Marcuse-Kubitza
01:39 AM Revision 8042: web/main/index.php: Changed Brad-Boyle to just Brad because people's names only have to be unique within VegPath
Aaron Marcuse-Kubitza
01:32 AM Revision 8041: web/main/.htaccess: Added fallback redirect to VegCore for paths without a namespace. This can be used to link to specific VegCore terms without needing to include the VegCore namespace.
Aaron Marcuse-Kubitza
01:13 AM Revision 8040: Added web/main/VegBIEN/
Aaron Marcuse-Kubitza
01:13 AM Revision 8039: Added web/main/servers/vegbiendev/
Aaron Marcuse-Kubitza
01:13 AM Revision 8038: Added web/main/.phpPgAdmin/
Aaron Marcuse-Kubitza
01:12 AM Revision 8037: web/main/.phpMyAdmin/.htaccess: Set [redirect] flag in case the dest server is on the same machine as VegPath itself
Aaron Marcuse-Kubitza
12:25 AM Revision 8036: web/main/*/ lowercase versions: Renamed with _ suffix to avoid svn conflicts on case-insensitive filesystems such as Mac HFS+
Aaron Marcuse-Kubitza
12:24 AM Revision 8035: web/main/.htaccess: Support lowercase versions of mixed-case dirnames without breaking case-insensitive filesystems such as Mac HFS+
Aaron Marcuse-Kubitza
12:10 AM Revision 8034: web/main/: Added lowercase symlinks for mixed-case dirs to work with subdomain translation, which uses subdomains lowercased by DNS
Aaron Marcuse-Kubitza
12:08 AM Revision 8033: web/main/index.php: Use absolute URLs for dependencies to work with subdomain translation, which adds components to the URL path
Aaron Marcuse-Kubitza

03/15/2013

11:30 PM Revision 8032: web/main/**/.htaccess: Use RewriteRule instead of RedirectMatch to handle incremental redirects internally instead of issuing a (much slower) redirect to the web browser each time. This also handles edge cases better, as [last] RewriteRules can be used to control when to forward control to a subdir, and doesn't require prepending the path to the dir the .htaccess file is in. Note that this requires all gateway dirs (dirs with subdirs) to contain special RewriteRules to avoid redirecting subdir paths and handle DirectoryIndex; see web/main/DwC/.htaccess. Also note that the regexp of a catch-all RewriteRule must exactly follow the template for internal or external redirects; see web/main/SALVIAS/db/.htaccess for internal redirects and web/main/DwC/history/.htaccess for external redirects.
Aaron Marcuse-Kubitza
07:31 PM Revision 8031: web/main/.htaccess: translate dotpaths: Allow the part before the [] escape to contain [], to support labels that end in [] (like PHP array vars in the query string) labels with a simple array-subscript syntax (a[b]). This also shortens the regexp and makes it more readable without the \[\] in [^.\[\]/] . Note that this also allows invalid combinations of [] exprs (e.g. more than one per level or unbalanced []), which will still be translated but will probably not have the desired result.
Aaron Marcuse-Kubitza
07:16 PM Revision 8030: web/main/.htaccess: translate dotpaths: Inline the [] escape regexp into the main regexp, because it is now approximately the same length as the []-matching portion of the main regexp and this greatly simplifies the code by removing the extra RewriteCond. Note that the translation rule is now a plain regexp (run repeatedly until no match), which can be used in *any* programming language that supports Perl-compatible regular expressions, not just mod_rewrite.
Aaron Marcuse-Kubitza
07:06 PM Revision 8029: web/main/.htaccess: translate dotpaths: discardpath explanation: Clarified that the infinite loop resulted from reappending PATH_INFO (the Apache-matched filename)
Aaron Marcuse-Kubitza
07:02 PM Revision 8028: web/main/.htaccess: translate dotpaths: Require any [] escape to have the ] at the end of the level, to simplify the [] regexp
Aaron Marcuse-Kubitza
06:54 PM Revision 8027: web/main/.htaccess: translate dotpaths: Use a lookahead assertion to ensure that at least one character is matched as the head of the dotpath. This ensures that (.*/)? + the rest of the regexp does not match a path with a trailing /, which is a sealed /-path and not subject to dotpath translation.
Aaron Marcuse-Kubitza
06:46 PM Revision 8026: web/main/.htaccess: translate dotpaths: Only support one [] escape per dot-level to (greatly) shorten the [] regexp. This does not pose a problem for encoding . because the entire level can simply be enclosed in [].
Aaron Marcuse-Kubitza
06:17 PM Revision 8025: web/main/.htaccess: Uncommented ErrorDocument
Aaron Marcuse-Kubitza
06:15 PM Revision 8024: web/main/.htaccess: translate dotpaths: Removed separate sealing of the /-path, which is now performed by the main RewriteRule because it appends a / even if there is no . suffix. This does not cause an infinite loop because a character is always added (/), which prevents the previously-matched head (after the last / but before any .) from being matched again.
Aaron Marcuse-Kubitza
06:05 PM Revision 8023: web/main/.htaccess: translate dotpaths: Fixed bug where it's actually the portion *before* the . (but after the last /) that should be subject to []-unescaping, rather than the portion *after* the . . Fixed bug where [] escapes were not being unescaped because the wildcard .* group matched the whole head portion instead of allowing the []-captures to match.
Aaron Marcuse-Kubitza
03:17 PM Revision 8022: web/main/.htaccess: translate subdomain to path: Don't use expr RewriteConds because they are not supported by Apache 2.2. Instead issue an external redirect with the subdomain part of the hostname removed, for the purpose of changing HTTP_HOST so that the replacement is not performed again if the mod_rewrite rules are run more than once.
Aaron Marcuse-Kubitza
02:46 PM Revision 8021: web/main/**/.htaccess: Use RewriteEngine, inheriting the web/main/.htaccess rules, in order to translate dotpaths that follow /-paths to existing dirs
Aaron Marcuse-Kubitza
02:45 PM Revision 8020: web/main/DwC/.htaccess: Moved DwC.history redirect to web/main/DwC/history/.htaccess
Aaron Marcuse-Kubitza
02:40 PM Revision 8019: web/main/SALVIAS/db/.htaccess: Removed trailing / because for DB redirects, this is apparently necessary
Aaron Marcuse-Kubitza
02:30 PM Revision 8018: web/main/.phpMyAdmin/.htaccess: Prepend http:// to the dest URL stem, instead of requiring the dest URL to provide the protocol, because the two // are replaced with one / by Apache when mod_rewrite is on, creating an invalid URL
Aaron Marcuse-Kubitza
01:55 PM Revision 8017: web/main/.htaccess: translate dotpaths: Fixed bug where can't remove all [] escapes, because this would also remove [] from protected parts of the URL (i.e. before the last /). Instead just remove up to two [] escapes per dot-level, e.g. a.[b.c]=[d.e] -> a/b.c=d.e . Note that this removes the restriction against URLs containing "[]", because these [] will be left as-is as long as they are followed at some point by a / .
Aaron Marcuse-Kubitza
01:41 PM Revision 8016: Removed web/.htaccess because it is not part of any site (web/main/.htaccess is the top-level .htaccess file for the default site)
Aaron Marcuse-Kubitza
01:17 PM Revision 8015: web/main/.htaccess: translate dotpaths: replace all unescaped . with / : Allow empty components on either side of the . , to ensure that *all* unescaped dots are consistently replaced with / . Note that hidden directories that start with . will have the . preserved, because they have (or get) a trailing slash, which prevents dotpath from operating on them.
Aaron Marcuse-Kubitza
01:13 PM Revision 8014: web/main/.htaccess: translate dotpaths: remove all [] escapes: Also remove empty [] to allow permalinking clients to easily generate escaped strings by URL-encoding the value and enclosing it in [], which will now also work even if the value is empty. This change will break any destination URLs that use "[]", such as PHP scripts that use this to indicate an array variable in the query string. In these URLs, the [] must now be %-encoded.
Aaron Marcuse-Kubitza
01:00 PM Revision 8013: web/main/.htaccess: RewriteRules with [redirect] and http:// : Removed [redirect] because it is implied when the replacement is an absolute URL
Aaron Marcuse-Kubitza
12:57 PM Revision 8012: web/main/.htaccess: RewriteRules with [redirect]: Added last because apparently redirect will not immediately cause the redirect, and will instead continue rewriting the URL ("You will almost always want to use [R] in conjunction with [L] (that is, use [R,L]) because on its own, the [R] flag prepends http://thishost[:thisport] to the URI, but then passes this on to the next rule in the ruleset, which can often result in 'Invalid URI in request' warnings." <http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_r>)
Aaron Marcuse-Kubitza
12:34 PM Revision 8011: web/main/.htaccess: RewriteRules: Added [discardpath,noescape] flags to all rules, because these are needed for correct interpretation of the replacement and should be the default setting. For some rules they may be optional, but it is better to include them on all rules to ensure there is never an unexpected effect from omitting them.
Aaron Marcuse-Kubitza
12:29 PM Revision 8010: web/main/.htaccess: Added "RewriteOptions inherit" because this[1] should be the default setting, to ensure that global filter rules also occur in subdirs. Note that the presence of this directive in *this* .htaccess file does not solve this problem, but putting it there will remind writers of subdir .htaccess files that they need to include that directive.
[1] The value should really be InheritBefore, but this is not supported in Apache 2.2, which we have (http://httpd.ap... Aaron Marcuse-Kubitza
12:02 PM Revision 8009: web/main/SALVIAS/.htaccess: RedirectMatch for below top-level: Use /(.*?)/?$ suffix for consistency with other sources' RedirectMatch patterns
Aaron Marcuse-Kubitza
12:00 PM Revision 8008: web/main/TCS,VegX/.htaccess: Fixed bug where need to match dirs by themselves in addition to when followed by a filename
Aaron Marcuse-Kubitza
11:58 AM Revision 8007: web/main/DwC/.htaccess: Fixed bug where need to match dirs by themselves (/DwC/, /DwC/history/) in addition to when followed by a filename
Aaron Marcuse-Kubitza
11:44 AM Revision 8006: web/main/.htaccess: mod_rewrite: Removed unneeded RewriteBase, because the default setting for RewriteBase is actually the dir of the .htaccess file itself, rather than always / . ("RewriteRule can be used in per-directory config files (.htaccess). There it will act locally, i.e., the local directory prefix is stripped at this stage of processing and your rewriting rules act only on the remainder. At the end it is automatically added back to the path." <http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteBase>)
Aaron Marcuse-Kubitza
11:30 AM Revision 8005: web/main/SALVIAS/*/.htaccess: Added trailing / to Redirect paths, which works now that a trailing / is automatically added to all paths by dotpath translation
Aaron Marcuse-Kubitza
11:28 AM Revision 8004: web/main/SALVIAS/: Moved db/ and users/ actions to their corresponding subdirs. The rules will still be run in the appropriate order, because if a .htaccess exists in a subdir, it will be used in preference to the .htaccess in the parent dir.
Aaron Marcuse-Kubitza
10:26 AM Revision 8003: my2pg: Removed SETs for standard_conforming_strings, escape_string_warning because these are also prepended by my2pg.data, which this script's output is piped to
Aaron Marcuse-Kubitza
10:00 AM Revision 8002: my2pg.data: \0 removal: Fixed bug where need to handle all backslashes before the 0, because they might escape just each other (an even number of \) rather than also encoding the \0 (an odd number of \)
Aaron Marcuse-Kubitza
08:20 AM Revision 8001: web/main/.htaccess: translate dotpaths to /-paths: Added examples
Aaron Marcuse-Kubitza
08:04 AM Revision 8000: web/main/.htaccess: translate dotpaths to /-paths: Translate *all* .-separated components in the path to / instead of just those that exist in the filesystem. This allows per-source rules to match just a / for the path-element separator instead of sometimes / and other times . (depending on whether that portion of the dotpath had been translated). Support [] escapes that preserve any . they surround, e.g. a.[b.c] -> a/b.c . (Supporting []-escapes requires that the /-path be "sealed" by appending a / , to prevent the . -> / translation rule from reinterpreting a newly-unescaped [] sequence as a dotpath. This also requires that the translation rule ignore anything before the last /, because it could have been []-unescaped by a previous round of mod_rewrite, e.g. in another dir. All the per-source rules need to be retrofitted to support the new trailing / .) Note also the discardpath flag (http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_dpi) on all RewriteRules that perform a replacement on the input string (i.e. that have a pattern other than ^.*$). This flag is needed to avoid infinite loops, because otherwise, a critical bug in mod_rewrite causes it to *reappend* the filename portion of the input string (PATH_INFO) to the result, causing it to be present twice, e.g. a/b -> a/b/b (https://issues.apache.org/bugzilla/show_bug.cgi?id=38642).
Aaron Marcuse-Kubitza
03:09 AM Revision 7999: web/main/nimoy/: Moved to web/main/servers/ so it can be grouped with other resources that are servers
Aaron Marcuse-Kubitza
 

Also available in: Atom