Activity
From 03/15/2013 to 04/13/2013
04/13/2013
- 05:47 PM Revision 8473: web/.htaccess: bugfix: moved auto-add trailing / if not external redirect to before don't rewrite existing paths because the trailing / also needs to be added on paths which *are* existing paths, but are just missing the trailing /
- 08:56 AM Task #565: partition the TaxonDetermination table by row into the different types of determinations
- changed vertical/horizontal to row/column for clarity
- 06:37 AM Revision 8472: web/.htaccess: auto-add trailing / if not external redirect (limited form of DirectorySlash). this allows URLs without trailing /, for dirs which have a special external redirect to the datasource's homepage when there is no trailing /, but still auto-adds the trailing / for all other dirs.
- 06:17 AM Revision 8471: README.TXT: Debugging: mod_rewrite: cached redirects issue: changed to limit the cache clearing to when a *DirectorySlash*-related redirect is changed, since only those redirects are permanent, not the RewriteRule redirects (unless you explicitly set them as permanent)
- 06:03 AM Revision 8470: README.TXT: Debugging: mod_rewrite: added tip that whenever you change a RewriteRule, you *must* clear your browser's cache to ensure that a cached redirect is not used instead (temporary redirects are not supposed to be cached, but sometimes Firefox caches them anyway)
- 04:41 AM Revision 8469: README.TXT: umask: Removed -S because this causes the new umask to be echoed, and is not needed even when setting a symbolic umask
- 04:18 AM Revision 8468: README.TXT: Datasource setup:: added step to use umask to prevent files from becoming web-accessible
- 04:16 AM Revision 8467: README.TXT: Testing: what to put in your .profile: added `umask -S ug=rwx,o=` to prevent files from becoming web-accessible
- 04:05 AM Revision 8466: README.TXT: fixed leading whitespace
- 02:46 AM Revision 8465: lib/util.run: run_cmd (): renamed to run_args_cmd to clarify that this runs only the command line args command, not any command. added doc comment to clarify this.
- 02:41 AM Revision 8464: lib/util.run: EXIT trap: use on_exit() function which calls run_cmd instead of having run_cmd be the handler, so that users can override this function to perform other commands (or no commands) after the script is read
- 02:28 AM Revision 8463: lib/util.run: echo_func (): use the caller's FUNCNAME via the $FUNCNAME[] array instead of requiring them to pass it in the function args as `echo_func "$FUNCNAME" "$@"`
- 02:11 AM Revision 8462: README.TXT: Maintenance: added steps to synchronize a Mac's settings with my testing machine's
- 02:10 AM Revision 8461: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: moved swap=1 right before src= dest=, which it applies to. aligned whitespace of the various commands.
- 01:49 AM Revision 8460: README.TXT: Maintenance: steps to synchronize vegbiendev, jupiter, and your local machine: added steps to install dependencies
- 01:44 AM Revision 8459: README.TXT: Maintenance: Added steps to synchronize vegbiendev, jupiter, and your local machine when changes are made on vegbiendev
- 01:34 AM Revision 8458: README.TXT: converted spaces to tabs for easier readability in a whitespace-highlighting text editor, and so that pressing backspace deletes an indent rather than just a character (creating a misaligned indent)
04/11/2013
- 10:29 PM Revision 8457: Moved validation/ under planning/workflow/
- 10:28 PM Revision 8456: Moved planning/use_cases/ under planning/goals/
- 10:26 PM Revision 8455: Moved planning/milestones/ under planning/timeline/
- 10:23 PM Revision 8454: Added inputs/VegBIEN/planning shortcut to fs/planning
- 10:16 PM Revision 8453: web/.fs: symlink via datasources/VegBIEN/fs instead
- 10:14 PM Revision 8452: Added inputs/VegBIEN/fs symlink to /servers/vegbiendev/fs
- 05:56 PM Revision 8451: Added web/.gbif shortcut to datasources/GBIF
- 04:52 PM Revision 8450: web/**/.htaccess: Use [qsappend] flag with all RewriteRules to ensure that the original query string is never lost
- 04:30 PM Revision 8449: inputs/VegBank/.htaccess: Added RewriteCond to check that the REQUEST_FILENAME is not an existing dir, which is apparently needed to prevent the verify/ subdir from being interpreted as a redirect
- 02:45 PM Revision 8448: web/**/.htaccess: Removed preprocessing commands that need to run for every dir, because these are now inherited from the outermost .htaccess using InheritBefore. This allows the inner .htaccess files to focus just on the fallback rewrites, without needing lots of boilerplate code (except for the obligatory RewriteEngine/RewriteOptions/SetEnvIf preamble).
- 02:34 PM Revision 8447: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex, by deleting the /index filename. This ensures that any fallback redirect is not invoked, and instead the action corresponding to a trailing / (usually the display of a directory listing) is performed.
- 02:07 PM Revision 8446: web/**/.htaccess: Use InheritBefore RewriteOption so that the outermost dir's rules are always run first. This is necessary so that all the HTTP_HOST and dotpath handlers, etc. are applied before any fallback handlers, *without* needing to copy and paste each rule into *every* .htaccess file. (Note that the InheritBefore RewriteOption is only available in Apache 2.4.) The outermost .htaccess file, however, must *not* use InheritBefore because this would inherit rules from the global Apache config, which normally should not be run and which cause an infinite redirect loop.
- 01:56 PM Revision 8445: web/**/.htaccess: fallback rules which should not be inherited by subdirs: Filter using a RewriteCond on %{ENV:innermost_dir} so that the rule is only run if the innermost dir was this dir. (This is needed when using InheritBefore.)
- 01:36 PM Revision 8444: web/**/.htaccess: Added SetEnvIf directives to set the $innermost_dir env var, which is used by InheritBefore RewriteRule inheritance. You can easily populate the current dir name in jEdit by right-clicking the buffer tab and selecting Insert File Path (which requires the BufferTabs plugin), and then picking out the dir right before the .htaccess. If multiple dirs in the dir path have the same name (e.g. a/b/a/), add a distinguishing prefix to the inner dirs (this is rare).
04/10/2013
- 11:17 PM Revision 8443: web/.htaccess: Support rules being inherited by subdirs using InheritBefore, by setting an env var $innermost_dir that stores the name of the innermost subdir to contain an .htaccess file. This env var can then be tested for in a RewriteCond in rules that should only apply locally. This works because SetEnvIf processes directives from the outer dir to the inner dir, so that the inner dir's setting comes last and is the final value of $innermost_dir. Note that you must use SetEnvIf rather than SetEnv because "The internal environment variables set by [SetEnv] are set after most early request processing directives are run, such as access control and URI-to-filename mapping. If the environment variable you're setting is meant as input into this early phase of processing *such as the RewriteRule directive*, you should instead set the environment variable with SetEnvIf." [emphasis added] (http://httpd.apache.org/docs/current/mod/mod_setenvif.html#setenvif)
- 10:20 PM Revision 8442: web/.htaccess: don't rewrite existing paths: Changed match string to what is used in subdirs, so that if this rule is inherited, what is considered an existing path (i.e. a file or subdir) is restricted appropriately
- 10:18 PM Revision 8441: web/**/.htaccess: no-replacement RewriteRules: Removed the standard [discardpath,noescape] flags because they add unnecessary clutter and are not needed for this type of rule. Note that these flags are still required on regular RewriteRules which have a replacement string.
- 10:15 PM Revision 8440: web/**/.htaccess: no-replacement RewriteRules: Use just ^ as the match string because there is no need to actually match characters. This also makes it easy to visually tell what is a no-replacement RewriteRule vs. a RewriteRule that matches the entire path.
- 10:11 PM Revision 8439: web/.htaccess: Renamed $REQUEST_URI_NO_SLASH to $REQUEST_URI_no_/ to take advantage of (some) special chars being allowed in env var names
- 09:42 PM Revision 8438: Added web/people/
- 09:08 PM Revision 8437: web/.htaccess: fallback redirect: Use new VegCore/ instead of specifying the full wiki URL
- 09:07 PM Revision 8436: schemas/VegCore/.htaccess: Fixed bug where need to issue explicit redirect because path contains a fragment (#...), which must be sent to the browser
- 09:03 PM Revision 8435: Added schemas/VegCore/.htaccess
- 08:57 PM Revision 8434: Added web/.vegcore shortcut to exchange_schemas/VegCore
- 08:34 PM Revision 8433: Added root .htaccess (accessed via web/servers/vegbiendev/fs or one of its shortcuts)
- 08:28 PM Revision 8432: web/.htaccess: translate subdomain to path: Use new REQUEST_URI_NO_SLASH to avoid adding an automatic trailing / when a subdir was reached as a subdomain without a path (e.g. VegBank.vegpath.org), which should be equivalent to going to the subdir's homepage (vegpath.org/VegBank) rather than to its directory listing (vegpath.org/VegBank/ ; note the trailing /)
- 08:25 PM Revision 8431: web/.htaccess: set REQUEST_URI_NO_SLASH for use by translating subdomain to path
- 08:19 PM Revision 8430: Added web/.fs symlink to .vegbiendev/fs
- 01:53 PM Revision 8429: web/svn-web, wiki: Made them hidden so they wouldn't appear in the dir listing (which should not contain shortcuts to inner paths, as these would create significant clutter). web/svn unfortunately can't be made hidden because there is already a .svn dir.
- 01:47 PM Revision 8428: Added web/.nimoy, .vegbiendev shortcuts
- 01:44 PM Revision 8427: web/DwC, VegX: Replaced with versions in exchange_schemas/
- 01:39 PM Revision 8426: Moved web/TCS/ to schemas/
- 01:36 PM Revision 8425: Added web/exchange_schemas/ symlink
- 01:35 PM Revision 8424: Removed web/exchange_schemas/ in preparation for making it a symlink
- 01:34 PM Revision 8423: Added schemas/.htaccess
- 01:33 PM Revision 8422: Added web/exchange_schemas/
- 01:28 PM Revision 8421: Merged web/DwC,VegX with the corresponding dir in schemas/
- 01:19 PM Revision 8420: Moved web/BIEN2, IH into aggregators/
- 01:18 PM Revision 8419: Moved web/BIEN2, IH into aggregators/
- 01:09 PM Revision 8418: Added web/aggregators, primary_databases with symlinks to the applicable datasources
- 01:04 PM Revision 8417: web/.tnrs: Updated to point to datasources/.TNRS/
- 01:01 PM Revision 8416: Removed web/TNRS/ in preparation for replacing it with a symlink
- 12:59 PM Revision 8415: Added inputs/.TNRS/.htaccess
- 12:13 PM Revision 8414: web/.redmine/: Updated for new VegBIEN/ location
- 12:12 PM Revision 8413: web/.htaccess: Use new %{REQUEST_SCHEME} to construct self-refential URLs
- 10:14 AM Revision 8412: web/VegBIEN/: Moved to datasources/
- 10:07 AM Revision 8411: Added inputs/.htaccess, which parses dotpaths and adds trailing / when datasources/ subdir is listed. (DirectorySlash cannot be used for this because it would affect all subdirs, too, which must not have DirectorySlash on. Turning DirectorySlash off in each subdir would not work, because the / is added *as soon as* a valid filesystem path is formed, even if the inner dir would not have added the /.)
- 09:48 AM Revision 8410: web/**/.htaccess: internal redirects: Use ?&$0 instead of just ?$0 to prevent the query string from being reinterpreted as a dotpath by the destination dir
- 09:45 AM Revision 8409: web/VegBIEN/Redmine/*/.htaccess: Use relative URL paths. This requires adding a RewriteBase directive because these dirs can be reached by symlinks.
- 09:40 AM Revision 8408: inputs/SALVIAS/.htaccess: Removed test RedirectMatch directive
- 09:38 AM Revision 8407: web/**/.htaccess: RewriteOptions: Redirect dir paths without the trailing / to the datasource's homepage, and let dir paths with the trailing / point to the directory index. This is the opposite of the way it was before, and is more intuitive because the trailing / indicates a directory listing, while the trailing / would generally be omitted when linking to the datasource itself.
- 07:49 AM Revision 8406: web/**/.htaccess: handle DirectoryIndex subrequests, which append "index" to the dir
- 06:15 AM Revision 8405: web/**/.htaccess: RewriteOptions: Removed inherit because if the destination is not found in the current dir, a 404 should be returned instead of trying to invoke the parent dir's fallback handler (which applies only to paths in the parent dir). Ideally, we would use InheritBefore to get the subdomain preprocessing, etc., but the parent dir's fallback handler would still need to be turned off somehow when the rule was being run by a subdir.
- 05:59 AM Revision 8404: web/.htaccess: Only forward specific terms (subpaths) to VegCore, not also the dir index itself
- 05:15 AM Revision 8403: web/**/.htaccess: RewriteOptions: Added AllowNoSlash (available in Apache 2.4) so dir paths without the trailing / can be parsed by mod_rewrite
- 05:12 AM Revision 8402: web/.htaccess: Turned off DirectorySlash so that the dir without the trailing / can have a separate meaning (e.g. as the home page of a datasource, rather than a listing of the short URLs available for it)
- 03:33 AM Revision 8401: web/main.conf: <Directory .>: Added `Require all granted`, which is needed by Apache 2.4 to prevent 403 Forbidden errors. This change breaks compatibility with older versions of Apache, but is unfortunately required by 2.4.
- 03:29 AM Revision 8400: web/**/.htaccess: parse dotpath in the query string: Use just dotpath instead of dotpath.php because the extension is added automatically by MultiViews
- 03:28 AM Revision 8399: web/**/.htaccess: parse dotpath in the query string: Remove any index.* suffix, which is added by MultiViews in Apache 2.4
- 03:28 AM Revision 8398: web/main.conf: Updated the path of the <Directory> directive for the DocumentRoot
- 03:21 AM Revision 8397: web/.htaccess: Set Options +Indexes because this isn't the default in Apache 2.4
- 01:46 AM Revision 8396: web/.htaccess: DirectoryIndex: Use index instead of hardcoding index.php because MultiViews will now add the appropriate extension automatically
- 01:44 AM Revision 8395: web/.htaccess: Turn on MultiViews to allow auto-adding of the file extension
- 12:15 AM Revision 8394: inputs/**/.htaccess: Use a direct filesystem path rather than going through /svn (the Redmine web interface) because this loads much faster, and avoids needing to manually specify svn-web instead of svn for files that should be displayed in the browser instead of downloaded. It also adds support for files not in svn. Note that the use of a relative RewriteRule replacement requires RewriteBase when the directory is not under the document root, in order to construct the correct self-referential URL (http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritebase).
04/09/2013
- 10:59 PM Revision 8393: inputs/**/.htaccess: Removed no longer needed trailing / in paths
- 10:55 PM Revision 8392: inputs/CTFS/*/.htaccess: Use new direct shortcuts to /Redmine subtrees
- 10:47 PM Revision 8391: Added wiki shortcut to Redmine/wiki
- 10:43 PM Revision 8390: web/.htaccess: use separate lowercase version when available: Support paths without a trailing /
- 10:41 PM Revision 8389: web/.htaccess: use separate lowercase version when available: Fixed bug where need to use [last] flag, in order to rewrite in the new subdir instead of applying the rest of this dir's rules (which would forward to VegCore)
- 10:11 PM Revision 8388: web/: datasources indexed in VegPath: Link to the corresponding inputs/ dir instead, which now stores the data and the short URLs in one place
- 10:10 PM Revision 8387: web/.htaccess: Use %{REQUEST_URI} ("The path component of the requested URI" <http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritecond>) instead of /$0, in order to preserve whether or not the path had a trailing slash. This also ensures that the rule functions correctly if it is invoked in a subdir context, which may occur for the "remove www subdomain" rule.
- 10:01 PM Revision 8386: web/.htaccess: Removed "don't redirect subdir paths" because this is now handled by "don't rewrite existing paths" as described in r8375
- 09:48 PM Revision 8385: web/: datasources indexed in VegPath: Added .htaccess files to corresponding inputs/ dir so the short URLs can also be accessed via VegBIEN's own dir for the datasource
- 09:36 PM Revision 8384: inputs/input.Makefile: Removed .PRECIOUS from %/header.csv, %/map.csv so that these scripts are deleted on error. This is useful for the runscripts and for non-data dirs whose header.csv cannot be made.
- 09:33 PM Revision 8383: inputs/input.Makefile: SVN: Only run %/add on subdirs with visible (non-hidden) files. Subdirs with only hidden files (e.g. .htaccess) are assumed to be non-data dirs.
- 09:23 PM Revision 8382: inputs/input.Makefile: Staging tables installation: %/install: $(logInstall): Only use log file if log dir exists, to support non-data dirs
- 09:13 PM Revision 8381: inputs/input.Makefile: Staging tables installation: %/install: ignore errors in $(exportHeader) and $(cleanup) if the table does not exist (i.e. a non-data dir)
- 09:11 PM Revision 8380: inputs/input.Makefile: Staging tables installation: %/install: Don't run $(import_install_) for empty dirs because there is no data to import
- 04:37 PM Revision 8379: lib/sql.py: parse_exception(): typed_name_re: Fixed bug where need to require the "" around the table/column name, because otherwise, the regexp will try to match as few characters as possible, causing it to match only the first letter of the name
- 04:13 PM Revision 8378: web/**/.htaccess: Removed "handle DirectoryIndex" rule, which does not appear to be needed with the new dotpath format
- 04:10 PM Revision 8377: web/VegBank/.htaccess: RewriteRule: Fixed bug where need to match tables by themselves as well as followed by columns
- 03:59 PM Revision 8376: web/**/.htaccess: Added "don't rewrite existing paths" and "handle DirectoryIndex" rules to all .htaccess files, not just those for dirs that contain subdirs, to facilitate linking directly to files in the filesystem without first needing to retrofit the applicable .htaccess file.
- 03:42 PM Revision 8375: web/**/.htaccess: Restrict redirection only of fully-existing paths, rather than of any path whose *subdir* exists, in order to avoid having to hardcode the absolute path to the directory containing the .htaccess file. This more restrictive approach still works because if the subdir exists and contains an .htaccess file, this (outer) .htaccess file's RewriteRules will be ignored completely until the inner .htaccess file's rules have been run, so that any subdir redirects will happen first. This "hole-punch" effect causes an existing filesystem path to always take priority over a virtual path (i.e. a redirect) defined in an outer .htaccess file.
- 02:57 PM Revision 8374: Added web/datasources/ symlink to inputs/
- 02:49 PM Revision 8373: web/servers/vegbiendev/fs: Updated symlink for new web/ dir nesting level
- 02:41 PM Revision 8372: Deleted no longer used www/
- 02:36 PM Revision 8371: Renamed www/ back to web/
- 02:34 PM Revision 8370: Removed www/logs/ because it was actually successfully moved into main/ (but some of its files weren't)
- 02:33 PM Revision 8369: Added www/logs/
- 02:24 PM Revision 8368: web/: Moved auxiliary files into the main/ subdir in preparation for having just the web/ dir. Renamed web/ to www/ so it can be replaced with web/main/.
- 02:09 PM Revision 8367: web/main/servers/vegbiendev/: Split it into db and fs forks, with db being the default. The fs fork links directly to /home/bien/svn on vegbiendev, which makes world-readable files directly web-accessible. (Permissions on /home/bien/svn and subdirs have been checked to ensure that private files are not world-readable.)
- 01:33 PM Revision 8366: Added web/main/IH/db/
- 01:26 PM Revision 8365: web/main/index.php: Updated fragment redirect for new dotpath format (using ? instead of a relative path)
- 01:20 PM Revision 8364: web/main/index.php: Updated path templates for new dotpath format (using ? instead of /)
- 01:06 PM Revision 8363: web/main/**/.htaccess: Support dotpaths in the query string instead of in the path, so that non-dotpath paths don't need to be suffixed with / to prevent their filenames from being interpreted as dotpaths. Putting dotpaths in the query string still requires only one character between the host and the path, but it's ? instead of / . ? is in many ways more natural, because the dotpath is a non-filesystem string to be parsed rather than something that's already a filesystem path. This change also avoids the need to strip trailing /s in many RewriteRules, because the dotpath mechanism is no longer appending them.
- 01:00 PM Revision 8362: Added web/main/dotpath.php, which parses any dotpath in the query string
- 12:58 PM Revision 8361: web/main/util.php: Added coalesce()
- 10:18 AM Revision 8360: web/main/svn*: Fixed symlinks to use .redmine instead of Redmine
- 09:37 AM Revision 8359: web/main/IH/.htaccess: Fixed whitespace
- 09:36 AM Revision 8358: web/main/IH/.htaccess: RewriteRule: Fixed bug where need to \-escape % because it's a special character in the replacement string (it references a regexp group matched by the last RewriteCond)
- 09:28 AM Revision 8357: Removed web/main/Redmine symlink so that it isn't listed in the directory listing as a visible (non-hidden) file. (There is already a .redmine symlink which matches all case variations via web/main/.htaccess logic.)
- 09:22 AM Revision 8356: Added web/main/svn* symlinks to VegBIEN/Redmine/svn*
- 09:18 AM Revision 8355: web/main/svn*/: Removed in preparation for replacing with symlinks, which will work now that absolute paths are used where needed
- 09:15 AM Revision 8354: web/main/**/.htaccess: internal redirects using relative paths with .. : Use absolute paths instead so that when the directory is reached through a symlink, the redirect will still work. Note that relative paths without .. do not need to be absolute paths because the subtree structure is the same (just the parent dirs are different).
- 08:01 AM Revision 8353: schemas/VegCore/VegCore.ERD.mwb: Regenerated exports
- 07:26 AM Revision 8352: schemas/VegCore/VegCore.ERD.mwb legend images: resized to medium-sized squares because apparently MySQL Workbench can't handle rectangular or very large images (it distorts them or resets them to their original size when the diagram is reloaded)
- 06:08 AM Revision 8351: schemas/VegCore/VegCore.ERD.mwb: Legend: Added IS-A, HAS-A, and "inherits from record" entries
- 06:08 AM Revision 8350: Added lib/MySQL_Workbench/connector.png, solid_line.png, dotted_line.png
- 05:00 AM Revision 8349: schemas/VegCore/VegCore.ERD.mwb: Fixed lines
- 04:59 AM Revision 8348: schemas/VegCore/VegCore.ERD.mwb: Renamed placename to named_place to match the table name
- 04:55 AM Revision 8347: schemas/VegCore/VegCore.ERD.mwb: Renamed measurement to trait because this is the more commonly used name for the entity
- 04:47 AM Revision 8346: schemas/VegCore/VegCore.ERD.mwb: Added VegCore logo
- 04:46 AM Revision 8345: Added schemas/VegCore/VegCore.logo.svg
- 04:09 AM Revision 8344: schemas/VegCore/VegCore.ERD.mwb: taxon: Added recursive parent fkey for (optionally) storing taxon hierarchies
- 03:42 AM Revision 8343: schemas/VegCore/VegCore.ERD.mwb: Repositioned taxon_determination
- 03:39 AM Revision 8342: schemas/VegCore/VegCore.ERD.mwb: Moved taxon next to qualified_taxon instead of above it, because inheritance (IS-A) is shown vertically while HAS-A is shown horizontally
- 03:25 AM Revision 8341: schemas/VegCore/VegCore.ERD.mwb: Populated legend
04/05/2013
- 12:23 AM Revision 8340: schemas/VegCore/VegCore.ERD.mwb: Added exports
- 12:23 AM Revision 8339: schemas/VegCore/VegCore.ERD.mwb: Fixed lines and settings for the Linux MySQL Workbench
04/04/2013
- 10:01 PM Revision 8338: schemas/VegCore/VegCore.ERD.mwb: Added table colors
- 10:01 PM Revision 8337: Removed backup file schemas/VegCore/VegCore.ERD.mwb.bak
- 09:48 PM Revision 8336: Added schemas/VegCore/VegCore.ERD.mwb, VegCore.my.sql with first VegCore ERD and MySQL schema. All tables are in the ERD, but contain only pkey and fkey columns.
- 09:52 AM Revision 8335: lib/sql.py: mk_select(): using subset function: Turn off enable_sort (within the transaction) to avoid unwanted slow sorts. This change (along with the subset functions themselves) should significantly reduce the long FIA.occurrence_all table subset time (~8 hours altogether) and with it the total import time, which had more than doubled as a result of the FIA refresh. Note that this issue would have been even more pronounced for larger datasets, such as the GBIF refresh, which would have taken ~2.5 days longer (400 million rows * ~30% are plants * (FIA: ~8 hours/16.7 million rows) * 1 day/24 hours).
- 09:30 AM Revision 8334: lib/sql.py: mk_select(): Use subset function when it's available for fast querying at large OFFSET values
- 09:29 AM Revision 8333: lib/sql.py: Added has_subset_func()
- 08:48 AM Revision 8332: inputs/FIA/occurrence_all/import: Run mk_subset_by_row_num_func() to make the subset functions available for fast querying at large OFFSET values
- 08:43 AM Revision 8331: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to add 1 to the 0-based offset_ to get the 1-based row_num (which is usually a serial column)
- 08:38 AM Revision 8330: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to subtract 1 from the end row_num because BETWEEN limits are inclusive of the bounds
- 08:33 AM Revision 8329: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where also need to COALESCE() offset_ to 0 when it's added to the limit_
- 08:20 AM Revision 8328: schemas/util.sql: mk_subset_by_row_num_func(): subset function which turns off enable_sort: Fixed bug where need to pass ($2, $3) instead of ($1, $2) to the regular subset function
- 08:14 AM Revision 8327: inputs/FIA/occurrence_all/import: Added occurrence_all-row_num column for use with mk_subset_by_row_num_func()
- 08:12 AM Revision 8326: schemas/util.sql: mk_subset_by_row_num_func(): Also create subset function which turns off enable_sort. This is used for limit values greater than ~100,000 to avoid unwanted slow sorts. The regular subset function is still needed to work with EXPLAIN, so that it produces expanded output instead of just a function scan.
- 07:27 AM Revision 8325: schemas/util.sql: Added mk_subset_by_row_num_func()
- 07:10 AM Revision 8324: schemas/util.sql: Added type_qual_name()
- 06:33 AM Revision 8323: schemas/util.sql: force_update_view(): Fixed bug where also need to drop view for "cannot change name of view column" errors
- 05:24 AM Revision 8322: inputs/FIA/occurrence_all/import: Use new force_update_view(), which only drops the view if its columns have changed and otherwise just uses CREATE OR REPLACE VIEW, rather than always first running DROP VIEW IF EXISTS
- 05:20 AM Revision 8321: schemas/util.sql: Added force_update_view()
- 04:23 AM Revision 8320: bin/make_analytical_db: Commented out export_analytical_db because we are not yet using the analytical DB in MySQL, and it doesn't make sense to generate a large, unused CSV export each time
- 04:19 AM Revision 8319: bin/export_analytical_db: Replaced analytical_aggregate with analytical_stem
- 03:53 AM Revision 8318: inputs/FIA/occurrence_all/: Updated header.csv for new column order
- 03:40 AM Revision 8317: inputs/FIA/occurrence_all/import: Use directional joins (LEFT/RIGHT JOIN) instead of inner joins to ensure that the PostgreSQL query planner always joins starting with the TREE table. Note that the directional joins are now needed for a different reason than when they were initially added, which had been to avoid slow sorts. The sorts (at least for LIMIT-only queries) went away when small tables such as COUNTY and REF_UNIT were added to the joins.
- 01:16 AM Revision 8316: inputs/FIA/*/map.csv: Changed newlines between table and field name to - because the newlines mess up the flow of queries and also break pgAdmin's display of EXPLAIN output. The - was chosen because it's a non-whitespace character that linewraps in browsers, phpPgAdmin, and Google spreadsheets (although unfortunately not in pgAdmin). It is better than space because you can set a text editor to treat it as a word character, allowing the entire column name (<table>-<field>) to be selected by double-clicking it.
04/03/2013
- 09:55 PM Revision 8315: Added planning/workflow/normalized_vs_denormalized/denormalized.generic_standardizations.png (a slide from Brad's bien3_architecture_denormalized.pptx PowerPoint), which shows the staging table preprocessing particularly well
- 09:45 PM Revision 8314: README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added instructions for what to do if the rightmost imports start getting truncated due to the 255-column limit in spreadsheets. (This will occur in 8 imports.)
- 09:32 PM Revision 8313: inputs/import.stats.xls: Removed the previous imports from the current tab because they are also in the 2012-6~9 tab, and should not be in two places
- 09:28 PM Revision 8312: inputs/import.stats.xls: Updated import times. MO and FIA have been refreshed.
04/02/2013
- 04:17 PM Revision 8311: Removed no longer needed inputs/GBIF/import. Use ./run instead.
- 04:17 PM Revision 8310: Removed no longer needed inputs/GBIF/_MySQL/import. Use ./run instead.
- 04:16 PM Revision 8309: inputs/GBIF/_MySQL/run: import: Run make directly instead of via ./import
- 04:15 PM Revision 8308: inputs/GBIF/_MySQL/run: Use new import.run, which defines all()
- 04:06 PM Revision 8307: Added planning/workflow/normalized_vs_denormalized/bien3_architecture_(de)normalized.pptx
- 03:57 PM Revision 8306: Added planning/workflow/normalized_vs_denormalized/BIEN-modArch-Dec2010 NS-SBD 1.4.ppt.url
- 03:50 PM Revision 8305: planning/workflow/: Moved normalized vs. denormalized files to separate normalized_vs_denormalized/ subfolder
- 03:21 PM Revision 8304: Regenerated inputs/ACAD/Specimen/logs/steps.by_col.log.sql
- 03:15 PM Revision 8303: inputs/GBIF/raw_occurrence_record/run: Override MySQL_export() so $filter can be customized
- 03:13 PM Revision 8302: inputs/GBIF/table.run: import(): Updated for lib/table.run template changes
- 03:09 PM Revision 8301: lib/table.run: template: import(): Also pass "$@" to superclass method
- 03:08 PM Revision 8300: lib/table.run: template: Use "$FUNCNAME" instead of hardcoding import
- 03:02 PM Revision 8299: Added inputs/GBIF/MySQL_export, used by ./table.run
- 02:57 PM Revision 8298: lib/util.run: echo_func: Fixed bug where need to use BASH_LINENO[0] for the line #s to match up with the files. For some reason the required array indexes for BASH_SOURCE (1) and BASH_LINENO (0) differ by one.
- 02:51 PM Revision 8297: inputs/GBIF/run: Use new import.run, which defines all()
- 02:51 PM Revision 8296: lib/table.run: Use new import.run, which defines all()
- 02:49 PM Revision 8295: Added lib/import.run
- 02:48 PM Revision 8294: lib/util.run: echo_func: Include the line # of the function to make it easier to find where the code being run is
- 02:32 PM Revision 8293: lib/table.run: Added all (default target)
- 02:26 PM Revision 8292: lib/util.run: run_cmd: If bash exited with an error, don't run the "$@" command. This test is necessary because `trap run_cmd EXIT` will run run_cmd as the result of *any* exit from the shell, including an error.
- 02:21 PM Revision 8291: *run: Use -e option to bash on the #! line instead of separate `set -o errexit` line so that there is no issue with the `set -o errexit` line getting separated from the #! line (errexit is required for the scripts to work properly)
- 02:09 PM Revision 8290: lib/util.run: run_cmd: When no command specified, default to running the `all` command, just like make
- 02:07 PM Revision 8289: lib/util.run: Run run_cmd at shell exit (using trap) instead of requiring every runscript to have `run_cmd ` at the end of it
- 01:49 PM Revision 8288: Added inputs/GBIF/run
- 01:48 PM Revision 8287: Added inputs/GBIF/raw_occurrence_record/run
- 01:47 PM Revision 8286: Added inputs/GBIF/table.run
- 01:45 PM Revision 8285: Added inputs/GBIF/_MySQL/run
- 01:42 PM Revision 8284: lib/util.run: fwd: Check that $subdirs is defined. Added $subdirs to usage.
- 01:39 PM Revision 8283: lib/util.run: fwd: Added usage
- 01:32 PM Revision 8282: lib/table.run: Switched from echo_run to echo_func
- 01:16 PM Revision 8281: lib/util.run: run_cmd: Echo the command being run, including the top-level run script. This is in addition to the echoing of the command in the function itself (using echo_func), which provides *both* the runscript that was run *and* the file where the invoked command was actually located (which may be different due to includes).
- 01:12 PM Revision 8280: lib/util.run: Echo the command at the beginning of each function using new echo_func, instead of having to type echo_run before every call to a function. Note that because echo_func uses BASH_SOURCE, the path to the file containing the function will be included in the debug message, which greatly facilitates locating which file a command is in.
- 01:08 PM Revision 8279: lib/util.run: Added echo_func
- 12:50 PM Revision 8278: lib/util.run: Added echo_cmd and use it in echo_run
- 12:46 PM Revision 8277: lib/util.run: echo_cmd(): Renamed to echo_run for clarity, because it also runs the command
- 12:39 PM Revision 8276: lib/util.run: Added inline_make()
- 12:39 PM Revision 8275: lib/util.run: Added echo_stdin()
- 12:30 PM Revision 8274: bin/my2pg_export: Put --password first because it's an authentication-related option
- 10:52 AM Revision 8273: Added lib/table.run, which includes the commands in import.sh but uses run scripts to allow running commands other than just import. (For example, map_table or postprocess can be run separately. Uninstall-related commands which would not belong in an import script can also be added, because import is only one of many commands a run script can offer.)
- 10:35 AM Revision 8272: Added lib/util.run with general functions and template for run scripts (a bash-based replacement for make). Unlike make, run scripts support full bash functionality including multiline commands. The run script template also includes syntax for various kinds of relative includes in bash.
- 12:03 AM Revision 8271: lib/common.Makefile: Added $(require_var)
04/01/2013
- 10:42 PM Revision 8270: bin/publish_analytical_db: Fixed bug where need to remove `ESCAPED BY '"'` because this would causing " followed by an escape sequence char to be interpreted specially (e.g. "n -> \n). MySQL automatically takes care of quote doubling when you specify `FIELDS OPTIONALLY ENCLOSED BY`.
- 10:13 PM Revision 8269: lib/common.Makefile: Compression: Added `%:: %.gz`, `%.gz: %`
- 08:07 PM Revision 8268: planning/workflow/import_process_comparison.odg: Moved "staging tables" under the method labels to reduce empty space
- 07:52 PM Revision 8267: planning/workflow/import_process_comparison.odg: Removed margins so the labels would align with the page margin on the Import process wiki page <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Import_process>
- 07:32 PM Revision 8266: Added planning/workflow/import_process_comparison.odg and .png export
- 06:12 PM Revision 8265: lib/db_xml.py: put_table(): Fixed bug where command to advance start to fetch next set was unintentionally deleted when removing the is_view check
- 06:11 PM Revision 8264: inputs/UNCC/Specimen/new_terms.csv: Updated for updated VegCore vocab
- 03:53 PM Revision 8263: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5: Regenerated after appending agent table to GBIFPortalDB-2013-02-20.data.sql
- 03:51 PM Revision 8262: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.gz.md5
03/28/2013
- 08:16 PM Revision 8261: Added inputs/GBIF/raw_occurrence_record/ from refresh
- 08:07 PM Revision 8260: inputs/GBIF/MySQL.schema.sql: Regenerated with inline enum type translated to CHECK constraint
- 08:07 PM Revision 8259: bin/my2pg: Translate inline enum type to CHECK constraint
- 07:43 PM Revision 8258: Added inputs/GBIF/**/MySQL.schema.sql
- 07:42 PM Revision 8257: Added inputs/GBIF/_MySQL/MySQL.*.sql.make
- 07:36 PM Revision 8256: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
- 07:29 PM Revision 8255: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
- 07:22 PM Revision 8254: inputs/input.Makefile: SVN: add: Removed Source/map.csv prerequisite because it is not related to adding unversioned files in the dir. It was originally a prerequisite in order to auto-create it when the datasource dir is first created, but the map.csv recipe does not currently create metadata-only map.csvs. In the future, metadata-only map.csvs will be replaced with constant columns added to the applicable tables.
- 07:19 PM Revision 8253: Added inputs/FIA/_archive
- 07:19 PM Revision 8252: inputs/input.Makefile: %/map.csv: Fixed bug where can only make header.csv if map.csv does not exist, because some subdirs are metadata-only and don't have a corresponding DB table
- 07:02 PM Revision 8251: README.TXT: Datasource setup: Install the staging tables: For a MySQL .sql export: Documented which password to use at each of the two password prompts my2pg_export will give you. You could also embed the value of the 2nd prompt in the _MySQL/*.make file using `--password="$(cat path/to/config/bien_password)"`.
- 06:56 PM Revision 8250: README.TXT: Datasource setup: Install the staging tables: Removed requirement that `make inputs/<datasrc>/reinstall quiet=1 &` be run on vegbiendev for MySQL .sql exports, because the hostname is now set to vegbiendev instead of localhost
- 06:38 PM Revision 8249: inputs/input.Makefile: sql/install: Use psql_script_vegbien instead of $(psqlNoSearchPath) (which uses psql_verbose_vegbien) because the insert statement for each data row should not be echoed
- 06:14 PM Revision 8248: inputs/FIA/occurrence_all/import: Run remake_VegBIEN_mappings at end to keep mappings to next stage of import process up to date
- 06:14 PM Revision 8247: inputs/FIA/occurrence_all/: Accepted new test output
- 06:13 PM Revision 8246: lib/import.sh: remake_VegBIEN_mappings(): Also remake VegBIEN.csv and test.xml.ref use `make test`
- 06:11 PM Revision 8245: lib/import.sh: Added remake_VegBIEN_mappings()
- 06:10 PM Revision 8244: inputs/input.Makefile: %/map.csv: make $*/header.csv first in case it doesn't exist (e.g. if it has been deleted so that it will be remade)
- 06:07 PM Revision 8243: inputs/FIA/occurrence_all/map.csv: Regenerated using new input table mappings
- 05:47 PM Revision 8242: lib/import.sh: Added make() and use it instead of the full make command
- 05:23 PM Revision 8241: inputs/input.Makefile: postprocess: Use %/postprocess instead of %/postprocess.sql/run so $*/import is also run
- 05:21 PM Revision 8240: inputs/FIA/: Ran inputs/FIA/import. This maps to VegCore's commonName.
- 05:19 PM Revision 8239: inputs/input.Makefile: %/postprocess: Also run the $*/import script, if it exists. Note that this is not the same as the %/import make target.
- 05:12 PM Revision 8238: inputs/input.Makefile: %/postprocess.sql/run: Factored out into separate %/postprocess command, which can eventually also perform other actions
- 04:59 PM Revision 8237: inputs/FIA/PLOT/map.csv: ELEV: Remapped to elevation_ft, assuming units based on the actual elevation of the region for a sample plot record
- 04:27 PM Revision 8236: inputs/VegBank/taxonobservation_/map.csv: Mapped int_currplantcommon to vernacularName
- 04:25 PM Revision 8235: mappings/VegCore.htm: Renamed salvias_plots table plotMetadata to PlotMetadata because of SALVIAS refresh on nimoy
- 04:18 PM Revision 8234: mappings/VegCore.htm: Regenerated from wiki. Added flower, fruit, commonName.
- 03:37 PM Revision 8233: mappings/Makefile: $(vocab); bin/redmine_synonyms: Support crossed out (deprecated) terms
- 03:24 PM Revision 8232: README.TXT: Maintenance: VegCore data dictionary: Added steps to update the data dictionary's Tables section if necessary
- 02:14 PM Revision 8231: inputs/GBIF/_MySQL/Makefile: %.data.sql: Added agent table
- 01:18 PM Revision 8230: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5
- 01:11 PM Revision 8229: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.sql
- 11:02 AM Revision 8228: Added web/main/svn*/, now using .htaccess to forward to Redmine/*
- 10:55 AM Revision 8227: Removed web/main/svn, svn-web symlinks because they need to be .htaccess-es in order for the relative mod_rewrite commands to work correctly
- 10:50 AM Revision 8226: Added web/main/svn, svn-web symlinks to Redmine/* for shorter URLs
- 10:49 AM Revision 8225: Added web/main/Redmine/svn-web/
- 08:28 AM Revision 8224: inputs/GBIF/: Added scripts for subsetting refresh
- 12:24 AM Revision 8223: lib/sql.py: table_order_by(): Documented that it returns None if table is a view, because table_cluster_on() would return None. This is necessary for inputs/FIA/occurrence_all/ sorting to work correctly, because specifying a manual sort order would prevent the query planner from just using fast nested loop joins and instead cause it to perform a slow sort. (This appears to be a bug in the query planner, because when the column list specified matches the joined-on indexes, there should be no need for post-nested loop re-sorting.)
- 12:20 AM Revision 8222: inputs/FIA/occurrence_all/test.xml.ref: Updated inserted row count for new row sort order
- 12:19 AM Revision 8221: lib/db_xml.py: put_table(): Fixed bug where also need to advance start to fetch next set when table is a view, because the views that are now being used with the import (inputs/FIA/occurrence_all/) are static rather than dynamic and do not return different rows after the previous set of rows has been imported
03/27/2013
- 11:43 PM Revision 8220: inputs/FIA/occurrence_all/import: Removed no longer applicable comment that directional joins are needed for PostgreSQL query planner to avoid slow sorts
- 11:40 PM Revision 8219: inputs/FIA/TREE/import: Reclustered table by TREE.parent path index, to facilitate path-order joins
- 11:39 PM Revision 8218: inputs/FIA/occurrence_all/import: Changed all RIGHT JOINs to inner joins so that tables would be joined in path order (i.e. general->specific). This optimizes the incremental joins so that the small tables are joined to each other before being joined to the large tables, rather than each row of the large tables being looked up in the small tables. This effect may not be noticeable for small LIMIT values, but would become apparent for large LIMIT values, such as the 1-million-row partitions used by db_xml.put_table() for column-based import. Note that inner joins used to cause the query planner to produce incorrect results containing slow sorts, but now this appears to no longer be an issue, perhaps because the result is not sorted by the TREE.ID index (which is not in the same order as the path indexes *.unique, *.parent).
- 10:46 PM Revision 8217: inputs/FIA/occurrence_all/import: Removed trailing whitespace
- 10:30 PM Revision 8216: Removed unused inputs/FIA/COND_unique/. Use COND instead.
- 09:52 PM Revision 8215: inputs/FIA/import: Use `set -o errexit` instead of putting ` || exit` after each command
- 09:52 PM Revision 8214: lib/import.sh: map_table(): Removed unneeded () around psql. This also fixes a bug where an error exit status from psql would not have aborted the script because `set -o errexit` does not apply to commands enclosed in (). For () you need to use ` || exit` instead (or ` || return` inside a function).
- 09:42 PM Revision 8213: lib/import.sh: Use `set -o errexit` so any command that exits with an error aborts the script. Note that a command's exit status can still be ignored using ` || true`. Removed no longer needed ` || return` in functions.
- 09:40 PM Revision 8212: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
- 09:33 PM Revision 8211: lib/import.sh: functions: abort if a command encounters an error
- 09:17 PM Revision 8210: schemas/VegCore/mk_derived: Added cultivated from oldGrowth
- 09:16 PM Revision 8209: schemas/util.sql: Added try_mk_derived_col()
- 08:35 PM Revision 8208: inputs/FIA/*/import: Run mk_derived after postprocessing commands
- 08:28 PM Revision 8207: inputs/FIA/import_order.txt: Added occurrence_all/
- 08:23 PM Revision 8206: mappings/VegCore-VegBIEN.csv: subplotID,subplot -> location.sourceaccessioncode: Fixed bug where need /_first to handle the case where both subplotID and subplot are provided
- 08:15 PM Revision 8205: Added inputs/FIA/map.csv, which maps shared columns to VegCore
- 08:12 PM Revision 8204: inputs/FIA/FIA_COND_unique/test.xml.ref: Updated now that PLOT, CONDID have been mapped
- 08:12 PM Revision 8203: inputs/FIA/*/map.csv for pre-refresh tables: Added back * before unmapped column names
- 08:03 PM Revision 8202: lib/csvs.py: stream_info(): Fixed bug where headers with multiline columns were not supported because only the first line (not the first multiline row) is sniffed for the dialect
- 06:56 PM Revision 8201: inputs/input.Makefile: %/header.csv: Fixed bug where newlines inside column names were incorrectly formatted by psql's table header formatting, by using COPY TO STDOUT instead
- 05:28 PM Revision 8200: schemas/util.sql: Added do_optionally_ignore()
- 04:28 PM Revision 8199: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
- 04:12 PM Revision 8198: lib/import.sh: Added mk_derived(). Added mk_derived to usage template.
- 04:11 PM Revision 8197: Added schemas/VegCore/mk_derived, which will be run in the import scripts
- 04:09 PM Revision 8196: lib/import.sh: psql(): Set psql vars :schema, :table, :table_str for use by the psql commands
- 03:22 PM Revision 8195: lib/import.sh: Export $schema, $table so they are available to programs invoked within an import script, which should not reset these vars if they include import.sh
- 03:20 PM Revision 8194: lib/import.sh: Only set $table, $schema if they don't already exist
- 03:11 PM Revision 8193: lib/import.sh: Added $root_dir and use it in $bin_dir
- 03:11 PM Revision 8192: inputs/FIA/*/import: Use new mk_*_col()
- 02:50 PM Revision 8191: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
- 02:43 PM Revision 8190: schemas/util.sql: Added mk_const_col()
- 02:37 PM Revision 8189: schemas/util.sql: Added type_qual()
- 02:34 PM Revision 8188: schemas/util.sql: mk_derived_col(): Added "idempotent" comment
- 02:23 PM Revision 8187: schemas/util.sql: Added mk_derived_col()
- 02:22 PM Revision 8186: inputs/FIA/COND/import: oldGrowth: Updated expr column names
- 01:49 PM Revision 8185: schemas/util.sql: Added typeof(text, regtype)
- 12:54 PM Revision 8184: inputs/FIA/*/import: Removed util. before function names because util is in the search_path
- 12:43 PM Revision 8183: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
03/25/2013
- 11:19 PM Revision 8182: schemas/functions.sql: Added existing_cols()
- 11:12 PM Revision 8181: schemas/functions.sql: col_type(): Fixed bug where a NULL col name crashed the undefined_column throw, because MESSAGE can't be NULL and the NULL name was nulling out the entire message
- 11:08 PM Revision 8180: schemas/functions.sql: Added col_exists()
- 10:31 PM Revision 8179: inputs/FIA/COND/map.csv: Mapped SLOPE, ASPECT
- 10:23 PM Revision 8178: web/main/.htaccess: remove linewraps (of the form table.path.vg/_-term) used to create a newline for Google spreadsheets
- 09:45 PM Revision 8177: inputs/FIA/*/map.csv: Replaced . between table and column name with newline, so that table viewers like pgAdmin will display both the table and column name at the left edge of the header cell, rather than displaying only the table name because the column name doesn't fit. This fixes the problem of seeing a bunch of columns whose names all start with a table name, and not knowing what each of them is. It also preserves the ability to see at a glance which table a column is in, which helps in navigating wide tables. Removed * before unmapped terms, because whether a term is mapped is generally obvious from the table name itself.
- 09:01 PM Revision 8176: inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)
- 08:45 PM Revision 8175: Added bin/fix_line_endings to standardize \r\n line endings to \n
- 08:12 PM Revision 8174: inputs/FIA/COND/import: Renamed COND.oldgrowth to VegCore name oldGrowth
- 07:52 PM Revision 8173: inputs/FIA/*/map.csv: Ensured that joined columns are globally unique, so they don't map to an ambiguous VegCore term in the future
- 07:38 PM Revision 8172: inputs/FIA/*/map.csv: Mapped terms to VegCore
- 07:22 PM Revision 8171: schemas/functions.sql: col_type(): Include column name in error message
- 06:57 PM Revision 8170: inputs/FIA/*/import: Updated column names to match map.csv
- 06:47 PM Revision 8169: schemas/functions.sql: col_type(): Raise undefined_column exception if column does not exist, instead of silently returning NULL
- 06:34 PM Revision 8168: inputs/FIA/import: Abort if any invoked script encounters an error
- 05:44 PM Revision 8167: planning/timeline/timeline.2013.xls: Updated for current progress
- 04:55 PM Revision 8166: inputs/FIA/*/map.csv: Removed no longer needed leading . from joined fields (globally-unique terms), because functions.to_global_col_names() is not used anymore
- 04:46 PM Revision 8165: Added inputs/FIA/occurrence_all/, which combines all the core tables in a denormalized view. Note that it is not necessary to materialize this view into a (large) denormalized table, because the unique indexes and left/right joins allow the rows to be denormalized on the fly.
- 04:36 PM Revision 8164: inputs/FIA/*/import: Use map_table to set column names based on the contents of map.csv, instead of using functions.to_global_col_names() and functions.rename_if_exists(). Added map.csv for all tables.
- 03:19 PM Revision 8163: inputs/FIA/: Changed postprocess.sql scripts to import scripts that can be run directly. Added top-level inputs/FIA/import to run all of them together.
- 03:05 PM Revision 8162: inputs/FIA/COND/postprocess.sql: Removed trailing whitespace
- 02:25 PM Revision 8161: Added lib/import.sh, for use by new, simpler import scripts used by FIA. Note that for now, input.Makefile is still used to create map.csv.
03/22/2013
- 11:13 PM Revision 8160: inputs/input.Makefile: Moved postprocess.sql from $(exportHeader) to %/install because that is not part of the $(exportHeader) functionality. Added %/header.csv and use it in $(exportHeader).
- 11:05 PM Revision 8159: inputs/input.Makefile: $(catSrcs): Fixed bug where need to use $(nonHeaderSrcs) instead of $(srcs) to exclude header.csv
- 08:07 PM Revision 8158: schemas/functions.sql: map: Added additional columns that are present in the standard map spreadsheet format (filter, notes). These columns are necessary to make COPY FROM work, because it requires the # of columns to be the same in the input data and the output table.
- 07:39 PM Revision 8157: inputs/input.Makefile: Moved $(cleanup) from $(exportHeader) to %/install because this is not part of exportHeader's functionality
- 07:29 PM Revision 8156: inputs/input.Makefile: $(mkSrcMap): Use header.csv instead of the header of the CSVs, so that the column list in the map spreadsheet matches the actual DB table
- 07:18 PM Revision 8155: inputs/input.Makefile: %.sql/run: Change to the directory the file is located in, so that includes (\i) are relative to the file, rather than relative to whatever happens to be the current directory
- 07:15 PM Revision 8154: inputs/input.Makefile: %/install: Always generate a header.csv, even for CSV inputs with their own header. This will include the *actual* column names in the staging table, which may differ from their names in the CSVs (e.g. the addition of row_num). Note that header.csv is not included in the CSVs list itself, and will not override the header or dialect in them.
- 06:09 PM Revision 8153: schemas/functions.sql: Added set_col_names()
- 06:08 PM Revision 8152: schemas/functions.sql: rename_if_exists(): Also ignore duplicate_column exceptions, which are generated when a column is renamed to itself (as well as when two columns are renamed to the same place)
- 06:02 PM Revision 8151: schemas/functions.sql: Added col_names(regclass), which unlike col_names(regtype) returns names in the order they are in the table
- 04:56 PM Revision 8150: schemas/functions.sql: Added map_values()
- 02:50 PM Revision 8149: schemas/functions.sql: map_get(): Fixed bug where can't use STRICT in EXECUTE INTO because there will sometimes be no match, causing a "query returned no rows" error
- 02:33 PM Revision 8148: schemas/functions.sql: rename_cols(): Support any renames type with an -> operator
- 02:27 PM Revision 8147: schemas/functions.sql: Added operator ->(regclass, text)
- 01:49 PM Revision 8146: schemas/functions.sql: Added map_get()
- 01:38 PM Revision 8145: schemas/functions.sql: table2hstore(): Made it STABLE instead of IMMUTABLE because the input table is not constant
- 01:36 PM Revision 8144: schemas/functions.sql: Added table2hstore()
- 01:34 PM Revision 8143: schemas/functions.sql: Added reset_map_table()
- 12:51 PM Revision 8142: schemas/functions.sql: Added truncate()
- 12:28 PM Revision 8141: schemas/functions.sql: mk_map_table(): Use the sql language instead of plpgsql because EXECUTE is not used directly, so plpgsql is not actually needed
- 12:25 PM Revision 8140: schemas/functions.sql: mk_map_table(): Store map table schema in separate `map` table and extend it using LIKE, for easier maintainability of the map schema
03/21/2013
- 08:27 AM Revision 8139: schemas/functions.sql: Added mk_map_table()
- 07:54 AM Revision 8138: schemas/functions.sql: ensure_prefix(): Made it IMMUTABLE instead of STABLE
- 07:37 AM Revision 8137: schemas/functions.sql: Added rename_cols()
- 06:01 AM Revision 8136: inputs/FIA/*/postprocess.sql: Avoid using :table, :table_str so that the commands in the script can also be run by pasting them into pgAdmin
- 02:13 AM Revision 8135: README.TXT: Full database import: Manual steps to run TNRS/remake analytical DB: Added `export version=<version>` to ensure that the import is run into the correct schema. Since these instructions are for running commands separately from the rest of the import, it's important to first ensure that the import environment is set up properly.
- 02:08 AM Revision 8134: schemas/vegbien.ERD.mwb: Added taxon_trait to ERD
- 02:04 AM Revision 8133: schemas/vegbien.ERD.mwb: Regenerated exports
- 01:58 AM Revision 8132: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
- 01:53 AM Revision 8131: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
- 01:27 AM Revision 8130: inputs/FIA/*/postprocess.sql: Added index on *.CN (autogen IDs)
- 01:25 AM Revision 8129: README.TXT: Full database import: Added steps to use `screen` to allow recovering from a closed terminal window
- 01:08 AM Revision 8128: inputs/FIA/TREE/postprocess.sql: TREE.unique index: Renamed to TREE.ID because this is on an autogenerated pkey rather than on domain values (for which a set of unique columns has not yet been found and may not exist)
- 01:03 AM Revision 8127: inputs/FIA/REF_SPECIES/postprocess.sql: Matched SPECIES_SYMBOL to .SYMBOL. Added .SYMBOL_TYPE for use in joining to REF_PLANT_DICTIONARY.
- 12:41 AM Revision 8126: Added inputs/FIA/REF_UNIT/postprocess.sql
- 12:36 AM Revision 8125: Added inputs/FIA/REF_RESEARCH_STATION/postprocess.sql
- 12:17 AM Revision 8124: Added inputs/FIA/COUNTY/postprocess.sql
- 12:02 AM Revision 8123: Added inputs/FIA/REF_PLANT_DICTIONARY/postprocess.sql
03/20/2013
- 05:16 PM Revision 8122: inputs/FIA/COND/postprocess.sql: Matched COND.HABTYPCD1, COND.HABTYPCD1_PUB_CD to REF_HABTYP_DESCRIPTION
- 05:07 PM Revision 8121: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to run postprocess.sql before exporting the header, because *it* can change the column names
- 05:02 PM Revision 8120: inputs/input.Makefile: Staging tables installation: $(exportHeader): export the header before running $(cleanup), because the header is not affected by the data cleanup operations and thus can be generated right away, to allow mapping while the cleanup operations run
- 04:30 PM Revision 8119: inputs/FIA/REF_HABTYP_DESCRIPTION/postprocess.sql: Prepare columns for joining with COND
- 03:23 PM Revision 8118: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to use psql_script_vegbien instead of the psql_verbose_vegbien used by $(psqlAsBien), to avoid echoing commands as part of the exported header
- 03:10 PM Revision 8117: Added planning/workflow/(de)normalized_import.mappings.png
- 03:04 PM Revision 8116: Added planning/workflow/denormalized_import.png, normalized_import.png
- 10:37 AM Revision 8115: web/main/IH/: Added lowercase alias
- 10:32 AM Revision 8114: Added web/main/IH/
- 10:12 AM Revision 8113: inputs/input.Makefile: Staging tables installation: Added postprocess target, which runs all the postprocess.sql files
- 09:34 AM Revision 8112: inputs/FIA/REF_SPECIES/postprocess.sql: Cast ID column to integer
- 08:52 AM Revision 8111: inputs/FIA/*/postprocess.sql: Cluster tables by their *.unique index for faster joins
- 08:51 AM Revision 8110: inputs/FIA/*/postprocess.sql: Cast ID columns to integer using new functions.set_col_types()
- 08:49 AM Revision 8109: bin/psql_verbose_vegbien: Run with client_min_messages = NOTICE to display notices for debugging. This is supposed to be the default, but apparently isn't.
- 08:47 AM Revision 8108: inputs/input.Makefile: BIEN commands: $(psqlAsBien): Use psql_verbose_vegbien instead of psql_script_vegbien so that timings and notices are displayed, which is useful for profiling and debugging
- 08:32 AM Revision 8107: schemas/functions.sql: Added col_cast and set_col_types()
- 07:45 AM Revision 8106: schemas/functions.sql: Added col_ref, col_type()
- 06:51 AM Revision 8105: schemas/functions.sql: Added cluster_once()
- 06:36 AM Revision 8104: schemas/functions.sql: Added cluster_index()
- 05:55 AM Revision 8103: schemas/functions.sql: create_if_not_exists(): Also handle duplicate_column exceptions
- 05:54 AM Revision 8102: schemas/functions.sql: Added rename_if_exists()
- 05:48 AM Revision 8101: inputs/FIA/COND/postprocess.sql: Renamed oldgrowth to COND.oldgrowth so it wouldn't be renamed by to_global_col_names()
- 04:28 AM Revision 8100: inputs/FIA/COND/postprocess.sql: Added oldgrowth column as part of the postprocessing instead of as part of the view that left joins the core tables together. This avoids needing to regenerate the oldgrowth field whenever the view is queried or materialized.
- 04:01 AM Revision 8099: inputs/FIA/TREE/postprocess.sql: Added index on columns that join to parent tables
- 03:00 AM Revision 8098: inputs/FIA/*/postprocess.sql: Removed table prefix from globally-unique columns that should be joined on
- 02:25 AM Revision 8097: schemas/functions.sql: Marked STRICT functions as such
- 02:22 AM Revision 8096: schemas/functions.sql: col_global_names(): Treat any column name that contains . as already being globally unique, and don't prepend the table name. This allows renaming the table columns after running col_global_names(), without causing the table name to be re-prepended the next time col_global_names() is run.
- 02:09 AM Revision 8095: schemas/functions.sql: Added contains()
- 02:07 AM Revision 8094: schemas/functions.sql: Added create_if_not_exists()
- 01:28 AM Revision 8093: inputs/FIA/*/postprocess.sql: Use functions.to_global_col_names() to ensure that all column names are globally unique. This makes it easy to join the tables together without worrying about column name collisions.
- 01:15 AM Revision 8092: inputs/FIA/*/postprocess.sql: Use new functions.create_if_not_exists() to allow re-running postprocess.sql idempotently
03/19/2013
- 11:48 PM Revision 8091: inputs/input.Makefile: Staging tables installation: %/install: Use new %.sql/run to run postprocess.sql
- 11:47 PM Revision 8090: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 11:47 PM Revision 8089: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 10:43 PM Revision 8088: schemas/functions.sql: Added to_global_col_names()
- 10:22 PM Revision 8087: schemas/functions.sql: col_global_names(): Use new functions.ensure_prefix() to only add the table name prefix if it doesn't already exist. This makes the function idempotent.
- 10:19 PM Revision 8086: schemas/functions.sql: Added ensure_prefix()
- 10:17 PM Revision 8085: schemas/functions.sql: Added has_prefix()
- 10:09 PM Revision 8084: schemas/functions.sql: Added col_global_names()
- 09:59 PM Revision 8083: schemas/functions.sql: Added name(regtype)
- 09:43 PM Revision 8082: schemas/functions.sql: Added col_names()
- 09:27 PM Revision 8081: root Makefile: Installation: Fixed bug where need to run schemas/public/install separately because schemas/install installs only the util schemas
- 09:26 PM Revision 8080: root Makefile: Installation: install util schemas (temp functions py_functions) before inputs, so that inputs can use util functions in their postprocess.sql or create.sql scripts. (However, they must not use util functions in views or index functions, because these would be cascadingly deleted whenever the util schemas are reinstalled before an import.)
- 08:07 PM Revision 8079: README.TXT: Single datasource import: Added by_col=1 to all commands
- 02:28 AM Revision 8078: mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field
- 02:05 AM Revision 8077: lib/sql_io.py: mk_errors_table(): Create a unique index on the MD5 of the value and error instead of on the values directly, because some strings are too long to index (e.g. row 2537268 of MO.Specimen causes an error "index row size 3032 exceeds maximum 2712 for index [...] Values larger than 1/3 of a buffer page cannot be indexed")
- 12:49 AM Revision 8076: inputs/import.stats.xls: Updated import times
03/16/2013
- 02:16 PM Revision 8075: bin/map: No mappings warning: Added explanation that this could also be due to no column name matches, and hint to check if you are importing the correct input table
- 01:45 PM Revision 8074: inputs/MO/: Renamed Specimen.2/ -> now available Specimen/
- 01:42 PM Revision 8073: inputs/MO/: Removed old import in Specimen/
- 01:33 PM Revision 8072: Refreshed MO
- 12:44 PM Revision 8071: csvs.py: TsvReader.next(): Fixed bug where empty line needs to be separately returned as [], because csv.reader would interpret it as EOF since the line ending has already been removed
- 12:25 PM Revision 8070: csvs.py: sniff(): TSVs: Turn off quoting because TSVs use \-escapes instead of quotes to escape delimeters, newlines, etc.
- 11:49 AM Revision 8069: csvs.py: InputRewriter.readline(): Surround function in a try block that prints all exceptions, so that debugging information is available if an error occurs when this stream is used as input for psycopg's copy_expert() (COPY FROM)
- 06:56 AM Revision 8068: Populated inputs/MO/import_order.txt
- 06:46 AM Revision 8067: Refreshed SALVIAS
- 06:33 AM Revision 8066: Added web/main/CTFS/
- 06:21 AM Revision 8065: inputs/SALVIAS/: Regenerated salvias_*.schema.sql from the MySQL version, to take advantage of my2pg improvements. The placeholder *_index columns which take the place of MySQL's inline index definitions have now been replaced by no-op CHECK constraints, so that there are no longer lots of dummy *_index columns in the map spreadsheets.
- 05:52 AM Revision 8064: Added web/main/Redmine/ alias to VegBIEN/Redmine/
- 05:51 AM Revision 8063: Added web/main/VegBIEN/Redmine/
- 05:48 AM Revision 8062: web/main/VegBIEN/.htaccess: Forward to new db/ subdir
- 05:47 AM Revision 8061: Added web/main/VegBIEN/db/
- 05:45 AM Revision 8060: web/main/**/.htaccess: Removed RewriteCond -l tests because one of the -d or -f tests will always also pass, making the -l test unnecessary
- 05:38 AM Revision 8059: web/main.conf: Added tolower RewriteMap
- 05:19 AM Revision 8058: web/main/.htaccess: use separate lowercase version when available: Also support input strings in mixed case which is not the default capitalization, in addition to all-lowercase strings
- 05:18 AM Revision 8057: web/main/.htaccess: use separate lowercase version when available: Generate the new dirname with a separate RewriteCond so its value can be used both in the -d test and in the replacement string, rather than separately for each
- 05:03 AM Revision 8056: web/main/.htaccess: translate dotpaths: Allow an unescaped . at the beginning of a filename, because this will never be a . separator. This adds support for hidden files in dir paths, which now won't be interpreted as dotpaths. However, regular files with extensions still need to have the filename escaped because it will otherwise be interpreted as a dotpath.
- 04:53 AM Revision 8055: web/main/.htaccess: Set Options +FollowSymLinks. It should be on by default ("All options except for MultiViews. This is the default setting." <http://httpd.apache.org/docs/2.2/mod/core.html#options>), but this makes sure it will always be enabled.
- 03:32 AM Revision 8054: web/main/.htaccess: Name the lowercased versions of dirs with a leading . (to make them hidden) instead of a trailing _ , to avoid having each dir listed twice in a row in the dir index
- 03:21 AM Revision 8053: Added web/main/TNRS/
- 03:15 AM Revision 8052: Added web/main/VegBank/
- 02:57 AM Revision 8051: Added web/main/BIEN2/
- 02:56 AM Revision 8050: web/main/index.php: Replaced - with . in namespaces to conform to new dotpath naming convention, which allows nesting of namespaces
- 02:35 AM Revision 8049: web/main/SALVIAS/.htaccess: Forward to new dd/ subdir
- 02:34 AM Revision 8048: Added web/main/SALVIAS/dd/
- 02:28 AM Revision 8047: web/main/DwC/.htaccess: Forward to new terms/ subdir
- 02:27 AM Revision 8046: Added web/main/DwC/terms/
- 01:51 AM Revision 8045: web/main/**/.htaccess: don't redirect subdir paths: Fixed bug where can only match non-empty string, because otherwise the rule would match this directory, which should still have its redirects processed
- 01:43 AM Revision 8044: web/main/index.php: Added back smaller spacing between the table columns
- 01:40 AM Revision 8043: web/main/main.css: blockquote: Removed right margin so there isn't a big space between the table columns in index.php, which results from nesting right-padded blockquotes inside one another
- 01:39 AM Revision 8042: web/main/index.php: Changed Brad-Boyle to just Brad because people's names only have to be unique within VegPath
- 01:32 AM Revision 8041: web/main/.htaccess: Added fallback redirect to VegCore for paths without a namespace. This can be used to link to specific VegCore terms without needing to include the VegCore namespace.
- 01:13 AM Revision 8040: Added web/main/VegBIEN/
- 01:13 AM Revision 8039: Added web/main/servers/vegbiendev/
- 01:13 AM Revision 8038: Added web/main/.phpPgAdmin/
- 01:12 AM Revision 8037: web/main/.phpMyAdmin/.htaccess: Set [redirect] flag in case the dest server is on the same machine as VegPath itself
- 12:25 AM Revision 8036: web/main/*/ lowercase versions: Renamed with _ suffix to avoid svn conflicts on case-insensitive filesystems such as Mac HFS+
- 12:24 AM Revision 8035: web/main/.htaccess: Support lowercase versions of mixed-case dirnames without breaking case-insensitive filesystems such as Mac HFS+
- 12:10 AM Revision 8034: web/main/: Added lowercase symlinks for mixed-case dirs to work with subdomain translation, which uses subdomains lowercased by DNS
- 12:08 AM Revision 8033: web/main/index.php: Use absolute URLs for dependencies to work with subdomain translation, which adds components to the URL path
03/15/2013
- 11:30 PM Revision 8032: web/main/**/.htaccess: Use RewriteRule instead of RedirectMatch to handle incremental redirects internally instead of issuing a (much slower) redirect to the web browser each time. This also handles edge cases better, as [last] RewriteRules can be used to control when to forward control to a subdir, and doesn't require prepending the path to the dir the .htaccess file is in. Note that this requires all gateway dirs (dirs with subdirs) to contain special RewriteRules to avoid redirecting subdir paths and handle DirectoryIndex; see web/main/DwC/.htaccess. Also note that the regexp of a catch-all RewriteRule must exactly follow the template for internal or external redirects; see web/main/SALVIAS/db/.htaccess for internal redirects and web/main/DwC/history/.htaccess for external redirects.
- 07:31 PM Revision 8031: web/main/.htaccess: translate dotpaths: Allow the part before the [] escape to contain [], to support labels that end in [] (like PHP array vars in the query string) labels with a simple array-subscript syntax (a[b]). This also shortens the regexp and makes it more readable without the \[\] in [^.\[\]/] . Note that this also allows invalid combinations of [] exprs (e.g. more than one per level or unbalanced []), which will still be translated but will probably not have the desired result.
- 07:16 PM Revision 8030: web/main/.htaccess: translate dotpaths: Inline the [] escape regexp into the main regexp, because it is now approximately the same length as the []-matching portion of the main regexp and this greatly simplifies the code by removing the extra RewriteCond. Note that the translation rule is now a plain regexp (run repeatedly until no match), which can be used in *any* programming language that supports Perl-compatible regular expressions, not just mod_rewrite.
- 07:06 PM Revision 8029: web/main/.htaccess: translate dotpaths: discardpath explanation: Clarified that the infinite loop resulted from reappending PATH_INFO (the Apache-matched filename)
- 07:02 PM Revision 8028: web/main/.htaccess: translate dotpaths: Require any [] escape to have the ] at the end of the level, to simplify the [] regexp
- 06:54 PM Revision 8027: web/main/.htaccess: translate dotpaths: Use a lookahead assertion to ensure that at least one character is matched as the head of the dotpath. This ensures that (.*/)? + the rest of the regexp does not match a path with a trailing /, which is a sealed /-path and not subject to dotpath translation.
- 06:46 PM Revision 8026: web/main/.htaccess: translate dotpaths: Only support one [] escape per dot-level to (greatly) shorten the [] regexp. This does not pose a problem for encoding . because the entire level can simply be enclosed in [].
- 06:17 PM Revision 8025: web/main/.htaccess: Uncommented ErrorDocument
- 06:15 PM Revision 8024: web/main/.htaccess: translate dotpaths: Removed separate sealing of the /-path, which is now performed by the main RewriteRule because it appends a / even if there is no . suffix. This does not cause an infinite loop because a character is always added (/), which prevents the previously-matched head (after the last / but before any .) from being matched again.
- 06:05 PM Revision 8023: web/main/.htaccess: translate dotpaths: Fixed bug where it's actually the portion *before* the . (but after the last /) that should be subject to []-unescaping, rather than the portion *after* the . . Fixed bug where [] escapes were not being unescaped because the wildcard .* group matched the whole head portion instead of allowing the []-captures to match.
- 03:17 PM Revision 8022: web/main/.htaccess: translate subdomain to path: Don't use expr RewriteConds because they are not supported by Apache 2.2. Instead issue an external redirect with the subdomain part of the hostname removed, for the purpose of changing HTTP_HOST so that the replacement is not performed again if the mod_rewrite rules are run more than once.
- 02:46 PM Revision 8021: web/main/**/.htaccess: Use RewriteEngine, inheriting the web/main/.htaccess rules, in order to translate dotpaths that follow /-paths to existing dirs
- 02:45 PM Revision 8020: web/main/DwC/.htaccess: Moved DwC.history redirect to web/main/DwC/history/.htaccess
- 02:40 PM Revision 8019: web/main/SALVIAS/db/.htaccess: Removed trailing / because for DB redirects, this is apparently necessary
- 02:30 PM Revision 8018: web/main/.phpMyAdmin/.htaccess: Prepend http:// to the dest URL stem, instead of requiring the dest URL to provide the protocol, because the two // are replaced with one / by Apache when mod_rewrite is on, creating an invalid URL
- 01:55 PM Revision 8017: web/main/.htaccess: translate dotpaths: Fixed bug where can't remove all [] escapes, because this would also remove [] from protected parts of the URL (i.e. before the last /). Instead just remove up to two [] escapes per dot-level, e.g. a.[b.c]=[d.e] -> a/b.c=d.e . Note that this removes the restriction against URLs containing "[]", because these [] will be left as-is as long as they are followed at some point by a / .
- 01:41 PM Revision 8016: Removed web/.htaccess because it is not part of any site (web/main/.htaccess is the top-level .htaccess file for the default site)
- 01:17 PM Revision 8015: web/main/.htaccess: translate dotpaths: replace all unescaped . with / : Allow empty components on either side of the . , to ensure that *all* unescaped dots are consistently replaced with / . Note that hidden directories that start with . will have the . preserved, because they have (or get) a trailing slash, which prevents dotpath from operating on them.
- 01:13 PM Revision 8014: web/main/.htaccess: translate dotpaths: remove all [] escapes: Also remove empty [] to allow permalinking clients to easily generate escaped strings by URL-encoding the value and enclosing it in [], which will now also work even if the value is empty. This change will break any destination URLs that use "[]", such as PHP scripts that use this to indicate an array variable in the query string. In these URLs, the [] must now be %-encoded.
- 01:00 PM Revision 8013: web/main/.htaccess: RewriteRules with [redirect] and http:// : Removed [redirect] because it is implied when the replacement is an absolute URL
- 12:57 PM Revision 8012: web/main/.htaccess: RewriteRules with [redirect]: Added last because apparently redirect will not immediately cause the redirect, and will instead continue rewriting the URL ("You will almost always want to use [R] in conjunction with [L] (that is, use [R,L]) because on its own, the [R] flag prepends http://thishost[:thisport] to the URI, but then passes this on to the next rule in the ruleset, which can often result in 'Invalid URI in request' warnings." <http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_r>)
- 12:34 PM Revision 8011: web/main/.htaccess: RewriteRules: Added [discardpath,noescape] flags to all rules, because these are needed for correct interpretation of the replacement and should be the default setting. For some rules they may be optional, but it is better to include them on all rules to ensure there is never an unexpected effect from omitting them.
- 12:29 PM Revision 8010: web/main/.htaccess: Added "RewriteOptions inherit" because this[1] should be the default setting, to ensure that global filter rules also occur in subdirs. Note that the presence of this directive in *this* .htaccess file does not solve this problem, but putting it there will remind writers of subdir .htaccess files that they need to include that directive.
- [1] The value should really be InheritBefore, but this is not supported in Apache 2.2, which we have (http://httpd.ap...
- 12:02 PM Revision 8009: web/main/SALVIAS/.htaccess: RedirectMatch for below top-level: Use /(.*?)/?$ suffix for consistency with other sources' RedirectMatch patterns
- 12:00 PM Revision 8008: web/main/TCS,VegX/.htaccess: Fixed bug where need to match dirs by themselves in addition to when followed by a filename
- 11:58 AM Revision 8007: web/main/DwC/.htaccess: Fixed bug where need to match dirs by themselves (/DwC/, /DwC/history/) in addition to when followed by a filename
- 11:44 AM Revision 8006: web/main/.htaccess: mod_rewrite: Removed unneeded RewriteBase, because the default setting for RewriteBase is actually the dir of the .htaccess file itself, rather than always / . ("RewriteRule can be used in per-directory config files (.htaccess). There it will act locally, i.e., the local directory prefix is stripped at this stage of processing and your rewriting rules act only on the remainder. At the end it is automatically added back to the path." <http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteBase>)
- 11:30 AM Revision 8005: web/main/SALVIAS/*/.htaccess: Added trailing / to Redirect paths, which works now that a trailing / is automatically added to all paths by dotpath translation
- 11:28 AM Revision 8004: web/main/SALVIAS/: Moved db/ and users/ actions to their corresponding subdirs. The rules will still be run in the appropriate order, because if a .htaccess exists in a subdir, it will be used in preference to the .htaccess in the parent dir.
- 10:26 AM Revision 8003: my2pg: Removed SETs for standard_conforming_strings, escape_string_warning because these are also prepended by my2pg.data, which this script's output is piped to
- 10:00 AM Revision 8002: my2pg.data: \0 removal: Fixed bug where need to handle all backslashes before the 0, because they might escape just each other (an even number of \) rather than also encoding the \0 (an odd number of \)
- 08:20 AM Revision 8001: web/main/.htaccess: translate dotpaths to /-paths: Added examples
- 08:04 AM Revision 8000: web/main/.htaccess: translate dotpaths to /-paths: Translate *all* .-separated components in the path to / instead of just those that exist in the filesystem. This allows per-source rules to match just a / for the path-element separator instead of sometimes / and other times . (depending on whether that portion of the dotpath had been translated). Support [] escapes that preserve any . they surround, e.g. a.[b.c] -> a/b.c . (Supporting []-escapes requires that the /-path be "sealed" by appending a / , to prevent the . -> / translation rule from reinterpreting a newly-unescaped [] sequence as a dotpath. This also requires that the translation rule ignore anything before the last /, because it could have been []-unescaped by a previous round of mod_rewrite, e.g. in another dir. All the per-source rules need to be retrofitted to support the new trailing / .) Note also the discardpath flag (http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_dpi) on all RewriteRules that perform a replacement on the input string (i.e. that have a pattern other than ^.*$). This flag is needed to avoid infinite loops, because otherwise, a critical bug in mod_rewrite causes it to *reappend* the filename portion of the input string (PATH_INFO) to the result, causing it to be present twice, e.g. a/b -> a/b/b (https://issues.apache.org/bugzilla/show_bug.cgi?id=38642).
- 03:09 AM Revision 7999: web/main/nimoy/: Moved to web/main/servers/ so it can be grouped with other resources that are servers
Also available in: Atom