Activity
From 02/18/2013 to 03/19/2013
03/19/2013
- 11:48 PM Revision 8091: inputs/input.Makefile: Staging tables installation: %/install: Use new %.sql/run to run postprocess.sql
- 11:47 PM Revision 8090: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 11:47 PM Revision 8089: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 10:43 PM Revision 8088: schemas/functions.sql: Added to_global_col_names()
- 10:22 PM Revision 8087: schemas/functions.sql: col_global_names(): Use new functions.ensure_prefix() to only add the table name prefix if it doesn't already exist. This makes the function idempotent.
- 10:19 PM Revision 8086: schemas/functions.sql: Added ensure_prefix()
- 10:17 PM Revision 8085: schemas/functions.sql: Added has_prefix()
- 10:09 PM Revision 8084: schemas/functions.sql: Added col_global_names()
- 09:59 PM Revision 8083: schemas/functions.sql: Added name(regtype)
- 09:43 PM Revision 8082: schemas/functions.sql: Added col_names()
- 09:27 PM Revision 8081: root Makefile: Installation: Fixed bug where need to run schemas/public/install separately because schemas/install installs only the util schemas
- 09:26 PM Revision 8080: root Makefile: Installation: install util schemas (temp functions py_functions) before inputs, so that inputs can use util functions in their postprocess.sql or create.sql scripts. (However, they must not use util functions in views or index functions, because these would be cascadingly deleted whenever the util schemas are reinstalled before an import.)
- 08:07 PM Revision 8079: README.TXT: Single datasource import: Added by_col=1 to all commands
- 02:28 AM Revision 8078: mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field
- 02:05 AM Revision 8077: lib/sql_io.py: mk_errors_table(): Create a unique index on the MD5 of the value and error instead of on the values directly, because some strings are too long to index (e.g. row 2537268 of MO.Specimen causes an error "index row size 3032 exceeds maximum 2712 for index [...] Values larger than 1/3 of a buffer page cannot be indexed")
- 12:49 AM Revision 8076: inputs/import.stats.xls: Updated import times
03/16/2013
- 02:16 PM Revision 8075: bin/map: No mappings warning: Added explanation that this could also be due to no column name matches, and hint to check if you are importing the correct input table
- 01:45 PM Revision 8074: inputs/MO/: Renamed Specimen.2/ -> now available Specimen/
- 01:42 PM Revision 8073: inputs/MO/: Removed old import in Specimen/
- 01:33 PM Revision 8072: Refreshed MO
- 12:44 PM Revision 8071: csvs.py: TsvReader.next(): Fixed bug where empty line needs to be separately returned as [], because csv.reader would interpret it as EOF since the line ending has already been removed
- 12:25 PM Revision 8070: csvs.py: sniff(): TSVs: Turn off quoting because TSVs use \-escapes instead of quotes to escape delimeters, newlines, etc.
- 11:49 AM Revision 8069: csvs.py: InputRewriter.readline(): Surround function in a try block that prints all exceptions, so that debugging information is available if an error occurs when this stream is used as input for psycopg's copy_expert() (COPY FROM)
- 06:56 AM Revision 8068: Populated inputs/MO/import_order.txt
- 06:46 AM Revision 8067: Refreshed SALVIAS
- 06:33 AM Revision 8066: Added web/main/CTFS/
- 06:21 AM Revision 8065: inputs/SALVIAS/: Regenerated salvias_*.schema.sql from the MySQL version, to take advantage of my2pg improvements. The placeholder *_index columns which take the place of MySQL's inline index definitions have now been replaced by no-op CHECK constraints, so that there are no longer lots of dummy *_index columns in the map spreadsheets.
- 05:52 AM Revision 8064: Added web/main/Redmine/ alias to VegBIEN/Redmine/
- 05:51 AM Revision 8063: Added web/main/VegBIEN/Redmine/
- 05:48 AM Revision 8062: web/main/VegBIEN/.htaccess: Forward to new db/ subdir
- 05:47 AM Revision 8061: Added web/main/VegBIEN/db/
- 05:45 AM Revision 8060: web/main/**/.htaccess: Removed RewriteCond -l tests because one of the -d or -f tests will always also pass, making the -l test unnecessary
- 05:38 AM Revision 8059: web/main.conf: Added tolower RewriteMap
- 05:19 AM Revision 8058: web/main/.htaccess: use separate lowercase version when available: Also support input strings in mixed case which is not the default capitalization, in addition to all-lowercase strings
- 05:18 AM Revision 8057: web/main/.htaccess: use separate lowercase version when available: Generate the new dirname with a separate RewriteCond so its value can be used both in the -d test and in the replacement string, rather than separately for each
- 05:03 AM Revision 8056: web/main/.htaccess: translate dotpaths: Allow an unescaped . at the beginning of a filename, because this will never be a . separator. This adds support for hidden files in dir paths, which now won't be interpreted as dotpaths. However, regular files with extensions still need to have the filename escaped because it will otherwise be interpreted as a dotpath.
- 04:53 AM Revision 8055: web/main/.htaccess: Set Options +FollowSymLinks. It should be on by default ("All options except for MultiViews. This is the default setting." <http://httpd.apache.org/docs/2.2/mod/core.html#options>), but this makes sure it will always be enabled.
- 03:32 AM Revision 8054: web/main/.htaccess: Name the lowercased versions of dirs with a leading . (to make them hidden) instead of a trailing _ , to avoid having each dir listed twice in a row in the dir index
- 03:21 AM Revision 8053: Added web/main/TNRS/
- 03:15 AM Revision 8052: Added web/main/VegBank/
- 02:57 AM Revision 8051: Added web/main/BIEN2/
- 02:56 AM Revision 8050: web/main/index.php: Replaced - with . in namespaces to conform to new dotpath naming convention, which allows nesting of namespaces
- 02:35 AM Revision 8049: web/main/SALVIAS/.htaccess: Forward to new dd/ subdir
- 02:34 AM Revision 8048: Added web/main/SALVIAS/dd/
- 02:28 AM Revision 8047: web/main/DwC/.htaccess: Forward to new terms/ subdir
- 02:27 AM Revision 8046: Added web/main/DwC/terms/
- 01:51 AM Revision 8045: web/main/**/.htaccess: don't redirect subdir paths: Fixed bug where can only match non-empty string, because otherwise the rule would match this directory, which should still have its redirects processed
- 01:43 AM Revision 8044: web/main/index.php: Added back smaller spacing between the table columns
- 01:40 AM Revision 8043: web/main/main.css: blockquote: Removed right margin so there isn't a big space between the table columns in index.php, which results from nesting right-padded blockquotes inside one another
- 01:39 AM Revision 8042: web/main/index.php: Changed Brad-Boyle to just Brad because people's names only have to be unique within VegPath
- 01:32 AM Revision 8041: web/main/.htaccess: Added fallback redirect to VegCore for paths without a namespace. This can be used to link to specific VegCore terms without needing to include the VegCore namespace.
- 01:13 AM Revision 8040: Added web/main/VegBIEN/
- 01:13 AM Revision 8039: Added web/main/servers/vegbiendev/
- 01:13 AM Revision 8038: Added web/main/.phpPgAdmin/
- 01:12 AM Revision 8037: web/main/.phpMyAdmin/.htaccess: Set [redirect] flag in case the dest server is on the same machine as VegPath itself
- 12:25 AM Revision 8036: web/main/*/ lowercase versions: Renamed with _ suffix to avoid svn conflicts on case-insensitive filesystems such as Mac HFS+
- 12:24 AM Revision 8035: web/main/.htaccess: Support lowercase versions of mixed-case dirnames without breaking case-insensitive filesystems such as Mac HFS+
- 12:10 AM Revision 8034: web/main/: Added lowercase symlinks for mixed-case dirs to work with subdomain translation, which uses subdomains lowercased by DNS
- 12:08 AM Revision 8033: web/main/index.php: Use absolute URLs for dependencies to work with subdomain translation, which adds components to the URL path
03/15/2013
- 11:30 PM Revision 8032: web/main/**/.htaccess: Use RewriteRule instead of RedirectMatch to handle incremental redirects internally instead of issuing a (much slower) redirect to the web browser each time. This also handles edge cases better, as [last] RewriteRules can be used to control when to forward control to a subdir, and doesn't require prepending the path to the dir the .htaccess file is in. Note that this requires all gateway dirs (dirs with subdirs) to contain special RewriteRules to avoid redirecting subdir paths and handle DirectoryIndex; see web/main/DwC/.htaccess. Also note that the regexp of a catch-all RewriteRule must exactly follow the template for internal or external redirects; see web/main/SALVIAS/db/.htaccess for internal redirects and web/main/DwC/history/.htaccess for external redirects.
- 07:31 PM Revision 8031: web/main/.htaccess: translate dotpaths: Allow the part before the [] escape to contain [], to support labels that end in [] (like PHP array vars in the query string) labels with a simple array-subscript syntax (a[b]). This also shortens the regexp and makes it more readable without the \[\] in [^.\[\]/] . Note that this also allows invalid combinations of [] exprs (e.g. more than one per level or unbalanced []), which will still be translated but will probably not have the desired result.
- 07:16 PM Revision 8030: web/main/.htaccess: translate dotpaths: Inline the [] escape regexp into the main regexp, because it is now approximately the same length as the []-matching portion of the main regexp and this greatly simplifies the code by removing the extra RewriteCond. Note that the translation rule is now a plain regexp (run repeatedly until no match), which can be used in *any* programming language that supports Perl-compatible regular expressions, not just mod_rewrite.
- 07:06 PM Revision 8029: web/main/.htaccess: translate dotpaths: discardpath explanation: Clarified that the infinite loop resulted from reappending PATH_INFO (the Apache-matched filename)
- 07:02 PM Revision 8028: web/main/.htaccess: translate dotpaths: Require any [] escape to have the ] at the end of the level, to simplify the [] regexp
- 06:54 PM Revision 8027: web/main/.htaccess: translate dotpaths: Use a lookahead assertion to ensure that at least one character is matched as the head of the dotpath. This ensures that (.*/)? + the rest of the regexp does not match a path with a trailing /, which is a sealed /-path and not subject to dotpath translation.
- 06:46 PM Revision 8026: web/main/.htaccess: translate dotpaths: Only support one [] escape per dot-level to (greatly) shorten the [] regexp. This does not pose a problem for encoding . because the entire level can simply be enclosed in [].
- 06:17 PM Revision 8025: web/main/.htaccess: Uncommented ErrorDocument
- 06:15 PM Revision 8024: web/main/.htaccess: translate dotpaths: Removed separate sealing of the /-path, which is now performed by the main RewriteRule because it appends a / even if there is no . suffix. This does not cause an infinite loop because a character is always added (/), which prevents the previously-matched head (after the last / but before any .) from being matched again.
- 06:05 PM Revision 8023: web/main/.htaccess: translate dotpaths: Fixed bug where it's actually the portion *before* the . (but after the last /) that should be subject to []-unescaping, rather than the portion *after* the . . Fixed bug where [] escapes were not being unescaped because the wildcard .* group matched the whole head portion instead of allowing the []-captures to match.
- 03:17 PM Revision 8022: web/main/.htaccess: translate subdomain to path: Don't use expr RewriteConds because they are not supported by Apache 2.2. Instead issue an external redirect with the subdomain part of the hostname removed, for the purpose of changing HTTP_HOST so that the replacement is not performed again if the mod_rewrite rules are run more than once.
- 02:46 PM Revision 8021: web/main/**/.htaccess: Use RewriteEngine, inheriting the web/main/.htaccess rules, in order to translate dotpaths that follow /-paths to existing dirs
- 02:45 PM Revision 8020: web/main/DwC/.htaccess: Moved DwC.history redirect to web/main/DwC/history/.htaccess
- 02:40 PM Revision 8019: web/main/SALVIAS/db/.htaccess: Removed trailing / because for DB redirects, this is apparently necessary
- 02:30 PM Revision 8018: web/main/.phpMyAdmin/.htaccess: Prepend http:// to the dest URL stem, instead of requiring the dest URL to provide the protocol, because the two // are replaced with one / by Apache when mod_rewrite is on, creating an invalid URL
- 01:55 PM Revision 8017: web/main/.htaccess: translate dotpaths: Fixed bug where can't remove all [] escapes, because this would also remove [] from protected parts of the URL (i.e. before the last /). Instead just remove up to two [] escapes per dot-level, e.g. a.[b.c]=[d.e] -> a/b.c=d.e . Note that this removes the restriction against URLs containing "[]", because these [] will be left as-is as long as they are followed at some point by a / .
- 01:41 PM Revision 8016: Removed web/.htaccess because it is not part of any site (web/main/.htaccess is the top-level .htaccess file for the default site)
- 01:17 PM Revision 8015: web/main/.htaccess: translate dotpaths: replace all unescaped . with / : Allow empty components on either side of the . , to ensure that *all* unescaped dots are consistently replaced with / . Note that hidden directories that start with . will have the . preserved, because they have (or get) a trailing slash, which prevents dotpath from operating on them.
- 01:13 PM Revision 8014: web/main/.htaccess: translate dotpaths: remove all [] escapes: Also remove empty [] to allow permalinking clients to easily generate escaped strings by URL-encoding the value and enclosing it in [], which will now also work even if the value is empty. This change will break any destination URLs that use "[]", such as PHP scripts that use this to indicate an array variable in the query string. In these URLs, the [] must now be %-encoded.
- 01:00 PM Revision 8013: web/main/.htaccess: RewriteRules with [redirect] and http:// : Removed [redirect] because it is implied when the replacement is an absolute URL
- 12:57 PM Revision 8012: web/main/.htaccess: RewriteRules with [redirect]: Added last because apparently redirect will not immediately cause the redirect, and will instead continue rewriting the URL ("You will almost always want to use [R] in conjunction with [L] (that is, use [R,L]) because on its own, the [R] flag prepends http://thishost[:thisport] to the URI, but then passes this on to the next rule in the ruleset, which can often result in 'Invalid URI in request' warnings." <http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_r>)
- 12:34 PM Revision 8011: web/main/.htaccess: RewriteRules: Added [discardpath,noescape] flags to all rules, because these are needed for correct interpretation of the replacement and should be the default setting. For some rules they may be optional, but it is better to include them on all rules to ensure there is never an unexpected effect from omitting them.
- 12:29 PM Revision 8010: web/main/.htaccess: Added "RewriteOptions inherit" because this[1] should be the default setting, to ensure that global filter rules also occur in subdirs. Note that the presence of this directive in *this* .htaccess file does not solve this problem, but putting it there will remind writers of subdir .htaccess files that they need to include that directive.
- [1] The value should really be InheritBefore, but this is not supported in Apache 2.2, which we have (http://httpd.ap...
- 12:02 PM Revision 8009: web/main/SALVIAS/.htaccess: RedirectMatch for below top-level: Use /(.*?)/?$ suffix for consistency with other sources' RedirectMatch patterns
- 12:00 PM Revision 8008: web/main/TCS,VegX/.htaccess: Fixed bug where need to match dirs by themselves in addition to when followed by a filename
- 11:58 AM Revision 8007: web/main/DwC/.htaccess: Fixed bug where need to match dirs by themselves (/DwC/, /DwC/history/) in addition to when followed by a filename
- 11:44 AM Revision 8006: web/main/.htaccess: mod_rewrite: Removed unneeded RewriteBase, because the default setting for RewriteBase is actually the dir of the .htaccess file itself, rather than always / . ("RewriteRule can be used in per-directory config files (.htaccess). There it will act locally, i.e., the local directory prefix is stripped at this stage of processing and your rewriting rules act only on the remainder. At the end it is automatically added back to the path." <http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteBase>)
- 11:30 AM Revision 8005: web/main/SALVIAS/*/.htaccess: Added trailing / to Redirect paths, which works now that a trailing / is automatically added to all paths by dotpath translation
- 11:28 AM Revision 8004: web/main/SALVIAS/: Moved db/ and users/ actions to their corresponding subdirs. The rules will still be run in the appropriate order, because if a .htaccess exists in a subdir, it will be used in preference to the .htaccess in the parent dir.
- 10:26 AM Revision 8003: my2pg: Removed SETs for standard_conforming_strings, escape_string_warning because these are also prepended by my2pg.data, which this script's output is piped to
- 10:00 AM Revision 8002: my2pg.data: \0 removal: Fixed bug where need to handle all backslashes before the 0, because they might escape just each other (an even number of \) rather than also encoding the \0 (an odd number of \)
- 08:20 AM Revision 8001: web/main/.htaccess: translate dotpaths to /-paths: Added examples
- 08:04 AM Revision 8000: web/main/.htaccess: translate dotpaths to /-paths: Translate *all* .-separated components in the path to / instead of just those that exist in the filesystem. This allows per-source rules to match just a / for the path-element separator instead of sometimes / and other times . (depending on whether that portion of the dotpath had been translated). Support [] escapes that preserve any . they surround, e.g. a.[b.c] -> a/b.c . (Supporting []-escapes requires that the /-path be "sealed" by appending a / , to prevent the . -> / translation rule from reinterpreting a newly-unescaped [] sequence as a dotpath. This also requires that the translation rule ignore anything before the last /, because it could have been []-unescaped by a previous round of mod_rewrite, e.g. in another dir. All the per-source rules need to be retrofitted to support the new trailing / .) Note also the discardpath flag (http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_dpi) on all RewriteRules that perform a replacement on the input string (i.e. that have a pattern other than ^.*$). This flag is needed to avoid infinite loops, because otherwise, a critical bug in mod_rewrite causes it to *reappend* the filename portion of the input string (PATH_INFO) to the result, causing it to be present twice, e.g. a/b -> a/b/b (https://issues.apache.org/bugzilla/show_bug.cgi?id=38642).
- 03:09 AM Revision 7999: web/main/nimoy/: Moved to web/main/servers/ so it can be grouped with other resources that are servers
03/14/2013
- 11:57 PM Revision 7998: web/main/index.php: $root: Fixed bug where need to use gethostbyaddr($_SERVER["SERVER_ADDR"]) instead of $_SERVER["SERVER_NAME"] because SERVER_NAME is actually set to whichever hostname the server is accessed through (e.g. path.vg or vegbiendev.nceas.ucsb.edu), rather than always being set to the reverse-DNS lookup of SERVER_ADDR, as is specified in the Apache docs ("If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address" <http://httpd.apache.org/docs/2.2/mod/core.html#servername>)
- 11:01 PM Revision 7997: web/main/.htaccess: Added back "don't rewrite existing paths" to ensure that rules after it will not rewrite any actual files (web page dependencies)
- 10:50 PM Revision 7996: web/main/nimoy/.htaccess: Removed no longer needed trailing /
- 10:49 PM Revision 7995: web/main/.phpMyAdmin/.htaccess: Add trailing / to provided URL in case the user didn't add it
- 10:46 PM Revision 7994: web/main/nimoy/.htaccess: Removed unnecessary index.php from end of phpMyAdmin URL
03/13/2013
- 11:09 PM Revision 7993: web/main/DwC,SALVIAS,TCS,VegX: Added .htaccess redirects and dot-path-matching empty dirs, which will replace the index.php redirects with a simpler syntax (http://httpd.apache.org/docs/2.2/mod/mod_alias.html#redirect)
- 09:15 PM Revision 7992: web/main/.htaccess: translate dot-paths to dirs: Documented that this only occurs when the dirs exist in the filesystem, and that each subdir Redirect directive needs a corresponding tree of (empty) dirs
- 08:49 PM Revision 7991: Removed no longer used web/main/_archive/index.htm. Use web/main/index.php instead, which issues the Location: header to support clients other than web browsers.
- 08:37 PM Revision 7990: web/vegpath/: Renamed to web/main/ because the content here is the main site hosted on vegbiendev. It will eventually contain internal links to VegBIEN resources in addition to external links (persistent URLs) to sources used in creating VegBIEN.
- 07:45 PM Revision 7989: web/vegpath/.htaccess: Removed "don't rewrite existing paths" RewriteRule because there are no longer any rules after it that require it
- 07:31 PM Revision 7988: web/vegpath/.htaccess: translate dot-paths to dirs: translate all levels of dot-paths present in the filesystem rather than just the first level, by using the [next] RewriteRule flag (http://httpd.apache.org/docs/current/rewrite/flags.html#flag_n). This avoids the need to translate further levels of dot-paths in each subdir's .htaccess.
- 07:07 PM Revision 7987: web/vegpath/.htaccess: translate subdomain to path: Simplified using new subdomain2path RewriteMap, which avoids the need for a separate RewriteRule+RewriteConds for each number of nested subdomains
- 07:06 PM Revision 7986: web/vegpath.conf: subdomain2path RewriteMap: Fixed bug where need `RewriteEngine on` in the VirtualHost config for a prg: RewriteMap to take effect ("External rewriting programs are not started if they're defined in a context that does not have RewriteEngine set to on." <http://httpd.apache.org/docs/current/rewrite/rewritemap.html#prg>)
- 06:00 PM Revision 7985: web/vegpath.conf: Added subdomain2path RewriteMap
- 06:00 PM Revision 7984: Added web/subdomain2path for use by RewriteMap
- 05:49 PM Revision 7983: web/vegpath.conf: Added spacing between sections and reordered directives
- 05:43 PM Revision 7982: web/vegpath/.htaccess: translate 2-level subdomain to prefix: interpret subdomains in (standard) reverse instead of forward order (e.g. history.DwC instead of DwC.history)
- 05:39 PM Revision 7981: web/vegpath/.htaccess: Don't also translate dash-paths to dirs (in addition to dot-paths) because this adds unnecessary complexity and was only used to allow expressing parts of subdomain paths in forward instead of reverse order (e.g. DwC-history instead of history.DwC)
- 05:31 PM Revision 7980: web/vegpath/.htaccess: translate dot-/dash-paths to dirs: don't match / inside a . or - component, because it has a lower precedence
- 05:31 PM Revision 7979: web/vegpath/.htaccess: translate dot-/dash-paths to dirs: changed comment to indicate that these rules can (eventually) translate any dot-/dash-path, not just the head
- 04:30 PM Revision 7978: web/vegpath/.htaccess: translate subdomain to prefix: Use last to avoid further rewriting, since these rules make the destination a /-path
- 04:28 PM Revision 7977: web/vegpath/.htaccess: Also translate dash-paths (e.g. DwC-history) to dirs
- 10:41 AM Revision 7976: web/vegpath/.htaccess: translate both 1-level and 2-level subdomains to dir prefixes
- 10:39 AM Revision 7975: web/vegpath/.htaccess: translate subdomain to prefix: HTTP_HOST RewriteCond: Don't capture part after . because it isn't used as a replacement (it used to be used for this when a now-unnecessary browser redirect was performed)
- 10:33 AM Revision 7974: web/vegpath/.htaccess: don't rewrite existing paths: Moved before other RewriteRules so that page dependencies (such as util.js) can be loaded more quickly without needing to pass through all the rewrite rules first
- 10:04 AM Revision 7973: web/vegpath/.htaccess: ErrorDocument: Use / instead of /index.php because index.php is the DirectoryIndex and doesn't need to be hardcoded
- 09:44 AM Revision 7972: web/vegpath/.htaccess: translate subdomain to prefix: Fixed bug where can only run this is if the dest subdir has a .htaccess file, because this prevents the RewriteRule from being re-run for the subdir (which would lead to infinite recursion because the rule is not idempotent)
- 09:37 AM Revision 7971: web/vegpath/.htaccess: translate subdomain to prefix: Fixed bug where need to prepend %{DOCUMENT_ROOT} to the file path when testing it for existence
- 09:16 AM Revision 7970: web/vegpath/.htaccess: translate path head to dir (if the dir exists)
- 08:56 AM Revision 7969: web/vegpath/.htaccess: translate subdomain to prefix: Set nosubreq to avoid infinite recursion (the RewriteRule is not idempotent)
- 08:43 AM Revision 7968: web/vegpath/.htaccess: translate subdomain to prefix: Don't translate all nested subdomains, because some of them may result from the machine's own name (vegbiendev.nceas.ucsb.edu) and should not be removed. Instead, remove only the leftmost level of subdomain, and only if it corresponds to a filesystem directory.
- 08:39 AM Revision 7967: web/vegpath/index.php: Use $alias whenever VegPath was reached via the machine's own name (vegbiendev.nceas.ucsb.edu) rather than via a shortened domain (path.vg), so that the right path root can seamlessly be used when path.vg is down
- 07:50 AM Revision 7966: web/vegpath/.htaccess: Translate subdomain to path prefix
- 07:30 AM Revision 7965: Added inputs/MO/_src/TropicosSpecimens_2013_03_12_*.txt.md5
- 07:25 AM Revision 7964: Added inputs/MO/_archive
- 07:25 AM Revision 7963: Added inputs/MO/_src/
- 07:14 AM Revision 7962: web/vegpath/index.php: Brad-Boyle root: Fixed link
- 07:12 AM Revision 7961: web/vegpath/index.php: CTFS root: Fixed link
- 07:11 AM Revision 7960: web/vegpath/index.php: CTFS-schema: Use $Redmine_svn
- 06:46 AM Revision 7959: web/vegpath/.htaccess: Remove www subdomain
- 06:25 AM Revision 7958: Added web/logs/
- 06:22 AM Revision 7957: web/vegpath/.htaccess: Use the simpler ErrorDocument directive instead of a catch-all RewriteRule to handle non-filesystem paths
- 06:11 AM Revision 7956: web/vegpath/index.php: $alias: Use HTTP_HOST and SCRIPT_NAME so the alias doesn't need to be hardcoded
- 06:09 AM Revision 7955: web/vegpath/util.php: Added ensure_prefix(), ensure_suffix()
- 05:32 AM Revision 7954: web/vegpath/index.php: Switching back to path.vg now that the .vg root DNS is back online
- 04:36 AM Revision 7953: web/vegpath/index.php: Use REQUEST_URI (removing the script's own dir) instead of PATH_INFO, to support being used for the Apache ErrorDocument directive
- 04:24 AM Revision 7952: web/vegpath/.htaccess: mod_rewrite: Don't run any further rewrite rules if path exists, to avoid needing to add RewriteConds to *each* RewriteRule to check if the path exists
- 01:39 AM Revision 7951: web/vegpath/index.php: Changed page title to contain "persistent URLs" because VegPaths are conceptually a type of PURL <http://en.wikipedia.org/wiki/PURL>
- 12:04 AM Revision 7950: web/vegpath/index.php: $alias: Switching back to j.mp/vegpath# because the .vg root DNS server is down. (The .vg registrar adamsnames.com also displays an under construction page.) If the registrar stays down, we may want to switch a different top-level domain.
03/12/2013
- 07:59 PM Revision 7949: web/vegpath/.htaccess: DirectoryIndex: Removed index.htm because HTML files should not be used for redirection (they don't issue the Location: header needed by clients other than web browsers)
- 07:52 PM Revision 7948: web/vegpath/.htaccess: mod_rewrite: Added `RewriteOptions inherit` so this .htaccess file can be copied to subdirs and will inherit the "unknown paths" rule
- 07:37 PM Revision 7947: web/vegpath/index.php: Use $PATH_INFO (i.e. index.php/$PATH_INFO) instead of the query string to pass the path to index.php, so that the query string can be used for other things
- 04:28 AM Revision 7946: web/vegpath/index.php: top-level path parsing: Use parse_dot_path() instead of parse_mixed_path() because XPaths are now (client-side) fragments, and don't need to be supported by the path mechanism
- 04:26 AM Revision 7945: web/vegpath/index.php: SALVIAS: Don't append #Plot_ when there is no subpath
- 03:19 AM Revision 7944: web/vegpath/index.php: SALVIAS: Factored out URL to $SALVIAS
- 03:16 AM Revision 7943: web/vegpath/index.php: SALVIAS: Appended Plot_ to dest URL because both data dictionary sections start with this, so it doesn't need to be included in the VegPath
- 02:56 AM Revision 7942: web/vegpath/index.php: XPath resources: Removed # at end of destination URL because the # is now part of the #/XPath suffix (and is added by the user)
- 02:53 AM Revision 7941: web/vegpath/index.php: XPath resources: Added # before XPath because the XPath is actually a location *within* the resource, and thus resolvable by the client (or user) themselves, rather than part of the redirected URL (which only needs to forward to the XML schema file)
- 02:41 AM Revision 7940: web/vegpath/index.php: Updated $alias to the newly-registered path.vg domain, which is shorter and more intuitive-looking than the previous value j.mp/vegpath (from bitly.com). Note that using a domain also allows redirects without JavaScript, which can be used by any web client (such as a Python script), not just a web browser.
03/11/2013
- 11:19 PM Revision 7939: web/vegpath/index.php: Renamed ref to ns to emphasize that the resources in this field are namespaces for the paths
- 11:16 PM Revision 7938: web/vegpath/index.php: add_ref(): Renamed $suffix to $path_pat to distinguish it from the actual suffix string sent in the URL
- 11:09 PM Revision 7937: web/vegpath/index.php: custom_separator(): Marked optional params as such
- 11:07 PM Revision 7936: web/vegpath/index.php: Terms: IH-db: Factored out phpMyAdmin URL into $IH_db
- 11:05 PM Revision 7935: web/vegpath/index.php: Terms: Added IH-db
- 11:05 PM Revision 7934: web/vegpath/index.php: php*Admin(): Added support for a fixed table that doesn't need to be specified in the path
- 10:43 PM Revision 7933: web/: Removed svn:executable permissions from non-executable files
- 10:36 PM Revision 7932: web/vegpath/index.php: Data: Added Institutions subsection
- 10:33 PM Revision 7931: web/vegpath/index.php: Placed Terms and Data sections side-by-side in two columns so that data resources would not need to come after a long list of terms resources
- 10:31 PM Revision 7930: web/vegpath/main.css: Added .indent
- 10:11 PM Revision 7929: web/vegpath/index.php: People: Added Brad's VegCore ID terms spreadsheets
- 10:10 PM Revision 7928: web/vegpath/index.php: Factored Redmine svn URLs out into $Redmine_svn
- 10:06 PM Revision 7927: Added schemas/VegCore/Brad_Boyle/ with Brad's VegCore IDs terms spreadsheets
- 09:51 PM Revision 7926: web/vegpath/index.php: URL fragment redirect: Removed loading indicator (blinking of resource being loaded) because for file downloads, the download will not redirect to a new page and the resource name will instead remain blinking
- 09:48 PM Revision 7925: web/vegpath/index.php: URL fragment redirect: Fixed bug where need to use parse_mixed_path() instead of parse_dot_path() to support XPaths
- 09:48 PM Revision 7924: web/vegpath/index.php: Terms: Added CTFS
- 09:23 PM Revision 7923: web/vegpath/index.php: URL patterns header: Reworded comment to clarify that the elements that can be left out are in italics
- 09:09 PM Revision 7922: web/vegpath/index.php: URL fragment redirect: Only perform fragment redirect if URL shortener requires it (as indicated by the $alias ending in #)
- 09:01 PM Revision 7921: web/vegpath/index.php: nimoy nodes: Factored nimoy phpMyAdmin() calls out into nimoy_db() function
- 08:57 PM Revision 7920: web/vegpath/index.php: nimoy nodes: Factored nimoy URL prefix out into $nimoy
- 08:51 PM Revision 7919: web/vegpath/index.php: SALVIAS: Added subnodes for each SALVIAS database on nimoy
- 08:37 PM Revision 7918: web/vegpath/index.php: VegCore: Changed suffix from .table.column to .term because VegCore terms are globally unique and don't need a table prefix
- 08:36 PM Revision 7917: web/vegpath/index.php: BIEN2: Added subnodes for each BIEN2 database on nimoy
- 08:04 PM Revision 7916: web/vegpath/index.php: Inlined VegBank() because it's only used once
- 08:01 PM Revision 7915: web/vegpath/index.php: VegBank(): Use custom_separator()'s new $main_url param
- 08:00 PM Revision 7914: web/vegpath/index.php: custom_separator(): Added $main_url param like in by_prefix(), for use when there is no path
- 07:55 PM Revision 7913: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:55 PM Revision 7912: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:53 PM Revision 7911: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:53 PM Revision 7910: web/vegpath/index.php: Added custom_separator()
- 07:52 PM Revision 7909: web/vegpath/index.php: Add # suffix to $alias instead of adding it in add_ref(), because not all aliases will use a # suffix
- 07:48 PM Revision 7908: web/vegpath/util.php: Added array_non_empty(), join_non_empty()
- 07:14 PM Revision 7907: web/vegpath/index.php: add_ref(): redirect: Don't display "Loading" as the page itself is not shown to the user when using a Location: header
- 07:12 PM Revision 7906: web/vegpath/index.php: URL fragment redirect: Fixed bug where can't prefix the fragment with / because this won't work if VegPath is not at the top-level of the domain (and in any case is unnecessary to form a relative path)
- 07:09 PM Revision 7905: web/vegpath/index.php: URL fragment redirect: Blink the namespace being loaded, rather than displaying "Loading" at the bottom of the page, to make it clearer what's being loaded
- 06:42 PM Revision 7904: web/vegpath/index.php: add_ref(): Don't display the URL pattern in monospace, because fixed-width font is not needed and variable-width font is easier to read. This requires decreasing the header level because variable-width font is bigger than fixed-width font.
- 06:41 PM Revision 7903: web/vegpath/main.css: Added entries for h6
- 06:28 PM Revision 7902: web/vegpath/index.php: parse_dot_path() calls: Fixed bug where need to test result components using \`!== ""\` instead of just using the value as a boolean because PHP incorrectly interprets the string "0" as false
- 06:24 PM Revision 7901: web/vegpath/util.php: Path: Made $tail default to ""
- 06:20 PM Revision 7900: web/vegpath/index.php: URL fragment redirect: Prepend / instead of ? because filesystem paths are now redirected using .htaccess, removing the need to set the query string
- 06:16 PM Revision 7899: web/vegpath/index.php: phpMyAdmin(): Fixed bug where can't use tbl_alter as the column target page because it requires a token param, which is only obtained after the user logs in
- 06:04 PM Revision 7898: web/vegpath/util.php: partition(): Fixed bug where strpos() actually does take the search-for param second rather than first (but returns false on no match)
- 06:00 PM Revision 7897: web/vegpath/util.php: partition(): Fixed bug where strpos() returns false rather than -1 on no match
- 05:59 PM Revision 7896: web/vegpath/util.php: partition(): Fixed bug where strpos() takes the search-for param first rather than second
- 05:37 PM Revision 7895: web/vegpath/index.php: php*Admin(): Append $target to $url in separate step from returning $url, to make it easier to debug-print the value of $url before returning it
- 05:30 PM Revision 7894: web/vegpath/index.php: BIEN2: Fixed to use phpMyAdmin with correct URL instead of vegbiendev phpPgAdmin placeholder
- 05:20 PM Revision 7893: web/vegpath/index.php: Added phpMyAdmin()
- 04:59 PM Revision 7892: web/vegpath/.htaccess: mod_rewrite: RewriteBase: Changed to / now that VegPath is hosted at vegbiendev.nceas.ucsb.edu
- 04:58 PM Revision 7891: web/vegpath/.htaccess: unknown paths: Don't redirect to index.php if the path already starts with index.php, to help avoid infinite redirect recursion
- 04:54 PM Revision 7890: web/vegpath/.htaccess: unknown paths: Removed no longer accurate VegRef from comment header
- 04:43 PM Revision 7889: vegpath/.htaccess: DirectorySlash: Lowercased On for consistency with other boolean directives
- 04:26 PM Revision 7888: Added web/vegpath.conf with Apache VirtualHost config
- 03:26 PM Revision 7887: vegpath/index.php: VegBank(): Fixed bug where name had incorrectly been replaced with $name
- 03:24 PM Revision 7886: vegpath/index.php: phpPgAdmin(): Factored out appending of "&subject="
- 03:23 PM Revision 7885: vegpath/: Archived no longer used vegpath/index.htm
- 03:05 PM Revision 7884: vegpath/index.*: IH: URL-encode ' in search URL to avoid needing to backslash-escape it
- 03:04 PM Revision 7883: vegpath/index.php: IH: Fixed search URL to undo incorrect search-and-replace of + with .
03/10/2013
- 06:51 AM Revision 7882: vegpath/index.*: Changed "Supported URLs" to "Supported URL patterns" because the provided short URLs take parameters that are filled into the destination URL
- 06:46 AM Revision 7881: vegpath/index.*: Removed tagline from window title because the window title (which becomes the bookmark name) only needs to be long enough to identify what the page is, and should not be cluttered with descriptions
- 06:43 AM Revision 7880: vegpath/index.*: Changed tagline to emphasize that the shortened URLs are especially intended to be human-readable, e.g. for use as column names
- 06:37 AM Revision 7879: VegRef/: Renamed to VegPath to make it clearer that the purpose of the web service is to provide short, globally unique paths to resources (e.g. for use as global IDs <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs>)
- 06:10 AM Revision 7878: VegRef/.htaccess: URL resolution: mod_rewrite: Fixed bug where also need to consider if %{REQUEST_FILENAME} is a dir or symlink, which occurs e.g. when redirecting to the VegRef/ dir itself
03/09/2013
- 10:50 PM Revision 7877: web/VegRef/.htaccess: Turned off RewriteEngine temporarily because it was preventing the internal VegRef redirection from working
- 10:48 PM Revision 7876: Added web/VegRef/, which provides a URL-shortening service for vegetation references. VegRef URLs are short and human-readable enough to be used as database column names. This also causes Google spreadsheets to display a clickable link for each column name, which takes the user to the data dictionary definition for the term. String row IDs (such as global IDs <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs>) can also be clickable VegRef URLs.
03/08/2013
- 10:26 PM Revision 7875: inputs/NCU/Specimen/map.csv: cultivated: Map false (0) to NULL because cultivated can only be true or NULL, not false
- 09:53 PM Task #584 (Resolved): enable running all the import steps from one runscript
- see [[Import process]]
- 09:14 PM Revision 7874: inputs/CTFS/_src/_README.TXT: Added e-mails from Brad Boyle about issues with species aggregation methods
03/07/2013
- 06:27 PM Revision 7873: inputs/UNCC/Specimen/map.csv: Populated specimenHolderInstitutions
- 03:21 PM Revision 7872: Added planning/goals/BIEN3_derived_data_products.docx.url
- 03:19 PM Revision 7871: Added planning/goals/BIEN3_derived_data_products.docx
03/06/2013
- 11:00 PM Revision 7870: inputs/ARIZ/omoccurrences/map.csv: cultivationStatus: Map to NULL if it's false, because cultivated can only be true or NULL
- 10:57 PM Task #581 (New): mechanism to dynamically autopopulate NOT NULL values
- * this is currently done using sql_io.put_table()'s col_defaults parameter, which only supports constant values
* th... - 10:01 PM Revision 7869: schemas/vegbien.sql: analytical_stem_view: Merged taxonName_verbatim and scientificNameAuthorship_verbatim into scientificName_verbatim
- 09:29 PM Revision 7868: schemas/vegbien.sql: analytical_plot: Added specimens fields at the end, for use by plots data that also tracks specimens (e.g. Madidi, SALVIAS)
- 09:21 PM Revision 7867: README.TXT: Full database import: In PostgreSQL: Removed tests related to unscrubbed_taxondetermination_view and TNRS taxondeterminations, which are no longer needed because TNRS is joined directly to analytical_stem_view
- 09:08 PM Task #580 (New): automap each word *in* a VegCore term instead of just the whole term
- * this avoids the need to create repetitive synonyms for each variation of a term (e.g. with different units)
- 08:45 PM Task #579 (New): in sql_gen.EnsureNotNull, use ARRAY[] instead of COALESCE() to wrap values where NULL should be compared literally
- * this avoids the need for different null sentinels for each datatype
* it would still be necessary to ensure that t... - 08:36 PM Revision 7866: mappings/VegCore.htm: Regenerated from wiki. Renamed specimenDonor to specimenDonorInstitution to clarify that this is an institution, not a collector.
- 08:30 PM Revision 7865: inputs/UNCC/Specimen/map.csv: Mapped flower, fruit -> reproductiveCondition using translation is from e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet on 2010-12-2~3
- 08:17 PM Revision 7864: make_analytical_db: Support the $log flag, which toggles output to the terminal or a log file
- 07:50 PM Revision 7863: inputs/import.stats.xls: Updated import times. The core import time has dropped by more than half (!) to ~12 hours, now that the TNRS scrubbing is added using a simple LEFT JOIN, instead of being pushed through the normalized schema. Not since October has the import been this fast!
- 07:27 PM Revision 7862: import_times: When there are no matching logs, exit with status 0 instead of 1 because this is a normal condition
- 07:26 PM Revision 7861: import_times: Fixed bug where sed would wait for user input from stdin when there were no matching logs
- 07:22 PM Revision 7860: README.TXT: Full database import: record the import times in inputs/import.stats.xls: If the row counts do not match up: Added step to get the datasource names from the log file footers
- 07:12 PM Revision 7859: inputs/UNCC/Specimen/map.csv: cultivated: assume anything non-NULL to be cultivated, so that we err on the side of excluding possibly-cultivated specimens
- 06:49 PM Revision 7858: inputs/UNCC/_src/_README.TXT: Added e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet about the meanings of the Leaf, Flower, Fruit, and Root codes
- 05:38 PM Revision 7857: README.TXT: Full database import: Added steps to manually back up the DB. (These steps are specific to the import process, and include filters to exclude previous imports. There are also general backup steps in the Backups > Full DB section.)
- 02:56 PM Revision 7856: inputs/ACAD/Specimen/map.csv: institutionCode: Replace "Acadia University" with the IH code ACAD
- 02:18 PM Revision 7855: inputs/.TNRS/data.sql: Re-ran TNRS on the test_taxonomic_names so that the sample data would contain all the accepted names under the current acceptedScientificName formula, and would not produce any entries for unscrubbed names in tnrs_input_name
- 02:08 PM Task #578 (New): replace all fields of type text with a domain type that ensures the value is non-empty (not '')
- * empty strings tend to cause bugs all over the database because they are sometimes, but not always, treated as NULL
- 01:59 PM Revision 7854: schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Don't exclude taxonlabels with a matched_label_id, because matched_label_id is no longer used to indicate that a name has a scrubbed name (this was used when all of TNRS was imported at the beginning of the import)
- 01:56 PM Revision 7853: schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Renamed to the VegCore table name AcceptedTaxon
- 01:54 PM Revision 7852: schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Use new MatchedTaxon view, which use VegCore column names where possible
- 01:46 PM Revision 7851: schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Use new MatchedTaxon, AcceptedTaxon views, which use VegCore column names where possible
- 01:22 PM Revision 7850: README.TXT: Full database import: Added instructions to run TNRS and view the TNRS log
- 10:21 AM Revision 7849: README.TXT: Full database import: Removed no longer necessary command to scrub unscrubbed taxondeterminations
03/05/2013
- 11:07 PM Revision 7848: *.sql: concat_ws() calls: Fixed bug where need to surround it with NULLIF(..., '') because concat_ws() returns '' instead of the required NULL when supplied with only NULL args
- 10:22 PM Revision 7847: schemas/vegbien.sql: analytical_stem_view: concat_ws() calls: Fixed bug where arguments were still enclosed in ARRAY[], because the name had been changed from concat_delim() but PostgreSQL removed the VARIADIC keyword without also removing the ARRAY[] wrapper
- 10:11 PM Revision 7846: schemas/vegbien.sql: Removed no longer needed unscrubbed_taxondetermination_view. Use taxonlabel joined to ScrubbedTaxon instead.
- 10:05 PM Revision 7845: inputs/.TNRS/schema.sql: ScrubbedTaxon: Include only matches with a match score of at least 0.8 (score_ok())
- 09:59 PM Revision 7844: inputs/.TNRS/schema.sql: Added back score_ok() and tnrs.tnrs_score_ok index
- 09:53 PM Revision 7843: *.sql: Replaced concat_delim() with concat_ws(), which is built-in as of PostgreSQL 9.1 (http://stackoverflow.com/questions/1943433/postgresql-concat-ws-like-function)
- 09:08 PM Revision 7842: inputs/U/Specimen/map.csv: Populated specimenHolderInstitutions
- 09:07 PM Revision 7841: mappings/VegCore.htm: Regenerated from wiki. Renamed specimenHolders to specimenHolderInstitutions to make it obvious that this is a list of institutions, such as would be in institutionCode in a DwC export.
- 09:06 PM Revision 7840: inputs/SpeciesLink/Specimen/map.csv: Fixed embedded line endings
- 09:05 PM Task #577 (Rejected): use views instead of map spreadsheets to store the datasource mappings
- * this allows the contents of the view to be appended directly to the flat Occurrence table, with the column names ma...
- 08:28 PM Revision 7839: inputs/NY/Ecatalog_all/map.csv: PlantFungDescription->occurrenceRemarks: Removed "plant fungus" label because this field actually contains the specimenDescription
- 08:06 PM Revision 7838: inputs/.TNRS/import_order.txt: Removed public.unscrubbed_taxondetermination_view
- 08:01 PM Revision 7837: inputs/.TNRS/: Moved no longer needed public.unscrubbed_taxondetermination_view/ to _archive/
- 07:55 PM Task #565 (Resolved): partition the TaxonDetermination table by row into the different types of determinations
- See TNRS.AcceptedTaxon, MatchedTaxon, ScrubbedTaxon
- 07:52 PM Revision 7836: root Makefile: Datasources: scrub: Removed no longer needed scrub-remake, which has been replaced by direct use of TNRS.ScrubbedTaxon
- 07:46 PM Revision 7835: schemas/vegbien.sql: analytical_stem_view: taxondetermination: Fixed bug where need to include only the is_datasource_current taxondetermination
- 07:43 PM Revision 7834: schemas/vegbien.sql: analytical_stem_view: Use new TNRS.ScrubbedTaxon directly instead of the scrubbed taxondeterminations added to the normalized DB
- 07:20 PM Revision 7833: inputs/.TNRS/schema.sql: MatchedTaxon: Renamed matchedTaxonomicStatus to preferred name taxonomicStatus
- 07:19 PM Revision 7832: mappings/VegCore.htm: Regenerated from wiki. Taxon: Moved scrubbing-related terms to MatchedTaxon.
- 07:16 PM Revision 7831: inputs/.TNRS/schema.sql: Added matched* prefix where needed
- 07:02 PM Revision 7830: inputs/.TNRS/schema.sql: MatchedTaxon: Added missing Name_submitted column
- 06:59 PM Revision 7829: inputs/.TNRS/schema.sql: MatchedTaxon: Name_matched: Fixed mapping to matchedTaxonName
- 06:56 PM Revision 7828: inputs/.TNRS/schema.sql: MatchedTaxon: Fixed bug where can't map Source to source because it refers just to the source of the matched name, rather than of the record (which would be TNRS itself)
- 06:55 PM Revision 7827: inputs/.TNRS/schema.sql: tnrs_populate_derived_fields(): Use new concat_delim()
- 06:43 PM Revision 7826: schemas/functions.sql: _join*(): Use new concat_delim()
- 06:41 PM Revision 7825: schemas/functions.sql: Added concat_delim() (also in public, but this copy is for use by the functions schema)
- 06:37 PM Revision 7824: inputs/.TNRS/schema.sql: Removed no longer used tnrs+accepted. Use ScrubbedTaxon instead.
- 06:22 PM Revision 7823: inputs/.TNRS/schema.sql: Added MatchedTaxon, AcceptedTaxon, ScrubbedTaxon views, which rename the columns to VegCore names
- 05:17 PM Revision 7822: Removed inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump*, which has been replaced by GBIFPortalDB-2013-02-20.dump
- 05:14 PM Revision 7821: mappings/VegCore.htm: Regenerated from wiki. Renamed MatchedTaxonDetermination/AcceptedTaxonDetermination to MatchedTaxon/AcceptedTaxon because they relate to a taxon *name* rather than to an application of that name to a TaxonOccurrence
- 02:43 PM Revision 7820: schemas/functions.sql: functions._map(): Made function STRICT because it always returns NULL if either argument is NULL
- 02:39 PM Revision 7819: inputs/UNCC/Specimen/map.csv: campus: Mapped to cultivated, interpreting non-NULL values as true
- 02:29 PM Revision 7818: inputs/UNCC/Specimen/map.csv: campus: Removed mapping to occurrenceRemarks because this is just a code, not a description. Added definition from Bob's validation comments.
- 02:19 PM Revision 7817: inputs/UNCC/Specimen/map.csv: accession: Removed comment that this is actually the catalogNumber, because they are now synonyms
- 01:56 PM Revision 7816: mappings/VegCore.htm: Regenerated from wiki. Split specimenIndexer into specimenStampingInstitution and specimenDonor (which are often the same).
- 12:25 PM Revision 7815: mappings/VegCore-VegBIEN.csv: Removed specimenIndexer->institutionCode mappings because the institutionCode should refer only to the specimenHolder
- 12:18 PM Revision 7814: Added inputs/GBIF/_src/GBIFPortalDB-2013-02-20.dump.gz (a new download they posted to the FTP site)
- 12:13 PM Revision 7813: schemas/vegbien.sql: analytical_specimen: Removed growthForm, threatened_bien at Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#UNCC > Brad Boyle's comments > growthForm, threatened_bien)
- 11:24 AM Revision 7812: inputs/UNCC/Specimen/map.csv: Removed mappings to reproductiveCondition because we don't know what the single-letter codes mean
- 11:16 AM Revision 7811: inputs/UNCC/Specimen/map.csv: Fixed embedded line endings
- 11:05 AM Revision 7810: schemas/vegbien.sql: _locationnarrative_is_cultivated(): Fixed bug where need to use NULL instead of false (and bool_or() instead of OR) for a non-match, because the absence of a match doesn't mean it's not cultivated
- 05:09 AM Task #576 (New): merge the datasources' mappings into the VegCore data dictionary
- * each mapping defines a synonym of a VegCore term
- 05:07 AM Task #468 (Resolved): rename/delete and recreate public schema in one transaction
- 05:04 AM Task #340 (Resolved): put together definitions of abstract BIEN terms
- See [[VegCore data dictionary]]
- 03:02 AM Revision 7809: mappings/VegCore.htm: Regenerated from wiki. Editing: Changed ambiguous terms templates to list the name of the alternative itself in the alternatives list, instead of the ambiguous term + a source suffix. This avoids the need to choose a source suffix for each alternative of an ambiguous term, as the ambiguous term itself (without the suffix) can be listed as the synonym. Coordinates: Disambiguated verbatimCoordinateSystem.
- 02:30 AM Revision 7808: mappings/Makefile: VegCore.thesaurus.csv: `uniq`ify the map after applying initial translations, to support mappings that appear twice (once as the result of an alternative's synonym entry and again as the result of an ambiguous term's alternatives listing)
03/04/2013
03/01/2013
- 08:17 PM Revision 7806: mappings/VegCore.htm: Regenerated from wiki. Added specimenDescription.
- 08:00 PM Revision 7805: mappings/VegCore.htm: Regenerated from wiki. Split apart identificationQualifier and taxonFit. Split apart taxonConfidence and identificationVerificationStatus. Moved identificationQualifier to new QualifiedTaxonName table.
- 07:48 PM Revision 7804: README.TXT: Full database import: To view progress: Use tail -100 to show more of the log files, and not tail -f to avoid blocking when the user just wants to check the status
- 04:56 PM Revision 7803: inputs/BRIT/specimen_flat/map.csv: Only map Notes_Plant to occurrenceRemarks if there is a separate Locality_Description, in order to filter out Notes_Plant values that concatenate everything together (often including a herbarium name that would mess up the cultivated flag parsing)
- 04:49 PM Revision 7802: my2pg.data: Replace \0 (with the backslash) with 0 because \0 isn't allowed in UTF-8
- 03:53 PM Revision 7801: root Makefile: VegBIEN DB: mk_db: Added command to import arbitrary DB users from a config file
02/28/2013
- 03:31 PM Revision 7800: inputs/import.stats.xls: Updated import times
- 03:29 PM Revision 7799: inputs/TEX/Specimen2/map.csv: _join() PLANT DATA and MISC NOTES to form occurrenceRemarks instead of _alt()ing them
- 03:27 PM Revision 7798: inputs/TEX/Specimen2/test.xml.ref: Accepted test
- 02:58 PM Revision 7797: inputs/BRIT/_MySQL/MySQL.*.sql.make: Include all tables, not just specimen_flat
- 02:57 PM Revision 7796: my2pg: Replace tinytext, mediumtext with text
- 11:48 AM Revision 7795: Added inputs/GBIF/_src/_README.TXT with Brad's GBIF filtering rules
- 11:30 AM Revision 7794: inputs/FIA/TREE/postprocess.sql: Use :table instead of hardcoding the table name
- 11:26 AM Revision 7793: inputs/FIA/*/postprocess.sql: UNIQUE constraint: Added UNITCD because although it may not be needed to scope PLOT, it is present in all the core tables and thus must be part of the joins' USING columns
- 10:58 AM Revision 7792: inputs/FIA/*/postprocess.sql for core tables: Added unique constraint on the datasource's primary key fields
- 02:38 AM Revision 7791: Added inputs/CTFS/AggregateObservation/ from BIEN2 data
- 02:16 AM Revision 7790: inputs/CTFS/: Switched global _no_import to table-specific _no_imports to allow adding new tables that *are* imported
02/27/2013
- 09:54 PM Revision 7789: Added schemas/TAPIR/SpeciesLink.tapir_request.txt symlink
- 09:53 PM Revision 7788: Added inputs/SpeciesLink/_src/tapir_request.txt
- 09:52 PM Revision 7787: Added schemas/TAPIR/
- 09:48 PM Revision 7786: Added planning/use_cases/rare_species.png
- 09:47 PM Revision 7785: Added planning/goals/BIEN deliverables.png
- 09:31 PM Revision 7784: Added planning/resources/iPlant_Overview_BIEN_2011.pdf.url
- 09:29 PM Revision 7783: planning/goals/Enquist_BIEN_Overview_2011.pptx: Replaced with .url file. Note that the Plone site <http://groups.nceas.ucsb.edu/bien/meetings> is not searchable by document filename.
- 09:26 PM Revision 7782: Added planning/workflow/Boyle_BIEN_data_quality.pdf.url
- 09:21 PM Revision 7781: Added planning/goals/BIEN_Science_2011.pptx.url, BIEN-Tech-Oct2011.ppt.url, Enquist_BIEN_Overview_2011.pptx
- 09:00 PM Revision 7780: Added schemas/VegX/run.py
- 08:53 PM Revision 7779: Added schemas/VegBIEN/growth_forms/Barbara_Dobrin.parsing.scripts/
- 08:35 PM Revision 7778: Added schemas/VegX/Veg-X_paper_in_press_Applied_Vegetation_Science.pdf.url
- 08:25 PM Revision 7777: Added schemas/VegX/JVS-VegX.pdf.url
- 08:22 PM Revision 7776: Added schemas/VegBank/VegBank_Data_Model.with_Collections_intro.ppt
- 08:13 PM Revision 7775: Added planning/goals/BIEN_White_Paper.pdf.url
- 08:11 PM Revision 7774: Added planning/meetings/2010-12-08.working_group/BIEN3-plan.docx
- 08:04 PM Revision 7773: Added schemas/VegBIEN/planning/taxonomy/On_the_Use_of_Taxonomic_Concepts_in_Support_of_Biodiversity_Research_and_Taxonomy.pdf.url
- 07:59 PM Revision 7772: Added schemas/BIEN2/ERD/
- 07:54 PM Revision 7771: Added schemas/BIEN2/bien2_core_schema.png
- 07:54 PM Revision 7770: Added schemas/BIEN2/mappings/BIEN2.viewFullOccurrence-DwC.xls
- 07:51 PM Revision 7769: Added schemas/VegBank/mappings/VegBank.observation-VegX.plotObservation.xls
- 07:49 PM Revision 7768: schemas/vegbien.ERD.mwb: Fixed table positions due to plantobservation field addition
- 07:35 PM Revision 7767: Added schemas/DiGIR/gbif-occurrencestore.digir.client.url
- 07:33 PM Revision 7766: Added schemas/DiGIR/gbif-crawler.digir.*.url
- 07:27 PM Revision 7765: inputs/.geoscrub/import_order.txt: Added Source
- 07:09 PM Revision 7764: Added schemas/VegX/
- 07:00 PM Revision 7763: Added schemas/VegBIEN/taxonomy/family_higher_plant_group.sql
- 06:57 PM Revision 7762: Added planning/use_cases/Trait_BIEN_data_use_case.docx
- 06:56 PM Revision 7761: Added planning/workflow/2010.working_group.white_board/
- 06:54 PM Revision 7760: Added planning/milestones/_archive/BIEN3 Milestones and Development Timeline_MLN2.*
- 06:51 PM Revision 7759: Added planning/workflow/BIEN-modArch-Dec2010 NS-SBD 1.4.ppt
- 06:50 PM Revision 7758: Added planning/milestones/_archive/BIEN3_milestones.2011-12-13.doc
- 06:48 PM Revision 7757: Added planning/milestones/BIEN Milestone – feedback.docx
- 06:46 PM Revision 7756: Added schemas/VegBIEN/planning/taxonomy/Nomenclature_excerpt.ppt
- 06:44 PM Revision 7755: Added planning/use_cases/
- 06:42 PM Revision 7754: Added planning/meetings/2012-11-26~30.working_group/
- 06:41 PM Revision 7753: Added schemas/BIEN2/Guide_to_the_BIEN2_database.pdf
- 06:39 PM Revision 7752: Added planning/meetings/2011-10-24~28.working_group/
- 06:36 PM Revision 7751: Added schemas/VegBIEN/planning/BIEN_OctWG-Whtbrd*.jpg
- 06:32 PM Revision 7750: Added schemas/BIEN2/
- 06:29 PM Revision 7749: Added schemas/VegBIEN/
- 06:20 PM Revision 7748: Moved schemas/vegbank.* to schemas/VegBank/
- 06:15 PM Revision 7747: Added planning/milestones/BIEN3 - Progress since the 2011 working group.docx
- 06:13 PM Revision 7746: planning/meetings/: Added Martha's conference call notes
- 06:06 PM Revision 7745: Moved planning/milestones.doc to planning/milestones/
- 06:04 PM Revision 7744: Moved planning/timeline.* to timeline/
- 06:01 PM Revision 7743: Renamed to_do/ to planning/
- 05:53 PM Revision 7742: Added schemas/VegCore/
- 05:53 PM Revision 7741: Added schemas/DwC/
- 05:47 PM Task #575 (New): put all wiki attachments in svn
- Wiki attachments are not version controlled, so it's better to link to the web URL of a file in the repository:
* ... - 05:35 PM Revision 7740: Added schemas/BIEN_logo.png from the bien.nceas.ucsb.edu website
- 05:29 PM Revision 7739: Renamed schemas/BIEN_logo.png to BIEN_logo.small.png
- 05:10 PM Revision 7738: Added inputs/SALVIAS/_archive/salvias_bien_19nov2009.tar.gz.url, .md5
- 04:57 PM Revision 7737: Added inputs/REMIB/_archive/remib_raw.0.header.specimens.txt
- 04:53 PM Revision 7736: Added inputs/CTFS/_src/ctfs_extract.tab.url, .md5
- 04:48 PM Revision 7735: Added inputs/NVS/_src/NVS_VegX.XSLT, _src/VegX/
- 04:43 PM Revision 7734: Added inputs/NVS/_src/_README.TXT
- 04:41 PM Revision 7733: Added inputs/TurboVeg/_src/DBASEDIC_rkp2011*.xlsx
- 04:40 PM Revision 7732: Added inputs/TurboVeg/_src/_README.TXT
- 03:45 PM Revision 7731: inputs/import.stats.xls: Updated import times
- 03:36 PM Revision 7730: README.TXT: Full database import: Don't run backups/fix_perms with sudo because owners are not rsynced to jupiter, so fixing them first is unnecessary. (It's not necessary to change the file owner to bien at any other time, either, because commands are not run as the bien user. It's in fact better to leave the file owner as yourself, to allow rsync to set the modification times.)
- 03:24 PM Revision 7729: README.TXT: Full database import: Added steps to set $version in all vegbiendev terminals
- 03:00 PM Revision 7728: lib/common.Makefile: rsync: $(rsync): Also exclude .lk* lock files
- 02:57 PM Revision 7727: README.TXT: Full database import: Updated log file sync steps for syncing via jupiter
- 02:56 PM Revision 7726: inputs/BRIT/specimen_flat/test.xml.ref: Updated inserted row count
02/26/2013
- 08:27 PM Revision 7725: schemas/py_functions.sql: Added _date(year integer, month integer, day integer)
- 08:08 PM Revision 7724: inputs/BRIT/specimen_flat/postprocess.sql: Added primary key
- 07:40 PM Revision 7723: schemas/functions.sql: Added _dms_to_dd() that takes text arguments and casts them to the appropriate type
- 07:11 PM Revision 7722: inputs/HVAA/Specimen/map.csv: Remapped fieldNotes to UNUSED. This also fixes a bug where fieldNotes was colliding with other occurrenceRemarks inputs because it was missing a filter.
- 06:46 PM Revision 7721: schemas/vegbien.sql: analytical_specimen/analytical_plot: Added occurrenceRemarks
- 06:32 PM Revision 7720: inputs/REMIB/Specimen/map.csv: Mapping NULL-equivalent lat/long to NULL: Fixed bug where need special *=* mapping to pass through values not in the map
- 06:30 PM Revision 7719: inputs/REMIB/Specimen/map.csv: Map NULL-equivalent text fields (country, state, etc.) to NULL
- 06:23 PM Revision 7718: inputs/REMIB/Specimen/map.csv: Map NULL-equivalent lat/long to NULL
- 05:59 PM Revision 7717: inputs/SpeciesLink/Specimen/map.csv: dwc_dwcore_Remarks: Remapped to occurrenceRemarks (it had been incorrectly automapped to taxonRemarks). Combined occurrenceRemarks inputs using /_alt.
- 05:47 PM Revision 7716: mappings/VegCore.htm: Regenerated from wiki. Remapped organismNotes to be a synonym of occurrenceRemarks, since notes on an organism are more generally notes on an occurrence.
- 05:28 PM Revision 7715: inputs/HVAA/Specimen/map.csv: Remapped dynamicProperties to occurrenceRemarks because it contains occurrenceRemarks data
- 05:23 PM Revision 7714: schemas/vegbien.sql: analytical_stem_view: Added occurrenceRemarks
- 05:19 PM Revision 7713: mappings/VegCore-VegBIEN.csv: Mapped occurrenceRemarks
- 04:53 PM Revision 7712: mappings/VegCore-VegBIEN.csv: Only use one of collection and collectionURL instead of concatenating them when both are provided (as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>)
- 04:53 PM Revision 7711: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:52 PM Revision 7710: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:44 PM Revision 7709: inputs/FIA/_src/_README.TXT: Added e-mail from Brad Boyle on 2013-2-8
- 04:42 PM Revision 7708: inputs/FIA/_src/: Added attachments from Brad's e-mail on 2013-2-8
- 04:37 PM Revision 7707: xml_func.py: Removed no longer used _range(). Use SQL functions._range() instead.
- 04:34 PM Revision 7706: schemas/functions.sql: Added _range()
- 04:30 PM Revision 7705: xml_func.py: Removed no longer used _avg(). Use SQL functions._avg() instead.
- 04:29 PM Revision 7704: schemas/functions.sql: Added _avg()
- 04:27 PM Revision 7703: mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions
- 04:15 PM Revision 7702: schemas/functions.sql: _dms_to_dd(text): Corrected concatenated form to interpret last 5 digits as MMmmm = MM.mmm instead of MMSSS, per Tom Wendt's explanation that this form is "degrees/minutes/decimalminutes"
- 04:05 PM Revision 7701: inputs/MO/Specimen/postprocess.sql: Use :table instead of hardcoding "MO"."Specimen"
- 04:04 PM Revision 7700: inputs/MO/Specimen/postprocess.sql: Also remove frameshifted rows where InstitutionCode is NULL
- 03:44 PM Revision 7699: inputs/UNCC/Specimen/map.csv: herbarium->specimenIndexer: Replace UNCCD (UNCC database?) with UNCC
- 03:27 PM Revision 7698: schemas/functions.sql: functions._dms_to_dd(): Added support for DMS values without separators
- 03:09 PM Revision 7697: README.TXT: Datasource setup: Update vegbiendev: Removed command to run the tests on vegbiendev because vegbiendev no longer has an empty public schema for running tests on (the public schema instead contains the latest published import)
- 03:04 PM Revision 7696: Mapped inputs/BRIT/
- 02:36 PM Revision 7695: my2pg: Remove (0) after char
- 02:35 PM Revision 7694: my2pg: Remove any (#) after bytea
- 02:33 PM Revision 7693: my2pg: Translate binary to bytea
- 02:30 PM Revision 7692: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Set $server to vegbiendev instead so the export command will work from any machine (though it will be faster from vegbiendev itself)
- 02:28 PM Revision 7691: inputs/BRIT/_MySQL/MySQL.*.sql.make: Fixed bug where needs to be set to just localhost and *--user* to bien instead
- 02:27 PM Revision 7690: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Fixed bug where $server needs to be set to just localhost and *--user* to bien instead.
- 02:25 PM Revision 7689: lib/MySQL.*.sql.make: Added --user option to demonstrate how to set the DB user vs. the ssh user
- 02:19 PM Revision 7688: inputs/BRIT/_MySQL/MySQL.*.sql.make: Fixed bug where $server needs to be set to bien@localhost
- 02:18 PM Revision 7687: README.TXT: Datasource setup: MySQL inputs: Editing _MySQL/*.make: Specified that $server needs to be set to bien@localhost. Install the staging tables: Specified that for a MySQL .sql export, this must be done on vegbiendev.
- 02:15 PM Revision 7686: inputs/BRIT/: Added _MySQL/MySQL.*.sql.make
- 02:08 PM Revision 7685: README.TXT: Datasource setup: MySQL inputs: .sql exports: Run mysql_bien in the background
- 02:03 PM Revision 7684: README.TXT: Datasource setup: MySQL inputs: .sql exports: Added steps to sync the export to vegbiendev. Specified that the database for the MySQL export should be created on vegbiendev.
- 01:58 PM Revision 7683: README.TXT: Datasource setup: Update vegbiendev: Updated steps for syncing via jupiter
- 01:56 PM Revision 7682: Added inputs/BRIT/ (not yet translated to PostgreSQL)
- 01:35 PM Revision 7681: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_DMS to coordinates.latitude_deg using new _dms_to_dd(text)
- 01:34 PM Revision 7680: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_DMS to coordinates.latitude_deg using new _dms_to_dd(text)
- 01:29 PM Revision 7679: schemas/functions.sql: Added _dms_to_dd(text)
- 01:28 PM Revision 7678: schemas/functions.sql: _dms_to_dd(): Fixed bug where need to use 1 as the multiplier when dir is NULL
- 01:19 PM Revision 7677: schemas/functions.sql: _dms_to_dd(): Fixed bug where need to apply the sign multiplier *after* the components are added, because it applies to the DMS as a whole rather than just to the degree component
- 12:58 PM Revision 7676: README.TXT: Full database import: Before running `make inputs/upload`, added step to do ./fix_perms on local machine so that files are not uploaded to jupiter with incorrect permissions
- 12:57 PM Revision 7675: README.TXT: Full database import: Before running `make inputs/upload`, added step to do svn up on jupiter so that unversioned files are not uploaded before their versioned equivalents are checked out from svn (requiring each file to be manually svn-added before the svn up can proceed)
- 12:48 PM Task #574 (New): mechanism to manually override the CSV dialect, encoding, etc.
- 12:37 PM Task #573 (New): automap the analytical DB columns
- * This will ensure the analytical DB schema stays up to date with VegCore
h3. Method
In @schemas/vegbien.sql@:
... - 12:30 PM Revision 7674: schemas/functions.sql: _dms_to_dd(): Added dir param to specify compass dir suffix, in order to calculate the sign correctly
- 11:05 AM Revision 7673: Moved compass_dir from schemas/vegbien.sql to schemas/functions.sql so it can be used by _dms_to_dd()
- 10:56 AM Revision 7672: Moved _dms_to_dd() from schemas/vegbien.sql to schemas/functions.sql so it can use functions schema functions
- 10:51 AM Revision 7671: schemas/vegbien.sql: Added compass_dir enum
- 10:42 AM Revision 7670: inputs/TEX/Specimen/map.csv: Remapped LAT/LONG_DEGREES,MINUTES,SECONDS to new latitude/longitude_deg,min,sec, allowing the DMS coordinates to be translated
- 10:40 AM Revision 7669: inputs/REMIB/Specimen/map.csv: Remapped lat/long_deg,min,sec to new latitude/longitude_deg,min,sec, allowing the DMS coordinates to be translated
- 10:39 AM Revision 7668: mappings/VegCore-VegBIEN.csv: latitude/longitude_deg,min,sec: Also mapped to the geoscrub coordinates entry
- 10:34 AM Revision 7667: mappings/VegCore-VegBIEN.csv: latitude/longitude_sec: Fixed name, which had been incorrectly automapped to verbatim*
- 10:31 AM Revision 7666: mappings/VegCore.htm: Regenerated from wiki. latitude/longitude: Moved *_deg,min,sec components after unit alternatives so that unit alternatives following them wouldn't incorrectly be interpreted as alternatives for the *components* instead of for the overall, unitless term.
- 10:22 AM Revision 7665: mappings/VegCore-VegBIEN.csv: Mapped latitude/longitude_deg,min,sec
- 10:15 AM Revision 7664: mappings/VegCore.htm: Regenerated from wiki. Merged coordinateUncertainty and coordinatePrecision, since datasources often use these interchangeably (the only difference is the DwC-recommended units).
- 10:08 AM Revision 7663: mappings/VegCore.htm: Regenerated from wiki. Source: Added authors. Renamed creator to firstPublisher. Moved dataOwner to Source as list. Coordinates: latitude/longitude, coordinatePrecision: Added units disambiguation. latitude/longitude_DMS: Added components. Global IDs: Changed "globally unique" tag to "global ID" and hyperlinked it to <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs> (which now includes detailed examples of the various types of global IDs).
- 09:58 AM Revision 7662: inputs/SpeciesLink/Specimen/map.csv: conceptual_darwin_2003_1_0_CoordinatePrecision: Remapped to coordinateUncertaintyInMeters, assuming units based on the range and precision of values
- 09:53 AM Revision 7661: inputs/HVAA/Specimen/map.csv: Remapped coordinatePrecision to UNUSED
- 09:50 AM Revision 7660: inputs/ARIZ/omoccurrences/map.csv: Remapped coordinatePrecision to UNUSED
- 09:05 AM Revision 7659: schemas/vegbien.sql: Added _dms_to_dd()
- 08:49 AM Revision 7658: inputs/bien2_traits/TraitObservation/map.csv: Remapped Source to sourceType
- 08:43 AM Revision 7657: Added inputs/TEX/
- 08:42 AM Revision 7656: units.py: convert(): Support ranges
- 08:35 AM Revision 7655: Moved parse_range() from xml_func.py to units.py
- 08:02 AM Revision 7654: units.py: conversions: Added ft->m conversion
- 07:12 AM Revision 7653: src_map: Reopen stdin (/dev/stdin) in universal newlines mode to support files with \r as the line ending
- 06:51 AM Revision 7652: csv2db: Open input stream in universal newlines mode to support files with \r as the line ending
02/22/2013
- 11:15 PM Revision 7651: mappings/VegCore.htm: Regenerated from wiki. Moved Forming IDs section to separate Global IDs wiki page (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs).
- 11:09 PM Revision 7650: input.Makefile: $(exts): Added .tab
- 10:29 PM Revision 7649: mappings/VegCore.htm: Regenerated from wiki. Moved preferred ID order to Forming IDs section as examples list. Made examples globally unique by prefixing the outer ID that scopes the value.
- 09:29 PM Revision 7648: mappings/VegCore.htm: Regenerated from wiki. Forming IDs: Added instructions for making a local ID global. Added note that these globally unique IDs are not the same as UUIDs. Forming preferred IDs: Reworded as a set of instructions rather than a definition. Clarified that this becomes the database primary key.
- 08:47 PM Revision 7647: mappings/VegCore.htm: Regenerated from wiki. Added section on forming preferred IDs, which are the ID which is most directly associated with the identified object.
- 08:17 PM Revision 7646: mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).
- 08:00 PM Revision 7645: inputs/NY/Ecatalog_all/test.xml.ref: Updated inserted row count
02/21/2013
- 05:13 PM Revision 7644: schemas/vegbien.sql: analytical_stem_view: scientificName_verbatim: Use the taxonname (which omits the family and author) instead when available, at Brad's request
- 04:02 PM Revision 7643: schemas/vegbien.sql: Disambiguated taxonlabel.taxonomicname, taxonverbatim.taxonomicname, and taxonverbatim.taxonname
- 03:14 PM Revision 7642: schemas/vegbien.sql: taxon_trait_view: Use taxonverbatim.taxonname (the binomial) in preference to taxonverbatim.taxonomicname (the binomial+author) or taxonlabel.taxonomicname (the family+binomial+author), at Mark's request
- 02:05 PM Revision 7641: to_do/timeline.2013.xls: Obtain and load any additional new data: Extended beyond April because it's ongoing
- 02:02 PM Revision 7640: README.TXT: Full database import: On nimoy: Moved analytical_aggregate table creation right after analytical_aggregate..csv download because it can happen while the file is downloaded
- 11:13 AM Revision 7639: to_do/timeline.2013.xls: Reload existing data in need of refresh: Added additional cell because this is still in progress
- 11:11 AM Revision 7638: to_do/timeline.2013.xls: Placed check mark in completed cells. Multi-cell tasks that are partially complete have check marks in part of their cells to indicate this.
02/20/2013
- 05:35 PM Revision 7637: Added inputs/FIA/REF_*/
- 05:33 PM Revision 7636: inputs/FIA/_src/Makefile: Table bundling: Also unpack FIADB_REFERENCE/
- 05:13 PM Revision 7635: Added inputs/FIA/_src/FIADB_REFERENCE.ZIP.md5, .url
- 04:11 PM Revision 7634: schemas/vegbien.sql: provider_count_view: Group data by dataprovider and dataset, rather than by each separately, to satisfy the GBIF attribution requirements as described by Brad
- 02:00 PM Revision 7633: schemas/vegbien.sql: analytical_plot/specimen: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
- 01:54 PM Revision 7632: schemas/vegbien.sql: analytical_stem_view: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
- 01:53 PM Revision 7631: schemas/vegbien.sql: sync_analytical_stem_to_view(): Also drop and re-create new dependent views analytical_plot, analytical_specimen
- 01:20 PM Revision 7630: inputs/TEAM/V*/map.csv: Mapped plotArea_ha based on "1ha Plot Number" field name
- 01:12 PM Revision 7629: inputs/TEAM/V*/map.csv: Id: Remapped to individualObservationID
- 01:10 PM Revision 7628: mappings/VegCore.htm: Regenerated from wiki. verbatimScientificName, verbatimTaxonName have been merged into concatenatedScientificName, and ID has been disambiguated.
- 01:09 PM Revision 7627: inputs/*/*/map.csv: Automapped taxonConfidence
- 12:28 PM Revision 7626: schemas/vegbien.sql: analytical_stem_view: plotArea_ha: Fixed bug where need to use parent plot's area when subplot's area is not provided
- 12:18 PM Revision 7625: inputs/SALVIAS/plotMetadata_/create.sql: Added primary key
- 11:42 AM Revision 7624: inputs/NY/NY_DATA_20130212.~.utils.sql: pkey: Made table an enum instead of a text field so that Ecatalog_LatLong could be sorted before Ecatalog
- 11:32 AM Revision 7623: inputs/NY/Ecatalog_all/create.sql: primary key: Combine table, key into one value so that the primary key has a single column. This is necessary because multi-column primary keys are not supported by the import algorithm.
- 11:28 AM Revision 7622: inputs/NY/NY_DATA_20130212.~.utils.sql: Added pkey record type for use by Ecatalog_all's generated pkey
- 10:47 AM Revision 7621: inputs/NY/Ecatalog_all/create.sql: Add table column which scopes the key field. Add primary key to allow sorting by it. inputs/NY/Ecatalog_all/map.csv: Documented that UniqueNYInternalRecordNumber is sometimes duplicated.
- 09:20 AM Revision 7620: make_analytical_db: Wait for tnrs.make to acquire the tnrs.make lock if being run right after import
- 09:13 AM Revision 7619: schemas/vegbien.sql: analytical_stem_view: taxonName_matched: Fixed bug where need to use the matched_taxonverbatim instead of the datasource_taxonverbatim
- 08:58 AM Revision 7618: Moved wait on tnrs.make lock from import_all to make_analytical_db, so that running make_analytical_db for a one-time import also waits on the lock
- 08:56 AM Revision 7617: schemas/vegbien.sql: taxondetermination: taxondetermination_unique: Added determinationtype so that when the matched and accepted determinations are the same, they still both get created rather than the second one being removed due to the unique constraint
- 07:59 AM Revision 7616: schemas/vegbien.sql: analytical_specimen: Removed speciesBinomialWithMorphospecies because it doesn't apply to specimens
- 07:58 AM Revision 7615: schemas/vegbien.sql: Added analytical_plot view
- 07:55 AM Revision 7614: schemas/vegbien.sql: Added analytical_specimen view
- 07:44 AM Revision 7613: schemas/vegbien.sql: analytical_stem_view: Moved recordedBy, recordNumber before dateCollected as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
- 07:40 AM Revision 7612: schemas/vegbien.ERD.mwb: Synced with schema
- 07:38 AM Revision 7611: schemas/vegbien.sql: Added reproductiveCondition
- 07:33 AM Revision 7610: mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition
- 07:28 AM Revision 7609: schemas/vegbien.sql: plantobservation: Added reproductivecondition
- 05:33 AM Revision 7608: mappings/VegCore.htm: Regenerated from wiki. matched*Fit_fraction has been renamed to matched*Confidence_fraction.
- 05:32 AM Revision 7607: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Updated for new mappings/VegCore.htm
- 05:10 AM Revision 7606: inputs/bien_web/observation/map.csv: Re-automapped taxonMorphospecies
- 05:08 AM Revision 7605: mappings/VegCore.htm: Regenerated from wiki. Data owner terms and taxon synonyms have been added, and morphospecies has been disambiguated.
- 04:51 AM Revision 7604: schemas/vegbien.sql: analytical_stem_view: Moved identifiedBy, dateIdentified, identificationRemarks right after the *_verbatim terms that they relate to, as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
- 02:25 AM Revision 7603: schemas/vegbien.sql: analytical_stem_view: Use new concat_delim() instead of array_to_string() surrounded by NULLIF(..., '')
- 02:19 AM Revision 7602: schemas/vegbien.sql: Added concat_delim()
- 01:43 AM Revision 7601: schemas/vegbien.sql: analytical_stem_view: Removed deprecated taxonNameWithMorphospecies now that we have speciesBinomialWithMorphospecies
- 01:17 AM Revision 7600: schemas/vegbien.sql: analytical_stem_view: speciesBinomial: Added morphospecies suffix to create speciesBinomialWithMorphospecies
- 12:15 AM Revision 7599: inputs/import.stats.xls: Updated import times
- 12:06 AM Revision 7598: README.TXT: Full database import: Check that unscrubbed_taxondetermination_view returns no rows: Documented that this takes 90 s with LIMIT 1
02/19/2013
- 11:16 PM Revision 7597: schemas/vegbien.sql: _taxon_family_require_std(): Also allow non-aceae families accepted by TNRS
- 09:38 PM Revision 7596: Added inputs/SALVIAS/_archive/salvias_plots.*.sql.zip.md5
- 09:35 PM Revision 7595: Added inputs/VegBank/_archive/vegbank_for_bien.tar.gz.url
- 09:29 PM Revision 7594: Added inputs/U/UtrechtHerbarium.csv.tar.gz.url
- 09:28 PM Revision 7593: Added inputs/TEAM/_archive/ci-team_extract.tar.gz.url
- 09:27 PM Revision 7592: Added inputs/SpeciesLink/_archive/specieslink*.txt.gz.url
- 09:22 PM Revision 7591: Added inputs/REMIB/_archive/remib_raw.csv.tar.gz.url
- 09:19 PM Revision 7590: Added inputs/NY/NYSpecimenDataAmericas.csv.tar.gz.url
- 09:17 PM Revision 7589: Added inputs/NCU/_archive/NCU-NCSC_2010-02-12.csv.tar.gz.url
- 09:14 PM Revision 7588: Added inputs/MO/mo_digirexport.tar.gz.url
- 09:13 PM Revision 7587: Added inputs/Madidi/_archive/2010-1-2/madidi_plots_original_12jan2010.zip.url
- 09:11 PM Revision 7586: Added inputs/GBIF/gbif_extract.tar.gz.url
- 09:10 PM Revision 7585: Added inputs/FIA/fia_extract.tar.gz.url
- 09:08 PM Revision 7584: Added inputs/CVS/_archive/CVS-allTaxonOccurrences_2010-01-12.txt.tar.gz.url
- 09:04 PM Revision 7583: Added inputs/ARIZ/ARIZ_DiGIR_21012010.csv.tar.gz.url
- 08:55 PM Revision 7582: Added inputs/UNCC/Specimen/UNCC.csv.url, UNCC.csv.md5
- 08:45 PM Revision 7581: Added inputs/XAL/_src/digir.xml.gz.md5
- 08:39 PM Revision 7580: Added inputs/UNCC/_src/ with UNCC.csv.zip.md5
- 08:23 PM Revision 7579: Added inputs/SpeciesLink/_src
Also available in: Atom