Project

General

Profile

  • svn:executable: *

# Date Author Comment
12995 03/30/2014 06:31 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: removed remake_diff_tables() because this is now done for each datasource in inputs/input.Makefile

12990 03/30/2014 06:02 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: removed no longer needed "${public}_validations" schema qualifier, now that it is in the search_path

12391 02/23/2014 08:11 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: added `public_validations.remake_diff_tables()`

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11840 12/05/2013 08:38 AM Aaron Marcuse-Kubitza

bin/make_analytical_db: don't regenerate family_higher_plant_group from the NCBI data because the lookup table is now prepopulated as part of the schema

11434 10/24/2013 05:07 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: removed no longer needed setting of $schema to $public, because this is now done by psql()

11423 10/24/2013 01:11 PM Aaron Marcuse-Kubitza

bugfix: bin/make_analytical_db: when running into a public schema other than "public", also pass this to `/run export_` (which currently uses $schema instead of $public)

10865 09/04/2013 05:27 PM Aaron Marcuse-Kubitza

bugfix: bin/make_analytical_db: `/run export_`: don't take input from the terminal, because this causes rm to prompt the user (from a background task) about overwriting the previous export

9925 06/19/2013 09:43 AM Aaron Marcuse-Kubitza

bin/make_analytical_db: use new mk_table() instead of TRUNCATE/INSERT

9924 06/19/2013 09:41 AM Aaron Marcuse-Kubitza

bin/make_analytical_db: added mk_table() and use it in mk_analytical_table()

9915 06/18/2013 06:22 PM Aaron Marcuse-Kubitza

bin/make_analytical_db: added `/run export_` to make the geoscrub_input CSV export

8320 04/04/2013 04:23 AM Aaron Marcuse-Kubitza

bin/make_analytical_db: Commented out export_analytical_db because we are not yet using the analytical DB in MySQL, and it doesn't make sense to generate a large, unused CSV export each time

8132 03/21/2013 01:58 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.

7864 03/06/2013 08:17 PM Aaron Marcuse-Kubitza

make_analytical_db: Support the $log flag, which toggles output to the terminal or a log file

7620 02/20/2013 09:20 AM Aaron Marcuse-Kubitza

make_analytical_db: Wait for tnrs.make to acquire the tnrs.make lock if being run right after import

7618 02/20/2013 08:58 AM Aaron Marcuse-Kubitza

Moved wait on tnrs.make lock from import_all to make_analytical_db, so that running make_analytical_db for a one-time import also waits on the lock

7345 01/23/2013 08:12 PM Aaron Marcuse-Kubitza

make_analytical_db: Don't run export_analytical_db if the SQL script exits with an error

7061 01/04/2013 10:03 PM Aaron Marcuse-Kubitza

make_analytical_db: ANALYZE each table after its created so that queries use index scans instead of seq scans

7043 01/03/2013 08:25 AM Aaron Marcuse-Kubitza

make_analytical_db: Fixed bug where can't give public_ select access to all analytical_db views because this apparently adds access rather than passing through the underlying table's permissions

7042 01/03/2013 08:18 AM Aaron Marcuse-Kubitza

make_analytical_db: Give public_ select access to analytical_db views. This causes the actual access to depend on the underlying table's permissions.

7031 01/02/2013 08:56 PM Aaron Marcuse-Kubitza

make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_

6981 12/20/2012 10:45 AM Aaron Marcuse-Kubitza

make_analytical_db: Automatically call export_analytical_db when finished

6950 12/19/2012 01:01 PM Aaron Marcuse-Kubitza

vegbien_dest callers: Removed no longer needed explicit setting $prefix to "", because this is now the default value

6908 12/19/2012 04:16 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Merged provider_view, provider_count, and owner_count into provider_count, using the combining query for Brad's data providers page at <http://bien.nceas.ucsb.edu/bien/people/data-providers/>

6895 12/18/2012 09:23 PM Aaron Marcuse-Kubitza

make_analytical_db: Also create taxon_trait materialized view

6869 12/14/2012 12:52 PM Aaron Marcuse-Kubitza

make_analytical_db: Also populate owner_count

6868 12/14/2012 12:51 PM Aaron Marcuse-Kubitza

make_analytical_db: Generate provider_count before analytical_aggregate because it's much faster

6866 12/14/2012 12:21 PM Aaron Marcuse-Kubitza

make_analytical_db: Also populate provider_count

6533 12/03/2012 02:10 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Removed no longer needed darwin_core table. Use analytical_stem instead, which is now identical.

6510 11/30/2012 05:20 PM Aaron Marcuse-Kubitza

make_analytical_db: Added step to create darwin_core materialized view

6466 11/26/2012 01:35 PM Aaron Marcuse-Kubitza

make_analytical_db: mk_analytical_table(): Use explicit schema references everywhere. This fixes a bug where the TRUNCATE/INSERT steps on the public schema's table would reference the analytical_db view instead because they were not schema-scoped.

6465 11/26/2012 01:33 PM Aaron Marcuse-Kubitza

make_analytical_db: mk_analytical_table(): Factored table references in different schemas out into vars

6462 11/25/2012 08:52 PM Aaron Marcuse-Kubitza

make_analytical_db: Moved set -x () around just psql_verbose_vegbien so embedded $() expressions wouldn't also be in set -x (verbose) mode

6461 11/25/2012 08:49 PM Aaron Marcuse-Kubitza

make_analytical_db: Fixed bug where need to use bash instead of sh because vegbien_dest requires it

6460 11/25/2012 08:37 PM Aaron Marcuse-Kubitza

make_analytical_db: Factored analytical_* table creation code out into mk_analytical_table() function

6459 11/25/2012 08:28 PM Aaron Marcuse-Kubitza

make_analytical_db: Create analytical_db views pointing to the analytical_* versions in the public schema

6085 11/08/2012 03:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added threatened_taxonlabel derived table with generating view threatened_taxonlabel_view

6056 11/08/2012 02:18 AM Aaron Marcuse-Kubitza

make_analytical_db: Run all commands synchronously so the log file output doesn't become jumbled

6055 11/08/2012 02:16 AM Aaron Marcuse-Kubitza

make_analytical_db: Fixed bug where log file needed to be appended to instead of overwritten

6054 11/08/2012 02:15 AM Aaron Marcuse-Kubitza

make_analytical_db: Wrap each individual command in `set -x` to avoid echoing low-level commands such as sleep, wait

6053 11/08/2012 02:02 AM Aaron Marcuse-Kubitza

make_analytical_db: Moved log file to inputs/analytical_db/logs/make_analytical_db.log.sql so it would be synced along with the other import logs

6050 11/08/2012 01:49 AM Aaron Marcuse-Kubitza

make_analytical_db: Moved log file to inputs/analytical_db/logs/make_analytical_db.log.sql so it would be synced along with the other import logs

6048 11/08/2012 01:37 AM Aaron Marcuse-Kubitza

make_analytical_db: Output the time at which it's run, so this is included in the log file

6047 11/08/2012 01:36 AM Aaron Marcuse-Kubitza

make_analytical_db: Store output in schemas/make_analytical_db.log

6045 11/08/2012 01:23 AM Aaron Marcuse-Kubitza

make_analytical_db: Use new psql_verbose_vegbien

6043 11/08/2012 01:18 AM Aaron Marcuse-Kubitza

make_analytical_db: Use psql_script_vegbien, which contains the necessary psq options, instead of setting those options manually

6042 11/08/2012 01:15 AM Aaron Marcuse-Kubitza

make_analytical_db: Run the SQL commands directly with psql instead of using the make_analytical_db() function. This provides incremental results and avoids running all commands in one transaction, thus preventing pgAdmin from freezing when the user attempts to access a table used in analytical DB creation (because the TRUNCATE statement fully locks the table until the entire analytical DB is built).

4395 08/31/2012 08:29 PM Aaron Marcuse-Kubitza

Added make_analytical_db