My best guess as to one major aspect of Supreme Leader Snoke's back story: He is Yoda's (twin?) brother, father, or son

I am guessing that Supreme Leader Snoke (whose image and voice we hear a few times in the new Star Wars flick) is quite likely a close relative of Yoda's, either his (twin?) brother, his father, or his son -- but even if not a relative, surely of the same species. Of course he is not a giant -- all we see of him in this first movie of the new trilogy is a huge holographic projection which gives no sense of his true scale.

The more I consider this possibility, the more I am convinced that this MUST be what the creators of the new trilogy have in mind (or if it isn't what they have in mind, it SHOULD have been what they had in mind).

Further along this line of thought: Is Maz Kanata also of the same species (perhaps also related to Yoda and his evil relative), and will she play an active counterweight to Snoke in one of the future episodes?

All shall surely be revealed...

Edit (24 hours after original post): Just created a theory posting (linked to this blog posting) on the "Star Wars speculation" subreddit. We'll see whether it generates any comment and further speculation...

Regarding the demotion of Saint Atticus

A couple of weeks ago I finished up a Harper Lee double-header, starting with a reread of her classic, "To Kill a Mockingbird", followed immediately with a reading of its newly-published sequel, "Go Set a Watchman". Before I began reading the sequel, I set out to studiously avoid exposing myself to any of the myriad of opinion pieces that sprang up just after its publication.

It was obvious from the headlines, though, that some folks were aghast at the prospect of Atticus Finch being presented as a "racist" in the newly published book. "Aghast" may be too weak a word: these people seemed REALLY pissed off! Despite my intentions to avoid them, I confess that some of the click-bait headlines lured me in to skim over the text of a few of the angrier jeremiads. (I figured that the biggest plot twist had already been spoiled for me, and I wondered what everybody was so worked up over.) Those that I skimmed did not seem to be expressing anger at Harper Lee herself; instead their ire was directed at Lee's relatives and publishers, as if they had somehow betrayed her (and all the rest of us) by conspiring to make this work public. One op-ed piece that I read, in the New York Times, seemed to go so far as to suggest that "Go Set a Watchman" was actually written BEFORE "To Kill a Mockingbird" -- that "Watchman" was some kind of shoddy first draft that a sagacious editor encouraged Lee to rework into the modern-day classic that we all know and love.

Well, now that I've read the work, I can unequivocally say that "Go Set a Watchman" could NOT have been written before "To Kill a Mockingbird". Major parts of "Watchman" simply make no sense unless they are PRECEDED by the character development (and the crucial first-person impressions of the young narrator, Scout) in "Mockingbird". But what is much more important to me than debunking conspiracy theories is to get to the question of why there would be such a vitriolic reaction to "Go Set a Watchman", which does indeed present, as its central dilemma, how Scout (now mostly called by her grown-up name of "Jean Louise") deals with the realization that her father is NOT the paragon of absolute virtue she had always taken him for. So as I was reading "Go Set a Watchman", I had two overriding questions at the forefront of my mind: (1) Why did Harper Lee take the story of Scout and Atticus in this direction, and (2) why would people be SOOOO upset about this?

Here's my guess: at some point following the publication and brilliant success of her debut novel, "To Kill a Mockingbird", it occurred to Harper Lee (either via a slow dawning of awareness, or perhaps in some thunderclap of sudden, distraught realization) that her book, and in particular her central character of Atticus Finch (very effectively drilled into all of our noggins by Gregory Peck's iconic portrayal), rather than inspiring people to carry on and fight the great fight to broaden and deepen civil rights for all, instead gave a completely unintended solace to the wrong people, inadvertently inspiring a sense of complacency in the very people she intended to spur to action. Horror of horrors, rather than sounding the bugle and heading into battle, some of Lee's target audience of folks-who-wanna-do-right-by-other-folks instead identified so thoroughly with Saint Atticus, that icon of ultimate understanding and benevolence, that their internal monologues might have gone something like this: "I am just like Atticus Finch. I AM Atticus Finch. I have evolved beyond racism. Within me, the American Civil War (which carried on for far too long after Appomattox) has finally arrive at a brilliant and triumphant denouement. The Great Work is now done."

Is it possible that Harper Lee came to just such a realization, and almost simultaneously came to the realization that somehow "Saint Atticus" must die, and be replaced with just plain old Atticus Finch, a real and imperfect human being who's still got some very hard work to do to truly set things right?

I wonder whether much of the ruckus and consternation regarding the publication of "Go Set a Watchman" is actually due to how closely a lot of us (particularly those of the white, privileged persuasion) identify with Atticus. For me, I see the upsetting of the Atticus apple cart to make perfect literary (and Jungian) sense. It HAD to be that way, and Lee does quite a good job of leading Jean Louise (and us) to the realization of her father's true character, in a completely sensible way!! But others may see the dethroning of Atticus to be an unacceptable last straw, a fiction-based confirmation that the brutal, complacency-shattering true events of our times (from Ferguson, to the tragedy of Sandra Bland, and beyond) offer proof-too-positive that the Civil War that we had hoped was over, still rages on. And that's a very difficult thing to have to wake up to.

Atticus Finch, in "To Kill a Mockingbird", despises individual white racists, referring to them explicitly as "trash"; but that same Atticus Finch staunchly supports institutional racism in "Go Set a Watchman". This SO TOTALLY MAKES SENSE -- it is the only direction that Harper Lee could have reasonably taken the narrative, and it makes this pair of books, with their fuller realization of the story of Scout and Atticus, into a powerful allegory for the challenges of our time.

A "Hello World" for HBase (a brief tutorial Java program)

As I continue my research and development efforts in the general vicinity of HBase, it's become clear to me how difficult it is to find good introductory resources -- even something as simple as a "Hello World" program -- to serve as the launchpad for a newbie's explorations into the workings of the HBase client API. Not finding any such thing (at least nothing that was usable), I decided to cook up my own.

Without further ado, here it is...

(Use the horizontal scroll bar down at the bottom of the frame, or you might find it easier to view on its original page on Github.)

	/**
	* This brief HELLO WORLD Java program is meant to enable you to very quickly
	* gain a rudimentary, hands-on understanding of how data (and metadata) is
	* stored and retrieved in HBase via the "client API".
	*
	* PART 1: CONCEPTS
	* ================
	* For those coming to the HBase world with previous experience in traditional
	* RDBMS databases, it is essential to realize that Tables, Rows, and Columns
	* in HBase, while bearing some resemblance to their namesakes in the RDBMS
	* world, differ markedly in their structures and functionality.
	*
	* **Column Families**
	* As you can see in the code below, when you use the Admin#createTable method,
	* besides providing a TableName, you also must specify at least one "Column
	* Family" (denoted in the code by the class HColumnDescriptor).
	* In HBase, Columns are all grouped by Column Family, with all Columns in a
	* family being physically stored together. Theoretically, you could have a
	* large number of Column Families, but the present HBase architecture
	* actually has a practical limitation of no more than three or four per Table.
	*
	* **Versioning**
	* In the code below, a "maxVersions" value of 3 is assigned to the
	* Column Family, which means that versioning has been enabled for all Columns
	* in the family: when a Column is updated, the 2 most recent *previous* values
	* for that Column are still retrievable, each designated by a timestamp.
	* These individual versioned instances are sometimes referred to as the Cells
	* of a Column. The retrieval of multiple versions (Cells) of the same Column is
	* performed below in the #getAndPrintAllCellVersions method.
	*
	* **Columns**
	* It is important to note (in the most striking departure from RDBMS norms)
	* that Columns themselves are NOT part of the Table definition. Columns are
	* "defined" on-the-fly as each row is <put> (i.e., inserted/updated) into the
	* database. There is also NO datatyping of each Column: HBase accepts any
	* byte-array of any length/format you wish to store in any Column. This means
	* that IT IS UP TO THE INDIVIDUAL APPLICATION TO MANAGE AND ENFORCE COLUMN
	* NAMES AND DATATYPES. In the RDBMS world, the database (i.e., database
	* administrator) manages column metadata; in the HBase world, the application
	* (i.e., application designer/programmer) manages column metadata.
	*
	* **Rows**
	* Rows are inserted, accessed, and physically ordered exclusively by Row ID
	* (the conceptual equivalent of an RDBMS primary key). When a "scan" is
	* performed to access multiple contiguous rows, those rows will always be
	* returned in Row ID order (either ascending or descending).
	*
	* PART 2: WORKING WITH THIS CODE
	* ==============================
	* Importantly, your ability to run this code requires that you have successfully
	* installed and started a standalone implementation of HBase on the machine
	* on which this program is to be run.
	*
	* The recommended steps to take to run this program are:
	* (1) Install a "standalone" configuration of the current stable release of
	* HBase on your machine following the instructions provided at:
	* https://hbase.apache.org/book.html#quickstart
	* (If you are installing on a Windows machine, it is strongly recommended
	* that you NOT bother trying to do an installation using the documented
	* Cygwin option [which has proven to be faulty and is apparently not
	* kept up-to-date with new releases of HBase], but instead install and
	* run a virtual Unix environment [e.g., Ubuntu] in a virtual machine
	* such as VirtualBox, and install HBase in that environment.)
	* (2) Copy this code into a new project in your favorite IDE, set up the
	* CLASSPATH as documented below, and use this code as your launchpad into
	* effective utilization of the HBase Client API. Run and modify this code
	* as extensively as you need to in order to build and deepen your
	* understanding of how to store and retrieve data (and metadata!) in HBase.
	* Refer to the HBase javadocs ( https://hbase.apache.org/apidocs/ )
	* to extend this code and explore functionality not demonstrated in the
	* code below.
	*
	* This code was developed in coordination with HBase release 1.0.1.1;
	* compatibility with subsequent releases is hoped for, but by no means
	* guaranteed.
	*
	* PART 3: CLASSPATH DETAILS
	* =========================
	* To fulfill CLASSPATH requirements to compile/run this program:
	* -- the CLASSPATH must include the directory in which hbase-site.xml (i.e.,
	* the HBase startup parameters file) is stored for your currently-running
	* instance of HBase (e.g., '/usr/local/hbase/hbase-1.0.1.1/conf').
	* [In NetBeans, this would be set in Project Properties/Libraries/Run.]
	* -- the CLASSPATH should also include the HBase library (e.g. "HBase_1.0.1.1"
	* [In NetBeans, you can include this library in your project's
	* "Compile-time Libraries" list.]
	*/
	package org.prettygoodexamples.hellohbase;
	import java.io.IOException;
	import java.util.Map.Entry;
	import java.util.NavigableMap;
	import org.apache.hadoop.hbase.HColumnDescriptor;
	import org.apache.hadoop.hbase.HTableDescriptor;
	import org.apache.hadoop.hbase.NamespaceDescriptor;
	import org.apache.hadoop.hbase.NamespaceNotFoundException;
	import org.apache.hadoop.hbase.TableName;
	import org.apache.hadoop.hbase.client.Admin;
	import org.apache.hadoop.hbase.client.Connection;
	import org.apache.hadoop.hbase.client.ConnectionFactory;
	import org.apache.hadoop.hbase.client.Delete;
	import org.apache.hadoop.hbase.client.Get;
	import org.apache.hadoop.hbase.client.Put;
	import org.apache.hadoop.hbase.client.Result;
	import org.apache.hadoop.hbase.client.Table;
	import org.apache.hadoop.hbase.util.Bytes;
	/**
	* Successful running of this application requires access to an active instance
	* of HBase. For install instructions for a standalone instance of HBase, please
	* refer to https://hbase.apache.org/book.html#quickstart
	*/
	public final class HelloHBase {
	protected static final String MY_NAMESPACE_NAME = "myTestNamespace";
	static final TableName MY_TABLE_NAME = TableName.valueOf("myTestTable");
	static final byte[] MY_COLUMN_FAMILY_NAME = Bytes.toBytes("cf");
	static final byte[] MY_FIRST_COLUMN_QUALIFIER
	= Bytes.toBytes("myFirstColumn");
	static final byte[] MY_SECOND_COLUMN_QUALIFIER
	= Bytes.toBytes("mySecondColumn");
	static final byte[] MY_ROW_ID = Bytes.toBytes("rowId01");
	public static void main(final String[] args) throws IOException {
	final boolean deleteAllAtEOJ = true;
	/**
	* ConnectionFactory#createConnection() automatically looks for
	* hbase-site.xml (HBase configuration parameters) on the system's
	* CLASSPATH, to enable creation of Connection to HBase via Zookeeper.
	*/
	try (Connection connection = ConnectionFactory.createConnection();
	Admin admin = connection.getAdmin()) {
	admin.getClusterStatus(); // assure connection successfully established
	System.out.println("\n*** Hello HBase! -- Connection has been "
	+ "established via Zookeeper!!\n");
	createNamespaceAndTable(admin);
	System.out.println("Getting a Table object for [" + MY_TABLE_NAME
	+ "] with which to perform CRUD operations in HBase.");
	try (Table table = connection.getTable(MY_TABLE_NAME)) {
	putRowToTable(table);
	getAndPrintRowContents(table);
	if (deleteAllAtEOJ) {
	deleteRow(table);
	}
	}
	if (deleteAllAtEOJ) {
	deleteNamespaceAndTable(admin);
	}
	}
	}
	/**
	* Invokes Admin#createNamespace and Admin#createTable to create a namespace
	* with a table that has one column-family.
	*
	* @param admin Standard Admin object
	* @throws IOException If IO problem encountered
	*/
	static void createNamespaceAndTable(final Admin admin) throws IOException {
	if (!namespaceExists(admin, MY_NAMESPACE_NAME)) {
	System.out.println("Creating Namespace [" + MY_NAMESPACE_NAME + "].");
	admin.createNamespace(NamespaceDescriptor
	.create(MY_NAMESPACE_NAME).build());
	}
	if (!admin.tableExists(MY_TABLE_NAME)) {
	System.out.println("Creating Table [" + MY_TABLE_NAME.getNameAsString()
	+ "], with one Column Family ["
	+ Bytes.toString(MY_COLUMN_FAMILY_NAME) + "].");
	admin.createTable(new HTableDescriptor(MY_TABLE_NAME)
	.addFamily(new HColumnDescriptor(MY_COLUMN_FAMILY_NAME)));
	}
	}
	/**
	* Invokes Table#put to store a row (with two new columns created 'on the
	* fly') into the table.
	*
	* @param table Standard Table object (used for CRUD operations).
	* @throws IOException If IO problem encountered
	*/
	static void putRowToTable(final Table table) throws IOException {
	table.put(new Put(MY_ROW_ID).addColumn(MY_COLUMN_FAMILY_NAME,
	MY_FIRST_COLUMN_QUALIFIER,
	Bytes.toBytes("Hello")).addColumn(MY_COLUMN_FAMILY_NAME,
	MY_SECOND_COLUMN_QUALIFIER,
	Bytes.toBytes("World!")));
	System.out.println("Row [" + Bytes.toString(MY_ROW_ID)
	+ "] was put into Table ["
	+ table.getName().getNameAsString() + "] in HBase;\n"
	+ " the row's two columns (created 'on the fly') are: ["
	+ Bytes.toString(MY_COLUMN_FAMILY_NAME) + ":"
	+ Bytes.toString(MY_FIRST_COLUMN_QUALIFIER)
	+ "] and [" + Bytes.toString(MY_COLUMN_FAMILY_NAME) + ":"
	+ Bytes.toString(MY_SECOND_COLUMN_QUALIFIER) + "]");
	}
	/**
	* Invokes Table#get and prints out the contents of the retrieved row.
	*
	* @param table Standard Table object
	* @throws IOException If IO problem encountered
	*/
	static void getAndPrintRowContents(final Table table) throws IOException {
	Result row = table.get(new Get(MY_ROW_ID));
	System.out.println("Row [" + Bytes.toString(row.getRow())
	+ "] was retrieved from Table ["
	+ table.getName().getNameAsString()
	+ "] in HBase, with the following content:");
	for (Entry<byte[], NavigableMap<byte[], byte[]>> colFamilyEntry
	: row.getNoVersionMap().entrySet()) {
	String columnFamilyName = Bytes.toString(colFamilyEntry.getKey());
	System.out.println(" Columns in Column Family [" + columnFamilyName
	+ "]:");
	for (Entry<byte[], byte[]> columnNameAndValueMap
	: colFamilyEntry.getValue().entrySet()) {
	System.out.println(" Value of Column [" + columnFamilyName + ":"
	+ Bytes.toString(columnNameAndValueMap.getKey()) + "] == "
	+ Bytes.toString(columnNameAndValueMap.getValue()));
	}
	}
	}
	/**
	* Checks to see whether a namespace exists.
	*
	* @param admin Standard Admin object
	* @param namespaceName Name of namespace
	* @return true If namespace exists
	* @throws IOException If IO problem encountered
	*/
	static boolean namespaceExists(final Admin admin, final String namespaceName)
	throws IOException {
	try {
	admin.getNamespaceDescriptor(namespaceName);
	} catch (NamespaceNotFoundException e) {
	return false;
	}
	return true;
	}
	/**
	* Invokes Table#delete to delete test data (i.e. the row)
	*
	* @param table Standard Table object
	* @throws IOException If IO problem is encountered
	*/
	static void deleteRow(final Table table) throws IOException {
	System.out.println("Deleting row [" + Bytes.toString(MY_ROW_ID)
	+ "] from Table ["
	+ table.getName().getNameAsString() + "].");
	table.delete(new Delete(MY_ROW_ID));
	}
	/**
	* Invokes Admin#disableTable, Admin#deleteTable, and Admin#deleteNamespace to
	* disable/delete Table and delete Namespace.
	*
	* @param admin Standard Admin object
	* @throws IOException If IO problem is encountered
	*/
	static void deleteNamespaceAndTable(final Admin admin) throws IOException {
	if (admin.tableExists(MY_TABLE_NAME)) {
	System.out.println("Disabling/deleting Table ["
	+ MY_TABLE_NAME.getNameAsString() + "].");
	admin.disableTable(MY_TABLE_NAME); // Disable a table before deleting it.
	admin.deleteTable(MY_TABLE_NAME);
	}
	if (namespaceExists(admin, MY_NAMESPACE_NAME)) {
	System.out.println("Deleting Namespace [" + MY_NAMESPACE_NAME + "].");
	admin.deleteNamespace(MY_NAMESPACE_NAME);
	}
	}
	}

view raw HelloWorldForHBase.java hosted with ❤ by GitHub

	2016-04-15 09:43:32,381 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
	2016-04-15 09:43:32,974 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x704a52ec connecting to ZooKeeper ensemble=localhost:2181
	2016-04-15 09:43:32,984 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
	2016-04-15 09:43:32,985 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=ubuntuVM
	2016-04-15 09:43:32,986 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_45
	2016-04-15 09:43:32,986 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
	2016-04-15 09:43:32,986 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/home/dv/jdk1.8.0_45/jre
	2016-04-15 09:43:32,986 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/dv/NetBeansProjects/column-manager/target/classes:/home/dv/.m2/repository/org/apache/hbase/hbase-common/1.0.1.1/hbase-common-1.0.1.1.jar:/home/dv/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/dv/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/dv/.m2/repository/org/apache/hbase/hbase-protocol/1.0.1.1/hbase-protocol-1.0.1.1.jar:/home/dv/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/dv/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/home/dv/.m2/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/home/dv/.m2/repository/asm/asm/3.1/asm-3.1.jar:/home/dv/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/dv/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/dv/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/home/dv/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/dv/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/home/dv/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/home/dv/.m2/repository/org/slf4j/slf4j-log4j12/1.7.7/slf4j-log4j12-1.7.7.jar:/home/dv/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-common/2.5.1/hadoop-common-2.5.1.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-annotations/2.5.1/hadoop-annotations-2.5.1.jar:/home/dv/.m2/repository/org/apache/commons/commons-math3/3.1.1/commons-math3-3.1.1.jar:/home/dv/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/dv/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/home/dv/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/dv/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/dv/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/dv/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/home/dv/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/home/dv/.m2/repository/net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar:/home/dv/.m2/repository/org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar:/home/dv/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/home/dv/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/dv/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/dv/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/dv/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/dv/.m2/repository/org/apache/avro/avro/1.7.4/avro-1.7.4.jar:/home/dv/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/home/dv/.m2/repository/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar:/home/dv/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/home/dv/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/dv/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/home/dv/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.jar:/home/dv/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/home/dv/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/home/dv/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/home/dv/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/home/dv/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.5.1/hadoop-yarn-api-2.5.1.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.5.1/hadoop-mapreduce-client-core-2.5.1.jar:/home/dv/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.5.1/hadoop-yarn-common-2.5.1.jar:/home/dv/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/home/dv/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/home/dv/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/dv/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/home/dv/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/home/dv/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/home/dv/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/dv/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.9/jersey-guice-1.9.jar:/home/dv/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/home/dv/.m2/repository/io/netty/netty/3.6.2.Final/netty-3.6.2.Final.jar:/home/dv/.m2/repository/com/github/stephenc/findbugs/findbugs-annotations/1.3.9-1/findbugs-annotations-1.3.9-1.jar:/home/dv/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/dv/.m2/repository/org/apache/hbase/hbase-client/1.0.1.1/hbase-client-1.0.1.1.jar:/home/dv/.m2/repository/org/apache/hbase/hbase-annotations/1.0.1.1/hbase-annotations-1.0.1.1.jar:/home/dv/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/dv/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/dv/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/dv/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/home/dv/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar:/home/dv/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/dv/.m2/repository/io/netty/netty-all/4.0.23.Final/netty-all-4.0.23.Final.jar:/home/dv/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/home/dv/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/home/dv/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/dv/.m2/repository/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.jar:/home/dv/.m2/repository/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar:/home/dv/.m2/repository/org/commonvox/collections/ordered-set/1.0.0/ordered-set-1.0.0.jar
	2016-04-15 09:43:32,993 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/dv/jdk1.8.0_45/jre/lib/amd64:/home/dv/jdk1.8.0_45/jre/lib/i386::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.13.0-85-generic
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=dv
	2016-04-15 09:43:32,994 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/dv
	2016-04-15 09:43:32,995 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/dv/NetBeansProjects/column-manager
	2016-04-15 09:43:32,996 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x704a52ec0x0, quorum=localhost:2181, baseZNode=/hbase
	2016-04-15 09:43:33,032 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
	2016-04-15 09:43:33,049 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
	2016-04-15 09:43:33,079 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x154175b89b60008, negotiated timeout = 40000
	*** Hello HBase! -- Connection has been established via Zookeeper!!
	Creating Table [myTestTable], with one Column Family [cf].
	Getting a Table object for [myTestTable] with which to perform CRUD operations in HBase.
	Row [rowId01] was put into Table [myTestTable] in HBase;
	the row's two columns (created 'on the fly') are: [cf:myFirstColumn] and [cf:mySecondColumn]
	Row [rowId01] was retrieved from Table [myTestTable] in HBase, with the following content:
	Columns in Column Family [cf]:
	Value of Column [cf:myFirstColumn] == Hello
	Value of Column [cf:mySecondColumn] == World!
	Deleting row [rowId01] from Table [myTestTable].
	Disabling/deleting Table [myTestTable].
	2016-04-15 09:43:35,002 INFO [main] client.HBaseAdmin: Started disable of myTestTable
	2016-04-15 09:43:36,252 INFO [main] client.HBaseAdmin: Disabled myTestTable
	2016-04-15 09:43:36,536 INFO [main] client.HBaseAdmin: Deleted myTestTable
	Deleting Namespace [myTestNamespace].
	2016-04-15 09:43:36,582 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
	2016-04-15 09:43:36,584 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x154175b89b60008
	2016-04-15 09:43:36,591 INFO [main] zookeeper.ZooKeeper: Session: 0x154175b89b60008 closed
	2016-04-15 09:43:36,591 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down

view raw output.txt hosted with ❤ by GitHub

Chaos Wrangler project, Phase 1: Coming up with a simplified picture of the HBase "Data Model"

Public domain image - Wikipedia, Jason Hise

It's only been a few days since I've launched my "Chaos Wrangler" project -- to see what I can do to help bring HBase metadata into the realm of controllability (or even just plain knowability) for enterprises who are beginning to use HBase as the back-end repository for their "Big Data" projects!

While my mind is already racing far down the road, envisioning what would be required to create both passive and active metadata discovery mechanisms and repositories to plug into HBase, it is vital that I take some time in these early days to be sure that I thoroughly understand the structures of HBase (both physical and conceptual). While I did find a useful resource or two, I found nothing that boiled things down into terms that I felt I could fully internalize, so I decided to trudge through the main reference guide on the Apache HBase site and derive for myself a simplified picture of what the HBase data model consists of and how it works.

As part of this exploratory process, I've begun writing a Java-based "analogy" of the HBase data model. I'm taking the concepts that are presented and discussed in the reference guide (e.g., "table", "column prefix", "cell", etc.) and building them into a small set of interrelated Java classes. This is proving useful for my understanding, and when I publish the code soon to gist.github.com/dvimont, my hope is that it might help to explain HBase to anybody who has prior understanding of basic Java and RDBMS concepts.

EDIT 2015-06-04: The code snippet mentioned in the preceding sentence is now available on Github. 

In the comments above my Java code, I began to write a few explanatory paragraphs, which soon included a character-based diagram of my conception of the HBase data model. Here are those comments, including the character-based diagram. The comments stand alone, apart from the Java code, as my first offering to the HBase community from the Chaos Wrangler project. My explanation is provided as a potential complement to more rigorous and comprehensive expositions of the HBase data model.

===============
The following is a Java-oriented analogy to the structures that comprise the HBase data model. It is presented under the assumption that the reader has a basic mastery of both Java and RDBMS concepts.

The task of coming to an understanding of HBase data structures is unfortunately made much more difficult than it otherwise might be, due to the fact that HBase uses the terms TABLE, COLUMN, and ROW, but the structures that are referred to by these names bear little resemblance to their RDBMS namesakes.

In fact, an HBase ROW is completely misnamed to the extent that the common term "row" is usually associated with a one-dimensional container of singular instances of data, whether it be a row in a spreadsheet or a row in an RDBMS table. In stark contrast, an HBase ROW could be conceived of as a container of an array of arrays of arrays!! (No wonder people run into difficulty understanding HBase data structures!)

Here is a conceptual representation of the hierarchy of structures in an HBase table. (Note that all relationships shown below are one-to-many -- e.g., one TABLE contains multiple COLUMN_FAMILY_PREFIX instances, one CELL contains multiple CELL_ENTRY instances, etc.):

A TABLE and its component COLUMN_FAMILY_PREFIXes are immutably defined when the TABLE is created. (While a TABLE could theoretically have an unlimited number of COLUMN_FAMILY_PREFIXes, the official reference says that the physical realities of the HBase architecture enforce a practical maximum of no more than two or three per TABLE!)  All lower-level constructs (from ROW on down) are created and maintained by an application via the HBase "put" method. Thus, crucially, all of these lower-level constructs (including so-called COLUMN_QUALIFIERs) are treated as application-managed DATA, and not as database-managed METADATA!!

There are a number of immediately apparent ramifications of this "shifting" of column-name-maintenance (not to mention column-datatype-maintenance) from the metadata-realm to the data-realm: not only does this potentially upend certain basic assumptions about application development and metadata management, but it could also call for a redefining/realignment of classical roles within IT organizations, which may otherwise be accustomed to an explicit separation of duties between DBAs (who are the traditional custodians of metadata) and developers (who are traditionally responsible for building applications that manipulate only data - not metadata!).

Yes Mr. Bloor, there IS a Hadoop "metadata mess", but it ain't "waiting in the wings" -- IT'S HERE (and here's what I intend to do about it)

I've spent the last week getting a great introduction to Hadoop technologies by studying the excellent book, HADOOP: THE DEFINITIVE GUIDE, by Tom White. At this point on my trip up the learning curve, I'm both impressed and distressed:

• impressed by the straightforward architecture Hadoop (and the technology stack built upon it) uses to manage both distributed data storage and data processing, but...
• distressed by the apparent lack of any means of documenting or even passively discovering fundamental information about the data being stored. 

It boils down to an apparent absence of any technology to serve as either an active or passive metadata repository (data dictionary, to use the old parlance).

Unfortunately, it may well be the case that a majority of the individuals who have bothered to read this far don't even know I'm talking about, perhaps not even being sure what "metadata" means in this context and why anyone would be worried about the lack of a means to manage it. If that's where you find yourself, please take a couple of minutes and read Robin Bloor's concise and clear exposition of the problem in his October 2013 analysis piece "Hadoop: Is There a Metadata Mess Waiting in the Wings?"; then please come on back here and read my remaining few paragraphs -- it's okay, I'll wait.

Given that the majority of my IT career up until 2008 was spent in the creation of tools for metadata extraction, transformation, and mapping between disparate architectures, I am particularly horrified at the state of metadata management in HBase (a distributed database that rides atop Hadoop and HDFS structures, and is somewhat reminiscent of ADABAS in its internal structures). Not only is it apparent that no tools exist to discover and maintain a directory to the fundamental structures of any HBase table (i.e., its columns), but there appears to be no awareness that such a thing is even needed (much less that it is, in my opinion, supremely vital). The only mention of such things that I've been able to find on the Web so far is in a posting to a HortonWorks Q&A page, in which someone very sensibly asks how one might obtain a list of the columns in an HBase table. They are very succinctly informed by one of HortonWork's experts that it simply CANNOT BE DONE!!

Yes, at this point in the evolution of HBase, apparently you can put anything you want to into it (in huge, exceedingly diverse quantities), but when the next person comes along and wants to utilize the data you've put there, you'd better be sure to hand them all those cocktail napkins upon which you jotted down the names and structures of the columns in your HBase tables. Otherwise you may be looking at a future where all those exabytes of data may just have to sit there as folks idly stare at it, scratch their heads, and wonder "what the hell is all this?".

I say all this by way of explanation of my soon-to-commence R&D project: to build a passive data dictionary for HBase which will discover the names (and where feasible, the structures) of the columns in any given HBase table, storing the results in (of course!) an HBase table. My working name of this open-source project is "Chaos Wrangler".

EDIT 2016-07-17: The open-source project, ColumnManager for HBase, is now in beta-02, with binaries and documentation available both on GitHub (where the project is housed) and via the Maven Central Repository!

My entry to the KHAN ACADEMY TALENT SEARCH

Here is the entry I just submitted to the Khan Academy Talent Search...

What subject do your videos teach, and what level do they target?

Long division, 3-12 math

Video #1

Video #2

Why are you interested in sharing your videos with the world through Khan Academy?

Usually I measure success in teaching on a fairly personal level, when I'm communicating concepts to an individual or a small group, and I can see the glint of recognition in human eyes (sometimes prefaced by the crinkled brow of confusion) right before me. But YouTube and Khan Academy give another way to find success in educating people. While I could just measure it in the raw "number of views" that a tutorial video gets, with YouTube I can also get direct feedback from people, sometimes with students asking for clarification, or requesting that I do another video on another topic that they're learning about. (See the comments section on my "Long Division With a Two-Digit Divisor" video to see what I'm talking about.) My presumption is that getting my content out on Khan Academy as well as YouTube would heighten the possibilities for this kind of interaction.

But here's what I really LOVE about this medium of learning: People come to get it when THEY need it, at the precise moment that they WANT to. Hungry, self-directed minds like these are the most open and receptive to a concise and intriguing explanation that can quickly lead to that "a-ha, now I get it" moment!

Is there anything else you'd like us to know?

The first video that I'm submitting, "Long Division With a Two Digit Divisor," was literally made for a single classroom-full of Grade 4 students that I was then working with. I posted it publicly on YouTube because I figured it might be of benefit to somebody else besides my students. After it started getting lots of views I decided to make two "prequels" explaining the basics of long division (one of which is my second submitted video, "Why Long Division Works").

In further-flung-yet-curiously-related matters:

• I've had lots of experience engaging adult learners in technical topics related to systems integration software that I've developed (http://treehouse.com/tReDPS.shtml).
• More recently I've produced a number of audiobooks, excerpts of which you can peruse on my YouTube channel (https://www.youtube.com/dvimont) and my website (http://commonvox.org).
• I'm also currently playing several roles in "radio play" style productions of a number of Shakespeare plays with a new group called THE ONLINE STAGE, and am composing music and serving as remote vocal coach to the voice actors in our production of THE TEMPEST: https://soundcloud.com/commonvox/sets/shakespeare-tunes

======================

Edit 2015-05-26:

Here's a cocktail napkin sketch of some ideas for projects I'd like to work on in the future (with folks like those at Khan Academy, or on my own) --

• Numeracy project = learning resources for self-tutoring in numeracy skills -- geared toward adults who are at the very lower end of the numeracy spectrum. Would need to be built with an eye to overcoming (or steering around) the fears that drive the innumerate to remain innumerate. (We dare not underestimate the overwhelming, self-defeating power of the "I'm no good at math" mantra.) Goal: elimination of innumeracy in the adult population.
• Literacy project = same as numeracy project, but with focus on literacy.
• Object Oriented version of MIT’s Scratch = for learners of all ages, a game-building mechanism like MIT’s Scratch, but one that is built upon valid Object-Oriented (OO) principles and which has a direct path into working with genuine, mature programming languages in a real IDE (integrated development environment). ScratchOO must get junior programmers working with valid OO concepts from day one of playing in the environment. Given that the path leads toward mature programming in a real IDE, perhaps the learning environment should be available as a plug-in to an IDE (like Eclipse or NetBeans). If, for example, ScratchOO were built on top of Java, then all the ScratchOO tools and widgets would be built in Java. The beginner would work with a simple set of these tools to build games, but as they advance, then can get into the actual Java code that lies "behind" the simple tools they've been working with, to customize and extend them, and become fully competent Java programmers.

Two new open source Java projects launched on GitHub & Sourceforge

In the world of Software Engineering, I've heard it said that "code is the new resume". 

If that's so, then I've just burnished up two fresh software engineering "CVs" and posted them to GitHub and SourceForge.

IndexedCollection: A Java class library which extends the standard Java collections framework to provide the IndexedCollection class, which offers simple, automatic, in-memory NoSQL composite-indexing of a standard Collection of objects. In many situations it could be a nice alternative (or a simpler complement) to complex ORM implementations, making for higher efficiency and lower TCO in the development and maintenance of Java applications. 
Github: https://github.com/dvimont/IndexedCollection 
SourceForge: https://sourceforge.net/projects/indexedcollection/ 
Gist (simple usage example code): https://gist.github.com/dvimont/4f0e11c79efb775bc893 
Side note -- I've been trying to benchmark my IndexedCollection against another package called CQEngine, but unfortunately that other package's documentation is so sparse, I can't even figure out how to use it for the simplest of queries. The sample code that is published for it won't run against the current release, and both the explanatory text on Google Code and its Javadocs documentation are also out-of-sync with the actual code! So, from a usability perspective, the project seems to have been abandoned, even though the package itself was updated as recently as early 2014. Oh well...

LibriVox Explorer: From a technical point of view, this Windows/Mac/Linux desktop application makes extensive use of the above-mentioned IndexedCollection Java class library, providing the best currently-available example of how IndexedCollections are intended to work. From the end-user viewpoint, LibriVox Explorer is a new way to experience the LibriVox collection of Public Domain audiobooks, and it takes full advantage of the "eye candy" potential of that collection's vast body of intriguing cover-art.
Download it now, and take it for a spin on your desktop:
http://commonvox.org/le-download 
Github: https://github.com/dvimont/LibriVoxExplorer 
SourceForge: https://sourceforge.net/projects/librivoxexplorer/