It’s linked to from this page on my blog (which has had 200 or so hits this month) and from this AWS Community Code page. The excessive hits on the bucket started on the 28th of Feb , the day the xlAWS code was published on Amazon and continued through most of March. Talking the size of the zip file, 133GB represents approximately 100,000 downloads. I don’t have server logging enabled on the bucket, so I can’t be sure how much is due to the other public files in the bucket (all belonging to the VBA/Proto SQLite xLite project), but as that project has been available for months and is accessible only through my website (who’s stats show a consistent 5-10 downloads per week) I’m guessing the downloads are for xlAWS.

Who would have though that there would be such interest in VBA/VB6 code for accessing AWS services! I wonder was it the Excel VBA side of the house or the dispossessed (and p*ssed off) VB6 developer hoards who downloaded it the most? Leave a comment if you downloaded and used the library, I’d love to know.

The “free for non-commercial purposes” product is no more, but you can still download a 30-day free trial and the really good news is the new price of $300 for the Proto Individual product.

Also, another business-oriented mash-up tool, RSSBus, is partially out of beta, RSSBus Server RC1 is now available. Still no information on final pricing for the server product. A new Beta 5 release of the free desktop edition is also available for download. As I’ve said before, RSSBus Server front-ended by Proto would be a very powerful combination.

But it is the use of SQLite as the client-side persistence engine that excites this datasmith’s old heart. Since first coming across SQLite (while learning Ruby on Rails) I’ve been convinced that this “good enough” micro-database on steroids was a winner. Since then, as well as using it with Ruby ,I’ve integrated it with excel to use as a micro-ETL tool, I was instrumental in getting SQLite support added to Kettle (Pentaho PDI), I wrote a SQLite custom module for Proto, in fact the first thing I now do when checking out a data manipulation tool is to check if it supports SQLite.

I’m looking forward to investigating further what Google Gears can add to my existing datasmithing tool set, and I’m certainly glad I invested the time in learning JavaScript. Exciting times.

This confirmation of Google’s continued commitment to D&S is opportune as I had decided a few weeks ago to invest some time finally getting my head around GData, in particular, the Spreadsheets and Calendar APIs. I investigated the APIs using VBA, as my main interest in interfacing with GoogleD&S & Calendar is as a back-end to Excel; i.e. using Excel as the front-end heavy duty data tool and Google Apps as the collaboration and orchestration tool (and Amazon S3 as the back-end data store). But I didn’t use VBA within Excel, instead as I was also investigating Proto I used this excellent Proto example as my starting point.

NOTE:

Proto Discount:

Byron Binkley CEO of Proto has kindly offered a total of 50 licenses (until July 13th 2007) at $79.50, a 90% discount to the listed price for Proto Individual (http://www.protosw.com/products/purchase). The offer applies to anybody mentioning this http://gobansaor.wordpress.com/2007/04/13/proto/blog post. I have no connection financial or otherwise with Proto Software, just a fan of the product.

RSSBus

Proto

So what to build for my Proto debut? Well looking at Proto one of the things that struck me was the lack of a relational join feature. In a previous post I explained that other mashup tools tend to use XML as their “pipe transport protocol” rather the table-oriented nature of EntrySets. But these entrysets cannot be manipulated using relational operators, not even the ability to join two tables by a shared key. Although the tool doesn’t offer this facility it does offer the flexibility to create a component to solve the problem and somebody has already done so; simulating an Excel VLOOKUP “join”. But I wanted the full power of a relational engine, INNER and OUTER joins, UNION and MINUS, aggregations via GROUP BY, access to very large datasets and I already had the answer; an Excel VBA module, xLite.

To say that xLite is a standard generic piece of code would be to exaggerate, it is like most operational VBA code, a template of ideas that are manipulated from project to project to serve the needs of the day. It had started life as an excel-to-Oracle tool (when it was called xORA), morphed into a SAP master data extraction tool (by the name of xSAP), finally getting a bit of structure when I discovered SQLite as the ultimate desktop database (having occasionally being seduced by that grand old dame, MS Access). Its further transformation into a Proto module required more renovation work; replacing Excel Range objects with variant arrays and replacing the excellent but COM based LiteX SQLite wrapper with the simpler but more flexible PSSQLITE.DLL thus allowing registry-free deployment.

By why SQLite, why not MS Access or SQLServer? See “What’s So Good about SSQLite” for why I use SQlite but in this case two reasons stand out;

• Manifest Typing:

SQlite doesn’t enforce column based types (i.e. VARCHARS,TEXT,INTEGER etc.) like Excel (more or less) it’s the content of the cell that determines its type – SQLite calls this manifest typing. This allows excel Ranges and Proto entrysets to be filtered through a relational join without type-conversion errors arising but at the same time allowing the database to recognise types where useful. For example;

Month	SalesThisMonth
Jan07	1000
Feb07	2000
Mar07	Accounts not available!!!
Total	3000

Excel and SQLite will both give a Total of 3000 for SalesThisMonth column, most databases would choke on the accounts-not-available value.

• In-Memory database:

SQLite does not need to be “installed” before it is used. Like MS Access the data resides in a single file but unlike MS Access even that single file can be dispensed with by using a “:memory:” database as I do within xLite.

The transformation from Excel to Proto VBA for both myself as the programmer and the code as the programmed proved to be trivial and rewarding. In this occasion I did most of the work and testing from the Excel IDE as I hadn’t yet fully comprehended the event firing mechanisms within Proto but I think in future I might develop and test under Proto and then port the code to Excel.

So any negatives?

With the development experience any negatives were generally to do with my inexperience with the platform (e.g. the above mentioned event firing causing me to “hang” the program once or twice) or features such as non-Excel “asset” management which are, I believe, on the product’s roadmap (e.g. I have to download my DLLs from the web rather than including them in the .PROTO manifest file as I would prefer).

My main two “problems” with Proto are not technical as such but market positioning misgivings:

• Treating Excel as the add-on;

Proto will not run without Excel being installed. Excel is the prime piece of data-manipulating real estate on users’ desktops. They trust it and depend on it. It would be much easier to “sell” Proto to end-users if Proto was marketed and packaged as an add-on to Excel (see for example Palo OLAP ).

• The Price;

Pricing software products is not easy (see this if you’ve never given the subject any consideration) and the last thing the good folks at Proto need is somebody on the other side of the ocean telling them they need to charge less per seat for their software. But I think they do if they intend to sell this software to individuals rather than to corporate IT purchasing departments. By individuals I mean the SMB owners/accountants/salesfolks, the large company departmental power-users, the freelance ‘data smiths’ like myself. Selling software into SMBs is not the same as selling to large companies (for some of the problems see http://supplychainventures.typepad.com/my_weblog/2006/04/tackling_the_sm.html) but equally selling something “new” into large company IT departments has in many cases become impossible unless that “new” product comes with a MS,Oracle,IBM or SAP badge on it.

But let’s not end on a negative, with its feet firmly on the ground where business computing currently operates (i.e Windows/MS Office, behind-the-firewall company intranets) but with an eye to the opening up of IT to the economics of the wider internet, Proto is one to watch.

UPDATE:

Byron Binkley CEO of Proto has pointed out in the comments below that the current dependency on Excel is temporary and will be removed in a future version. But he also reponds to my pricing concerns by offering ….

….a total of 50 licenses over the next 90 days (through July 13th 2007) at $79.50, a 90% discount to the listed price for Proto Individual (http://www.protosw.com/products/purchase)…

…this offer applies to anybody mentioning this blog post. This is pretty typical of my dealings to date with this young company; you raise a problem and they respond with a solution. Takes all the fun out of complaining.

Both languages can also transcend their “ugly duckling” status. If you want to take VBA development to a new level have a look at Professional Excel Development by Bullen et al. GMail, Google Docs & Spreadsheets and the other AJAX powered applications of Web2.0 show the power of JavaScript. Both languages are now starting to move beyond their original home bases, particularly in the case of JavaScript.

VBA has been used in the past as the scripting language for corporate applications such as Siebel; now in an inspired move, Proto opted to use VBA as the scripting environment in their corporate focused mashup tool.

JavaScript is however showing the real expansionist streak. Along side its traditional client-side role in web page DOM manipulation, Mozilla Firefox extensions, widgets and within Flash applications under its ActionScript implementation, it can now be found providing the imperative flourish to many client and server side declarative tools (e.g. Kettle ETL, OpenLaszlo and Grazr). You can even use JavaScript to provide the complete server-side logic for a web application/service by using Zimki.(deadpool)

My decision to spend time to improve my JavaScript skills was as a result of my experience with Zimki. I initially created a web service to sign requests for uploading files to my Amazon S3 account; allowing me to give clients a small executable (developed in VBA’s big brother VB6) with which to securely upload data to S3 without the need to expose my Amazon account credentials. This turned out to be so simple and elegant with Zimki’s Rhino based JavaScript environment that I decided this language deserved some quality time. I had I guess, like many others, confused the mess that is the browser-wars scared DOM with JavaScript (or more correctly, EMACScript) the language. Having looked further, I’ve discovered an elegant,simple yet powerful language that I’m tempted to think may become the universal glue-language.

UPDATE:

As Tom from Fotango (the Canon Europe company behind Zimki also in deadpool) points out, Zimki uses Mozilla’s C implementation of JavaScript, SpiderMonkey rather than Rhino.

UPDATE: Sep 2007

Cannon Europe are to close the Zimki service on Dec 24th, 2007

Another potential partnership that the folks at RSSBus should investigate is a link up withProto.  Both products are firmly positioned behind the corporate firewall with their emphasis on commercial data sources i.e. – dare I say it – mashup tools for adults. 

Proto is visually appealing (like Yahoo Pipes) and is Excel centric in both its table-based design philosphy and its choice of macro language, VBA; both sound design decisions in the market the product is aiming at – the power user.   It is however, lacking in some of the ‘heaving lifting’ tools that power-users depend on but are usually supplied to them by IT; this is where RSSBus comes in.  With its XML scripting language and its extensibility capability provided by C#/VB.NET its natural user base is the corporate IT developer.  RSSBus’s impressive list of pre-built connectors and the ease with which developers can add new ones, makes it an ideal partner to Proto’s more approachable user-focused front-end.