(Unless you’re a certain SME manufacturing company in the South-West of England that couldn’t find a suitable and affordable ERP package for its type of business; well then, developing an open-source one to help you and your fellow SMEs is, it seems, a viable option, resulting in uzERP!)

Many organisations will commission changes to these packages to better fit their way of doing business . In fact, such modifications are a major source of revenue for package providers (and as a result a major cost to those customers who will not “bend” to the packages’ default process flows).

The advice for those who either don’t have the money to commission changes (or who wish to avoid such costs) is to fit in with the package’s way of doing things (which as most will be “industry best practices” can often be a good thing) or use “off-line” processes to handle the exceptions. In some such cases a bespoke professionally delivered software solution might be the result; but for many the more usual “solution” to to those out-of-scope processes is “Simple!, you could easily do that in Excel“. Yet another recruit for the Excel -driven world of Shadow IT.

Whether such “systems” are the casualties of scope warfare or the preferred means of getting things done by Excel-savvy end-users they are part of every organisation’s IT landscape. They’re not going away, yet they receive scant attention from professional IT who often regard them as “somebody else’s problem”.

The end result is sub-optimal; although the business logic encoded in these spreadsheets is often of the highest quality and extremely fit-for-purpose (as it’s the work of those with the greatest insight of the problems at hand) the processes of moving data in and out (and when inside, efficently processing such data) is usually highly time-consuming, manual and error prone. This is where I come in.

I can wrap such “Simple Systems” is a professional “envelope” that will automate the flow of data in, out and within the process while keeping the likely-to-change business logic in Excel. Even when the Excel formula-based business logic needs to be converted to “code” (perhaps for performance or automation reasons) most likely that code will be in VBA (a language familiar to many citizen programmers) which can also be made configurable via a simple Excel “table” rather than some obscure XML dialect.

To get started, send me your existing workbooks (with or without data) with a short note describing the process they support and what you would like to automate or change. I’ll likely respond with some further questions re the process and perhaps made some further suggestions for automation. When we’ve agreed a “deliverable” I’ll get back to you with a fixed price quote.

The idea is to turn “Simple Systems” into “Really Simple System” not works of software art. The main interface will continue to be Excel enhanced by Excel-based tools (mainly my microETL add-in and perhaps add-ins such as PowerPivot for added reporting goodness). Separating data-flow from business logic is usually the major change. It’s also important to be clear what parts of the process need to remain “user-configurable” and which parts are to be “professionally” automated.

The user-configurable parts will need to continue to be Excel-formula based or as code that’s at a level the user is happy to engage with, or as code controlled via configuration tables. The professional parts will be developed by a professional (me!) with the expectation that a professional (i.e somebody proficient in Excel, VBA and SQL) would be required to modify the (always supplied) source code.

Highly configurable solutions are sometimes more expensive than “black boxed” ones, with the most user-configurable option – leave as is – the cheapest (until you take into account the payroll costs of those staff repeating the same easily-automated task over and over again!).

Contact Tom …

Using Pygame with its similar problem domain to that of Scratch would, I figured, make the transition to a grown-up platform easier, and so it has; concepts such as sprite, coordinates, animation etc. are common to both.  I took him through the “Pummel the Chimp” tutorial, expecting his young eyes to glaze over within 10 minutes, but no, a hour later he was still engaged and learning.  Why? He already has a deep understanding of programming, particularly object oriented programming, all thanks to Scratch.

Most of this knowledge he acquired without any help for me, I simply introduced him to Scratch and explained one or two concepts (variables and messages/method calls) which he initially had trouble with, the rest he picked up from looking at other Scratch projects and from writing his own.

So if your kids (or even you) have an itch to learn the essence of programming in a fun and effective way, then Scratch it.

It’s linked to from this page on my blog (which has had 200 or so hits this month) and from this AWS Community Code page. The excessive hits on the bucket started on the 28th of Feb , the day the xlAWS code was published on Amazon and continued through most of March. Talking the size of the zip file, 133GB represents approximately 100,000 downloads. I don’t have server logging enabled on the bucket, so I can’t be sure how much is due to the other public files in the bucket (all belonging to the VBA/Proto SQLite xLite project), but as that project has been available for months and is accessible only through my website (who’s stats show a consistent 5-10 downloads per week) I’m guessing the downloads are for xlAWS.

Who would have though that there would be such interest in VBA/VB6 code for accessing AWS services! I wonder was it the Excel VBA side of the house or the dispossessed (and p*ssed off) VB6 developer hoards who downloaded it the most? Leave a comment if you downloaded and used the library, I’d love to know.

It was going to be called xlS3, but while doing the exercise SimpleDB appeared on the scene, so I decided to try accessing it from Excel, particularly as both products have a lot in common; both “simple”, both “schema-less” data stores. Like the S3Helper code, the simpleDBHelper module is less of comprehensive library, more a collection of useful functions which (hopefully) make working with AWS a bit easier.

To use this code library, you’ll need to have a good grasp of the S3 and SimpleDB APIs and be reasonably proficient with VBA. This is not an end-user tool, it’s for VBA (or VB6) developers. There’s a README and some basic examples within the Excel VBA project to help you get started. Code is released “in the spirit” of LGPL, you can use it how you wish, but if you add something new to the “library” (or find/fix a bug) do let the rest of us know.

As I’ve not been able to find a pure VBA implementation of the HMAC-SHA1 hash algorithm (and I couldn’t see an implementation within the standard “Microsoft Enhanced Cryptographic Provider” ) I’ve wrapped the open source XySSL SHA1 HMAC C code in a VBA friendly DLL. This DLL (and the source, under LGPL) is included in the zip file as AWS authentication requires SHA1 HMAC signatures.

You’ll also obviously require an AWS account. Credentials are stored within the workbook’s custom properties and can be encrypted via a “key file” if required. If you intend to use this code within VB6 (or Proto) you’ll need to provide your own implementation of the AWSKeyData class in order to use a non-Excel persistence store.

You can download the project ZIP file from here.

Have fun.

UPDATE

The simpleDB signature code no longer works as it used Signature Version 1 (which it turns out was insecure), you must now use Signature Version 2. It should be simple enough to change, but as I no longer use SimpleDB I’ve not done so. Also, see below for the many non-DLL methods for calculating HMACSHA1 hashes.

An alternative for calculating HMAC-SHA1 signatures in VBA/VB6 is a Google Checkout supplied COM DLL see http://bit.ly/9CIKtM

There’s the bones of a pure VBA HMAC-SHA1 implementation here http://www.eggheadcafe.com/software/aspnet/32187540/hmac-sha1-challenge.aspx

>>>>>>UPDATE: 17-April -1011

Here’s pure VB6 HMAC-SHA256 implementation it should be easily changed to provide and it now includes a HMAC-SHA1 facility (see comment below)  http://www.vbforums.com/showthread.php?t=635398

<<<<<<<<<

File this under: “Ya learn something new everyday”:

It seems you can access a significant subset of .NET libraries from VB6/VBA/VbScript (including System.Security.Cryptography.HMACSHA1) see this and dotNet in VBScript.

AppJet, funded by Paul Graham‘s Y-Combinator, is very similar to Zimki, being a server-side JavaScript platform. No details yet as to what sort of paid options will be offered (all accounts are free at the moment). Unlike Zimki there’s no plans to create an open-source version. I like the easy “build a Facebook app” feature; and I guess this is the sort of light-weight applications that they hope to attract.

Although Heroku uses Ruby-on-Rails technology, rather than JavaScript, it is closer to the original Zimki idea; but rather than take the hard (and ultimately unsuccessful in Zimki’s case) road of building an open-source platform from scratch, Heroku takes an already popular open-source project and offers it wrapped in a full on-line development and deployment environment. Again, being in beta, there’s no indication as to what pricing model it will operate under, but I would think that it will attract more “serious” projects than AppJet since anything developed under Heroku is pure Rails which means it can be migrated to any other Rails hosting environment; so no lock-in. The online editor is excellent and whatever about its merits as a hosting service it’s by far the easiest way to learn and explore Ruby and Rails, even easier than this…

If Facebook apps are your goal but you wish to use Ruby rather than AppJet’s JavaScript then not to panic, as being Ruby some bright young spark (no, not me I’m afraid) will already have done a lot of the hard graft for you…

Why do I need another scripting language? Well JavaFX Script is a DSL (domain-specific language) and the domain in question is the GUI and as a GUI DSL, JavaFX is impressive.

In my quest for a micro ETL environment the lack of a fast and powerful GUI tool has been a problem. Admittedly, in my Excel/Sqlite xLite suite GUI generation is not a problem as VBA forms are both fast to develop in and feature rich, but the other two options (Java and Ruby) both lack a cost-effective GUI tool (I know there’s Swing and Tk but for somebody who’s used to the speed of development of tools such as VB forms or Oracle’s Application Express, neither appealed). JavaFX changes all that, it’s elegant, powerful, fast to develop in and portable. Further pushes me down the road of using Talend generated Java as my micro-ETL environment of choice.

It’s easy to dismiss JavaFX as hype but the technology behind the hype looks sound. The language borrows powerful features from other languages, object literals from JavaScript, list-comprehensions from Python/Erland/Haskell and first-class functions from JavaScript/Lisp all combined with the full power and glory of Java. The end result:

..is a declarative and statically typed programming language. It has first-class functions, declarative syntax, list-comprehensions, and incremental dependency-based evaluation. The JavaFX Script language makes intensive use of the Java2D swing GUI components and allows for easy creation of GUIs.

Go the community page for FAQs, tutorials and reference information. Timothy O’Brien has a good first pass at calling JavaFX from a Java program and this article by John Smart explains the basics of using ScriptEngineManager to call JavaScript.

More Douglas Crawford JavaScript/DOM related presentations can be found here.

The feature that impressed me most about JavaScript is its functional programming and closure support, a good introduction to the functional programming capabilities of JavaScript can be found here.

Both languages can also transcend their “ugly duckling” status. If you want to take VBA development to a new level have a look at Professional Excel Development by Bullen et al. GMail, Google Docs & Spreadsheets and the other AJAX powered applications of Web2.0 show the power of JavaScript. Both languages are now starting to move beyond their original home bases, particularly in the case of JavaScript.

VBA has been used in the past as the scripting language for corporate applications such as Siebel; now in an inspired move, Proto opted to use VBA as the scripting environment in their corporate focused mashup tool.

JavaScript is however showing the real expansionist streak. Along side its traditional client-side role in web page DOM manipulation, Mozilla Firefox extensions, widgets and within Flash applications under its ActionScript implementation, it can now be found providing the imperative flourish to many client and server side declarative tools (e.g. Kettle ETL, OpenLaszlo and Grazr). You can even use JavaScript to provide the complete server-side logic for a web application/service by using Zimki.(deadpool)

My decision to spend time to improve my JavaScript skills was as a result of my experience with Zimki. I initially created a web service to sign requests for uploading files to my Amazon S3 account; allowing me to give clients a small executable (developed in VBA’s big brother VB6) with which to securely upload data to S3 without the need to expose my Amazon account credentials. This turned out to be so simple and elegant with Zimki’s Rhino based JavaScript environment that I decided this language deserved some quality time. I had I guess, like many others, confused the mess that is the browser-wars scared DOM with JavaScript (or more correctly, EMACScript) the language. Having looked further, I’ve discovered an elegant,simple yet powerful language that I’m tempted to think may become the universal glue-language.

UPDATE:

As Tom from Fotango (the Canon Europe company behind Zimki also in deadpool) points out, Zimki uses Mozilla’s C implementation of JavaScript, SpiderMonkey rather than Rhino.

UPDATE: Sep 2007

Cannon Europe are to close the Zimki service on Dec 24th, 2007

UPDATE: June 2010

See how I combined VBA and JavaScript, to make JavaScript an Excel scripting language http://blog.gobansaor.com/2010/06/04/javascript-as-an-excel-scripting-language-via-jsdb/

Tables, aka datasets, aka datagrids, aka CSV or TSV files and the peoples favourite, the Excel Range, are alive and well and continue to drive most IT based business processes.

Look at this recent comparison of mashup tools over on TechCrunch, if you look at the comparison table only one product uses tables (datagrids) as its “pipe transport protocol” (Input/Output feeds) the rest are all XML variations. The product in question is Proto which is significant as it’s the only mashup tool I know (other than RSSBus which wasn’t on the Techcrunch list) that deals with the current reality of Excel/”behind-the-firewall data sources” while also looking to the near-future where more and more (but not all) data migrates to the cloud.

I not saying that XML is always the wrong choice; RSS, OPML, MicroFormats and the unloved but powerful HTML <table> tag are here to stay and more and more tools offer out-of-the-box parsing for these standards (did you know that Excel can parse a HTML table?). Likewise for formal robust interfaces between systems a bespoke XML protocol can be useful for professional developers. But if a mashup product ignores tables as an input/output format, that product is ignoring the most common business focused data format there is (and the skills of those who day-in day-out manipulate such datasets).

One other line on the TechCrunch table (see how useful tables are) that intrigued me was the assigned level of Advanced to the skill level required to use Proto. There are at least 2 million VBA users out there without any formal programming training who construct corporate internal “mashups” everyday, for that constituency Proto would be by far the easiest tool to use.