VBA is now a one trick pony (Office automation), but Python opens all sorts of API scripting and product automation doors. My use of IronPython as HAMMER’s scripting language is one such door, a door to the otherwise mainly-closed-to-VBA world of multi-threading and easy .NET library integration.

The HAMMER function can be called from VBA using the Application.Run command like so…

retArray = Application.Run(“DATASMITH.HAMMER”,inArray,”Select dept,count(*) from table1″,”SQL”)

… the 1st parameter is the function name, parameter two is the 1st argument to the function, parameter three is the 2nd and so on.

By utilising HAMMER’s IronPython functionality (requires NET 4.0 runtime), VBA routines can access the full power of the .NET platform with data passed back and forth using tables. Admittedly for many complex .NET utilities utilising VB.NET or C# may be a better approach (due to better IDE and debugging features of such languages) but for standard library calls, IronPython is an ideal option. It also has the benefit that the “code” can be stored within the workbook.

HAMMER also offers the power of multi-threading to VBA via its internal threading functionality (requires Excel >= 2007 and Net4.0 runtime). The multi-threading example in the hammerThreads.xlsx workbook could easily be wrapped in VBA code, perhaps to allow it to be controlled by a user-form.

I’ve added two new commands specially designed for use within VBA scripted HAMMER scenarios:

• APPDB – Opens an application-wide shared in-memory database. This will allow tables (and Python objects) created in one function call to be accessible in an other function call (assuming both issue the APPDB command as their 1st command). This replicates the functionality of microETL which by default exposes a application-wide SQLite in-memory instance and a common Python workspace.
• CLOSEAPPDB – This will close and clear the shared c#-SQLite and IronPython instances. Equivalent of microETL’s xLiteReset() function.

Be careful not to…

• use the APPDB instance from in-cell UDF calls to HAMMER that are likely to be scheduled as multi-threaded (the helper functions HammerToSheet & HammerToFit are safe, as they are always single-threaded),
• or use within “internal threaded” HAMMER commands

…as although c#-SQLite is thread safe, the implementation logic is not.

Here’s a list of the HAMMER commands implemented so far …

You can download the latest version of HAMMER here …

The saying “If you only have a hammer, you tend to see every problem as a nail“ could be rephrased as ”If you only have a datasmith’s HAMMER, you tend to see every problem as a table“!. But that’s OK, as nearly every commercial IT problem can be expressed as a set of tables, and in any case, HAMMER is not intended to be a datasmith’s only tool. Its close integration with Excel, recognises the prime importance of spreadsheets to most data wranglers and its use of CSV & the SQLite database format as its persistence & transport mechanism, (à la “SQLite as the MP3 of data“) recognises that datasets will often need to be shared with other tools.

Finding the perfect tool for the job it an IT obsession that most of our customer’s care little for; it’s only when our choice of tool affects their bottom-line (i.e excessive cost or wasted time) that end users take much notice of the backroom technology. The most important skill for a datasmith, is the ability to understand the structures and forms that data can take and to be able to derive and build business value from that data. The technology is an aid, nothing more, technology (and indeed  applications) will come and go, data lives on.

HAMMER encapsulates the  minimum set of technical skills an aspiring datasmith should learn:

• Spreadsheets, in particular Excel. No need to know every nook and cranny, nor expect it to handle every task. But do learn about pivot tables and array formulas; if you have Excel >= 2007, learn about Excel  tables (aka Lists). If you have Excel 2010, make sure to download PowerPivot. Become comfortable with “formula programming”, don’t expect  a ribbon command to solve every problem.
• SQL – learn the basics of selecting, joining and group-by; again no need to become a master; SQLite is not only an excellent way to learn SQL, it’s also a powerful tool to have once you’ve picked up the basics.
• Learn a scripting language –  Python is one of the best and is relatively easy to learn. Again, mastery is nice, but not essential, learn the basics of IF-THEN-ELSE logic, loops and iterations, array and list handling and string manipulation.  Your efforts at coding do not have to be pretty or elegant, just functional. Python skills will also transfer across platforms, CPython (the original and the best), IronPython (.NET and HAMMER) and Jython (JVM, here’s an cool example of Python as a scripting language to automate GUIs).

All this talk of picking the right tools brings to mind the old golf pro-am story where the amateur was constantly pointing out to the pro (Jack Nicklaus I think) what club to play. At a particularly tricky shot to the green, the pro had enough when his choice of club was again “criticised”. So, he proceeded to take out every club including woods & putters, placed a ball for each one and hit every ball on to the green.

We’re not all as talented as Jack Nicklaus, so having at least an good enough tool for the job at hand  is important. But it does show, that focusing on the end-game is what matters, not becoming fixated with a  particular way of doing things.

Enough of the moralising, and talking of being fixated on a particular tool  ;) here’s this week’s list of new features to be found in HAMMER:

New commands:

TXT – will load a text file such that each line is a new row in a single columned (column named “text”) table.

LOADDB – like OPENDB, opens the named database but only if the database file exists (OPENDB will create a new empty database if file not found). Intended primarily as end-point to request-process-response loop, see DELEGATE/SUBMIT  below.

SAVEDB – saves the current “main” database using the previous argument as it’s name. (shorthand for … “main”,”filename.db”,”SAVENAMEDDB”).

SUBMIT – same as SAVEDB, but the value of the prior argument has a “.request” appended to make a file name in the format “databasename.request” for saving to. Also, if the previous argument = “” or “{GUID}” will generated a globally unique name using  a GUID. The main use-case for this command is to send the saved database for processing by an external program, maybe a Talend or Keetle ETL job , or a HAMMER.EXE job.

DELEGATE – same a SUBMIT, but doesn’t do any processing (i.e. all commands other than DELEGATE, which is expected to be the last command, are ignored), instead it’ll package the request in the saved database with the expectation that an external HAMMER.EXE, or another Excel-based HAMMER call, with do the processing.

Changes and helper functions to support DELEGATE/SUBMIT processing:

The DELEGATE and SUBMIT commands are how HAMMER implements its version of steam-powered servers.

The intention is that databases named with extensions of “.request” are picked up and processed by “servers” waiting for such files. (The transport of such files between server and the served is a separate issue, might be simply a matter of placing files on a shared drives, or DropBox!) Theses servers may then populate that database with new data ( or not, e.g. might generate a CSV). When finished, a text file of the same name but with the “.request” replaced by a “.response” is written to indicate that processing is complete.

Both the Excel udf HAMMER and HAMMER.EXE (non-excel command line version of UDF function), have been changed such that when a single argument is passed, that argument is expected to be a DELEGATE generated “.request” database. The database’s “pickled” arguments and data will then unpacked and the “delegated request” processed.

HAMMER.exe, if started with no arguments, implements a simple DELEGATE “server”, i.e. will wait for “.requests” files in its default folder and process each file as above.

Three RTD helper functions have been added to HAMMER udf:

waitOnHammerResponse(database (without the .response),[folder name]) – will return “Waiting” until it finds a database.response file, will then return the original .request filename (i.e. not the response file as it’s simply a marker, the database.request will  contain original data and any new data generated).

waitOnHammerRequest(database (without the .request),[folder name]) – will return “Waiting” until it finds a database.request file, will then then return the request filename.

waitOnFile(filenaname,[folder name]) – like the functions above but without the response/request expectations.

Here’s a list of the HAMMER commands implemented so far …

Get the latest version of HAMMER here …

Fact (aka transactional data) can of course be handled, particularly when already reduced or when by-nature low volume, such facts will in many cases fit easily in-memory.

But when facts start to run into the millions of records, traditional in-memory manipulation becomes a problem. Obviously such large volumes datasets should in the first instance be handled IT-side utilising tools that are designed to handle such volumes, i.e. enterprise-class databases. But sometimes data, even large transactional databases, are “thrown-over-the-wall” with limited support offered (or accepted). But again, there’s plenty of cut-down versions of enterprise RDBMS available (SQL Express, Oracle Express etc.) plus the FOSS offerings such as MySQL or PostgresSQL, that can be configured user-side to help tame these beasts.

If you’re using PowerPivot, you have another option, PowerPivot itself. With its ability to quickly load and compress large volumes of data and its ability to perform many data cleansing tasks by means of row-context DAX formulas, often that’s all that will be required.

The typical problems that a transactional dataset can throw up, such as data split over two tables (header and detail) or needing to replace a surrogate date key with an actual date (to enable certain DAX date functionality to work) can easily be fixed within PowerPivot.

One thing to note about fact data presented as a Header-Detail set, is that traditional star-schema design requires that such data be flattened to the lower “grain” of the detail line, but PowerPivot doesn’t actually require you to do that. Some dimensions can link to the header (example Customer on Invoice Header) and others to the line (example Product on Invoice Line). The detail line table is still the “hub” of the “star” but one of its dimensions (the header table) is its route to a multiple other dimensions. Not classic star schema design, but it’ll work , and good for quick and dirty analysis (might be situations though where things might not pan-out as expected see this, perhaps best stick with pure star-schemas for complex work).

There’ll come a time however, when you’ll be faced with the problem of manipulating large datasets outside of traditional  RDBMS servers and outside of PowerPivot. Combining sets of data from multiple sources, as in my previous post, would be a prime example. Such projects often operate on a “need-to-know-basis” often with those supplying the data ‘outside the loop’. Today’s additions to HAMMER should help.

Three new commands, ATTACHDB, ATTACHDBINMEMORY and SAVENAMEDB, will allow external disk-based databases to be attached to the default HAMMER database.

ATTACHDB requires a filename  followed by an alias name for the attached db. Having an attached external database would allow, for example, a large fact table in CSV format to be loaded (and indexed) without touching memory. This could also be done using the previously introduced OPENDB command, but the benefit of ATTACHDB is that other non-memory-threatening processing can continue to take place in-memory.

The ATTACHDBINMEMORY also attaches an external database, but this time loads it into memory, so any changes made will not be automatically persisted back to disk. To do that, use the SAVENAMEDDB command.

This requires an attached database alias, followed by the file name to save the database to. SAVEASNAMEDB has other uses such as making backups or making copies of data for use in an external debugging platform (it can be much easier to debug Python using a proper IDE).

Along side the facility to load data via CSV/TSV, I’ve also added an ADO command. This requires a valid ADO connection string, followed by either a table/view name or a SQL Select statement. It uses ADODB 2.7, to enable handling of modern Access file formats, I’ll eventually make an ADO.NET version to remove this dependency.

Finally, I’ve managed to get a DBAPI compliant SQLite provider working in the PYTHON step. The provider is called XLite (it’s a modified version of Jeff Hardy’s SQLite3 library) and exposes most of the same functionality as CPython’s SQlite3 provider.

The library can open external SQLite databases, so offering another means of accessing non in-memory data and accesses the HAMMER default databases via the pre-connected pyDB variable. Having the ability to lazy-load rows via a cursor loop is also very useful in reducing memory foot-print when dealing with large tables. (see the IrelandFOIExample_hammer_test2.xlsx for an example of using XLite).

Update:

I’ve removed the dependency on a specific ADODB library (figured out how do equivalent of CreateObject in .NET for COM libraries). Also included is a first pass at a command line version (hammer.exe). Example:

hammer.exe mydata.csv CSV myotherdata.csv CSV JOIN > new.csv

hammer.exe inv.csv “CSV” “sum(qty)” REDUCE sum.csv TOCSV

Here’s a list of the HAMMER commands implemented so far …

Download  the latest version of HAMMER from here …

To be honest, when I started, my aim was to simply port  a lot of my useful framework code to .NET as I’m doing more and more work in pure .NET these days.

It’s a lot easier to do so with a project driving the work rather than the drudgery of line by line conversions simply for the sake of conversion.

But over the last few months of investigating the process of moving some of more useful code over to .NET; discovering C#-SQLite as a near drop-in replacement for SQLite; realising that IronPython 2.7 marks the coming-of-age of Python under .NET; discovering the ease of multi-threading programming under Net 4.o; I began to see the potential of building a tool more suited to this new emerging world of technological plenty; of 64bit, multi-core high-RAM machines playing host to ground-breaking tools such as PowerPivot, than the 32bit, single-threaded  world that microETL was born into.

I wanted to bring microETL back to its roots as an in-cell function. To make it sit more easily with the spreadsheet functional style of programming, a style of programming that has been more successful at attracting “civilians” to programming that any other method. To make the tool more in tune with the Excel-way of doing things.

At the same time, I also wanted the tool to be “detachable” from Excel, so that it could perform ETL functions server-side without the need for Excel. Ideally capable of being run from the command-line from a single no-install-required executable.

And so, the Datasmith’s HAMMER was born.

So where does this fit in with PowerPivot? When I design something like HAMMER I normally craft it around a serious of “use cases”. Below would be a one of my PowerPivot & HAMMER  use cases:

The Data Quality Team

The sales & marketing department of a wholesaling company have discovered PowerPivot and are excited about using it to help manage an upcoming merger with a recently purchased competitor, also in the wholesaling business. There’s a large overlap between customers and products supplied, and the sales structures, pricing and KPIs used by both companies are very different. The transition phase from two separate companies to a single operating unit will need detailed  planning and constant monitoring.  Oh, and there’s also the problem of the brand new ERP system that’ll replace the existing systems currently in use.

The group can see how PowerPivot models will help with this task but are worried about the sourcing and managing the data. They decide to appoint a small data quality team, with a mixture of IT and business experience; which will be responsible for mapping old  to old,old to new, managing the data deltas (business will still goes on, new customers, new products etc.).

Most of this work revolves around “master data”. Transactional data may be much larger in volume, but get the master data right then transactional data will be easy to handle as long as the processing capacity to handle the volumes is available (which thanks to PowerPivot and really fast high-RAM PCs it is).

In golf, there’s a saying “You drive for show, but you putt for dough”. Likewise in data: a lot of emphasis is put on the big-data transactional datastores, but real analytical benefit comes from the less sexier master-data, aka the dimensions. As a result the most important output from the data quality team will be a set of conformed dimensions.

Each week (and at month-end) the team will get extracts of the master and transactional data in various formats from both legacy systems and will also get the WIP master datasets from the team working on the new ERP system. From these they’ll construct point-in-time conformed-dimensions combined old with old and old with new; and will rebuild transactional feeds with new conformed keys alongside existing keys. These datatsets are then sent to the various analysis teams to enable them build Excel and PowerPivot models that will hopefully all “sing from the same hymn sheet”.

And how will this be accomplished? With the datasmith’s HAMMER of course (well it is my day-dream use-case!).  Both in-Excel and command-line HAMMER “scripts” will be used to wrangle the datasets and the complete sets packaged as SQLite database files.

When the analysis groups receive these files, they too will use HAMMER to extract the datasets they require (either straight to CSV for loading into PowerPivot) or into Excel for some further transformation work prior to modelling. To handle slowing-changing dimension scenarios, many of the teams will squirrel away each week’s data in other to be able to model yet-unknown-but-very-likely changes of organisational structures.

Although this is a use-case, it’s based on reality, a reality that would have been so much easier to manage if I, and the teams I worked with, had access to technology such as HAMMER and PowerPivot.

So, the version that the setup.xls will install if it detects a sub-2007 version of Excel, will offer SQL and noSQL (JOIN,UNION etc.) but no Python or multi-threading.

So what about new features? Excel being the original noSQL  database, I continue to add more noSQL commands for those who wish to avoid SQL or find its syntax somewhat long-winded. The JOIN  & LOJOIN (outer join) commands are good examples, simply load two tables with the column names that you wish to join on, sharing the same names, simple.  Another example is the REDUCE (aka GROUPBY aka DISTINCT) command I’ve added this version. It essentially performs a SELECT … FROM … GROUP BY; again load or generate a table, then follow with a list of the columns you wish to ‘reduce’ the table by, plus any aggregates you wish to perform. Examples:

• =HAMMER(myHugeList,”dept,sum(overtime)”,”REDUCE”)
• =HAMMER(AccessLogs!A1:C9999,”areaAccessed,byWhom”,”REDUCE”)
• =HAMMER(invHead,invLine,”JOIN”,”count(invID),sum(netAmt)”,”REDUCE”)

If noSQL is not your cup of tea and you wish to utilise the full power of a SQL database; a new command “OPENDB” will allow you to open an existing (or create a new) SQLite database file. This will allow SQLite data sources to be accessed and written to via standard in-cell formula, no VBA required! The command expects the previous argument to be the database name. If no such argument exists it will create a temporary on-disk database. This command usually only makes sense as the 1st command as it’ll close and wipe any previously opened databases. If no “OPENDB” command is issued (i.e. the default) an in-memory database (aka :memory:)  is used . Examples:

• =HAMMER(“C:\data\myDB.db”,”OPENDB”,A10:C:9910) will copy the data for range A10:C9910 and save in a table called table3 in the myDB.db SSQLite database.
• =HAMMER(“C:\data\myDB.db”,”OPENDB”,”SELECT * from table3″) will fetch the same data back into Excel.

Wow steady on, what if there’s a need to store or fetch data from disk without using SQLite? No problem, use the “TOCSV” command, outputs the last table loaded or generated in CSV format to the file name specified. (There’s also a “SQLTOCVS” command which expects a SQL statement to specify the data to extract followed by the file name to extract to).

Two other commands “CSV” and “TSV” will load comma and tab separated data into HAMMER.  Although the CSV functionality is useful within Excel, the main driver for these command is to enable HAMMER to work outside Excel as a command-line data processor; you heard it here first folks!

I’ve also added the 1st set of my helper functions, these two functions are only available in the 2007/2010 versions as they use multi-threading. The two functions are:

• hammerToFit – wraps HAMMER, but will auto-resize the array area (or create a brand new array-selection if none) to fit the returned table. Note: to achieve this, the HAMMER function will be called twice if the existing array area needs adjusting.
• hammerToSheet – again wraps HAMMER, but will paste the resulting table to a new sheet.

Although both helper functions utilise threads to achieve these little tricks (hence they’re not available sub-2007) when HAMMER functionality is called via these wrappers the function operates as a single threaded function – there’s a good reason for this which I’ll explain some other time. Internal HAMMER threading does however still work.

Here’s a list of the HAMMER commands implemented so far …

Download the latest version of HAMMER here …

I’ve been intending for some time to build an offshoot of microETL that would be less powerful but perhaps more approachable and have fewer moving parts. This week I finished it, still needs some more testing, but the basic product is in place. It’s a single file add-in (a .xll) , called DATASMITH. At its heart is a single function called HAMMER, there will be other helper functions that in the main will wrap the HAMMER function, but in essence it is the datasmith’s HAMMER.

If microETL is a datasmith’s forge or indeed a mirco-foundry,HAMMER is a datasmith’s everyday portable tool (with perhaps Excel as the anvil, and your CPUs as the fire?). Talking of CPUs; multi-core CPUs are now the norm and since version 2007, Excel can utilise such multi-cores. MicroETL being VBA-based cannot however take advantage of this, to do so requires an .xll add-in; another reason to build the HAMMER.

So what will this new functionality look like:

Examples:

=HAMMER(Invoices[#All],InvoiceLine[#All],”JOIN”)

…will take the two ranges (you need the [#All] to pick-up the header and data sections of a 2007/2010 Excel Table) and join them using the columns with the same name as the join fields. The JOIN command expects the last two preceding arguments to be tables (aka arrays with a header line).

=HAMMER(“SALES”,DeptSales!A1:F2101,”SELECT * from table2 where dept=’:1′”,”SQL”)

… the 1st argument is loaded as Arg(1) (:1 in SQL), the 2nd argument is loaded (being an array) into a table named table2; if they were the other way round, it would be table1 and Arg(2). The 3rd argument is loaded as Arg(3) and the 4th is a command: SQL. SQL looks back at the preceding argument (Arg(3) in this case) and executes its contents as SQLite SQL. If the preceding argument was a table, it would expect to find a list of SQL statements for execution in the first column. The output of the last issued SELECT statement is then returned to Excel as an array.

=HAMMER(DeptTargets!A1:D20,DeptSales!A1:F2101,SalesTagetScript,”PYTHON”)

…this is similar to the previous SQL example but this time the command is PYTHON which will execute the Python Script passed in via the command’s preceding argument. The script will most likely return a table to Excel, but it could also, if not the last argument, create a table associated with its position, in this case table4, which could then be accessed by subsequent PYTHON or SQL scripts.

Up to 25 arguments can be passed, the last table produced is returned to Excel (either via a load, which wouldn’t be terribly useful, or more likely as a result of a command such as JOIN, SQL or PYTHON). HAMMER functions can of course be nested and can also issue “internal” HAMMER requests. The “flow” of commands is from left to right, with the preceding args usually setting the stage for subsequent commands. Alongside the all-powerful SQL and PYTHON commands, I’ll most likely add a set of “noSQL” offerings such as JOIN, LOJOIN (left-outer), DISTINCT, REDUCE (a SELECT .. GROUP BY… with PYTHON as the MAP?), UNION, INTERSECT. These will likely also be available through helper functions such as =JOIN(ThisTable,ThatTable).

Unlike microETL, there’s no persistence across function calls i.e. HAMMER will play by Excel’s Functions no-side-effects rule. Each call to a function will build up and tear down its in-memory SQLite environment (including calls to HAMMER from within PYTHON).(UPDATE: Specify the 1st COMMAND as “APPDB” to simulate microETL’s persistence across function calls)  Likewise each call to PYTHON will be a separate engine instance. Likewise each call to a HAMMER function will create its own PYTHON engine. Tables  and Python artefacts created by prior “steps” in a single HAMMER call are available to subsequent steps.

HAMMER embeds Python under the guise of IronPython, so a lot of the power and speed of CPython will not be available, but on the other hand, the full power of the .NET CLR will be, not a bad swap.

And a huge advantage, HAMMER will be an asynchronous function (i.e. will run in its own thread). This will allow multiple long running transforms to be handled within the same Excel instance, a major shortcoming of microETL. This requires Excel 2007 or 2010, but will still work synchronously for Excel 97-2003. Having said that, there’s a requirement for Net4.o for IronPython so this add-in is more suited for modern versions of Excel and for OS >= XP SP3.

HAMMER is an array function, yeah I know, normal folk tend to steer clear of Excel arrays. Which is to be expected, they’re not the most intuitive end-user-facing construct that the industry has ever come up with. But, they are the “Excel way” for passing tables in and out of formulas and, once mastered, open up a new world of power to Excel users. I will be adding helper functions to make using arrays a bit easier (an autoSize wrapper function, that’ll resize the selection area if it’s too small or too big, at the cost of a 2nd pass at the enclosed functions, might also port microETL’s SQL “paste” functionality).

I had intended to have an example to download with this post, but have discovered a last-minute bug that needs fixing. So it’ll be most likely next week before I’ve a version to show.

UPDATE:

I’ve managed to sort out the bug, so here’s an example…

http://bit.ly/datasmith

Here’s a list of the HAMMER commands implemented so far …

Use setup.xls to install (or simply open in Excel), see IrelandFOIexample_hammer (2007/2010) for examples (there’s also a copy of the same functionality via microETL in IrelandFOIexample_microETL.xls – note the speed difference).  There’s Excel 2007/2010 32bit and  Excel 2010 64bit versions included and there’s also an un-tested 2003 version.

The existing xLiteLoadUnLoad(filename[,unload]) function loads a SQLite file into “main” i.e. the primary database (which is usually a :memory: db) overwriting any existing data. Michael’s suggestion was to allow loading into another in-memory database with a different alias; thus keeping the main database intact but allowing the benefits of in-memory access to the externally attached database. This feature has now been added.

I’ve kept the existing xLiteLoadUnLoad as is, but added a new optional argument to the xLiteAttachDB function so.

•  xliteAttachDB(databaseName,alias)  becomes xliteAttachDB(databaseName,alias,[loadInMemory=False]). The optional loadInMemory argument defaults to FALSE, so acts like the old version (i.e. issues a standard SQLite Attach statement). But if set to TRUE; the function will first Attach a “:memory: database” named as the alias, then will load the external database file into that in-memory database. Once this happens the on-disk database is not referenced, so any changes will not be reflected back to disk. To enable changes to be persisted to disk, I’ve added another new function…
• xLiteDBSaveAs(alias,outDatabaseFile) will save a copy of the database named alias (with could be “main” if you wished to backup the default in-memory database) to the file outDatabaseFile. I’ve also added a …
• xLiteDetachDB(alias) to issue a SQLite DETACH statement. You might ask why not simply use the SQL() function to issue DETACH (or indeed ATTACH) statements? Statements such as ATTACH/DETACH cannot be issued by the SQL() functions as its pre-processor (for table() functionality) wraps SQL statements in a SAVEPOINT (nested SQL transaction). You can however use the fastSQL() or xliteRawSQL() functions to issue such commands.

There’s another (this time breaking) change to the SQLScript TIMER  (see here …) command. The existing function used an ActiveX control (the Internet Explorer control as a provider for JavaScript timer functionality); ActiveX controls do not work under 64bit Excel, so I’ve reverted back to using Application.OnTime as my timer mechanism. The breaking change is the 3rd argument, which previously expected a value indicating the number of thousands-of-a-second to wait, now it represents whole seconds.

To download the latest version see the http://www.gobansaor.com/microetl page.

Update:

For another method of loading SQLite databases within Excel/VBA see my new .NET-centric micro ETL tool  http://blog.gobansaor.com/category/hammer/

Within the microETL project, VBA routines can gain access to the App instance via the gAppDB global variable (gHelperDB for the Helper instance) while Python routines use the global dictionary element connDict(“App”) or connDict(“xlite”) for the App database (and connDict(“Helper”) for the Helper).

SQLScript (and in-cell formulas) can gain access to the Helper instance via the xLiteSQL(inSQL,outRange,NoHeader,PasteOver,useHelperDB) command, by setting useHelperDB to True. (The SQL() function is a thin-wrapper on this function to handle positional substitution tokens (:1, :2 …), the main functionality,such as the table() preprocessor commands and output to a range, are provided by xLiteSQL.)

These default in-memory databases are blank databases that are  instantiated in memory at workbook start-up and any tables or data created during the workbook session are discarded at shutdown. The standard use-case assumes that any data to be saved or fetched will come from Excel or perhaps via external CSVs or corporate database connections as would be the case with a normal Excel workbook, i.e. the main use for SQLite is assumed to be a provider of a SQL layer on top of Excel not a primary data-store.

However, SQLite is a superb data-store, one which is used by 1000s of applications (quite likely many of apps on your smart-phone or iPad use SQLite as their database). Its ACID nature and small foot-print makes it an ideal means of storing client-side information that would otherwise need to be communicated to a “proper” server-side database such as MySQL or SQL Server.

MicroETL supports the use of disk-based databases for both the main App and companion Helper instances via:

• xliteAttachDB(databaseName,alias); this command will attach a database to the App database and assign it an alias. This is a wrapper on the SQLite Attach command  http://www.sqlite.org/lang_attach.html. Note: the SQL Attach command can not be used via the SQL() command, but can be used by the “raw” fastSQL() function.
• xLiteLoadUnLoad(filename[,unload]); this function is both useful & dangerous, it will replace the main (App) database’s contents with the contents of the SQLite file provided by the first parameter. Typically the target database is the default empty in-memory (:memory:) database so no harm done; but if the default database was a file-based database, whose contents you wished to preserve, this might not be what you wanted. The second parameter if set to TRUE (FALSE being the default) works in the opposite direction i.e. replaces the specified database’s contents with those of the main (App) database.
• Setting a custom property of xLiteDB to a SQLite filename in a driver workbook, this database will then be used as the default App database. As well as file based databases, the xLiteDB property can be set to ”:temp:” or “:memory:”, :temp: is like :memory: but will use RAM & a temporary file while :memory: only uses RAM. The driver workbook is the first workbook to initiate microETL, so be careful of  multiple workbooks with different App “expectations” being accessed in the same Excel instance.
• The similar custom property of xLiteHelperDB controls what Helper (i.e. Logging) database to use.
• Another option is to use the custom properties xLiteAttachDB & xLiteAttachDBAlias to auto-attach a database at startup
• Yet another method is provided by the custom properties xLiteDBSeed and xLiteHelperDBSeed. Does the the equivalent of xLiteLoadUnLoad(databaseName,FALSE) at startup. There’s an xLiteClose() function which release’s resources held by the App & Helper databases and if they were seeded via xLiteDBSeed or XLiteHelperDBSeed, will save back any changes to the seed databases.

VBA is of course at the heart of microETL. I’m often asked why not port the code to one of Microsoft’s more “modern” languages? There are three  main reasons for not doing so now (and one major one for not doing so in the past).

First the past. When MS introduced .NET the needs of those who program mainly around the Office platform were ignored, .NET add-ins for Excel were a pain to write, even more painful to deploy and even then, were slow compared to VBA add-ins. Things have improved, it’s now much easier to target Office from within Visual Studio (although still just as expensive). But even now, the best method of using .NET with Excel is a  superb 3rd-party open source tool, ExcelDNA. This is the tool I’ve used (and will continue to use) whenever I needed to merge Excel with the .NET platform.

I’m currently developing a .NET offshoot of microETL called HAMMER; using ExcelDNA, C#-SQLite and IronPython; not so much a replacement for microETL but a new type of tool to take advantage of the new capabilities offered by Excel 2007/2010 (multi-threading and 64bit memory addressing mainly) and .NET4 (IronPython and ease of multi-threading programming). It’s also, thanks to ExceDNAPack, a single file deployment. To keep an eye on its development follow the HAMMER tag on my blog.

So if ExcelDNA provides a pain-free (and cost-free) method of using C# or VB.NET with Excel, why stick with VBA? As I said, three reasons:

• VBA is to Excel what JavaScript is to the web browser. It’s only remaining purpose as a language (with the death of VB6) is as an Office automation language and ,in effect, when you’re programming in Excel you’re largely working with the Excel Object model (in much the same way and the DOM is the focus of attention on JavaScript progamming). VBA is perfectly suited to this task and although it has its problems as a language (again like JavaScript) once you know them they’re not a problem. VBA is here for the long haul, Office 2010′s addition of the 64bit support to the language (Vb7 !) confirmed that. As the code is already in VBA, works very well as VBA and has a long future ahead of it, why change?
• VBA is easy for a large subset of end-users to learn due to Excel’s macro-recording facility. The resulting code may not be pretty but it works both as a automation tool and as a learning tool. Millions of citizen programmers owe their skills to the VBA macro recorder. One of microETL’s goal is to make adding functionality as easy as possible and one such method of doing that is via simple VBA functions that can be called from SQLScript (microETL’s tabular scripting extension). C# or VB.NET are professional languages designed for professionals to use, VBA comes from a long line of “pro-am” scripting languages which open the world of programming to a much larger population.
• Multiple deployment options. MicroETL can be installed like a normal add-in and consists of three files, the VBA add-in, the SQLite3 dll and the xLiteSQLite dll to wrap SQLite to make it VBA accessible. Python functionality and other options will add further files. But by embedding a small call-on-open macro in a workbook the add-in can be called without installation. This is how I use it for once-off tasks, as I can package the code and the associated data and workbooks in a single zip file for archiving purposes or for deployment (often running off a USB stick).  Because I can deploy my code as code-at-a-moment-in-time I can modify the codebase to suit the current requirements without affecting other microETL projects that are using, or might in the future use, a different version fo the code. In fact, microETL has a noSQL compile option that removes SQLIte and other DLL dependencies (with a reduction in power obviously) enabling the code to be embedded in a workbook allowing for a single-file deployment (which often, is the only option). This could not be done with a .NET add-in.

Okay, so VBA is useful, but why embed SQLite and Python? As I’ve said above, microETL exists to make my life easier. I’m a database programmer, I’ve been using SQL (or its predecessors ) for  three decades. I like set-based logic (such as SQL  or PowerPivot’s DAX) because:

• I’m good at it (practice does make perfect) enabling me to solve many data related problems efficiently and more accurately (fast is no good if the result is wrong) that would be the case with functional (Excel fromulas) or procedural (code) approaches.
• I believe anybody who spends a significant proportion of their working lives managing datasets should learn basic SQL. If they don’t it’s like driving a car but never getting beyond 2nd gear.

SQLite allows me to bring the beautiful world of set logic to user-centric Excel world of functional programming (yes, all you Excel jocks, you’re functional programmers). It also brings some nice side-effects:

• It is, like Excel itself, an outstanding single-file document-oriented datastore.
• By default, microETL uses SQLite in-memory database functionality (you can also attach disk-based ones) so it offers a means of exploiting the vast amount of cheap RAM that modern PCs now offer. Excel has always stored its working datasets in-memory and it too now offers much greater capacity (1,000,000 rows per sheet in Excel 2007/2010). SQLite has no practical limit other than the available memory and the addressable-limit of the OS (2GB for 32bit Windows). With 64bit Windows (microETL is Excel 64bit enabled) that addressable limit is now effectively gone. This makes microETL a superb platform for all sorts of ETL tasks.
• As SQLite offers Excel a secondary in-memory datastore it can be used to share models across programming platform boundaries. This allows me to embed Python in Excel and use the in-memory SQLite database as a shared data conduit. I’ve done the same with JavaScript (but only on 32bit) and with .NET. The result is the non-native languages can be used as if they were not operating with Excel, no need to mangle in Excel interface code, simple!

Which brings me to Python. Why embed a Python interpreter into microETL? Can I not just use VBA as I quite obviously know it inside out? Well of course for all things Excel-focused that is what I do, and for most other transformation or calculations that require some programming I would also opt for VBA. No reason not to, it works and works quickly. However, Python (and in particular C-Python) offers access to world of superb code. If something can be automated, or interfaced to, or turned into a computing algorithm, somebody (usually very bright) will have done it in Python. Python also works well with C, and again if a C library is useful somebody will have wrapped it in Python. Python means that if I’m asked “Can you integrate Excel with ……?”, I can say, “Yeah, no problem”.

Python as a language has the other advantage of being a superb tool to work through a problem with; it doesn’t get in the way; very little of my mind has to be diverted to the language itself, the problem at hand gets the attention (C would be an example of a language on the other extreme, you better have the solution to your domain problem figured out before you start to program in C as the programming task itself will command most of your attention).

Python is an ideal language for business logic. It is easy to learn, eminently readable and generally immune to programming gotchas. Because of this, it’s very popular as a citizen programmers’ language, especially amongst engineers, scientists and quants, so it blends in naturally with microETL the ultimate datasmithing tool!

So if microETL is built on a foundation of “civilian friendly” technologies such as Excel formulas, VBA, SQL and Python is it a tool that anybody could use? In theory yes, in practice it depends.

Only a small proportion of the population has the ability to master procedural programming and an even smaller precentage has the interest in so doing. (A much large proportion of the population shows the ability and the interest to tackle Excel’s native programming method: functional no-side-effects programming using Excel formulas, which accounts for the appeal of spreadsheets to the general populace!).

So although I’ve abstracted away a large level of the complexity of automating Excel by means of the SQLScript and SQL functions, it’s still programming.  But even if the full power of the tool is beyond most, the tabular sequential nature of  SQLScript and the relative approachability of SQL should mean that many can at least follow the logic and data flows and might even be able to modify existing scripts. Those users with a good knowledge of VBA and or SQL should find the tool enables them to be more productive and cost effective. While those with Excel formula skills can usually work in tandem with a framework of microETL functionality with relative ease.

For the latest versions and articles on using microETL follow the microETL tag on this blog …

Example: click for a larger image (animated GIF)

=SQL() in action

SQL(sqlStatement,selectDestination(OR arg1), arg2 …. argn)

The 1st argument is the SQL statement(s) to execute. These can be any valid SQLite statements (see SQL as understood by SQLite); exceptions are the ATTACH statement and in some circumstances the DROP TABLE statement. The statement will be issued against the default microETL in-memory database i.e. an empty database that is instantiated in memory at workbook start-up and any tables or data created during the workbook session are discarded at shutdown.

So, if the database is ephemeral where will my data be stored? In the spreadsheet most likely. The other alternatives  are to use the SQLite UDFs load_CSV() and write_CSV() to load/save CSV files, or see http://blog.gobansaor.com/2011/05/11/accessing-sqlite-databases-from-excel-via-microetl/ for other options.

In order to make transferring data between Excel and the SQL environment easier and simpler a pre-processor has been added to the function. There are three types of pre-processor commands:

• The TABLE(…) command tells SQL() to go fetch the relevant Excel range, loads the “table” into SQLite; when the SQL command finishes it writes back the data (if changed) to the originating Excel address, and cleans up the temporary SQLite-side table. Nearly all SQL commands can be issued against the resulting tables, including UPDATE, DELETE, DROP and CREATE TRIGGERS etc.
• Two other pre-processor commands are CELL(…) and CELLS(…). Similar to TABLE(…), data is loaded from the Excel addresses specified and placed in the SQL  statement. CELL takes a single cell reference while CELLS will iterate through a multi-cell range and insert a comma-separated list into the SQL statement (in a format suitable for IN (…) expressions).
• Positional substitution tokens in the form :1 to :n

Note these are pre-processor commands not SQLite commands and can only be accessed using the SQL() function. The pre-processor “functions” may appear to work like a normal SQLite function call but are less flexible in some ways e.g. no spaces allowed, parser expects “TABLE(” not “TABLE  ( ” but more flexible in others so a function requiring a table name can use the pre-processor to provide it e.g. =SQL(“Select load_CSV(‘TABLE(Sheet!A1)’,'c:\data\test.csv’);)”).

The sequence of events when a TABLE() pre-processor command is detected in a SQL statement is as follows:

• SQLite SAVEPOINT is set if the SQL()  statement is not already running in the context of a previous SQL() function call. This wrapping in a transaction is the reason for ATTACH commands not been allowed. The purpose of the transaction is two-fold; 1st it will clean up any temporary tables and other updates in the case of a function failure, 2nd, SQLite inserts/updates are much faster when executed within a transaction, especially if writing to a disk-based database.
• If the text between “TABLE(” and the next “)” is a valid range representation (or a named range pointing at such) a check is carried out to see whether that same range has been referenced before in this context (same context being within the same set of SQL statements or those of a prior “wrapping” SQL() function call). If already “handled” replace the TABLE(…) with the name of the temporary table associated with this range. If not, load the table data (as long as the range is an Excel table, or points to the top left-hand corner of a potential table) into a new temporary SQLite table and replace the TABLE(…) with its name.
• When the last SQL statement within the current context is executed and if the temporary table has been modified (INSERT’ed, UPDATEd,DELETEd or DROPped ) the table is written back to the source range (or range cleared if a DROP statement).
• The SAVEPOINT is released.

The remaining arguments to the SQL() function are also processed by the pre-processor, the arguments passed replace positional substitution tokens in the form :1 to :n where :1 is replaced by 2nd argument and :n is replaced by the n+1th argument.

The 2nd argument (that is the arg corresponding to the :1 token) is special. If no “:1″ token is found within the SQL Statement, the argument is assumed to be a destination for a SQL SELECT statement (think of it as a SELECT INTO). It can be either a string representation of a workbook range or a named range pointing at one. The range can be either a full range, top left-hand corner range or an Excel Table. (If none of the above, the SELECT will populate an ADO detached RecordSet and use the argument as the name of that recordset).

A number of helper functions have been added to SQLite within microETL. Note: these are not preprocessor commands and so can be used by any SQL issued within microETL. (There are a number of SQL issuing functions other than SQL() and there’s also the SQL calls issued within Python functions.)

Using the Menu option SQL

The main SQL helper functions are …

x(formula,p1,p2,p3…)

Where formula is a spreadsheet formula or built-in function, followed by a variable list of parameters. The formula can take advantage of microETL “placeholders” e.g. x(“20+ :1 + :2″,20,40) will return 80, x(“upper(:1:)”,”abc”) returns ABC.

The formula’s text, with any placeholders replaced, is passed to Excel’s Application object’s Evaluate method, so not withstanding some of the limitations associated with this function (see here) it’s possible to use most Excel formulas within a SQL statement.

Only built-in functions (i.e not User Defined Functions) can be called in this manner.  To call a UDF use the udf() (or u() /mdf())  function.

When specifying the formula do not include the leading “=”, this will ensure formula is evaluated inline along with the rest of the SQL statement. If preceded by “=”, the formula is passed back as text. This can sometimes be useful either for testing purposes or to get around Application.Evaluate limitations.

Example: click for larger view:

udf(functionName,p1,p2,p3…)

This allows any VBA public function (i.e. a UDF) in the active workbook  to be called from within a SQLite SQL statement. The u() and mdf() functions are similar but the mdf() function will only run functions in the microETL project while u() will default to the microETL project but will also take references to UDFs in other workbooks.

Py(address,p1,p2,p3…)

The address can be either a string representation of an Excel range e.g. “Sheet2!A3:Sheet2!D6″ or a named range. If the range is a single cell the function will expect to find the full Python script within, if a multi-cell range then the script will utilise cells as Python indents.

The scripts are in fact anonymous parameter-less Python functions. Parameters are passed via xLite “placeholders” which are replaced before the script is passed to the Python interpreter e.g.

If in  this case 2nd parameter is less than 40, return 1st parameter increased by 10% otherwise return unchanged.

SQLScript(address,p1,p2,p3…)

See previous post http://blog.gobansaor.com/2011/03/04/sqlscript-microetls-sql-sequencer-utility/

SQL(sqlStatement,p1,p2,p3…)

This allows the microETL.SQL() function (along with its pre-processor goodness) to be called from an environment where the pre-processor is not supported e.g. within Python. It’s also useful to process a series of SQL statements held in a table.

Various Hierarchy functions

See this Handling Flat, Parent-Child and Nested Set Hierarchies and this.

write_CSV(table,filename)

This will write out the contents of the table in classic CSV format (UTF-8 encoded). Useful for external SQLite tables that may be too big to fit in memory but need transformation before loading as a CSV into the likes of PowerPivot.

load_CSV(table,filename)

Will load UFT-8 encoded CSV files into a table. Again, if a CSV source is too big to load into memory (thinking fact tables here) but needs some SQL love’n'care can be loaded into an external table (i.e. in a disk-based database) bypassing the need to fit in Excel’s working memory.

sqliteDate(dateTimeString)

Will attempt to convert a string to an ISO DateTime string ( the format used internally by SQLite i.e. “YYYY-MM-DD hh:mm:ss”). Usually microETL will handle date conversions between Excel and SQlite and back again (but when a date is pasted back it’ll be in Excel numeric representation, the column may need to be formatted to the required date/time representation). Sometimes however the date format of imported “string dates” may not be recognised (for example, Excel loaded CSVs often fail to recognise dates correctly). This function will usually solve the problem.

ISODate(Year,Month,Day)

Will take a date passed as 3 numbers (e.g. ISODate(2010,12,31)) and convert to SQLite date format.

REGEXP

I’ve also added REGEXP (regular expression) support so you can do stuff like so…

SELECT * FROM Foo WHERE account_id REGEXP ‘[0][0-9]*[x]‘

or

=SQL(“SELECT regexp(‘[0][0-9]*[x]‘,’0987f85x’);”) -> returns 0 for false (should be all digits between leading zero and final ‘x’).