I am busy at the moment writing various implementations of NeuClear Ledger in mind with the next release 0.4 which should be released later this week. One of the the implementation plugins will be a relatively complete version that I’ve written in Prevayler

What are Ledgers?

Within NeuClear we use ledger’s as one of the most important building blocks. Ledger’s are the fundamental datastructure used in any kind of double entry accounting system.

The NeuClear Ledger API is fairly simple and can be used to implement ledgers using SQL, in memory etc

The API basically allows you to perform accounting transactions on the ledger implementation. It will then allow you to view balanaces etc.

The NeuClear system has a special requirement for what we called Held Transactions, which are simply temporary transactions that put a hold on the available funds of a system. It was important that we implemented this on the ledger side and not higher up in the NeuClear framework.

Our Requirements for a ledger for a transaction processor

The Prevalent Ledger is intended to be run on our NeuClear Transaction Processor. The Transaction Processor does the following:

1. Listens to Transfer Requests
2. Verifies the Signatures of said Requests
3. Verifies the Balance of Transferer
4. Performs book entry transaction on Ledger
5. Signs and returns a receipt

For performance and security reasons this processor will not allow people to view statements or check for historical records. This will be handled on external Auditing Servers, who use a different implementation of the NeuClear Ledger Hibernate

Thus the Prevalent model works fine. We have a few very limited types of queries we need to do in the normal run of things and require absolute speed and reliability.

30 second overview of Prevayler

Prevayler stores the entire database in memory and performs periodic snapshots to disk in a serialized form. To avoid dataloss and manage transactions any transactions are handled in serialized command objects. That get serialized to a transaction log on disk before they get executed.

For more info see the 1 Minute Introduction to Prevayler or the 2 minute tutorial both at on the Codehaus Wiki.

Implementing the Prevalent Ledger

A prevayler database needs a single reference point for all the data stored within. In Prevayler terminology this is the Prevalent System. In my implementation I created a central class LedgerSystem, which

The Data Objects

The most basic part of the ledger functionality is to get the balances of books and add transactions. In this implementation we keep no history just update the balances. To do this I created a super simple class BalanceTable which allows me to get the balance of a book as well as add an amount to a book.

For memory and speed reasons I chose to use TObjectDoubleHashMap from Trove4j as it stores primitive doubles and supposedly uses a lot less memory.

I wont get into the other parts of LedgerSystem here, but they are also fairly simple. The important part is to remember that the system is your main database and it needs to be memory efficient as well as Serializable. I made most of the methods in the System as well as its contained objects package scope, to not allow any outside manipulation of data.

The Commands

To create a Transfer I simply created a new command class which implements “TransactionWithQuery”. This simple one is called PostTransaction. All the command classes must be serializable. I pass all parameters to it’s constructor. The real juice of the object is done in the executeAndQuery() method which gets passed an execution time and the System.

public Object executeAndQuery(Object prevalentSystem, Date executionTime) throws Exception {
         LedgerSystem system=(LedgerSystem) prevalentSystem;
         TransactionTable table=system.getTransactionTable();
         if (table.exists(tran.getId()))
             throw new TransactionExistsException(null,tran.getId());
         if (table.exists(tran.getRequestId()))
             throw new TransactionExistsException(null,tran.getRequestId());
 
         table.register(tran.getId(),executionTime);
         table.register(tran.getRequestId(),executionTime);
 
         Iterator iter=tran.getItems();
         while (iter.hasNext()) {
              TransactionItem item = (TransactionItem) iter.next();
              system.getBalanceTable().add(item.getBook(),item.getAmount());
          }
 
         return new PostedTransaction(tran,executionTime);
     }

As you can see it’s pretty simple and straightforward. No real black magic. When you execute the above command, Prevayler first stores the serialized form to disk as its log and then executes the object.

Integrating it

All implementations of NeuClear Ledger must extend the Abstract class Ledger. This I did in PrevalentLedger.

The real juice here is the Constructor which loads the data:

    public PrevalentLedger(final String id, final String basedir) throws
		 IOException, ClassNotFoundException {
         super(id);
         prevayler = PrevaylerFactory.createPrevayler(new LedgerSystem(id), basedir);
         system=(LedgerSystem) prevayler.prevalentSystem();
     }

basedir is the directory used to store the snapshots and transaction logs. The new LedgerSystem created in createPrevayler is only used if there isn’t already one available as a snapshot in the basedir.

The Ledger class has various abstract methods that I needed to implement:

   public PostedTransaction performTransaction(UnPostedTransaction trans) 
	throws UnBalancedTransactionException, LowlevelLedgerException, InvalidTransactionException {
         try {
              return (PostedTransaction) prevayler.execute(new PostTransaction(trans));
          } catch (Exception e) {
              if (e instanceof InvalidTransactionException)
                  throw (InvalidTransactionException)e;
              if (e instanceof UnBalancedTransactionException)
                  throw (UnBalancedTransactionException)e;
              if (e instanceof LowlevelLedgerException)
                  throw (LowlevelLedgerException)e;
              throw new LowlevelLedgerException(e);
          }
     }

This creates a new Instance of PostTransaction and lets Prevayler execute it. Note the ugly Exception handling. This is due to the fact that the executeWithQuery method is declared with throws Exception. I’m sure there is a more elegant way, but I couldnt get my head around it at the moment.

Conclusion

It was remarkably easy to implement the Prevalent Ledger. I wrote it to the stage it is today in probably less than 8 hours over a 2 day period. Very refreshing.

For some things the logic got a tiny bit out of hand. See for example HoldTable and it’s pal AccountHeld this would have been very quick to do in SQL. I’m sure there is a better way here, I’ll have to look at it another time.

I guess my main fear about using Prevayler is when I make database changes, how easy will it migrate? I have heard there are various XML backends for Prevayler, which may suit this purpose well.

I am now working on a new implementation of Ledger, written using Hibernate. There is a bit more of a learning curve as I havent used Hibernate before, but I can see that it is an excellent tool.

I am happy to announce the 0.12 release of NeuClear XMLSIG. Major new
features are:

• Safer Application Oriented API
• Improved verification of Reference Types
• Simpler API
• Support for X509 Certificates
• Improved Interoperability

Also released is the 0.6 release of NeuClear Commons which is required by NeuClear XMLSig.

For a list of all changes see the 0.12 Release Notes on JIRA

NeuClear XMLSig is a java library that use the following libraries:

• dom4j xml library
• Bouncy Castle Crypto

It is not designed for completeness. The features that we support are
generally features needed for NeuClear. If any one is interested in
implementing the missing features for full interopability please let me know.

For more information see NeuClear XMLSig

Example Sign XML Element

// First we'll create a KeyPair
KeyPair kp=CryptoTools.createTinyRSAKeyPair();

Document doc=DocumentHelper.parseText("<test><test2/></test>");
Element elem=doc.getRootElement();

EnvelopedSignature envsig=new EnvelopedSignature(kp,elem);
System.out.println(doc.asXML());

This will leave you with the Element looking like this:

<test><test2/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<ds:Reference><ds:Transforms><ds:Transform ds:Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform ds:Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/></ds:Transforms>
<ds:DigestMethod ds:Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<ds:DigestValue>
B/P2qMqBvrZCYSa5RDuKHY9s8j4=
</ds:DigestValue></ds:Reference>
</ds:SignedInfo>
<ds:SignatureValue>gIYL0CM6xeniLgqRqPqsFEFx7Rrv7vKvW/sBlgoCyn7BlX+OTizulwGOFmw3Q9H5vFcSfFjTO8Y1TJcnIMnKzg==</ds:SignatureValue>
<ds:KeyInfo><ds:KeyValue><ds:RSAKeyValue><ds:Modulus>
zi2oiVe/tXSsGe8U1zT6znn2kFG3FXwjZ+NW8l2GcTd9zt1Y3dpScbUzzvbeQqfUw0uzPetRrK6h
fhCeo4D0Uw==
</ds:Modulus>
<ds:Exponent>
AQAB
</ds:Exponent>
</ds:RSAKeyValue>
</ds:KeyValue>
</ds:KeyInfo>
</ds:Signature></test>

Verify Enveloped XML Element

It is very simple to verify a signature. Just feed the signed xml element to the constructor of EnvelopedSignature:

try {
     EnvelopedSignature verified=new EnvelopedSignature(elem);
} catch (InvalidSignatureException e) {
     System.err.println("Invalid Signature");
}

Philosophy

The XML Signature standard was designed very much as a low level standard. Most API’s reflect this and allow all manners of types of signatures, which could potentially lead to major security holes.

The NeuClear XMLSig library was developed for application developers and allows you to work with most major types of XML Signatures safely and without knowing the quite complex XML Signature Standard

For more examples on how to use it see the new Busy Developers Guide to NeuClear XMLSig

Steven Levy has an interesting story on Newsweek entitled Geek War on Terror, which is about a new cypherpunkish approach to analyzing transactions from individuals in a anonymous manner. Very interesting. The technology isn’t new it just hasn’t really been applied in this way before. I am naturally sceptical about anything that allows big brother to check on us more, but the approach is interesting from a technical and moral standpoint.

Background

SRD the company behind the new technology has a background in amongst other things fraud prevention in Casinos.

Their 3 key technologies as I can work out are:

• ERIK which is what they call Identity Recognition Software
• NORA is their FOAF like technology or what they like to call “Who Knows Who”
• ANNA is their anonymous information sharing technology

While they appear to have customers in gaming and financial services their real target market now is Patriot act compliance. And I have to say that I like that they have at least thought about anonymity in this approach.

Who’s Who?

Entered data in CRM systems can often be very unreliable. It is not uncommon to have several different records for the same client. Often with minor spelling mistakes or different addresses.

ERIK attempts to create an unique ID for each client in a CRM system. The technology sounds like an advanced multidimensional form of SOUNDEX, obviously it is a bit more complicated. The multi dimensionality comes by it using more than just names. It adds various other identity information such as residence, age etc as well and somehow generates the Unique ID.

Who knows who

Besides knowing exactly who’s who the key application is to find out relations between people. Call it a fancy Friend of A Friend (FOAF) analyzer. A simple example would be if Bob and Carol lived at the same address, that would be a connection between the two. I am not sure to what extend they add relations. But the core patriot act application is of course to dig out friends of known terrorists. This sounds like it could be pretty effective, but also very scary.

To do this properly you need to aggregate data from as many sources as possible. This might be fine within a large organization, but it gets scary when you think about governments having access to all of our financial transactions etc.

Which is why they have invented…

ANNA: Anonymous Entity Resolution Technology

Anna is the cypherpunkish aspect of SRD’s technology chain. Essentially ANNA takes the ID generated from ERIK and creates a secure hash of it. In NeuClear we use the hash of the public key as the main identifier, this is very similar, except it is based on the persons real life persona.

The data from financial and other institutions can now be shared anonymously only identified by the fingerprint generated above. This would most likely be shared using NORA servers.

There would be private applications where this information could be shared via independent clearing houses running NORA. You could imagine equifax etc running this kind of business.

The patriot act application would have the government doing this. This they argue would mean that the government could anonymously match FOATs (Friend of a Terrorist) getting on planes, renting cars, opening bank accounts etc.

Conclusion

If this works it sounds like a beautiful technological solution that would protected peoples privacy while allowing intelligence agencies to do some serious fishing.

There in I think lies the problem. You can imagine any bunch of “vital” agencies such as the IRS and SEC wanting in on the act as well. You might end up as a FOASO (Friend of a Sex Offender) because you worked in the same office as someone who had was once convicted.

I dont trust government, and I dont trust that only legitimate use will be made of it. Just look how the IRS managed to get Tax Evasion added to the list of Money Laundering crimes and how they are using that to bully foreign governments under the guise of anti drug and or anti terrorist war fare.

Never the less the technology has interesting ideas and should be analyzed be real security and privacy experts like Bruce Schneier before it gets put into government use.

A very good friend of mine, paid for a notebook with a personal check in November from QLI, they still havent delivered it or a refund to him. Now I see the following on their web site:

After 6 years of serving the Linux Community, QLI technologies will be ceasing operations as of March 15th 2004. All orders in progress will be refunded and current warranty returns are being processed

They have obviously been in trouble for a while. However I dont buy that orders will be refunded. They stone walled my friend all through January and February while they were still taking orders. That sounds more like fraud to me.

Raymond Sanders the owner of QLI posts a poignant blog entry about some people being fuckwads about their closing. I would imagine that being some of the people who appear have been defrauded and might be upset. Hmm. I really do hope that Ray manages to provide refunds or the like. If he does I will let you know.

iftop to the rescue. Today I’ve been having really bad bandwidth and started getting paranoid. Sometimes bad bandwidth can be one of the first real obvious symptoms of a worm or a Denial of Service attack. How do you find out what is going on?

Well one of my favorite little tools in my networking toolbelt is iftop which works just like top but for bandwidth. It shows all the open connections on your network interface and allows you to see their source and destination IPs as well as bandwidth information.

In its simplicity and beauty this allows you like top to see in an instant what is going and where the bandwidth is going.

iftop works on Linux and bsd. I’m guessing that Mac OS/X should work. The main dependency besides curses is libpcap. Under Gentoo linux all you have to do is to:

# emerge iftop

On a related note. All users of PF (the most powerfull firewall in the world) on OpenBSD should install pftop which provides a top like view into the world of PF’s state tables. Very interesting and cool.

I was initially sceptical about Confluence from Atlassian as I didnt see the need for a commercial wiki.

However having played with their demo site I have to admit it really is an exceptionally cool web application. Which it rightly should be since its from Atlassian.

My OSS project NeuClear is a very complex project in the financial and commerce space. It requires some pretty heavy conceptual shifts and thus excellent documentation is vital. I had started documenting it on a Movable Type blog like the one you’re viewing now. But I knew from the beginning it was wrong and I went searching for a better approach.

Confluence has allowed me to setup what will be a very complex site in a very short time. It is slick, very professional looking and easy to use. The installation of it was fairly painless as well. I am glad they went with Hibernate and not the old ofbiz entity engine craziness they use in JIRA it seems to have made the installation process a bit simpler.

With respect to the NeuClear site. I am using 2 spaces. One for the non technical aspects such as conceptual thoughts, legal and business aspects etc. The second space NeuClear Tech will hold all the technical aspects of NeuClear such as implementation howtos, standards documents etc. This tech space is currently empty as I am trying to make the non technical space useful first.

BTW. Thanks to the folks at Atlassian for giving us and other OSS projects a free license.

I have been using the Linux 2.6 kernel for quite some time now as well as IntelliJ IDEA. I had problems with this combo before which seemed to have been fixed. Now I think a variation of this has reared its ugly face again.

What happens is that when I add a jar file it imports it and reports that everything is fine. However none of the symbols are available and if you attempt to browse the libraries, the classes show up as empty class names.

I’ve reported the bug and I also am pretty certain that the following report 30327 is related.

The temporary fix to this for anyone who has this problem is to append the following to your idea.lax file?s lax.nl.java.option.additional property:

-Didea.jars.nocopy=true

This kind of fixed the old problems as well, but with this it makes it fully usable as far as I can see.

This would indicate to me, that IDEA somehow garbles the classes in the jar files when they are copied over to the system jar cache. The reason that I think that is that the class browser can see the directory structure within the jars, but not for some reason parse the class files for their symbols.

I was checking my logs and realized that all the comment spam I was receiving was done using http GET. Thus a quick temporary fix is to disable GET from comment postings as MT’s forms all use POST.

Just insert this little snippet at line 50 of your MT-BIN/lib/MT/App/Comment.pm file, note you can put it really more or less wherever you want in the validation part of the post method. I placed it there as thats before any real heavy work is done:

if ($q->request_method() ne "POST") {
        return $app->handle_error($app->translate("You are not allowed to post comments."));
}

Remember this wont stop spam for ever, it’s more like a temporary bandaid.

I get lots of questions by people all the time about Panama. So rather than keep feeding the answers out by email and writing here about Panama I have just launched The Panama FAQ.

I may still write a few personal items here in this blog about Panama, but most stuff in the future will go there.

How cool is this? The Java Optimized Processor provides an design for a Java CPU that can be embedded on your own fpga chips, or I would assume even larger scale ASIC’s.

Found this via OpenCores which I found via Kuri5hin’s A Verilog Introduction.

The design is very small and would be cool for use in certain microcontroller applications. The author has created a tiny board which contains an ethernet chip as well.

Now, if you dont have an underlying OS for your java, you need to write a low level Microcontroller OS in Java to do networking. So the author implemented a nice looking little TCP/IP stack in JAVA called EJIP. This comes with such things as simple web server.

Great work, now combine this with an external DAC and an internal USB2.0 with some FFT for MP3 decoding (Yes I am showing my ignorance, I have no idea how MP3 works and if it at all uses FFT’s) and you would have a home customized mp3 player customizable in Java.

In any respect OpenCores is exceptionally cool as is the JOP. If I only had time I would buy one of the home kits and experiment. Chip design was one of my favorite subjects way back at uni and I’ve had an unhealthy fascination for FPGA’s since I was 17 years old and tried to design a parallel multimedia computer using them and the Transputer. Needless to say this early attempt at world domintation didnt take off, but I certainly had fun trying.

Kuro5hin have a superb quick introduction to the syntax of the Japanese language , from a geeks perspective.

I did Japanese in high school (gymnasium) in Denmark and loved it. By now of course I’ve forgotten most of it, but can still helpfully manage to embarras myself around Japanese people.

I remember being fond of the grammar, because while it was very different from most European grammar, it was simplicity itself. Do to its clever use of post positions you can really do some kewl perl like one liners in Japanese, where you would have to use several lines in lower level languages like English.