PayPal IPN, PDT & Analytics tracking, getting there.

As I’ve previously written about, I’ve been having some trouble with PayPal PDT ( Payment Data Transfer ) and Google Analytics e-commerce tracking.

If you’ve added Analytics to your PayPal thank-you page, and used PDT to get the data to be sent to Analytics and got that working nicely then you will have discovered that not every sale gets tracked, because not every shopper can be guaranteed to land on your thank-you page.

Now I’ve been doing some testing on an idea to get around this using IPN. The basic premise is this.

  1. Google Analytics works through a request for __utm.gif from the Analytics server, to which is attached all the tracking information.
  2. On the page just before leaving for PayPal set the analytics script to local only.
  3. Use url rewriting to hide a script behind your local __utm.gif.
  4. Record all the details for the request for the local __utm.gif in a database, referenced to the session.
  5. Send the session ID information through PayPal’s custom variable.
  6. User finishes sale.
  7. IPN script picks up sale, checks session ID and looks up the stored request.
  8. IPN script rebuilds request and forwards to Google’s __utm.gif.

Now to test this before getting involved in databases and IPNs I just made two pages. One with Analytics set to remote, the next page with analytics set to local. Then I made a local __utm.gif just forward the request for the remote __utm.gif through a server side HTTP request.

Wait a day.

No joy, nothing showing up for the e-commerce tracking, or the visit to the second page, just the first page visits where the script was set to remote.

Then I fiddle for weeks trying to improve the request. Adding all the cookies etc. etc.

No joy.

So I took a look at the urchin.js file to determine whether or not the local request was being built any differently to the remote one. And it was. If you have a look through urchin.js searching for the local/remote variable (forget the name of it right now) then you’ll see that there are extra things appended to the end of the querystring for the remote call to _utm.gif. The extra stuff is the content of all the analytics cookies.

So I fiddle urchin.js and made a local copy that built the same request for remote and local, including all the cookie data. I had a couple of people hit the two pages and a day later…

Success, there is e-commerce transaction data, items, totals, everything in Analytics now.

I used different sources and the like for a couple of test and those have also come through. Even more surprising though is that where I have had other people test the two pages for me, their locations have been tracked and assigned to the e-commerce transactions correctly. Which I didn’t expect as it’s always the web server making that request, and it’s not moving around!

So that’s something to work on. I’ll keep you updated on the next stage of implementation.

Making AirClick USB Plugins

Airclickusb_medIt should be fairly easy for someone with programming experience to create their own “Plugins” to extend the AirClick. It runs on the Microsoft .NET Framework 2.0, which means you can write plugins in any .NET capable language. Just add a reference to the actual AirClick.exe file, define that you are using the “AirClick” namespace, then create a derived class from the AirClick’s “Plugin” class. As you will see there will be several virtual methods you will have to define. These include accessor methods for the plugin’s name, author, description, executable name, and version. (NOTE: The only one of these attributes used at the present is the name, which appears in the AirClick menu. The others are to insure compatibility with potential future updates.)

To “tell” the AirClick software which button presses you are interested in knowing about, add a call to the “AddButton” method for each button press in your class’s constructor. Button presses are defined by an instance of the “Button” class. Here is a quick rundown of a Button’s constructor parameters:

ID – a unique integer ID given to the button. This ID is passed back also a paramter in the “Notify” method to tell a plugin which one of his buttons has been pressed.

Key – the button on the AirClick remote you want to know about. Values can be back, forward, play, voldown, or volup.

Modifier – a modifer key that must be held down along with the “Key”. Values can be back, forward, play, voldown, volup, or none.

Time – the number of milliseconds the “Key” must be held down in order to trigger this button press.

OnRelease – true if button is triggerd when it is released, false if it is triggered on push.

Name – short description of this button, such as “Play” or “Next Chapter”. (Presently not used, just there for potential future updates.)

ShowInMenu – whether or not this button’s name should appear in a list of this plugin’s functionality. (Presently not used, just there for potential future updates.)

Finally, the last thing you have to do is define the “Notify” method. This method is called by the AirClick software when a desired button press is found, and the ID of the found button pressed is passed to it.

There are a few more classes built in to more easily accomodate the making of Plugins. Their is the OSD class for using the AirClick’s on-screen display. It can display various combinatinations of strings, a progress bar, and generic player graphics. The player graphics are represented by the “DisplayGraphic” enumeration. The “Volume” class can get or set the system’s master volume, and the “WinMessages” class can send several different kinds of Windows messages to a windows with the given caption or class name. The messages it presently supports sending are “WM_APPCOMMAND”, “WM_APP”, “WM_COMMAND”, “WM_VSCROLL”, “WM_HSCROLL, “WM_SYSCOMMAND”, and “WM_USER”.

Once, you have built the plugin, just change the extension from .dll to .acp and place it in the AirClick’s “Plugins” folder. Restart the AirClick software, and it should appear in the AirClick’s menu.

PayPal E-commerce Tracking with Google Analytics

Updated post: http://www.teknohippy.net/2008/03/04/paypal-ipn-pdt-analytics-tracking-getting-there/

Trying to track E-commerce transactions with Google Analytics (GA) and PayPal can be problematic.

PayPal has two ways of returning data to your server. Payment Data Transfer (PDT) and Instant Payment Notification (IPN).

PDT can be used to setup GA tracking by including the relevant GA scripts on a receipt page, processing the PDT data server-side and populating those scripts. Instruct PayPal that this page is your “return” page and when visitors return to it the e-commerce and goal data will be tracked.

The big problem with this is that PayPal users do not have to return to the receipt page, even if you’ve turned on the auto return feature. The pause before redirecting is very long and many people just abandon on the PayPal pages.

IPN on the other hand will always get triggered. The problem is it calls a script on your server from PayPal’s server-side scripts. This means your IPN script never runs in the users browser and so the GA JavaScript is never executed and no tracking will happen.

Now at it’s heart all the GA script really does is assemble a call to an image on the GA servers, __utm.gif to be more specific. Appended to this request is a lengthy querystring with all the information for Analytics to log.

It should therefore be possible to capture users info or the actual __utm.gif querystring on the page just before leaving for PayPal, temporarily store this in a database against the users SessionID, and pass the SessionID to PayPal as a custom pass-through variable.

The easiest way to collect the querystring from the pre-PayPal page is to set “_userv = 2;” in the tracking code. This means that __utm.gif will be requested from both the Analytics server and your server. Trap the call to _utm.gif with some server-side script and you’ve got the whole querystring very easily, as well as the corresponding SessionID.

The user is now off to PayPal to complete their transaction.

Then in your IPN script, lookup the SessionID and recover the __utm.gif querystring, change the utmp path variable for the path of the reciept page and then use your server-side script to make an http request to __utm.gif.

This will then at least track and goal reporting will work. The IP address will be wrong but the rest of the data will be correct.

More complex is building the querystring for the  __utmSetTrans() call. I’ve had a look into what’s sent and a separate call is made to __utm.gif for the transaction and each item within it.

I suppose it would be possible to set _userv to 0 and only have a local __utm.gif request on the checkout page. Then the Ecommerce tracking data could also be included. All the script hiding behind the local __utm.gif needs to do is record all requests when visitors hit the checkout page. It can pass on the actualy request for the tracking (but not the transaction), when it happens. Then in the IPN script the tracking request path can be rewritten and replayed, followed by the transaction requests. Sounds like it should work.

I guess I should go make it happen and test it!

Winamp 5.0 Bento Skin

The new Winamp skin is actually usable! I’ve always hated the modern skin and so have not installed it for who knows how long, preferring the classic sking and it’s darker colour.

Well I’ve just installed Winamp 5.0 and thought I’d try out the new Bento skin. The good news is it’s actually usable! Woo! Well done.

Elite IV is not Vapourware

In a recent CVG interview, David Braben talks about the next version of Elite. Confirmation that Elite IV (No better name exists publically yet) is not vapourware as some have been saying.

Obviously they’ve not been listening to David and the passion with which he speaks of a future Elite.

Elite will be Frontier’s next big next-gen game after the political thriller The Outsider ships. The Outsider is going to be a game that breaks a lot of new ground in gameplay and immersiveness, ideas and technology that is required to make Elite what Braben dreams it could be.

Also confirmed is that the next incarnation of Elite will be a single player game at first, with a small LAN based multiplayer element. Followed later, after more development and technical advances, by a proposed Massive Multiplayer Online version of the game. Something that a lot of Elite fans have been fantasising about for many years.

David Braben on the PS3

In a recent interview with CVG, Elite creator David Braben speaks of how he believes Sony got their decisions about the PS3 absolutely right.

There’s been plenty of bashing of Sony’s handling of the PS3 launch and little to be said from anyone in defence.

Braben though thinks differently:

“I’m very wary …at the moment there’s a fashion to knock Sony and I think it’s a big mistake. The fact that they have succeeded very well twice, they’re one of the few companies to have gone outside core gamers. Don’t write them off.”

Braben continues to express his belief that the choice to go with a completely open net access service, as opposed to a closed service such as Micosoft’s XBox Live, will be key to the PS3’s success.

“That shows a very different approach. It’s not a narrow gateway, highly controlled by one person. I think that is very very positive.”

He also believes that the PS3 should be considered more than a console.

“When you put it together with the fact that it supports a keyboard and things like that, people may start to regard it more as a cheap PC than an expensive console. Or a new thing that’s somewhere in between.”

Well it remains to be seen, but I respect the positive attitude. I’m going to a talk by Braben on Monday, I wonder if this will arise in the after lecture questions.

Hacking AirClick USB for Windows/PC

(updated below)

In the aftermath of changing to my new Cherry MX switched keyboard I have found myself missing the handy media buttons on the old membrane keyboard.

So I had a little google about (using google) and found the Griffin Technology AirClick USB. A handy little Human Interface Device, it has five buttons and a number of plugins that allow you to control various applications. Primarily I was interested in Winamp control, playing, pausing and changing tracks. I had a deeper look into it and one of the available plugins in Sean Wilson’s new version 2 software is a winamp plugin. Cool, so I bought one.

Works great. Until I went to change volume that is, it’s just too slow for my purposes.

In the release notes there is the following addition to the latest version mentioned:

Easier creation of additional Plugins using any .Net supporting language

Well fantastic, I can do that, perhaps I can write my own winamp plugin. Plenty of searching around their site later and there’s no help or info on doing any such thing. I emailed support and got back a plugin that used sendkeys to just send configurable keys to Windows. Not a lot of use if you happen to be typing at the time!

I found the developer of the Mac software and his blog. He’s posted ways to hack the Mac version, no go for PC though.

I then noticed that the .acp plugin files in AirClick’s plugin folder were just renamed .net 2.0 IL dlls so I emailed support again asking if there was any supporting source code for writing your own plugins.

While I was waiting for a response I had a quick fiddle with Lutz Roeder’s Reflector and Denis Bauer’s FileDisassembler plugin and generated a C# project from the winamp.acp plugin. It compiled with no trouble, so I renamed the .dll to .acp, dropped it in the pugins folder, restarted AirClick and everything worked. Fantastic. Now I can get on with hacking it.

A little bit of coding later and I’ve now got a volume control that smoothly accelerates from small changes to larger the longer you hold down a button.

Then the reply came back from support:

Unfortunately we do not have source code for the AirClick. This is info that doesn’t leave the doors of research and development.

Oh dear, really, well I’m screwed then.

*GRINS*

So top marks to Mr.Wilson and his code, but the usual non-technical nounce of the average support department proves to be a barrier yet again. Great product on the other hand.

Updated Information

I decided it was worth trying to contact Griffin again, so I penned a very similar email to the last one. This time I got a direct response from Sean Wilson, the creator of the PC AirClick software.

I have not written a formal SDK or Visual Studio template for writing

AirClick plugins yet, but I do have a txt file I wrote up real quick

to send to people who are interested in writing their own. I will

attach it with this email.  – Sean Wilson

Great stuff! So here’s a copy of that file for all you plugin makers.

Making Plugins

And here’s my modified Winamp plugin.

USB Keyboards again.

Well I’ve got meself a decent mechanically switched keyboard, and frankly I really don’t know why I didn’t do it before. I’ve concluded that my recent problems with USB keyboards are just that they both had lower than average quality membranes. My new mechanical switch enabled Cherry MX 3000 is working wonders for my typing speed. I had found that I had to slow down too much with the old keyboard, now I’m back up to previous speeds, perhaps even a little faster.