New ranges on the way

Sep 14 2012 Published by under MediaCreeper

After a long period of inactivity, at least visible inactivity, we have got a bunch of new ranges and are also starting a round of evaluation of old ranges.

Work has started to clean up code, to prepare release of CreeperFramework as open source on github. Further ahead the database layer is gonna get an overhaul, a switch from MySQL to mongodb.

An evaluation of the usage of the API-site to determine if it’s worth the effort to convert it to node.js or not.

No responses yet

CreeperPing, the WordPress plugin

Feb 22 2012 Published by under MediaCreeper

I got a message from a friend on twitter the other day, about the WordPress plugin I was developing a while ago. Up to now, it has stalled, completely, as the development of the API-site/service of the CreeperFramework went high priority the plugin were left behind, none of the API-calls that made it tick are the same, so I’ll have to rewrite those parts.

The CreeperPing plugin for WordPress is a simple way to evade filtering and blocking that some media houses do, as communication happens server-to-server, then they can block MediaCreeper all they want, their trails will still be visible..

I’ve never written a complete WordPress plugin from scratch, only hacked up existing ones beyond recognition, this will be an interesting journey. I’ve got professional WP-developers to hold my hand while coding and hopefully it’ll be done in the near future, a few weeks at most.

No responses yet

Request for suggestions

Jun 15 2011 Published by under MediaCreeper

It’s been a while since I had the time to really do something constructive with the code, the project has been put into a maintenance phase, not moving in any direction, as I’ve been caught up in work — quitting employment, beginning a new.

I’ll put this request for suggestions out; what would you like to seen added and/or changed on MediaCreeper? All suggestions will be taken under consideration, I’ll even blog about each suggestion and evaluate it, with argumentation for/against inclusion into the code base.

Issues that is already on the roadmap..

  • Optimize database-connections (in transitional work for moving off MySQL to a non-sql database in the future, probably MongoDb)
  • Make web site more dynamic, add ajax to update listings
  • Clean up code base to prepare for public release, as open source
  • Sync API-calls with WordPress plugin CreeperPing
  • Adding full support for IPv6 — currently spotty support

.. there is possibly more issues that I’ve forgotten about, adding them as they bubble up.

No responses yet

Better IPv6

Apr 25 2011 Published by under MediaCreeper

Just updated a chunk of code that handled the parsing of IPv6 (6to4) .. where IPv4 are tunneled through IPv6 … and IPv4 addresses are embedded within IPv6 addresses. The chunk of code has been tested against approximately 16000 IPv6 addresses and test patterns.

Possibly we have missed a few hits where a small number of addresses were parsed wrong and dismissed as negative hits. We will keep an eye on the statistics for this and tweak it further if necessary.

CreeperFramework (which is the platform MediaCreeper is built upon) does not yet support IPv6 fully, work is in progress for this and will be supported in the near future. CreeperFramework will be released as open source when code base is mature for a public release.

No responses yet

Old logs

Apr 17 2011 Published by under MediaCreeper

To correct and fill in missing data, I wrote a small shell-script to parse through old logs, some 9Gb of old logs. There is some more work required to massage the parsed data to something useful, so it’ll be online as soon as I’ve had a few more hours to spend on it.

When done, it should be possible to render diagrams of traffic on MediaCreeper from November 16th 2008 to today’s date, on total hits, positive hits and unique IP’s.

The last couple of weeks has been quite busy, so I haven’t had that much time to work on CreeperFramework and MediaCreeper. I have a few new ideas to try, but I’ll put them on hold for the moment.

The end of this month will be the last days on my current job, a new job from the May 2nd which is based in my city, less travel, better salary, more sleep and more time for MediaCreeper.

No responses yet

Maintenance update

Mar 13 2011 Published by under MediaCreeper

Just updated a lot of code and templates of the CreeperFramework, preparing for further updates. No new ranges in this update but I’ve got serveral up my sleeve.

Some optimization, prettification, validation of HTML, tweaks on SEO-stuff, javascript and will turn on compression in the next update.

I’ll put some effort into completing the WordPress plugin, CreeperPing .. as the project has gone stale. Need to put this one out the door.

Have given evercookie‘s a thought ..

No responses yet

API considered stable (enough)

Mar 07 2011 Published by under MediaCreeper

The MediaCreeper API (http://api.mediacreeper.com/) has been online for a couple of weeks now and are considered stable. It has been linked on the Swedish OpenData-site, a sort of repository of links to open APIs and data collections.

The MediaCreeper API is licensed under CreativeCommons Attribution-NonCommercial-ShareAlike or CC BY-NC-SA for short.

There are a non-public part of the API that isn’t completed yet, the CreeperPing. It’s a WordPress plugin that will interact with MediaCreeper over a Server-to-Server protocol, capable of circumventing blocking and blacklisting of CreeperFramework (MediaCreeper and variants).

No responses yet

First block?

Nov 25 2010 Published by under MediaCreeper

I’ve seen signs that at least one media organization has actively blocked MediaCreeper, it’s the Swedish newspaper Aftonbladet. They have almost disappeared from the toplist and latest pages, only a few hits per day.

I’ve foreseen this situation and have a solution, it’s a bit complicated, a WordPress-plugin (called “Creeper-Ping”) that acts like a proxy — it will forward each page-load to the MediaCreeper server to verify if it’s a hit/miss and serve back the banner accordingly. This plug in is still in early beta and I have it installed on my Swedish blog for testing and development purposes only, it seems like I have to spend a few hours this weekend to get it into production quality.

It’s prepared for several more sites just like MediaCreeper, as the underlying framework is going to be released as open source as soon as I bang it into shape.

I’ve been thinking about how to distribute the collection of traffic across several domains, with a little help from trusted friends and perhaps a few domains or subdomains, a simple PHP-proxy could be hosted on any server, in any directory .. as long as there is some coordination, this would make it possible to use Javascript to include the banner, which would get loaded from the proxy-script, which in turn would report it to MediaCreeper as a hit or miss.

No responses yet

Progress of today

Sep 04 2010 Published by under MediaCreeper

I’ve been working on a few things today.

The English translation has been tweaked here and there, as I will put it into production soon. Some wording didn’t feel right and I have rewritten parts. I hope to juggle these texts with someone who has English as first language/native tongue.

The RSS-Engine was a bit broken, when someone for some reason was caught as a positive hit in the monitored ranges and didn’t supply a referrer (masked off or filtered), the RSS would come out with text-macros intact, which isn’t pretty or even useful. Added a few lines of code to handle that and adapted the engine further, to handle multilingual stuff and removed/replaced some hard-coded stuff from the templating.

The Multilingual/Language code has been somewhat overhauled, as far as detection and logic-wise, at the moment any other languages other than the supported, Swedish and English, will default (fall backwards) to English… as most people on the Internet at least can understand a little bit of English. Swedish is a relatively small language, about 9 million people around the globe talks, reads and writes it.

I’ve added about 50 ranges for Norway, Denmark and Finland but not activated them yet, just have to throw the switch and they’re active.

Some meta-headers in the HTML has been added/adjusted to ensure that spiders and bots find their way in, with correct URLs. Now CreeperFramework’s version number and revision date is visible in meta-headers.

All this on my development server, nothing deployed yet.

No responses yet

Translations

Jul 14 2010 Published by under MediaCreeper

I’ve built MediaCreeper with support for multiple languages, at the moment there is two complete language definitions; Swedish and English.

These are the two languages I can handle and speak, read and write. English will be enough to reach out to the world but I can imagine that there will be requests for German, French… Mandarin?

It’s quite simple really, all strings (yes, strings) of all languages are kept in a single XML-document (will be split into several if necessary) that is read when a page is prepared for rendering (in CreeperFramework’s TemplateEngine), a selector is used to point at a language out of several — so it’s easy to add more.

The few things needed to create a new language/translation is a language-template file (an XML-fragment really) and a good text editor that can handle UTF-8 encoding properly.

I think it is even possible to handle RTL-languages, I have nevere tried it, all the bits should be there though.

Interested in translating MediaCreeper into another language? Write me a mail, mediacreeper (-at-) gmail.com

No responses yet