Hacking blogger's atom.xml feed

So, in the past couple of days, one of my projects has been to figure out why the atom.xml feed for Goodnight Burbank was not listing all 30 of the episodes - we only could see 25 episodes, from the most recent modified to the 25th recently modified. This only seemed to be happening with the advent of the new blogger - so, I thought there must be people who have had this problem before.

Surprisingly, I discovered people do not seem to be having this problem - at least not asking the question. For instance:

1. When I contacted my friendly neighborhood blogger support staff, they pointed me to the google Reader and proved to me that the feed was actually fully there (which, when I looked, it was!). So, obviously, there was nothing wrong - even if I and other feedreaders could not see past the 25th most recently modified post.
2. When I contacted my engineering friends at google, I was told that "Feeds are not supposed to have all of the episodes/posts on a single feed. Usually atom feeds only have 10 posts/episodes. You are lucky to have 25!" And then when I pointed to the episodes I saw in the google Reader, I was told that, more than likely, there is a cache that exists and it is already indexed.

This had me frustrated. Obviously, the constraints of programming were out of my hands, since blogger is being developed and maintained by engineers at google. So, taking one of my friend's advice, I decided to begin the process of migrating the site to a WordPress installation.

Wordpress.com or my own?
This was a question, since the hosted version of Wordpress is managed by others - but I noticed that modification of templates and other aspects would be a bit of a problem (not sure how much it would cost). So, I downloaded a version of Wordpress 2.1.2 and tried the install.

In following the famous "5 Minute Install Instructions", I learned the the zipped version of the file seemed to be missing the special pages (like wp-config-special.php and install.php), so I then downloaded the .tar.gz version (fortunately, I was on a Linux server). And, true to the word, once all was in place - it was much less than 5 minutes.

Migrating from blogger to Wordpress
Now, to migrate off of the blogger platform and onto the new Wordpress install. But, how to do it? That took a little bit of detective work, and I found Ady's plugin which I tried to follow the instructions, but could not figure out - after I installed and activated it, what to do next?

Well, being an old hacker, I accidentally found the "Plugin Editor" and started reading the code - and discovered the Management options that were now added to my Wordpress installation. So, I went to it and, after reading Ady's post with greater understanding, I brought down the old blog. But, what surprised me was that the script found all 30 posts, not just the 25 that I could see on the atom.xml feed. This was puzzling.

So, again, I went back into the code and discovered an interesting snippet that Ady had recovered from somewhere - and it did not look at all like what I expected from google. The link now says:

http://goodnightburbank.blogspot.com/feeds/posts/full?alt=rss&max-results=&start-index=1

where Ady's code manually figured out the maximum number of posts by paging through the XML feed itself with a loop. Interestingly, I had seen a similar feed link when I looked at my google Reader subscription, I just thought it was a translator for the atom.xml request.

Help Pages?
Where was this? I spent over three weeks going through this. And, scouring the blogspot help pages, there was nothing listed - including this new feed format as shown above.

Well, to give credit where credit is due, I must point to Sumesh of Digital Dreams for his post on Hacking Blogger which was the first post I saw that explained the new format.

Note to Blogger Team - can you please publish these APIs (name/value pairs) in a way that is easier to find and reference? Especially since I would rather not have to manually change the number of posts on my feed request when I submit to feedburner for Goodnight Burbank.

Conclusion - sticking with blogger...for now
So, when all was said and done, it was easier to stick with blogger than to set up the Wordpress site - and make sure that the feed I am feeding to feedburner is correctly modified. Will I change over some other time? Maybe. But not tonight. I have more work to do.

Kyle Shannon speaks on Web 2.0 and social media

This evening, in my Emerging Programming Paradigms class at Polytechnic, the topic was social media and how it will impact the business processes that my students are facing now and in the coming days. Fortunately for me, a friend came to discuss his perception of Web 2.0 and social media by putting it into a more corporate context than I could.

Below is a live-blogging of Kyle's presentation:

What is Web 2.0 and social media and why should you care?
by Kyle Shannon

Web 1.0 companies that survived the bubble is the ones that have been able to "atomize" their services and allow for syndication of functionality to other locations. The goal has been to make people become the advocates for the company by offering services that can be syndicated (think google AdWords, Amazon Affiliates, etcetera).

Web 1.0 world - one person with a blog could wreak a lot of havoc. Remember the Intel chip problem - one blog post, one Excel sheet, mailed to all of the blog's fan base, eventually got to a reporter and from there, it made a story - Intel now has to recall the processors.

RSS allows for syndication of content - which can now go alongside of the atomized services - which is where the eyeballs are.

Used to be "there" - website is not going to go away - functionality is going to go wherever they are. It will be where the functionality is needed for others.

Rather than thinking of Web 2.0 as the next set of APIs, just go out and have fun with it. Check some of the sites that show Web 2.0 companies:

• programmableweb.com
• mashable.com

Homework assignment - go to programmableweb.com - lists all of the mashup sites and all of the APIs that are currently available and look at the company names that are providing APIs. Try to find one that existed before 1990. They are all small companies.

Best Web 2.0 applications - go play with this stuff - get a sense of "where these guy's heads are at". The dark side is users are having fun and building up expectations which is experienced in the Web 2.0 offerings. "Why doesn't Acme Co. do this?" Think of it as an end-user perspective.

Examples of business applications: zillow.com - combines databases and google Maps to add functionality that neither had before.

Kyle believes that microtransactions become more important - ringtone, widgets that does something that I am willing to pay 10c

As entertaining as watching cat videos (140K videos of cats on YouTube), there is still a large amount of amateur content that might have a higher level of creation that has yet been untapped.

Kyle also believes that the Web will become much more visual - convergence, blending of entertainment and commerce and personal content. The Web will become a more immersive experience...

Think back to desktop publishing - a lot of people who could not design, and most part went back to the designers. A number of the amateurs (minor league) of content developers will form.

His suggestion: Get on the beta list of Joost - doing for television what they did for telephony (using the same system). Most people look at it and say, "Oh - this is TV!" But you can also open widgets which have chat and information and commerce and so on.

Yi-Tan Tech Conference Call - Climate Change

Jerry: "It's not Easy Being Green"

Climate Change - speaker: John Holdren

Pip on the issue: for the past couple of years, have been watching the issue - watching/evaluating monumental shifts from thought to idea to action. One of the statistics last year was the number of mentions of global warming and alternative energy - # of mentions of global warming and alternative energy was flat in 2005, 2006 altNRG went from 70% and global change 260% - cement is happening.

Effect for business re business change is going to be more impactful on business - down to the minutia (washing towels). When are clients going to feel "icky" when you fly around the world for the client call.

Political tipping point on climate changes - including the drumbeat of science (every day an article showing changes are coming out more frequently) being reinforced on personal experiences and reinforced on TV and the blogs. Rapidly changing attitudes in the business world and showing in the ads - and the religious change. US government changing POV from pre 2005 (not going to sign Kyoto if China not there) to post-2005 (Senate resolution very different).

Economic change - will finally get action in 2009 - some form of carbon tax - it is now an irresistible force. Systems effect is changing - first order effects (geophysical), second order (biophysical) to third order (disease vectors) to four order (how it impacts US). Blerb from Peter Schwartz.

How is this impacting us today? Energy economy climate nexus - sits in the middle of economics, politics, technology and others. Can not understand this problem unless we look at the problem from all sides of the issue. The more of those pieces one thinks about, the more challenging this problem becomes. Greater the number of areas of knowledge grows the level of concern - this is systematic symptoms.

What organizations are getting traction on this now? John: in the private sector - this challenge is not a threat, but an opportunity. GE, Alcoa, Duke Energy, etc coming out claiming a need to have the government regulate greenhouse gases. They see that government policy is inevitable, they also see that there is an opportunity. There is an overall change in the business community. Pulls the run out from the naysayers. GlobalWarmingRealEstate.com - a wiki spoof site.

David Isen: Some of the positive feedback loops that could accelerate the issues? John: Jim Hansen is taking serious action - issues are becoming more likely. One of the feedback effects is the melting of the ice in the Artic, then the highly reflective ice is replaced by highly absorbing water - which is a positive feedback look accelerating the temperature. Fate of the carbon in the permafrost and tundra as there is in the atmosphere. If there is a major defrosting of the tundra, we could have a pulse of drastic CO2 content and methane in the atmosphere which could have a pulse effect.

Howard: disproving the skeptics. John: skeptics are in retreat. The only danger is that skeptics can make a comment in one sentence, where it takes paragraphs to reverse the comment. But now, people are behind it. Pip: people are

Judy: cold futures - more desirable 2025: how can we weather global climate change. Came up with a dozen strategies. How can we actually do the things we need to do - Millennium Development Goals. John: for the UN, climate change - Confronting Climate Change on the UN website. Talks about the synergisms and the win-win tactics that can be applied to adapt and accelerate the attainment of the Millennium Development Goals FAST. Huge amount of leverage in energy end use - REDUCE the amount of energy we use.

Example of a win-win: reducing the black soot emissions from two-stroke and burning - BIG impact on the action. Immediate benefits on health.

John summary: must address this problem - it will not wreck the economy to fix it, it will if we do not.

Pip: consuming less is a big deal - without making a big difference. Corporates have a huge impact. Can control the moral level - they are not all BAD people - they can help the company and the shareholders. Governments - thinking either or versus and. And helping people who need it now (thinking globally).

Lots of people have been trying to make social issues into the business. GRI - confluence of libertarian (what is needed for the business) and the measure of success of a business in the long-run.

Carbon-0ffsets - very difficult for consumers to determine how to consume less. Intrigued by the idea of buying your way to carbon neutrality. People find it difficult as a way to contribute or to make happen. A lot of people do not want to put into the switching costs, they are willing to buy their way out of them. People can buy "green power" which supports geothermal, solar and wind which can provide a revenue stream to build up green power.

A book I read: Power to the People, by Vijay V. Vaitheeswaran. Gives a lot of thought into issues at hand - including distributed power generation and the issues people have been addressing. Vijay is switching over from power to other topics, but is still at the Economist.

Don't overlook the people in the second half of their life - they want to be able to give back both in business and in popular culture. A whole spectra of activities - boomers may want to be remembered as the peopel who made the first effort in solving this problem.

American Chemical Society - sustainability of food and water, various scientific organizations are building momentum.

Yi-Tan : Software Development Challenges

Scott Rosenberg - why isn't software easy?

Scope of the project: tell the story of six different projects. Is software as hard as it seems and why?

Complexity of the project was so difficult - almost impossible. Became a story of a project that was in trouble and incomplete - and that is the way it happened.

Jerry: spellbound

Why are projects so difficult to manage, why are budgets so overrun? He followed Chandler from 2003 to 2005/6. Just around the time of Web 2.0 happening. Chandler knew this was happening which caused them to change course. Also followed

Web 2.0 is proof of concepts - almost all in perpetual beta state. When it becomes "successful" - big one buys it up and then either improves it or destroys it (e.g. myspace technical challenges)

Broadest conclusion - be humble - there are huge swamps and trouble ahead. Know the history, know what has happened - the biggest pragmatic take-away - work incrementally. Get som ething running quickly, feedback loop with users, etcera. Grand plan is almost impossible thing to complete in software.

People seem to have the "principle of exception" - yes, that is true, but! - people tend to exempt themselves from the reality around.

Al Chang - talking about browsers - client software is a challenge. Very important to pick the "right model".

Scott: when you select a form (like a sonnet in poetry), you can make it happen. Free verse - you get lost.

Al - when you think as a screenplay, then all of the players have to play to the script. Novels are a challenge - because you can keep writing and it never has to end.

Xanadu project - ever receding nightmare.

Scott: Firefox project. Started in 1997-98, years to get started. Many people wrote them off. They finally took it in pieces and took big bites and stayed with it. Took them much, much longer. Question - why?

Jerry: YAGNI - you aren't going to need it. Premature optimization is the root of all evil.

Scott: axe sharpening - taking forever to get your tools, setting up the right environment, right infrastructure - can almost become a job in of themselves. Often becomes a prject in and of themselves. Do not spend all of the time focusing on cleaning - and just do.

Al: search engine - make it simple. Mind also has a hard time handling the scale issue.

Scott: Quantity changes quality - as you scale/grow - does not grow as predictably. The dynamics of systems and communities change the scale.

Jerry: what were the major issues in the Chandler project? Scott: major personality clashes and major "bets" on the project. For example, they chose to use python for the project - not unheard of, but definitely unorthodox. They believed the gain would be from the build versus the dearth of libraries. Today, it is helping much better.

Jerry: AI intelligence with common sense reasoning project - the project hopefully will bootstrap. Scott: the naysayers are right until you prove them wrong.

Jerry: Apollo computers - problems with timing. Scott: software tied to hardware. Software-only project - is infinitely malleable. But, in theory, you can keep changing the platform (hardware) if you want to take care of the platforms. Agile changes.

When you have a lot of users - then you have the concept of legacy and you do not want to break compatibility.

Bill Seitz: every architectural decisions made in the beginning ends up causing the challenge later. Scott: fear is valuable up to a point.

Al: when things are infinitely malleable, people tend to

All about constraint management, rather than getting a particular project done. Used the constraints to make decisions.

Had to leave the call early.