(Very) beta release of the WattsBuster(tm) app

Discussion in 'Environment & Conservation' started by caerbannog, Oct 19, 2012.

  1. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    (Disclaimer -- this is not a polished "end-user" product; think of it as a rough "proof of concept" prototype -- making this a truly user-friendly app would require more time and expertise than I currently have to spare ;) )

    The WattsBuster(tm) package allows users to "roll their own" global-average temperature results by pointing/clicking on GHCN temperature stations on a global map. As additional stations are selected, WattsBuster(tm) updates its global-average temperature calculations "on the fly" with data from the newly-selected stations.

    The WattsBuster(tm) package can be installed/configured to run on Linux, BSD/OS-X, and Windows/Cygwin-X systems. Installation/configuration is easiest on Linux platforms (OS-X and Windows platforms require additional prep work -- details below).

    WattsBuster(tm) borrows heavily from several open-source software projects: Quantum GIS, gnuplot, and a neat C++ TCP socket class that I found on-line.

    Experimentation with WattsBuster(tm) will reveal that the NASA/GISS global-warming results can be confirmed with raw or "homogenized" data from as few as 30 or so globally-scattered temperature stations, rural *or* urban.

    WattsBuster(tm) installation/configuration is recommended for folks who are comfortable working in an "old-school" Unix/Linux command-line environment (or for those who are willing to learn how to work in a Unix/Linux command-line environment).

    Here are a couple of images that show WattsBuster(tm) in action:

    The first image shows the WattsBuster(tm) "front-end" (QGIS with a customized plugin), where stations are selected via mouse-clicks (selected stations are shown in yellow):

    [​IMG]


    The second image shows the WattsBuster(tm) "back-end", where global-average results computed from the selected stations are displayed:

    [​IMG]

    In the "back-end" display shown above, two plots are shown.

    The upper plot shows results for 50 randomly-selected "long-record" (i.e. with pre-1885 data and/or post-2010 data) temperature stations. Results are shown for raw (red) *and* homogenized (green) data. The corresponding NASA/GISS results are shown in dark blue. As you can see, the long-term global-temperature trends produced from the raw and homogenized data (50 stations) match the official NASA/GISS long-term temperature trend (produced from thousands of stations) quite well.

    The lower plot shows how many of the selected stations actually reported data for any given year. (NOTE: a station that reported data for 6 months in a given year is counted as "half a station" for that year). As you can see, 40-45 stations reported data for most years from 1960 to 1980, while 20 or fewer of the selected stations reported data prior to 1900.

    Experimentation with WattsBuster(tm) will reveal that for time-periods where you have at least 30 or so globally-scattered stations reporting data, the WattsBuster(tm) global-warming trend will match the official NASA/GISS warming trend very nicely (with raw *or* homogenized data from rural *or* urban stations).

    With WattsBuster(tm), you can confirm for yourselves the following:

    1) Rural and urban stations produce very similar results (thereby debunking Anthony Watts' claims about UHI).

    2) Raw and homogenized data produce similar warming trends, thereby debunking Watts' claims that much/most of the warming is due to "homogenization". (Yes, homogenized data will often produce a slightly larger warming trend than will raw data, but the differences between the raw and homogenized data results will be small).

    3) The NASA/GISS results can be reproduced very closely from *raw* data. In fact, the NASA/GISS results will generally match the WattsBuster(tm) raw data results a bit more closely than the WattsBuster(tm) homogenized data results. This debunks the Watts' claims about the NASA/GISS results relying on data "homogenization". WattsBuster(tm) computes global-average results via a straightforward averaging procedure that does not incorporate *any* data homogenization/adjustment steps -- that can be verified by inspecting the WattsBuster(tm) source-code.

    4) WattsBuster(tm) can be used to debunk Watts' "station dropout" claim. Watts has claimed that because the number of temperature stations used by NASA has "dropped" from about 6000 in the 1970's to about 1500 now, the temperature results have somehow been "skewed". But with WattsBuster(tm), you can reproduce the NASA/GISS global-warming results very nicely from just a *few dozen* stations (none of which has been "dropped"), thereby completely debunking Watts' "dropped stations" claim.

    This package should be considered to be sort of a "developers' release". In its current state, it is not very "user friendly" (by today's standards, at least).

    But if you are comfortable in a Unix command-line environment (i.e. comfortable installing/using compilers such as g++, editors such as vi/gedit/emacs, etc.), you shouldn't have too much trouble getting WattsBuster(tm) up and running on your laptop.

    OS-X users will need to install X11 and XCode (available for free from Apple) and then compile gnuplot from source (source-code available at gnuplot.info). They will also need to edit a C++ header file slightly prior to compilation (details available in the WattsBuster(tm) README files).

    Windows users will need to install Cygwin/X (available at x.cygwin.com) -- g++ and gnuplot must be included in the Cygwin installation.

    An alternative is to use VirtualBox (virtualbox.org) to install/run Linux in a virtual machine on your OS-X or Windows system. WattsBuster(tm) can then be installed on the Linux virtual machine. There are the usual O/S installation/configuration issues with this approach, however -- if you don't have prior experience installing/configuring VirtualBox and Linux, this is not recommended.

    If you are a bit handy with network configuration (i.e. know how to determine IP addresses, configure firewalls to open ports, etc.), you can run the QGIS "front end" and global-temperature computation "back end" modules on different computers on a network -- the "front end" and "back end" communicate via a TCP connection.

    Additional details (including installation/operation instructions and links to all code and data) are available here.

    One final note -- on a Linux system test-install, the QGIS installation package set the owner of the QGIS plugin directory to root (superuser). The ownership of that directory (~/.qgis) had to be reassigned to the current user via "sudo chown" to allow QGIS plugins to be installed (just a quick heads-up for Linux users out there).
     
  2. Taxcutter

    Taxcutter New Member

    Joined:
    Dec 18, 2011
    Messages:
    20,847
    Likes Received:
    188
    Trophy Points:
    0
    Garbage in...garbage out.
     
  3. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    Challenge to "Taxcutter" -- Download the software/data from http://tinyurl.com/WattsBusterProject. Install and run it per the instructions provided. (The zip file at the above link contains all code and raw/homogenized temperature data, in one convenient package.)

    Anyone who has the technical chops to pass judgement on the work of climate scientists should have no trouble at all installing and running the software.

    Create an image file of the output results, upload it to imageshack.com, and post a link to it here.

    The above challenge applies to other skeptics here as well -- please don't reply in this thread until you've made an honest attempt to install and run the software.


    For convenience, here's a quick system-prep summary for Windows and OS-X users.

    1) Windows and OS-X: Go to www.qgis.org and install QGIS for your system (it's a quick point/click operation).

    2) Windows users: Go to x.cygwin.com and install Cygwin/X -- make sure that you select g++ and gnuplot when
    running the Cygwin/X package manager. (Mostly a point/click operation, with a bit of searching/navigation
    to find the g++/gnuplot software).

    3) OS-X users: Install g++ and X11 for OS-X (start here for g++: https://github.com/kennethreitz/osx-gcc-installer; Google
    a bit to find X11 for OS-X -- it's all free). Then go to gnuplot.info, download the gnuplot source-code, unpack it and
    compile/install per the instructions provided -- gnuplot compiles very cleanly on OS-X with g++.
     
  4. Windigo

    Windigo Banned

    Joined:
    Jul 8, 2008
    Messages:
    15,026
    Likes Received:
    1,139
    Trophy Points:
    113
    As I said before V3 raw is not raw. It just means that the GCHN has not yet applied their adjustments to it. It is already heavily adjusted by the individual national weather services. That was plainly obvious when you first started plagiarizing the work done by skeptics and alarmists on airvent and it still is today. The match between V3 raw and V3 adjusted is no secret and has been the subject of much discussion on airvent and is well known by watts and others long before you came on the scene spouting your plagiarized tripe.

    If you want to find out about caerbannog just go to the airvent at http://noconsensus.wordpress.com/ and you can see where he got most of his plagiarized "work".
     
  5. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    Folks, I've never been to airvent. What I did was implement my own gridding/averaging routine (from scratch -- I never even looked at anyone else's code) based on very generic, publicly-available descriptions of the gridding/averaging process. Furthermore, if you inspect the code/comments, you will see that my gridding implementation differs from implementations used by others (I had specific reasons for doing this).

    What Windigo is suggesting is that the NWS offices of some 180 nations around the world must somehow be "coordinating" their data adjustments in real time so that no matter what subset of stations is processed, you will get similar warming results every time (provided that you have reasonable global coverage).

    Remember folks, in addition to *monthly* GHCN data, you can download the *daily* data.

    So any *adjustments* supposedly made to the raw data by all those 180 NWS offices have to be made "on the fly" before new data goes out *each day*. These "adjustments" have to be coordinated amongst the 180 independent nations *on a daily basis* (i.e. in "real time") to ensure that stations from random subsets of nations all produce a consistent global-warming trend.

    Anyone who buys into wacky conspiracy nonsense like that, folks, is simply beyond delusional.

    Folks, go ahead and try WattsBuster(tm) for yourselves -- plot raw vs. adjusted data for individual stations -- you will often see significant differences between raw and adjusted data for individual stations (some adjustments are up, others down).

    Then start averaging stations randomly from all around the world -- average together data from random subsets of nations. When you average enough stations together ("enough" being no more than a few dozen), the raw and adjusted results will converge pretty closely to the official NASA results, in spite of the fact that for individual stations, raw and adjusted data may differ significantly. You will see this no matter *what* subset of stations you choose.

    In fact, for all time periods where you have decent global coverage (i.e. a couple of dozen stations or more, scattered around the world), you will see raw *and* adjusted data global-average temperature trend that lines up with the NASA results quite nicely. Only in the fevered mind of the most delusional loonball could all those 180 independent nations coordinate their secret "raw data" adjustments in real time to ensure "random station subset" results consistent with both the NASA results and the results you get from processing the adjusted GHCN data.
     
  6. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    Just following up with another item...

    When someone accuses you committing an act of plagiarism when you post code that implements a straightforward averaging routine that could be taught to high-schoolers, then that tells you a lot about that certain someone's grasp of the subject material in question...
     
  7. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    For those folks out there who might be interested in seeing WattsBuster(tm) in action but have better things to do than than muck around in a user-hostile Unix environment trying to get it set up and running, I've created an animated GIF file that shows WattsBuster(tm) at work.

    I created the file by capturing snapshots of the WattsBuster(tm) output, starting with data from a single rural station and then adding rural stations one at a time, up to 40 stations.

    I selected rural stations at random, with only one condition -- the stations had to have data going back to 1885 or earlier, or data up until 2010 or later. This ensured that I had stations with long-enough data histories to compute decent results for the entire time-span 1885 to the present.

    Once again, the results were computed from rural stations, selected at random (I don't even remember what stations I clicked on). The animation is the result of a "first try" run. I didn't cherry-pick the "best result" from multiple attempts -- I just posted what I got from the first attempt.

    The upper data plot in the GIF display shows raw data results (red) and "homogenized" data results (green) plotted against the official NASA/GISS results (dark blue).

    The lower plot shows the number of selected stations that reported data for any given year. (Raw data station count in red, adjusted data station count in green). You can also see the total station count in the legend in the upper-right corner of the lower plot (starting at 1, going to 40).

    Each new frame in the animation shows the results updated with each new random station in the order that it was selected. (The update rate is 1 frame per second). As you watch the animation, you will see a nice visual demonstration of how few stations it takes -- with either "homogenized/adjusted" *or* raw data -- to replicate the warming seen in the NASA/GISS results (which were computed from *thousands* of stations).

    You can download it from: https://docs.google.com/open?id=0B0pXYsr8qYS6aV94WTdoa2E4Umc and view the animation right in your browser.

    (It's a big file -- I didn't embed it because the web-master would ban me for life if I did ;) )

    As you can see from the animation, those infamous data "homogenization/adjustment" steps that skeptics keep going on about pretty much cancel each other out once you've averaged a dozen or two stations together. The fact that global-warming "skeptics" were unable to figure this out in all the years that they've been complaining about temperature data adjustments should tell you all you need to know about them (the skeptics, not the adjustments).
     
  8. Windigo

    Windigo Banned

    Joined:
    Jul 8, 2008
    Messages:
    15,026
    Likes Received:
    1,139
    Trophy Points:
    113
    When you first jumped on the internet saying that you had dont something no one else has done before when Jeff Id, RomanM, Nick Stokes and many others had already done the same thing says a lot about you.

    Your "work" is nothing new. Its been dont multiple times by multiple people years before you did. And I've also seen you rip off Von Storch's criticism of M&M near verbatim. The ball drops once it might be a coincidence but when I saw that you had also ripped off Von Stroch that synched it for me. Your are just a plagiarist.
     
  9. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0

    Folks,

    I've always been scrupulous about crediting others for material that I've borrowed from them. Nothing has been plagiarized, stolen, whatever. Any ideas, insights that I have represented as my own in posts here or elsewhere were all, in fact, my own.

    Look through the WattsBuster package -- I was very careful to give proper credit to the authors of code that I borrowed for the project.

    Regarding analysis of temperature data, I've always taken care to explain that my approach is a much less-sophisticated (i.e. "dumbed down", if you will) version of the processing approaches taken by Nick Stokes, Tamino, and others. I've made that quite clear in my posts here and elsewhere. I never claimed to have done anything that others had not done (better).

    What I've "brought to the table" here is a straightforward bit of software that is easy to build and use, and implements an algorithm so straightforward that there's no place in it to "hide" any data manipulation.

    I've always made available all the code I wrote/used to generate results I've posted here or elsewhere. IOW, I've always played with my cards face-up on the table.

    As for my critique of M&M, that derived from insights from extensive use of the SVD algorithm in underwater acoustic array processing software that I wrote. I saw a major problem with the M&M approach before I had read any of Von Storch's material -- a problem that was obvious to me based on my own experience with the SVD algorithm at my place of employment, and a problem that should be obvious to *anyone* who understands how to interpret the output the SVD algorithm.

    I'm won't say anything about Windigo (if I were to describe him/her accurately here, I'd rightfully be banned) except that I would like invite folks to compare the content/honesty of my posts here with Windigo's. And with that, Windigo becomes the first entry on my ignore list.
     
  10. Windigo

    Windigo Banned

    Joined:
    Jul 8, 2008
    Messages:
    15,026
    Likes Received:
    1,139
    Trophy Points:
    113
    You could start by ceasing to lie to your audience. The V3 raw is not "raw". It is only raw in the sense that the GISS hasn't done their own adjustments to it yet. This is well known. Anyone who has paid any attention to reconstructions done on airvent would know that. You know that. After all you stole it from there. I will repeat again, from you plagiarized reconstructions stolen from airvent, to your near verbatim ripoff of Von Stroch you are just a fraudulent plagiarist! All you do is put your own spin on someone else work.
     
  11. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    I've got some new material that a few of the "reasonable regulars" here might be interested in...

    bowerbird, politicalcenter, livefree, mannieD, poor debater .... please check your PM's. ;)
     
  12. caerbannog

    caerbannog Banned

    Joined:
    Jun 17, 2011
    Messages:
    190
    Likes Received:
    7
    Trophy Points:
    0
    Just saw this little Christmas goodie over at skepticalscience.com (if it appears as a "thumbnail" image, click on it to view the full-sized version):

    north_pole_cartoon_med.jpg

    And those who are interested in a science-oriented virtual "stocking stuffer" might want to check this out. To figure out what to do with it, just RTFM (read the fantastic material) that's included at no extra charge.
     

Share This Page