Computer Science Teachers Days

Computer Science Teacher Outreach (Summer 2011)

Teachers and graduate students learning about Arduino

Resources & Links

Physical Computing

  1. Processing – the Processing site contains downloads, tutorials and manuals
  2. Arduino – Information about the Arduino microcontroller can be found at many locations, but the http://arduino.cc/en website is a definitive starting point
  3. Arduino and Processing - Many art projects using physical computing and the Processing language; this resource provides information on accessing an Arudino from within Processing
  4. This Digital Oscilloscope provides a concrete example of what can be done with Processing and an Arduino
  5. We’re distributing the SparkFun Inventor’s Kit for Arduino at the meeting
  6. Google has announced the “Android Open Accessory Kit“, which allows an Arduino platform to connect to Google Android phones for applications that involve mobile computing and physical sensors.
  7. The Scribbler robot is developed by Parallax
  8. The Institute for Personal Robotics in Education has considerable information on using the Scribbler (and add-ons) for education, both at the college level and K-12.

HTML5 and JavaScript resources

At Google, we looked at the http://www.html5rocks.com website, which is a great resource for learning about HTLM5.

This website ( http://bit.ly/mHVZiZ ) lists “66+ game development engines in JavaScript and HTML5″ – most of these are free or open source and some are commercial.

Tim provided the HTML5 demonstration that does uses a laptop tilt sensor to pan an image – you can download that here –

html5-demo-tilt-sensor

Accessing and Visualizing Data from the Web

These are notes to supplement the eCSite workshop session on June 10, 2011. We present some examples to illustrate some of the techniques.

The picture below provides a kind of map of some of the paths that link various information sources on the Web to tools that can provide ways of processing and visualizing data. No doubt other paths are possible, and more paths and tools will become available over time, but one can already see that there are many ways to take data from the Web and display it, often with remarkably little work.

Example: Showing the locations of mine plants in the US

See the fusion table containing the data. Select Map on the Visualize menu to view on a map.

Procedure

  1. Download from data.gov as KML (http://explore.data.gov/Geography-and-Environment/Active-Mines-and-Mineral-Plants-in-the-US/ir5y-579r). KML is a data format for location information.
  2. Create Fusion Table from downloaded KML file: On the New table menu, select Import table and use the dialog to create your new table from the downloaded KML file on your computer.
  3. Visualize as map: select Map on Visualize menu.
  4. Use controls on Map viewer to zoom in on an area of interest, for example Boulder, and click on a marker for detail.

Comment: This is what happens when the planets are completely aligned: everything just works!

Example: Boulder County Restaurant Inspections

In the mine locations example we are using data from a downloaded data file. It is also possible to access information directly from a Web page, as this next example shows. As this example also shows, a good deal more fiddling can be needed to get things to work.

Click to see the finished spreadsheet for this example. Click to see the fusion table for the example.

In outline, here is the process:

  • Pull the data from a Web page into a spreadsheet
  • Build a Fusion Table from the spreadsheet
  • Visualize the Fusion Table as a map, as in the previous example

Method notes

The data we want are available at http://www.bouldercounty.org/help/health/disease/pages/restaurantinspect.aspx. To pull the data into a spreadsheet, we need to know the URL from which we are getting them, and we need to know which list or table on that page the data are in. Sometimes that information will be easy to find: look at the URL of the page where you see the data, and View Source for that page to find what list or table the data are in.

But in this case, the data are not actually in the page we are looking at, but rather in a frame that is shown as part of the page. That means that we don’t actually have the URL that we need, which is that of the source of the frame, not that or the enclosing frame, and we can’t see the HTML for the frame to verify which table we want.

To solve this problem, select the frame in your browser view, and ask the browser to display just the frame in a separate window (the details of this may differ among browsers). Doing that in this case we see the URL http://www.decadeonline.com/results.phtml?agency=BOU&coding=BCPH_RATING&rating=u&forceresults=1&offset=0&businessname=&businessstreet=&city=&zip=&facilityid=&soundslike=&sort=FACILITY_NAME for the frame. This shows, not surprisingly, that the Boulder County data are actually being served from a commercial service provider, not a county server. We can now View Source for the frame, and see that there is only one table on the page, so that’s the one we want to refer to in our spreadsheet.

In the example, we can also see that while we want all of the Unsatisfactory ratings, the Web page is only showing the first 50, in alphabetical order. We’ll be able to solve that problem with a little work, but let’s first work through how we will show the data this page shows. Then we’ll come back and do the dirty work to get all of the ratings.

Here’s the process:

  1. Create a new Google spreadsheet
  2. Place this formula in a cell somewhere towards the upper left: =importHtml(“http://www.decadeonline.com/results.phtml?agency=BOU&coding=BCPH_RATING&rating=u&forceresults=1&offset=0&businessname=&I7businessstreet=&city=&zip=&facilityid=&soundslike=&sort=FACILITY_NAME”,”table”,0)What that does is import data from the indicated URL. If all goes well, you will see the data you want filled in on your spreadsheet.
  3. Give your spreadsheet a name, and save it.
  4. Go to http://www.google.com/fusiontables and create a new Fusion Table by selecting New table and Import table. Indicate that you want to import from a Google spreadsheet, and browse to the spreadsheet you’ve just created. Leave all of the columns checked.
  5. You’ll see your new Fusion Table with the data in it. Choose Visualize and Map to see it on a map. You can now explore the data using the Google Map controls, as in the previous example.

Unfortunately, all is not well on our map. You’ll see some dots well outside Boulder county. What’s going on?

A look at the addresses tells the tale. Because all of the inspections are in Boulder County, and Boulder County is wholly within the state of Colorado, the state isn’t included in the data in the database! Google Maps tries to cope anyway, and it does remarkably well, somehow working out that nearly all of the locations are in fact in Colorado. But it misses on some.

Fixing this problem isn’t completely easy. Some of the approaches you might try, such as putting formulae into the data cells to concatenate CO onto each address, won’t work, because the importHTML formula that pulls the data from the spreadsheet places its own formulas in these cells. What we did was to copy the data area of the spreadsheet into a new area to the right, being careful NOT to copy the cell with the importHtml formula in it. We put in appropriate headings for the new columns, and then put a formula in the top address cell that looks like this:=CONCAT(C6,” CO”), where C6 is the cell that contains the original address data for the restaurant in that row. Copying that formula down the column fills in the augmented address for each restaurant, as desired, and because these cells to the right aren’t affected by the importHtml formula, our corrections don’t get overwritten.

Now you can create a new fusion table, as before, being careful to check the new address column and uncheck the old address column when doing the import.

Most of the spurious dots are now gone, but a close look will show that there are still some dots showing up outside Boulder County. This is due to some addresses having “UN” in them, which probably stands for “unincorporated”, but throws off Google Maps. Since there aren’t many of these, we found it easiest to manually edit the corresponding addresses in the Fusion Table: if you click on an address in the table you can just edit it.

Seeing All the Data

We mentioned earlier that the Web frame we have extracted the data from only has the first 50 records. Some experimentation reveals that to get the next 50, one has to change the “offset” value in the URL to http://www.decadeonline.com/results.phtml?agency=BOU&coding=BCPH_RATING&rating=u&forceresults=1&offset=51&businessname=&I7businessstreet=&city=&zip=&facilityid=&soundslike=&sort=FACILITY_NAME. Now we can put another insertHtml formula into our spreadsheet, below our last data record, to pull the next 50 records into the spreadsheet.

We also have to make a copy of the new data, and copy our concat formula down the column, to put the state information into the new addresses, as we did with the old ones.

When we now create a new fusion table from the expanded spreadsheet, we are almost finished. But unfortunately the line of the spreadsheet on which we placed the new importHTML formula comes across into the fusion table as a bogus record. Fortunately we can just delete that record from the fusion table before visualizing the data in a map.

Survey and charting with Google spreadsheets.

Google has created a “form” tool that allows survey or similar data to be gathered by asking questions on a Web page. Here’s the process for creating an example. Here is the survey:

Loading…

The associated spreadsheet is at this link to the spreadsheet. [The example survey has two items contrasting categorical and relational belonging, inspired by Ji, L.J., Zhang, Z., & Nisbett, R.E. (2004) Is it Culture, or is it language? Examination of language effects in cross-cultural research on categorization. Journal of Personality and Social Psychology, 87(1), 57-65.]

  1. Create a Google Form, by choosing Form on the Create new menu at https://docs.google.com. The create dialog talks you through choosing questions of various forms (multiple choice, text input, and so on).
  2. When you finish the form the system shows you a piece of html that you can embed in any Web page that links to the survey. Or you can email the link to participants, from the form creation page.
  3. A spreadsheet to hold the survey results is automatically created, and there is a link to it is included in the form creation dialog. When participants respond to the survey, records representing their responses are automatically collected in the spreadsheet.
  4. In the example we used spreadsheet formulae to convert participants’ textual survey responses to numerical form, to ease further analysis. You can see these formulae in the example spreadsheet.
  5. We then inserted a scatter chart into the spreadsheet, as can be seen in the example.

Example: A Bit of Google Apps Script

See example spreadsheet.

Google spreadsheets can have Javascript programs attached to them that can perform a wide range of function. In this simple example the program extracts a zip code from the spreadsheet, looks up the current temperature on weather.com, and inserts the temperature into the spreadsheet.

Process:

  1. Create spreadsheet and put in appropriate headings in the A1 and B1 cells.
  2. Select Script editor on the Tools menu.
  3. Enter the desired Javascript code. In the example the code is:
    function example() { 

    var ss = SpreadsheetApp.getActiveSpreadsheet();

    var sheet = ss.getSheets()[0];

    var zip=sheet.getRange(“A2″).getValue()

    Browser.msgBox(“The Zip is: ” + zip);

    var data = UrlFetchApp.fetch(“http://www.weather.com/weather/today/”+zip).getContentText();

    var start=data.indexOf(“realTemp:”)

    var temp=data.slice(start).split(‘”‘)[1]

    sheet.getRange(“B2″).setValue(temp)

    }

  4. Put a zip code in cell A2.
  5. Press Run on the script editor page.
  6. Respond to the message box that displays the zip code.
  7. The current temperature should appear in cell B2

How It Works

The first two lines of the code get access to the current sheet of the current spreadsheet. The next line gets the value in cell A2, and the following line puts up a message box with the zip code in it (just to check the operation of the code.) The following line constructs a URL by concatenating the zip code onto a base URL for weather.com, copied from an example looked up manually. The next two lines find where in the page the word “realTemp” appears (this word was found by examing the page source), and then breaks up the following text at the double quote marks and picks out the piece that contains the temperature. Finally, the last line puts that number into cell B2.

Other Tools and Resources

Web pages can contain programs written in the Javascript language. See http://www.w3schools.com/js/default.aspJavascript is also the base language for Google Apps Script, a facility that allows programs to operate on and connect together various Google Apps.

HTML5 allows drawing on Web pages programmed in Javascript using the canvas feature, and lots of other really cool stuff. See http://slides.html5rocks.com or http://www.w3schools.com/html5/default.asp or http://diveintohtml5.com/ or other things from Ted Chen’s talk.

Google App Engine is a free hosting facility that supports server-side functionality such as database updating and access. It’s programmable in Python or Java. See http://code.google.com/appengine/docs/whatisgoogleappengine.html

data.gov lists many free data sets of government data.

WordPress Appliance - Powered by TurnKey Linux