May 12, 2008

Movement

So there is movement - but not here....

I have been using www.crankybunny.com to use as my playground for blog development. This is for two reasons: I wanted to experiment with another hosting provider (HostNine) and I wanted to use a different blogging software (WordPress).

The hosting has proven to be less than expected and I will probably stay with 1and1.com in the long run (although their customer service can be a big drag at times) but the blogging package - Wordpress - has proven to be very cool and easier to deal with.

Now that I am focusing on the travel blogging site again, I'm going re-bury myself in the whole geolocated photos/map thing that I obsess about. I still think that the ZenPhoto platform works the best for me and I will continue working in that vein. I have a few geolocated sets here but the layout leaves A LOT to be desired.

But no matter - the wheels are rolling again.

I have also posted my updated resume on Crankybunny and have been using the site as a general dumping ground for some geospatial related notes.

The idea is to figure out how to do the photos with ZenPhoto and Wordpress, migrate Steady As She Goes to WordPress and keep SASG as my travel blog - I'd like to have something in place that I don't have to deal with for a few years so I can just travel and not worry about the technology for awhile. I really don't want to use Flickr or Google Web Albums because I like having the images on my site and I like to display them in my own manner.

May 07, 2008

Movement

Steadily I may go,
but to where?
I don't know.

I have deleted all the comments by accident.

September 25, 2007

Maroc and Moving Servers

I have processed about half of the Morocco photos and GPS tracks while also figuring out how to install WordPress onto a differe web hosting service. Busy, busy.

When the Moroccan photos are selected and labeled (this weekend) I will post them here for one last set of photos and then I will focus on getting the whole site moved to the new host. Let's just say that it is a daunting prospect to move servers and blog packages (MovableType to WordPress) and try to keep it all nice and neat.

My camera was on it's last legs on this trip and I have since upped the ante and bought a Nikon D40 (fancy!) but David took a lot of really great photos that will be mixed in with mine.

Then comes the writing. There were some many things to write about for this trip. Just about everything that could go wrong did, but we survived and enjoyed visiting the country - ten days was certainly not long enough for a country like Morocco - especially when you are on your one limping along in bad French.

Cheers.

June 29, 2007

Data Processing

So the website transformation has not occurred yet, but I have downloaded WordPress which is the platform I will move Steady As She Goes to. But before I can move I have to figure out how to back up the TypePad files and figure out how to set up the url.

In the meantime I have been working on a few projects that I hope will function as a proof of concept for some business ideas that I have. These projects are directly connected with the direction of the new blog – information & location & travel (which really can be considered a combination of information & location).

So while there is no migration and web design going on I have learned an awful lot (and still have and awful lot to learn ) about Regular Expressions (and how much I suck at using them), data extraction programs, data extraction and manipulation between programs, sgml formats, MySql and myphpadmin, web hosting interactions (what do I actually have and what will I need), and I am still learning Google Maps API programming. It’s mind-boggling.

So for this project I am cleaning up data in order to geocode it but it isn’t very easy. Here are the steps so far:

1)Downloading Files – This sounds easy until you realize that there are approximately 2900 of them in various folders – not all in one place. This required many emails just to find the files in the first place on the Library of Congress website, then IM chats with a tech friend to figure out why I couldn’t access the ftp site (my browser was timing out – ended up using WSftp). Fortunately, two of the main folders had directory listing so I could create a batch file and download en-mass – but the two did not – numbingly good time that was.
2) Evaluating File Format – These files are “machine readable” according to the Library of Congress. They were written in sgml which is indeed machine readable but does not have much of an organizational structure. And there are no available sgml readers out there anymore – too bad for us. I used TextPad to look at them. The files were meant to be digital facsimiles of the paper documents, which they are, but that doesn’t help me convert them to a database format for serving up over the internet. Fortunately, most of the data I wanted was between specific tags – usually it was much more than what I needed but it worked. Another bonus is that the bulk of the text to be used was stored between tags and is readable by html. I can store this text with these tags in a mediumtext mysql table(?) – although I don’t know what to call the mysql equivalent of an Excel cell or actually how to get it from TextPad to MySql.
3) Extracting data from Files – This one is a doosey and if I knew Regular Expressions and maybe Grep or Awk (and had the systems to use them) and maybe a little Perl – not sure – it would have been a hell of a lot easier than what I went through – on 418 of the files – not even all of the files. First I decided on a subset of information to represent – if I knew more programmey stuff I would have tried to pull info from all of the files, but not right now – 418 is enough. Next I poured over the sgm files to find someway to extract data between tags. As it turns out – I could use a text search in a program called DataExtractor to get the actual data and then a similar Regular Expression search in TextPad to get the line numbers (which correspond to the file numbers). It was easy to just pulling specific data from 418 files, but very difficult to know what data corresponded to what file.
4) Cleaning up the Data – This is the very painful part. I have to actually compare what was pulled to the complete file. For example, I will pull everything between "2. Place of Interview*<(slash)p>“ which may give me any one of the following results: the search string and the actual address of the interview, the search string only because the address was on the next line with its own set of

tags, nothing because there was a different number of spaces between “2.” and “Place”, nothing because it was “1.” not “2.” and a handful of other problems. Multiply this by 3 data sets to pull. I got it down to 4 different pulls per set and a final integrity check just to fill in the blanks – spots that received nothing but I know had data. And this is even cleaning yet – this is just filling in the blanks. Cleaning will be a painstaking review of each entry, splitting it up into different fields so that I will be able to geocode the addresses.
5) Geocoding – This, when I get to it, will be interesting because many of these places probably don’t exist anymore. Such-and-such Nursing Home, Portland, OR – I can bet that that place doesn’t exist so I will have to ferret out a location for it. It’s just research, but a team of elves would be a good thing here.
6) Making a MySQL database out of the text and xls data – The is my second big knowledge gap and the problem actually lies in getting formatted text into a database format.
7) Making the webpage – The page will most likely involve, forms, Google Maps API programming, javascrip or Perl scripting all of which I know some but not enough about.

So the problem that I am running into the most is that I know what I want to do and I know that it can be done, but I am not qualified to do it. I am not a programmer and it is not easy for me to whip off a perl script to move this stuff to that place or to write javascript code to pull data from the data base to just show information by state. I get so frustrated with this because I think that there are some very cool things that can be created for the benefit of everyone – to help everyone tell their story and to education people about place.

The solution is to just keep moving forward, keep learning what I can, and hope that I will eventually find a tech partner to make these ideas a business. Anyone interested?

(P.S. I know that I didn’t divulge what I was actually doing but I need to create, at the minimum, a prototype before I go shooting off my mouth to the world in general)

Also - please note that I didn't figure out how to show paragraph tags within the text of the entry so I put a bunch of stupid parenthesis around them.

May 02, 2007

The Future of the Site (and Me!)

Well. I’m back now and trying to figure out what to do next in terms of life, living, and location – all the easy things.

But with regards to this website, I have decided that I really like the name and url so I will be revamping the site in the coming months. But the problem is – what should the focus be?

I think that I will continue to focus on Travel and Geo related topics. Such as: how to make my Treo 680 into a tracking device, how my self-studies in Google Maps API & and KML/XML are going, updated geo-photo KML files, and things like that. That should keep me busy. One big problem will be – how to keep the site as a record of the South America trip while adding other static pages like the resume and such. I like the idea of the blog as the main page because it keeps the content fresh (when one is actually posting) but there are design difficulties with merging old and new trips and goals. Perhaps the maps and photos and writing can all be some sort of blog – hmmm that’s an idea.

While I have my brand new “GTD” list of projects I am going to post a short list here.

1) Back up and delete crankybunny TypePad account so that I have that url to use for Google Maps API testing and Treo GPS server stuff. All very new territory for me.
2) Continue pulling together raw data from South America trip
3) Complete self-study in Google Map/Earth Hack book
4) Start and complete self-study in O’Reilly Google Maps book (slightly different stuff and I could use the practice)
5) Develop selling point outline for my two “great ideas”
6) Continue processing and linking photos to GPS tracks to create trip KML files
7) Look into methods for making Maps/Files page into a blog type entry system (that way it would be searchable)

So, off I go, back to work editing polygons – something I haven’t done in a quite a long time.

Oh - how could I forget...on the travel front - David and I are going to Morocco for a week beginning on Friday. We will be spending 3-4 of our 8 days in Marrakesh exploring the city by foot and then we will rent a car and head off to the High Atlas mountains, Dades Gorge, and the edge of the Sahara to ride some camels. How exciting!! We will post writing and photos upon our return and of course, I will take the GPS and track everything!

TO HELP WITH MY TRIP EXPENSES:

Thank you for your Support

Google Stuff

Powered by
Movable Type 3.2