Importing a Shapefile into PostgreSQL

I tried to import a shapefile to a PostgreSQL database using SPIT in QGIS, but kept getting an error because the gid column already existed in the .dbf. So I used the command line instead:

shp2pgsql Downloads/Archive/data_dev.shp > data.sql

then ran the resulting data.sql in PGAdmin.

Trying to execute
SELECT ST_Intersects(ST_GeomFromText('MULTIPOLYGON(((-170.82 -14.40,-170.82 -14.29,-170.62 -14.29,-170.62 -14.40,-170.82 -14.40)))',4326), the_geom)
as uni_data FROM data_dev

threw
ERROR: Operation on two geometries with different SRIDs

I had to set the SRID on the table using
ALTER TABLE data_dev DROP CONSTRAINT "enforce_srid_the_geom" RESTRICT;
UPDATE data_dev SET the_geom = SETSRID (the_geom, 4326);
ALTER TABLE data_dev ADD CONSTRAINT "enforce_srid_the_geom" CHECK (SRID(the_geom)=4326);

(As discussed here.)

Posted in PostgreSQL | 1 Comment

Taking the top 5 of grouped ranked data in PostgreSQL

Working from a list of my ancestors’ occupations I wanted to list the top 5 occupations in each occupational category per decade, so that given

year	category    description   total
...
1851	Agriculture  Ag Lab	   4
1851	Agriculture  Farm Bailiff  1
1851	Agriculture  Farm Laborer  2
1851	Crafts       Blacksmith	   1
1851	Crafts       Carpenter	   3
1851	Crafts       Shoe Maker	   1
1852	Crafts       Shoe Maker	   1
1853	Crafts       Blacksmith	   1
1853	Crafts       Shoe Maker	   1
1854	Crafts       Blacksmith	   1
....

I would get

decade	category    description        total   rank
...
1850	Agriculture  Ag Lab		4	1
1850	Agriculture  Farm Laborer	2	2
1850	Agriculture  Farm Bailiff	1	3
1850	Crafts       Blacksmith		3	1
1850	Crafts       Carpenter		3	2
1850	Crafts       Shoe Maker		3	3
...

Using the windows functions introduced in version 8.4, this was the solution I ended up with:

  • The outermost loop selects the top 5
  • The middle loop ranks the data
  • The innermost loop sums the data by decade

SELECT * FROM (
  SELECT decade, 
              category, 
              description, 
              total, 
              rank() OVER (PARTITION BY decade, 
                                        category
                             ORDER BY total DESC, description) AS pos
     FROM (
          SELECT  
                 CASE
                      WHEN year BETWEEN 1840 AND 1849 THEN 1840
                      WHEN year BETWEEN 1850 AND 1859 THEN 1850
                      WHEN year BETWEEN 1860 AND 1869 THEN 1860
                      WHEN year BETWEEN 1870 AND 1879 THEN 1870 
                      WHEN year BETWEEN 1880 AND 1889 THEN 1880 
                      WHEN year BETWEEN 1890 AND 1899 THEN 1890 
                      ELSE 1900 
                 END as decade, 
		 category,  
		 description,  
		 SUM(total) as total
          FROM occupations
          GROUP BY 1,2,3) as orig
     ) AS ss
WHERE pos < 6;

The method for grouping by decade is suggested on grokbase.com.

The ranking then filtering is discussed on the PostgreSQL site:

SELECT depname, empno, salary, enroll_date
FROM
  (SELECT depname, empno, salary, enroll_date,
          rank() OVER (PARTITION BY depname ORDER BY salary DESC, empno) AS pos
     FROM empsalary
  ) AS ss
WHERE pos < 3;

Posted in PostgreSQL, SQL | Leave a comment

Handling Transactions with SubSonic 3

I was pulling data from a web service and splitting the information across multiple tables in my database, so I wanted to rollback the entire operation if one of the saves failed. SubSonic 3 can handle transactions and I did get it working as needed, but I encountered a couple of issues along the way.

  • When I attempted to save to the second table I was getting an error about a DataReader already being open. Like the guy who reported the same problem here, I was using SubSonic 3.0.03. I replaced it with version 3.0.0.4, which worked fine.
  • The example in the SubSonic documentation on transactions shows the TransactionScope code nested inside the SharedDbConnectionScope code. I found that with this setup there was no rollback on error, and the DB tables were being updated. As indicated here and here, the nesting needs to be the other way round.
Posted in Uncategorized | Leave a comment

Creating a Rails 3 Project with PostgreSQL and TextMate on a Mac OS X Machine

I was advised to use RVM to manage Ruby on the Mac.

  1. Sign up as an Apple developer
  2. Download and install XCode
  3. Follow the instructions for installing RVM.
    1. Install RVM by executing $ bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head ) in the terminal window.
    2. I didn’t have one, so I had to create a .bash_profile file by following the instructions at redfinsolutions.com:
      1. Start up Terminal
      2. cd ~/ (to go to your home folder)
      3. touch .bash_profile to create the file.
      4. open -e .bash_profile to open it in TextEdit.
    3. Paste [[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # This loads RVM into a shell session. into the file and save
    4. Close the Terminal window, open another one, and type $ type rvm | head -1. If everything has worked, you should see rvm is a function
        Then you can run

      • rvm install 1.9.2
      • rvm use 1.9.2

Next you follow the instructions to set up rvm with TextMate. In order to run the script,

  1. Create the file in the same way as creating the .bash_profile file, naming it [myfilename].command
  2. Execute chmod u+rwx [myfilename].command to give yourself read-write-execute permissions
  3. Double-click on it in Finder to execute it.

Finally, install the pg gem:

  1. Find out the pg_config path by executing mdfind pg_config|grep bin|uniq
  2. Install the gem using gem install pg -- --with-pg-config=[the path returned by the previous line]

Notice that sudo isn’t used, because RVM is being used.

See this tutorial for setting up a Rails app with a PostgreSQL DB.

Posted in PostgreSQL, Ruby on Rails | Leave a comment

Creating a Rails 3 Project with RubyMine and PostgreSQL on a Windows 7 Machine

Ruby gems and Windows don’t always get on – only try doing Rails projects on a Windows 7 machine if you’re feeling brave.

Downloads:
RubyMine
RubyInstaller for Windows (contains RubyGems, so you don’t need to download the gems separately)

In RubyMine,
File > New Project…
Specify the project name and set the Project type to ‘Rails application’. Click ‘OK’
Set the Ruby Interpreter to the Ruby installation (click the button next to the dropdown)
Click the button next to the Rails Version dropdown to install the Rails gem
Tick the ‘Preconfigure for selected database’ box and select ‘postgresql’
Wait for RubyMine to generate the project files
Replace the
gem 'pg'
line in the project’s Gemfile with
gem 'pg', '0.9.0', :require => 'pg'
– the latest version at this time won’t work with Windows 7
Tools > Bundler > Install

You’ll now be able to run rake tasks.

Posted in Ruby on Rails | 1 Comment

CSS3 Effects for IE using PIE

Short and sweet, this post. PIE.htc makes it easy to persuade IE to render some CSS3 effects – including border-radius and box-shadow.

Posted in Web Design | Leave a comment

Adventures in BDD for an ASP.NET MVC 2 and jQuery Project

I’ve played with Cucumber testing for a Ruby on Rails project, and I wanted to be able to a) automate and b) document some complex scenario testing for one of my ASP.NET MVC apps.

I downloaded SpecFlow 1.5 and NUnit 2.5.9. Because I am testing a web application, I also needed a browser automation library. I already use Selenium for recorded tests in FireFox, but decided to try WatiN 2.0.

When testing with a Cucumber-style setup, there are three parts:

  1. The feature specification, written in Gherkin

    Feature: Searching
       In order to view documents
       As a user
       I want to be able to search for a particular document
    
    Scenario: Search by Document Title
       Given I am on the search page
       And I am logged in
       Then I should see a textbox labelled "Title"
       When I click the button labelled "Search"
       Then I am on the document page
    

  2. The step definitions, using the library that will drive your application (your model for a console app, or your browser automation library for a web app – this example uses WatiN)

           [Given(@"I am on the (.*) page")]
            public void GivenIAmOnThePage(string pageName)
            {
                var browser = new IE();
                browser.GoTo("http://localhost:xxxx/Home/" + pageName);
                Assert.That(browser.Title, Is.EqualTo(pageName));
            }
    


    (notice that you can use regular expressions and capturing groups to pass parameters into your method)

  3. The actual code that provides the functionality being tested

I started out by following the SpecFlow screencast. The basic steps in VS 2008, using Red-Green-Refactor, are as follows:

  1. Add a new project to your solution (use a Class Library) called [ProjectName]Specs.
  2. Add references to the nunit.framework.dll and techtalk.SpecFlow.dll (I had to add a reference to WatiN.Core.dll as well)
  3. Add > New Item… > SpecFlow Feature File
  4. Run the tests in NUnit. They will show green but with 1 inconclusive test
  5. Add > New Item… > SpecFlow Step Definition. Call it StepDefinitions
  6. Go to the text output tab in NUnit and copy the provided step definition stubs into your Step Definition file
  7. Implement the steps
  8. Run your tests again and check that they pass

The demonstrator appears to have TestDriven.net installed, but you can run the tests by opening the NUnit GUI from your Start menu and opening your spec project’s dll. Then, next time you build the project in VS the tests will automatically run in NUnit. (Remember to check all 3 boxes under Tools > Settings > Test Loader > Assembly Reload in NUnit.)

For examples of implementing the steps with WatiN, I looked at Steve Sanderson’s article on using SpecFlow with ASP.NET MVC, and Deran Schilling’s articles part 1 and part 2 are useful too, particularly the solution for the
System.Threading.ThreadStateException: The CurrentThread needs to have it's ApartmentState set to ApartmentState.STA to be able to automate Internet Explorer. at WatiN.Core.IE.CheckThreadApartmentStateIsSTA()
error which IE threw.

To debug the step definitions, when you have NUnit running select Debug > Attach to Process… in VS and choose nunit.exe from the processes list.

But my experiment ground to a halt when I needed to wait for an AJAX response from my jQuery validation. I tried to use Omar Al Zabir’s WatinN to Automate Browser and Test Sophisticated ASP.NET AJAX Sites (see also Robert Koritnik’s response to How to wait for jQuery Ajax requests to complete from WatiN? on StackOverflow), which involves injecting JavaScript to keep track of the AJAX requests in progress, but it kept failing with a JavaScript TypeError saying that the variable was undefined. My applications make extensive use of AJAX, so this was a show stopper.

The solution was inspired by Dima Kovalenko’s article Selenium wait for AJAX (the right way!!!). Because my project uses jQuery I didn’t need to inject JavaScript to count AJAX calls – I could use WatiN’s Eval method to test for $.active == 0 instead.

Posted in Testing | Leave a comment

Useful Tools for Working With Web Site Colours

ColorZilla – plugin for Firefox, I find the eyedropper very useful
Color Converter – displays the colour as well as providing the code in different formats
colorbrewer2.org – great for colour schemes for charts as well as maps, includes colorblind-safe options

–Update 10 June 2012–
html-color-names.com – gives the hex codes for the HTML colours

Posted in Web Design | Leave a comment

Checking if FullText Service is installed on SQL Server 2008

SELECT FullTextServiceProperty('IsFullTextInstalled')

This will return 1 if the service is installed.

Posted in SQL Server | Leave a comment

Concatenating column from many side of one-to-many into single record using STUFF

SELECT OneTable.col1, OneTable.col2, 
     STUFF (
          (
               SELECT ' ' + mt.col1
               FROM ManyTable mt
               WHERE mt.foreignkeycol = OneTable.primarykeycol 
               FOR xml path('')
          ),
          1, 1, ''
     ) AS concatenatedcol
FROM  OneTable


Note: remember the '' in the inner SELECT statement, otherwise the concatenated column will contain XML tags (with the first bracket stripped).

Posted in SQL | Leave a comment