Tag Archives: Dev

From Jekyll to Hugo

Sometimes simpler is better.

After being away from Orbital for a bit, I started revving my engines a bit by trying a new project on the subway and hacknight. Project Munchausen is about doing some writing and exploring some ideas – so I tried to get myself away from doing too much programming on it. Since it is just writing, I wanted to avoid falling into the trap of using a big content management system like WordPress.

I started exploring with the first Static Site Generator I know of: Jekyll. Jekyll powers the Pages feature on GitHub. I got really far, but felt really constrained. It felt like Jekyll was trying to push me to turn my site into a blog, like the downhill path of Jekyll was to create a blog.

I didn’t want to create a blog – I want to make something that’s an exploration of a subject. It will change over time, so it should have an RSS feed generated, but I didn’t want to only have a timeline view of things.

I don’t know how I happened on Hugo but it seemed perfect. The templates are simple, Go is a pretty simple language to use, and the organization of the content is really easy to understand.

In Hugo, you have a directory named content. Under that directory, any other directories are Sections.  Content goes in a section as a Markdown file. Templates can iterate over those sections and their contents.  Easy peasy.

So far, I’ve gotten farther with Hugo in a few subway rides than I have with long sessions wrangling Jekyll. My goal is to put something up on the web in a month.

How (not) to do Tagging in Angular.js

I’ve been working on getting tagging working in Orbital Feed Reader. Tagging feeds was really brilliant in Google Reader and I want that in the feed reader I use. So I’m building it.

The Setup

You can store tagging in your database a few different ways, but here’s the way I chose:

                         +----------+
      +----------+       | feed_tag |
      | feeds    |       |----------|
      |----------|       |  tag_id  +---+
      |  id      +-------+  feed_id |   |   +--------+
      |  name    |       +----------+   |   | tags   |
      |  url     |                      |   |--------|
      |  unread  |                      +---+  id    |
      +----------+                          |  name  |
                                            |        |
                                            +--------+

You’ve got feeds and tags, those have an ID number. You’ve got a table linking those by ID number. Easy.

The Goal

I wanted to be able to toggle between either a simple list of feeds or a list of tags with feeds underneath.

<script type="text/ng-template"  id='feedline.html'>
  <div class="feed" id="feed-{{feed.feed_id}}" 
    ng-class="{'is-editable': editable, 'is-selected': feed.isSelected}" 
    ng-click="select(feed)"  >
      {{feed.feed_name}} 
      <span class="feedcounter">{{feed.unread_count}}</span>
      <a ng-show="editable" ng-click="editFeed(feed)">⚙</a>
  </div>
</script>
<ul id='feeds' ng-hide="showByTags" >
  <li ng-repeat="feed in feeds" ng-include="'feedline.html'" ></li>
</ul>
<ul id='tags' ng-show="showByTags">
  <li class="tag" ng-repeat="(tag, feeds) in tags" >
    <div id="{{tag}}" ng-click="select(tag)" 
      ng-class="{'is-selected':tag.isSelected}" >
      #{{tag}} 
      <span class="feedcounter">{{tagUnreadCount(tag)}}</span> 
    </div>
    <ul>
      <li ng-repeat="feed in feeds" ng-include="'feedline.html'"></li>
    </ul>
  </li>
</ul>

My first stab at doing this was very wrong and I wanted to share that so you don’t go down that dumb path on your project.

The Wrong way to do tagging

I decided I would pull the feeds in two different ways. I would pull feeds from the database as a normal list, but then I would also pull a list of tags with feeds underneath.
This was my first mistake. Hitting the DB twice for the same data means I’m doing lots of duplication and sending lots of information over the internet (slooooow) twice. It also means keeping lists in sync which is also a code smell that means you are probably not keeping things DRY.

So I pull feeds from the db once like this:

SELECT
  feeds.id AS feed_id,
  COALESCE(u_feeds.feed_name,feeds.feed_name ) AS feed_name,
  feeds.feed_url,
  COALESCE(u_feeds.icon_url, feeds.icon_url ) AS icon_url,
  COALESCE(u_feeds.site_url, feeds.site_url ) AS site_url,
  feeds.lASt_updated,
  feeds.lASt_error,
  u_feeds.private,
  SUM(IF(COALESCE(ue.isRead,1)=0,1,0)) AS unread_count
FROM user_feeds AS u_feeds
INNER join feeds AS feeds
  ON u_feeds.feed_id = feeds.id
LEFT OUTER JOIN user_entries AS ue
  ON ue.feed_id=feeds.id
  AND u_feeds.owner = current_user_ID
GROUP BY feeds.id,
  feeds.feed_url,
  feeds.feed_name,
  feeds.icon_url,
  feeds.site_url,
  feeds.lASt_updated,
  feeds.lASt_error,
  u_feeds.private

And then pull feeds by tag like this:

SELECT
COALESCE(tags.name,'Untagged') AS tag,
COALESCE(tags.id, null) AS tag_id,
feeds.id AS feed_id,
COALESCE(u_feeds.feed_name,feeds.feed_name ) AS feed_name,
feeds.feed_url,
COALESCE(u_feeds.icon_url, feeds.icon_url ) AS icon_url,
COALESCE(u_feeds.site_url, feeds.site_url ) AS site_url,
feeds.last_updated,
feeds.last_error,
u_feeds.private,
SUM(IF(COALESCE(ue.isRead,1)=0,1,0)) AS unread_count
FROM user_feed_tags AS uft
INNER JOIN tags AS tags
  ON tags.id = uft.tag_id
INNER JOIN user_feeds AS u_feeds
  ON uft.user_feed_id = u_feeds.id
INNER JOIN feeds AS feeds
  ON u_feeds.feed_id = feeds.id
LEFT OUTER JOIN user_entries AS ue
  ON ue.feed_id=feeds.id
WHERE u_feeds.owner = current_user_ID
GROUP BY
feeds.id,
feeds.feed_url,
feeds.feed_name,
feeds.icon_url,
feeds.site_url,
feeds.last_updated,
feeds.last_error,
u_feeds.private,
tags.name

UNION
SELECT
'Untagged' AS tag,
null AS tag_id,
feeds.id AS feed_id,
COALESCE(u_feeds.feed_name,feeds.feed_name ) AS feed_name,
feeds.feed_url,
COALESCE(u_feeds.icon_url, feeds.icon_url ) AS icon_url,
COALESCE(u_feeds.site_url, feeds.site_url ) AS site_url,
feeds.last_updated,
feeds.last_error,
u_feeds.private,
SUM(if(COALESCE(ue.isRead,1)=0,1,0)) AS unread_count

FROM user_feeds AS u_feeds
LEFT OUTER JOIN user_feed_tags AS uft
  ON uft.user_feed_id = u_feeds.id
INNER JOIN feeds AS feeds
  ON u_feeds.feed_id = feeds.id
LEFT OUTER JOIN user_entries AS ue
  ON ue.feed_id=feeds.id
WHERE u_feeds.owner = current_user_ID
  AND ISNULL(uft.user_feed_id)
GROUP BY
feeds.id,
feeds.feed_url,
feeds.feed_name,
feeds.icon_url,
feeds.site_url,
feeds.last_updated,
feeds.last_error,
u_feeds.private

For feeds by tag I then would use underscore.js’s excellent _.groupBy() method. It’s easy to see tags in angular using groupBy and I thought it was good enough.

It worked! Sure, it was inefficient, but I’m pragmatic – I just wanted to get this done and out the door so I could use it. No point in prematurely optmizing, right?

Next step was making sure I could show the unread count per tag. And keep it all in sync between the tags and the feeds themselves. This is when I smelled my stinky code smells. Every time I marked an entry in a feed as read I would have to search it out multiple times on multiple lists. This means this information was stored in multiple places – and it’s always good to follow the DRY principle. Don’t Repeat Yourself.

The Right(er) way to do tagging

Let’s regroup and try this again. I want to have ONE place where I pull info from my DB about a feed, and I want to only update information in one place but display it in multiple ways.

I changed it so I pull a straight list of the feeds, but use the group_concat SQL function to also pull in all the tags associated with a feed.

SELECT 
  u_feeds.id AS feed_id,
  COALESCE(u_feeds.feed_name,feeds.feed_name ) AS feed_name,
  feeds.feed_url, 
  COALESCE(u_feeds.icon_url, feeds.icon_url ) AS icon_url,
  COALESCE(u_feeds.site_url, feeds.site_url ) AS site_url,
  feeds.last_updated,
  feeds.last_error,
  u_feeds.private,
  SUM(IF(COALESCE(ue.isRead,1)=0,1,0)) AS unread_count,
  GROUP_CONCAT(DISTINCT COALESCE(tags.name,'Untagged')) as tags
FROM $user_feeds AS u_feeds
INNER JOIN $feeds AS feeds
  ON u_feeds.feed_id = feeds.id
  AND u_feeds.owner =  $current_user->ID.
LEFT OUTER JOIN $user_entries AS ue
  ON ue.feed_id=feeds.id
LEFT OUTER JOIN $user_feed_tags uft
  ON uft.user_feed_id = u_feeds.id
LEFT OUTER JOIN $tags tags
  ON uft.tag_id = tags.id
GROUP BY 
  feed_id,
  feed_url,
  feed_name,
  icon_url,
  site_url,
  last_updated,
  last_error,
  private

When I want to show a list of feeds I can just go with the same solution I had before.
But when I want to group by tags I can apply a couple of interesting tricks to solve this for me.

    refresh : function refresh(callback){
  _isLoading = true;
  $http.get(opts.ajaxurl + '?action=orbital_get_feeds')
  .success( function( data ){
    //Here is our simple feed list
    _feeds= data;

    //Now lets get a list of all the unique tags in those feeds
    var taga = _.unique(_.pluck(_feeds, 'tags').join().split(","));

    //For each tag, lets build up a list of the feeds that have that tag
    _.each(taga, function(tag){
      _tags[tag] = _.filter(_feeds,function(feed){
                      return _.contains(feed.tags.split(","),tag);
                    });
    })
    _isLoading = false;
    //Should we do some extra work?
    if(callback){
      callback(_feeds);
    }
  });
},

What’s going on there?
I want to get an array of all my unique tags, so first thing I do, I use _.pluck() to give me an array of the tags element of each feed.
This gives me:

[
  ["art,gifs"],
  ["art"],
  ["Untagged"],
  ["gifs"],
  ["Untagged"],
  ["Untagged"],
]

Then I join() those together (the default join is with a comma):

"art,gifs,art,Untagged,gifs,Untagged,Untagged"

I split(“,”) that on commas:

["art","gifs","art","Untagged","gifs","Untagged","Untagged"]

And finally we just grab the _.unique() values:

["art","gifs","Untagged"]

So now we have a list of the tags for our feeds – how exciting! Let’s build a list of the feeds for each of these tags.
So, for _.each() tag in our list of tags we go back to our list of feeds and _.filter() it – we want only a feed where the feed.tags _.contains() the tag.

Why’s this so much better? Well – if we update the `unread_count` for a feed or change the name of a feed – it updates the underlying object. That means your change percolates up automatically through angular databinding to every part of your UI. No more keeping lots of lists in sync.

And now I’ve bored even myself to death with this so I guess that’s it.

SVG icons in Orbital

Screen Shot 2013-08-06 at 5.51.25 PM

In Orbital Feed Reader I’m trying to make sure that I’m doing things in smart ways – I don’t have much time to work on it. One thing that people do when they create software is make icons – lots of them for different sizes. This has always seemed strange to me – most icons aren’t photos, so I don’t get the point.  There’s a graphics format called SVG – Scalable Vector Graphics  – that automatically resizes itself at any resolution.  It’s smooth and very small – and all modern browsers support SVG.

So I’m trying to use SVG for any icons that aren’t available as unicode symbols.

Those two icons are free icons for Feeds and OPML and they are both SVG.  This was such an easy choice that I don’t know why everyone doesn’t do it. The only thing that was a little off is that the OPML icon had a little bit more blank space than I wanted, so I edited it in the free, open source SVG editor Inkscape. Here’s the new version I made.


Creative Commons License BY-SA 3.0
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

The folks over at liquidorb made this icon available under a CC-BY-SA license so I’m giving them credit and making my derivative work available under the same license.

WordpRSS Status Update: pretty sweet

Current image of wordprssI’ve been using my kaizen hack time to work on a social feed reader for WordPress. Right now, here’s what it can do:

  • Install itself and set up database tables
  • Put in a few sample feeds
  • Pull feed entries down into the database.
  • Display the list of feeds
  • When you select a feed, display the items

Continue reading WordpRSS Status Update: pretty sweet

Ditz – the distributed issue tracker

If you are using a distributed version control system 1 you get some really cool benefits and some really strange problems.

When I was toying around on a project with Aaron, I fell in love with ditz. We needed a quick way to keep track of bugs, without taking the time to set up a central bug repository. We wanted a bug tracker that could live in the same place as the code, where adding a friend to kick in on code didn’t require more accounts being set up and maintained.

Ditz did all of that, works straight from the command line, and even outputs some sweet html pages for display to the world.

Ditz is kind of abandonware right now as the original author has gone on to other things – but the state it is in right now is just perfect for my personal projects. If you are using it, I’ve added an RSS feed for the html output.

And the really good news? I just convinced the maintainer to make me a co-maintainer. So that means that I can integrate features! Once we get enough in for a new release, I’ll post an update right here!

  1. like git or darcs or mercurial, etc.   (back)

My Valentines / Music Hack Day NYC app!

-or-
Why I wasn’t around this weekend.

I was at NYC Music Hack Day!

You can try out the idea here in our valentune.es beta. We send love and music over wires and wireless to your special sweetheart.

You put in your sweetheart’s Name & cellphone, a couple of key words about your them and then ask the app to get busy.

It goes out to the MusixMatch lyrics service – finds lyrics that describe your sweetheart and then gives you a list of songs we can find streaming full mp3s online for. You choose what you don’t want in the play list and hit send.

We call your sweetheart up using the Twilio API , give them your sweet message and then play them your custom made mixtape over the phone! Yes, my lovely wife Sam was the first non-developer to get a call from the valentun.es robot.

Getting SongsPick songs on the iPhone!

Don’t have a web browser handy? Surely you jest! If you’ve got an iPhone you can use the iPhone app.

If not, the website is mobile compliant and works great on Android phones. The website uses CSS3 and WebFonts for style, but degrades well to older browsers. How old? IT WORKS ON LYNX!in case you need web 3.0 from a terminal

So all of this was accomplished by 6 talented and dedicated people over the course of 24 hours in NYC. The whole team was positive and awesome. I’m really proud of what we did so quickly. It’s got bugs 1 , but it works. I didn’t sleep the whole 24 hour hack session and I worked on a lot of the bits.

  • The Musixmatch call to find lyrics that talk about your sweetheart.
  • The Skreemr call to find playable songs with those lyrics
  • Adapting Jeff’s web mockups to Django templates
  • Calling into Alex’s Twilio wrapper
  • Pitching the idea and the vision and recruiting the group
  • Helping design the process flow and settling on using Django with Nate
  • Helping coordinate who would do what and figuring out the pitch with Jeff
  • Presenting the whole thing in 2 minutes to over 300 people with Anna and Nate

It was a busy 24 hours!

  1. the Title element has the wrong spelling of valentunes and the message doesn’t always go through and the songs sometimes take a long time to play, etc…   (back)

Installing the Android Developer Tools for Eclipse

If you’re trying to install the Android Developer Tools on Ubuntu and it keeps failing on a strange error:

An error occurred while installing the items
session context was:(profile=PlatformProfile, phase=org.eclipse.equinox.internal.provisional.p2.engine.phases.Install, operand=null –> [R]org.eclipse.ant.ui 3.4.1.v20090901_r351, action=org.eclipse.equinox.internal.p2.touchpoint.eclipse.actions.InstallBundleAction).
The artifact file for osgi.bundle,org.eclipse.ant.ui,3.4.1.v20090901_r351 was not found.

Here’s your solution:

sudo apt-get install eclipse-pde

Worked for me, found it here.

New article on how to set CGI:IRC as Firefox’s IRC protocol handler

I really like to overdo things.  In the overdoing spirit of the Thanksgiving season, I overdid something recently.

When I clicked on an irc link to a chatroom, it didn’t open in my preferred IRC client, which is on my webserver.   I like to run my applications off of my webserver because I don’t have to worry about carrying thumbdrives or dealing with install policies on strange computers.  If I’ve got a browser, my own private cloud just works.

A little digging, a little coding, and I wrote up a handler.  Yes, that was a bit more work than I expected, but this way I can give thanks to you by giving you this gift of a very small effort.

If you’ve followed my advice before about getting your own webserver, this could be a 15 minute task for you.  Install CGI:IRC, install my little handler, baboomp.  Then we can chat on IRC

Say hi!

Why the Business Hates the Software Developers

Inspired by “Why I Hate Frameworks”

So I wanted a custom spice rack, one that would really help my restaurant, but I didn’t want to build it myself.  I’m no carpenter.  I hired one though.
“Can you build me a spice rack? Here’s what I need – ”
“Just a sec,” he interrupted. “I’m going to start building some hammers while you talk.”
“Don’t you already have some hammers?”
“Sure, but those were for past projects.  There’s better ways to build hammers these days.”
“Here – I have a hammer already.  It got left behind by the guy who hung the pictures in the restaurant.  Use this.”
“SIR! I don’t think that using a framing hammer is going to help us develop this spicerack.  Please!”

Using Ruby for command line web lookups

Common Problem

You frequently have to look up customer information on the company website.  Firing up a web browser takes time and invites you to start dawdling away on facebook and such.  If only anyone in the company had bothered to write a decent webservice or command line utility to look up customer information.

Find the right url

The first step is to dig into the company website and find out what happens when you click search.  You are looking for an element in there of type “form”.  A form is what gets submitted when you click search.  It will submit information to a page, and that page is the “action” attribute of the form element.  Then you need to find the inputs to that form.  Look for elements of type “input”.  These guys are the information you are sending to the action page.  Once you have the “action” and the “input” names, you can come up with a URL that represents this lookup.  It’s dead easy and it always follows the same pattern.

If you have a form like this:


<form action="/admin/clientsearch.asp" method="post">
<table border="0" align="center">
<tbody>
<tr>
<th class="QueryHeader">Search Options</th>
</tr>
<tr>
<td align="center">
<table border="0">
<tbody>
<tr>
<td><strong>Search For:</strong>

<input name="SEARCHPARAM" size="15" type="text" /></td>
</tr>
</tbody></table>
</td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td align="center"><input type="submit" value="Search" /><input type="reset" value="Reset" /></td>
</tr>
</tbody></table>
</form>

You can see the “action” is”/admin/clientsearch.asp” and that the input is named “SEARCHPARAM”.  From this we know that the URL is going to be “http://www.example.com/admin/clientsearch.asp?SEARCHPARAM=”.  That’s how simple it is.

Automate the lookup using Ruby

If this is a task you have to do often, try using Ruby to automate it.  Ruby has a utility for doing repetitive tasks called Rake or Sake and a utility for parsing web pages called Hpricot.  Install them like so:


gem install rake hpricot sake

Now we write up a file called “Rakefile.rb” and put in a rake task

desc "sets up the following tasks"
task :setup do
require 'open-uri'
require 'hpricot'
end

desc "lookup a client"
task :clients, :client_name do |t, args|
doc = Hpricot(open("http://www.example.com/admin/clientsearch.asp?SEARCHPARAM=#{args.client_name}", :http_basic_authentication => ['username', 'password']))
puts doc.search("//center[2]/table")[0].to_plain_text
end
task :clients => :setup #put in here bc named args seem to conflict with dependencies.

Most of what’s going on there is happening on line 9. We are opening a url, then passing it to our parser. If you don’t have a username and password for this website, you can remove the whole “http_basic_authentication” argument to open.

Where is my data

In line 10 you’ll see a little handy XPath going on to narrow down the document to what we care about. If you aren’t so hot with XPath, there is an easy way to find it out. In Firefox, install an extension called Firebug. Do a search on your webpage, then activate firebug by clicking the bug icon in your statusbar. Click inspect in Firebug and then click where your data is. Firebug will display a bunch of elements on the top. Move your mouse along them and you’ll find one element that highlights your data in blue. Right click on this and “Copy XPath”. That’s what you will put in “doc.search()”.

Using it

From the commandline type rake clients[myclient] and ruby will do the lookup and return the information you care about.  That will only work if you are in the same directory as your rakefile.rb.  We can install these tasks into sake by typing sake -i rakefile.rb. This makes these tasks system wide, so you can call sake clients[myclient].

A couple of caveats

  1. You may have to do a little tweaking to get open-uri to play nice with expired https certificates. Shouldn’t be a problem for most folks.
  2. The world of screen-scraping as it is called, doesn’t end there. If you need more advanced techniques for screen scraping a page, behold the power of the internet.