When I started at my current job I noticed we we had lots of room for improvement about how we imported and exported data. Folks had been using the MicroSoft SSIS platform as a way to Extract, Transform and Load data in and out of our database to various files.
SSIS is great for lots of things and has a lot of upsides. It is very drag and drop, folks don’t have to know a lot of programming to get it to do things, and it has lots of functions built in. If you need more programming power, you can execute C# or VB scripts to do the fiddly bits.
But I hate it. ( Don’t worry, we’ll get to the love soon.)
My biggest problems with SSIS:
- It is unversionable. Try reading a git diff of an SSIS change. The xml is designed for a machine to read, not a human. If you want to know what has changed over time in your world, it’s a problem.
- You can only use Visual Studio to edit it. Many of our SSIS packages include VB or C# scripts. That sounds fine – but apparently these compile to an undiffable, uneditable blob in the xml that is only recompiled if you save using visual studio. So if you want to change something across many SSIS package scripts, you have to open and resave each one.
- It hides options under rocks. Finding out how something works requires lots of delving into lotsa windows and dialogues.
- It changes things unexpectedly. Click in the wrong dialogue and it helpfully re-infers datatypes from a file for you. You don’t know until you go to execute.
- It slapped my momma. Etc.
I wanted to move my team to something that was better for people.
We need something:
- That we can diff
- That we can do code reviews and pull requests on
- That is simple, expressive and clear.
- That is powerful.
To me that sounds like a programming language. I encouraged folks on the team to try accomplishing a couple of tasks that might use an SSIS package instead to use Python. Immediately, things got better. Our code reviews made sense. Code quality improved with every single pull request.
We used pymssql to connect to SqlServer and inserted records as needed after processing them. Navigating and transforming XML docs was easy, CSV files were eaten up by the native DictReader.
And then Derrick found PETL. It’s beautiful. You point it at data and make simple moves to completely transform it. I’m smitten.
I had dozens of files to read from, each a quarterly file for a year – only noted in the file name. Each had a crappy heading line that preceded column headers. I needed to put them into 1 file for loading into SalesForce Wave. Whacking together a solution with PETL was effortless. Line 36 is where the PETL starts, and it’s so small and good that it is nice to see how much it encapsulates.
At work, I’m trying to convince people that we should auto-generate at least a suggested code name for our project names. It’s an important thing for compliance and secrecy. You’d rather someone is overheard talking in the elevator about “Project Icy Gneiss” than about “the restructuring of Acme Corp”.
I wanted to make the point that if you just have a small list of adjectives and nouns you quickly get a vast space of possible names – more than we’ll exhaust. But a working demo is more persuasive than logic.
I knocked this together last night: Projects-a-Plenty.
Used bootstrap & angular which is kind of overkill on something this tiny.
This is a tweak to fix some problems people told me about on the Orbital Feed Reader GitHub Issues page.
Some folks were getting a very bad experience during the install – everything goes blank – or they were able to install and then couldn’t add feeds or read feeds. Not good for a feed reader! I traced down a problem where if you had a specific new default collation on your database, the way I initialized the database tables made everything go boom. It’s fixed now, but it’s part of the problem of supporting a lot of different configurations without a giant automated test lab.
The other was that someone pointed out there is an improved PressThis page in the latest version of WordPress. Everything was working, but you’d get a nudge message saying that there was a new way to use PressThis and why are you going this old way? Updated and I think it’s a really good experience now!
I also did some cleanup and logging improvements so that I can work with folks who have problems in a better way.
Sometimes simpler is better.
After being away from Orbital for a bit, I started revving my engines a bit by trying a new project on the subway and hacknight. Project Munchausen is about doing some writing and exploring some ideas – so I tried to get myself away from doing too much programming on it. Since it is just writing, I wanted to avoid falling into the trap of using a big content management system like WordPress.
I started exploring with the first Static Site Generator I know of: Jekyll. Jekyll powers the Pages feature on GitHub. I got really far, but felt really constrained. It felt like Jekyll was trying to push me to turn my site into a blog, like the downhill path of Jekyll was to create a blog.
I didn’t want to create a blog – I want to make something that’s an exploration of a subject. It will change over time, so it should have an RSS feed generated, but I didn’t want to only have a timeline view of things.
I don’t know how I happened on Hugo but it seemed perfect. The templates are simple, Go is a pretty simple language to use, and the organization of the content is really easy to understand.
In Hugo, you have a directory named content. Under that directory, any other directories are Sections. Content goes in a section as a Markdown file. Templates can iterate over those sections and their contents. Easy peasy.
So far, I’ve gotten farther with Hugo in a few subway rides than I have with long sessions wrangling Jekyll. My goal is to put something up on the web in a month.
Last night I pushed out the final changes to uncage version 0.1.9 and get it out in the wild.
This version is mostly about giving you freedom. You should be able to easily import your stuff from other platforms into Orbital Feed Reader to use it. If you decide that you don’t want to use Orbital Feed Reader any more, you should have an easy way to leave. Any software that doesn’t tell you how to leave it is something you should be suspicious of.
Get Feeds IN
For Feed Readers, the standard way to exchange lists of subscriptions is a standard called OPML. It’s an old format, but it’s how they all exchange lists of feeds. I’ve buffed up the OPML import to handle larger numbers of feeds. After you import the file you have a chance to look over the feeds and edit them, tag them or exclude them entirely. Hit save and they’ll all flow into the backend.
Get Feeds OUT
But we also don’t want Orbital to be a lobster trap for you. If you import that kind of data I also want you to be able to export that kind of data. Now Orbital is good at exporting your feeds. If you are logged in – we’ll export a file with all your public and private subscriptions and let you take that wherever you want. Some folks want to share their OPML publicly – if you aren’t logged in, hitting the same URL gives you only the public feeds. All your private stuff should stay safe!
Read Lotsa Feeds
Now that Orbital can handle tons of feeds coming in I wanted it to be easy to actually sort through them. Feed Tags now start off collapsed and you can expand them anytime by using the little twiddlers next to each. That’s a technical term. The twiddlers make it much more manageable to see 200 feeds over many subject areas.
The most common feedback I’ve heard from folks on Orbital is that it’s not very good looking. It does the job, but it isn’t a pleasure to look at.
I assume these people are all supermodels, so it looks like Orbital is doing great among the high fashion set.
I got out my scissors, called in the wardrobe department, applied polish and buffed with a chamois. This is the new smooth look, designed to blend in with the new admin themes of WordPress.
Hey, good lookin’!
Beauty comes in all thicknesses
Along the way, I’ve added some features – now it is responsive, cleaner and is even usable on my Galaxy S5. Check out this hot resizing action!
Future attractions and graceful aging
Coming soon are some cool things. First, I’m working on version 0.1.9, code named Born Free. It will deal with the how to leave Orbital and how to get into Orbital. I’d never want to trap you!
Second, WordPress is going to upgrade to version 4.0 soon. I’ll be getting Orbital ready for that, and the new REST API should make it easy to build a better mobile experience for Orbital.
Party time is all about friends. I’m in bed recovering from the 4th of July and a party that turned into a sleepover. But I’m happy because my friends could hang out til we passed out.
Orbital Feed Reader now supports multiple users!
Continue reading Orbital Feed Reader 0.1.7 Codenamed “PARTY TIME” is out!
Two new reviews for Orbital bring a smile to my face!
WordPress Plugins A-Z gave Orbital a 4/5 rating:
This is a great feed reader especially if you are doing research for a post or if you are like many people out there who write additional articles based on what they read. This make it easy to get a pull quote from an article and drop it right into a post. While the layout and functionality is a little raw in its appearance it seems to work very well and to give it a good go I added it to my secret blog site as a place to test some ranting on article and stories. And it seems to work quite well..
That’s great feedback and I feel like I do need to work on the style and appearance – it’s tough and I need help.
Jeff Chandler over at WPTavern also reviewed Orbital, comparing it to Feedly and Google Reader. It’s a lukewarm review – he doesn’t like the interface and feels like Feedly plus PressThis is find for him. Still, he’s offered to help me by looking at revamps of the interface.
CSS Diner is a .fun way to learn CSS selectors and :not(.boring) – thanks @flukeout!
I just stumbled on The Codeless Code, a collection of fables and Koans for programmers. It’s brilliant, if a terrible rabbit hole of procrastination – I’ve just burned a few hours clicking through random stories. Here’s a good one to get the flavor, Case 41 is called Garbage:
Java master Suku was investigating the software of a distant temple. On her monitor large swaths of text glowed not black but green, indicating that reams of code had been commented out.
“Curious,” said Suku. “I have opened a fine clock, and discovered orange rinds and fish bones.”
The head monk explained that the inactive code was no longer necessary, yet he had ordered his clan to leave it in place: for if it someday became desirable to restore the logic then the code could simply be uncommented, rather than wastefully written from scratch.
“Understandable,” nodded Suku.
The next day the monks assembled again for the Java master’s inspection. With concern they reported that the head monk could not be found. Suku gestured up into the rafters where the missing monk dangled from a strong rope, by the neck. A foul odor wafted downwards.
“Your head monk’s services are no longer necessary,” said Suku. “Yet let his corpse moulder above you from this day forward. After all, his methods may someday become desirable again, and it would be wasteful to train another monk from scratch.”
In her final report, Suku noted a swift change in the clan’s coding practices, and credited the good example set by the deceased head monk. We should consider his reinstatement, she wrote.