Category Archives: Python

Backing up a SalesForce instance

SalesForce is an interesting beast. You gotta work within its limits, and it is great within them. As soon as you want to venture outside of the normal flow, things get complicated.

They suck  into SalesForce, but never out – it’s designed as a lobster trap for your information.

Weirdly, there’s not much on the SalesForce AppExchange that helps you easily back up your data on site. There are some tools that help you easily back up to another cloud, but little that helps you get your data back within your own walls.

Still, there’s a little layer over the SalesForce API in Python called simple-salesforce. Here’s a quick script I threw together to help put all your data into csv files.

I love PETL

When I started at my current job I noticed we we had lots of room for improvement about how we imported and exported data.  Folks had been using the MicroSoft SSIS platform as a way to Extract, Transform and Load data in and out of our database to various files.

SSIS is great for lots of things and has a lot of upsides. It is very drag and drop, folks don’t have to know a lot of programming to get it to do things, and it has lots of functions built in. If you need more programming power, you can execute C# or VB scripts to do the fiddly bits.

But I hate it. ( Don’t worry, we’ll get to the love soon.)

My biggest problems with SSIS:

  • It is unversionable. Try reading a git diff of an SSIS change. The xml is designed for a machine to read, not a human. If you want to know what has changed over time in your world, it’s a problem.
  • You can only use Visual Studio to edit it. Many of our SSIS packages include VB or C# scripts. That sounds fine – but apparently these compile to an undiffable, uneditable blob in the xml that is only recompiled if you save using visual studio. So if you want to change something across many SSIS package scripts, you have to open and resave each one.
  • It hides options under rocks. Finding out how something works requires lots of delving into lotsa windows and dialogues.
  • It changes things unexpectedly. Click in the wrong dialogue and it helpfully re-infers datatypes from a file for you. You don’t know until you go to execute.
  • It slapped my momma. Etc.

I wanted to move my team to something that was better for people.

We need something:

  • That we can diff
  • That we can do code reviews and pull requests on
  • That is simple, expressive and clear.
  • That is powerful.

To me that sounds like a programming language.  I encouraged folks on the team to try accomplishing a couple of tasks that might use an SSIS package instead to use Python. Immediately, things got better. Our code reviews made sense. Code quality improved with every single pull request.

We used pymssql to connect to SqlServer and inserted records as needed after processing them. Navigating and transforming XML docs was easy, CSV files were eaten up by the native DictReader.

And then Derrick found PETL. It’s beautiful. You point it at data and make simple moves to completely transform it. I’m smitten.

I had dozens of files to read from, each a quarterly file for a year – only noted in the file name. Each had a crappy heading line that preceded column headers. I needed to put them into 1 file for loading into SalesForce Wave. Whacking together a solution with PETL was effortless. Line 36 is where the PETL starts, and it’s so small and good that it is nice to see how much it encapsulates.

How to migrate your WordPress Blog between hosts.

My boss Mike needed to move his wine review blog from a friend’s hosting on lunarpages. I suggested he try dreamhost and he liked it – in a few minutes he had signed up for a free trial and used their 1-click install to set up a new install of wordpress.

Before he moved his domain to point from lunarpages to dreamhost I got him to prep by writing down a few important pieces of info. I’m trying to make sure I make this easier for other friends like I did when I helped Tove’s Thread For Thought move from to her own host.

Things to do before you change your domain to point to your new hosting

      Write down the name of your THEME. If you want to use the same theme, it’s important to write this down before you make the switch.
      Export your blog content from wordpress.
      Download your images. The wordpress export guide pretends this is easy, but it isn’t. If you are using the same domain name, I’m not sure what the easy way to do this is.

How to download your images

I wrote a python script that does this for you.
Make sure your system supports python. Next install BeautifulSoup – a great html parser for python.
Once that’s done, download this little script and change home and filesUrl to be your domain name.
Run the script, it should crawl your domain and download all of the images you host. Now follow the same steps of editing your export if needed and upload it all into your new blog at your new domain.

Hope that helps!

Project Idea – Syncing ebook reader

The bookworm logo

Here’s the setup. O’Reilly hosts a django based open source ebook reading website called bookworm.  You can run bookworm on your own server.  I opened a ticket on bookworm’s bugtracker to provide an api method to update where you are in a book.  Next you update Aldiko (Not open source, but perhaps we can write a plugin for it) and FbreaderJ to use that method when they exit to update where you stopped reading.

Upshot: You open a book on your phone and read it.  It syncs with your server with a bookmark of where you stopped reading.  Then you go to your website, and begin reading from where you left off.  And so on. Perhaps your phone also detects when it gets a new ebook and uploads that to your server or downloads a new book from your server when one shows up as well.

My Valentines / Music Hack Day NYC app!

Why I wasn’t around this weekend.

I was at NYC Music Hack Day!

You can try out the idea here in our beta. We send love and music over wires and wireless to your special sweetheart.

You put in your sweetheart’s Name & cellphone, a couple of key words about your them and then ask the app to get busy.

It goes out to the MusixMatch lyrics service – finds lyrics that describe your sweetheart and then gives you a list of songs we can find streaming full mp3s online for. You choose what you don’t want in the play list and hit send.

We call your sweetheart up using the Twilio API , give them your sweet message and then play them your custom made mixtape over the phone! Yes, my lovely wife Sam was the first non-developer to get a call from the robot.

Getting SongsPick songs on the iPhone!

Don’t have a web browser handy? Surely you jest! If you’ve got an iPhone you can use the iPhone app.

If not, the website is mobile compliant and works great on Android phones. The website uses CSS3 and WebFonts for style, but degrades well to older browsers. How old? IT WORKS ON LYNX!in case you need web 3.0 from a terminal

So all of this was accomplished by 6 talented and dedicated people over the course of 24 hours in NYC. The whole team was positive and awesome. I’m really proud of what we did so quickly. It’s got bugs 1 , but it works. I didn’t sleep the whole 24 hour hack session and I worked on a lot of the bits.

  • The Musixmatch call to find lyrics that talk about your sweetheart.
  • The Skreemr call to find playable songs with those lyrics
  • Adapting Jeff’s web mockups to Django templates
  • Calling into Alex’s Twilio wrapper
  • Pitching the idea and the vision and recruiting the group
  • Helping design the process flow and settling on using Django with Nate
  • Helping coordinate who would do what and figuring out the pitch with Jeff
  • Presenting the whole thing in 2 minutes to over 300 people with Anna and Nate

It was a busy 24 hours!

  1. the Title element has the wrong spelling of valentunes and the message doesn’t always go through and the songs sometimes take a long time to play, etc…   (back)

Tweetability and Readability

Robin wrote a good story: Last Beautiful.
I thought Robin’s writing was “twitterish” – most of the sentences were under 140 characters.
Maybe microblogging pushed him to write better, snappier sentences?
Sounds like a hypothesis – let’s make that story easy to parse:
Time to break out the python and turn this story into tweets:

Continue reading Tweetability and Readability

Android Scripting Environment is heating up

The Android Scripting Environment is to the Android phone as Greasemonkey is to Firefox.   It’s a simple place to do quick hacks for your phone without having to build a full plugin or application.

Two cool scripts that just got released are Silent Night and Droid Track. Continue reading Android Scripting Environment is heating up

Mixins, Django, Python and adding to Models

I’m working with my buddy ADubs, esq. on a project where we make it easier to add federation to various models. We’re properly lazy programmers, so we don’t want to rewrite federation for each model we are creating, we just want to mix it in as a general set of functionality.  Below the fold, more details, sample code, etc.
Continue reading Mixins, Django, Python and adding to Models