Category Archives: Python

Week 3008

Welp, I’m getting this in a bit late, so I’ll also put in some Music Monday.

But since the world is a pit, you get Cotton Eye Joe Gregorian Chant Nightcore Hardcore Dubstep remix. Do better this week, world! The Senate confirmation hearings for Brett Kavanaugh were a reminder of how little women’s pain matters. The guy is also clearly lying.

Code

My team won one of the 2nd place slots in the Hackathon as “Most Complete Hack”! We are talking about bringing it through to complete prod deployment.

Read up on Conflict Free Replicated Data Types which are a really interesting way to do mergable data structures. Underlies lots of interesting stuff like Redis.

Experimented with Mozilla’s Configman – the docs are NOT good enough, because they actually have a great drop in replacement for the standard ArgumentParser in there.

Bike

Got in all but 1 day last week. Rode in the rain 2 days! The first convinced me that I need to either own rainproof pants or just plan on riding in shorts in the rain. I bought a pair of Showers Pass transit pants and they worked great on Friday. Also got to randomly ride in with my buddy Lance again and reminded how nice that is.

Family

Zeebus is really smart and understands a lot of what we say! She picks things up and puts them where you ask, she understands that we need sleep, she’s got a lot going on upstairs.

She puts on Max’s bike helmet to let us know she wants to go riding!

When she slapped Sam hard in the face she realized she had hurt her- then Z gave her a big hug and patted her. That’s what Max does when he plays too rough with her!

Max and I did a really good hike together. We went a little crazy and climbed up next to a little waterfall and did some semi-safe bouldering nearby.  Max and I actually climbed way up a small cliff and got to the top. We also found a cool Puffball mushroom. But we didn’t eat it because we didn’t know how to tell if it was poisonous or not. (now we know and we’ll eat it next time!)

Week 3006

Work

We are  having a hackathon! I’m excited.  Its my first since Music Hack Day NYC. We’re going to try out Amazon Lex, Lambda, and some containers.

Some folks are already using my code for working with ELK, so that’s nice.

I’ve been focusing around automating my testing and such, and I really am centering around just learning to write better makefiles. Make is installed everywhere, like vim and other things I like. It just works. And makefiles do almost everything you want. The downside is that it is an ugly syntax. The upside is that if you learn one ugly syntax, you don’t need to learn everything about Rake, Yarn, etc.

Around the web

Best Practices for Staging Environments – this article by the excellent Alice Goldfuss came up at work as we wrestle with big calcs and datasets for our clients.

Why you hate Contemporary Architecture – This is grrrreat. One little note for the computer nerds. If you’ve heard of the Gang of 4 Design Patterns book, read the article and come back. The Design Patterns book was based in part on Christopher Alexander’s “A Pattern Language” which is a great guide to things that seem to work in architecture.

I discovered two things that similar to a POC I was working on (voracious-etl)

  • Datasette provides a readonly JSON api for any SQLite DB.
  • Dataset provides an ORM for any CSV or JSON file

The plugin system at the core of pytest is a library: Pluggy. I like that and I might use it earlier.

A good podcast: Flash Forward. Explores a new future every week. The most recent focuses on fungal enslavement, which I love. If you like that one, I highly recommend the mindblowing Parasite Rex by Carl Zimmer or Sensation by Nick Mamatas

 

 

Politics

None of the big progressive candidates made it in the democratic primaries. Cuomo still won. Tish won, which is OK. However, I see that  some progressive candidates made it through.

I hope they can do some real work and win in November. I’ll be calling and doing work to support them.  I long for a day when I can start moaning about free speech vs hate speech and trying to reign in some liberal excesses. However, right now, the work has to get done.

I also note that children are still in cages and parents are getting deported without hearings, but it isn’t in the headlines anymore. I’m still pissed about it. I’m still pissed that the probable governor of NY seems to only work for progressive issues when pushed and won’t use his clemency powers.

Exercise

I biked around 37 miles this week! Had to take off Monday and Tuesday due to rain and being pretty sick. But otherwise, I rode in. Got to stop on Christie street and help a guy who’d been knocked down by a cab.

Worked to a slightly lower pistol squat and my butt hurts sooo bad. Also, my dragon flag work is getting better. I can kind of hold it.

I started doing partial handstand pushups against the wall and they feel pretty good. I can do 5 at a time part way down and up. Next I’ll try lower and lower, then try freestanding ones.

 

Dataclasses coming in Python 3.7

I’ve been loving my time writing in Python. I started with 3.4 I think, and every release has brought something new and useful to the table. All the speed and async improvements are great, but the thing that I loved most in Python 3.6 was the new f string formatting. Removing boilerplate and providing the simplest easiest path just makes every task easier.  Less code on a page means fewer places to make mistakes. So it’s much better to see simple than complex code.


foo = 'bar'
# this is so clear and direct
message = f'Meet me at the {foo}'
# versus
message = 'Meet me at the {location}'.format(location = foo)

In 3.7, I’m excited about dataclasses. It’s like the attrs library – just a simple place to store data where you don’t have to re-implement all the standard dunder methods (__repr__, __str__, __eq__ etc). Adding a dataclass decorator and a list of the fields gives you a class with a standard constructor and all the other bells and whistles. The more you can use the standard library to accomplish high level concepts without having to type more code and write more bugs, the better. It’s coming in Python 3.7, but you can use dataclasses today with Python 3.6 using this backport on github – totally same functionality.

I’ve been playing around with using them here.

Backing up a SalesForce instance

SalesForce is an interesting beast. You gotta work within its limits, and it is great within them. As soon as you want to venture outside of the normal flow, things get complicated.

They suck  into SalesForce, but never out – it’s designed as a lobster trap for your information.

Weirdly, there’s not much on the SalesForce AppExchange that helps you easily back up your data on site. There are some tools that help you easily back up to another cloud, but little that helps you get your data back within your own walls.

Still, there’s a little layer over the SalesForce API in Python called simple-salesforce. Here’s a quick script I threw together to help put all your data into csv files.

I love PETL

When I started at my current job I noticed we we had lots of room for improvement about how we imported and exported data.  Folks had been using the MicroSoft SSIS platform as a way to Extract, Transform and Load data in and out of our database to various files.

SSIS is great for lots of things and has a lot of upsides. It is very drag and drop, folks don’t have to know a lot of programming to get it to do things, and it has lots of functions built in. If you need more programming power, you can execute C# or VB scripts to do the fiddly bits.

But I hate it. ( Don’t worry, we’ll get to the love soon.)

My biggest problems with SSIS:

  • It is unversionable. Try reading a git diff of an SSIS change. The xml is designed for a machine to read, not a human. If you want to know what has changed over time in your world, it’s a problem.
  • You can only use Visual Studio to edit it. Many of our SSIS packages include VB or C# scripts. That sounds fine – but apparently these compile to an undiffable, uneditable blob in the xml that is only recompiled if you save using visual studio. So if you want to change something across many SSIS package scripts, you have to open and resave each one.
  • It hides options under rocks. Finding out how something works requires lots of delving into lotsa windows and dialogues.
  • It changes things unexpectedly. Click in the wrong dialogue and it helpfully re-infers datatypes from a file for you. You don’t know until you go to execute.
  • It slapped my momma. Etc.

I wanted to move my team to something that was better for people.

We need something:

  • That we can diff
  • That we can do code reviews and pull requests on
  • That is simple, expressive and clear.
  • That is powerful.

To me that sounds like a programming language.  I encouraged folks on the team to try accomplishing a couple of tasks that might use an SSIS package instead to use Python. Immediately, things got better. Our code reviews made sense. Code quality improved with every single pull request.

We used pymssql to connect to SqlServer and inserted records as needed after processing them. Navigating and transforming XML docs was easy, CSV files were eaten up by the native DictReader.

And then Derrick found PETL. It’s beautiful. You point it at data and make simple moves to completely transform it. I’m smitten.

I had dozens of files to read from, each a quarterly file for a year – only noted in the file name. Each had a crappy heading line that preceded column headers. I needed to put them into 1 file for loading into SalesForce Wave. Whacking together a solution with PETL was effortless. Line 36 is where the PETL starts, and it’s so small and good that it is nice to see how much it encapsulates.

How to migrate your WordPress Blog between hosts.

My boss Mike needed to move his wine review blog from a friend’s hosting on lunarpages. I suggested he try dreamhost and he liked it – in a few minutes he had signed up for a free trial and used their 1-click install to set up a new install of wordpress.

Before he moved his domain to point from lunarpages to dreamhost I got him to prep by writing down a few important pieces of info. I’m trying to make sure I make this easier for other friends like I did when I helped Tove’s Thread For Thought move from WordPress.org to her own host.

Things to do before you change your domain to point to your new hosting

      Write down the name of your THEME. If you want to use the same theme, it’s important to write this down before you make the switch.
      Export your blog content from wordpress.
      Download your images. The wordpress export guide pretends this is easy, but it isn’t. If you are using the same domain name, I’m not sure what the easy way to do this is.

How to download your images

I wrote a python script that does this for you.
Make sure your system supports python. Next install BeautifulSoup – a great html parser for python.
Once that’s done, download this little script and change home and filesUrl to be your domain name.
Run the script, it should crawl your domain and download all of the images you host. Now follow the same steps of editing your export if needed and upload it all into your new blog at your new domain.

Hope that helps!

Project Idea – Syncing ebook reader

The bookworm logo

Here’s the setup. O’Reilly hosts a django based open source ebook reading website called bookworm.  You can run bookworm on your own server.  I opened a ticket on bookworm’s bugtracker to provide an api method to update where you are in a book.  Next you update Aldiko (Not open source, but perhaps we can write a plugin for it) and FbreaderJ to use that method when they exit to update where you stopped reading.

Upshot: You open a book on your phone and read it.  It syncs with your server with a bookmark of where you stopped reading.  Then you go to your website, and begin reading from where you left off.  And so on. Perhaps your phone also detects when it gets a new ebook and uploads that to your server or downloads a new book from your server when one shows up as well.

My Valentines / Music Hack Day NYC app!

-or-
Why I wasn’t around this weekend.

I was at NYC Music Hack Day!

You can try out the idea here in our valentune.es beta. We send love and music over wires and wireless to your special sweetheart.

You put in your sweetheart’s Name & cellphone, a couple of key words about your them and then ask the app to get busy.

It goes out to the MusixMatch lyrics service – finds lyrics that describe your sweetheart and then gives you a list of songs we can find streaming full mp3s online for. You choose what you don’t want in the play list and hit send.

We call your sweetheart up using the Twilio API , give them your sweet message and then play them your custom made mixtape over the phone! Yes, my lovely wife Sam was the first non-developer to get a call from the valentun.es robot.

Getting SongsPick songs on the iPhone!

Don’t have a web browser handy? Surely you jest! If you’ve got an iPhone you can use the iPhone app.

If not, the website is mobile compliant and works great on Android phones. The website uses CSS3 and WebFonts for style, but degrades well to older browsers. How old? IT WORKS ON LYNX!in case you need web 3.0 from a terminal

So all of this was accomplished by 6 talented and dedicated people over the course of 24 hours in NYC. The whole team was positive and awesome. I’m really proud of what we did so quickly. It’s got bugs 1 , but it works. I didn’t sleep the whole 24 hour hack session and I worked on a lot of the bits.

  • The Musixmatch call to find lyrics that talk about your sweetheart.
  • The Skreemr call to find playable songs with those lyrics
  • Adapting Jeff’s web mockups to Django templates
  • Calling into Alex’s Twilio wrapper
  • Pitching the idea and the vision and recruiting the group
  • Helping design the process flow and settling on using Django with Nate
  • Helping coordinate who would do what and figuring out the pitch with Jeff
  • Presenting the whole thing in 2 minutes to over 300 people with Anna and Nate

It was a busy 24 hours!

  1. the Title element has the wrong spelling of valentunes and the message doesn’t always go through and the songs sometimes take a long time to play, etc…   (back)

Tweetability and Readability

Robin wrote a good story: Last Beautiful.  http://ur1.ca/sees
I thought Robin’s writing was “twitterish” – most of the sentences were under 140 characters. http://ur1.ca/seex
Maybe microblogging pushed him to write better, snappier sentences? http://ur1.ca/sewq
Sounds like a hypothesis – let’s make that story easy to parse: http://ur1.ca/sefl
Time to break out the python and turn this story into tweets: http://gist.github.com/348616

Continue reading Tweetability and Readability