Tag Archives: Python

One more thing to do after upgrading python

This is mainly a note-to-self to remind me for my next upgrade – but hope it helps you too. TL;DR – install python-dev tools, export old installs into a requirements file, import into new python site-packages.

I upgraded the python on my laptop from 3.6 to 3.7 so I could use dataclasses for a little project. Then I got on a plane to Zurich and planned to get some work done on the 9 hour flight. Unfortunately I spent most of that time wrangling my python install – little cli tools I like to use like black, glances, sphinx, cookiecutter, etc – none of them worked!

When you do a python pip install of a library, it puts the library in a directory called site-packages under lib/python3.<your-version>/site-packages/<your-package-name>. If the library has defined command line entry-points, you will find it has also installed a file under bin/<your-package-name>. If bin is in your PATH, it means you can just type something like cookiecutter and it will call a function in the library to do stuff for you! Here’s what one of those files looks like:

➜  ~ bat ~/.local/bin/cookiecutter
───────┬───────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /home/matt/.local/bin/cookiecutter
───────┼───────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/python3
   2   │
   3   │ # -*- coding: utf-8 -*-
   4   │ import re
   5   │ import sys
   6   │
   7   │ from cookiecutter.__main__ import main
   8   │
   9   │ if __name__ == "__main__":
  10   │     sys.argv[0] = re.sub(r"(-script\.pyw?|\.exe)?$", "", sys.argv[0])
  11   │     sys.exit(main())
───────┴───────────────────────────────────────────────────────────────────────────────────────────────
➜  ~

It tells the shell to execute it using /usr/bin/python3 – that’s a link to the latest version of python3 – in this case it’s now python3.7, while the library was installed under python3.6. So the command line now fails! Python3.7 doesn’t know about libs installed for 3.6 at all.

On the flight I realized I could either copy from the python3.6 site-lib to the python3.7 one or just switch python3 to point back to python3.6 to make things work. But once you have internet access, here’s the magic to get it all working.

First, install the python3.7 dev stuff so you can compile anything that needs compilation.

➜  ~ sudo apt-get install python3.7-dev

Then let’s export everything we used to use under 3.6 and then reinstall it under 3.7.

➜  ~ python3.6 -m pip freeze > old_requirements.txt
➜  ~ python3.7 -m pip install --user -r old_requirements.txt 

Week 3011

Getting ready for my first trip to Hyderabad!

Health

Been working on dragon flags and handstand pushups at the gym. Only 3 biking days this week :(. On the other hand, they were all good days. It’s cold enough now to break out the gloves here in NYC, so I’m looking forward to warmer days in Hyd for the next few weeks. I wonder if I can rent a bike there easily?

Family

One of the reasons for that: I got to take Max Lazer to go see Tinga Tinga. He got to see it, I just got to be part of the parental escort through the subways to Times Square. He handled it great, but it was amazing to see the way the teachers handle 22 kindergartners getting through the subway system.

Z is teething and it’s so tough on her. I hope it ends soon. On the other hand, I managed to get both kids to sleep in the same room at the same time on Wednesday and Thursday we got Z to sleep the whole night through!

My Kobo broke and I sent it off to be replaced, but I must have an e-reader for the trip so I am now the owner of a Kindle Paperwhite as well. As someone who’s now owned and used a Nook, Kobo and Kindle, I’ll write up my take on it. Should have some time on the 16 hours or so of flight time!

Code

Been helping folks at work learn some more advanced Python. I get excited every time I talk about list or dictionary comprehensions because they were a real light when I was learning python in the beginning.

You often need to transform data structures and the first way is usually to just iterate a list, construct new instances of something else, then pop them into a list of new things.

Say you have a list of cool coders you want to talk about.

 


from collections import namedtuple
Coder = namedtuple('Person', 'name login url')
cool_coders = [Coder(name="Grace Hopper", login="ghopper", url="https://en.wikipedia.org/wiki/Grace_Hopper"),
Coder(name="Ada Lovelace", login="ladyada", url="https://en.wikipedia.org/wiki/Ada_Lovelace"),
Coder(name="Jenn Schiffer", login="jennmoney", url="https://jennmoney.biz/"),
Coder(name="Nina Zharenko", login="nnja", url="https://www.nnja.io/"),
]

intro = "Here's a list of some cool coders and more you can learn about them: "

Maybe you want to produce a markdown sentence listing them.

coder_markup = []
for programmer in cool_coders:
    coder_markup.append( "[{}]({})".format(programmer.name, programmer.url) )
markup = intro + ", ".join(coder_markup)

Sure, that could be more complicated and have more changes or filtering that needs to be done. But what’s nice about a comprehension is that it helps cut down boilerplate and it makes things clearer.

coder_markup2 = ["[{}]({})".format(coder.name, coder.url) for coder in cool_coders]
markup = intro + ", ".join(coder_markup2)

And of course, if we were on python 3.6 we could make that even smaller by just using fstrings.

I’ve been using tmux pretty heavily, but now I get annoyed that on reboot I have to rebuild all of my workspaces. So now I’ve installed tmuxinator. It should be useful for setting up project/workspaces and then firing them up after a reboot.

Of course – I don’t just install tmuxinator. I set it so that it can auto-install on any account I’ve got with my jumpstart script.  If you know any cool tricks or experiences you want to share about either, I’m eager to learn.

 

Week 3006

Work

We are  having a hackathon! I’m excited.  Its my first since Music Hack Day NYC. We’re going to try out Amazon Lex, Lambda, and some containers.

Some folks are already using my code for working with ELK, so that’s nice.

I’ve been focusing around automating my testing and such, and I really am centering around just learning to write better makefiles. Make is installed everywhere, like vim and other things I like. It just works. And makefiles do almost everything you want. The downside is that it is an ugly syntax. The upside is that if you learn one ugly syntax, you don’t need to learn everything about Rake, Yarn, etc.

Around the web

Best Practices for Staging Environments – this article by the excellent Alice Goldfuss came up at work as we wrestle with big calcs and datasets for our clients.

Why you hate Contemporary Architecture – This is grrrreat. One little note for the computer nerds. If you’ve heard of the Gang of 4 Design Patterns book, read the article and come back. The Design Patterns book was based in part on Christopher Alexander’s “A Pattern Language” which is a great guide to things that seem to work in architecture.

I discovered two things that similar to a POC I was working on (voracious-etl)

  • Datasette provides a readonly JSON api for any SQLite DB.
  • Dataset provides an ORM for any CSV or JSON file

The plugin system at the core of pytest is a library: Pluggy. I like that and I might use it earlier.

A good podcast: Flash Forward. Explores a new future every week. The most recent focuses on fungal enslavement, which I love. If you like that one, I highly recommend the mindblowing Parasite Rex by Carl Zimmer or Sensation by Nick Mamatas

 

 

Politics

None of the big progressive candidates made it in the democratic primaries. Cuomo still won. Tish won, which is OK. However, I see that  some progressive candidates made it through.

I hope they can do some real work and win in November. I’ll be calling and doing work to support them.  I long for a day when I can start moaning about free speech vs hate speech and trying to reign in some liberal excesses. However, right now, the work has to get done.

I also note that children are still in cages and parents are getting deported without hearings, but it isn’t in the headlines anymore. I’m still pissed about it. I’m still pissed that the probable governor of NY seems to only work for progressive issues when pushed and won’t use his clemency powers.

Exercise

I biked around 37 miles this week! Had to take off Monday and Tuesday due to rain and being pretty sick. But otherwise, I rode in. Got to stop on Christie street and help a guy who’d been knocked down by a cab.

Worked to a slightly lower pistol squat and my butt hurts sooo bad. Also, my dragon flag work is getting better. I can kind of hold it.

I started doing partial handstand pushups against the wall and they feel pretty good. I can do 5 at a time part way down and up. Next I’ll try lower and lower, then try freestanding ones.

 

Week 3004

Z-Ray is walking like crazy now. Her tiny feet are hilarious and she now just toddles everywhere. This morning she just ran into a room and hugged me.  She brought me my shoes!  She definitely understands when I ask her to put her shoes into the shoe bench.

I love her so much I want to squeeze her for hours.

Max Lazer is going to kindergarten next week! He’s not excited yet, but I think he’ll love it. Just got a call that his ear is large and swollen, but hopefully he’s ok. We’ve been reading a lot of choose your own adventure – I like that he’s seeing how different choices control consequences. When Z cries, he’s very loving and helpful and is doing his best to comfort her or distract her.

Code

I’m thinking about how we will enhance our python support at work. As part of that I’m researching better ways to standardize configuration of objects.  You want to be able to initialize an object foo from a class Foo.

class Foo(object):
    def __init__(bar='baz')
        self.bar = bar
foo = Foo()

print(f"bar = {foo.bar}")
# "bar = baz"

Swell!  But some people may think that bar should be set to “bazzz”.

foo2 = Foo(bar='bazzz')
print(f"bar = {foo2.bar}")
# "bar = bazzz"

Easy enough. But what if I work in a place where we need to change hundreds of uses of bar among tons of different scripts and applications and whatnot?

I may want to get it from an environment variable automatically on startup. Or maybe I work in a place where we like configuring things with a .env file. Or a settings.toml.

Or maybe we want to just this once pass it in as a commandline parameter. Flexibility is really good to have.

Ideally, we can say there’s a hierarchy.

  1. A parameter passed into the constructor is the highest priority.
  2. An environment variable is the next.
  3. A config file is the next most concrete. (I could be convinced to switch these, but as long as there’s a consistent pattern, it’s good enough)
  4. The default value of the parameter in a constructor if there’s not other choice.

There are lots of good libraries out there for achieving this, at least partially. Ideally, I think I’d want to have either something we can apply as a class decorator or as through multiple inheritance.

Some good python configuration libraries I came across that I really like:

For my problem with dreamhost, I’m making progress and have been able to get a set of forward rules pretty much ready. I’m hoping to publish it pretty soon.

I don’t think I’ll get around to auto fixing the mailbox filter rules though.

Reading

I got my Kobo back after it fizzled. I’ve missed having an e-reader so much! I did get to read “The Moon is Down” by Steinbeck because we had a paperback on our shelves, but it’s so nice to be able to read a book in bed or in the bath or in the dark while a kid settles down. It’s also nice to be able to just squeeze in A Road to Common Lisp in between things because of the integration with Pocket.

Bike

Only 50 miles this week. I skipped a day because there were some team drinks on Tuesday.

I also was gently hit by a car. An Uber driver pulled into the bike lane while I was riding to drop off his passenger. As it was happening I was pounding on his window and yelling stop. I was powerless, a thing I’ve known intellectually but I never really believe. He didn’t stop, he knocked me over. I ran up to his window and explained that I am a person, that people die this way and that my children want to see me. I’m fine. Scrapes and bruises and a tendency to replay it in my head.

It spurred me to donate to Transportation Alternatives. They advocate for better laws, for protected bike lanes that don’t get used as parking spaces. For bike lanes that aren’t truck unloading zones. Things that can really save my life and help me get home.

I also finally replaced my speaker.  It’s louder, but I’m not sure how awesome.  It’s designed with push buttons instead of a volume dial, it fucking talks to you instead of doing things and it isn’t clear what it’s doing. Physical interfaces are SO MUCH BETTER than this. On the other hand, it is louder and my old one was broken.

Politics

A good thing you can do about weeknotes is talk about what is going on in the world so you can notice slow changes and not think things were always the way they are.

The government has started confiscating passports from US Citizens who are Latino. There are a very small number of cases where some midwives fraudulently filled out birth certificates. This is the pretext that a racist administration is using to attack Hispanics. Midwife births are more common for those who cannot afford hospitals or who live in areas that are too poor to support a hospital. This is not a thing that started under the current administration – but the character of what is happening has changed.

I am renewed in my determination to work continuously to replace these terrible people with less terrible people. I would love to do that with you! If you wonder what you would do in a time of crisis, the answer is that it is what you are doing right now.

Boob sweat

My buddy Dawn Hunter is smart and charming as hell. She is working on a startup that helps with sweaty ta-tas. I’m no expert, but for people that have boobs and want more comfort with them, she’s writing tips on managing boob sweat and gathering interest from people who might want a product. Get in touch with her!

Dataclasses coming in Python 3.7

I’ve been loving my time writing in Python. I started with 3.4 I think, and every release has brought something new and useful to the table. All the speed and async improvements are great, but the thing that I loved most in Python 3.6 was the new f string formatting. Removing boilerplate and providing the simplest easiest path just makes every task easier.  Less code on a page means fewer places to make mistakes. So it’s much better to see simple than complex code.


foo = 'bar'
# this is so clear and direct
message = f'Meet me at the {foo}'
# versus
message = 'Meet me at the {location}'.format(location = foo)

In 3.7, I’m excited about dataclasses. It’s like the attrs library – just a simple place to store data where you don’t have to re-implement all the standard dunder methods (__repr__, __str__, __eq__ etc). Adding a dataclass decorator and a list of the fields gives you a class with a standard constructor and all the other bells and whistles. The more you can use the standard library to accomplish high level concepts without having to type more code and write more bugs, the better. It’s coming in Python 3.7, but you can use dataclasses today with Python 3.6 using this backport on github – totally same functionality.

I’ve been playing around with using them here.

Backing up a SalesForce instance

SalesForce is an interesting beast. You gotta work within its limits, and it is great within them. As soon as you want to venture outside of the normal flow, things get complicated.

They suck  into SalesForce, but never out – it’s designed as a lobster trap for your information.

Weirdly, there’s not much on the SalesForce AppExchange that helps you easily back up your data on site. There are some tools that help you easily back up to another cloud, but little that helps you get your data back within your own walls.

Still, there’s a little layer over the SalesForce API in Python called simple-salesforce. Here’s a quick script I threw together to help put all your data into csv files.

I love PETL

When I started at my current job I noticed we we had lots of room for improvement about how we imported and exported data.  Folks had been using the MicroSoft SSIS platform as a way to Extract, Transform and Load data in and out of our database to various files.

SSIS is great for lots of things and has a lot of upsides. It is very drag and drop, folks don’t have to know a lot of programming to get it to do things, and it has lots of functions built in. If you need more programming power, you can execute C# or VB scripts to do the fiddly bits.

But I hate it. ( Don’t worry, we’ll get to the love soon.)

My biggest problems with SSIS:

  • It is unversionable. Try reading a git diff of an SSIS change. The xml is designed for a machine to read, not a human. If you want to know what has changed over time in your world, it’s a problem.
  • You can only use Visual Studio to edit it. Many of our SSIS packages include VB or C# scripts. That sounds fine – but apparently these compile to an undiffable, uneditable blob in the xml that is only recompiled if you save using visual studio. So if you want to change something across many SSIS package scripts, you have to open and resave each one.
  • It hides options under rocks. Finding out how something works requires lots of delving into lotsa windows and dialogues.
  • It changes things unexpectedly. Click in the wrong dialogue and it helpfully re-infers datatypes from a file for you. You don’t know until you go to execute.
  • It slapped my momma. Etc.

I wanted to move my team to something that was better for people.

We need something:

  • That we can diff
  • That we can do code reviews and pull requests on
  • That is simple, expressive and clear.
  • That is powerful.

To me that sounds like a programming language.  I encouraged folks on the team to try accomplishing a couple of tasks that might use an SSIS package instead to use Python. Immediately, things got better. Our code reviews made sense. Code quality improved with every single pull request.

We used pymssql to connect to SqlServer and inserted records as needed after processing them. Navigating and transforming XML docs was easy, CSV files were eaten up by the native DictReader.

And then Derrick found PETL. It’s beautiful. You point it at data and make simple moves to completely transform it. I’m smitten.

I had dozens of files to read from, each a quarterly file for a year – only noted in the file name. Each had a crappy heading line that preceded column headers. I needed to put them into 1 file for loading into SalesForce Wave. Whacking together a solution with PETL was effortless. Line 36 is where the PETL starts, and it’s so small and good that it is nice to see how much it encapsulates.

How to migrate your WordPress Blog between hosts.

My boss Mike needed to move his wine review blog from a friend’s hosting on lunarpages. I suggested he try dreamhost and he liked it – in a few minutes he had signed up for a free trial and used their 1-click install to set up a new install of wordpress.

Before he moved his domain to point from lunarpages to dreamhost I got him to prep by writing down a few important pieces of info. I’m trying to make sure I make this easier for other friends like I did when I helped Tove’s Thread For Thought move from WordPress.org to her own host.

Things to do before you change your domain to point to your new hosting

      Write down the name of your THEME. If you want to use the same theme, it’s important to write this down before you make the switch.
      Export your blog content from wordpress.
      Download your images. The wordpress export guide pretends this is easy, but it isn’t. If you are using the same domain name, I’m not sure what the easy way to do this is.

How to download your images

I wrote a python script that does this for you.
Make sure your system supports python. Next install BeautifulSoup – a great html parser for python.
Once that’s done, download this little script and change home and filesUrl to be your domain name.
Run the script, it should crawl your domain and download all of the images you host. Now follow the same steps of editing your export if needed and upload it all into your new blog at your new domain.

Hope that helps!

Tweetability and Readability

Robin wrote a good story: Last Beautiful.  http://ur1.ca/sees
I thought Robin’s writing was “twitterish” – most of the sentences were under 140 characters. http://ur1.ca/seex
Maybe microblogging pushed him to write better, snappier sentences? http://ur1.ca/sewq
Sounds like a hypothesis – let’s make that story easy to parse: http://ur1.ca/sefl
Time to break out the python and turn this story into tweets: http://gist.github.com/348616

Continue reading Tweetability and Readability