Category Archives: Dev

Week 3008

Music, Pals, Python, Software, , ,

Welp, I’m getting this in a bit late, so I’ll also put in some Music Monday.

But since the world is a pit, you get Cotton Eye Joe Gregorian Chant Nightcore Hardcore Dubstep remix. Do better this week, world! The Senate confirmation hearings for Brett Kavanaugh were a reminder of how little women’s pain matters. The guy is also clearly lying.

Code

My team won one of the 2nd place slots in the Hackathon as “Most Complete Hack”! We are talking about bringing it through to complete prod deployment.

Read up on Conflict Free Replicated Data Types which are a really interesting way to do mergable data structures. Underlies lots of interesting stuff like Redis.

Experimented with Mozilla’s Configman – the docs are NOT good enough, because they actually have a great drop in replacement for the standard ArgumentParser in there.

Bike

Got in all but 1 day last week. Rode in the rain 2 days! The first convinced me that I need to either own rainproof pants or just plan on riding in shorts in the rain. I bought a pair of Showers Pass transit pants and they worked great on Friday. Also got to randomly ride in with my buddy Lance again and reminded how nice that is.

Family

Zeebus is really smart and understands a lot of what we say! She picks things up and puts them where you ask, she understands that we need sleep, she’s got a lot going on upstairs.

She puts on Max’s bike helmet to let us know she wants to go riding!

When she slapped Sam hard in the face she realized she had hurt her- then Z gave her a big hug and patted her. That’s what Max does when he plays too rough with her!

Max and I did a really good hike together. We went a little crazy and climbed up next to a little waterfall and did some semi-safe bouldering nearby.  Max and I actually climbed way up a small cliff and got to the top. We also found a cool Puffball mushroom. But we didn’t eat it because we didn’t know how to tell if it was poisonous or not. (now we know and we’ll eat it next time!)

Week 3006

Pals, Python, , , , , , ,

Work

We are  having a hackathon! I’m excited.  Its my first since Music Hack Day NYC. We’re going to try out Amazon Lex, Lambda, and some containers.

Some folks are already using my code for working with ELK, so that’s nice.

I’ve been focusing around automating my testing and such, and I really am centering around just learning to write better makefiles. Make is installed everywhere, like vim and other things I like. It just works. And makefiles do almost everything you want. The downside is that it is an ugly syntax. The upside is that if you learn one ugly syntax, you don’t need to learn everything about Rake, Yarn, etc.

Around the web

Best Practices for Staging Environments – this article by the excellent Alice Goldfuss came up at work as we wrestle with big calcs and datasets for our clients.

Why you hate Contemporary Architecture – This is grrrreat. One little note for the computer nerds. If you’ve heard of the Gang of 4 Design Patterns book, read the article and come back. The Design Patterns book was based in part on Christopher Alexander’s “A Pattern Language” which is a great guide to things that seem to work in architecture.

I discovered two things that similar to a POC I was working on (voracious-etl)

  • Datasette provides a readonly JSON api for any SQLite DB.
  • Dataset provides an ORM for any CSV or JSON file

The plugin system at the core of pytest is a library: Pluggy. I like that and I might use it earlier.

A good podcast: Flash Forward. Explores a new future every week. The most recent focuses on fungal enslavement, which I love. If you like that one, I highly recommend the mindblowing Parasite Rex by Carl Zimmer or Sensation by Nick Mamatas

 

 

Politics

None of the big progressive candidates made it in the democratic primaries. Cuomo still won. Tish won, which is OK. However, I see that  some progressive candidates made it through.

I hope they can do some real work and win in November. I’ll be calling and doing work to support them.  I long for a day when I can start moaning about free speech vs hate speech and trying to reign in some liberal excesses. However, right now, the work has to get done.

I also note that children are still in cages and parents are getting deported without hearings, but it isn’t in the headlines anymore. I’m still pissed about it. I’m still pissed that the probable governor of NY seems to only work for progressive issues when pushed and won’t use his clemency powers.

Exercise

I biked around 37 miles this week! Had to take off Monday and Tuesday due to rain and being pretty sick. But otherwise, I rode in. Got to stop on Christie street and help a guy who’d been knocked down by a cab.

Worked to a slightly lower pistol squat and my butt hurts sooo bad. Also, my dragon flag work is getting better. I can kind of hold it.

I started doing partial handstand pushups against the wall and they feel pretty good. I can do 5 at a time part way down and up. Next I’ll try lower and lower, then try freestanding ones.

 

Dataclasses coming in Python 3.7

Dev, Python, , ,

I’ve been loving my time writing in Python. I started with 3.4 I think, and every release has brought something new and useful to the table. All the speed and async improvements are great, but the thing that I loved most in Python 3.6 was the new f string formatting. Removing boilerplate and providing the simplest easiest path just makes every task easier.  Less code on a page means fewer places to make mistakes. So it’s much better to see simple than complex code.


foo = 'bar'
# this is so clear and direct
message = f'Meet me at the {foo}'
# versus
message = 'Meet me at the {location}'.format(location = foo)

In 3.7, I’m excited about dataclasses. It’s like the attrs library – just a simple place to store data where you don’t have to re-implement all the standard dunder methods (__repr__, __str__, __eq__ etc). Adding a dataclass decorator and a list of the fields gives you a class with a standard constructor and all the other bells and whistles. The more you can use the standard library to accomplish high level concepts without having to type more code and write more bugs, the better. It’s coming in Python 3.7, but you can use dataclasses today with Python 3.6 using this backport on github – totally same functionality.

I’ve been playing around with using them here.

How to not fail at support

Dev, Pals,

Support is a really neglected part of a lot of development jobs. You want to just keep writing code and making releases. It’s easy to get into the code and forget that the users are the reason you get paid to write it! If you find you’re doing your best and solving problems, but don’t seem to get the credit or positive results to show it, this might help you. Also, the title is “How to not fail at support” – because this is not about solving the problem or techniques for doing the fix – this is about how to not lose the encounter even when you solve the problem.

All Code has Bugs

You’re going to have them. Nothing is perfect. Your plan for any software has to include support or you’re gonna have a bad day.

Why people get angry at you when you are trying to help them

People need help and when people need help they are vulnerable. They are trying to do something and it is important to their ability to succeed at their job. If they don’t need it to succeed at their job, they won’t bother you about it. If they do need help, they want it as soon as possible and they want to get back to doing their work.

H.A.R.E.M solves the problem

The best way method I’ve found to keep sane and make sure every user is happy with support is to follow the H.A.R.E.M method. There is no silver bullet for software development, but there is definitely a silver bullet for handling human beings who need support without them getting angry about it. I didn’t come up with this, but I evangelize it like crazy.

It sounds stupid simple, because it is. Doing it is harder than understanding it.

  • Hear the request
  • Acknowledge the request
  • Restate the request
  • Estimate how long until you will get back to them.
  • Meet or beat your estimate.

That’s it! It handles the customer service part perfectly. Do this and all you have to do is go find that race condition that disappears when a debugger is running!

Hear the Request

You need to have someone actually monitoring incoming support requests. This is harder than it sounds. Not hearing the request is a terrible problem – because they’ve reported an issue and no one is actually working on it.

If you don’t have a clear, easy dedicated, and monitored pipe for users to tell you about their problems, you need to fix that immediately. When someone doesn’t know where to go for help or when they go for support and no one looks at the queue, you have a big problem. Your users will hate you, and you’ll deserve it. If you’ve ever called for help and it hasn’t arrived, you understand the feeling of betrayal and hopelessness that you are associating with your labor.

Poor software with excellent support is a better deal for users than good software with no support. We know, because they actually use poor software with excellent support more.

Think through the experience of asking for help. You are trying to get help. It’s bad enough so that you can’t fix it yourself – so you are helpless. You want to get help – to ask for support. If you don’t know where to go to reliably get help, you will feel despair and anger.

Acknowledge the request

If someone tries to get support but they feel like they are shouting into the darkness, that’s the same as if they aren’t getting support. One of the big reasons people call instead of entering a ticket is that people don’t know if the ticket is getting looked at. If someone is calling, they know that a person has heard them.

Think yourself through the experience of asking for help. You are trying to get help. It’s bad enough that you can’t fix it yourself – so you are helpless. You want to get help – to ask for support. You ask for the support and hear nothing. You will feel anxiety until you know that someone has heard you. You’ll “check in on them.” You’ll call to follow-up. You’ll do anything to make sure that someone is actually going to work on your problem.

When you acknowledge the request – you just say “I heard you and I’m working on the problem.” That is such a soothing thing to hear. Someone knows the house is on fire and help is coming!

This is super important!

  1. Users can chill out and go work on other things until you solve the problem
  2. If people know that submitting an email or ticket online gets acknowledged, then they can do that without tying up someone on the phone.

A script

“Hi Allie, I saw your ticket and we’re re-imprinting Asimov’s three rules on the floor robot’s positronic brain.”

“Hi Bob, I saw your message. I’ve unlocked your account for you and you can reset your password at the help portal. Happy to help, let me know if you’ve got any more issues with it.”

“Hi Carla, I saw your request. You’ve hit the hard drive limits under our policy.  I’m attaching a quick guide to how to slim your hard drive footprint that other folks have found helpful. If that doesn’t work we can send a note to your manager and the CTO to see if they will approve an exception.”

Restate the request

Support requests are difficult. Don’t work on the wrong one. I’ve spent hours troubleshooting the wrong problem because I didn’t understand what they were trying to tell me. This waste is a waste of your time and the poor person who is waiting for you to fix their problem. They don’t care about the time you just spent and how hard you worked. No one benefits.

Always. Restate. The. Requests.

It sounds stupid. It sounds awkward. Do it anyway. It’s a “check for understanding“. In conversation you can just say “I want to make sure I’ve got this right, so let me restate what you’ve told me.” That language usually helps them understand why you insist on saying back to them what they said to you.

Think yourself through the experience of asking for help. You are trying to get help. It’s bad enough that you can’t fix it yourself – so you are helpless. You want to get help – to ask for support. You ask for the support and hear that someone will help. Relief.. Then they tell you the house isn’t on fire – but you are seeing the flames! Or they say they’ve fixed the water leak, but you are seeing a waterfall in your bedroom. Are they insane or incompetent or lazy or lying? You will feel angry, frustrated or betrayed.

A script

“Hi Allie, I saw your request – I just to make sure I understand. You’re saying the floor robots are ignoring Asimov’s 3rd Law – that they are killing themselves out of boredom?”

“OH MY. Sorry, you’re saying that they are ignoring Asimov’s 1st Law and killing everyone on the floor out of boredom! I’ll send Will Smith over immediately. Please shelter somewhere safe.”

“Hi Carla, I saw your request to increase your hard drive size – you’re telling me that you have to use a program that only works with large files on your primary drive. You’ve hit the hard drive limits under our policy.  I’m attaching a quick guide to how to slim your hard drive footprint that other folks have found helpful. If that doesn’t work we can send a note to your manager and the CTO to see if they will approve an exception.”

Estimate how long until you will get back to them.

A key metric for user satisfaction is “First Call Resolution”. FCR makes people happy. For phone calls that means getting solved during a conversation. For emails and tickets, that means someone resolves the issue within 1 hour of the request coming in and while in the first response to the request. Yay – those are automatic wins!

But there are plenty of issues that are bigger than a single call or take more time. This is where real rancor and unhappiness can develop even if you are solving problems and doing everything else right.

Tell people how long until you will get back to them. This isn’t a guarantee that you’ll have the problem fixed – just a message of how long they should expect until you can update them. This is very important because it gives them a chance to make decisions and be empowered. Just telling them this lets them be less helpless.

If you think this reporting issue will take 1 hour, and someone has to be on the phone with a client in the next 20 minutes, they can decide to calculate the numbers themselves or to postpone the call. Letting someone know enough to seek other solutions is good for them.

An estimate is basically a promise to the user: You can bug me if I don’t come back to you before this time. If that isn’t soon enough you can seek other solutions. If it is ok, just relax and work on something else until I call you – I’ve got this.

A script

“Hi Zelda, I’m not sure why the delivery drones are returning to the warehouse without releasing the payload. Just to be clear – the address is correct, the payment is fulfilled and they order hasn’t been cancelled, but the drone just goes to the address, circles, and comes back with the package? This is tricky. I know you are under a time crunch. I’ll start looking at this and I’ll get back to you by 2pm with either a solution or I’ll let you know how much longer it will take me.”

Meet or beat your estimate.

When you make an estimate, you’ve made a promise. Keep your promises.

Don’t break your promises. Do what you say. Be clear. Meet or beat your estimate. That’s why your estimate includes a statement saying you might come back with another estimate. Do that if you think it might take longer than your original estimate. Better to keep the user in the loop and up to date than to have them know that they can’t trust you.

A script

“Hi Zelda – Just calling to check in with you. This is trickier than we thought. The GPS system connected to the drones doesn’t show the address! That’s something we are working with a vendor to correct, but it won’t be fixed by 2pm. I’ve asked them to keep us up to date and I’ll chase them. I’ll get back to you by 4pm with either good news or I’ll get you a better estimate.”

Conclusion

The reason this works is simple: it’s humane, it respects the needs of the person asking for help and it treats them with good manners. It won’t solve problems for you – you’ve still got to do that, but it will get rid of feedback that your people feel abandoned by your support team or that they don’t want to bother reporting issues.

Backing up a SalesForce instance

Dev, Python, ,

SalesForce is an interesting beast. You gotta work within its limits, and it is great within them. As soon as you want to venture outside of the normal flow, things get complicated.

They suck  into SalesForce, but never out – it’s designed as a lobster trap for your information.

Weirdly, there’s not much on the SalesForce AppExchange that helps you easily back up your data on site. There are some tools that help you easily back up to another cloud, but little that helps you get your data back within your own walls.

Still, there’s a little layer over the SalesForce API in Python called simple-salesforce. Here’s a quick script I threw together to help put all your data into csv files.

I love PETL

Dev, Python, ,

When I started at my current job I noticed we we had lots of room for improvement about how we imported and exported data.  Folks had been using the MicroSoft SSIS platform as a way to Extract, Transform and Load data in and out of our database to various files.

SSIS is great for lots of things and has a lot of upsides. It is very drag and drop, folks don’t have to know a lot of programming to get it to do things, and it has lots of functions built in. If you need more programming power, you can execute C# or VB scripts to do the fiddly bits.

But I hate it. ( Don’t worry, we’ll get to the love soon.)

My biggest problems with SSIS:

  • It is unversionable. Try reading a git diff of an SSIS change. The xml is designed for a machine to read, not a human. If you want to know what has changed over time in your world, it’s a problem.
  • You can only use Visual Studio to edit it. Many of our SSIS packages include VB or C# scripts. That sounds fine – but apparently these compile to an undiffable, uneditable blob in the xml that is only recompiled if you save using visual studio. So if you want to change something across many SSIS package scripts, you have to open and resave each one.
  • It hides options under rocks. Finding out how something works requires lots of delving into lotsa windows and dialogues.
  • It changes things unexpectedly. Click in the wrong dialogue and it helpfully re-infers datatypes from a file for you. You don’t know until you go to execute.
  • It slapped my momma. Etc.

I wanted to move my team to something that was better for people.

We need something:

  • That we can diff
  • That we can do code reviews and pull requests on
  • That is simple, expressive and clear.
  • That is powerful.

To me that sounds like a programming language.  I encouraged folks on the team to try accomplishing a couple of tasks that might use an SSIS package instead to use Python. Immediately, things got better. Our code reviews made sense. Code quality improved with every single pull request.

We used pymssql to connect to SqlServer and inserted records as needed after processing them. Navigating and transforming XML docs was easy, CSV files were eaten up by the native DictReader.

And then Derrick found PETL. It’s beautiful. You point it at data and make simple moves to completely transform it. I’m smitten.

I had dozens of files to read from, each a quarterly file for a year – only noted in the file name. Each had a crappy heading line that preceded column headers. I needed to put them into 1 file for loading into SalesForce Wave. Whacking together a solution with PETL was effortless. Line 36 is where the PETL starts, and it’s so small and good that it is nice to see how much it encapsulates.

Quick Project Names Demo

Dev, Javascript, ,

At work, I’m trying to convince people that we should auto-generate at least a suggested code name for our project names. It’s an important thing for compliance and secrecy. You’d rather someone is overheard talking in the elevator about “Project Icy Gneiss” than about “the restructuring of Acme Corp”.

I wanted to make the point that if you just have a small list of adjectives and nouns you quickly get a vast space of possible names – more than we’ll exhaust.  But a working demo is more persuasive than logic.

I knocked this together last night: Projects-a-Plenty.

projects-a-plenty

Used bootstrap & angular which is kind of overkill on something this tiny.

Mistakes were made, tears were shed

Orbital 0.2.2 – Mistakes Were Made

Dev, Projects,

 

This is a tweak to fix some problems people told me about on the Orbital Feed Reader GitHub Issues page.

Some folks were getting a very bad experience during the install – everything goes blank – or they were able to install and then couldn’t add feeds or read feeds. Not good for a feed reader!  I traced down a problem where if you had a specific new default collation on your database, the way I initialized the database tables made everything go boom.  It’s fixed now, but it’s part of the problem of supporting a lot of different configurations without a giant automated test lab.

The other was that someone pointed out there is an improved PressThis page in the latest version of WordPress.  Everything was working, but you’d get a nudge message saying that there was a new way to use PressThis and why are you going this old way?  Updated and I think it’s a really good experience now!

I also did some cleanup and logging improvements so that I can work with folks who have problems in a better way.

From Jekyll to Hugo

Dev, , , , ,

Sometimes simpler is better.

After being away from Orbital for a bit, I started revving my engines a bit by trying a new project on the subway and hacknight. Project Munchausen is about doing some writing and exploring some ideas – so I tried to get myself away from doing too much programming on it. Since it is just writing, I wanted to avoid falling into the trap of using a big content management system like WordPress.

I started exploring with the first Static Site Generator I know of: Jekyll. Jekyll powers the Pages feature on GitHub. I got really far, but felt really constrained. It felt like Jekyll was trying to push me to turn my site into a blog, like the downhill path of Jekyll was to create a blog.

I didn’t want to create a blog – I want to make something that’s an exploration of a subject. It will change over time, so it should have an RSS feed generated, but I didn’t want to only have a timeline view of things.

I don’t know how I happened on Hugo but it seemed perfect. The templates are simple, Go is a pretty simple language to use, and the organization of the content is really easy to understand.

In Hugo, you have a directory named content. Under that directory, any other directories are Sections.  Content goes in a section as a Markdown file. Templates can iterate over those sections and their contents.  Easy peasy.

So far, I’ve gotten farther with Hugo in a few subway rides than I have with long sessions wrangling Jekyll. My goal is to put something up on the web in a month.

Orbital Feed Reader 0.1.9 Codenamed “BORN FREE” is out!

Dev, Software, wordpress, ,

Last night I pushed out the final changes to uncage version 0.1.9 and get it out in the wild.
MAX LAZER tears off on his own to explore Oregon

This version is mostly about giving you freedom. You should be able to easily import your stuff from other platforms into Orbital Feed Reader to use it. If you decide that you don’t want to use Orbital Feed Reader any more, you should have an easy way to leave. Any software that doesn’t tell you how to leave it is something you should be suspicious of.

Get Feeds IN

Share your OPML

For Feed Readers, the standard way to exchange lists of subscriptions is a standard called OPML. It’s an old format, but it’s how they all exchange lists of feeds. I’ve buffed up the OPML import to handle larger numbers of feeds. After you import the file you have a chance to look over the feeds and edit them, tag them or exclude them entirely. Hit save and they’ll all flow into the backend.

 

Get Feeds OUT

But we also don’t want Orbital to be a lobster trap for you. If you import that kind of data I also want you to be able to export that kind of data. Now Orbital is good at exporting your feeds. If you are logged in – we’ll export a file with all your public and private subscriptions and let you take that wherever you want. Some folks want to share their OPML publicly – if you aren’t logged in, hitting the same URL gives you only the public feeds. All your private stuff should stay safe!

Read Lotsa Feeds

Now that Orbital can handle tons of feeds coming in I wanted it to be easy to actually sort through them. Feed Tags now start off collapsed and you can expand them anytime by using the little twiddlers next to each. That’s a technical term. The twiddlers make it much more manageable to see 200 feeds over many subject areas.