JavaScript in the Data Center?

First I saw that PayPal was using Node.js for production services.  Then Airbnb.  What the hell is going on?

KrakenLogo

Well, a pattern is emerging:

  1. A new product or service team starts operating in lean-startup-style.
  2. This means rapid-prototyping so you can have quick build/measure/learn cycles.  There are a many choices for a rapid-prototyping stack, including: Ruby-on-Rails, Django, Node.js (and many more).  [Notice that Java/Spring/JSP – the current Data Center stack champion – is not one of them.]
  3. After piloting and iterating on the product/service, it’s blessed (usable, feasible and valuable) and it’s time to scale!  Now a decision is necessary: rebuild in the current de-facto enterprise stack (e.g. Java), or reuse/refactor the pilot code?  Well you can certainly reuse CSS style definitions, and if you used client-side templating then maybe you can re-use that, and then the inevitable question keeps poking it’s head: “Why not just use the pilot code/stack as the basis for the product?”

Now it’s very true that during build/measure/learn phase the prototyping engineer was optimizing for developer time.  That means the goal of the code was to learn about the problem and solution – not to follow best engineering practices.  As a consequence:

  • Not all use-cases / features have been implemented
  • Test coverage may be poor or non-existent
  • The code and database queries have not been optimized, and have been written for speed of development, not speed of execution
  • It has not been internationalized / localized / translated
  • It doesn’t necessarily support all required browsers / devices
  • There may be orphaned chunks of code
  • etc

But what if you bake-in as many best-practices into your prototyping stack?  Have a boilerplate rapid prototyping app with:

  • centralized logging (e.g. sumologic)
  • performance monitoring and analytics (e.g. statsd)
  • a deployment framework that can deploy the rapid-prototyping stack with load-balancing, auto-scaling, zero-downtime-deployment, etc. (e.g. aws, cloudformation, etc)
  • ready-to-go for internationalization
  • cross-browser-libraries loaded (e.g. jquery, etc).
  • build on a responsive-design framework (e.g. bootstrap3, etc)

There will still be orphaned chunks of code, un-optimized data access and manipulation patterns, strings to be translated, etc.  This is all manageable.  The biggest remaining hurdle to using the pilot-codebase as the foundation for the product is the technology choice, and the objections:

  • “That RoR/Django/Node.js stack is unreliable”
  • “It doesn’t scale”
  • “Who’s going to maintain it?”
  • “We can’t have 1,000,000 stacks in our services architecture”

Since there are examples of RoR/Django/Node.js stacks in production for large-scale products/services I think the first two objections are FUD.  But the second two are more legitimate.  And here’s what I’m seeing: companies are choosing and standardizing on a second-stack.  They’re opening up the service-architecture and data center and letting that stack play with the other kids.

I used to think that the momentum was in favor of Python / Django, but that was probably  due to personal experience and bias.  Now I think that Node.js might be the one.

FYI: Here’s that Bill Scott / PayPal presentation (from O’Reilly Fluent Conf 2013):

here are the slides (slideshare).

 

BONUS GIFT: Here is another great presentation on Node.js at PayPal.

 

Posted in Uncategorized | Tagged , | 1 Comment

Goodbye Bazaarvoice, Hello Indeed

After 3 amazing years at Bazaarvoice, it’s time for me to move on.  My last day was Oct 31st.  See how R&D dressed up to say goodbye! …

bvhalloween

Bazaarvoice has been very good to me.  For the past 3 years I’ve had a series of dream roles: lead engineer in the Bazaarvoice Labs, manager of the Bazaarvoice Labs, then off to London to start and lead the UK engineering group.  After relocating back to Austin it seemed sensible to be open to new opportunities.  Making a move would only happen if there was something outstanding that aligned with my passions: engineering leadership, people leadership, and product management.  There aren’t a lot of roles that require all three but I’m delighted to say that my new job does – today was my first day as Director of Indeed Labs.

indeedlabs

Like at Bazaarvoice, I’ll be surrounded by great people, solid technology and lots of opportunity.  The thing that particularly excites me is working at a direct-to-consumer (B2C) company whose mission is to help people get jobs.  What a great mission !

Indeed is growing fast.  We’re looking for great Dev Ops, SW Developers, UX Designers and more.  If you’re interested in an impactful role at a great company that’s helping people every day, get in touch.

 

Posted in Uncategorized | 1 Comment

Seek contact with reality, early and often

There was an interesting question on Quora the other day: What are the early symptoms that a startup is going to fail?

There were a variety of answers, some funny, some insightful, some both:

Screen Shot 2013-11-06 at 6.30.32 PM

William Pietri provided a superb answer that is worth reading in full.  Here’s his top 10 reasons/symptoms a startup is going to fail:

  1. No demonstrated user need.
  2. Fear of testing hypotheses. 
  3. No love for the audience.
  4. No love for the domain.
  5. No love for the team.
  6. A desire for perfection.
  7. Not thinking about revenue.
  8. Caring too much about what other people think.
  9. Being in it for the wrong reasons.
  10. A high Dunning-Kruger quotient.

One piece of wisdom that particularly resonated based on personal experience <sigh>

“If a startup team doesn’t seek contact with reality early and often, they will have a bunch of surprises in store. It hurts to find out your ideas are dumb, so you have to really want to know the truth more than you want to feel comfortable.”

 

Posted in Uncategorized | Tagged | Leave a comment

Be Vulnerable

Screen Shot 2013-10-29 at 2.25.13 PM

Shame on me.  I haven’t watched Jack Dorsey present before (live or on video), and I had made certain assumptions about him because of his business success (co-founder of Twitter, founder and CEO of Square) and because he was (in)famous for working full-time at both of them for a bit.

Well I was surprised and impressed by what he presented at Startup School 2013.

What I especially love about this talk is that towards the end he openly shares the personal development items that he’s working on (his list of Daily Do’s and Daily Don’ts).  He has a note on his phone that he checks every morning, periodically during the day, and at night.  Here are some of the items on the list:

Daily Do:

  • Stay present
  • Be vulnerable
  • Drink only lemon-water and red wine
  • Exercise (Squats, Pushups, Planks & Run)
  • Stand up straight
  • Say hello to everyone
  • Do a video journal entry
  • Get 7 hours of sleep

Daily Don’t:

  • Don’t avoid eye contact
  • Don’t be late
  • Don’t set expectations and fail to meet them
  • Don’t eat sugar
  • Don’t drink hard liquor or beer on weekdays

The “be vulnerable” really stuck out for me.  It’s a necessary piece for having real relationships outside of work, and for having high-trust relationships within work.

A quick story: In the past I really struggled with what to do about a particular member on my team who wasn’t excelling the way he could, and the issue boiled down to his need to maintain  emotional barriers between himself and others.  My coaching advice was centered around encouraging him to allow himself to be a little more vulnerable.  It felt odd at the time to be talking about vulnerability, but now that I have a little more experience I really think that this barrier would impact his career, and that this was sound advice.

So I’m glad to see an extremely accomplished CEO sharing something as touch-feely as the importance of being vulnerable to an audience of aspiring entrepreneurs.  And the rest of the items on the lists are great too.

 

 

Posted in Uncategorized | Tagged | Leave a comment

Can I explore your API in a browser address bar?

I like this best practices guide to designing APIs by Vinay Sahni.

In it he offers some excellent advice, but one particular design principle is worth heeding and has lots of implications, namely the API should be “explorable via a browser address bar.”  This means several things (some of which he mentions, some of which he doesn’t):

  • Major versioning in the url (api.xx.com/v1/)
  • Use SSL/HTTPS with apikey params (or basic auth)
  • Nicely formatted JSON output
  • Content-Type overriding via resource extension to the url (/users.json)
  • Pagination in query params (offset & length)
  • Overriding HTTP methods (…&_method=DELETE to delete from browser)

Vinay does make some recommendations that preclude exploring the API in a browser, e.g. use JSON as the body content when creating resources with a POST.  But there aren’t alternatives for POSTing content from a browser anyway, and using GET to create resources is bad bad bad.

If you do want to stay in the browser when POSTing to an API, then there are two great Chrome extensions you should check out:

Remember:

An API is a user interface for developers. Put the effort in to ensure it’s not just functional but pleasant to use.

There are some other great resources for API design considerations.  I recommend checking out:

Posted in Uncategorized | Tagged , , | 1 Comment

Hire people you would be willing to work for

The Y Combinator Startup School 2013 videos are online!

Screen Shot 2013-10-27 at 11.08.54 PM

I started with Paul Graham’s interview with Zuckerberg.  There’s not a ton of new info in that video IMHO, but there are a couple of gems, such as:

“One of the reasons I think great stuff gets built is because it’s kind of irrational at the time, so it selects for the people who care the most about doing it.”

This aligns with what Facebook Director of Engineering Jocelyn Goldfein said in an interview at Stanford:

“Innovative ideas by definition look like bad ideas.  If they looked like good ideas they would be obvious ideas.  And so to be innovative, to be un-obvious, something about them has to look stupid, or dumb or impossible.”

The other very interesting part of the discussion was around management / leadership.  Paul asks Mark “How did you learn how to manage people?”  and the answer is interesting (beyond the “Make and learn from mistakes” part)…

“Hire good people.  But what’s the right heuristic to figure out if someone’s really good?  – Would you work for that person?

I believe that.  If you look at my management team today – if we were in an alternate universe and I hadn’t started the company it would be an honor to work for any of these people.  And if you build a company that has those kind of values then I think you’ll build a pretty strong company.”

This is the second time in a week that I’ve come across that advice regarding hiring.  I’ll have to keep it in mind next time I interview someone for a management role.

I think it mainly applies for hiring management, but it’s an interesting question: what’s the equivalent advice for hiring engineers?

 

Posted in Uncategorized | Tagged | Leave a comment

Do the right thing, Wait to get fired

I stumbled upon this bit of wisdom in Team Geek: A Software Developers Guide to Working Well with Others, and it resonated.  It comes from Google engineer Chade-Meng Tan:

Do the right thing, wait to get fired.

New Google employees (we call “Nooglers”) often ask me what makes me effective at what I do.  I tell them only half-jokingly that it’s very simple: I do the Right Thing for Google and the world, and then I sit back and wait to get fired.  If I don’t get fired, I’ve done the Right Thing for everyone.  If I do get fired, this is the wrong employer to work for in the first place.  So, either way, I win.  That is my career strategy.

This requires you to have confidence in your judgement, to assume authority and responsibility, to make decisions, to take risks – in other words to do what you have been hired to do!

You’ve got to recognize that companies are schizophrenic.  They build process, and rules, and structure and they ask you to follow them:

  • “Employee evaluation is done this way…”
  • “The rules on conferences attendance are as follows…”
  • “Products are to be QA’d and deployed in this manner…”
  • “New projects require the approval of the following people…”

In general these rules are better than the alternative – no guideposts or structure.  They help new managers and teams to function effectively.  They push employees to do things in a good way.

But greatness rarely happens by following rules, process and structure.  That is why companies also want to find employees ready to take risks, make decisions, try new things, move fast and even break things.

This means recognizing when the process is too heavyweight – and a simpler alternative is better in this case.  It means making the call to assume some technical debt or operational risk because getting this new product/feature out to some real customers for feedback now is most important.  It means approving an over-budget trip for a particular employee to a conference they’re passionate to attend.  It means setting aside some time to work on your idea for a new tool that will help the support team to diagnose customer problems.  It also means slowing down to refactor something – even if it means hurting your reputation for getting things done quickly.  You do this because it’s the right thing for your team, your company, and/or your customers.

When you break rules, and do what you think is the right thing you are taking a risk.  Sometimes this will pay off, sometimes it wont.  It’s ok to fail – but try hard not to fail repeatedly at the same thing or for the same reasons.  Know when you’re taking a risk, and learn from the outcomes.  The best engineers and managers I’ve known have all been willing to break rules and take risks.  You should too.

Update:

Follow the discussion of this article on Hacker News or the discussion of this article on Reddit.

Posted in Uncategorized | 63 Comments

Fairness Matters

cucumber-grape-monkey-experimentIf you haven’t seen Frans De Waal’s TED talk on Moral behavior in animals you should.  One of the best parts is the Capuchin monkey fairness experiment video, where we see that capuchin monkeys – like people – aren’t rational (in the classical economic sense), but instead express outrage at outcomes that are mutually beneficial, but unfair (i.e. when someone else benefits even more for no apparent reason).  In this case the monkeys are happy to do a little work (fetch a rock) for a cucumber treat – until they see their neighbor getting a grape for the same work.  At which point they refuse to participate and throw the cucumbers at the boss experimenter.

In the workplace fairness is particularly important in compensation, and work assignments.   Happiness with salary is mostly relative – in the software field salaries are extremely good – and most developers would still be happily coding even if salaries were 20 or even 30 percent lower than they are.  But if a developer feels that they are being treated unfairly with regard to compensation – if someone in their team is getting 25 percent more for doing the same job and performing at the same level – then expect them to be updating their LinkedIn profile pretty quickly.

barbara-smaller-o-k-if-you-can-t-see-your-way-to-giving-me-a-pay-raise-how-about-givin-new-yorker-cartoon (1)

Source: Barbara Smaller, New Yorker Magazine

But it’s not just fairness of outcomes you have to worry about.  In Fair Process: Managing in the Knowledge Economy, W. Chan Kim and Renee Mauborgne point out the importance of fair process – fairness in the way decisions are made and communicated.  They define fair process as consisting of three principles:

  1. Engagement – reaching out to stakeholders for input, having them participate in the decision making process where possible.
  2. Explanation – explaining the reasoning behind decisions.
  3. Expectation Clarity – communicating clearly the new expectations that result from decisions and change.

I’ve experience this firsthand as a software developer.  At one company where I worked I didn’t even know we had been through a merit cycle – or how it impacted me – until I discovered (a few months after the fact) that I had been granted some new stock options.  I couldn’t complain about the outcome – I wasn’t expecting anything at the time – but the lack of clarity about the process was damaging to trust and goodwill.

I’ve also witnessed this as a manager.  With good intentions and the business goals foremost in mind, I have seen technical leadership get together and completely rearrange the development teams to execute on new goals.  The problem is that too little reaching out to the developers was done, and even the explanation / context setting was minimally communicated.  You don’t need to achieve consensus, or make decisions democratically – but you do need to push down decision-making as low as possible, listen to stakeholders and over-communicate the business context and rationale for decisions.  This isn’t the industrial age – we need full engagement by all team members, so we have to work to engage everyone when forming decisions.

Fair process responds to a basic human need. All of us, whatever our role in a company, want to be valued as human beings and not as “personnel” or “human assets.” We want others to respect our intelligence. We want our ideas to be taken seriously. And we want to understand the rationale behind specific decisions. People are sensitive to the signals conveyed through a company’s decisionmaking processes. Such processes can reveal a company’s willingness to trust people and seek their ideas—or they can signal the opposite.

So how would fair process be followed in merit cycles?  First of all managers should ensure their team knows when it happens, and how it happens.  Like most compensation communication, this is best handled 1:1.  Then make sure the employee has the opportunity to make their case – ask them to prepare a description of their accomplishments during the time period, and discuss it together.  If you have to assign a rating to the employee as part of the process, be sure to share that rating with the employee – even if it means a difficult conversation about why they are “meeting expectations” instead of “exceeding.”  Then share with the employee 1:1 the resulting decision about any merit/bonus compensation.  If the employee is disappointed, make sure you communicate what kinds of behaviors / activities / impact they should aim for in the next time period.

With regard to reorganizations, first it’s important to accept that there’s no clean way to do it, and that you cannot resolve all anxieties.  However, there are better and worse ways to enact change.  It’s necessary to limit the decision makers, and at the same time take time to listen and solicit ideas from a broad-base.  Groundwork needs to be done, then the context for changes needs to be presented, in-person, with plenty of time for answering questions and addressing concerns – and be sure to provide any promised follow-up answers promptly.  It may seem like this takes a lot of effort and will add unnecessary time – but in the full-context of enacting change, this up-front effort pays dividends in morale, trust, and willingness to adapt and execute on the change.

Fair Process is good for business.

Further Reading / Viewing:

Posted in Uncategorized | Leave a comment

Inside Facebook with Jocelyn Goldfein, FB Engineering Director

jocelyn.goldfeinA friend pointed me to this frank and insightful talk by Jocelyn Goldfein, Engineering Director at Facebook.  In this hour-long interview at Stanford she shares her perspective on engineering culture, and a few stories about how things are done at Facebook.  Here are some of the things I found noteworthy…

Aim for impact, and your career takes care of itself:

“There was never a moment when I walked into my boss’ office and said ‘where is my next promotion coming from?’.  I was always just thinking ‘how can I have the most impact I can have on the company, on my team, on our users?’.  And I think that by just trying to have impact my career kind of took care of me.  I never really thought of myself as consciously managing my career.”

You must make sacrifices to reinforce your values:

Culture arises from so many small things.  ’Culture is the behaviors that you reward and punish.’  You’ve also got to show people what behaviors you want.  …   The concrete floors [at Facebook] show that we’re not finished.  The open space is another huge one …  You know if you program you need flow time.   The idea of having programmers in open space is controversial.  For many years the gold standard was offices.  Why would we sacrifice engineer productivity?  It’s not to save money.  It’s because one of the key values of Facebook is to be open.

What happens when you really break something:

I wanted to try automating bug triage.  I called it ‘The Task Reaper’.  I wasn’t careful enough with my testing.  I accidentally ran the test on live data.  It pinged 14,000 bugs, and sent 14,000+ emails.  I basically launched a DOS on our email infrastructure.  It also brought email down.  I was kind of terrified.  There was definitely a vocal company response.

They just got in the trenches and fixed things. … No one acted like I didn’t have the right to try. …  It was like a lightbulb went off in my head.  This is what lets this company still innovate. … We are willing to take risks, we are willing to face up to the consequences of failure if it was in the spirit of trying for something, of trying to innovate.  It’s not that you can come to Facebook and be incompetent or do things wrong all the time … there’s a lot of feedback and actions if that’s happening.  We expect you to deal with the consequences of your failures, but we also rally and have your back when you fail. And we expect you to rally and have our back when we fail.

On Innovation:

Writing “Be Bold” on the wall is one thing – I think that’s helpful – but actually seeing that you can try things and not have them work out and still thrive definitely contributes. What I tell BootCampers during on-boarding is Innovative ideas by definition look like bad ideas.  If they looked like good ideas they would be obvious ideas.  And so to be innovative, to be un-obvious, something about them has to look stupid, or dumb or impossible.

…  

As a company you’d rather be trying 10 things and having 9 of them fail and have one of them be a 20-fold success – that’s just good ROI for anybody who owns the portfolio.  The problem is that it’s bad ROI for the actual individuals on the 9 things that failed.  And if you’re a human being I really think the biggest thing that kills innovation is not that companies suddenly wake up one day and say ‘I don’t want to innovate’, companies all want to innovate, they want their employees to be bold and to try risky things, the problem is innovative ideas are going to fail at a pretty high rate.  And if you’re a human you hate failure.  You are going to try something, … have a horrible experience, and not try it again.  Maybe if you have exceptional grit you will try two things that fail.  But you have to be willing to try 10 things in a row that fail if you’re going to be a successful entrepreneur, and I do think that is what distinguishes successful entrepreneurs is that particular brand of insanity that is ready to keep trying and keep trying in spite of failure and keep believing in yourself.  But ordinary humans do not.  And so i think you just have to provide incredible cultural back pressure to support people in that instance of failure if you want them to keep trying crazy ideas.

On Permission:

It’s a try/catch model of decision-making, not an if/then.

On Mark Zuckerberg’s style as CEO:

Mark has organized the company so he can spend the bulk of his time on product and product strategy, and he has very able lieutenants so that allow him to do that.  He has set up his calendar so that each day of the week has a theme and the theme is one of the departments – so one day might be mobile, and one day might be platform, etc.  There will be a block of four hours and teams rotate through that block and just present stuff to him and talk stuff through with him.  And its amazing – because he’s probably the most gifted product thinker in the company and maybe in the valley … certainly in this space.

But he can only pay attention to so many things at a time too, and so if he’s not paying attention he doesn’t expect you to sit around and wait around for him.  He expects you to run forward while he’s not looking, and if he comes back and finds you havent moved from where he left you he’ll be pretty disappointed.

Posted in Uncategorized | 3 Comments

The Signal and the Noise

the-signal-and-the-noiseI recently finished Nate Silver’s The Signal and the Noise, and frankly I’m disappointed.  There were surprisingly few insights to be found in the 450-odd pages; I only highlighted two passages – an all-time low – and remarkable as I like to read with a highlighter in hand.

On the positive side the writing is clear, and the few technical discussions are easy to follow.  Silver includes some interesting stories as he covers the history and reliability of predictions in finance, sports, politics, weather, gambling and economics.  The general conclusion is that the predictions are not very accurate.  The one optimistic tidbit is that we seem to be getting better at predicting the weather.  

His main advice appears to be: Practice Bayesian Thinking [this showed up in Nassim Taleb’s Fooled by Randomness also].  I find it hard to really understand what that means in practice – and when I do look into it I tend to find myself on the side of the frequentists.

At best I’ll chalk it up to unmet high expectations – I was familiar with his outstanding reputation with predictions in Baseball and Politics.  Or maybe it’s because I read this just after reading Fooled by Randomness and found that to be very thought-provoking (although not as enjoyable to read).  Perhaps it’s because Silver is just so generally sensible and doesn’t throw out any controversial opinions or perspectives.   Either way I find myself unsatisfied.

Posted in Uncategorized | Leave a comment